repoman is mostly a cpu-bound task even with generated metadata. [sf] ~/portage/gentoo-haskell:time ~/portage/gentoo-qa/run_repoman.sh real 5m20.469s user 4m35.703s sys 0m43.763s [sf] ~/portage/gentoo-haskell:cat ~/portage/gentoo-qa/run_repoman.sh #!/bin/sh repoman full -d > repoman-QA-`date +%F-%T`.log Would be nice to make repoman use of --jobs/EJOBS variables to speed things down a multicore boxes. Thanks!
Given the current design, doing one fork per package ($CATEGORY/$PN) would make sense, since it would allow dependency match caches to be shared between multiple versions of the same package. We'd just have to split out the dependency checking part (which consumes the bulk of the cpu time) and run it from within instances of portage's ForkProcess class. The results could be sent back to the main process via a pipe, similarly to how our MergeProcess class sends back elog messages.
as workaround one could run repoman with sys-process/parallel until this is fixed.
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/8d1fd04c29ff87ff96a15731299aedb3 https://github.com/gentoo/portage/pull/606
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=89ceaeb6b4619dcddace5dc70a126c360ccdd8fe commit 89ceaeb6b4619dcddace5dc70a126c360ccdd8fe Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-17 05:18:21 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-17 05:32:14 +0000 sys-apps/portage: Bump to version 3.0.3 #448462 repoman: avoid "RuntimeError: Event loop is closed" after SIGINT with --jobs > 1. #699256 emerge: Remove deprecated --changelog option #699256 emerge: Add short -l option for --load-average #711174 SpawnProcess: Fix _cancel to cleanup PipeLogger quickly Bug: https://bugs.gentoo.org/733180 Bug: https://bugs.gentoo.org/448462 Bug: https://bugs.gentoo.org/699256 Bug: https://bugs.gentoo.org/711174 Package-Manager: Portage-3.0.3, Repoman-2.3.23 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.3.ebuild | 263 ++++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=09578270a9cdb7ad6ed533b4f3dec816c5ab8868 commit 09578270a9cdb7ad6ed533b4f3dec816c5ab8868 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-17 05:59:17 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-17 06:12:54 +0000 app-portage/repoman: Bump to version 3.0.0 #448462 Add --jobs and --load-average options which allow dependency checks for multiple profiles to run in parallel Bug: https://bugs.gentoo.org/448462 Package-Manager: Portage-3.0.3, Repoman-3.0.0 Signed-off-by: Zac Medico <zmedico@gentoo.org> app-portage/repoman/Manifest | 1 + app-portage/repoman/repoman-3.0.0.ebuild | 64 ++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=26524e5fbc8019e22d15765a35570df7c7a315ca commit 26524e5fbc8019e22d15765a35570df7c7a315ca Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-08-15 02:51:38 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-08-17 05:39:06 +0000 repoman: Add --jobs and --load-average options (bug 448462) Add --jobs and --load-average options which allow dependency checks for multiple profiles to run in parallel. The increase in performance is non-linear for the number of jobs, but it can be worthwhile (I measured a 35% decrease in time when running 'repoman -j8 full' on sys-apps/portage). For the -j1 case (default), all dependency checks run in the main process as usual, so there is no significant performance penalty for the default case. Bug: https://bugs.gentoo.org/448462 Signed-off-by: Zac Medico <zmedico@gentoo.org> repoman/lib/repoman/argparser.py | 9 ++ repoman/lib/repoman/modules/scan/depend/profile.py | 117 +++++++++++++++++---- repoman/man/repoman.1 | 9 +- 3 files changed, 116 insertions(+), 19 deletions(-)
I've got support to run the dependency checks for multiple ebuilds in parallel, but it needs some work because it's not showing a performance improvement: https://github.com/gentoo/portage/compare/master...zmedico:bug_448462_parallel_repoman
repoman support has been removed per bug 835013. Please file a new bug (or, I suppose, reopen this one) if you feel this check is still applicable to pkgcheck and doesn't already exist.
