I'm trying to get single repoman run to fit in 50 minute budget on travis.ci for ::haskell overlay. Today I'm sharding by --include-arches=. This brings scan times down to 8-13 minutes per arch: https://travis-ci.org/gentoo-haskell/gentoo-haskell/builds/613243862 with a notable exception of amd64, which still does not fit into 50-minute budget. I suspect it's because amd64 has 3x more profiles compared to other arches. I'd like to shard it further: run repoman per each individual profile. Would it be feasible to implement profile filtering similar to --include-arches=? Something like --include-profiles=default/linux/amd64/17.0/no-multilib.
Created attachment 596746 [details, diff] 0001-repoman-add-include-profiles-PROFILES.patch
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=e9bb1e9681685f4e4d7174f51751356fd3f67d0c commit e9bb1e9681685f4e4d7174f51751356fd3f67d0c Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2019-11-19 00:21:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-11-20 10:29:14 +0000 repoman: add --include-profiles=PROFILES repoman slows down ~linearly with amount of profiles being scanned. In case of amd64 we have 28 stable profiles. To speed up processing and fit into time budged of various CIs we can split the work across different processes that handle different profiles. Example benchmark on ::haskell overlay: $ ./repoman full --include-arches=amd64 ~65 minutes $ ./repoman full --include-profiles=default/linux/amd64/17.0 ~4 minutes This allows for a crude sharding of work across processes and allows for cheap tree-wide scans for early failures. Bug: https://bugs.gentoo.org/700456 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> repoman/lib/repoman/actions.py | 4 ++++ repoman/lib/repoman/argparser.py | 7 +++++++ repoman/lib/repoman/modules/scan/depend/__init__.py | 3 ++- repoman/lib/repoman/modules/scan/depend/profile.py | 9 +++++++-- repoman/lib/repoman/scanner.py | 5 +++++ repoman/lib/repoman/tests/simple/test_simple.py | 1 + repoman/man/repoman.1 | 4 ++++ 7 files changed, 30 insertions(+), 3 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0af2015bc57a267f5f67610a73bde5da16bcc6d4 commit 0af2015bc57a267f5f67610a73bde5da16bcc6d4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-11-27 04:18:16 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-11-27 04:19:23 +0000 app-portage/repoman: Bump to version 2.3.19 #667432 Rename DCO_SIGNED_OFF_BY config variable to SIGNED_OFF_BY. #700456 Add --include-profiles=PROFILES Bug: https://bugs.gentoo.org/667432 Bug: https://bugs.gentoo.org/700456 Package-Manager: Portage-2.3.80, Repoman-2.3.19 Signed-off-by: Zac Medico <zmedico@gentoo.org> app-portage/repoman/Manifest | 1 + app-portage/repoman/repoman-2.3.19.ebuild | 63 +++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+)
Using per-profile charding allows me to shrink scan time down to 5 minutes per profiles: https://travis-ci.org/gentoo-haskell/gentoo-haskell/builds/651819453?utm_source=github_status&utm_medium=notification Thank you!
