Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 700456 - repoman: add a way to limit repoman to subset of profiles (similar to --include-arches= for arches)
Summary: repoman: add a way to limit repoman to subset of profiles (similar to --inclu...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Repoman (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-18 08:26 UTC by Sergei Trofimovich (RETIRED)
Modified: 2020-02-18 13:47 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
0001-repoman-add-include-profiles-PROFILES.patch (0001-repoman-add-include-profiles-PROFILES.patch,6.14 KB, patch)
2019-11-19 00:21 UTC, Sergei Trofimovich (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sergei Trofimovich (RETIRED) gentoo-dev 2019-11-18 08:26:26 UTC
I'm trying to get single repoman run to fit in 50 minute budget on travis.ci for ::haskell overlay. Today I'm sharding by --include-arches=.

This brings scan times down to 8-13 minutes per arch:
    https://travis-ci.org/gentoo-haskell/gentoo-haskell/builds/613243862
with a notable exception of amd64, which still does not fit into 50-minute budget.

I suspect it's because amd64 has 3x more profiles compared to other arches.

I'd like to shard it further: run repoman per each individual profile.

Would it be feasible to implement profile filtering similar to --include-arches=?

Something like --include-profiles=default/linux/amd64/17.0/no-multilib.
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2019-11-19 00:21:56 UTC
Created attachment 596746 [details, diff]
0001-repoman-add-include-profiles-PROFILES.patch
Comment 2 Larry the Git Cow gentoo-dev 2019-11-20 10:30:12 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e9bb1e9681685f4e4d7174f51751356fd3f67d0c

commit e9bb1e9681685f4e4d7174f51751356fd3f67d0c
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2019-11-19 00:21:13 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-11-20 10:29:14 +0000

    repoman: add --include-profiles=PROFILES
    
    repoman slows down ~linearly with amount of profiles being scanned.
    In case of amd64 we have 28 stable profiles.
    
    To speed up processing and fit into time budged of various CIs we can
    split the work across different processes that handle different profiles.
    
    Example benchmark on ::haskell overlay:
        $ ./repoman full --include-arches=amd64
        ~65 minutes
        $ ./repoman full --include-profiles=default/linux/amd64/17.0
        ~4 minutes
    This allows for a crude sharding of work across processes and allows for
    cheap tree-wide scans for early failures.
    
    Bug: https://bugs.gentoo.org/700456
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 repoman/lib/repoman/actions.py                      | 4 ++++
 repoman/lib/repoman/argparser.py                    | 7 +++++++
 repoman/lib/repoman/modules/scan/depend/__init__.py | 3 ++-
 repoman/lib/repoman/modules/scan/depend/profile.py  | 9 +++++++--
 repoman/lib/repoman/scanner.py                      | 5 +++++
 repoman/lib/repoman/tests/simple/test_simple.py     | 1 +
 repoman/man/repoman.1                               | 4 ++++
 7 files changed, 30 insertions(+), 3 deletions(-)
Comment 3 Larry the Git Cow gentoo-dev 2019-11-27 04:21:33 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0af2015bc57a267f5f67610a73bde5da16bcc6d4

commit 0af2015bc57a267f5f67610a73bde5da16bcc6d4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-11-27 04:18:16 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-11-27 04:19:23 +0000

    app-portage/repoman: Bump to version 2.3.19
    
     #667432 Rename DCO_SIGNED_OFF_BY config variable to SIGNED_OFF_BY.
     #700456 Add --include-profiles=PROFILES
    
    Bug: https://bugs.gentoo.org/667432
    Bug: https://bugs.gentoo.org/700456
    Package-Manager: Portage-2.3.80, Repoman-2.3.19
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 app-portage/repoman/Manifest              |  1 +
 app-portage/repoman/repoman-2.3.19.ebuild | 63 +++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)
Comment 4 Sergei Trofimovich (RETIRED) gentoo-dev 2020-02-18 13:47:09 UTC
Using per-profile charding allows me to shrink scan time down to 5 minutes per profiles:
    https://travis-ci.org/gentoo-haskell/gentoo-haskell/builds/651819453?utm_source=github_status&utm_medium=notification

Thank you!