In catalyst builds, we update the stage3 before using it to build a new stage1. Updating the entire stage is expensive and unnecessary (since we're going to build the latest packages in stage1 and then rebuild everything in stage3). What we definitely do need to update in the original stage3 however, is any package that would trigger a subslot rebuild. For example of why the seed needs to be updated, in the past a new version of dev-libs/mpfr caused builds to fail in stage2. The seed stage3 had mpfr SLOT="0/4", so the sys-devel/gcc it built and installed into the stage1 was linked against libmpfr.so.4. But it built and installed mpfr with SLOT="0/6" into the stage1, leading to gcc invocations in stage2 to fail. Our current options to update the seed are too large a hammer (e.g., "--update --deep --newuse @world" or "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc") and spend too much time updating seed stages for no gain. What I would like is a portage set that contains packages that would trigger subslot rebuilds if upgraded. Is that doable?
Oh, and one other thought: making the set work with --ignore-built-slot-operator-deps y would be preferable. We want to upgrade the packages but not actually trigger any rebuilds.
About half the time I spend doing MIPS catalyst stuff is manually rebuilding seed stages from the last set of stage3's I built to prepare for new stage 1's just to avoid problems similar to what Matt describes. Since gcc's build time seems to double every few releases, I try to skip gcc upgrades in the seeds unless it is a brand new major version, but that sometimes leads to surprises when I get to stage2 or stage3 that can sometimes add a week to my catalyst runs. Anything to smooth that out will help me quite a bit.
Our new set should select packages with new slot/subslot available, yes? That sounds easy enough.
Yeah, that sounds exactly right.
(In reply to Zac Medico from comment #3) > Our new set should select packages with new slot/subslot available, yes? > That sounds easy enough. With a little guidance, I'm interested in giving it a try. Could you say what you would do?
It will be similar to DowngradeSet in the sense that it iterates over installed packages and checks the best visible version for each slot. The diff is something like this: > diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py > index 52367c4a6..efc822dc3 100644 > --- a/lib/portage/_sets/dbapi.py > +++ b/lib/portage/_sets/dbapi.py > @@ -167,7 +167,7 @@ class VariableSet(EverythingSet): > > singleBuilder = classmethod(singleBuilder) > > -class DowngradeSet(PackageSet): > +class SubslotChangedSet(PackageSet): > > _operations = ["merge", "unmerge"] > > @@ -193,7 +193,7 @@ class DowngradeSet(PackageSet): > ebuild = xmatch(xmatch_level, slot_atom) > if not ebuild: > continue > - if vercmp(cpv.version, ebuild.version) > 0: > + if pkg.sub_slot != ebuild.sub_slot: > atoms.append(slot_atom) > > self._setAtoms(atoms)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=6c70596d6d6382160490bb16ef559ccfdd982fae commit 6c70596d6d6382160490bb16ef559ccfdd982fae Author: Matt Turner <mattst88@gentoo.org> AuthorDate: 2021-01-16 02:47:09 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-17 10:35:03 +0000 Add @changed-subslot package set This set is the upgradable packages for which the highest visible version has a different subslot than the currently installed version. The primary purpose of this feature is for use in catalyst builds. We update the "seed" stage3 before using it to build a new stage1. Updating the entire stage is expensive and unnecessary (since we're going to build the latest packages in stage1 and then rebuild everything in stage3). What we definitely do need to update in the original stage3 however, is any package that would trigger a subslot rebuild. For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc will link with libmpfr.so.4, but the latest version of dev-libs/mpfr will be built and libmpfr.so.6 included into the stage1. Since the old libmpfr.so.4 is not included in the stage1, gcc will not work, breaking subsequent stage builds. Our current options to update the seed are too large a hammer (e.g., "--update --deep --newuse @world" or "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc") and spend too much time updating seed stages for no gain beyond updating only packages for whom the subslot has changed. With this set, catalyst will likely use emerge @changed-subslot --ignore-built-slot-operator-deps y to update the seed stage. Thank you to Zac Medico for showing me how to do this. Bug: https://bugs.gentoo.org/739004 Signed-off-by: Matt Turner <mattst88@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> cnf/sets/portage.conf | 5 +++++ doc/config/sets.docbook | 8 ++++++++ lib/portage/_sets/dbapi.py | 39 +++++++++++++++++++++++++++++++++++++-- 3 files changed, 50 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b03adb22f0432ee992aab733ce5dfa13e9b6801a commit b03adb22f0432ee992aab733ce5dfa13e9b6801a Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2021-01-19 11:07:56 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2021-01-19 11:16:53 +0000 sys-apps/portage: Bump to version 3.0.14 #571126 KeyError during package moves "global updates" triggered by FEATURES=binpkg-multi-instance (same root cause as bug 765847) #739004 Add @changed-subslot package set #765847 Spurious package file renames during application of package moves with FEATURES=binpkg-multi-instance #766012 Copy on write when applying a package move to a binary package, so that the old version of a binary package will remain available until eclean-pkg deletes it Bug: https://bugs.gentoo.org/766117 Bug: https://bugs.gentoo.org/571126 Bug: https://bugs.gentoo.org/739004 Bug: https://bugs.gentoo.org/765847 Bug: https://bugs.gentoo.org/766012 Package-Manager: Portage-3.0.14, Repoman-3.0.2 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.14.ebuild | 268 +++++++++++++++++++++++++++++++++ 2 files changed, 269 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=b3f782a0725da0f4858cba3e887f973e324755c7 commit b3f782a0725da0f4858cba3e887f973e324755c7 Author: Matt Turner <mattst88@gentoo.org> AuthorDate: 2021-01-17 23:29:05 +0000 Commit: Matt Turner <mattst88@gentoo.org> CommitDate: 2021-01-23 16:19:07 +0000 targets: Update the @changed-subslot set by default In portage commit 1789fdf2ee81 (Add @changed-subslot package set) I added this: the set of upgradable packages for which the highest visible version has a different subslot than the currently installed version. Updating the entire stage is expensive and unnecessary (since we're going to build the latest packages in stage1 and then rebuild everything in stage3). What we definitely do need to update in the original stage3 however, is any package that would trigger a subslot rebuild. For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc will link with libmpfr.so.4, but the latest version of dev-libs/mpfr will be built and libmpfr.so.6 included into the stage1. Since the old libmpfr.so.4 is not included in the stage1, gcc will not work, breaking subsequent stage builds. Our current options to update the seed are too large a hammer (e.g., "--update --deep --newuse @world" or "--update --deep --newuse --complete-graph --rebuild-if-new-ver gcc") and spend too much time updating seed stages for no gain beyond updating only packages for whom the subslot has changed. Bug: https://bugs.gentoo.org/739004 Signed-off-by: Matt Turner <mattst88@gentoo.org> targets/stage1/chroot.sh | 2 ++ 1 file changed, 2 insertions(+)
