The choices.sort(key=lambda x: (not x.all_installed_slots, len(x.slot_map))) code from bug 632026 is causing these perl-cleaner dependencies to prefer paludis in the catalyst stage1 build: || ( ( sys-apps/portage app-portage/portage-utils ) sys-apps/pkgcore sys-apps/paludis ) It prefers paludis because len(x.slot_map) is 1 for sys-apps/paludis, compared to 2 for ( sys-apps/portage app-portage/portage-utils ). The command that fails is for the packages listed in profile packages.build, which includes virtual/package-manager. So, catalyst never really makes a commitment to a specific package manager, relying entirely on || deps and virtuals.
While I'm sure you're on the right track, this doesn't seem isolated to that one case with the extra package count. On a fresh stage1 build that was working w/o conflicts or surprises a few days ago, there are 4 new choices being made, each causing some grief: 1) paludis picked over ( portage portage-utils) to satisfy virtual/package-manager 2) pkgconf picked over pkgconfig to satisfy virtual/pkgconfig, even though pkgconf requires an extra USE flag to get there 3) realpath picked over coreutils, to satisfy eselect, resulting in blocker 4) static-dev being preferred over virtual/udev, to satisfy virtual/dev-manager, resulting in bail-out in pkg_pretend
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/c4f76b6d6ea802be8c78ba5e968cf154 https://github.com/gentoo/portage/pull/244
Created attachment 515332 [details] pretend emerge calculation, simulating catalyst stage1, with patch applied (In reply to Ben Kohler from comment #1) > While I'm sure you're on the right track, this doesn't seem isolated to that > one case with the extra package count. > > On a fresh stage1 build that was working w/o conflicts or surprises a few > days ago, there are 4 new choices being made, each causing some grief: > > 1) paludis picked over ( portage portage-utils) to satisfy > virtual/package-manager > 2) pkgconf picked over pkgconfig to satisfy virtual/pkgconfig, even though > pkgconf requires an extra USE flag to get there > 3) realpath picked over coreutils, to satisfy eselect, resulting in blocker > 4) static-dev being preferred over virtual/udev, to satisfy > virtual/dev-manager, resulting in bail-out in pkg_pretend With my patch applied, I did a pretend emerge calculation in a stage4-amd64-minimal-20180118T214502Z chroot, and it looks good. There's no pkgconf, realpath, or static-dev issues showing up. The patch did solve the paludis issue.
With this patch, all 4 issues I cited have disappeared. 1-3 made the correct other choice, and for #4 it seems virtual/dev-manager dropped out of the deps entirely. Thanks!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=e2134e9f72a86734552bb67e9414a017cfc4ea51 commit e2134e9f72a86734552bb67e9414a017cfc4ea51 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-01-20 00:28:42 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-01-20 23:08:42 +0000 dep_zapdeps: prefer choices with fewer new slots (bug 645002) Prefer choices with fewer new slots, rather than choices with the lowest total number of slots. This fixes a case triggered by the catalyst stage1 build, where paludis was selected to satisfy perl-cleaner dependencies because that choice happened to have a smaller number of slots: || ( ( sys-apps/portage app-portage/portage-utils ) sys-apps/pkgcore sys-apps/paludis ) Bug: https://bugs.gentoo.org/645002 Fixes: 9fdaf9bdbdf5 ("dep_check: use DNF to optimize overlapping virtual || deps (bug 632026)") Tested-by: Ben Kohler <bkohler@gmail.com> pym/portage/dep/dep_check.py | 14 ++-- .../resolver/test_virtual_minimize_children.py | 85 +++++++++++++++++++++- 2 files changed, 93 insertions(+), 6 deletions(-)}
(In reply to Ben Kohler from comment #4) > With this patch, all 4 issues I cited have disappeared. 1-3 made the correct > other choice, and for #4 it seems virtual/dev-manager dropped out of the > deps entirely. > > Thanks! Great, thanks for testing!
I'm not sure how this worked on my first run, but on my second run it's still preferring static-dev to satisfy virtual/dev-manager. In 2.3.13-r1 it did select eudev, but I'm not entirely sure there's a portage bug to fix here-- do we just need to be more explicit about our udev preference here? Strictly counting dep complexity, static-dev is the easier choice..
(In reply to Ben Kohler from comment #7) > I'm not sure how this worked on my first run, but on my second run it's > still preferring static-dev to satisfy virtual/dev-manager. In 2.3.13-r1 it > did select eudev, but I'm not entirely sure there's a portage bug to fix > here-- do we just need to be more explicit about our udev preference here? > > Strictly counting dep complexity, static-dev is the easier choice.. I've posted a patch to fix that over on bug 645190.
(In reply to Zac Medico from comment #8) > (In reply to Ben Kohler from comment #7) > > I'm not sure how this worked on my first run, but on my second run it's > > still preferring static-dev to satisfy virtual/dev-manager. In 2.3.13-r1 it > > did select eudev, but I'm not entirely sure there's a portage bug to fix > > here-- do we just need to be more explicit about our udev preference here? > > > > Strictly counting dep complexity, static-dev is the easier choice.. > > I've posted a patch to fix that over on bug 645190. Zac, I hit both problems (paludis and static-dev). Does this patch fix both?
(In reply to Anthony Basile from comment #9) > > Zac, I hit both problems (paludis and static-dev). Does this patch fix both? Combined patch is here: https://github.com/gentoo/portage/compare/e81497dec819dbfc8e85e533db30d751fadaad81...74d0f516a346c7fb6c52a2508ca16b8949b3b65f.patch
This is released in portage-2.3.20.
(In reply to Zac Medico from comment #11) > This is released in portage-2.3.20. i can verify that it fixes all the issues that i had with 2.3.19-r1. i can now build stages again. thanks Zac.
Since the behavior depends on the order of dependency evaluation, it still randomly pulls in paludis in some catalyst stage1 builds. I have a patch to fix this that I will post soon. The patch adds some some conditional logic involving the fix for bug 632026, making results more deterministic in cases when the DNF form is not needed.
New patch for review: https://archives.gentoo.org/gentoo-portage-dev/message/767148c64dee151f765374bfafb6c5f6 https://github.com/gentoo/portage/pull/248
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=0ddfb5802d98c618bd20742be49a520c9a54b394 commit 0ddfb5802d98c618bd20742be49a520c9a54b394 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-02-02 06:08:59 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-02-02 12:01:25 +0000 dep_zapdeps: sort by new_slot_count for DNF only (bug 645002) Sorting of choices by new_slot_count causes the order of choices specified in the ebuild to be discarded, and new_slot_count may have variance related to the order that packages are added to the graph. This variance can contribute to outcomes that appear to be random, like when catalyst stage1 builds sometimes pull in paludis to satisfy perl-cleaner dependencies. Meanwhile, the order specified in the ebuild has no variance, and the preferences that it specifies can serve as a crucial sources of guidance. Therefore, take advantage of the order specified in the ebuild whenever possible, and use new_slot_count only when it is needed to select optimal choices from DNF (as in bug 632026). This fixes random outcomes in the unit test for bug 645002. Previously, the unit test pulled in paludis randomly unless portage-utils was added to the arguments. With this patch, the portage-utils argument is no longer needed for the unit test to succeed consistently. The perl-cleaner dependencies do not have any overlapping || deps, so the DNF conversion and new_slot_count sorting do not apply, and the first choice is preferred regardless of the number of slots that it pulls in: || ( ( sys-apps/portage app-portage/portage-utils ) sys-apps/pkgcore sys-apps/paludis ) The _overlap_dnf function now returns the argument object when there is no overlap, so OverlapDNFTestCase has to be adjusted to account for this. Bug: https://bugs.gentoo.org/645002 Fixes: 9fdaf9bdbdf5 ("dep_check: use DNF to optimize overlapping virtual || deps (bug 632026)") pym/portage/dep/dep_check.py | 49 ++++++++++++++++++---- pym/portage/tests/dep/test_overlap_dnf.py | 2 +- .../resolver/test_virtual_minimize_children.py | 12 +++--- 3 files changed, 47 insertions(+), 16 deletions(-)}
