Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 645002

Summary: sys-apps/portage-2.3.19-r1: catalyst stage1 build pulls in paludis to satisfy perl-cleaner dependency
Product: Portage Development Reporter: Zac Medico <zmedico>
Component: Core - DependenciesAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: alexander, bkohler, releng, viklevin2
Priority: Normal Keywords: InVCS, REGRESSION
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=632026
https://bugs.gentoo.org/show_bug.cgi?id=645190
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 645194    
Attachments: pretend emerge calculation, simulating catalyst stage1, with patch applied

Description Zac Medico gentoo-dev 2018-01-19 04:32:39 UTC
The choices.sort(key=lambda x: (not x.all_installed_slots, len(x.slot_map))) code from bug 632026 is causing these perl-cleaner dependencies to prefer paludis in the catalyst stage1 build:

	|| (
		( sys-apps/portage app-portage/portage-utils )
		sys-apps/pkgcore
		sys-apps/paludis
	)

It prefers paludis because len(x.slot_map) is 1 for sys-apps/paludis, compared to 2 for ( sys-apps/portage app-portage/portage-utils ).

The command that fails is for the packages listed in profile packages.build, which includes virtual/package-manager. So, catalyst never really makes a commitment to a specific package manager, relying entirely on || deps and virtuals.
Comment 1 Ben Kohler gentoo-dev 2018-01-19 14:02:45 UTC
While I'm sure you're on the right track, this doesn't seem isolated to that one case with the extra package count.

On a fresh stage1 build that was working w/o conflicts or surprises a few days ago, there are 4 new choices being made, each causing some grief:

1) paludis picked over ( portage portage-utils) to satisfy virtual/package-manager
2) pkgconf picked over pkgconfig to satisfy virtual/pkgconfig, even though pkgconf requires an extra USE flag to get there
3) realpath picked over coreutils, to satisfy eselect, resulting in blocker
4) static-dev being preferred over virtual/udev, to satisfy virtual/dev-manager, resulting in bail-out in pkg_pretend
Comment 3 Zac Medico gentoo-dev 2018-01-20 00:57:06 UTC
Created attachment 515332 [details]
pretend emerge calculation, simulating catalyst stage1, with patch applied

(In reply to Ben Kohler from comment #1)
> While I'm sure you're on the right track, this doesn't seem isolated to that
> one case with the extra package count.
> 
> On a fresh stage1 build that was working w/o conflicts or surprises a few
> days ago, there are 4 new choices being made, each causing some grief:
> 
> 1) paludis picked over ( portage portage-utils) to satisfy
> virtual/package-manager
> 2) pkgconf picked over pkgconfig to satisfy virtual/pkgconfig, even though
> pkgconf requires an extra USE flag to get there
> 3) realpath picked over coreutils, to satisfy eselect, resulting in blocker
> 4) static-dev being preferred over virtual/udev, to satisfy
> virtual/dev-manager, resulting in bail-out in pkg_pretend

With my patch applied, I did a pretend emerge calculation in a stage4-amd64-minimal-20180118T214502Z chroot, and it looks good. There's no pkgconf, realpath, or static-dev issues showing up. The patch did solve the paludis issue.
Comment 4 Ben Kohler gentoo-dev 2018-01-20 14:47:34 UTC
With this patch, all 4 issues I cited have disappeared. 1-3 made the correct other choice, and for #4 it seems virtual/dev-manager dropped out of the deps entirely.

Thanks!
Comment 5 Larry the Git Cow gentoo-dev 2018-01-20 23:14:49 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e2134e9f72a86734552bb67e9414a017cfc4ea51

commit e2134e9f72a86734552bb67e9414a017cfc4ea51
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-01-20 00:28:42 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-01-20 23:08:42 +0000

    dep_zapdeps: prefer choices with fewer new slots (bug 645002)
    
    Prefer choices with fewer new slots, rather than choices with the lowest
    total number of slots. This fixes a case triggered by the catalyst stage1
    build, where paludis was selected to satisfy perl-cleaner dependencies
    because that choice happened to have a smaller number of slots:
    
      || (
        ( sys-apps/portage app-portage/portage-utils )
        sys-apps/pkgcore
        sys-apps/paludis
      )
    
    Bug: https://bugs.gentoo.org/645002
    Fixes: 9fdaf9bdbdf5 ("dep_check: use DNF to optimize overlapping virtual || deps (bug 632026)")
    Tested-by: Ben Kohler <bkohler@gmail.com>

 pym/portage/dep/dep_check.py                       | 14 ++--
 .../resolver/test_virtual_minimize_children.py     | 85 +++++++++++++++++++++-
 2 files changed, 93 insertions(+), 6 deletions(-)}
Comment 6 Zac Medico gentoo-dev 2018-01-20 23:15:44 UTC
(In reply to Ben Kohler from comment #4)
> With this patch, all 4 issues I cited have disappeared. 1-3 made the correct
> other choice, and for #4 it seems virtual/dev-manager dropped out of the
> deps entirely.
> 
> Thanks!

Great, thanks for testing!
Comment 7 Ben Kohler gentoo-dev 2018-01-20 23:20:58 UTC
I'm not sure how this worked on my first run, but on my second run it's still preferring static-dev to satisfy virtual/dev-manager.  In 2.3.13-r1 it did select eudev, but I'm not entirely sure there's a portage bug to fix here-- do we just need to be more explicit about our udev preference here?

Strictly counting dep complexity, static-dev is the easier choice..
Comment 8 Zac Medico gentoo-dev 2018-01-21 00:17:59 UTC
(In reply to Ben Kohler from comment #7)
> I'm not sure how this worked on my first run, but on my second run it's
> still preferring static-dev to satisfy virtual/dev-manager.  In 2.3.13-r1 it
> did select eudev, but I'm not entirely sure there's a portage bug to fix
> here-- do we just need to be more explicit about our udev preference here?
> 
> Strictly counting dep complexity, static-dev is the easier choice..

I've posted a patch to fix that over on bug 645190.
Comment 9 Anthony Basile gentoo-dev 2018-01-21 00:23:39 UTC
(In reply to Zac Medico from comment #8)
> (In reply to Ben Kohler from comment #7)
> > I'm not sure how this worked on my first run, but on my second run it's
> > still preferring static-dev to satisfy virtual/dev-manager.  In 2.3.13-r1 it
> > did select eudev, but I'm not entirely sure there's a portage bug to fix
> > here-- do we just need to be more explicit about our udev preference here?
> > 
> > Strictly counting dep complexity, static-dev is the easier choice..
> 
> I've posted a patch to fix that over on bug 645190.

Zac, I hit both problems (paludis and static-dev).  Does this patch fix both?
Comment 10 Zac Medico gentoo-dev 2018-01-21 00:29:27 UTC
(In reply to Anthony Basile from comment #9)
> 
> Zac, I hit both problems (paludis and static-dev).  Does this patch fix both?

Combined patch is here:

https://github.com/gentoo/portage/compare/e81497dec819dbfc8e85e533db30d751fadaad81...74d0f516a346c7fb6c52a2508ca16b8949b3b65f.patch
Comment 11 Zac Medico gentoo-dev 2018-01-21 01:56:10 UTC
This is released in portage-2.3.20.
Comment 12 Anthony Basile gentoo-dev 2018-01-21 19:13:27 UTC
(In reply to Zac Medico from comment #11)
> This is released in portage-2.3.20.

i can verify that it fixes all the issues that i had with 2.3.19-r1.  i can now build stages again.  thanks Zac.
Comment 13 Zac Medico gentoo-dev 2018-02-02 06:08:15 UTC
Since the behavior depends on the order of dependency evaluation, it still randomly pulls in paludis in some catalyst stage1 builds.

I have a patch to fix this that I will post soon. The patch adds some some conditional logic involving the fix for bug 632026, making results more deterministic in cases when the DNF form is not needed.
Comment 15 Larry the Git Cow gentoo-dev 2018-02-03 00:01:47 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=0ddfb5802d98c618bd20742be49a520c9a54b394

commit 0ddfb5802d98c618bd20742be49a520c9a54b394
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-02-02 06:08:59 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-02-02 12:01:25 +0000

    dep_zapdeps: sort by new_slot_count for DNF only (bug 645002)
    
    Sorting of choices by new_slot_count causes the order of
    choices specified in the ebuild to be discarded, and
    new_slot_count may have variance related to the order that
    packages are added to the graph. This variance can
    contribute to outcomes that appear to be random, like when
    catalyst stage1 builds sometimes pull in paludis to
    satisfy perl-cleaner dependencies.
    
    Meanwhile, the order specified in the ebuild has no
    variance, and the preferences that it specifies can serve
    as a crucial sources of guidance. Therefore, take advantage
    of the order specified in the ebuild whenever possible, and
    use new_slot_count only when it is needed to select optimal
    choices from DNF (as in bug 632026).
    
    This fixes random outcomes in the unit test for bug 645002.
    Previously, the unit test pulled in paludis randomly unless
    portage-utils was added to the arguments. With this patch,
    the portage-utils argument is no longer needed for the unit
    test to succeed consistently. The perl-cleaner dependencies
    do not have any overlapping || deps, so the DNF conversion
    and new_slot_count sorting do not apply, and the first
    choice is preferred regardless of the number of slots that
    it pulls in:
    
    || (
      ( sys-apps/portage app-portage/portage-utils )
      sys-apps/pkgcore
      sys-apps/paludis
    )
    
    The _overlap_dnf function now returns the argument object
    when there is no overlap, so OverlapDNFTestCase has to be
    adjusted to account for this.
    
    Bug: https://bugs.gentoo.org/645002
    Fixes: 9fdaf9bdbdf5 ("dep_check: use DNF to optimize overlapping virtual || deps (bug 632026)")

 pym/portage/dep/dep_check.py                       | 49 ++++++++++++++++++----
 pym/portage/tests/dep/test_overlap_dnf.py          |  2 +-
 .../resolver/test_virtual_minimize_children.py     | 12 +++---
 3 files changed, 47 insertions(+), 16 deletions(-)}