Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 463976 - sys-apps/portage: rebuild triggered by dependency upgrade has incorrect merge order, caused by circular dependencies (affects slot-operator/sub-slot rebuilds and --rebuild-* options)
Summary: sys-apps/portage: rebuild triggered by dependency upgrade has incorrect merge...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 567552 569938 685280 787806 789966 (view as bug list)
Depends on: 908717
Blocks: 155723 build-order
  Show dependency tree
 
Reported: 2013-03-31 10:42 UTC by Roman Žilka
Modified: 2023-08-08 04:55 UTC (History)
12 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Debug output of the original emerge command (emerge-d.out.xz,60.12 KB, application/x-xz)
2013-03-31 10:43 UTC, Roman Žilka
Details
emerge --info (emerge--info.out,5.80 KB, text/plain)
2013-03-31 10:46 UTC, Roman Žilka
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Roman Žilka 2013-03-31 10:42:21 UTC
Today I got this:

# emerge -pvuDN --complete-graph y --with-bdeps y --rebuild-if-new-rev y world

[ebuild  rR    ] sys-apps/util-linux-2.22.2  USE="cramfs crypt ncurses nls {test} udev unicode -ddate -old-linux -perl (-selinux) -slang -static-libs -suid" 0 kB
[ebuild     U  ] sys-fs/udev-init-scripts-25 [23] 0 kB
[ebuild     U  ] sys-apps/hwids-20130329 [20130302] USE="udev" 0 kB
[ebuild  N     ] app-text/build-docbook-catalog-1.19.1  0 kB
[ebuild  N     ] app-text/docbook-xsl-stylesheets-1.78.0  USE="-ruby" 0 kB
[ebuild     U  ] sys-fs/udev-200 [197-r8] USE="firmware-loader%* kmod openrc -acl -doc -gudev -hwdb -introspection -keymap (-selinux) -static-libs" 0 kB
[ebuild  rR    ] x11-base/xorg-server-1.13.1  USE="minimal nptl udev xorg xvfb -dmx -doc -ipv6 -kdrive (-selinux) -static-libs -suid -tslib -xnest" 0 kB

I can't seem to find out what triggered the rebuild of util-linux, but whatever it is, it makes no sense to rR-rebuild something before everything else. Maybe it's the dependency on virtual/udev that's somehow tied to sys-fs/udev, but wrongly processed by rebuild-if-new-rev.

Reproducible: Always
Comment 1 Roman Žilka 2013-03-31 10:43:27 UTC
Created attachment 343816 [details]
Debug output of the original emerge command
Comment 2 Roman Žilka 2013-03-31 10:46:18 UTC
Created attachment 343818 [details]
emerge --info
Comment 3 Zac Medico gentoo-dev 2013-03-31 17:59:18 UTC
The --rebuild-if-new-rev code "sees through" virtuals, so that explains why a virtual/udev dependency and sys-fs/udev upgrade would trigger a rebuild of util-linux.

The odd build order is probably due to circular dependencies, which is common in system packages. If so, then I don't think there's any good way to fix this, other than to stop using --rebuild-if-new-rev.

Ideally, we'd be able to automatically trigger all desirable rebuilds without using options like --rebuild-if-new-rev. For example, with EAPI 5, we have automatic rebuilds triggered by slot-operators and sub-slots:

  http://wiki.gentoo.org/wiki/Sub-slots_and_Slot-Operators
Comment 4 Roman Žilka 2013-03-31 22:08:49 UTC
First off, I looked at the wiki article and I must say that if this strategy is adopted by all/most ebuilds, the tree will be a better place.

But as of now, I suppose the circular deps situation can cause trouble with revdep-if-new-rev in case pkgX and pkgY are to be upgraded while depending on each other. But that's not the case here with udev and util-linux. Only udev needs to be upgraded, so why not merge udev first and util-linux afterwards? rebuild-if-unbuilt cannot be satisfied, but rebuild-if-new-* can be, it seems.

And even if both udev and util-linux were in need of upgrade while depending on each other, rebuild-if-new-* can function: one of the two packages is upgraded, then the other and finally the first one is re-compiled. I suppose that would require implementing multiple builds of a single package in a single emerge run. Alternatively, one of the two packages is upgraded, then the other and the next call of "emerge --deep --rebuild-if-new-* world" (or something similar) knows to rR-merge the former package. Actually, this feature would be a nice thing to have even outside any circular deps matters. I mean, the feature of the unfinished, but necessary rR builds being remembered and included in the merge set next time emerge is run with rebuild-if-new-*. I wonder, though, what kind of amount of people use rebuild-if-new-* at all.
Comment 5 Roman Žilka 2013-03-31 22:12:52 UTC
This also reminds me that the stable sys-fs/udev is now -200, while virtual/udev stays at -197. I have no idea if that's any kind of problem at all; just FYI.
Comment 6 Zac Medico gentoo-dev 2013-04-01 01:52:22 UTC
(In reply to comment #4)
Yeah, it seems like we could handle this better. Also, it's possible for the same issue to affect rebuilds involving slot-operators and sub-slots.
Comment 7 Zac Medico gentoo-dev 2015-12-29 17:32:09 UTC
*** Bug 569938 has been marked as a duplicate of this bug. ***
Comment 8 Andreas K. Hüttel archtester gentoo-dev 2016-02-29 13:50:23 UTC
*** Bug 567552 has been marked as a duplicate of this bug. ***
Comment 9 Andreas K. Hüttel archtester gentoo-dev 2016-02-29 13:50:48 UTC
*** Bug 575960 has been marked as a duplicate of this bug. ***
Comment 10 Andreas K. Hüttel archtester gentoo-dev 2016-05-24 21:40:38 UTC
*** Bug 567552 has been marked as a duplicate of this bug. ***
Comment 11 Zac Medico gentoo-dev 2019-05-08 17:13:33 UTC
*** Bug 685280 has been marked as a duplicate of this bug. ***
Comment 12 Andreas K. Hüttel archtester gentoo-dev 2021-05-13 13:47:20 UTC
*** Bug 787806 has been marked as a duplicate of this bug. ***
Comment 13 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-05-13 14:00:25 UTC
*** Bug 789966 has been marked as a duplicate of this bug. ***
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-05-13 14:00:54 UTC
I'm not sure why but this seems to be hit more often right now on fresh installs.

bug 787806, bug 789966
Comment 15 Larry the Git Cow gentoo-dev 2023-06-16 03:34:50 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=224207c7d1988a354d004507bb7ecfb90b4ef097

commit 224207c7d1988a354d004507bb7ecfb90b4ef097
Author:     YiFei Zhu <zhuyifei1999@gmail.com>
AuthorDate: 2023-06-13 00:47:52 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-06-16 03:34:46 +0000

    depgraph: Do not allow slotted deps to be satisfied by wrong slots
    
    This may be part of what caused the "perl rebuild bug". The
    "priority" of perl, when seen by perl modules, may have "satisfied"
    set to an installed perl of a wrong slot. Compounding this factor
    with bug #756199 where find_smallest_cycle would select a single
    node, in this case because the dependency of perl is satisfied and
    the priority then gets ignored, the "cycle" becomes a perl module
    alone and gets rebuilt early.
    
    I also updated the test from the previous patch to account for
    this change. No other tests seems affected.
    
    For a larger scale test, I reproduced this initially with a stage3
    chroot from a Jan 1 2022 stage3 snapshot, and testing in an
    equivalent dockerfile would work too:
    
      FROM gentoo/stage3:amd64-openrc-20220101
      RUN emerge-webrsync
      COPY . /portage
    
    Before this patch (USE flags omitted):
    
      # cd /portage && bin/emerge -puDN @world 2>&1 | grep -i perl
      [ebuild     U  ] app-admin/perl-cleaner-2.30-r1 [2.30]
      [ebuild  rR    ] virtual/perl-File-Temp-0.231.100
      [ebuild  rR    ] dev-perl/Locale-gettext-1.70.0-r1
      [ebuild  rR    ] dev-perl/MIME-Charset-1.12.2-r1
      [ebuild  rR    ] dev-perl/Module-Build-0.423.100
      [ebuild  rR    ] dev-perl/Text-CharWidth-0.40.0-r2
      [ebuild     U  ] dev-lang/perl-5.36.0-r2 [5.34.0-r3]
      [ebuild  N     ] virtual/perl-CPAN-2.330.0
      [ebuild     U  ] virtual/perl-ExtUtils-MakeMaker-7.640.0 [7.620.0]
      [ebuild     U  ] virtual/perl-File-Spec-3.840.0 [3.800.0]
      [...]
    
    After this patch:
    
      # cd /portage && bin/emerge -puDN @world 2>&1 | grep -i perl
      [ebuild     U  ] app-admin/perl-cleaner-2.30-r1 [2.30]
      [ebuild     U  ] dev-lang/perl-5.36.0-r2:0/5.36 [5.34.0-r3:0/5.34]
      [ebuild  N     ] virtual/perl-CPAN-2.330.0
      [ebuild     U  ] virtual/perl-ExtUtils-MakeMaker-7.640.0 [7.620.0]
      [ebuild     U  ] virtual/perl-Data-Dumper-2.184.0 [2.179.0]
      [ebuild     U  ] virtual/perl-File-Spec-3.840.0 [3.800.0]
      [ebuild     U  ] virtual/perl-Test-Harness-3.440.0-r1 [3.430.0]
      [ebuild  rR    ] dev-perl/Pod-Parser-1.630.0-r1
      [ebuild  rR    ] dev-perl/Text-CharWidth-0.40.0-r2
      [ebuild  rR    ] dev-perl/Text-WrapI18N-0.60.0-r2
      [...]
    
    Bug: https://bugs.gentoo.org/463976
    Bug: https://bugs.gentoo.org/592880
    Bug: https://bugs.gentoo.org/596664
    Bug: https://bugs.gentoo.org/631490
    Bug: https://bugs.gentoo.org/764365
    Bug: https://bugs.gentoo.org/793992
    Signed-off-by: YiFei Zhu <zhuyifei1999@gmail.com>
    Closes: https://github.com/gentoo/portage/pull/1055
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/_emerge/depgraph.py                             | 18 ++++++++++++++++++
 lib/portage/tests/resolver/test_perl_rebuild_bug.py |  4 ++--
 2 files changed, 20 insertions(+), 2 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ff457f742b513a65e6609054e5310f8beee488ec

commit ff457f742b513a65e6609054e5310f8beee488ec
Author:     YiFei Zhu <zhuyifei1999@gmail.com>
AuthorDate: 2023-06-13 08:08:51 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-06-16 03:34:46 +0000

    tests: resolver: Test the broken behavior of Perl rebuild bug
    
    Bug: https://bugs.gentoo.org/463976
    Bug: https://bugs.gentoo.org/592880
    Bug: https://bugs.gentoo.org/596664
    Bug: https://bugs.gentoo.org/631490
    Bug: https://bugs.gentoo.org/764365
    Bug: https://bugs.gentoo.org/793992
    Signed-off-by: YiFei Zhu <zhuyifei1999@gmail.com>
    Signed-off-by: Sam James <sam@gentoo.org>

 .../tests/resolver/test_perl_rebuild_bug.py        | 121 +++++++++++++++++++++
 1 file changed, 121 insertions(+)
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-08-08 04:55:44 UTC
I think this was actually

commit 44afa8445dc46464200fe46c1e09e0c7475067bf
Author: YiFei Zhu <zhuyifei1999@gmail.com>
Date:   Sun Jun 11 19:23:09 2023 -0700

    depgraph: Don't ignore downgrades as missed_updates

but we had to revert it

commit a2da08b6dfbfd2ef803aac6302f9de1f7c00a143
Author: Sam James <sam@gentoo.org>
Date:   Wed Jun 21 19:54:24 2023 +0100

    Revert "depgraph: Don't ignore downgrades as missed_updates"

    This reverts commit 44afa8445dc46464200fe46c1e09e0c7475067bf.

    This is just so we can make a release for the other fixes in master like
    the Perl bits and keep a linear history.

    Signed-off-by: Sam James <sam@gentoo.org>

for bug 908717.