Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 754903 - sys-apps/portage: emerge should minimize the use of installed packages to break cycles
Summary: sys-apps/portage: emerge should minimize the use of installed packages to bre...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Interface (emerge) (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 155723 build-order 756028
  Show dependency tree
 
Reported: 2020-11-16 12:03 UTC by Zac Medico
Modified: 2021-01-26 00:47 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2020-11-16 12:03:10 UTC
When working on an implementation for bug 754777, I found that the prioritization in DepPrioritySatisfiedRange may cause it to break PDEPEND edges later than it should in dependency cycle handling. This issue could make PDEPEND more likely to trigger bug 199856. I'm working on a patch that will fix it to break the PDEPEND edges earlier, and the new priorities currently look like this:

> --- a/lib/_emerge/DepPrioritySatisfiedRange.py
> +++ b/lib/_emerge/DepPrioritySatisfiedRange.py
> @@ -8,17 +8,18 @@ class DepPrioritySatisfiedRange:
>  
>         not satisfied and buildtime                    HARD
>         not satisfied and runtime              7       MEDIUM
> -       not satisfied and runtime_post         6       MEDIUM_SOFT
> -       satisfied and buildtime_slot_op        5       SOFT
> -       satisfied and buildtime                4       SOFT
> -       satisfied and runtime                  3       SOFT
> -       satisfied and runtime_post             2       SOFT
> +       satisfied and buildtime_slot_op        6       MEDIUM_SOFT
> +       satisfied and buildtime                5       MEDIUM_SOFT
> +       satisfied and runtime                  4       MEDIUM_SOFT
> +       runtime_post                           3       MEDIUM_POST
> +       satisfied and runtime_post             2       MEDIUM_POST
>         optional                               1       SOFT
>         (none of the above)                    0       NONE
>         """
>         MEDIUM      = 7
>         MEDIUM_SOFT = 6
> -       SOFT        = 5
> +       MEDIUM_POST = 3
> +       SOFT        = 1
>         NONE        = 0
>  
>         @classmethod
Comment 2 Larry the Git Cow gentoo-dev 2020-11-22 06:13:06 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=5095c2023595a75e2848f1ad3dbe25b5fb451a44

commit 5095c2023595a75e2848f1ad3dbe25b5fb451a44
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-11-16 05:55:54 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-11-22 03:19:29 +0000

    find_smallest_cycle: enhance search prioritization
    
    Enhance the find_smallest_cycle function to prioritize its
    search so that it will minimize the use of installed packages
    to break cycles. When installed packages must be used to
    break cycles, it will now prefer to do this for runtime
    dependencies over buildtime dependencies, since it's
    preferable to build against latest versions of buildtime
    dependencies whenever possible. This should solve some cases
    of bug 199856 which have been triggered by unsafe reliance
    on installed packages to break cycles.
    
    The included unit test case demonstrates correct merge order
    for a dependency calculation involving 6 independent cycles.
    This test case fails with the master branch, due to a buildtime
    dependency cycle of 3 packages being merged earlier than cycles
    of 2 packages. We can generalize this to say that the master
    branch may use an installed package to break an arbitrarily
    sized cycle in a somewhat random location, even though that
    cycle may be composed of smaller independent cycles which
    would be safer to break individually.
    
    Bug: https://bugs.gentoo.org/754903
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/DepPriorityNormalRange.py          |  2 +
 lib/_emerge/DepPrioritySatisfiedRange.py       | 53 ++++++++++++++++----------
 lib/_emerge/depgraph.py                        | 43 ++++++++++++---------
 lib/portage/tests/resolver/test_merge_order.py | 10 +++++
 4 files changed, 70 insertions(+), 38 deletions(-)
Comment 3 Larry the Git Cow gentoo-dev 2020-11-22 06:47:37 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b723fb347408dcd76ab4121f9cf94b0c51d48223

commit b723fb347408dcd76ab4121f9cf94b0c51d48223
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-11-22 06:43:39 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-11-22 06:47:30 +0000

    sys-apps/portage: Bump to version 3.0.10
    
     #753497 profiles should not inherit deprecated messages
     #754903 minimize use of installed packages to break cycles
     #755950 compare_files: handle missing xattr support
    
    Bug: https://bugs.gentoo.org/753497
    Bug: https://bugs.gentoo.org/754903
    Bug: https://bugs.gentoo.org/755950
    Bug: https://bugs.gentoo.org/756028
    Package-Manager: Portage-3.0.10, Repoman-3.0.2
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.10.ebuild | 267 +++++++++++++++++++++++++++++++++
 2 files changed, 268 insertions(+)
Comment 4 Zac Medico gentoo-dev 2020-12-02 02:17:08 UTC
We've had a report of an issue that may be triggered by the patch associated with this bug report (which later may be corrected by fixes related to bug 756199 and/or bug 754777). The issue is that xorg-server is rebuilt too late by this command:

> # emerge -av1 =xorg-server-1.20.8-r1 @x11-module-rebuild
> 
> These are the packages that would be merged, in order:
> 
> Calculating dependencies... done!
> [ebuild   R    ] x11-drivers/xf86-input-synaptics-1.9.1::gentoo  0 KiB
> [ebuild   R    ] x11-drivers/nvidia-drivers-455.38-r1:0/455::gentoo  USE="X driver gtk3 kms (libglvnd) multilib tools uvm wayland -compat -static-libs" ABI_X86="32 (64) (-x32)" 0 KiB
> [ebuild   R    ] x11-drivers/xf86-input-libinput-0.30.0::gentoo  0 KiB
> [ebuild     UD ] x11-base/xorg-server-1.20.8-r1:0/1.20.8::gentoo [1.20.10:0/1.20.10::gentoo] USE="ipv6 (libglvnd%*) systemd udev wayland xorg xvfb -debug -dmx -doc (-elogind) -kdrive -libressl -minimal (-selinux) -static-libs -suid -unwind -xcsecurity -xephyr -xnest" 6.162 KiB
> 
> Total: 4 packages (1 downgrade, 3 reinstalls), Size of downloads: 6.162 KiB
> 
> Would you like to merge these packages? [Yes/No] n
> 
> Quitting.
Comment 5 Zac Medico gentoo-dev 2020-12-02 02:42:53 UTC
(In reply to Zac Medico from comment #4)
> We've had a report of an issue that may be triggered by the patch associated
> with this bug report (which later may be corrected by fixes related to bug
> 756199 and/or bug 754777). The issue is that xorg-server is rebuilt too late
> by this command:
> 
> > # emerge -av1 =xorg-server-1.20.8-r1 @x11-module-rebuild

I've confirmed that reverting 5095c2023595a75e2848f1ad3dbe25b5fb451a44 restores the previous behavior. Also, my current patch for bug 754777 fixes it.
Comment 6 Larry the Git Cow gentoo-dev 2020-12-02 08:32:53 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=07a604537e746814613dc171a5c09072ef0266af

commit 07a604537e746814613dc171a5c09072ef0266af
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-12-02 07:08:04 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-12-02 08:06:27 +0000

    find_smallest_cycle: don't merge satisfied PDEPEND too early
    
    After PDEPENDs have been neglected by the find_smallest_cycle function,
    do not try to merge them too early if they are already satisfied by
    an installed package. This fixes incorrect merge order for PDEPEND
    cycles involving xorg-server and xorg-drivers, which was triggered
    by commit 5095c2023595a75e2848f1ad3dbe25b5fb451a44 because it gave
    PDEPEND higher priority than satisfied buildtime dependencies.
    
    Fixes: 5095c2023595 ("find_smallest_cycle: enhance search prioritization")
    Reported-by: josef64 in #gentoo-portage
    Bug: https://bugs.gentoo.org/754903
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/DepPrioritySatisfiedRange.py       |  1 +
 lib/_emerge/depgraph.py                        |  8 ++++----
 lib/portage/tests/resolver/test_merge_order.py | 27 ++++++++++++++++++++++++--
 3 files changed, 30 insertions(+), 6 deletions(-)
Comment 7 Larry the Git Cow gentoo-dev 2020-12-02 08:54:23 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=161da5c962d6e5a63f55ce3743ec48c7d7a43ab8

commit 161da5c962d6e5a63f55ce3743ec48c7d7a43ab8
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-12-02 08:41:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-12-02 08:54:17 +0000

    sys-apps/portage: Revbump to 3.0.11-r1 for regression fix
    
     #754903 emerge: don't merge satisfied PDEPEND too early
    
    Bug: https://bugs.gentoo.org/754903
    Bug: https://bugs.gentoo.org/756028
    Package-Manager: Portage-3.0.11, Repoman-3.0.2
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest                                          | 1 +
 .../portage/{portage-3.0.11.ebuild => portage-3.0.11-r1.ebuild}    | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)