Summary: | Wrong merge order with dev-qt/{qtcore,qtxmlpatterns} and kde-frameworks/syntax-highlighting | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Sam James <sam> |
Component: | Core - Dependencies | Assignee: | Portage team <dev-portage> |
Status: | CONFIRMED --- | ||
Severity: | normal | CC: | esigra, ionen, zmedico |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=756199 https://bugs.gentoo.org/show_bug.cgi?id=754777 https://bugs.gentoo.org/show_bug.cgi?id=199856 https://github.com/gentoo/portage/pull/1231 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 689644, 930802 | ||
Attachments: |
emerge -p -uvDU @world
emerge_debug.xz diff of output before/after |
Description
Sam James
2024-01-04 07:10:03 UTC
Created attachment 881450 [details]
emerge -p -uvDU @world
If I re-run emerge -p -uvDU @world, I get kde-frameworks/syntax-highlighted queued still, rather than dev-qt/qtxmlpatterns first:
```
$ emerge -p -uvDU @world
These are the packages that would be merged, in order:
Calculating dependencies ... done!
Dependency resolution took 22.39 s (backtrack: 1/20).
[ebuild U ] kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo [5.110.0:5/5.110::gentoo] USE="-debug -doc -test" 0 KiB
[...]
[ebuild U ] dev-qt/qtgui-5.15.11-r2:5/5.15.11::gentoo [5.15.10-r2:5/5.15.10::gentoo] USE="X accessibility* dbus egl jpeg libinput png udev vulkan wayland -debug -eglfs -evdev -gles2-only -ibus -linuxfb -test -tslib -tuio -vnc" 0 KiB
[ebuild U ] dev-qt/qtwayland-5.15.11-r1:5/5.15.11::gentoo [5.15.10-r6:5/5.15.10::gentoo] USE="-compositor% -debug -test (-X%*) (-vulkan%*)" 0 KiB
[ebuild U ] dev-qt/qtwidgets-5.15.11-r1:5/5.15::gentoo [5.15.10-r3:5/5.15::gentoo] USE="X dbus gtk png -debug -gles2-only -test" 0 KiB
[ebuild U ] dev-qt/qtx11extras-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="-debug -test" 0 KiB
[ebuild U ] kde-plasma/kwayland-5.113.0:5/5.113::gentoo [5.110.0:5/5.110::gentoo] USE="-debug -doc -test" 0 KiB
[ebuild U ] dev-qt/qtdeclarative-5.15.11-r2:5/5.15::gentoo [5.15.10-r3:5/5.15::gentoo] USE="jit vulkan widgets -debug -gles2-only -localstorage -test" 0 KiB
[ebuild U ] kde-frameworks/kguiaddons-5.113.0:5/5.113::gentoo [5.110.0:5/5.110::gentoo] USE="X dbus wayland -debug -doc (-kf6compat) -test" 0 KiB
[ebuild U ] dev-qt/qtsvg-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="-debug -test" 0 KiB
[ebuild U ] kde-plasma/kwayland-integration-5.27.10:5::gentoo [5.27.8:5::gentoo] USE="-debug" 0 KiB
[ebuild U ] dev-qt/qtprintsupport-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="cups -debug -gles2-only -test" 0 KiB
[ebuild U ] kde-frameworks/kidletime-5.113.0:5/5.113::gentoo [5.110.0:5/5.110::gentoo] USE="X wayland -debug -doc -xscreensaver" 0 KiB
[ebuild U ] dev-qt/qtopengl-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="-debug -gles2-only -test" 0 KiB
[ebuild U ] kde-apps/kate-23.08.3:5::gentoo [23.04.3:5::gentoo] USE="-debug -handbook" 0 KiB
[ebuild U ] kde-frameworks/kconfigwidgets-5.113.0:5/5.113::gentoo [5.110.0:5/5.110::gentoo] USE="man -debug -designer -doc -test" 0 KiB
[ebuild U ] dev-qt/linguist-tools-5.15.11:5::gentoo [5.15.10:5::gentoo] USE="qml -debug -test" 0 KiB
[ebuild U ] dev-qt/qtmultimedia-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="alsa gstreamer* pulseaudio qml widgets -debug -gles2-only -openal -test" 0 KiB
[ebuild U ] dev-qt/qtgraphicaleffects-5.15.11:5::gentoo [5.15.10:5::gentoo] USE="-debug -test" 0 KiB
[ebuild U ] dev-qt/qtquickcontrols-5.15.11:5::gentoo [5.15.10:5::gentoo] USE="widgets -debug -test" 0 KiB
[ebuild U ] kde-plasma/layer-shell-qt-5.27.10:5::gentoo [5.27.8:5::gentoo] USE="-debug" 0 KiB
[ebuild N ] dev-qt/qtwebchannel-5.15.11:5/5.15::gentoo USE="qml -debug -test" 0 KiB
[ebuild U ] dev-qt/designer-5.15.11:5/5.15::gentoo [5.15.10:5/5.15::gentoo] USE="declarative -debug -test" 0 KiB
[ebuild U ] dev-qt/qtxmlpatterns-5.15.11:5::gentoo [5.15.10:5::gentoo] USE="qml -debug -test" 0 KiB
[...]
```
Created attachment 881451 [details]
emerge_debug.xz
I was expecting to see a runtime cycle in the --debug output but there doesn't seem to be a relevant one.
(In reply to Sam James from comment #2) > Created attachment 881451 [details] > emerge_debug.xz > > I was expecting to see a runtime cycle in the --debug output but there > doesn't seem to be a relevant one. ``` Parent: (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) Depstring: >=dev-qt/qtdeclarative-5.15.9:5 >=dev-qt/qtgui-5.15.9:5 >=dev-qt/qtnetwork-5.15.9:5 >=dev-qt/qtxmlpatterns-5.15.9:5 dev-qt/qtcore:5 Priority: buildtime Candidates: ['>=dev-qt/qtdeclarative-5.15.9:5', '>=dev-qt/qtgui-5.15.9:5', '>=dev-qt/qtnetwork-5.15.9:5', '>=dev-qt/qtxmlpatterns-5.15.9:5', 'dev-qt/qtcore:5'] ebuild: dev-qt/qtxmlpatterns-5.15.11::gentoo [...] Child: (dev-qt/qtxmlpatterns-5.15.11:5/5::gentoo, ebuild scheduled for merge) USE="qml -debug -test" ABI_X86="(64)" Parent Dep: >=dev-qt/qtxmlpatterns-5.15.9:5 required by (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) Exiting... (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) ``` and ``` (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) depends on (dev-qt/qtdeclarative-5.15.11-r2:5/5.15::gentoo, ebuild scheduled for merge) (buildtime) (dev-qt/qtgui-5.15.11-r2:5/5.15.11::gentoo, ebuild scheduled for merge) (buildtime) (dev-qt/qtnetwork-5.15.11-1:5/5.15::gentoo, installed) (buildtime) (kde-frameworks/kf-env-5-2:5/5::gentoo, installed) (runtime) (dev-qt/qtcore-5.15.11-r1-1:5/5.15.11::gentoo, installed) (buildtime) (dev-qt/qtxmlpatterns-5.15.11:5/5::gentoo, ebuild scheduled for merge) (buildtime) (dev-lang/perl-5.38.2-r1-1:0/5.38::gentoo, installed) (buildtime) (dev-qt/linguist-tools-5.15.11:5/5::gentoo, ebuild scheduled for merge) (buildtime) (app-alternatives/ninja-1-1:0/0::gentoo, installed) (buildtime) (dev-util/cmake-3.27.7-1:0/0::gentoo, installed) (buildtime) (dev-libs/libpcre2-10.42-r1-3:0/3::gentoo, installed) (buildtime) (kde-frameworks/extra-cmake-modules-5.113.0-1:0/0::gentoo, installed) (buildtime) (dev-qt/qtxmlpatterns-5.15.11:5/5::gentoo, ebuild scheduled for merge) depends on (dev-qt/qtcore-5.15.11-r1-1:5/5.15.11::gentoo, installed) (buildtime) (dev-qt/qtnetwork-5.15.11-1:5/5.15::gentoo, installed) (buildtime) (dev-qt/qtdeclarative-5.15.11-r2:5/5.15::gentoo, ebuild scheduled for merge) (buildtime) (dev-lang/perl-5.38.2-r1-1:0/5.38::gentoo, installed) (buildtime) (virtual/pkgconfig-2-r1-2:0/0::gentoo, installed) (buildtime) ``` Came up on the forums too at https://forums.gentoo.org/viewtopic-t-1166369.html. Not been able to make my testcase reproduce it yet. (In reply to Sam James from comment #2) > Created attachment 881451 [details] > emerge_debug.xz The log shows a bunch of single node runtime cycles like this, so find_smallest_cycle is definitely misbehaving: runtime cycle digraph (1 nodes): (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) (no children) runtime cycle leaf: (kde-frameworks/syntax-highlighting-5.113.0:5/5.113::gentoo, ebuild scheduled for merge) Can you test https://github.com/gentoo/portage/pull/1229 to see if that helps? Maybe it's not the best fix, but it might suffice until we come up with a better way to make find_smallest_cycle behave. Using a stage3 I found these cycles related to your USE=emacs setting: * Error: circular dependencies: (dev-util/desktop-file-utils-0.27:0/0::gentoo, ebuild scheduled for merge) depends on (app-editors/emacs-29.1-r6:29/29::gentoo, ebuild scheduled for merge) (buildtime) (x11-libs/gtk+-3.24.38:3/3::gentoo, ebuild scheduled for merge) (buildtime) (dev-util/desktop-file-utils-0.27:0/0::gentoo, ebuild scheduled for merge) (runtime) It might be possible to break this cycle by applying any of the following changes: - dev-util/desktop-file-utils-0.27 (Change USE: -emacs) - app-editors/emacs-29.1-r6 (Change USE: -gui) - app-editors/emacs-29.1-r6 (Change USE: -gtk) (dev-util/ninja-1.11.1-r3:0/0::gentoo, ebuild scheduled for merge) depends on (app-editors/emacs-29.1-r6:29/29::gentoo, ebuild scheduled for merge) (runtime) (media-libs/lcms-2.15:2/2::gentoo, ebuild scheduled for merge) (buildtime) (app-alternatives/ninja-1:0/0::gentoo, ebuild scheduled for merge) (buildtime) (dev-util/ninja-1.11.1-r3:0/0::gentoo, ebuild scheduled for merge) (runtime) It might be possible to break this cycle by applying any of the following changes: - app-alternatives/ninja-1 (Change USE: +samurai -reference) - dev-util/ninja-1.11.1-r3 (Change USE: -emacs) - app-editors/emacs-29.1-r6 (Change USE: -lcms) (gnome-base/librsvg-2.56.3:2/2::gentoo, ebuild scheduled for merge) depends on (dev-util/desktop-file-utils-0.27:0/0::gentoo, ebuild scheduled for merge) (runtime) (app-editors/emacs-29.1-r6:29/29::gentoo, ebuild scheduled for merge) (buildtime) (gnome-base/librsvg-2.56.3:2/2::gentoo, ebuild scheduled for merge) (buildtime) It might be possible to break this cycle by applying any of the following changes: - app-editors/emacs-29.1-r6 (Change USE: -svg) - app-editors/emacs-29.1-r6 (Change USE: -gui) - dev-util/desktop-file-utils-0.27 (Change USE: -emacs) Created attachment 881487 [details] diff of output before/after (In reply to Zac Medico from comment #6) > Can you test https://github.com/gentoo/portage/pull/1229 to see if that > helps? Maybe it's not the best fix, but it might suffice until we come up > with a better way to make find_smallest_cycle behave. Unfortunately no meaningful difference I think. (if needed I can probably give ssh to a container with the same env or tarball it up, but obv. it'll be big) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b5340fc8a3a71cca6161fddfbcf9db982965b7ee commit b5340fc8a3a71cca6161fddfbcf9db982965b7ee Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-05 09:38:32 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-05 09:38:32 +0000 dev-util/desktop-file-utils: move Emacs files to app-emacs/desktop-entry-mode Bug: https://bugs.gentoo.org/921333 Signed-off-by: Sam James <sam@gentoo.org> .../desktop-file-utils-0.27-r1.ebuild | 27 ++++++++++++++++++++++ 1 file changed, 27 insertions(+) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=50a7f72871ffc4a535cf555aaf49ce95eb67d710 commit 50a7f72871ffc4a535cf555aaf49ce95eb67d710 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-05 09:35:57 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-05 09:37:20 +0000 app-emacs/desktop-entry-mode: new package split out from dev-util/desktop-file-utils[emacs], add 0.27 Split out from dev-util/desktop-file-utils[emacs] to help break a circular dep with USE=emacs: ``` (dev-util/desktop-file-utils-0.27:0/0::gentoo, ebuild scheduled for merge) depends on (app-editors/emacs-29.1-r6:29/29::gentoo, ebuild scheduled for merge) (buildtime) (x11-libs/gtk+-3.24.38:3/3::gentoo, ebuild scheduled for merge) (buildtime) (dev-util/desktop-file-utils-0.27:0/0::gentoo, ebuild scheduled for merge) (runtime) It might be possible to break this cycle by applying any of the following changes: - dev-util/desktop-file-utils-0.27 (Change USE: -emacs) - app-editors/emacs-29.1-r6 (Change USE: -gui) - app-editors/emacs-29.1-r6 (Change USE: -gtk) ``` Bug: https://bugs.gentoo.org/921333 Signed-off-by: Sam James <sam@gentoo.org> app-emacs/desktop-entry-mode/Manifest | 1 + .../desktop-entry-mode-0.27.ebuild | 24 ++++++++++++++++++++++ .../files/50desktop-entry-mode-gentoo.el | 5 +++++ app-emacs/desktop-entry-mode/metadata.xml | 13 ++++++++++++ 4 files changed, 43 insertions(+) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=aac64dad7bd5e8e6fcc4e63baff0f54807712280 commit aac64dad7bd5e8e6fcc4e63baff0f54807712280 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-05 09:11:36 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-05 09:37:19 +0000 dev-util/ninja: move Emacs files to app-emacs/ninja-mode Bug: https://bugs.gentoo.org/921333 Signed-off-by: Sam James <sam@gentoo.org> dev-util/ninja/ninja-1.11.1-r4.ebuild | 118 ++++++++++++++++++++++++++++++++++ dev-util/ninja/ninja-9999.ebuild | 25 ++----- 2 files changed, 124 insertions(+), 19 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4ff501f192595dcce7ca90809c1285bfadd8915b commit 4ff501f192595dcce7ca90809c1285bfadd8915b Author: Sam James <sam@gentoo.org> AuthorDate: 2024-01-05 09:09:28 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-01-05 09:37:12 +0000 app-emacs/ninja-mode: new package split out from dev-util/ninja[emacs], add 1.11.1 Split out from dev-util/ninja[emacs] to help break a circular dep with USE=emacs: ``` (dev-util/ninja-1.11.1-r3:0/0::gentoo, ebuild scheduled for merge) depends on (app-editors/emacs-29.1-r6:29/29::gentoo, ebuild scheduled for merge) (runtime) (media-libs/lcms-2.15:2/2::gentoo, ebuild scheduled for merge) (buildtime) (app-alternatives/ninja-1:0/0::gentoo, ebuild scheduled for merge) (buildtime) (dev-util/ninja-1.11.1-r3:0/0::gentoo, ebuild scheduled for merge) (runtime) It might be possible to break this cycle by applying any of the following changes: - app-alternatives/ninja-1 (Change USE: +samurai -reference) - dev-util/ninja-1.11.1-r3 (Change USE: -emacs) - app-editors/emacs-29.1-r6 (Change USE: -lcms) ``` Bug: https://bugs.gentoo.org/921333 Signed-off-by: Sam James <sam@gentoo.org> app-emacs/ninja-mode/Manifest | 1 + app-emacs/ninja-mode/metadata.xml | 18 ++++++++++++++++++ app-emacs/ninja-mode/ninja-mode-1.11.1.ebuild | 19 +++++++++++++++++++ 3 files changed, 38 insertions(+) (In reply to Larry the Git Cow from comment #10) These don't seem to help as-is here (via just the ebuild metadata) but I don't want to merge them in case the bug goes away. Since kde-frameworks/syntax-highlighting actually has an soname dependency on libQt5XmlPatterns.so.5, changing its dev-qt/qtxmlpatterns dependency to a slot-operator dependency would be simple way to increase the dependency priority so that find_smallest_cycle will give a better result. As it is, find_smallest_cycle sees a bunch of dependencies of similar priority, so it's not really clear which node to choose. If we increase he priority of a runtime + buildtime dep to be stronger than a plain buildtime dep, then it will provide some more contrast. However, we'll have to merge the DEPEND and RDEPEND DepPriority instances into a single instance that has both runtime and buildtime attributes set to True. (In reply to Zac Medico from comment #12) > Since kde-frameworks/syntax-highlighting actually has an soname dependency > on libQt5XmlPatterns.so.5, changing its dev-qt/qtxmlpatterns dependency to a > slot-operator dependency would be simple way to increase the dependency > priority so that find_smallest_cycle will give a better result. > dev-qt/* has by policy <subslot> meaning private ABI unfortunately.. Now I've realized that since the "satisfied" attribute of the DepPriority can be an overriding factor in the find_smallest_cycle behavior, it is essential that we include a model for broken dependencies in the calculation of the "satisfied" attribute. In order to model the broken dependencies, we'll have to start by creating a dependency graph for the initial installed state (like emerge --depclean does), and then update the effective state after each package is selected for merge in the merge order calculation. The DepPriority "satisfied" attribute, instead of being a constant boolean, will be a dynamic object that is linked to the effective state of installed packages at a given moment during the merge order calculation. In https://github.com/gentoo/portage/pull/1231 I've extended the _calc_depclean function to support a new dep_check action that can be used to check for broken dependencies of installed packages. Since it will be a lot of work to fully account for broken dependencies, it is tempting to first implement an approximate fix that only solves the problem for dependencies that are initially broken, and does not account for those that break as a result of the update process. However, maybe it's better to avoid this kind of approximation because it will not account for dependencies that become satisfied as a result of the update process, and can therefore trigger "false positive" circular dependency errors. (In reply to Zac Medico from comment #16) > In https://github.com/gentoo/portage/pull/1231 I've extended the > _calc_depclean function to support a new dep_check action that can be used > to check for broken dependencies of installed packages. > > Since it will be a lot of work to fully account for broken dependencies, it > is tempting to first implement an approximate fix that only solves the > problem for dependencies that are initially broken, and does not account for > those that break as a result of the update process. i.e. that'd fix the 'resume' case, but not the original breakage (which I wasn't able to capture)? The problem is that the larger the depgraph, the more likely things will go wrong, and with large upgrades where python-exec changes USE flags for example, resuming isn't always possible (your 'emerge' gets bricked partway through). > However, maybe it's > better to avoid this kind of approximation because it will not account for > dependencies that become satisfied as a result of the update process, and > can therefore trigger "false positive" circular dependency errors. Could you elaborate on that last bit? I don't get yet when they'd show up (In reply to Sam James from comment #17) > > However, maybe it's > > better to avoid this kind of approximation because it will not account for > > dependencies that become satisfied as a result of the update process, and > > can therefore trigger "false positive" circular dependency errors. > > Could you elaborate on that last bit? I don't get yet when they'd show up Suppose that you unmerge sys-devel/llvm as a way to trigger some breakage for testing purposes, and it breaks the corresponding dependency dev-lang/rust. Then suppose that you have USE=system-bootstrap enabled for dev-lang/rust, and you have masked dev-lang/rust-bin. Now if you try to perform a world update, and we have some approximate logic that says your installed dev-lang/rust is broken and can't satisfy dependencies, then you'll get a "false positive" circular dependency error if a new version of dev-lang/rust is pulled in. The approximation does not account for the fact that your installed dev-lang/rust becomes un-broken as soon as the world update installs your missing sys-devel/llvm package. On the bright side, circular dependency backtracking might temporarily mask the dev-lang/rust update, allowing a successful calculation that installs sys-devel/llvm and un-breaks dev-lang/rust. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=4b885b9ca063c990b1e218c73a786e9d434717e8 commit 4b885b9ca063c990b1e218c73a786e9d434717e8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-01-08 06:04:37 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-01-08 08:08:51 +0000 _calc_depclean: add dep_check action Add a dep_check action which can be used to check the dependencies of all installed packages. The plan is for depgraph to use this action to check for broken dependencies prior to the merge order calculation. The new frozen_config parameter will allow depgraph to pass a shared frozen config to _calc_depclean. The result of the dep_check action becomes stale as soon as there is any change to the installed packages. So, in order to account for dependencies that may become broken or satisfied during the process of updating installed packages, the merge order calculation will need to refresh the dep_check calculation for every merge order choice that it makes. This refresh will need to be optimized to identify the portion of the graph that would become stale due to a given change, so that it can avoid unnecessary repetition of work. Bug: https://bugs.gentoo.org/921333 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/actions.py | 21 ++++++- lib/_emerge/depgraph.py | 7 ++- lib/portage/tests/resolver/ResolverPlayground.py | 43 ++++++++++++-- lib/portage/tests/resolver/meson.build | 1 + lib/portage/tests/resolver/test_broken_deps.py | 76 ++++++++++++++++++++++++ 5 files changed, 138 insertions(+), 10 deletions(-) (In reply to Zac Medico from comment #18) (thanks!) |