Bug 702806 – Specify behavior of *TARGET* USE_EXPAND variable settings so that package managers can use them to infer which updates are desirable (related: python-any-r1 pulls in latest Python)

• Status: UNCONFIRMED
• Product: Gentoo Hosted Projects

[Description alfredthetomato 2019-12-14 03:26:06 UTC]

With previous versions of sys-apps/portage, such as ≤portage-2.3.80, emerge -uDN @world acted as expected and would consider only the highest version of whichever slot satisfied *_TARGETS, namely PYTHON_SINGLE_TARGET and {PYTHON,RUBY}_TARGETS.

With sys-apps/portage-2.3.81, emerge -uDN @world wants to update to the latest slot available, despite the profile default *_TARGETS being set.

Wrt dev-lang/python, it will happily emerge the new python:3.7 slot, despite PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" being set.

However, when it comes to dev-lang/ruby:

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-lang/ruby-2.6.5::gentoo
# required by virtual/rubygems-15::gentoo[ruby_targets_ruby26]
# required by dev-ruby/rdoc-6.2.0::gentoo[-test,ruby_targets_ruby26]
# required by dev-lang/ruby-2.4.9::gentoo[rdoc]
# required by dev-ruby/rubygems-3.0.6::gentoo[ruby_targets_ruby24]
>=dev-ruby/did_you_mean-1.3.1 ruby_targets_ruby26

... (it continues with everything with a RUBY_TARGETS use_expand flag)

Essentially, it will not proceed until the user sets RUBY_TARGETS: ruby26, despite the profile defaults of RUBY_TARGETS="ruby24 ruby25". Adding --backtrack=9999 does not help.

I would expect sys-apps/portage to only consider those versions of dev-lang/{python,ruby} which satisfy the slot requirements set with the *_TARGETS use_expand flags. This is the behaviour of previous versions.

This is not so much an issue with dev-lang/python, as it will install an otherwise unused slot. With dev-lang/ruby, however, portage will not proceed until the user sets RUBY_TARGETS: ruby26 and deviates from profile defaults.

Reproducible: Always

[Comment 1 Zac Medico 2019-12-14 03:47:27 UTC]

Maybe related to the change from bug 701996:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=fa7b6ea6ecdc135a01a65e249276e6d75b92791e

[Comment 2 alfredthetomato 2019-12-14 05:54:54 UTC]

Reverting that commit restored previous behaviour.

[Comment 3 Zac Medico 2019-12-14 08:38:54 UTC]

(In reply to alfredthetomato from comment #0)
> The following USE changes are necessary to proceed:
>  (see "package.use" in the portage(5) man page for more details)
> # required by dev-lang/ruby-2.6.5::gentoo
> # required by virtual/rubygems-15::gentoo[ruby_targets_ruby26]
> # required by dev-ruby/rdoc-6.2.0::gentoo[-test,ruby_targets_ruby26]
> # required by dev-lang/ruby-2.4.9::gentoo[rdoc]
> # required by dev-ruby/rubygems-3.0.6::gentoo[ruby_targets_ruby24]
> >=dev-ruby/did_you_mean-1.3.1 ruby_targets_ruby26

It looks like you have ruby_targets_ruby26 enabled for dev-ruby/rdoc and virtual/rubygems. Does this command confirm it?

   emerge -pv --nodeps dev-ruby/rdoc virtual/rubygems

[Comment 4 Zac Medico 2019-12-14 20:47:19 UTC]

(In reply to alfredthetomato from comment #2)
> Reverting that commit restored previous behaviour.

We're not reverting that commit unless we have a clear test case showing that there's something wrong with it.

As noted in comment #3, your output seems to indicate that you had ruby_targets_ruby26 enabled for dev-ruby/rdoc and virtual/rubygems. You have to ensure that you've disabled ruby_targets_ruby26 globally if you don't want it.

Also, you can't install dev-lang/ruby-2.6.5 unless ruby_targets_ruby26 is enabled, so you need to ensure that you don't have dev-lang/ruby or something like that in /var/lib/portage/world. It's okay to have specific slots in there, like dev-lang/ruby:2.5 for example.

[Comment 5 alfredthetomato 2019-12-15 04:52:37 UTC]

dev-lang/ruby is being pulled in as a dependency only. It is not in @world.

The output of emerge -pv --nodeps dev-ruby/rdoc virtual/rubygems shows that I have the default RUBY_TARGETS set, with or without that commit:

These are the packages that would be merged, in order:

[ebuild  N    ~] dev-ruby/rdoc-6.2.0::gentoo  USE="-doc -test" RUBY_TARGETS="ruby24 ruby25 -ruby26" 653 KiB
[ebuild  N     ] virtual/rubygems-15::gentoo  RUBY_TARGETS="ruby24 ruby25 -ruby26" 0 KiB

Total: 2 packages (2 new), Size of downloads: 653 KiB

[Comment 6 alfredthetomato 2019-12-15 05:01:29 UTC]

A simple test-case for the described issue would be to emerge something which requires dev-lang/ruby as a dependency:

   emerge -p dev-libs/ocl-icd

It will happily proceed with the commit reverted, but with it applied:

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-lang/ruby-2.6.5::gentoo
# required by virtual/rubygems-15::gentoo[ruby_targets_ruby26]
# required by dev-lang/ruby-2.4.9::gentoo
# required by dev-ruby/bundler-1.17.3::gentoo[ruby_targets_ruby24]
>=dev-ruby/did_you_mean-1.3.1 ruby_targets_ruby26

With the commit applied, it will refuse to proceed until ruby_targets_26 is added to USE. Putting -ruby_targets_26 into USE does not help.

You mentioned that dev-lang/ruby:2.6 cannot be installed without ruby_targets_26 enabled. I also mentioned that dev-lang/python:3.7 had been installed after running emerge -uDN @world. Reverting the commit and running depclean will remove the extra dev-lang/python:3.7 slot.

The underlying issue is more likely due to portage adding the new dev-lang/ruby:2.6 and dev-lang/python:3.7 slots to the dependency graphs despite profile default *_TARGETS being set, which do not call for those newer slots.

[Comment 7 Zac Medico 2019-12-15 06:09:09 UTC]

(In reply to alfredthetomato from comment #6)
> The underlying issue is more likely due to portage adding the new
> dev-lang/ruby:2.6 and dev-lang/python:3.7 slots to the dependency graphs
> despite profile default *_TARGETS being set, which do not call for those
> newer slots.

The thing is, portage has no way of knowing that a person's *_TARGETS settings should correlate with the latest version of some particular package such as dev-lang/ruby or dev-lang/python. There's no rule linking these things together. So, it's a person's responsibility to manage latest versions of these packages with mechanisms like ACCEPT_KEYWORDS, package.accept_keywords and package.mask.

[Comment 8 Zac Medico 2019-12-15 06:18:20 UTC]

So, assuming you have ACCEPT_KEYWORDS="amd64 ~amd64", something like this will give you the package versions that you want if we assume that you have *_TARGETS enabled for "stable" versions of dev-lang/python and dev-lang/ruby:

$ cat /etc/portage/package.accept_keywords
dev-lang/python -~amd64
dev-lang/ruby -~amd64

[Comment 9 alfredthetomato 2019-12-16 18:27:08 UTC]

(In reply to Zac Medico from comment #7)
> The thing is, portage has no way of knowing that a person's *_TARGETS
> settings should correlate with the latest version of some particular package
> such as dev-lang/ruby or dev-lang/python. There's no rule linking these
> things together. So, it's a person's responsibility to manage latest
> versions of these packages with mechanisms like ACCEPT_KEYWORDS,
> package.accept_keywords and package.mask.

Portage seemed to manage quite well and did not require user intervention before the commit.

Regardless, I came up with a patch to keep the fix for bug 701996 in place whilst restoring the previous behaviour with regards to *_TARGETS dependency resolution. Portage successfully installs with FEATURES=test. Running emerge -pv =virtual/wine-0-r6 dev-libs/ocl-icd with USE=staging just pulls in wine-staging (no wine-vanilla) and will proceed without requiring RUBY_TARGETS changes.

[Comment 10 alfredthetomato 2019-12-16 18:30:10 UTC]

Created attachment 599856 [details, diff]
portage-bug_702806.patch

Patch to restore previous *_TARGETS behaviour.

[Comment 11 alfredthetomato 2019-12-16 19:04:04 UTC]

Created attachment 599860 [details, diff]
portage-bug_702806-r1.patch

Previous patch changed depclean behaviour, updated patch fixes that.

[Comment 12 Zac Medico 2019-12-16 19:12:26 UTC]

(In reply to alfredthetomato from comment #11)
> Created attachment 599860 [details, diff] [details, diff]
> portage-bug_702806-r1.patch
> 
> Previous patch changed depclean behaviour, updated patch fixes that.

This patch is not valid because you're actually dropping dependencies here:

> +					if disjunctions_appended:
> +						disjunctions = []

[Comment 13 alfredthetomato 2019-12-16 19:42:52 UTC]

Created attachment 599862 [details, diff]
portage-bug_702806-r2.patch

Updated to hopefully not drop dependencies.

[Comment 14 Zac Medico 2019-12-16 20:15:25 UTC]

(In reply to alfredthetomato from comment #13)
> Created attachment 599862 [details, diff] [details, diff]
> portage-bug_702806-r2.patch
> 
> Updated to hopefully not drop dependencies.

Do you know why this changes the behavior? I don't want to merge this patch as-is because it goes against the intention of the fix for bug 701996, which is to group all of of the disjunctions (and virtuals) into a single _queue_disjunction call.

The behavior that you're observing is entirely expected, and I don't want to merge a patch to change the behavior unless there is a very good reason (and it must not conflict with other concerns such as the fix for bug 701996).

[Comment 15 alfredthetomato 2019-12-16 21:54:33 UTC]

(In reply to Zac Medico from comment #14)
> Do you know why this changes the behavior? I don't want to merge this patch
> as-is because it goes against the intention of the fix for bug 701996, which
> is to group all of of the disjunctions (and virtuals) into a single
> _queue_disjunction call.
> 
> The behavior that you're observing is entirely expected, and I don't want to
> merge a patch to change the behavior unless there is a very good reason (and
> it must not conflict with other concerns such as the fix for bug 701996).

Calling _queue_disjunction per || dependency but grouping the virtuals will result in the same behaviour before the fix for bug 701996. Conversely, calling _queue_disjunction per virtual dependency, but grouping || dependencies will result in the same behaviour after the commit. Therefore, it is sufficient to group just the || disjunctive dependencies together to fix bug 701996.

So my patch results in _queue_disjunction called two times instead of one: firstly with || disjunctions grouped and then for the second time for the virtual dependencies.

[Comment 16 Zac Medico 2019-12-16 22:12:38 UTC]

(In reply to alfredthetomato from comment #15)
> Calling _queue_disjunction per || dependency but grouping the virtuals will
> result in the same behaviour before the fix for bug 701996. Conversely,
> calling _queue_disjunction per virtual dependency, but grouping ||
> dependencies will result in the same behaviour after the commit. Therefore,
> it is sufficient to group just the || disjunctive dependencies together to
> fix bug 701996.

The intention of the code is to group all of the || dependencies together with the virtual dependencies so that they can all be expanded into DNF together, for the purposes of bug 632026 (there's an _expand_new_virtuals function that will expand a virtual dependency into its component || dependencies).

In this context, it's counter-productive to split virtual dependencies into separate _queue_disjunction calls.

> So my patch results in _queue_disjunction called two times instead of one:
> firstly with || disjunctions grouped and then for the second time for the
> virtual dependencies.

It can be called any number of times, depending on the composition of the dependency string, and this goes completely against intentions for solving cases like bug 632026, since we want to be able to generate a single DNF that combines all of the disjunctions found in the dependency string.

[Comment 17 Zac Medico 2019-12-16 22:25:45 UTC]

I don't think that your patch guarantees any particular behavior. Even if it appears to produce the "desired" result for the current state of your system at this time, something in the dependencies could easily change that causes it to produce a different behavior.

[Comment 18 Zac Medico 2020-02-01 22:22:29 UTC]

*** Bug 705700 has been marked as a duplicate of this bug. ***

[Comment 19 Zac Medico 2020-02-01 22:27:56 UTC]

I suppose it's possible to add some kind of special logic for *_TARGETS, as long as it doesn't re-introduce issues like bug 693790 or bug 706278.

[Comment 20 Zac Medico 2020-11-08 01:26:57 UTC]

*** Bug 753470 has been marked as a duplicate of this bug. ***

[Comment 21 Rafał Mużyło 2020-11-17 17:14:35 UTC]

*** Bug 754912 has been marked as a duplicate of this bug. ***

[Comment 22 Zac Medico 2020-12-07 08:38:39 UTC]

*** Bug 758878 has been marked as a duplicate of this bug. ***

[Comment 23 Stéphane Veyret 2021-01-17 08:42:31 UTC]

*** Bug 765598 has been marked as a duplicate of this bug. ***

[Comment 24 Erik Quaeghebeur 2021-02-11 16:55:36 UTC]

I also find this bug (the current behavior) to be an irritant and am glad that I stumbled on this bug report.

I do not have dev-lang/python in my world file, but it gets pulled in by other Python programs. They all have the default PYTHON_TARGETS set (python3_8), except python_exec, but that seems to be a special case, as it also has pypy3 set without pulling in dev-lang/pypy3. Nevertheless, Python 3.9 gets pulled in next to Python 3.8, updated, etc. without me ever using it. I see no logic there.

I do understand that portage evolved to exhibit this behavior, while fixing other bugs. But still, please consider getting this corner of portage to behave a but more as expected.

[Comment 25 Zac Medico 2021-05-16 00:16:00 UTC]

*** Bug 790431 has been marked as a duplicate of this bug. ***

[Comment 26 Zac Medico 2021-05-16 00:52:13 UTC]

(In reply to Erik Quaeghebeur from comment #24)
> I also find this bug (the current behavior) to be an irritant and am glad
> that I stumbled on this bug report.
> 
> I do not have dev-lang/python in my world file, but it gets pulled in by
> other Python programs. They all have the default PYTHON_TARGETS set
> (python3_8), except python_exec, but that seems to be a special case, as it
> also has pypy3 set without pulling in dev-lang/pypy3. Nevertheless, Python
> 3.9 gets pulled in next to Python 3.8, updated, etc. without me ever using
> it. I see no logic there.
> 
> I do understand that portage evolved to exhibit this behavior, while fixing
> other bugs. But still, please consider getting this corner of portage to
> behave a but more as expected.

The root problem is that you've grown accustomed to an inconsistent behavior of old portage versions. The old behavior was truly a bug, and there's no simple way to revert to the old behavior without introducing a risk of inconsistent results and missed updates.

That said, there's probably some way to model the behavior that you want, and I'm open to contributions of that nature.

[Comment 27 Zac Medico 2021-05-16 18:42:53 UTC]

Maybe you can overlay a masking layer that models the  *TARGET* USE_EXPAND vars correlation.

[Comment 28 Zac Medico 2021-05-16 18:56:02 UTC]

The behavior of *TARGET* USE_EXPAND settings is not covered by PMS, so first we'll have to specify the behavior of *TARGET* USE_EXPAND settings.

[Comment 29 Zac Medico 2021-05-16 19:23:36 UTC]

This is another case like bug 283587, where we'll have extend the package manager specification in order to allow deeper integration with the Gentoo ecosystem.

[Comment 30 Larry the Git Cow 2021-11-11 21:04:21 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3d081cb997d18d200c71e1ff394e17db2f01f38d

commit 3d081cb997d18d200c71e1ff394e17db2f01f38d
Author:     Sam James <[e-mail]>
AuthorDate: 2021-11-11 21:02:52 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2021-11-11 21:02:52 +0000

    sys-libs/glibc: drop Python 3.10 from PYTHON_COMPAT to avoid circular deps
    
    Needed for now to try help upgrades and avoid circular dependencies
    with glibc -> python -> libcrypt -> libxcrypt -> glibc -> ...
    
    Bug: https://bugs.gentoo.org/699422
    Bug: https://bugs.gentoo.org/702806
    Bug: https://bugs.gentoo.org/809410
    Signed-off-by: Sam James <[e-mail]>

 sys-libs/glibc/glibc-2.33-r7.ebuild | 5 ++++-
 sys-libs/glibc/glibc-2.34-r1.ebuild | 5 ++++-
 sys-libs/glibc/glibc-9999.ebuild    | 5 ++++-
 3 files changed, 12 insertions(+), 3 deletions(-)