Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 651018 - sys-apps/portage: emerge --emptytree @world 'forgets' a dependency that doesn't satisfy USE requirements
Summary: sys-apps/portage: emerge --emptytree @world 'forgets' a dependency that doesn...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: Normal major with 2 votes (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, PullRequest
: 653970 891223 (view as bug list)
Depends on: 925214
Blocks: 300071
  Show dependency tree
 
Reported: 2018-03-20 19:52 UTC by calimeroteknik
Modified: 2024-02-29 17:46 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description calimeroteknik 2018-03-20 19:52:58 UTC
In order to install dev-java/icedtea-bin, I once put the following lines in /etc/portage/package.use:
dev-java/icedtea-bin -webstart headless-awt
app-text/ghostscript-gpl cups


Today I ran my routine update/check process:
# (no emerge --sync here)
emerge -av --update --deep --newrepo --newuse --changed-deps @world
emerge -av --depclean # note this
(no errors happened)

Then I removed this line from /etc/portage/package.use:
app-text/ghostscript-gpl cups

And re-ran:
~ # emerge -av --update --deep --newrepo --newuse --changed-deps @world

These are the packages that would be merged, in order:

Calculating dependencies... done!

Total: 0 packages, Size of downloads: 0 KiB

Nothing to merge; quitting.


To be frank, I expected an error, so in doubt, I tried the following:

~ # emerge -av1 app-text/ghostscript-gpl

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] app-text/ghostscript-gpl-9.21  USE="-cups*" 

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

app-text/ghostscript-gpl:0

  (app-text/ghostscript-gpl-9.21:0/0::gentoo, ebuild scheduled for merge) pulled in by
    app-text/ghostscript-gpl (Argument)

  (app-text/ghostscript-gpl-9.21:0/0::gentoo, installed) pulled in by
    >=app-text/ghostscript-gpl-9.09[cups] required by (net-print/cups-filters-1.17.9:0/0::gentoo, installed)
                                    ^^^^                                                                                                        


It might be possible to solve this slot collision
by applying all of the following changes:
   - app-text/ghostscript-gpl-9.21 (Change USE: +cups)


My portage tree wasn't super fresh, and this was portage 2.3.19.
So I updated:
emerge -av --sync # this told me a new version of portage was available
emerge -av1 portage # this updated 2.3.19 -> 2.3.24
emerge -av --update --deep --newrepo --newuse --changed-deps @world
emerge -av --depclean # note this
(no errors happened running all of this)

Re-running emerge -av1 app-text/ghostscript-gpl produces the same result as previously (character for character, I ran diff on the two outputs).

And, the output of `emerge -epv @world`, which does not produce any error, does not include ghostscript-gpl.
It does however include cups-filters, which pulls ghostscript-gpl:

~ # emerge -cpv ghostscript-gpl

Calculating dependencies... done!
  app-text/ghostscript-gpl-9.21 pulled in by:
    net-print/cups-filters-1.17.9 requires >=app-text/ghostscript-gpl-9.09[cups]

>>> No packages selected for removal by depclean
Packages installed:   509
Packages in world:    88
Packages in system:   44
Required packages:    509
Number to remove:     0


If I put "app-text/ghostscript-gpl cups" back into /etc/portage/package.use, then `emerge -epv @world` includes ghostscript-gpl as expected.

Possibly related: https://bugs.gentoo.org/show_bug.cgi?id=559224
Comment 1 Zac Medico gentoo-dev 2018-03-20 22:12:11 UTC
This is practically the same issue as bug 266836, just triggered by --emptytree instead of --newuse.
Comment 2 Zac Medico gentoo-dev 2018-03-29 07:00:27 UTC
This problem is well understood, it's due to the depgraph _add_dep method trying very hard to satisfy dependencies:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/_emerge/depgraph.py?h=portage-2.3.26#n2765

Rather than come up empty-handed, it selects an installed package to satisfy the dependency. This sort of behavior could be desirable in some cases. For example, consider that some people might want to be able to continue despite an ebuild or binary package unavailable, while the installed instance satisfies the dependency. We should definitely provide an option to make emerge bail out here, though.
Comment 3 Zac Medico gentoo-dev 2018-03-29 07:08:11 UTC
See this code where for --emptytree it does not pass up the installed package if the matched_packages list is empty:

https://gitweb.gentoo.org/proj/portage.git/tree/pym/_emerge/depgraph.py?h=portage-2.3.26#n6220

if installed and not find_existing_node:
	want_reinstall = reinstall or empty or \
		(found_available_arg and not selective)
	if want_reinstall and matched_packages:
		continue

If you want emerge to bail out, you need to reject that installed package.
Comment 4 Zac Medico gentoo-dev 2018-04-26 19:44:19 UTC
*** Bug 653970 has been marked as a duplicate of this bug. ***
Comment 5 Raffaello D. Di Napoli 2024-01-20 20:18:57 UTC
(Coming from bug 922343 which zmedico says to be related, and I see why; my repro ebuilds from that bug also reproduce this one.)

(In reply to Zac Medico from comment #2)
> Rather than come up empty-handed, it selects an installed package to satisfy
> the dependency.

Shouldn’t --emptytree disregard any installed packages? Isn’t that the whole point to requesting an empty tree for dependency resolution, IOW why are installed packages being considered at all?
Comment 6 Zac Medico gentoo-dev 2024-01-20 20:40:21 UTC
(In reply to Raffaello D. Di Napoli from comment #5)
> (In reply to Zac Medico from comment #2)
> > Rather than come up empty-handed, it selects an installed package to satisfy
> > the dependency.
> 
> Shouldn’t --emptytree disregard any installed packages? Isn’t that the whole
> point to requesting an empty tree for dependency resolution, IOW why are
> installed packages being considered at all?

In practice, --emptytree is more about rebuilding packages than disregarding installed packages. The installed packages are used to satisfy dependencies for the rebuilds, and in some cases installed packages are used to solve circular dependencies (there is a large runtime dependency cycle involving glibc). It's also possible to use the --exclude option to exclude installed packages from being rebuilt, but these installed packages can still be used to satisfy dependencies.
Comment 7 Raffaello D. Di Napoli 2024-01-20 21:57:07 UTC
(In reply to Zac Medico from comment #6)
> (In reply to Raffaello D. Di Napoli from comment #5)
> > (In reply to Zac Medico from comment #2)
> > > Rather than come up empty-handed, it selects an installed package to satisfy
> > > the dependency.
> > 
> > Shouldn’t --emptytree disregard any installed packages? Isn’t that the whole
> > point to requesting an empty tree for dependency resolution, IOW why are
> > installed packages being considered at all?
> 
> In practice, --emptytree is more about rebuilding packages than disregarding
> installed packages.

Not to be pedantic, but from man emerge:

> --emptytree, -e
>    Reinstalls target atoms and their entire deep dependency tree, as though no
>    packages are currently installed.

“As though no packages are installed” makes me think installed packages should be disregarded. Otherwise the man page is wrong or at least ambiguous.
Comment 8 Zac Medico gentoo-dev 2024-01-20 22:19:15 UTC
(In reply to Raffaello D. Di Napoli from comment #7)
> “As though no packages are installed” makes me think installed packages
> should be disregarded. Otherwise the man page is wrong or at least ambiguous.

Yes, I agree. The language came from this commit:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=1d73d80236e380f26f32f1737cc2f36d980715d1

commit 1d73d80236e380f26f32f1737cc2f36d980715d1
Author: Zac Medico <zmedico@gentoo.org>
Date:   2010-03-16 06:17:57 +0000

    Bug #309699 - Update --emptytree docs. Thanks to Ben Kohler <bkohler@gmail.com>
    for wording suggestions.
    
    svn path=/main/trunk/; revision=15831
Comment 9 Gábor Oszkár Dénes 2024-02-14 18:41:20 UTC
See also: https://github.com/gentoo/portage/pull/1272
Comment 10 Gábor Oszkár Dénes 2024-02-14 18:55:58 UTC
(In reply to Zac Medico from comment #6)
> (In reply to Raffaello D. Di Napoli from comment #5)
> > (In reply to Zac Medico from comment #2)
> > > Rather than come up empty-handed, it selects an installed package to satisfy
> > > the dependency.
> > 
> > Shouldn’t --emptytree disregard any installed packages? Isn’t that the whole
> > point to requesting an empty tree for dependency resolution, IOW why are
> > installed packages being considered at all?
> 
> In practice, --emptytree is more about rebuilding packages than disregarding
> installed packages. The installed packages are used to satisfy dependencies
> for the rebuilds, and in some cases installed packages are used to solve
> circular dependencies (there is a large runtime dependency cycle involving
> glibc). It's also possible to use the --exclude option to exclude installed
> packages from being rebuilt, but these installed packages can still be used
> to satisfy dependencies.

I have a limited experience with Portage codebase, but I'm not sure this is true. Is it true that all installed packages in `depgraph:_wrapped_select_pkg_highest_available_imp` have `pkg.operation == "nomerge"`? And `"nomerge"` packages are not selected for install at the end, right? In this case no installed packages would be reinstalled.

I might be wrong, this is only a guess.
Comment 11 Zac Medico gentoo-dev 2024-02-14 19:05:40 UTC
(In reply to Gábor Oszkár Dénes from comment #10)
> Is it true that all installed packages in
> `depgraph:_wrapped_select_pkg_highest_available_imp` have `pkg.operation ==
> "nomerge"`? And `"nomerge"` packages are not selected for install at the
> end, right? In this case no installed packages would be reinstalled.

Right, `pkg.operation == "nomerge"` is always true for installed packages and if _wrapped_select_pkg_highest_available_imp selects such a package then it intends not to reinstall it.
Comment 12 Gábor Oszkár Dénes 2024-02-14 19:30:38 UTC
(In reply to Zac Medico from comment #11)
> Right, `pkg.operation == "nomerge"` is always true for installed packages
> and if _wrapped_select_pkg_highest_available_imp selects such a package then
> it intends not to reinstall it.

Thanks. Now it makes sense, I misunderstood slightly one of your last comments. I also updated the PR to handle --excluded.

Is there any other circumstance when we allow the installed package to be used as a satisfied dependency? I do not think there's any other elegant way to handle this, other than adding it in the conditional.
Comment 13 Zac Medico gentoo-dev 2024-02-14 19:56:25 UTC
(In reply to Gábor Oszkár Dénes from comment #12)
> Is there any other circumstance when we allow the installed package to be
> used as a satisfied dependency? I do not think there's any other elegant way
> to handle this, other than adding it in the conditional.

There may be some places where depgraph will call _select_pkg_from_installed to satisfy unsatisfied dependencies, and the depgraph _solve_non_slot_operator_slot_conflicts method could possibly eliminate a replacement package in favor of an installed package in order to solve a slot conflict.
Comment 14 Larry the Git Cow gentoo-dev 2024-02-23 04:39:32 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=92ff02b9189f8350f44e134d538319e4037f3f71

commit 92ff02b9189f8350f44e134d538319e4037f3f71
Author:     Gábor Oszkár Dénes <gaboroszkar@protonmail.com>
AuthorDate: 2024-02-14 17:40:24 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-23 04:39:26 +0000

    emerge: Skip installed packages with emptytree in depgraph selection
    
    Running emerge with emptytree tries to find the best match for every
    atom it needs to install. Sometimes the best matches would be
    already installed packages (with `operation=nomerge`), but these
    packages would be silently skipped with full emptytree installation.
    This change makes sure that emerge attempts to install every package.
    If the package has unmet requirements, emerge will complain.
    
    Bug: https://bugs.gentoo.org/651018
    Signed-off-by: Gábor Oszkár Dénes <gaboroszkar@protonmail.com>
    Closes: https://github.com/gentoo/portage/pull/1272
    Signed-off-by: Sam James <sam@gentoo.org>

 NEWS                                               |   6 +
 lib/_emerge/depgraph.py                            |  13 ++
 lib/portage/tests/resolver/test_depth.py           |   8 +-
 .../test_emptytree_reinstall_unsatisfiability.py   | 137 +++++++++++++++++++++
 lib/portage/tests/resolver/test_useflags.py        |   6 +-
 5 files changed, 166 insertions(+), 4 deletions(-)
Comment 15 Larry the Git Cow gentoo-dev 2024-02-24 03:36:41 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=01d06eb1d9dc8c4b16cbc9a6567ed0c07df5901a

commit 01d06eb1d9dc8c4b16cbc9a6567ed0c07df5901a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-02-24 02:45:58 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-02-24 03:28:24 +0000

    Fix python 3.9 CI jobs since 92ff02b9189f
    
    Use emerge -e to fix this error we've seen since
    92ff02b9189f for python 3.9 CI jobs:
    
    https://github.com/gentoo/portage/actions/runs/8014796128/job/21893963019
    
    test_portage_baseline[binhost emerge-gpkg] - AssertionError: 'emerge' failed with args '('-e', '--getbinpkgonly', 'dev-libs/A')'
    
    emerge: there are no binary packages to satisfy "dev-libs/B[flag]".
    (dependency required by "dev-libs/A-1::test_repo" [binary])
    (dependency required by "dev-libs/A" [argument])
    
    Fixes: 92ff02b9189f ("emerge: Skip installed packages with emptytree in depgraph selection")
    Bug: https://bugs.gentoo.org/651018
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/tests/emerge/conftest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 16 Gábor Oszkár Dénes 2024-02-24 08:21:28 UTC
(In reply to Larry the Git Cow from comment #15)
> The bug has been referenced in the following commit(s):
> 
> https://gitweb.gentoo.org/proj/portage.git/commit/
> ?id=01d06eb1d9dc8c4b16cbc9a6567ed0c07df5901a

Why did it only break Python 3.9 pipeline? My intuition tells me the fix for that should be to change how the tests run *only* under this Python version, but Zac Medico seemed to be changing that for every Python version?
Comment 17 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-02-24 08:40:22 UTC
(In reply to Gábor Oszkár Dénes from comment #16)
> (In reply to Larry the Git Cow from comment #15)
> > The bug has been referenced in the following commit(s):
> > 
> > https://gitweb.gentoo.org/proj/portage.git/commit/
> > ?id=01d06eb1d9dc8c4b16cbc9a6567ed0c07df5901a
> 
> Why did it only break Python 3.9 pipeline? My intuition tells me the fix for
> that should be to change how the tests run *only* under this Python version,
> but Zac Medico seemed to be changing that for every Python version?

The intutition should be to figure out why it changes.

If Zac thinks the change is right anyway, I'm happy to trust him, but I wouldn't say the right fix is to then just conditionalise it per-Python version without understanding why. That's kind of the worst of both worlds (it means it really is acting wrongly sometimes, rather than "well, we actually need -e anyway, and only py3.9 noticed" (which is still not great but it's a different category)).
Comment 18 Gábor Oszkár Dénes 2024-02-24 08:50:59 UTC
> If Zac thinks the change is right anyway, I'm happy to trust him

Sorry, I didn't mean to say that I don't trust his change. I was just simply trying to understand it. Of course he knows how Portage works and what these tests do.

> I wouldn't say the right fix is to then just conditionalise it per-Python
> version without understanding why.

Of course, understanding why this happens should be first before actually fixing it (or suggesting a fix). I wouldn't suggest adding a condition to check for Python version either, but rather, if a behavior of the program is different, we should change that exact behavior (which is most probably Python specific, maybe a function signature or default argument change, etc.).

All in all I just wanted to be terse, and ask the question, not to challange Zac's actions.
Comment 19 Zac Medico gentoo-dev 2024-02-24 12:58:10 UTC
(In reply to Gábor Oszkár Dénes from comment #16)
> (In reply to Larry the Git Cow from comment #15)
> > The bug has been referenced in the following commit(s):
> > 
> > https://gitweb.gentoo.org/proj/portage.git/commit/
> > ?id=01d06eb1d9dc8c4b16cbc9a6567ed0c07df5901a
> 
> Why did it only break Python 3.9 pipeline? My intuition tells me the fix for
> that should be to change how the tests run *only* under this Python version,
> but Zac Medico seemed to be changing that for every Python version?

My change is valid for every Python version because the intent is to populate the binary packages completely for use in the emerge -e --getbinpkgonly command that comes next.

I don't fully understand why it behaved differently under Python 3.9, so I would encourage further investigation here, but it's annoying to have test breakage like this in the master branch so I would encourage experimentation in a another branch like Sam was doing in https://github.com/gentoo/portage/pull/1277 for example.
Comment 20 Gábor Oszkár Dénes 2024-02-25 08:10:36 UTC
For reference: see https://github.com/gentoo/portage/pull/1281#issuecomment-1962731145 for the resolution of why tests failed only intermittently.
Comment 21 Larry the Git Cow gentoo-dev 2024-02-25 08:33:01 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6ad9d3103abc02f60d9e123ae21fa4a2e69b7e38

commit 6ad9d3103abc02f60d9e123ae21fa4a2e69b7e38
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-02-25 08:32:40 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-25 08:32:47 +0000

    sys-apps/portage: add 3.0.63
    
    Closes: https://bugs.gentoo.org/925214
    Closes: https://bugs.gentoo.org/651018
    Closes: https://bugs.gentoo.org/922935
    Closes: https://bugs.gentoo.org/925240
    Closes: https://bugs.gentoo.org/925311
    Closes: https://bugs.gentoo.org/925333
    Closes: https://bugs.gentoo.org/925350
    Closes: https://bugs.gentoo.org/925456
    Closes: https://bugs.gentoo.org/925460
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.63.ebuild | 246 +++++++++++++++++++++++++++++++++
 2 files changed, 247 insertions(+)
Comment 22 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-02-28 16:00:02 UTC
*** Bug 891223 has been marked as a duplicate of this bug. ***