Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 48195 - Dep calculation does not use installed package information
Summary: Dep calculation does not use installed package information
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High critical (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 40089 42334 52280 77048 77669 79578 94724 124056 128161 128503 128969 130018 130560 131276 132388 133210 133865 139604 141304 141484 141795 143658 143665 147481 147730 148300 (view as bug list)
Depends on:
Blocks: 155723 147007
  Show dependency tree
 
Reported: 2004-04-18 00:56 UTC by Joan
Modified: 2007-02-16 07:29 UTC (History)
39 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
allow installed packages to satisfy dependencies (installed_deps.patch,1.38 KB, patch)
2006-07-06 18:01 UTC, Zac Medico
Details | Diff
allow installed packages to satisfy dependencies (installed_deps.patch,3.37 KB, patch)
2006-07-07 00:44 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joan 2004-04-18 00:56:07 UTC
emerge --depclean removes the libacl library, then the system crashes.

Reproducible: Always
Steps to Reproduce:
1.
2.
3.
Comment 1 Robert Moss (RETIRED) gentoo-dev 2004-04-21 17:29:47 UTC
Just wanted to elaborate a little here. Please change to all hardware versions as this is not only a PPC problem - it's certainly at least also x86.

The base system installation links against libacl.so and libattr.so. emerge depclean removes libacl and libattr, so if you're using stage2 (I think) or stage 3 (definitely) then you're going to screw yourself over. In particular, coreutils links against libacl.so and libattr.so. Perhaps it would make more sense for either the stage 2 or stage 3 coreutils to be compiled without ACL support, or for libacl and libattr to be added to the base system profile? Otherwise, we're going to end up with a lot of broken installations, very quickly indeed...

Of course, this breaks revdep-rebuild and even emerge, so you can't re-emerge coreutils if it breaks this way.
Comment 2 Nicholas Jones (RETIRED) gentoo-dev 2004-04-22 15:57:05 UTC
So... killing libacl is a problem with the default system and depclean...
We won't have a good fix in portage for a while as it requires reverse
deps and some kind of advanced cleaning knowledge...
Comment 3 Jason Stubbs (RETIRED) gentoo-dev 2004-04-23 02:07:01 UTC
This sort of problem is the main reason why --depclean gives a big scary warning and pauses for 10 seconds before doing anything.

Nick, my rewrite of deptree solves this problem. It's still not completely safe, though, as programs may link to software that they don't *DEPEND on. I would call those issues ebuild bugs, though.

The only way to do it in a 100% safe manner is to ignore USE and *DEPEND and calculate what can be removed in the same manner as revdep-rebuild.

Release people, are stage3 tarballs compiled with the default USE flags for the profile? If not, they probably should be. If so, has a change in default USE flags caused this?

Will re-assign to dev-portage after that issue is sorted.
Comment 4 Björn Michaelsen 2004-04-28 04:19:39 UTC
stage3 and probably GRP (2004.0) used USE="acl" while later the default USE doesnt include USE="acl", so doing a Stage 3+GRP install and a depclean afterwards without changing any USE-Flags results in a broken system (own expirience).
As for a complete revdep-rebuild-like check, this takes time, - maybe it is enough to check the dependencies of "emerge system" to make sure that a least portage and the compiler toolchain etc. works - everything else isnt to hard to fix if portage works
Comment 5 Glenn Ross 2004-05-01 09:05:49 UTC
*** Bug 40089 has been marked as a duplicate of this bug. ***
Comment 6 Björn Michaelsen 2004-05-04 01:08:07 UTC
rescue binaries are available here:
http://dev.gentoo.org/~avenj/bins/
(attr and acl)
Comment 7 Thomas Kuehling 2004-05-07 11:47:37 UTC
I just had the problem on amd64-arch. I copied the libacl.* files out of the stage3-amd64-2004.1 package into the correct path on my system and now everything works again.

The files i meant you could find here: http://debianer.org/~fek/bug_48195/
Comment 8 Brian Harring gentoo-dev 2004-05-28 11:19:56 UTC
*** Bug 52280 has been marked as a duplicate of this bug. ***
Comment 9 John Davis (zhen) (RETIRED) gentoo-dev 2004-06-13 20:11:26 UTC
yeah, for some reason, GRP and stages use acl... this is pre "me" working on catalyst, so I have no idea why that flag is set ..
Comment 10 Robert Moss (RETIRED) gentoo-dev 2004-07-20 14:42:11 UTC
I realise this is possibly the most disgusting script you'll ever have seen, but it could be useful in this case...

http://home.jesus.ox.ac.uk/~rmoss/portage/rtdeps

What that basically does is work out what a particular (specified) package links against, and generates a list of run-time dependencies. It doesn't yet know how to deal with scripting languages, but I'm not sure if that's really feasible. So, what could be done to solve this is a rewrite of this in Python to be added to the depclean code, which checked the base system packages and created a depgraph using those. This would ensure that there was absolutely no chance whatsoever of linker-related issues such as the above and would be genuinely "safe." I don't know whether this is anything like the revdep-rebuild stuff as I haven't looked at it.

I'm afraid I don't really know any Python, so rewriting that script in said language is probably beyond me, at least for now, I'm afraid :-(
Comment 11 ta2002 2004-09-10 23:14:24 UTC
This sounds like the same problem I ran into today:

# qpkg -I -v kdebase 
kde-base/kdebase-3.2.3-r1 * 

# dog -l 7-15 /usr/portage/kde-base/kdebase/kdebase-3.2.3-r1.ebuild 
DESCRIPTION="KDE base packages: the desktop, panel, window manager, konqueror..." 

KEYWORDS="x86 ppc sparc ~mips ~alpha hppa amd64 ~ia64" 
IUSE="ldap pam motif encode cups ssl opengl samba java" 

DEPEND="media-sound/cdparanoia 
        >=kde-base/kdelibs-3.2.3-r1 
        ldap? ( net-nds/openldap ) 
        pam? ( sys-libs/pam ) 

# qpkg -q cdparanoia-3.9.8-r1 
media-sound/cdparanoia-3.9.8-r1 * 
DEPENDED ON BY: 
        kdebase-3.2.3-r1 
        kdemultimedia-3.2.3 
        mplayer-1.0_pre4-r7 
media-sound/cdparanoia-3.9.8-r1 
DEPENDED ON BY: 
        kde-base/kdebase-3.2.3-r1 
        kde-base/kdemultimedia-3.2.3 
        media-video/mplayer-1.0_pre4-r7 

# emerge -p depclean 

[...] 

Calculating depclean dependencies ... done! 

>>> These are the packages that I would unmerge: 

 media-sound/cdparanoia 
    selected: 3.9.8-r1 
   protected: none 
     omitted: none 

>>> 'Selected' packages are slated for removal. 
>>> 'Protected' and 'omitted' packages will not be removed. 

[...]


It seems that qpkg knows that cdparanoia belongs on the
system - but portage itself does not. :(
Comment 12 Chris Gianelloni (RETIRED) gentoo-dev 2004-11-19 16:29:57 UTC
We have solved this is relation to the release by ensuring our USE flags used to create the stages/LiveCD do not cause this sort of problem and by removing USE=acl from the GRP USE
Comment 13 Alec Warner archtester Gentoo Infrastructure gentoo-dev Security 2005-08-06 07:50:49 UTC
*** Bug 77669 has been marked as a duplicate of this bug. ***
Comment 14 Jason Stubbs (RETIRED) gentoo-dev 2005-11-20 19:02:21 UTC
*** Bug 42334 has been marked as a duplicate of this bug. ***
Comment 15 Jason Stubbs (RETIRED) gentoo-dev 2006-02-25 06:08:46 UTC
*** Bug 124056 has been marked as a duplicate of this bug. ***
Comment 16 Jason Stubbs (RETIRED) gentoo-dev 2006-04-02 05:22:49 UTC
*** Bug 128503 has been marked as a duplicate of this bug. ***
Comment 17 Donnie Berkholz (RETIRED) gentoo-dev 2006-04-04 12:47:03 UTC
*** Bug 128809 has been marked as a duplicate of this bug. ***
Comment 18 Jakub Moc (RETIRED) gentoo-dev 2006-04-05 14:59:51 UTC
*** Bug 128969 has been marked as a duplicate of this bug. ***
Comment 19 Jakub Moc (RETIRED) gentoo-dev 2006-04-15 04:10:46 UTC
*** Bug 130018 has been marked as a duplicate of this bug. ***
Comment 20 Simon Stelling (RETIRED) gentoo-dev 2006-04-15 05:39:48 UTC
*** Bug 77048 has been marked as a duplicate of this bug. ***
Comment 21 Donnie Berkholz (RETIRED) gentoo-dev 2006-04-19 20:58:50 UTC
*** Bug 128161 has been marked as a duplicate of this bug. ***
Comment 22 Jakub Moc (RETIRED) gentoo-dev 2006-04-20 00:58:46 UTC
*** Bug 130560 has been marked as a duplicate of this bug. ***
Comment 23 Marius Mauch (RETIRED) gentoo-dev 2006-05-06 02:50:25 UTC
*** Bug 132388 has been marked as a duplicate of this bug. ***
Comment 24 Antti Mäkelä 2006-05-06 07:18:12 UTC
Look, is anything going to be done about this any time soon? There are gazillion duplicates and this one has been open since 2004. Considering that the bug is critical, there is very little progress..

For myself, whenever this issue arises I just copy the needed ebuild from /var/db/pkg/category/package to /usr/local/portage/category/package, do an ebuild digest and continue. Could someone tell me if assigning /var/db/pkg to the list of PORTAGE_OVERLAY directories would work and fix the problem permanently or will it break something?
Comment 25 Jakub Moc (RETIRED) gentoo-dev 2006-05-13 08:26:19 UTC
*** Bug 133210 has been marked as a duplicate of this bug. ***
Comment 26 Jakub Moc (RETIRED) gentoo-dev 2006-05-20 05:33:20 UTC
*** Bug 133865 has been marked as a duplicate of this bug. ***
Comment 27 Zac Medico gentoo-dev 2006-07-06 14:18:02 UTC
(In reply to comment #24)
> Could someone tell me if assigning /var/db/pkg to
> the list of PORTAGE_OVERLAY directories would work and fix the problem
> permanently or will it break something?

That's an amusing idea.  I might work with FEATURES=-strict but I certainly wouldn't recommend it.

Just now I was looking into what it would take to fix this bug correctly.  The resolver needs to be refactored so that is views packages in the same way whether they are ebuilds, binary packages, or installed packages.  Currently, it handles ebuilds and binaries as separate cases (rather than as abstract packages), which makes it unnecessarily difficult to add one or more new package types.
Comment 28 Marius Mauch (RETIRED) gentoo-dev 2006-07-06 14:54:44 UTC
(In reply to comment #27)
> (In reply to comment #24)
> > Could someone tell me if assigning /var/db/pkg to
> > the list of PORTAGE_OVERLAY directories would work and fix the problem
> > permanently or will it break something?
> 
> That's an amusing idea.  I might work with FEATURES=-strict but I certainly
> wouldn't recommend it.

It wouldn't work, the directory structure is different (cat/package vs. cat/package-version)
Comment 29 Zac Medico gentoo-dev 2006-07-06 18:01:27 UTC
Created attachment 91086 [details, diff]
allow installed packages to satisfy dependencies

Here's a pretty simple patch that seems to correct the problem.  However, I don't want to apply it until bug 122089 is fixed.
Comment 30 Zac Medico gentoo-dev 2006-07-06 18:54:11 UTC
Well, that patch doesn't work right.  But anyway, I'll keep trying.
Comment 31 Zac Medico gentoo-dev 2006-07-07 00:44:10 UTC
Created attachment 91100 [details, diff]
allow installed packages to satisfy dependencies

With the little testing I've done on this patch, it seems to be completely functional (including --newuse support).  Now I just need to finish up the /var/db/pkg maintenance tool for bug 122089.
Comment 32 Zac Medico gentoo-dev 2006-07-07 16:25:15 UTC
I'll be keeping the latest versions of the patch in my dev space until it's been merged:

http://dev.gentoo.org/~zmedico/portage/branches/2.1/patches/installed_deps/

One notable change that I've made is to include built time dependencies in the calculation (though it's not necessarily required for installed packages).  This makes the resolver behave more like it does without the patch.  This behavior should eventually be controllable via the fix for bug 132355.

If you'd like to try the patch then you will likely need to run the maintenance script that I've prepared for bug 122089.  It's available here:

http://dev.gentoo.org/~zmedico/portage/branches/2.1/bin/vdb-tools.py

Comment 33 Jakub Moc (RETIRED) gentoo-dev 2006-07-12 23:49:25 UTC
*** Bug 139604 has been marked as a duplicate of this bug. ***
Comment 34 Jakub Moc (RETIRED) gentoo-dev 2006-07-21 10:14:09 UTC
*** Bug 141304 has been marked as a duplicate of this bug. ***
Comment 35 Jakub Moc (RETIRED) gentoo-dev 2006-07-23 05:22:37 UTC
*** Bug 141484 has been marked as a duplicate of this bug. ***
Comment 36 Jakub Moc (RETIRED) gentoo-dev 2006-07-26 07:15:05 UTC
*** Bug 141795 has been marked as a duplicate of this bug. ***
Comment 37 Jakub Moc (RETIRED) gentoo-dev 2006-08-12 05:31:57 UTC
*** Bug 143658 has been marked as a duplicate of this bug. ***
Comment 38 Zac Medico gentoo-dev 2006-08-13 10:57:52 UTC
*** Bug 143665 has been marked as a duplicate of this bug. ***
Comment 39 Jakub Moc (RETIRED) gentoo-dev 2006-09-13 11:22:29 UTC
*** Bug 147481 has been marked as a duplicate of this bug. ***
Comment 40 Jason Stubbs (RETIRED) gentoo-dev 2006-09-14 08:26:28 UTC
*** Bug 131276 has been marked as a duplicate of this bug. ***
Comment 41 Jakub Moc (RETIRED) gentoo-dev 2006-09-15 14:29:40 UTC
*** Bug 147730 has been marked as a duplicate of this bug. ***
Comment 42 Jakub Moc (RETIRED) gentoo-dev 2006-09-20 03:57:17 UTC
*** Bug 148300 has been marked as a duplicate of this bug. ***
Comment 43 Zac Medico gentoo-dev 2006-10-06 03:31:15 UTC
This is fixed in svn r4605.  The implementation only uses the installed package metadata (*DEPEND) when there is no other choice available.  Global updates are automatically performed on *DEPEND if necessary in order to clean up stale atoms (we do single packages as necessary because it's too time consuming to do the whole vdb at once).
Comment 44 Zac Medico gentoo-dev 2006-10-06 04:42:56 UTC
This has been released in 2.1.2_pre2-r5.
Comment 45 Zac Medico gentoo-dev 2006-10-06 11:26:20 UTC
*** Bug 150251 has been marked as a duplicate of this bug. ***
Comment 46 Zac Medico gentoo-dev 2006-10-06 18:05:33 UTC
*** Bug 150251 has been marked as a duplicate of this bug. ***
Comment 47 Jakub Moc (RETIRED) gentoo-dev 2006-11-19 00:44:12 UTC
*** Bug 155634 has been marked as a duplicate of this bug. ***
Comment 48 Marius Mauch (RETIRED) gentoo-dev 2007-01-11 14:29:52 UTC
*** Bug 79578 has been marked as a duplicate of this bug. ***
Comment 49 Marius Mauch (RETIRED) gentoo-dev 2007-01-12 06:09:36 UTC
*** Bug 94724 has been marked as a duplicate of this bug. ***
Comment 50 Antti Mäkelä 2007-02-12 11:30:38 UTC
Ok, I now tested the new portage functionality in 2.1.2. I have ck-sources 2.6.19_p2 (sys-kernel/ck-sources-2.6.19_p2) installed. The tree no longer has it.

I have =sys-kernel/ck-sources-2.6.19_p2 in my package.keywords. When I do emerge -uvDaN world, emerge wants to install gentoo-sources.

Only after I put gentoo-sources to package.mask, will emerge decide that it does not want to install any kernel sources after all (as I intended).

However, still not sure if this is intended behavior...Still, thanks for the fix!
Comment 51 Chris Slycord 2007-02-13 19:56:55 UTC
(In reply to comment #50)
> Ok, I now tested the new portage functionality in 2.1.2. I have ck-sources
> 2.6.19_p2 (sys-kernel/ck-sources-2.6.19_p2) installed. The tree no longer has
> it.
> 
> I have =sys-kernel/ck-sources-2.6.19_p2 in my package.keywords. When I do
> emerge -uvDaN world, emerge wants to install gentoo-sources.
> 
> Only after I put gentoo-sources to package.mask, will emerge decide that it
> does not want to install any kernel sources after all (as I intended).
> 
> However, still not sure if this is intended behavior...Still, thanks for the
> fix!
> 

That is the intended behavior. If you want ck-sources-2.6.19_p2 then you must put it in your local overlay.
Comment 52 Antti Mäkelä 2007-02-13 23:19:13 UTC
(In reply to comment #51)
> That is the intended behavior. If you want ck-sources-2.6.19_p2 then you must
> put it in your local overlay.

  No I don't. I just need to mask gentoo-sources to prevent the fallback to even that. Before this fix went in, yes, I did have to copy stuff to local overlay. 
  Now ck-sources-2.6.19_p2 dependency gets satisfied because it is in the installed packages.

  I just found a bit strange is that it's apparently only the ultimate fallback (since emerge suggests gentoo-sources unless explicitly denied).
Comment 53 Chris Slycord 2007-02-14 04:21:33 UTC
(In reply to comment #52)
> (In reply to comment #51)
> > That is the intended behavior. If you want ck-sources-2.6.19_p2 then you must
> > put it in your local overlay.
> 
>   No I don't. I just need to mask gentoo-sources to prevent the fallback to
> even that. Before this fix went in, yes, I did have to copy stuff to local
> overlay. 
>   Now ck-sources-2.6.19_p2 dependency gets satisfied because it is in the
> installed packages.
> 
>   I just found a bit strange is that it's apparently only the ultimate fallback
> (since emerge suggests gentoo-sources unless explicitly denied).
> 

Sure, I guess you could mask gentoo-sources. But the behavior that it exhibited is not wrong whatsoever. You have a kernel source installed that isn't in portage. Then you do something like "emerge world --update --deep" and it tries to install the newest stable kernel source. Sure, the installed kernel sources satisfy the dependency... but when you tell it "--update --deep" it's gonna update packages and their dependencies. So in this case, it updates the kernel sources.

If I'm wrong and you really did something else, then please explain.
Comment 54 Avuton Olrich 2007-02-14 04:31:20 UTC
man portage, search for package.provided.
Comment 55 Antti Mäkelä 2007-02-14 07:05:44 UTC
(In reply to comment #53)
> to install the newest stable kernel source. Sure, the installed kernel sources
> satisfy the dependency... but when you tell it "--update --deep" it's gonna
> update packages and their dependencies. So in this case, it updates the kernel
> sources.

  That explains it. Should have checked --tree. Kinda odd though that even build-time dependencies are not satisfied with installed packages that are gone from the tree.

  Anyway, just to make clear what I'm going for:

Situation before this patch went in:

1) Install ck-sources, version X
2) Version X, vanishes from tree, X-r1 comes in (and don't wont to bother with it)
3) Having package.keywords accept only ck-sources X (X-r1 masked by ~) causes emerge to pull gentoo-sources in instead of ck-sources.
4) Masking gentoo-sources causes emerge to fail with "No packages satisfying...." error

Situation now that the patch is in:

4) Masking gentoo-sources, no errors occur, no new kernel versions are pulled in.

What I thought this version would do was that without even doing the masking no new sources are pulled in. 

I mean, kernel-sources are, as far as I know, build-time dependencies (For any package - they are sources after all - hardly anything could depend on them runtime). Unless --with-bdep is active there should be no reason for emerge to try to pull in latest&greatest. But I guess the BDEP in packages is specified as virtual/linux-sources - and first check is still based on the availability of packages in portage tree, and tree's gentoo-sources take precedence over installed ck-sources gone from tree. Only after all else fails, are the installed packages checked.

In fact, now that my emerge -uvDaN world (with gentoo-sources masked) has completed, I removed the gentoo-sources from package.mask and re-ran emerge -uvDaN world - and it no longer tries to pull in the gentoo-sources (Nothing to merge, so no bdeps to satisfy either..)

Based on this, what I'd like to actually have, is to tweak the "installed packages satisfy dependencies" to include that they can also satisfy virtuals. Anyway - this seems to only apply to virtuals so I can package.mask them nicely.

(In reply to comment #54)
> man portage, search for package.provided.

  Doesn't really apply since at least according to the manpage wildcards are not allowed. I'd like to "provide", say, all ck-sources version 2.6.19-*, but when 2.6.20 hits the tree upgrading to that is ok (I just want to skip all the minor updates that happen every now and then and concentrate on upgrading the kernel only every few months (or years)).

  Anyway, I can live with masking virtual-candidates. Especially since the other big beef, sun-jre/sdk, with manual fetching (that I didn't like to do every time a minor release came out) is now automatically downloaded.
Comment 56 Duncan 2007-02-14 10:37:17 UTC
(In reply to comment #55)
> (In reply to comment #54)
> > man portage, search for package.provided.
> 
>   Doesn't really apply since at least according to the manpage wildcards are
> not allowed. I'd like to "provide", say, all ck-sources version 2.6.19-*, but
> when 2.6.20 hits the tree upgrading to that is ok (I just want to skip all
> the minor updates that happen every now and then and concentrate on upgrading
> the kernel only every few months (or years)).

In that case, use 99, version 99 (or 999 or 9999) if you don't want it to /ever/ be pulled in, -r99 if you want the next version but no revisions pulled in.  That /usually/ works, unless someone has a ranged dep with a specified top end (so <0.00-r00).

Thus in this case put ck-sources-2.6.19-r99 in package.provided, and it should satisfy any reasonable -r number that comes out, while still bumping to whatever-sources-2.6.20 when it comes out.

Using multiple 9s it's also immediately apparent what you did, where if you package.provided a random version/release, it looks as if you wanted specifically that release.  Therefore, if there's ever a problem (say a < dependency isn't met), the multiple 9s should stick out and remind you what you did, and why, while an ordinary number wouldn't necessarily prod your memory.
Comment 57 Chris Slycord 2007-02-16 04:27:28 UTC
(In reply to comment #55)
> (In reply to comment #53)
> > to install the newest stable kernel source. Sure, the installed kernel sources
> > satisfy the dependency... but when you tell it "--update --deep" it's gonna
> > update packages and their dependencies. So in this case, it updates the kernel
> > sources.
> 
>   That explains it. Should have checked --tree. Kinda odd though that even
> build-time dependencies are not satisfied with installed packages that are gone
> from the tree.

What? Just because it's installing a newer kernel source doesn't mean that the dependency wasn't met.

Here's something you can try to illustrate what I'm saying
1) install ck-sources version
2) install yourpackage that depends on virtual/linux-sources
3) watch as your installed sources get removed from portage
4) Do "emerge yourpackage" and you'll see that it reinstalls just fine
5) Now do "emerge --update --deep yourpackage" and watch as it wants to install gentoo-sources

See, the dependency is met (hence why "emerge yourpackage" works), but when you specify --update AND --deep you end up updating all dependencies of the packages in world (and dependencies of dependencies too). So since yourpackage depends on linux-sources, linux-sources is included in the update.

This really is by design.
Comment 58 Antti Mäkelä 2007-02-16 07:29:00 UTC
(In reply to comment #57)
> 5) Now do "emerge --update --deep yourpackage" and watch as it wants to install gentoo-sources
> See, the dependency is met (hence why "emerge yourpackage" works), but when you
> specify --update AND --deep you end up updating all dependencies of the

  Yes - but I thought that --with-bdeps defaults to no when doing a --deep (and to yes on --depclean), so if it's just a build-time dependency (as kernel sources tend to be), that does not get updated. 
  But I didn't do a --tree so cannot check anymore if whatever package I updated had sources listed as runtime or buildtime dependency. 

  But ok, thanks for clearing this up.