Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 141118

Summary: multiple levels of indirection in virtual deps cause undesirable results
Product: Portage Development Reporter: Matthew Kennedy (RETIRED) <mkennedy>
Component: Core - DependenciesAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: askwar, caster, djaara83, jakub, java, truedfx
Priority: High Keywords: Inclusion, InVCS
Version: 2.1   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=632026
https://bugs.gentoo.org/show_bug.cgi?id=526160
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 155723, 531786, 147007    
Attachments: recursively collapse new style virtuals

Description Matthew Kennedy (RETIRED) gentoo-dev 2006-07-19 21:46:52 UTC
localhost / # emerge mozilla-firefox -pv

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N F  ] dev-java/sun-jre-bin-1.5.0.07  USE="X alsa nsplugin" 16,455 kB
[ebuild  N    ] virtual/jre-1.5.0  0 kB
[ebuild  N    ] www-client/mozilla-firefox-1.5.0.4  USE="java mozdevelop -debug -gnome -ipv6 -xinerama -xprint" LINGUAS="-ar -ca -cs -da -de -el -en_GB -es -es_AR -es_ES -fi -fr -ga -ga_IE -he -hu -it -ja -ko -mk -nb -nb_NO -nl -pl -pt_BR -ro -ru -sk -sl -sv -sv_SE -tr -zh_CN -zh_TW" 0 kB

Total size of downloads: 16,455 kB
localhost / # emerge sun-jdk -pv

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   Rf  ] dev-java/sun-jdk-1.5.0.07  USE="X alsa nsplugin -doc -examples -jce" 0 kB

Total size of downloads: 0 kB
localhost / # emerge info
*** Deprecated use of action 'info', use '--info' instead
Portage 2.1.1_pre3-r1 (default-linux/x86/2006.0, gcc-4.1.1/vanilla, glibc-2.4-r3, 2.6.17-1-k7 i686)
=================================================================
System uname: 2.6.17-1-k7 i686 AMD Athlon(tm) XP 3200+
Gentoo Base System version 1.12.1
ccache version 2.4 [enabled]
app-admin/eselect-compiler: 2.0.0_rc2-r1
dev-lang/python:     2.4.3-r1
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.4-r2
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.18.1
sys-devel/autoconf:  2.13, 2.60
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2
sys-devel/binutils:  2.17
sys-devel/gcc-config: 2.0.0_rc1
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.11-r5
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/eselect/compiler /etc/gconf /etc/java-config/vms/ /etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-O2 -mcpu=i686 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig ccache cvs distlocks metadata-transfer notitles parallel-fetch sandbox sfperms sign strict userfetch userpriv usersandbox"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
LANG="en_US.UTF-8"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 X alsa apache2 apm arts avi berkdb bitmap-fonts cli crypt cups dlloader dri eds emboss encode esd foomaticdb fortran gcj gdbm gif gnome gstreamer gtk gtk2 imlib isdnlog jpeg kde libg++ libwww mad mikmod motif mp3 mpeg ncurses nls nptl nptlonly nsplugin ogg opengl oss pam pcre pdflib perl png pppd python qt qt3 qt4 quicktime readline reflection sdl session spell spl ssl tcpd tiff truetype truetype-fonts type1-fonts udev unicode vorbis xml xmms xorg xv zlib elibc_glibc input_devices_keyboard input_devices_mouse kernel_linux userland_GNU video_cards_nv"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, LINGUAS, MAKEOPTS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY



This is a completely new build from stage3.
Comment 1 Jakub Moc (RETIRED) gentoo-dev 2006-07-21 00:43:14 UTC
*** Bug 141247 has been marked as a duplicate of this bug. ***
Comment 2 Harald van Dijk (RETIRED) gentoo-dev 2006-07-21 00:58:15 UTC
New-style virtuals are not automatically provided, so the fact that you have sun-jdk installed means nothing to portage. They are just packages which install no files now. You don't have virtual/jdk installed (I assume; if you do, something is wrong), so portage (as specified by the virtual/jre ebuild) prefers sun-jre-bin instead. You can
  emerge --oneshot virtual/jdk
manually to get portage to pick that and not pull in sun-jre-bin. This is certainly bad design, though. One way to make this work would be adding virtual/jdk's dependency list to the virtual/jre ebuilds instead of depending on it. Another way (more work, but IMO better since it will handle more cases) would be for portage to give virtual/xxx ebuilds special treatment and always prefer them if their dependencies are already installed.
Comment 3 Jakub Moc (RETIRED) gentoo-dev 2006-07-21 01:05:41 UTC
There's a better description in Bug 141247. This looks like another dupe of Bug 1343.
Comment 4 Zac Medico gentoo-dev 2006-07-21 01:27:35 UTC
Indirect dependencies make this situation different than usual:

virtual/jre-1.5.0
RDEPEND=|| ( =dev-java/sun-jre-bin-1.5.0* =virtual/jdk-1.5.0* )

virtual/jdk-1.5.0
RDEPEND=|| ( =dev-java/sun-jdk-1.5.0* =dev-java/ibm-jdk-bin-1.5.0* =dev-java/jrockit-jdk-bin-1.5.0* )

One possible way to make it behave better would be to put the jdk packages directly in the virtual/jre dependencies (with lower priority than the jre packages).
Comment 5 Zac Medico gentoo-dev 2006-07-21 16:38:48 UTC
(In reply to comment #2)
> would be for portage to give virtual/xxx ebuilds special treatment and always
> prefer them if their dependencies are already installed.

Since new-style virtuals are a type of "meta-package", I'd prefer that we introduce some type of package metadata into the EAPI that distiguishes meta-packages (those that do not install anything) from normal packages.
Comment 6 Harald van Dijk (RETIRED) gentoo-dev 2006-07-21 23:28:07 UTC
(In reply to comment #5)
> (In reply to comment #2)
> > would be for portage to give virtual/xxx ebuilds special treatment and always
> > prefer them if their dependencies are already installed.
> 
> Since new-style virtuals are a type of "meta-package", I'd prefer that we
> introduce some type of package metadata into the EAPI that distiguishes
> meta-packages (those that do not install anything) from normal packages.

Virtuals are similar to other existing metapackages, but with one big difference: those other metapackages are not supposed to be used as dependencies, virtuals are. It's not a big deal if portage has less than perfect support for random metapackages used as dependencies, I think.

You mention EAPI, would that mean that updated new-style virtuals would no longer work with older portage versions?
Comment 7 Zac Medico gentoo-dev 2006-07-22 00:44:05 UTC
(In reply to comment #6)
> You mention EAPI, would that mean that updated new-style virtuals would no
> longer work with older portage versions?

Any version of portage should reject any package having an EAPI that it doesn't recognize.  All currently versions of portage only recongnize EAPI=0.

Anyway, this bug doesn't seem like real problem to me.  All that needs to be done to get better behavior is to collapse the jdk dependencies into the jre dependencies, like this:

virtual/jre-1.5.0
RDEPEND="|| ( =dev-java/sun-jre-bin-1.5.0* =dev-java/sun-jdk-1.5.0* =dev-java/ibm-jdk-bin-1.5.0* =dev-java/jrockit-jdk-bin-1.5.0* )"

That way, an installed jdk will satisfy the virtual/jre dependency without portage attempting to pull in sun-jre-bin.

Comment 8 Harald van Dijk (RETIRED) gentoo-dev 2006-07-22 00:58:12 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > You mention EAPI, would that mean that updated new-style virtuals would no
> > longer work with older portage versions?
> 
> Any version of portage should reject any package having an EAPI that it doesn't
> recognize.  All currently versions of portage only recongnize EAPI=0.

Then any solution relying on EAPI could not be implemented for months, until all current portage versions are history, and isn't a good idea here.

> Anyway, this bug doesn't seem like real problem to me.  All that needs to be
> done to get better behavior is to collapse the jdk dependencies into the jre
> dependencies, like this:

Yes, that was my first suggestion too (comment #2), but the reason it's a workaround rather than a fix is because ebuilds can do || ( ... virtual/xxx ) too, which can make sense for at least the new-style virtual/pmake. And in that case, it's definitely wrong to not use the virtual in the ebuild. (I considered virtual/x11, but it wouldn't be affected either way.)
Comment 9 Zac Medico gentoo-dev 2006-07-22 01:16:25 UTC
(In reply to comment #8)
> Then any solution relying on EAPI could not be implemented for months, until
> all current portage versions are history, and isn't a good idea here.

It's possible to make minor additions to the EAPI without actually bumping the version.

> Yes, that was my first suggestion too (comment #2), but the reason it's a
> workaround rather than a fix is because ebuilds can do || ( ... virtual/xxx )
> too, which can make sense for at least the new-style virtual/pmake. And in that
> case, it's definitely wrong to not use the virtual in the ebuild.

I'm not sure how useful it is to have multiple levels of indirection in dependency relationships like this.  Are there any ebuilds using || ( ... virtual/pmake ) now?  Perhaps there should be a virtual/anymake that pulls in any type of make.

Comment 10 Harald van Dijk (RETIRED) gentoo-dev 2006-07-22 02:03:23 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > Then any solution relying on EAPI could not be implemented for months, until
> > all current portage versions are history, and isn't a good idea here.
> 
> It's possible to make minor additions to the EAPI without actually bumping the
> version.

Okay. My original question was whether EAPI would mean current portage versions would fail to handle new-new-style virtuals. You said portage would fail on any EAPI other than 0. But that was intended to be an unrelated comment, rather than implying an answer of "yes" ? :)

> > Yes, that was my first suggestion too (comment #2), but the reason it's a
> > workaround rather than a fix is because ebuilds can do || ( ... virtual/xxx )
> > too, which can make sense for at least the new-style virtual/pmake. And in that
> > case, it's definitely wrong to not use the virtual in the ebuild.
> 
> I'm not sure how useful it is to have multiple levels of indirection in
> dependency relationships like this.  Are there any ebuilds using || ( ...
> virtual/pmake ) now?  Perhaps there should be a virtual/anymake that pulls in
> any type of make.

Presumably, the extra alternative to virtual/pmake would be a make clone similar to pmake, not supporting all the features, but enough for the package in question. Not just "any" make would work. In particular, GNU make would not work (otherwise, since gmake is a system dep on all archs, no ebuild dep is necessary). I don't know what Flameeyes's (or anyone else's) requirements for virtual/pmake are, but if NetBSD make doesn't match them, || ( ... virtual/pmake ) ebuilds would be a good idea for (parts of) the NetBSD base system. Even though I haven't said much about it, I do still work on NetBSD. :)

Anyway, it was just an example. Another potential problem is if monolithic X would be updated to no longer provide virtual/x11 the old-style way: || ( ... virtual/x11 ) ebuilds would all fail, and there are a whole lot of them.

If you must have something that currently breaks, though, this bug is the only one I know of, sorry.
Comment 11 Josh Nichols (RETIRED) gentoo-dev 2006-07-23 20:35:20 UTC
(In reply to comment #4)
> Indirect dependencies make this situation different than usual:
> 
> virtual/jre-1.5.0
> RDEPEND=|| ( =dev-java/sun-jre-bin-1.5.0* =virtual/jdk-1.5.0* )
> 
> virtual/jdk-1.5.0
> RDEPEND=|| ( =dev-java/sun-jdk-1.5.0* =dev-java/ibm-jdk-bin-1.5.0*
> =dev-java/jrockit-jdk-bin-1.5.0* )
> 
> One possible way to make it behave better would be to put the jdk packages
> directly in the virtual/jre dependencies (with lower priority than the jre
> packages).
> 

I'm going to go with this option for now. It would be nice though, to be able to just have the virtual/jdk dep instead in jre's depend instead of listing everything that is a jdk.
Comment 12 Jason Stubbs (RETIRED) gentoo-dev 2006-07-26 05:00:11 UTC
With "new style" virtuals you'd either have to list all jdks under virtual/jre or add virtual/jdk to whatever ebuilds depend on virtual/jre. A third solution would be to add PROVIDE="virtual/jre" to virtual/jdk-*.ebuild if "old style" virtuals will not one day be deprecated.
Comment 13 Jakub Moc (RETIRED) gentoo-dev 2006-07-27 14:35:56 UTC
*** Bug 141934 has been marked as a duplicate of this bug. ***
Comment 14 Harald van Dijk (RETIRED) gentoo-dev 2006-07-29 00:41:41 UTC
> If you must have something that currently breaks, though, this bug is the only
> one I know of, sorry.

Here's one:
- Locale-gettext-1.05 depends on >=virtual/perl-Test-Simple-0.54
- virtual/perl-Test-Simple-0.63 depends on || ( ~dev-perl/perl-5.8.8 ~perl-core/Test-Simple-0.63 )
- virtual/perl-Test-Simple-0.64 depends on ~perl-core/Test-Simple-0.64

When installing Locale-gettext, portage wants to pull in virtual/perl-Test-Simple-0.64, and because of that perl-core/Test-Simple-0.64, even when perl-5.8.8-r2 is installed already. This -- again -- may be the correct behaviour for the current design, but I think the design could use an improvement, if that's doable.
Comment 15 Zac Medico gentoo-dev 2006-08-01 23:27:08 UTC
(In reply to comment #14)
> This -- again -- may be the
> correct behaviour for the current design, but I think the design could use an
> improvement, if that's doable.

The current resolver requires hints so that it chooses the correct path without ever having to do any such comparisons between separate paths.  It would require a non-trivial amount of work to make it handle these types of situaltions. Basically, it would need to explore multiple paths and compare them to see which ones can be satisfied the easiest.  As an example, take || ( virtual/jre virtual/jdk ).  The resolver would have to explore both paths before it would know which virtual is easiest to satisfy (jre/jdk are a poor example, but it should convey the idea).
Comment 16 Carsten Lohrke (RETIRED) gentoo-dev 2006-10-23 18:13:40 UTC
(In reply to comment #9)
> I'm not sure how useful it is to have multiple levels of indirection in
> dependency relationships like this.  Are there any ebuilds using || ( ...
> virtual/pmake ) now?  Perhaps there should be a virtual/anymake that pulls in
> any type of make.
> 

Allowing multiple level of indirection may be preferable (think of even more complicated dependencies between virtuals or even necessary. And Portage could determine the "best" ebuld providing the virtual by comparing the possible differing trees for the lowest number of ebuilds that would be installed.

Comment 17 Zac Medico gentoo-dev 2006-10-23 18:26:23 UTC
I'm hoping to fix this bug, along with bug 1343, in 2.1.3.
Comment 18 Zac Medico gentoo-dev 2006-11-14 21:13:08 UTC
In order to fix bug #154223, I've created a bridge between new-style and old-style virtuals.  In it's current form, the virtuals bridge doesn't solve this bug.  However, the bridge could be extended to do on-the-fly rewriting of virtual dependencies so that they include the multiple levels new-style virtual dependencies.  This rewriting of dependencies would collapse the multiple levels of indirection such that this bug would be solved in the simplest possible way.
Comment 19 Zac Medico gentoo-dev 2006-11-15 00:32:33 UTC
Created attachment 101968 [details, diff]
recursively collapse new style virtuals

This patch (applies against 2.1.2_rc1-r7) adds a new expand_new_virtuals() function that does essentially the same thing that dep_virtual() already does, except that it recurses through the dependencies of new-style virtuals and collapses them together via recursive dep_check() calls.  Basically, it converts dependency on a new-style virtual into a normal dependency on-the-fly.
Comment 20 Zac Medico gentoo-dev 2006-11-15 02:17:45 UTC
The patch seems to work pretty well, so I'm considering it for inclusion in 2.1.2.  One notable side-effect is that new-style virtuals will no longer be installed.  They'll be used to expand the dependencies of other packages, and nothing more.  I guess I'll have to add a note to the depclean message so that people don't get too surprised when it wants to remove all of their new-style virtuals.
Comment 21 Zac Medico gentoo-dev 2006-11-15 20:58:45 UTC
This is fixed in svn r5057.
Comment 22 Zac Medico gentoo-dev 2006-11-17 20:53:31 UTC
In trunk r5060:5067 I fixed it so that new-style virtuals are installed like they've always been, but whether or not they are installed does not count in preference selection (the have zero cost for installation).  Also, I integrated the functionality of dep_virtual() into _expand_new_virtuals() and made new-style virtuals preferred over old-style virtuals whenever possible.

This has been released in 2.1.2_rc2.