Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 188782

Summary: portage-2.1.2.11 reverses merge order of dependencies
Product: Portage Development Reporter: Andrew Evans <gentoo>
Component: Core - DependenciesAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: major CC: fellows
Priority: High Keywords: InVCS, REGRESSION
Version: unspecified   
Hardware: x86   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 181949, 188739, 189285    
Attachments: improve merge order for cases where installed packages have unsatisfied dependencies
give higher priority to dependencies on packages specified as arguments

Description Andrew Evans 2007-08-14 05:16:26 UTC
After upgrading from portage-2.1.2.9 to portage-2.1.2.11, portage installs dependencies in the wrong (reverse) order on my system.

Reproducible: Always

Steps to Reproduce:
Incorrectly working example:

$ emerge --version
Portage 2.1.2.11 (default-linux/x86/2006.1/desktop, gcc-4.1.2, glibc-2.5-r4, 2.6.21-gentoo-r4 i686)

$ emerge --pretend --oneshot --update --newuse --deep shared-mime-info

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ] x11-misc/shared-mime-info-0.21-r1
[ebuild  N    ] dev-perl/XML-Parser-2.34-r1

This causes the merge of x11-misc/shared-mime-info to fail in the configure stage, since it looks for dev-perl/XML-Parser (required indirectly through dev-util/intltool) and dies when it isn't found.

Correctly working example:

$ emerge --version
Portage 2.1.2.9 (default-linux/x86/2006.1/desktop, gcc-4.1.2, glibc-2.5-r4, 2.6.21-gentoo-r4 i686)

$ emerge --pretend --oneshot --update --newuse --deep shared-mime-info

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ] dev-perl/XML-Parser-2.34-r1
[ebuild  N    ] x11-misc/shared-mime-info-0.21-r1




$ emerge --info
Portage 2.1.2.11 (default-linux/x86/2006.1/desktop, gcc-4.1.2, glibc-2.5-r4, 2.6.21-gentoo-r4 i686)
=================================================================
System uname: 2.6.21-gentoo-r4 i686 AMD Athlon(tm) 64 X2 Dual Core Processor 4600+
Gentoo Base System release 1.12.9
Timestamp of tree: Mon, 13 Aug 2007 19:33:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
dev-java/java-config: 1.3.7, 2.0.32
dev-lang/python:     2.4.4-r4
dev-python/pycrypto: 2.0.1-r6
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.61
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.17
sys-devel/gcc-config: 1.3.16
sys-devel/libtool:   1.5.23b
virtual/os-headers:  2.6.21
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-pipe -O3 -march=athlon64"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/revdep-rebuild /etc/splash /etc/terminfo"
CXXFLAGS="-pipe -O3 -march=athlon64"
DISTDIR="/usr/portage/distfiles"
FEATURES="distlocks nostrip sandbox sfperms strict userfetch userpriv usersandbox"
GENTOO_MIRRORS="http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac alsa avahi bash-completion berkdb bindist bitmap-fonts bonjour boundschecking browserplugin cairo caps cdr cli cracklib crypt cups dbus dri dvd dvdr dvdread emacs emboss encode fam firefox foomaticdb gif gstreamer gtk gtk2 hal iconv imlib isdnlog jpeg kde mad midi mikmod mmx mmxext mozilla moznopango mp3 mpeg mudflap mythtv ncurses nptl nptlonly nsplugin nvidia ogg ooo-kde opengl openmp pam pcre pda pdf png ppds pppd qt3 qt4 quicktime readline reflection rtc sdl session spell spl sse sse2 ssl tcpd truetype truetype-fonts type1-fonts vorbis win32codecs x86 xml xorg xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIRC_DEVICES="hauppauge" USERLAND="GNU" VIDEO_CARDS="nvidia"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Zac Medico gentoo-dev 2007-08-14 06:33:07 UTC
Created attachment 128016 [details, diff]
improve merge order for cases where installed packages have unsatisfied dependencies

Normally this type of thing doesn't happen since XML-Parser should already be installed if intltool is installed.
Comment 2 Carsten Lohrke (RETIRED) gentoo-dev 2007-08-14 15:53:24 UTC
(In reply to comment #1)
> Normally this type of thing doesn't happen since XML-Parser should already be
> installed if intltool is installed.

This happens all the time, when doing a rebuild because of library ABI breakage. forums.g.o is full of the hassles (expat, curl upgrade) of revdep-rebuild breaking because of Portage not getting the build order right. I went yesterday evening through all open bugs, because I thought the bug regarding getting the ordering of the to be (re-)built ebuilds right must be still open, but actually didn't found it...


Should this patch fix these issues or is there more about it?
Comment 3 Andrew Evans 2007-08-14 16:55:11 UTC
Yes, I was bitten by the expat upgrade too. portage-2.1.2.9 correctly merges expat before the packages that require/link against it. portage-2.1.2.11 doesn't.
Comment 4 Zac Medico gentoo-dev 2007-08-14 18:45:49 UTC
(In reply to comment #2)
> Should this patch fix these issues or is there more about it?

This patch reverts it to behavior that 2.1.2.9 had. It's better, but not as good as can be. In order to make it really optimal, we need to make it aware of ABI breakage so that it knows when an installed package is unusable to satisfy a dependency. Right now, the user would have to unmerge the broken packages to prevent the resolver from treating them as satisfied depenedencies.
Comment 5 Carsten Lohrke (RETIRED) gentoo-dev 2007-08-14 19:25:24 UTC
(In reply to comment #4)
> In order to make it really optimal, we need to make it aware of
> ABI breakage so that it knows when an installed package is unusable to satisfy
> a dependency. Right now, the user would have to unmerge the broken packages to
> prevent the resolver from treating them as satisfied depenedencies.

There're a lot of possible breaking rebuilds where libs do not matter at all, e.g. scripting langauges, java, etc.. To let revdep-rebuild not fail, what Portage should do is actually merging the dependencies in order of the tree the dependencies form and not as they're given on the command line. 

To fix the revdep-rebuild issues now, there is no need to have Portage be aware of the ABI breakage. For proper reverse dependency support (and getting rid of revdep-rebuild) a lot more is needed, but Portage still shouldn't care about any libs; The relevant data needs to be part of the ebuilds, as we can't rely on upstream developers to change their library major version number to reflect an ABI change. There have been violations of this good practice and there will be in future. Also, as already pointed out - and I'm prettsy sure you are aware of it - libraries are only part of the picture.


To give you an example of what's the broken status now: On my box due to the expat upgrade I had to rebuild musicbrainz and tunepimp (which depends on muscbrainz). revdep-rebuild basically spit out "emerge --oneshot tunepimp musicbrainz" and of course tunepimp failed.

The funny thing about it is that actually Portage does the wanted bits already under certain conditions (this is with v.2.1.3.5)...


WRONG:

emerge tunepimp musicbrainz -p

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] media-libs/tunepimp-0.5.3
[ebuild   R   ] media-libs/musicbrainz-2.1.4


RIGHT:

emerge tunepimp musicbrainz -pt

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild   R   ] media-libs/musicbrainz-2.1.4
[ebuild   R   ] media-libs/tunepimp-0.5.3


Funny is that Portage does --pretend --tree in corrent order, but actually trying to build with --tree, but without --pretend, the order is wrong again. If Portage would actually build in order as --pretend --tree output prints it, there would (given there are no bugs in it - and issues with circular dependencies aside) be no problem with revdep-rebuild.
Comment 6 Carsten Lohrke (RETIRED) gentoo-dev 2007-08-14 19:27:57 UTC
Ah, missed the "reversed" in the -pt output...
Comment 7 Zac Medico gentoo-dev 2007-08-14 19:50:03 UTC
(In reply to comment #5)
> To give you an example of what's the broken status now: On my box due to the
> expat upgrade I had to rebuild musicbrainz and tunepimp (which depends on
> muscbrainz). revdep-rebuild basically spit out "emerge --oneshot tunepimp
> musicbrainz" and of course tunepimp failed.

Use the attached patch together with the --deep option and it should give you a pretty good order in most cases. I can probably make it a little better still without ABI awareness. I'll look into it.
Comment 8 Carsten Lohrke (RETIRED) gentoo-dev 2007-08-14 20:29:01 UTC
(In reply to comment #7)
> Use the attached patch together with the --deep option and it should give you a
> pretty good order in most cases. I can probably make it a little better still
> without ABI awareness. I'll look into it.

Well, it's only a minor annoyance for me, but when you look in the forums or even lookup the bug reports marked as duplicates with "expat" in the comments lately, you get an idea what sort of big problem this is for (parts of) our user base. So that you're having a look is very appreciated. :)
Comment 9 Zac Medico gentoo-dev 2007-08-15 01:57:41 UTC
Created attachment 128134 [details, diff]
give higher priority to dependencies on packages specified as arguments

This patch includes the previous one and it also gives higher priority to dependencies on packages that have been specified as arguments (such as those given by revdep-rebuild).
Comment 10 Zac Medico gentoo-dev 2007-08-15 22:37:20 UTC
This patch is released in portage-2.1.3.6.  I'm also going to do a 2.1.2.12 release with the same patch, but I probably will wait until tomorrow to release that.
Comment 11 Zac Medico gentoo-dev 2007-08-17 22:49:44 UTC
portage-2.1.2.12 has been released with this patch.
Comment 12 Zac Medico gentoo-dev 2007-08-24 18:50:50 UTC
Closing since 2.1.2.12 is stable on most archs now.