Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 326787 - emerge --jobs does not emerge buildtime dependencies soon enough(?)
Summary: emerge --jobs does not emerge buildtime dependencies soon enough(?)
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Dependencies (show other bugs)
Hardware: All Linux
: High minor (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 331387 (view as bug list)
Depends on:
Blocks: 335925
  Show dependency tree
 
Reported: 2010-07-04 01:19 UTC by Richard
Modified: 2010-09-04 08:39 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard 2010-07-04 01:19:30 UTC
I switched my system to a new profile. This profile flipped the ldap flag, among other things, so I recompiled my system with emerge --jobs --keep-going -avDuN world. While it was compiling, net-nds/openldap appears to have finished and app-crypt/mit-krb5 started compiling before net-nds/openldap was installed. This caused the build process to fail.

Reproducible: Didn't try

Steps to Reproduce:




I have seen similar situations happen before when installing and recompiling many packages at once and it is usually resolved by doing something like emerge --resume. This appears to be a bug in portage. Here is the terminal output of the emerge process:

>>> Verifying ebuild manifests
>>> Starting parallel fetch
>>> Emerging (1 of 94) dev-lang/swig-2.0.0
>>> Emerging (2 of 94) gnome-base/gnome-common-2.28.0
>>> Emerging (3 of 94) media-libs/libid3tag-0.15.1b-r2
>>> Emerging (4 of 94) media-libs/lcms-2.0a
>>> Emerging (5 of 94) net-nds/openldap-2.4.21
>>> Emerging (6 of 94) sys-libs/e2fsprogs-libs-1.41.12
>>> Emerging (7 of 94) media-libs/libexif-0.6.19
>>> Emerging (8 of 94) app-portage/eix-0.20.6
>>> Emerging (9 of 94) app-editors/nano-2.2.4
>>> Emerging (10 of 94) app-cdr/cdrdao-1.2.3
>>> Emerging (11 of 94) media-gfx/sane-backends-1.0.21
>>> Emerging (12 of 94) media-libs/flac-1.2.1-r3
>>> Emerging (13 of 94) media-libs/libmng-1.0.10
>>> Emerging (14 of 94) media-libs/pdflib-7.0.4_p4-r1
>>> Emerging (15 of 94) media-libs/speex-1.2_rc1
>>> Emerging (16 of 94) media-libs/libsdl-1.2.14-r2
>>> Emerging (17 of 94) dev-libs/glib-2.24.1
>>> Emerging (18 of 94) x11-libs/cairo-1.8.10
>>> Emerging (19 of 94) app-text/aspell-0.60.6-r1
>>> Installing (2 of 94) gnome-base/gnome-common-2.28.0
>>> Installing (3 of 94) media-libs/libid3tag-0.15.1b-r2
>>> Installing (4 of 94) media-libs/lcms-2.0a
>>> Installing (7 of 94) media-libs/libexif-0.6.19
>>> Installing (10 of 94) app-cdr/cdrdao-1.2.3
>>> Installing (16 of 94) media-libs/libsdl-1.2.14-r2
>>> Installing (13 of 94) media-libs/libmng-1.0.10
>>> Installing (15 of 94) media-libs/speex-1.2_rc1
>>> Installing (8 of 94) app-portage/eix-0.20.6
>>> Installing (1 of 94) dev-lang/swig-2.0.0
>>> Installing (12 of 94) media-libs/flac-1.2.1-r3
>>> Installing (19 of 94) app-text/aspell-0.60.6-r1
>>> Installing (18 of 94) x11-libs/cairo-1.8.10
>>> Emerging (20 of 94) media-video/ffmpeg-0.6
>>> Emerging (21 of 94) media-libs/libmpeg2-0.5.1-r1
>>> Emerging (22 of 94) media-sound/mpg123-1.12.2
>>> Emerging (23 of 94) media-libs/sdl-mixer-1.2.11-r1
>>> Emerging (24 of 94) media-libs/libdv-1.0.0-r2
>>> Emerging (25 of 94) dev-libs/zziplib-0.13.59-r1
>>> Emerging (26 of 94) www-client/links-2.3_pre1
>>> Emerging (27 of 94) media-libs/sdl-sound-1.0.3
>>> Emerging (28 of 94) app-dicts/aspell-en-6.0.0
>>> Installing (28 of 94) app-dicts/aspell-en-6.0.0
>>> Installing (23 of 94) media-libs/sdl-mixer-1.2.11-r1
>>> Installing (22 of 94) media-sound/mpg123-1.12.2
>>> Installing (24 of 94) media-libs/libdv-1.0.0-r2
>>> Installing (27 of 94) media-libs/sdl-sound-1.0.3
>>> Installing (21 of 94) media-libs/libmpeg2-0.5.1-r1
>>> Installing (11 of 94) media-gfx/sane-backends-1.0.21
>>> Installing (25 of 94) dev-libs/zziplib-0.13.59-r1
>>> Installing (26 of 94) www-client/links-2.3_pre1
>>> Installing (14 of 94) media-libs/pdflib-7.0.4_p4-r1
>>> Installing (20 of 94) media-video/ffmpeg-0.6
>>> Emerging (29 of 94) net-misc/curl-7.21.0
>>> Emerging (30 of 94) media-video/transcode-1.1.5-r1
>>> Emerging (31 of 94) games-strategy/freeciv-2.2.0
>>> Installing (29 of 94) net-misc/curl-7.21.0
>>> Installing (30 of 94) media-video/transcode-1.1.5-r1
>>> Installing (31 of 94) games-strategy/freeciv-2.2.0
>>> Emerging (32 of 94) app-crypt/mit-krb5-1.8.2
>>> Failed to emerge app-crypt/mit-krb5-1.8.2, Log file:
>>>  '/var/tmp/portage/app-crypt/mit-krb5-1.8.2/temp/build.log'
>>> Installing (9 of 94) app-editors/nano-2.2.4
>>> Installing (6 of 94) sys-libs/e2fsprogs-libs-1.41.12
>>> Installing (17 of 94) dev-libs/glib-2.24.1
>>> Installing (5 of 94) net-nds/openldap-2.4.21
>>> Jobs: 31 of 94 complete, 1 failed               Load avg: 1.8, 7.0, 10.9
*** Resuming merge...

Here is an excerpt of the build.log file, also taken from the terminal:

checking return type of setrpcent... (cached) void
checking return type of endrpcent... (cached) void
checking for a recent enough OpenSSL... yes
checking for PKCS7_get_signer_info in -lcrypto... yes
checking for daemon... yes
checking for gethostbyname_r... (cached) yes
checking for getservbyname_r... (cached) yes
checking for gmtime_r... (cached) yes
checking for localtime_r... (cached) yes
checking ldap.h usability... no
checking ldap.h presence... no
checking for ldap.h... no
configure: error: ldap.h not found

!!! Please attach the following file when seeking support:
!!! /var/tmp/portage/app-crypt/mit-krb5-1.8.2/work/krb5-1.8.2/src/config.log
 * ERROR: app-crypt/mit-krb5-1.8.2 failed:
 *   econf failed
 * 
 * Call stack:
 *     ebuild.sh, line   54:  Called src_configure
 *   environment, line 2881:  Called econf '--with-ldap' '--without-krb4' '--enable-shared' '--with-system-et' '--with-system-ss' '--enable-dns-for-realm' '--enable-kdc-replay-cache' '--disable-rpath'
 *     ebuild.sh, line  544:  Called die
 * The specific snippet of code:
 *                      die "econf failed"
 * 
 * If you need support, post the output of 'emerge --info =app-crypt/mit-krb5-1.8.2',
 * the complete build log and the output of 'emerge -pqv =app-crypt/mit-krb5-1.8.2'.
 * The complete build log is located at '/var/tmp/portage/app-crypt/mit-krb5-1.8.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/app-crypt/mit-krb5-1.8.2/temp/environment'.
 * S: '/var/tmp/portage/app-crypt/mit-krb5-1.8.2/work/krb5-1.8.2/src'

Here is my emerge --info:

# emerge --info
Portage 2.1.8.3 (default/linux/x86/10.0/desktop/kde, gcc-4.4.4, glibc-2.11.2-r0, 2.6.34 i686)
=================================================================
System uname: Linux-2.6.34-i686-Genuine_Intel-R-_CPU_T2400_@_1.83GHz-with-gentoo-2.0.1
Timestamp of tree: Sat, 03 Jul 2010 15:00:01 +0000
ccache version 2.4 [enabled]
app-shells/bash:     4.1_p7
dev-java/java-config: 2.1.11
dev-lang/python:     2.6.5-r2, 3.1.2-r3
dev-util/ccache:     2.4-r8
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.6.1-r1
sys-apps/sandbox:    2.2
sys-devel/autoconf:  2.13, 2.65-r1
sys-devel/automake:  1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.4-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
virtual/os-headers:  2.6.34
ACCEPT_KEYWORDS="x86 ~x86"
ACCEPT_LICENSE="*"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=prescott --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=2048 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=prescott --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=2048 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests buildpkg ccache distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -march=prescott --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=2048 -pipe -fomit-frame-pointer"
GENTOO_MIRRORS="http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ http://gentoo.netnitco.net http://distro.ibiblio.org/pub/linux/distributions/gentoo/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/sunrise /var/lib/layman/vmware /var/lib/layman/java-overlay /usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac acpi alsa apm berkdb bluetooth branding bzip2 cairo cdr cleartype cli consolekit corefonts cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam fftw firefox flac fortran gd gdbm gif gnutls gpm hal iconv java java6 jpeg kde lcms ldap libnotify lzma mad midi mikmod mmap mmx mng modules mp3 mp4 mpeg mudflap ncurses networkmanager nls nptl nptlonly nsplugin ogg opengl openmp pam pango pcre pdf perl png ppds pppd python qt3support qt4 readline reflection samba sdl session spell spl sqlite sse sse2 sse3 ssl startup-notification svg sysfs tcpd threads tiff truetype type1 unicode usb vorbis x264 x86 xcb xml xorg xulrunner xv xvid xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" NETBEANS_MODULES="*" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Richard 2010-07-04 01:20:05 UTC
By the way, the old profile was default/linux/x86/10.0.
Comment 2 Zac Medico gentoo-dev 2010-07-06 00:31:37 UTC
I've seen reports like this before, I've checked the code, and I couldn't find anything wrong. It works like this:

In the Scheduler class there is a method called _choose_pkg which chooses the next build. It uses the _dependent_on_scheduled_merges  method to search a directed graph for unsatisfied dependencies.  Members of the graph are marked complete by the _task_complete method. This code is single-threaded using a poll loop.

If you can reproduce the initial state and post emerge --pretend --debug output for that command that results in failure, it might give some clue about what causes this.
Comment 3 Richard 2010-07-06 03:30:56 UTC
(In reply to comment #2)
> I've seen reports like this before, I've checked the code, and I couldn't find
> anything wrong. It works like this:
> 
> In the Scheduler class there is a method called _choose_pkg which chooses the
> next build. It uses the _dependent_on_scheduled_merges  method to search a
> directed graph for unsatisfied dependencies.  Members of the graph are marked
> complete by the _task_complete method. This code is single-threaded using a
> poll loop.
> 
> If you can reproduce the initial state and post emerge --pretend --debug output
> for that command that results in failure, it might give some clue about what
> causes this.
> 

Unfortunately, I did not perform a backup prior to the upgrade, so the initial state has been lost.

Perhaps the issue is present in one of the helper functions, such that it is behaving close to how it should be, but not perfectly, leading to this bug. Have you tried doing unit testing to verify that all of the helper functions are behaving correctly (e.g. there is no off by 1 error in one of them that would cause this)?
Comment 4 Zac Medico gentoo-dev 2010-07-07 23:19:01 UTC
(In reply to comment #3)
> Perhaps the issue is present in one of the helper functions, such that it is
> behaving close to how it should be, but not perfectly, leading to this bug.

I looked again, and now I think I found the bug. Here's the fix:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e89da47d4ef51fba3d5a03f1b379d9ad02190d22

> Have you tried doing unit testing to verify that all of the helper functions
> are behaving correctly (e.g. there is no off by 1 error in one of them that
> would cause this)?

No, there aren't any tests for this yet. I suppose we could write a test that schedules a fake dependency graph and double checks the choices. It might be hard to create a state that would trigger this bug though. It's not a lot of code and I just needed to look at it more carefully.
Comment 5 Richard 2010-07-10 20:16:04 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > Perhaps the issue is present in one of the helper functions, such that it is
> > behaving close to how it should be, but not perfectly, leading to this bug.
> 
> I looked again, and now I think I found the bug. Here's the fix:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e89da47d4ef51fba3d5a03f1b379d9ad02190d22
> 
> > Have you tried doing unit testing to verify that all of the helper functions
> > are behaving correctly (e.g. there is no off by 1 error in one of them that
> > would cause this)?
> 
> No, there aren't any tests for this yet. I suppose we could write a test that
> schedules a fake dependency graph and double checks the choices. It might be
> hard to create a state that would trigger this bug though. It's not a lot of
> code and I just needed to look at it more carefully.
> 

(In reply to comment #4)
> (In reply to comment #3)
> > Perhaps the issue is present in one of the helper functions, such that it is
> > behaving close to how it should be, but not perfectly, leading to this bug.
> 
> I looked again, and now I think I found the bug. Here's the fix:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e89da47d4ef51fba3d5a03f1b379d9ad02190d22

Is there any chance that this fix will be backported to portage-2.1.8.x? I had planned to test it after it had been pushed into the portage testing tree through a revision of a sys-apps/portage, but since that has not happened so far, I have manually applied it to my systems. So far, I have had no issues.
Comment 6 Zac Medico gentoo-dev 2010-07-11 18:35:22 UTC
Yes, this fix will be included in 2.1.8.4 which will be released later this month.
Comment 7 Zac Medico gentoo-dev 2010-08-06 06:57:14 UTC
*** Bug 331387 has been marked as a duplicate of this bug. ***
Comment 8 Zac Medico gentoo-dev 2010-08-23 06:26:54 UTC
This is in 2.2_rc68, but I'll leave this bug open until it's in an unmasked version.
Comment 9 Zac Medico gentoo-dev 2010-09-04 08:39:29 UTC
This is fixed in 2.1.9.