Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 403895 - sys-apps/portage- emerge with parallel jobs quickly degrades to building only one package at a time
Summary: sys-apps/portage- emerge with parallel jobs quickly degrades to bui...
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
Keywords: InVCS
Depends on:
Blocks: 184128 423075
  Show dependency tree
Reported: 2012-02-15 13:18 UTC by Joe Breuer
Modified: 2012-06-25 21:54 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

Output of emerge -pe --tree @system (system-tree,45.55 KB, text/plain)
2012-02-15 16:34 UTC, Joe Breuer

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Breuer 2012-02-15 13:18:23 UTC
Since I have a significant number of cores available at some locations (here: 8 local, further 16 via distcc) I'd like to reduce the wall-clock time of emerges by having emerge build enough jobs in parallel to saturate the cores.

After initially starting out with building a number of jobs in parallel, emerge quickly tends to devolve to emerging and installing one package at a time, linearly.

This is long-standing, I'm seeing this behavior for at least 18 months.

Reproducible: Always

Steps to Reproduce:
1. configure emerge for parallel builds (make.conf):
EMERGE_DEFAULT_OPTS="--with-bdeps y --jobs=24 --load-average=6"
MAKEOPTS="--jobs=32 --load-average=6"

2. emerge -bDNauvt @system
[66 packages to build]

3. Observe emerge output:
>>> Starting parallel fetch
>>> Emerging (1 of 66) media-libs/libpng-1.5.8
>>> Emerging (2 of 66) media-libs/libogg-1.3.0
>>> Emerging (3 of 66) sys-libs/timezone-data-2011n
>>> Emerging (4 of 66) sys-apps/sdparm-1.06
>>> Emerging (5 of 66) media-libs/jpeg-8c-r1
>>> Emerging (6 of 66) dev-util/byacc-1.9-r3
>>> Emerging (7 of 66) virtual/modutils-0
>>> Installing (6 of 66) dev-util/byacc-1.9-r3
>>> Installing (3 of 66) sys-libs/timezone-data-2011n
>>> Installing (2 of 66) media-libs/libogg-1.3.0
>>> Installing (1 of 66) media-libs/libpng-1.5.8
>>> Installing (4 of 66) sys-apps/sdparm-1.06
>>> Installing (7 of 66) virtual/modutils-0
>>> Installing (5 of 66) media-libs/jpeg-8c-r1
>>> Emerging (8 of 66) media-libs/tiff-4.0.0-r1
>>> Installing (8 of 66) media-libs/tiff-4.0.0-r1
>>> Emerging (9 of 66) sys-libs/slang-2.2.4
>>> Installing (9 of 66) sys-libs/slang-2.2.4
>>> Emerging (10 of 66) sys-kernel/linux-headers-3.1
>>> Installing (10 of 66) sys-kernel/linux-headers-3.1
>>> Emerging (11 of 66) dev-perl/Net-Daemon-0.480.0
>>> Installing (11 of 66) dev-perl/Net-Daemon-0.480.0
>>> Emerging (12 of 66) perl-core/Time-Local-1.200.0
>>> Installing (12 of 66) perl-core/Time-Local-1.200.0
>>> Emerging (13 of 66) virtual/perl-Time-Local-1.200.0-r1
>>> Installing (13 of 66) virtual/perl-Time-Local-1.200.0-r1
>>> Emerging (14 of 66) perl-core/Test-Harness-3.230.0
>>> Installing (14 of 66) perl-core/Test-Harness-3.230.0
>>> Emerging (15 of 66) virtual/perl-Test-Harness-3.230.0-r2
>>> Installing (15 of 66) virtual/perl-Test-Harness-3.230.0-r2
>>> Emerging (16 of 66) dev-libs/libusb-1.0.9_rc3
>>> Installing (16 of 66) dev-libs/libusb-1.0.9_rc3
>>> Emerging (17 of 66) app-text/libpaper-1.1.24-r1
>>> Installing (17 of 66) app-text/libpaper-1.1.24-r1
>>> Emerging (18 of 66) dev-libs/openssl-1.0.0g
>>> Installing (18 of 66) dev-libs/openssl-1.0.0g
>>> Emerging (19 of 66) media-libs/libdvdread-4.2.0
>>> Installing (19 of 66) media-libs/libdvdread-4.2.0
>>> Emerging (20 of 66) media-libs/libdvdnav-4.2.0
>>> Installing (20 of 66) media-libs/libdvdnav-4.2.0
>>> Emerging (21 of 66) kde-base/kde-env-4.7.4
>>> Installing (21 of 66) kde-base/kde-env-4.7.4
>>> Emerging (22 of 66) dev-libs/pth-2.0.7-r3
>>> Installing (22 of 66) dev-libs/pth-2.0.7-r3
>>> Emerging (23 of 66) sys-apps/attr-2.4.46-r1
>>> Installing (23 of 66) sys-apps/attr-2.4.46-r1
>>> Emerging (24 of 66) sys-apps/coreutils-8.14
>>> Installing (24 of 66) sys-apps/coreutils-8.14
>>> Emerging (25 of 66) dev-util/intltool-0.50.0
>>> Jobs: 24 of 66 complete, 1 running              Load avg: 0.28, 0.88, 0.53

Actual Results:  
emerge, after typically one round of parallel emerges, degrades to 'emerge one package, install one package' in a linear fashion.
The load average shows that the available system (8 local cores, further 16 via distcc) is not utilized.

Expected Results:  
Packages should be emerged in parallel achieving a load average matching that configured using --load-average.

Portage (default/linux/amd64/10.0/desktop/kde, gcc-4.5.3, glibc-2.13-r4, 3.1.6-gentoo x86_64)
System uname: Linux-3.1.6-gentoo-x86_64-Intel-R-_Core-TM-_i7-2630QM_CPU_@_2.00GHz-with-gentoo-2.0.3
Timestamp of tree: Wed, 15 Feb 2012 12:15:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [enabled]
app-shells/bash:          4.1_p9
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.1.4-r3
dev-util/cmake:           2.8.6-r4
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.3-r1
sys-devel/gcc-config:     1.4.1-r1
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo LOCAL jmbreuer
ACCEPT_LICENSE="* -@EULA dlj-1.1 PUEL skype-eula sun-bcla-java-vm"
CFLAGS="-march=core2 -mtune=generic -O2 -pipe -momit-leaf-frame-pointer"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c /usr/share/X11/xkb"
CXXFLAGS="-march=core2 -mtune=generic -O2 -pipe -momit-leaf-frame-pointer"
EMERGE_DEFAULT_OPTS="--with-bdeps y --jobs=24 --load-average=6"
FEATURES="assume-digests binpkg-logs buildpkg collision-protect distcc distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en de"
MAKEOPTS="--jobs=32 --load-average=6"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTDIR_OVERLAY="/usr/local/portage /var/lib/layman/jmbreuer"
USE="X Xaw3d a52 aac accessibility acl acpi alsa amd64 apache2 berkdb bluetooth branding bzip2 cairo cdda cdr cjk cli consolekit cracklib crypt css cups curl cvs cxx dbus declarative dga dri dts dv dvd dvdr emboss encode exif fam fame fastcgi ffmpeg firefox flac flash fontconfig foomaticdb fortran fuse gd gdbm gdu gif glitz glut gphoto2 gpm gtk iconv ieee1394 imap imlib innodb ipv6 java jce jpeg kde kerberos kipi kpathsea lame lcms leim libgda libnotify libsamplerate lzo mad mbox mjpeg mmx mmxext mng modplug modules mono motif mozilla mp3 mp4 mpeg mplayer mudflap mule multilib ncurses network network-cron networkmanager nls nptl nptlonly nsplugin offensive ogg openal openexr opengl openmp pam pango pcap pcmcia pcre pda pdf phonon plasma plotutils png policykit ppds pppd qt3support qt4 readline rtsp samba sdl semantic-desktop sensord session slang slp sound spell sse sse2 ssl ssse3 startup-notification stream subversion svg sysfs tcpd theora tiff truetype trusted udev unicode usb v4l v4l2 vhosts vim-syntax vlm vorbis webdav wifi wmf x264 xanim xcb xcomposite xemacs xinerama xml xorg xscreensaver xulrunner xv xvid xvmc zlib zsh-completion" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en de" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" SANE_BACKENDS="net" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

I'll happily provide further information / help in debugging, as requested.
Comment 1 Joe Breuer 2012-02-15 13:25:43 UTC
To forestall one possible argument regarding the low load average: No, I/O bandwidth is NOT the issue here - the disk is mostly idle during the emerge, there only are occasional accesses.
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2012-02-15 14:05:43 UTC
Given a particular depgraph, you cannot ALWAYS have multiple parallel "chains" of emerges going. eg. if 10 packages depend on foo, you must wait for foo to finish emerging.
Comment 3 Joe Breuer 2012-02-15 15:07:30 UTC
I fully understand darkside's argument, but:

A bit further down in the list there are, for example, the following packages:
- eselect
- xvid
- openmotif
- man
- curl
- libvorbis

And they also built one after the other.

I do not see how these packages would have a linear dependency.

On proper reflection, especially openmotif, xvid and libvorbis seem a bit out-of-place in a @system build...

To pick an arbitrary example, openmotif is required by:
app-editors/xemacs-21.4.22-r2 (motif ? >=x11-libs/openmotif-2.3:0)
app-text/xdvik-22.84.16 (motif ? >=x11-libs/openmotif-2.3:0)
dev-util/ddd-3.3.12-r2 (>=x11-libs/openmotif-2.3:0)
media-libs/mesa-7.11.2 (motif ? x11-libs/openmotif)
sci-electronics/pcb-20100929 (!gtk ? >=x11-libs/openmotif-2.3:0)

I cannot readily figure out why this would be built as part of / a dependency for the system set in the first place.

Is there something like 'equery depgraph' going in the OPPOSITE direction, to help figure out where @system dependencies come from?
Or something which would in effect be 'equery depgraph @system'?

Currently, the @world build underway seems to be reasonably well-behaved (load avgs 4.95, 5.73, 5.10); but I've seen @world build degrade in a similar fashion as well.
Comment 4 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2012-02-15 15:48:53 UTC
(I'm not a portage dev, so I'll probably stop following this bug)

(In reply to comment #3)
> Is there something like 'equery depgraph' going in the OPPOSITE direction, to
> help figure out where @system dependencies come from?
> Or something which would in effect be 'equery depgraph @system'?

emerge -p --tree world, would probably help you.
Comment 5 Joe Breuer 2012-02-15 16:34:54 UTC
Created attachment 302053 [details]
Output of emerge -pe --tree @system

@darkside: Of course, thanks. Always forget that one ;-)

I still think I'm onto something here:

# emerge -pe --tree @system
These are the packages that would be merged, in reverse order:

Calculating dependencies  .... done!
[nomerge       ] media-plugins/gst-plugins-soup-0.10.30 
[nomerge       ]  net-libs/libsoup-2.36.1 
[nomerge       ]   net-libs/glib-networking-2.30.1-r1 
[nomerge       ]    net-libs/libproxy-0.4.7 
[nomerge       ]     kde-base/kdelibs-4.7.4 
[nomerge       ]      kde-base/khelpcenter-4.7.4 
[nomerge       ]       kde-base/kdesu-4.7.4 
[nomerge       ]        x11-libs/qt-webkit-4.7.4 
[nomerge       ]         media-libs/phonon-4.5.1-r1 
[nomerge       ]          media-libs/phonon-gstreamer-4.5.0 
[nomerge       ]           media-plugins/gst-plugins-meta-0.10-r6 
[nomerge       ]            media-plugins/gst-plugins-dv-0.10.30 
[nomerge       ]             media-libs/libdv-1.0.0-r2 
[nomerge       ]              media-libs/libsdl-1.2.14-r6 
[nomerge       ]               virtual/opengl-7.0 
[ebuild   R    ]                media-libs/mesa-7.11.2 
[ebuild   R    ]                 sys-devel/llvm-2.9-r2 
[nomerge       ] media-libs/phonon-gstreamer-4.5.0 
[ebuild   R    ]  media-plugins/gst-plugins-soup-0.10.30 
[ebuild   R    ]   net-libs/libsoup-2.36.1 
[ebuild   R    ]    net-libs/glib-networking-2.30.1-r1 
[ebuild   R    ]     net-libs/libproxy-0.4.7 

To me, this LOOKS LIKE the circular dependency between gst-plugins-soup and phonon-gstreamer bloats my @system set with all the stuff required by that sub-tree. I did not see a package requiring either gst-plugins-soup and phonon-gstreamer, apart from those related to the circular dependency.

I'm attaching the full output, perhaps I'm just blind and do not see the "real" dependency.
Comment 6 Zac Medico gentoo-dev 2012-02-15 17:23:44 UTC

*** This bug has been marked as a duplicate of bug 383439 ***
Comment 7 Zac Medico gentoo-dev 2012-06-24 08:20:36 UTC
I've found a bug that's triggered by the --load-average option:;a=commit;h=fe03b5fd790fc09bd6594c55bb174e80aaac1e5a
Comment 8 Zac Medico gentoo-dev 2012-06-25 21:54:36 UTC
This is fixed in and 2.2.0_alpha113.