Bug 402335 – emerge exceeds maximum recursion depth limit

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Richard Yao (RETIRED) 2012-02-06 02:27:05 UTC]

Created attachment 300983 [details]
Output generated

Doing `emerge --jobs --keep-going -ve @world 2>&1 | tee >(wgetpaste)` will result in portage exceeding the maximum recursion depth limit on my system.

The system python is python 3.2.

# emerge --info portage
Portage 2.1.10.45 (default/linux/amd64/10.0/desktop/kde, gcc-4.5.3, glibc-2.13-r4, 3.1.6 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-3.1.6-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q9550_@_2.83GHz-with-gentoo-2.1
Timestamp of tree: Sun, 05 Feb 2012 23:15:01 +0000
ccache version 3.1.7 [enabled]
app-shells/bash:          4.2_p20
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/ccache:          3.1.7
dev-util/cmake:           2.8.7-r2
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.10.3, 1.11.3
sys-devel/binutils:       2.22.52.0.1
sys-devel/gcc:            4.4.6-r1, 4.5.3-r2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.2 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo local_overlay mythtv sunrise vmware bitcoin sabayon x-gentoofan
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -mtune=core2 -mcx16 -msahf -msse4.1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=6144 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=core2 -mtune=core2 -mcx16 -msahf -msse4.1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=6144 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs buildpkg ccache distlocks ebuild-locks fixlafiles multilib-strict news parallel-fetch parallel-install protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox"
FFLAGS="-march=core2 -mtune=core2 -mcx16 -msahf -msse4.1 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=6144 -O2 -pipe"
GENTOO_MIRRORS="http://mirror.lug.udel.edu/pub/gentoo/ http://gentoo.osuosl.org/ ftp://mirrors.rit.edu/gentoo/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--sort-common"
LINGUAS="en"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /usr/local/mythtv_portage/Gentoo /var/lib/layman/sunrise /var/lib/layman/vmware /var/lib/layman/bitcoin /var/lib/layman/sabayon /var/lib/layman/gentoofan"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="X a52 aac acpi alsa amd64 bash-completion berkdb branding bzip2 cairo cdda cdr cjk cli consolekit cracklib crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam ffmpeg fftw firefox flac fontconfig fortran gdbm gdu gif gles gnutls gpm iconv ipv6 java jpeg kde kipi lcms ldap libnotify lzma mad mmap mmx mng modules mp3 mp4 mpeg mudflap multilib ncurses nls nptl nptlonly nsplugin ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pppd qt3support qt4 readline sdl session spell sse sse2 sse3 ssl ssse3 startup-notification svg sysfs tcpd theora tiff truetype udev unicode usb vdpau vorbis x264 xcb xcomposite xinerama xml xorg xscreensaver xulrunner xv xvid xvmc zlib zsh-completion" ALSA_CARDS="hda-intel hpet hrtimer" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="*" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" PHP_TARGETS="php5-3" QEMU_SOFTMMU_TARGETS="mips mips64 mips64el mipsel x86_64" QEMU_USER_TARGETS="mips mipsel x86_64" RUBY_TARGETS="ruby19" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

sys-apps/portage-2.1.10.45 was built with the following:
USE="(consolekit) (ipc) (multilib) (policykit) -build -doc -epydoc -python2 -python3 (-selinux) -xattr" LINGUAS="-pl"

[Comment 1 Zac Medico 2012-02-06 04:22:57 UTC]

Your system got overloaded, which is no surprise since it really doesn't make sense to use --jobs without --load-average.

[Comment 2 Richard Yao (RETIRED) 2012-02-06 04:43:47 UTC]

(In reply to comment #1)
> Your system got overloaded, which is no surprise since it really doesn't make
> sense to use --jobs without --load-average.

The odd thing is that I have reached load averages of over 100 when emerging KDE in the past without any loss of function. Is there any particular reason why emerge should not function under these circumstances?

[Comment 3 Zac Medico 2012-02-06 05:25:56 UTC]

We may be able to re-organize the stack handling, or simply use sys.setrecursionlimit() to increase the limit. The default limit is 1000.

Still, I suspect that your system was loaded to an unreasonable level. You had a sustained load average of 11.8, and 103 concurrent builds executing when it exceeded the limit.

[Comment 4 Zac Medico 2012-02-06 19:21:27 UTC]

(In reply to comment #3)
> We may be able to re-organize the stack handling

We can add a timeout method to the scheduler, like the glib.timeout_add() function from pygobject, and use that instead of recursion inside AbstractEbuildProcess._exit_command_callback().

[Comment 5 Zac Medico 2012-02-07 03:11:08 UTC]

I've got a set of patches for this that I'll push as soon as git.overlays.gentoo.org comes back online:

http://dev.gentoo.org/~zmedico/tmp/bug_402335/

[Comment 6 Richard Yao (RETIRED) 2012-02-07 09:02:41 UTC]

(In reply to comment #5)
> I've got a set of patches for this that I'll push as soon as
> git.overlays.gentoo.org comes back online:
> 
> http://dev.gentoo.org/~zmedico/tmp/bug_402335/

These resolve my issue. Thanks. :)

[Comment 7 Zac Medico 2012-02-07 23:08:46 UTC]

Thanks for testing. The patches are in git now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e9d1125f6730c85c4b384a580da55da68338acf1
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=839f369f259a7c019c1f544bf4c153058355f6d4
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=445a6ea22c132f4c06c2cc7c48ec6e7af7116962

[Comment 8 Zac Medico 2012-02-12 07:11:01 UTC]

This is fixed in 2.1.10.46 and 2.2.0_alpha86.

[Comment 9 Zac Medico 2012-02-12 07:11:01 UTC]

This is fixed in 2.1.10.46 and 2.2.0_alpha86.

[Comment 10 Zac Medico 2012-02-14 02:39:24 UTC]

Here's a fix for a similar recursion problem that can be triggered by Ctrl-C:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=07823ba56f63309da9547e02e96b043005932be0