Bug 337193 – Portage leaves processes running if it runs out of space

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Alec Meyers 2010-09-14 00:22:06 UTC]

While compiling openoffice, I ran out of space and found that even though portage crashed, some processes were left running. This happened to me twice.

Reproducible: Always

Steps to Reproduce:
1. Mount a small tmpfs partition over /var/tmp/portage
2. Emerge something that won't fit
Actual Results:  
Some portage processes are left running:

root     25236  0.0  0.0   3972   608 pts/2    S    16:57   0:00 [app-office/openoffice-3.2.1] sandbox "/usr/lib64/portage/bin/misc-functions.sh" die_hooks
root     25237  0.0  0.0  21076  2620 pts/2    S    16:57   0:00 /bin/bash /usr/lib64/portage/bin/misc-functions.sh die_hooks
root     25333  0.0  0.1  68716  6672 pts/2    S    16:57   0:00 /usr/bin/python2.6 /usr/lib64/portage/bin/ebuild-ipc.py exit 1


Expected Results:  
Nothing should be left running after portage quit.

Portage 2.2_rc81 (default/linux/amd64/10.0/desktop, gcc-4.5.1, glibc-2.12.1-r1, 2.6.35.4 x86_64)
=================================================================
System uname: Linux-2.6.35.4-x86_64-Intel-R-_Core-TM-_i7_CPU_M_620_@_2.67GHz-with-gentoo-2.0.1
Timestamp of tree: Mon, 13 Sep 2010 18:15:02 +0000
app-shells/bash:     4.1_p7
dev-java/java-config: 2.1.11
dev-lang/python:     2.6.5-r3
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.6.3
sys-apps/sandbox:    2.3-r1
sys-devel/autoconf:  2.67
sys-devel/automake:  1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.5.1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
virtual/os-headers:  2.6.35 (sys-kernel/linux-headers)
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA PUEL skype-eula dlj-1.1 AdobeFlash-10.1"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe -fno-ident -floop-interchange -floop-strip-mine -floop-block -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe -fno-ident -floop-interchange -floop-strip-mine -floop-block -ggdb"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests distlocks fixlafiles fixpackages news preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.chem.wisc.edu/gentoo http://gentoo.osuosl.org/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
LINGUAS="en"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_COMPRESS="gzip"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage/tree/official"
PORTDIR_OVERLAY="/usr/portage/tree/layman/qting-edge /usr/portage/tree/local"
SYNC="rsync://rsync21.us.gentoo.org/gentoo-portage"
USE="X a52 aac aalib acl acpi aio alsa amd64 audiofile avi bash-completion berkdb bluetooth branding bzip2 cairo cdparanoia cdr chm cli cracklib crypt css cups curl custom-optimization cxx dbus dell dirac dri dts dvd dvdr emboss encode exif expat ffmpeg fftw firefox flac foomaticdb fortran gif git gmp gpm gtk iconv imagemagick imlib inotify java5 java6 jpeg kde kdehiddenvisibility kvm laptop latex lcms logrotate lqr lzma lzo mad mikmod mmx mmxext mng modules mp3 mp4 mpeg mplayer mudflap multilib musepack ncurses nptl nptlonly nsplugin nvidia ogg opengl openmp openssl pam pango pch pcre pdf perl plasma png ppds pppd python qt qt3support qt4 quicktime rar readline reflection resolvconf rtmp samba schroedinger sdl session sndfile speex spell sqlite sqlite3 srt sse sse2 sse3 sse4 ssl ssse3 startup-notification subversion svg sysfs tcpd theora threads tiff tokyocabinet truetype unicode usb v4l v4l2 vaapi vcd videos vorbis vpx webkit wifi wmf x264 xcb xcomposite xorg xscreensaver xulrunner xv xvid xvmc xz zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" QEMU_SOFTMMU_TARGETS="arm i386 x86_64" QEMU_USER_TARGETS="arm i386 x86_64" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="intel" XFCE_PLUGINS="logout menu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS

[Comment 1 Alec Meyers 2010-09-14 00:23:02 UTC]

Created attachment 247207 [details]
Full output of "ps aux"

[Comment 2 Alec Meyers 2010-09-14 00:27:01 UTC]

Created attachment 247208 [details]
End of build.log

[Comment 3 Zac Medico 2010-09-14 00:39:26 UTC]

We have a "balancing act" of sorts going on between bug #335777 and bug #336142. In 2.2_rc81, the timeout is currently set at 15 minutes as mentioned in bug #336142, comment #21. There are some fixes in git, mentioned in bug #336142, comment #23 that should make it safer to decrease the timeout to a smaller value.

Ideally, we'd be able to eliminate the timeout entirely, but I haven't found a good way to do that yet that doesn't create the possibility of leaving processes running indefinitely.

[Comment 4 Zac Medico 2010-09-14 00:47:18 UTC]

(In reply to comment #3)
> Ideally, we'd be able to eliminate the timeout entirely, but I haven't found a
> good way to do that yet that doesn't create the possibility of leaving
> processes running indefinitely.

Two possible solutions to this that come to mind are:
1) Ensure the the parent process always kills the descendent processes (ebuild-ipc being the one that's hung in this case).
2) Make the ebuild-ipc process detect when the parent process has died, so that it won't timeout as long as the parent process is still alive.

[Comment 5 Zac Medico 2010-09-14 08:31:26 UTC]

(In reply to comment #4)
> 2) Make the ebuild-ipc process detect when the parent process has died, so that
> it won't timeout as long as the parent process is still alive.

This approach is implemented in git now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f055cd8f241263c67cf53ed1a7b52ffa8e671fc2

It tries a non-blocking lock on a lock file that's held by the parent process, in order to check if it's still alive.

[Comment 6 Zac Medico 2010-09-14 08:32:23 UTC]

The timeout is 15 seconds now which should be much more reasonable.

[Comment 7 Zac Medico 2010-09-15 00:44:06 UTC]

This is fixed in 2.1.9.6 and 2.2_rc82.