Sometimes, running "emerge -Dup world" creates an OOM condition, although the log shows that there is still virtual memory remaining. While running "emerge udev" today, I lost a Firefox session, and in return got the following log entry: ---------------begin "dmesg" output oom-killer: gfp_mask=0xd0 DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 30, high 90, batch 15 cpu 0 cold: low 0, high 30, batch 15 HighMem per-cpu: empty Free pages: 2584kB (0kB HighMem) Active:37830 inactive:2512 dirty:0 writeback:0 unstable:0 free:646 slab:5444 mapped:40302 pagetables:256 DMA free:872kB min:144kB low:180kB high:216kB active:7936kB inactive:496kB present:16384kB pages_scanned:3772 all_unreclaimable? no lowmem_reserve[]: 0 175 175 Normal free:1712kB min:1624kB low:2028kB high:2436kB active:143384kB inactive:9552kB present:180032kB pages_scanned:61200 all_unreclaimable? no lowmem_reserve[]: 0 0 0 HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 DMA: 8*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 872kB Normal: 30*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1712kB HighMem: empty Swap cache: add 26, delete 26, find 0/0, race 0+0 Free swap = 393008kB Total swap = 393112kB Out of Memory: Killed process 10672 (firefox-bin). Out of Memory: Killed process 10690 (firefox-bin). Out of Memory: Killed process 10691 (firefox-bin). Out of Memory: Killed process 10693 (firefox-bin). Out of Memory: Killed process 10694 (firefox-bin). Out of Memory: Killed process 10696 (firefox-bin). Out of Memory: Killed process 10697 (firefox-bin). Out of Memory: Killed process 10698 (firefox-bin). Out of Memory: Killed process 10705 (firefox-bin). ---------------end "dmesg" output If the system has "Free swap = 393008kB", why is the OOM-Killer chopping down processes? I have tried setting "vm.swappiness = 0" in /etc/sysctl.conf, I have tried trimming the kernel down by using modules everywhere possible, I have tried running Memtest86 in case I have bad memory somewhere. None of these has stopped the OOM-Killer, or shown me where the problem lies. Two symptoms seem to be the case in every instance: 1. Firefox is the OOM-Killer's favorite target during "emerge". 2. The ebuild must include compilation. "gentoo-sources" (source only) and "openoffice-bin" (pre-compiled) have not yet caused OOM-Killer problems. Everything I've said above is true on my laptop, with 192MB, as of today. However, I have seen the same behavior on my desktop system, with 768MB of memory! "emerge blah" is less likely to cause dead programs, but I have seen it happen there as well. When compiling, Firefox gets zapped. After this happens, I can usually re-start the killed program (Firefox), and have no problem, even though "emerge" is still compiling something. Reproducible: Sometimes Steps to Reproduce: 1. Start X, or log in to XDM/GDM/KDM. Start Firefox. 2. Switch to a virtual console, log in as root, and run "emerge -Dup some-big-package". 3. Switch back to X and surf the net. Actual Results: Sometimes, Firefox dies. Sometimes, not. (laptop config follows:) anna ~ # emerge info Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r1, 2.6.12-gentoo-r10 i686) ================================================================= System uname: 2.6.12-gentoo-r10 i686 Pentium III (Coppermine) Gentoo Base System version 1.6.13 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.3.5-r2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6 sys-devel/binutils: 2.15.92.0.2-r10 sys-devel/libtool: 1.5.18-r1 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -pipe -march=pentium3 -mcpu=pentium3 -fomit-frame-pointer -ffast-math" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /var/bind /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -pipe -march=pentium3 -mcpu=pentium3 -fomit-frame-pointer -ffast-math" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://mirror.usu.edu/mirrors/gentoo/ http://prometheus.cs.wmich.edu/gentoo http://modzer0.cs.uaf.edu/public/gentoo/" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X acpi alsa apm arts avi bash-completion berkdb bitmap-fonts bonobo bzlib cdr crypt cscope cups curl dga eds emboss encode esd fam foomaticdb fortran gcj gdbm gif gnome gnutls gpm gstreamer gtk gtk2 gtkhtml guile howl imagemagick imlib ipv6 java jpeg kde lesstif libg++ libwww mad mikmod mmap mmx motif mozilla mp3 mpeg ncurses nls objc ogg oggvorbis opengl oss pam pcmcia pdflib perl png posix python qt quicktime readline sdl sharedmem slang spell sse ssl svga sysvipc tcltk tcpd threads tiff truetype truetype-fonts type1-fonts vorbis xml xml2 xmms xv zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTDIR_OVERLAY (desktop config follows:) joseph gus3 # emerge info Portage 2.0.51.22-r2 (default-linux/x86/2005.0, gcc-3.3.6, glibc-2.3.5-r1, 2.6.12-gentoo-r10 i686) ================================================================= System uname: 2.6.12-gentoo-r10 i686 AMD Athlon(tm) XP 2400+ Gentoo Base System version 1.6.13 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled] dev-lang/python: 2.3.5-r2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6 sys-devel/binutils: 2.15.92.0.2-r10 sys-devel/libtool: 1.5.18-r1 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=athlon-xp -mmmx -m3dnow -msse -mfpmath=sse" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=athlon-xp -mmmx -m3dnow -msse -mfpmath=sse" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distcc distlocks sandbox sfperms strict" GENTOO_MIRRORS="http://gentoo.mirrors.tds.net/gentoo http://open-systems.ufl.edu/mirrors/gentoo http://gentoo.chem.wisc.edu/gentoo/" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X Xaw3d a52 aac aalib acl acpi aim alsa apache2 apm arts audiofile avi bash-completion bcmath berkdb bidi bindist bitmap-fonts bonobo bzlib caps cdparanoia cdr cjk crypt cscope ctype cups curl dbm dbx directfb dvd dvdread eds emboss encode esd ethereal examples exif expat fam fastcgi fbcon ffmpeg flac foomatic foomaticdb fortran ftp gcj gd gdbm gif glut gmp gnome gnustep gphoto2 gpm gstreamer gtk gtk2 gtkhtml guile hal iconv imagemagick imap imlib ipv6 java javascript joystick jpeg junit kde lesstif libcaca libg++ libwww lirc lm_sensors mad mikmod mime mmap mmx mng motif mozilla mozsvg mp3 mpeg mysql ncurses nls nptl nvidia objc odbc ogg oggvorbis opengl oss pam pdflib perl php pic png posix ppds python qt quicktime readline sasl sdl sharedmem slang slp sndfile sockets sox spell sse ssl streamzap svg svga symlink sysfs sysvipc tcltk tcpd tetex theora threads tidy tiff tokenizer truetype truetype-fonts type1-fonts unicode usb vcd vorbis win32codecs wmf wxwindows xine xinerama xml xml2 xmlrpc xmms xosd xpm xprint xrandr xsl xv xvid yahoo zlib userland_GNU kernel_linux elibc_glibc" Unset: ASFLAGS, CTARGET, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTDIR_OVERLAY
This isn't portage's fault, nor problem really. Portage requires a chunk of memory, but nothing that's going to hog a significant portion of mem (compared to firefox fex). OOMK waxes the process hogging the most memory; if you run out of memory, it's going to get killed. plain and simple (this is assuming OOMK is in use). If portage is consuming insane amounts of memory, sure, then there is a problem, but willing to bet it's not. Assuming you're lacking swap also, if oomk is kicking in.
Actually, this looks more like your OOMK is being whacky. Bouncing it to kernel crew, either way it's not portage doing it (or you can tar and feather me) :P
Can you try to reproduce this with gentoo-sources-2.6.13-r1 and in case this is reproducible in gentoo-sources, try vanilla-sources-2.6.13.1 please?
sorry, forgot to mention, vanilla-sources-2.6.14_rc1 will be in portage soon (it's already in CVS) so try with this one too.
Do you mean try to get the OOMK while emerging kernel sources, or build the kernel sources and then try to get the OOMK on the new kernel?
Build the new kernel, boot it, then reproduce the problem.
No worries, Brian. No tar and feathers. Not even tar and bzip2. ;-) I got an OOMK incident today, with having just "too much" open. I don't understand why, as swap was only about 10% used. But running Firefox, Thunderbird, and OpenOffice.org with two documents, then trying to launch Acrobat Reader, ended up zapping acroread. There was no emerge or gcc running on my system. I suppose I need to look into tuning the OOMK. I'll search Google, but if anyone can point me to a document explaining, I would appreciate it greatly.
