Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 346749 - sys-apps/portage-2.2.0_alpha5: traceback while installing gentoo-sources (lstat OSError: [Errno 12] Cannot allocate memory)
Summary: sys-apps/portage-2.2.0_alpha5: traceback while installing gentoo-sources (lst...
Status: IN_PROGRESS
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal with 4 votes (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 320859 412191 (view as bug list)
Depends on:
Blocks: 240320
  Show dependency tree
 
Reported: 2010-11-25 13:03 UTC by Philipp Riegger
Modified: 2016-09-24 13:27 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Riegger 2010-11-25 13:03:02 UTC
I'm running Gentoo as Xen guest on CentOS host. While installing sys-kernel/gentoo-sources-2.6.36-r3 I get the following:

...
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/compiler-intel.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/neighbour.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/unistd.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/mISDNhw.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/threads.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/hpet.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/tc_em_cmp.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/Kbuild
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/tc_em_text.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/tc_em_nbyte.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/tc_ematch/tc_em_meta.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/bio.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/pch_dma.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/sort.h
Traceback (most recent call last):
  File "/usr/lib/portage/pym/portage/dbapi/_MergeProcess.py", line 43, in _spawn
    self.cfgfiledict, self.conf_mem_file)
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3439, in _merge_process
    if self.mergeme(srcroot, destroot, outfile, secondhand, "", cfgfiledict, mymtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3699, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3699, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3699, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3699, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3699, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3543, in mergeme
    mydstat = os.lstat(mydest)
  File "/usr/lib/portage/pym/portage/__init__.py", line 226, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 12] Cannot allocate memory: '/usr/src/linux-2.6.36-gentoo-r3/include/linux/interrupt.h'

>>> Failed to install sys-kernel/gentoo-sources-2.6.36-r3, Log file:

>>>  '/var/log/portage/sys-kernel:gentoo-sources-2.6.36-r3:20101125-122252.log'
 * 
 * The following package has failed to build or install:
 * 
 *  (sys-kernel/gentoo-sources-2.6.36-r3::gentoo, ebuild scheduled for merge), Log file:
 *   '/var/log/portage/sys-kernel:gentoo-sources-2.6.36-r3:20101125-122252.log'
 * 

 * GNU info directory index is up-to-date.





There should be enough disk space, inodes and ram:
playground ~ # df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                8.8G  7.6G  835M  91% /
/dev/root             8.8G  7.6G  835M  91% /
rc-svcdir             1.0M   52K  972K   6% /lib/rc/init.d
udev                   10M  132K  9.9M   2% /dev
shm                   375M     0  375M   0% /dev/shm
playground ~ # df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
rootfs                584064  385908  198156   67% /
/dev/root             584064  385908  198156   67% /
rc-svcdir              95860      61   95799    1% /lib/rc/init.d
udev                   95860     520   95340    1% /dev
shm                    95860       1   95859    1% /dev/shm
playground ~ # free
             total       used       free     shared    buffers     cached
Mem:        766884     667228      99656          0     105748     457476
-/+ buffers/cache:     104004     662880
Swap:      1004056       4856     999200
playground ~ # 




emerge --info:
Portage 2.2.0_alpha5 (hardened/linux/x86, gcc-4.5.1, glibc-2.12.1-r3, 2.6.35-gentoo-r13 i686)
=================================================================
System uname: Linux-2.6.35-gentoo-r13-i686-Intel-R-_Xeon-TM-_CPU_3.60GHz-with-gentoo-2.0.1
Timestamp of tree: Thu, 25 Nov 2010 09:15:01 +0000
app-shells/bash:     4.1_p9
dev-lang/python:     2.6.6-r1, 3.1.2-r4
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.1-r1
sys-apps/openrc:     0.6.5
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.68
sys-devel/automake:  1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.5, 4.5.1-r1
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.4
sys-devel/make:      3.82
virtual/os-headers:  2.6.36.1 (sys-kernel/linux-headers)
Repositories: gentoo playground_local java-overlay sunrise stoile_overlay wacfg
ACCEPT_KEYWORDS="x86 ~x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-Os -march=i686 -pipe -mno-tls-direct-seg-refs"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/openvpn/easy-rsa"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-Os -march=i686 -pipe -mno-tls-direct-seg-refs"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going --alphabetical --binpkg-respect-use y --load-average=3 --with-bdeps y"
FEATURES="assume-digests binpkg-logs buildpkg collision-protect distlocks fixlafiles fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox"
GENTOO_MIRRORS="http://ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo http://gentoo.inode.at/ http://distfiles.gentoo.org http://www.ibiblio.org/pub/Linux/distributions/gentoo"
INSTALL_MASK="*.la"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1 -s"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /var/lib/layman/java-overlay /var/lib/layman/sunrise /var/lib/layman/stoile /var/lib/layman/wacfg"
SYNC="rsync://rsync.de.gentoo.org/gentoo-portage"
USE="acl bash-completion berkdb bzip2 caps cli cracklib crypt cxx dri gdbm gs hardened iconv idn ipv6 jpeg modules mudflap ncurses nls nptl nptlonly pam pcre perl pic png pppd python readline session ssl svg sysfs tcpd tiff unicode urandom x86 xattr xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-2" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, FFLAGS, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS



I don't see anything in dmesg, syslog or anywhere else
Comment 1 Zac Medico gentoo-dev 2010-11-25 17:09:13 UTC
The difference in alpha5 is that the merge code now runs in a forked process. The fork is not supposed to consume too much memory, because kernels typically use copy-on-write memory pages for forked processes. Are you using ulimit or something like that to limit process virtual memory?
Comment 2 Philipp Riegger 2010-11-25 20:52:42 UTC
I do not use ulimit or anything like that, at least not that I know of.

Another thing is that I can't guarantee, that this is alpha5 specific. I will try some older portage versions.
Comment 3 Philipp Riegger 2010-11-25 21:49:56 UTC
2.2.0_alpha4 just died:

>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/intel_mid_dma.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/clockchips.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/drbd_limits.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/hdreg.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/textsearch.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/kprobes.h
Traceback (most recent call last):
  File "/usr/bin/emerge", line 43, in <module>
    retval = emerge_main()
  File "/usr/lib/portage/pym/_emerge/main.py", line 1698, in emerge_main
    myopts, myaction, myfiles, spinner)
  File "/usr/lib/portage/pym/_emerge/actions.py", line 443, in action_build
    retval = mergetask.merge()
  File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1129, in merge
    rval = self._merge()
  File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1447, in _merge
    self._main_loop()
  File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1596, in _main_loop
    self._poll_loop()
  File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 138, in _poll_loop
    handler(f, event)
  File "/usr/lib/portage/pym/_emerge/EbuildIpcDaemon.py", line 82, in _input_handler
    reply_hook()
  File "/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py", line 149, in _exit_command_callback
    self.scheduler.schedule(self._reg_id, timeout=self._exit_timeout)
  File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 232, in _schedule_wait
    handler(f, event)
  File "/usr/lib/portage/pym/_emerge/SpawnProcess.py", line 203, in _output_handler
    self._unregister_if_appropriate(event)
  File "/usr/lib/portage/pym/_emerge/AbstractPollTask.py", line 49, in _unregister_if_appropriate
    self.wait()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
    self._wait_hook()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib/portage/pym/_emerge/EbuildPhase.py", line 182, in _ebuild_exit
    self.wait()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
    self._wait_hook()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib/portage/pym/_emerge/EbuildBinpkg.py", line 46, in _package_phase_exit
    self.wait()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
    self._wait_hook()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib/portage/pym/_emerge/EbuildBuild.py", line 276, in _buildpkg_exit
    self.wait()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
    self._wait_hook()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib/portage/pym/_emerge/CompositeTask.py", line 105, in _default_final_exit
    return self.wait()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
    self._wait_hook()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1426, in _build_exit
    self._schedule()
  File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 52, in _schedule
    return self._schedule_tasks()
  File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1623, in _schedule_tasks
    if q.schedule():
  File "/usr/lib/portage/pym/_emerge/SequentialTaskQueue.py", line 55, in schedule
    task.start()
  File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 23, in start
    self._start()
  File "/usr/lib/portage/pym/_emerge/PackageMerge.py", line 43, in _start
    self.returncode = self.merge.merge()
  File "/usr/lib/portage/pym/_emerge/MergeListItem.py", line 147, in merge
    retval = self._install_task.install()
  File "/usr/lib/portage/pym/_emerge/EbuildBuild.py", line 324, in install
    rval = merge.execute()
  File "/usr/lib/portage/pym/_emerge/EbuildMerge.py", line 26, in execute
    blockers=self.find_blockers)
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3988, in merge
    mydbapi=mydbapi, prev_mtimes=prev_mtimes)
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3839, in merge
    mydbapi=mydbapi, prev_mtimes=prev_mtimes)
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3860, in _merge
    cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes)
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3225, in treewalk
    if self.mergeme(srcroot, destroot, outfile, secondhand, "", cfgfiledict, mymtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
    join(offset, x), cfgfiledict, thismtime):
  File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3516, in mergeme
    mydstat = os.lstat(mydest)
  File "/usr/lib/portage/pym/portage/__init__.py", line 226, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 12] Cannot allocate memory: '/usr/src/linux-2.6.36-gentoo-r3/include/linux/rcupdate.h'
Comment 4 Zac Medico gentoo-dev 2010-11-26 03:21:56 UTC
Okay, so apparently the addition of the fork is unrelated to your problem. Please post the output of `ulimit -a`.
Comment 5 Philipp Riegger 2010-11-26 10:32:35 UTC
For alpha5 it happened every time (so far), for alpha4 it happened the second time, I have not reproduced it with aplha3 yet. Will continue trying.

I run a hardened system (switched from normal to hardened recently). I don't run a hardened kernel.

Here's the ulimit:

playground ~ # ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 5988
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5988
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
playground ~ # 
Comment 6 Zac Medico gentoo-dev 2010-11-26 13:25:08 UTC
The ulimit output looks normal. I suspect that you've got some sort of kernel bug, because I've never seen lstat fail like that. Usually that type of error is triggered by a fork, like in bug 236491 and bug 320859.
Comment 7 Ortwin Glueck 2010-12-08 16:27:09 UTC
Here a simple:
emerge -1 gentoo-sources
process would eat up all memory on a 512MB machine that has no swap. This occurs while
installing from the sandbox image into the live filesystem.

This is not a kernel bug. The author of emerge apparently just assumed infinite memory :-)
Comment 8 Philipp Riegger 2011-04-06 19:33:55 UTC
Ok, this is really annoying. Why does portage use lots and lots of RAM while copying files to the filesystem? I would really like to be able to install a new kernel...
Comment 9 Zac Medico gentoo-dev 2011-04-06 19:40:33 UTC
(In reply to comment #8)
> Why does portage use lots and lots of RAM while
> copying files to the filesystem?

It's an unsolved mystery because it only affects a minority of people.
Comment 10 Philipp Riegger 2011-04-06 19:47:39 UTC
Btw., I'm at portage 2.2.0 alpha29 and get the error while installing the latest 2.6.37.x vanilla sources. Is there anything I can do to help? Try to switch to python2, use valgrind, whatever? I just don't know too much about python so I have no idea how to debug this.
Comment 11 Zac Medico gentoo-dev 2011-04-06 20:03:51 UTC
I've heard that Heapy (from dev-python/guppy package) is useful for analyzing memory consumption:

  http://guppy-pe.sourceforge.net/#Heapy

Maybe we can add a handler for ENOMEM that will automatically use Heapy to dump heap information to a file for later analysis.
Comment 12 Philipp Riegger 2011-04-06 20:12:43 UTC
Hmm, I just reinstalled portage.

Before: USE="-python2 python3"
Now: USE="python2 -python3"

And I successfully installed vanilla sources. Which did not work before several times in a row (5 times or something). Not sure, if this is by mistake. watching top (sorted by memory usage) while installing still shows the RAM usage rising while copying the files to the filesystem (/usr/src in this case), but it feels a little slower with python2. I cannot proof this in anyway, of course, no statistics, tests or anything. I'll watch this and see if I hit the bug again with python2.
Comment 13 Zac Medico gentoo-dev 2012-04-17 15:39:48 UTC
*** Bug 412191 has been marked as a duplicate of this bug. ***
Comment 14 Maxim Kammerer 2012-07-04 17:16:47 UTC
I often observe this bug with stable portage 2.1.10.65 and hardened-sources-3.2.11 (on non-hardened system). However, the bug disappears with FEATURES=-userpriv.

# free -m
             total       used       free     shared    buffers     cached
Mem:          1969       1530        439          0         73        979
-/+ buffers/cache:        477       1492
Swap:         2000         52       1947

# portageq envvar FEATURES
assume-digests binpkg-logs collision-protect compress-build-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg
Comment 15 Maxim Kammerer 2012-07-20 00:30:21 UTC
(In reply to comment #12)
> And I successfully installed vanilla sources. Which did not work before
> several times in a row (5 times or something).

Same here (FEATURES=-userpriv did not help, however — it was probably accidental previoooously).
Comment 16 Josh G 2013-01-24 21:03:08 UTC
I ran into this bug on a i686 VPS and an ARM NAS I've been getting Gentoo running on.  Both have 128MB of memory and 512MB swap.

Traced it to Python 3 using >3x the memory during the Merge stage that Python 2 did.  Python 2 could do the merge needing 107MB in "Virt" memory while Python 3 crashed out for me with 305 MB of "Virt" memory.  Readings were taken from top.

Interestingly, the kernel's OOM was never triggered by the memory allocation failure.

Not sure why there's such a big difference between the two python versions, if time allows, I hope to profile it in Guppy (http://guppy-pe.sourceforge.net) to see if I could find where/why it happens.

-J
Comment 17 Marios Andreopoulos 2013-02-22 03:11:11 UTC
I run on this behavior frequently but most of my machines have enough RAM so that portage's RAM consumption during emerge of {gentoo,hardened}-sources isn't a problem.

Sadly I can not run Gentoo on a 512MB VPS anymore and it is a bit off an issue on a 1024MB VPS, since in order to emerge hardened-sources, many of my running services and/or caches move to the swap, slowing the server.

It happens on a native ~amd system and on three ~amd hardened systems (one native, one kvm based, one xen based).

Here is a screenshot of portage using 90% of memory on a 512MiB KVM guest, moments before it runs out of memory.
http://www.andmarios.com/attachments/portage1.png

If I can do something to help, please ask.


# emerge --info
Portage 2.1.11.52 (hardened/linux/amd64, gcc-4.6.3, glibc-2.16.0, 3.7.4-hardened-r1-gwhp x86_64)
=================================================================
System uname: Linux-3.7.4-hardened-r1-gwhp-x86_64-Intel_Core_i7_9xx_-Nehalem_Class_Core_i7-with-gentoo-2.2
KiB Mem:     1022700 total,     75512 free
KiB Swap:     524284 total,    271572 free
Timestamp of tree: Fri, 22 Feb 2013 00:45:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p42
dev-lang/python:          2.7.3-r3, 3.2.3-r2
dev-util/cmake:           2.8.10.2-r1
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6
sys-devel/autoconf:       2.69
sys-devel/automake:       1.12.6, 1.13.1
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.6.3
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.7 (virtual/os-headers)
sys-libs/glibc:           2.16.0
Repositories: gentoo x-personal
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch webrsync-gpg"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/portage/local/personal"
SYNC=""
USE="acl amd64 berkdb bzip2 cli cracklib crypt cxx dri gdbm gmp hardened iconv icu ipv6 justify mmx modules mudflap multilib ncurses nls nptl openmp pam pax_kernel pcre readline session sse sse2 ssl tcpd udev unicode urandom zlib" ABI_X86="64" APACHE2_MODULES="actions access_compat alias auth_basic authn_alias authn_anon authn_core authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner auth_digest authn_dbd authz_user autoindex cache cache_disk cgi deflate dir disk_cache env slotmem_shm expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif socache_shmcb speling status reqtimeout lbmethod_byrequests unique_id unixd usertrack vhost_alias proxy proxy_fcgi proxy_http proxy_balancer lbmethod_bytraffic" APACHE2_MPMS="event" COLLECTD_PLUGINS="aggregation apache conntrack contextswitch cpu csv curl curl_json curl_xml dbi df disk dns entropy ethstat exec filecount fscache interface iptables irq load logfile match_regex match_timediff match_value md memory mysql netlink notify_email ntpd openvpn ping powerdns processes protocols rrdtool rrdcached swap syslog table tail tcpconns threshold unixsock uptime users varnish vmem" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby19" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 18 Maxim Kammerer 2013-08-20 23:18:40 UTC
Just wanted to note that the bug is not fixed yet.
Comment 19 Zac Medico gentoo-dev 2015-03-05 05:55:25 UTC
*** Bug 320859 has been marked as a duplicate of this bug. ***
Comment 20 Tom Wijsman (TomWij) (RETIRED) gentoo-dev 2016-09-02 20:21:16 UTC
(In reply to Maxim Kammerer from comment #18)
> Just wanted to note that the bug is not fixed yet.

Reproduced by a #gentoo user with 2GB RAM while emerging =sys-kernel/gentoo-sources-4.4.6 with similar error "OSError: [Errno 12] Cannot allocate memory"; so, this bug still seems to be lingering around
Comment 21 Alice Ferrazzi Gentoo Infrastructure gentoo-dev 2016-09-24 13:27:46 UTC
please update portage to 2.2.28 and try again
I had similar problem and upgrading solved it