Summary: | sys-apps/portage-2.2.0_alpha5: traceback while installing gentoo-sources (lstat OSError: [Errno 12] Cannot allocate memory) | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Philipp Riegger <bugs+gentoo> |
Component: | [OLD] Core system | Assignee: | Portage team <dev-portage> |
Status: | IN_PROGRESS --- | ||
Severity: | normal | CC: | dschridde+gentoobugs, kernel, mk, mpagano, odi, opensource, sde, semen, tiago, tomwij |
Priority: | High | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: | https://bugs.gentoo.org/show_bug.cgi?id=320859 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 240320 |
Description
Philipp Riegger
2010-11-25 13:03:02 UTC
The difference in alpha5 is that the merge code now runs in a forked process. The fork is not supposed to consume too much memory, because kernels typically use copy-on-write memory pages for forked processes. Are you using ulimit or something like that to limit process virtual memory? I do not use ulimit or anything like that, at least not that I know of. Another thing is that I can't guarantee, that this is alpha5 specific. I will try some older portage versions. 2.2.0_alpha4 just died:
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/intel_mid_dma.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/clockchips.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/drbd_limits.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/hdreg.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/textsearch.h
>>> /usr/src/linux-2.6.36-gentoo-r3/include/linux/kprobes.h
Traceback (most recent call last):
File "/usr/bin/emerge", line 43, in <module>
retval = emerge_main()
File "/usr/lib/portage/pym/_emerge/main.py", line 1698, in emerge_main
myopts, myaction, myfiles, spinner)
File "/usr/lib/portage/pym/_emerge/actions.py", line 443, in action_build
retval = mergetask.merge()
File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1129, in merge
rval = self._merge()
File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1447, in _merge
self._main_loop()
File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1596, in _main_loop
self._poll_loop()
File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 138, in _poll_loop
handler(f, event)
File "/usr/lib/portage/pym/_emerge/EbuildIpcDaemon.py", line 82, in _input_handler
reply_hook()
File "/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py", line 149, in _exit_command_callback
self.scheduler.schedule(self._reg_id, timeout=self._exit_timeout)
File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 232, in _schedule_wait
handler(f, event)
File "/usr/lib/portage/pym/_emerge/SpawnProcess.py", line 203, in _output_handler
self._unregister_if_appropriate(event)
File "/usr/lib/portage/pym/_emerge/AbstractPollTask.py", line 49, in _unregister_if_appropriate
self.wait()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
self._wait_hook()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
self._exit_listener_stack.pop()(self)
File "/usr/lib/portage/pym/_emerge/EbuildPhase.py", line 182, in _ebuild_exit
self.wait()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
self._wait_hook()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
self._exit_listener_stack.pop()(self)
File "/usr/lib/portage/pym/_emerge/EbuildBinpkg.py", line 46, in _package_phase_exit
self.wait()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
self._wait_hook()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
self._exit_listener_stack.pop()(self)
File "/usr/lib/portage/pym/_emerge/EbuildBuild.py", line 276, in _buildpkg_exit
self.wait()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
self._wait_hook()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
self._exit_listener_stack.pop()(self)
File "/usr/lib/portage/pym/_emerge/CompositeTask.py", line 105, in _default_final_exit
return self.wait()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 41, in wait
self._wait_hook()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 114, in _wait_hook
self._exit_listener_stack.pop()(self)
File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1426, in _build_exit
self._schedule()
File "/usr/lib/portage/pym/_emerge/PollScheduler.py", line 52, in _schedule
return self._schedule_tasks()
File "/usr/lib/portage/pym/_emerge/Scheduler.py", line 1623, in _schedule_tasks
if q.schedule():
File "/usr/lib/portage/pym/_emerge/SequentialTaskQueue.py", line 55, in schedule
task.start()
File "/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line 23, in start
self._start()
File "/usr/lib/portage/pym/_emerge/PackageMerge.py", line 43, in _start
self.returncode = self.merge.merge()
File "/usr/lib/portage/pym/_emerge/MergeListItem.py", line 147, in merge
retval = self._install_task.install()
File "/usr/lib/portage/pym/_emerge/EbuildBuild.py", line 324, in install
rval = merge.execute()
File "/usr/lib/portage/pym/_emerge/EbuildMerge.py", line 26, in execute
blockers=self.find_blockers)
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3988, in merge
mydbapi=mydbapi, prev_mtimes=prev_mtimes)
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3839, in merge
mydbapi=mydbapi, prev_mtimes=prev_mtimes)
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3860, in _merge
cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes)
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3225, in treewalk
if self.mergeme(srcroot, destroot, outfile, secondhand, "", cfgfiledict, mymtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
join(offset, x), cfgfiledict, thismtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
join(offset, x), cfgfiledict, thismtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
join(offset, x), cfgfiledict, thismtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
join(offset, x), cfgfiledict, thismtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3672, in mergeme
join(offset, x), cfgfiledict, thismtime):
File "/usr/lib/portage/pym/portage/dbapi/vartree.py", line 3516, in mergeme
mydstat = os.lstat(mydest)
File "/usr/lib/portage/pym/portage/__init__.py", line 226, in __call__
rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 12] Cannot allocate memory: '/usr/src/linux-2.6.36-gentoo-r3/include/linux/rcupdate.h'
Okay, so apparently the addition of the fork is unrelated to your problem. Please post the output of `ulimit -a`. For alpha5 it happened every time (so far), for alpha4 it happened the second time, I have not reproduced it with aplha3 yet. Will continue trying. I run a hardened system (switched from normal to hardened recently). I don't run a hardened kernel. Here's the ulimit: playground ~ # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 5988 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 5988 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited playground ~ # The ulimit output looks normal. I suspect that you've got some sort of kernel bug, because I've never seen lstat fail like that. Usually that type of error is triggered by a fork, like in bug 236491 and bug 320859. Here a simple: emerge -1 gentoo-sources process would eat up all memory on a 512MB machine that has no swap. This occurs while installing from the sandbox image into the live filesystem. This is not a kernel bug. The author of emerge apparently just assumed infinite memory :-) Ok, this is really annoying. Why does portage use lots and lots of RAM while copying files to the filesystem? I would really like to be able to install a new kernel... (In reply to comment #8) > Why does portage use lots and lots of RAM while > copying files to the filesystem? It's an unsolved mystery because it only affects a minority of people. Btw., I'm at portage 2.2.0 alpha29 and get the error while installing the latest 2.6.37.x vanilla sources. Is there anything I can do to help? Try to switch to python2, use valgrind, whatever? I just don't know too much about python so I have no idea how to debug this. I've heard that Heapy (from dev-python/guppy package) is useful for analyzing memory consumption: http://guppy-pe.sourceforge.net/#Heapy Maybe we can add a handler for ENOMEM that will automatically use Heapy to dump heap information to a file for later analysis. Hmm, I just reinstalled portage. Before: USE="-python2 python3" Now: USE="python2 -python3" And I successfully installed vanilla sources. Which did not work before several times in a row (5 times or something). Not sure, if this is by mistake. watching top (sorted by memory usage) while installing still shows the RAM usage rising while copying the files to the filesystem (/usr/src in this case), but it feels a little slower with python2. I cannot proof this in anyway, of course, no statistics, tests or anything. I'll watch this and see if I hit the bug again with python2. *** Bug 412191 has been marked as a duplicate of this bug. *** I often observe this bug with stable portage 2.1.10.65 and hardened-sources-3.2.11 (on non-hardened system). However, the bug disappears with FEATURES=-userpriv. # free -m total used free shared buffers cached Mem: 1969 1530 439 0 73 979 -/+ buffers/cache: 477 1492 Swap: 2000 52 1947 # portageq envvar FEATURES assume-digests binpkg-logs collision-protect compress-build-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg (In reply to comment #12) > And I successfully installed vanilla sources. Which did not work before > several times in a row (5 times or something). Same here (FEATURES=-userpriv did not help, however — it was probably accidental previoooously). I ran into this bug on a i686 VPS and an ARM NAS I've been getting Gentoo running on. Both have 128MB of memory and 512MB swap. Traced it to Python 3 using >3x the memory during the Merge stage that Python 2 did. Python 2 could do the merge needing 107MB in "Virt" memory while Python 3 crashed out for me with 305 MB of "Virt" memory. Readings were taken from top. Interestingly, the kernel's OOM was never triggered by the memory allocation failure. Not sure why there's such a big difference between the two python versions, if time allows, I hope to profile it in Guppy (http://guppy-pe.sourceforge.net) to see if I could find where/why it happens. -J I run on this behavior frequently but most of my machines have enough RAM so that portage's RAM consumption during emerge of {gentoo,hardened}-sources isn't a problem. Sadly I can not run Gentoo on a 512MB VPS anymore and it is a bit off an issue on a 1024MB VPS, since in order to emerge hardened-sources, many of my running services and/or caches move to the swap, slowing the server. It happens on a native ~amd system and on three ~amd hardened systems (one native, one kvm based, one xen based). Here is a screenshot of portage using 90% of memory on a 512MiB KVM guest, moments before it runs out of memory. http://www.andmarios.com/attachments/portage1.png If I can do something to help, please ask. # emerge --info Portage 2.1.11.52 (hardened/linux/amd64, gcc-4.6.3, glibc-2.16.0, 3.7.4-hardened-r1-gwhp x86_64) ================================================================= System uname: Linux-3.7.4-hardened-r1-gwhp-x86_64-Intel_Core_i7_9xx_-Nehalem_Class_Core_i7-with-gentoo-2.2 KiB Mem: 1022700 total, 75512 free KiB Swap: 524284 total, 271572 free Timestamp of tree: Fri, 22 Feb 2013 00:45:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p42 dev-lang/python: 2.7.3-r3, 3.2.3-r2 dev-util/cmake: 2.8.10.2-r1 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6 sys-devel/autoconf: 2.69 sys-devel/automake: 1.12.6, 1.13.1 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.8 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.7 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo x-personal ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=native" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=native" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch webrsync-gpg" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/personal" SYNC="" USE="acl amd64 berkdb bzip2 cli cracklib crypt cxx dri gdbm gmp hardened iconv icu ipv6 justify mmx modules mudflap multilib ncurses nls nptl openmp pam pax_kernel pcre readline session sse sse2 ssl tcpd udev unicode urandom zlib" ABI_X86="64" APACHE2_MODULES="actions access_compat alias auth_basic authn_alias authn_anon authn_core authn_dbm authn_default authn_file authz_core authz_dbm authz_default authz_groupfile authz_host authz_owner auth_digest authn_dbd authz_user autoindex cache cache_disk cgi deflate dir disk_cache env slotmem_shm expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif socache_shmcb speling status reqtimeout lbmethod_byrequests unique_id unixd usertrack vhost_alias proxy proxy_fcgi proxy_http proxy_balancer lbmethod_bytraffic" APACHE2_MPMS="event" COLLECTD_PLUGINS="aggregation apache conntrack contextswitch cpu csv curl curl_json curl_xml dbi df disk dns entropy ethstat exec filecount fscache interface iptables irq load logfile match_regex match_timediff match_value md memory mysql netlink notify_email ntpd openvpn ping powerdns processes protocols rrdtool rrdcached swap syslog table tail tcpconns threshold unixsock uptime users varnish vmem" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby19" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON Just wanted to note that the bug is not fixed yet. *** Bug 320859 has been marked as a duplicate of this bug. *** (In reply to Maxim Kammerer from comment #18) > Just wanted to note that the bug is not fixed yet. Reproduced by a #gentoo user with 2GB RAM while emerging =sys-kernel/gentoo-sources-4.4.6 with similar error "OSError: [Errno 12] Cannot allocate memory"; so, this bug still seems to be lingering around please update portage to 2.2.28 and try again I had similar problem and upgrading solved it |