zoom ~ # emerge --info Portage 2.1_pre5-r1 (default-linux/amd64/2005.1, gcc-3.4.4, glibc-2.3.5-r2, 2.6.15-gentoo-r5 x86_64) ================================================================= System uname: 2.6.15-gentoo-r5 x86_64 AMD Athlon(tm) 64 Processor 3200+ Gentoo Base System version 1.6.14 dev-lang/python: 2.3.5-r2, 2.4.2 sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="amd64" AUTOCLEAN="yes" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=athlon64 -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-march=athlon64 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig buildpkg ccache cvs digest distlocks fixpackages multilib-strict sandbox sfperms strict" GENTOO_MIRRORS="ftp://ftp.ussg.iu.edu/pub/linux/gentoo" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="amd64 acl alsa avi bitmap-fonts crypt eds emboss encode fastcgi foomaticdb fortran ftp gif gpm gstreamer gtk2 imlib innodb jpeg libwww lzw lzw-tiff maildir mp3 mpeg mysql ncurses nptl nptlonly opengl pam pdflib perl png python quicktime readline ruby sdl spell ssl tcpd tetex tidy tiff truetype-fonts type1-fonts usb userlocales xml xv zlib elibc_glibc kernel_linux userland_GNU" Unset: ASFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, LANG, LC_ALL, LDFLAGS, LINGUAS Before I started an "emerge --sync && emerge -au world" , memory usage was at 155MB. After starting the sync process, the memory usage climbed to 512MB. After the sync was complete, memory usage remains at 514MB, even though 'emerge' and 'rsync' no longer show up in the process table.
Is it reproducable? Technically I want to say "Portage doesn't have memory leaks" because it would be an error in Python's garbage collection. It is my understanding that emerge --sync would end, killing one python interpreter, linux should reclaim that memory to the heap and it would start over for the emerge -au world. You also mention that memory usage is 514MB even after emerge and rsync exited...This is after emerge -au world finished? What process was the RAM given to then? ps -auxfw should give a comprehensive list. Was perhaps "python" still running?
(In reply to comment #1) > Is it reproducable? Technically I want to say "Portage doesn't have memory > leaks" because it would be an error in Python's garbage collection. Certain obscene uses will trip up gc actually, leading to python holding onto things for a bit longer then you'd expect. Either way, that statement isn't accurate- gc is effectively smart ptrs (ref counting) with periodic cycle detection. So... if we have refs to objects left, garbage collection can't do anything about it. > It is my > understanding that emerge --sync would end, killing one python interpreter, > linux should reclaim that memory to the heap and it would start over for the > emerge -au world. He stated first emerge instance used 155, second used half a gig (eg, his statement is accurate if it's leaking). The *rough* emerge mem usage for 32bit arches is ~20 mb for emerge --metadata runs; for 64bit, would expect it to be in the range of 25-30 (sizeof(ptr)), so if you're seeing 155 something is weird. Custom backend? A guess offhand is that this is either 1) cache leakage (the 5-7x increase I don't see in --metadata runs though) 2) config instance leakage. This one strikes me as probable, since repoman has repeatedly exposed config leakage/bloat (occasion versions of repoman in the past would clear 2 gb for a full tree run).
Reading your comments on the bug, I think I need to clarify something. The figures 155MB before syncing, and 514MB after syncing were totals. Totals for ALL processes. I used top to tell. I wasn't looking at memory usage for single process. When I ran the sync, I watched the total memory usage climb rapidly, but never return to original levels after 'emerge sync && emerge -au world' was finished.
And... a full 180 from me. :) If memory isn't being used for anything (eg, not used by an app or by the kernel data structures) linux uses the free memory to buffer data from disk- disk access is slow, the memory isn't (currently) used for anything so it uses it for buffers. It'll release the memory to any app that requests memory, but the unused memory it uses for io buffering; after a sync, and after a emerge -Du world portage has accessed a _lot_ of files, thus if you've got the memory, a lot will be buffered.
crap... wrong resolution.
there we go...
