when I try emerge -a --depclean portage completly saturises one core and starts eating memory. Lots of memory. I have 4gb ram and 16gb swap. After a few minutes all ram is used up and I am 5gb in swap - that was the point where I killed emerge. It starts slow - the first minute below 10% but then it goes up very fast. It is ok until it displays: >>> These are the packages that would be unmerged: and the fun starts. Nothing seems to happen, one core at 100%, memory consumption goes up by almost 1%/minute Reproducible: Always Steps to Reproduce: 1. have system with lots of stuff installed 2. emerge -a --depclean 3. watch memory consumption go up emerge --info WARNING: repository at /usr/local/portage is missing a repo_name entry Portage 2.2_rc8 (default/linux/amd64/2008.0/desktop, gcc-4.3.1, glibc-2.8_p20080602-r0, 2.6.26.3r4 x86_64) ================================================================= System uname: Linux-2.6.26.3r4-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_6000+-with-glibc2.2.5 Timestamp of tree: Thu, 04 Sep 2008 13:36:01 +0000 ccache version 2.4 [enabled] app-shells/bash: 3.2_p39 dev-java/java-config: 1.3.7, 2.1.6-r1 dev-lang/python: 2.5.2-r7 dev-util/ccache: 2.4-r7 sys-apps/baselayout: 2.0.0 sys-apps/openrc: 0.2.5 sys-apps/sandbox: 1.2.18.1-r3 sys-devel/autoconf: 2.13, 2.62-r1 sys-devel/automake: 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 2.2.4 virtual/os-headers: 2.6.26 ACCEPT_KEYWORDS="amd64 ~amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=k8 -O2 -msse3 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/kde/4.1/env /usr/kde/4.1/share/config /usr/kde/4.1/shutdown /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d" CXXFLAGS="-march=k8 -O2 -msse3 -pipe" DISTDIR="/var/portage/distfiles" FEATURES="autoconfig candy ccache distlocks parallel-fetch preserve-libs sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="ftp://ftp.tu-clausthal.de/pub/linux/gentoo/" LANG="de_DE@euro" LC_ALL="de_DE@euro" LDFLAGS="-Wl,-O1" LINGUAS="de" MAKEOPTS="-j3" PKGDIR="/var/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/var/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow 3dnowext 7zip S3TC X Xaw3d a52 aac acpi addbookmarks aiglx aim aio akode alias alsa altenburgcards amarokamd64 amr amrnb amrwb aotuv aqua_theme archive asf athena audiofile autoreplace bash-completion berkdb binary-drivers bluetooth boost branding bzip2 cairo captury cdda cddb cdinstall cdparanoia cdr cdrom cdsound cegui clamav cli connectionstatus contactnotes cpudetection cracklib crypt css cups curl dbus depth32 dhcp dio divx dnd double-precision dpms dri dts dv dvb dvbplayer dvbsetup dvd dvdr dvdread eds emboss emerald emovix emul-linux-x86 encode evo exif exiv2expat exscalibar extra-cardsets fame fasttrack ffmpeg fftw file filter_default filter_legacy filter_light firefox fits flac fontconfig foomaticdb fortran ftp gadu gammu gd gdbm gif gimp gimpprint glitz glut glx gnokii gnutella gpgme gpm hal hddtemp highlight history iconv icq icu id3 id3tag ieee1394 imagemagick inotify ipv6 irc irmc isdnlog jabber java javascript jce joystick jpeg jpeg2k kcal kde kdecards kdeenablefinal kdehiddenvisibility kdepim kdm kexi kipi lame lapack lcms lesstif libnotify libsamplerate libwww lirc lm_sensors logitech-mouse lzo lzw mad magic mailbox maildir maps matroska mbox mdnsresponder-compat metric midi mikmod mime mixer mjpeg mmap mmx mmxext mng mp3 mp4 mpeg mpeg2 mplayer msn mtp mudflap mule multilib musepack music nano-syntax ncurses neXt net network newspr ngui nls no-old-linux nocd nosendmail nova nowin nowlistening nptl nptlonly nsplugin obex offensive ogg ogg123 ogm openexr opengl openmp oscar pam paste64 pch pcre pda pdf perl physfs pixmaps plasma player plugins png pnm posix ppds pppd python qt3 qt3support qt4 quicktime rar readline recode reflection reiser4 reiserfs remix resolvconf rogue rtc samba sametime sasl scanner scenarios schedule scsi sdl sdl-sound sdlaudio sendfile server session slp smp sms sndfile sockets sounds spellspl sqlite sqlite3 srt sse sse-filters sse2 sse3 ssh ssl ssse3 startup-notification statistics stats stencil-buffer stroke subp subtitles svg sysfs sysvipc szip tcpd texteffect textures tga theora threads threadsafe tiff timidity tk tomsfastmath toolbar tools transcode translator truetype tv tv_check tv_combiner tv_pick_cgi type1 unicode unzip usb userlocales utempter v4l v4l2 vcd videos visualization vorbis webdav webpresence wifi wireshark wmf wmp wxwindows x264 xanim xcb xcomposite xemacs xext xface xine xml xorg xosd xpm xrandr xrender xscreensaver xskatcards xv xvid xvmc yahoo yiff yv12 zip zlib zsh-completion zvbi" ALSA_CARDS="emu10k1" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_fileauthz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de" LIRC_DEVICES="devinput" USERLAND="GNU" VIDEO_CARDS="vesa vga v4l fglrx radeon" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
I've been monitoring memory consumption for the past month here and finally realized, it's Portage who's caching large amounts of memory here. Although, ATop doesn't slate Portage (Python) as using the memory. But after syncing and/or compiling (using Portage/Python), large amounts of memory are apparently consumed by comparing "free -m" output. (I've finally concluded this after dropping to tty and DWM!) My best guess, it's Python and/or un-optimized Python coding. Might be a function or two in Portage not optimized to make proper use of CPU/Memory resources??
The --depclean-lib-check code (enabled by default) used to consume lots of memory when looking up owwners when there were lots of libraries to uninstall. It's been fixed to avoid consuming so much memory, but it can be very time consuming. You can use --depclean-lib-check=n to disable it.
So can we close this? (In reply to comment #2) > The --depclean-lib-check code (enabled by default) used to consume lots of > memory when looking up owwners when there were lots of libraries to uninstall. > It's been fixed to avoid consuming so much memory, but it can be very time > consuming. You can use --depclean-lib-check=n to disable it.
(In reply to comment #3) > So can we close this? Yes, this we can consider this fixed now that. Note that the --depclean-lib-check code is disabled automatically when preserve-libs is enabled, since the fix for bug 286714: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=c64d4abee145d083c70273be8fd23bd56dffe7ec Also, the --depclean-lib-check (and preserve-libs) code now uses a more efficient method to look up the owners of libraries and consumers: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f393413c3f823ef4a60acfcc41c3920933510fc1
FYI: emerge: error: no such option: --depclean-lib-check
(In reply to comment #5) > FYI: emerge: error: no such option: --depclean-lib-check Currently, it's only available in portage-2.2.x.
