Bug 162622 – equery memory leak

• Product: Portage Development
• Status: RESOLVED FIXED

[Description hvjunk 2007-01-18 08:04:18 UTC]

doing just "equery list > package.list"
sees a process that grows and grows.
doing a single query (like "equery list gentoolkit > /tmp/t")
uses less than 15Mbyte, but the "equery list > package.list"
keeps growing till over 300MByte (~800 opackages installed)

Being just piping out, I don't see the need for arrays to grow,
unless there are a new set of variables being created and not
cleared for each entry :(

Reproducible: Always

Steps to Reproduce:
1. fire up top in one screen/window
2. in another screen/window: equery list > /tmp/package.list
3. keep eyes on equery's size as the time goes by...

Actual Results:  
equery grows and grows in size as it outputs the packages installed
and have at times killed other processes as I've ran out of memory

Expected Results:  
equery's size should be a constant, but still much, much less than 300Mbyte
for more than 800 packages installed.

[Comment 1 hvjunk 2007-01-18 13:19:00 UTC]

Okay, this appears to be a general equery memory management isuse,
as "equery depends" also run into >>500Mbyte very quickly and then
bogs down machines when it starts to swap.

app-portage/gentoolkit-0.2.2p

Portage 2.1.1-r2 (default-linux/amd64/2006.1/desktop, gcc-4.1.1, glibc-2.4-r4, 2
.6.19-gentoo-r3 x86_64)
=================================================================
System uname: 2.6.19-gentoo-r3 x86_64 Mobile AMD Athlon 64 Processor 3000+
Gentoo Base System version 1.12.6
Last Sync: Wed, 17 Jan 2007 10:00:02 +0000
distcc 2.18.3 x86_64-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disab
led]
ccache version 2.3 [disabled]
app-admin/eselect-compiler: [Not Present]
dev-java/java-config: 1.3.7, 2.0.30
dev-lang/python:     2.3.5-r2, 2.4.3-r4
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache:     2.3
dev-util/confcache:  [Not Present]
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.61
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.16.1-r3
sys-devel/gcc-config: 1.3.14
sys-devel/libtool:   1.5.22
virtual/os-headers:  2.6.11-r2
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=k8 -pipe -O2"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shu
tdown /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/shar
e/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms
/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-march=k8 -pipe -O2"
DISTDIR="/home/ftp/gentoo/distfiles/"
FEATURES="autoconfig collision-protect confcache digest distlocks metadata-trans
fer parallel-fetch sfperms strict"
GENTOO_MIRRORS=" ftp://ftp.is.co.za/linux/distributions/gentoo ftp://gentoo.chem
.wisc.edu/gentoo/ http://gentoo.mirrors.pair.com/"
LANG="af en en_ZA af_ZA"
LINGUAS="af en en_ZA af_ZA"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress 
--force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/d
istfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://10.11.12.2/gentoo-portage/"
USE="amd64 7zip X a52 aac acpi adns aiglx aim aio alsa alsa_cards_ali5451 alsa_c
ards_als4000 alsa_cards_atiixp alsa_cards_atiixp-modem alsa_cards_bt87x alsa_car
ds_ca0106 alsa_cards_cmipci alsa_cards_emu10k1x alsa_cards_ens1370 alsa_cards_en
s1371 alsa_cards_es1938 alsa_cards_es1968 alsa_cards_fm801 alsa_cards_hda-intel 
alsa_cards_intel8x0 alsa_cards_intel8x0m alsa_cards_maestro3 alsa_cards_trident 
alsa_cards_usb-audio alsa_cards_via82xx alsa_cards_via82xx-modem alsa_cards_ymfp
ci alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_p
lugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoo
p alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm
_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_
ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter al
sa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugi
ns_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa
_pcm_plugins_shm alsa_pcm_plugins_softvol amr animgif aoss apm applet arts artsw
rappersuid auctex background berkdb bitmap-fonts bluetooth browserplugin bundled
-libs buttons cairo cap cardbus cdda cddb cdio cdparanoia cdr cgi cli cracklib c
rypt css cups curl curlwrappers dar64 dbus dbx dga dio discard-path diskcache di
vx4linux dlloader dmx dnd doc dri dv dvd dvdr dvdread eds elibc_glibc emboss emu
l-linux-x86 encode esd exif fam firefox flac font-server fortran fping freetype 
ftp gcj gd gdbm gif gnome gogo gpm gstreamer gtk gtk2 hal high-ints html httpd i
conv ieee1394 imagemagick imlib input_devices_evdev input_devices_keyboard input
_devices_mouse ipv6 isdnlog java jpeg kde kernel_linux lame lcd_devices_bayrad l
cd_devices_cfontz lcd_devices_cfontz633 lcd_devices_glk lcd_devices_hd44780 lcd_
devices_lb216 lcd_devices_lcdm001 lcd_devices_mtxorb lcd_devices_ncurses lcd_dev
ices_text lcms ldap libg++ linguas_af linguas_af_ZA linguas_en linguas_en_ZA lm_
sensors mad mikmod mmap mng motif mozbranding mozdevelop mozsvg mp3 mpeg mplayer
 mudflap ncurses nls nptl nptlonly ntfs ofx ogg oggvorbis on-the-fly-crypt openg
l openssl oss pam pam_chroot pam_console pcre pdf perl php png postgres ppds ppp
d print python qt3 qt4 quicktime rar rdesktop readline reflection reiserfs restr
ict-javascript rle rrdtool rtc sdl sensord session shorten skey sms sockets sox 
spell spl sse-filters ssl staticsocket stroke subp subtitles svg sysfs tcpd tete
x theora threads tiff toolbar transcode transparent-proxy truetype truetype-font
s type1 type1-fonts udev unicode userland_GNU vcd vcdimager video_cards_ati vide
o_cards_fbdev video_cards_fglrx video_cards_radeon video_cards_vesa vim-pager vo
rbis vorbis-psy wma wma123 wmf wmp x11vnc xemacs xface xforms xinerama xml xmlre
ader xorg xosd xprint xscreensaver xv xvid xvmc zip zlib"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, PORTAGE_RSY
NC_EXTRA_OPTS

[Comment 2 Michael Rzechonek 2007-01-18 14:07:50 UTC]

(In reply to comment #1)
> Okay, this appears to be a general equery memory management isuse,
> as "equery depends" also run into >>500Mbyte very quickly and then
> bogs down machines when it starts to swap.

Works for me...

equery list > /dev/null  3,27s user 0,20s system 91% cpu 3,774 total
and never gets above 20M of RAM

gentoolkit-0.2.3_pre2

$ equery --version
equery(0.1.4) - Gentoo Package Query Tool

[Comment 3 Paul Varner (RETIRED) 2007-01-19 02:45:36 UTC]

I don't know what to tell you, I do not see the same behavior on any of my boxes. The most memory that I have seen used is about 20MB. Are you willing to install gentoolkit-0.2.3_pre3 and let me know if you see the same sort of behavior?

[Comment 4 hvjunk 2007-01-19 09:57:25 UTC]

Okay, 0.2.3_pre3 don't appear to have this problem...

SOmething else perhaps to "investigate" is that I'm
also in the process of recompiling everything for GCC 4.1
In this process it might've happened that python got upgraded/recompiled
but the gentoolkit stuff didn't got python-compiled.

I've just reinstalled the gentoolkit-0.2.2 and it appears to have "fixed" this
issue.