Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 56170 - excessive ressource usage when running "repoman scan"
Summary: excessive ressource usage when running "repoman scan"
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Repoman (show other bugs)
Hardware: x86 Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks:
 
Reported: 2004-07-05 13:46 UTC by Patrick Lauer
Modified: 2004-08-11 16:45 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Lauer gentoo-dev 2004-07-05 13:46:47 UTC 
"repoman scan" on the whole tree uses > 1.5GB memory and causes a swapstorm on machines with "normal" amounts of RAM

Reproducible: Always
Steps to Reproduce:
1. cd gentoo-cvs
2. repoman scan
3. wait ~5hrs while repoman thrashes around

Actual Results:  
Memory Usage >1.5Gig, system swapping, repoman uses almost no CPU since it's
swapping :-(

Expected Results:  
repoman should have used a reasonable amount of memory (<<1GB)

Portage 2.0.51_pre12 (default-x86-1.4, gcc-3.3.3, glibc-2.3.3.20040420-r0, 2.6.7)
=================================================================
System uname: 2.6.7 i686 AMD Athlon(tm) XP 2600+
Gentoo Base System version 1.5.1
distcc 2.14 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
ccache version 2.3 [enabled]
Autoconf: sys-devel/autoconf-2.59-r4
Automake: sys-devel/automake-1.8.5-r1
Binutils: sys-devel/binutils-2.14.90.0.8-r1
ACCEPT_KEYWORDS="x86 ~x86"
AUTOCLEAN="yes"
CFLAGS="-O2 -march=athlon-xp -pipe"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3.1/share/config /usr/kde/3.2/share/config /usr/kde/3/share/config
/usr/lib/mozilla/defaults/pref /usr/share/config
/usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/
/usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/
/usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -march=athlon-xp -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs buildpkg ccache sandbox"
GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo
http://gentoo.inode.at/ ftp://gentoo.inode.at/source/"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow X aalib alsa apm arts avi berkdb bonobo cdr crypt cups dga dvd encode
esd faad ffmpeg foomaticdb freetype gdbm gif gnome gpm gtk2 gtkhtml imlib java
jpeg kde libg++ libwww mikmod motif mozilla mpeg mysql nas ncurses nls oggvorbis
opengl oss pam pdflib perl png python qt quicktime readline sdl slang spell ssl
tcltk tcpd tetex truetype x86 xml2 xmms xv zlib"
Comment 1 Marius Mauch (RETIRED) gentoo-dev 2004-07-06 13:53:04 UTC 
seems that the problem is on lines 731-732 in repoman (rev. 1.66):
    portage.portdb=portage.portdbapi(portdir, dep_settings)
    arch_caches[arch]=[dep_settings, portage.portdb]
With those lines I get a memory usage of about 30 MB (and increasing at 0.5-1 MB per package) when scanning app-accessiblity, without them it's less than 10 MB (and stays nearly constant).
Comment 2 Marius Mauch (RETIRED) gentoo-dev 2004-07-06 14:02:32 UTC 
And actually I don't see where portdb is being used, the other code in this loop only refers to portage.db["/"]["porttree"], so if noone has a reason for it I'm going to remove it.
Comment 3 Brian Harring (RETIRED) gentoo-dev 2004-08-11 16:45:16 UTC 
The execessive memory issues of repoman were fixed by genone/nakano around pre14 iirc; either way, they're fixed in pre16/pre17 of .51 .