I noticed that a portageq was taking a large amount of cpu time during emerges, and that emerges seemed to be slower than I remembered. I had a quick look at portageq, and reimplemented it as a python server backend with a C client. This saved a lot on the overhead of initiating a new instance of python for every call. During some debugging, I was stracing my new daemon, and I noticed that (seemingly) for every line of output during the configure stage of a typical 'emerge' process, there was a portageq request for the envvar NOCOLOR. Reproducing this on a 'vanilla' system is hard to prove, as linux doesnt have fine enough process monitoring to tell you how often a process is being executed. I don't know enough about the internals of emerge to track down where this is happening, but a grep across /usr/lib/portage/bin for NOCOLOR shows it appearing in one place only inside emerge. Reproducible: Didn't try Steps to Reproduce: 1. download and start portageqd (attached) 2. download and compile portageq (attached) (make portageq) 3. replace /usr/lib/portage/bin/portageq with the version you just compiled 4. Strace the portageqd process (strace -ff -p `pgrep portageqd`) Actual Results: First of all, compiles run a lot quicker as the server / C client framework is much faster than the original portageq written in python. Second, you can see the high number of requests for 'envvar NOCOLOR' Expected Results: This envvar should be retrieved at most once per ebuild. I could probably work this out, but it's 1am and I feel like sleeping. I figure someone else will know exactly where to fix this, given that the major problem seems to be the repeated requests for 'envvar NOCOLOR' and that it only seems to happen during the configure stage of an emerge. Portage 2.0.51-r8 (default-linux/x86/2004.3, gcc-3.4.3, glibc-2.3.4.20041102-r0, 2.6.9-gentoo-r4 i686) ================================================================= System uname: 2.6.9-gentoo-r4 i686 AMD Athlon(tm) XP 1800+ Gentoo Base System version 1.6.8 Python: dev-lang/python-2.3.4,dev-lang/python-2.2.3-r5 [2.3.4 (#1, Sep 30 2004, 17:42:12)] dev-lang/python: 2.3.4, 2.2.3-r5 sys-devel/autoconf: 2.13, 2.59-r6 sys-devel/automake: 1.5, 1.8.5-r2, 1.6.3, 1.7.9, 1.4_p6, 1.9.3 sys-devel/binutils: 2.15.92.0.2-r2 sys-devel/libtool: 1.5.10-r2 virtual/os-headers: 2.4.19-r1, 2.4.22 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CFLAGS="-mtune=athlon-xp -march=athlon-xp -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-mtune=athlon-xp -march=athlon-xp -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/Linux/distributions/gentoo" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.au.gentoo.org/gentoo-portage" USE="X a52 aalib acl acpi alsa apm arts audiofile avi berkdb bitmap-fonts bonobo cdr crypt cscope cups curl directfb divx4linux dv dvd dvdr dvdread encode esd evo f77 faad fam fame fbcon fftw flac foomaticdb fortran gb gd gdbm ggi gif gnome gphoto2 gpm gstreamer gtk gtk2 gtkhtml guile imagemagick imap imlib ipv6 javascript jpeg junit ldap libg++ libwww lzo mad mikmod mmx mng motif mozilla mpeg mpi mysql nas ncurses nls nocd offensive oggvorbis opengl oss pam pdflib perl png pnp postgres python qt quicktime readline samba scanner sdl slang speex spell sse ssl svga tcltk tcpd tetex theora tiff transcode truetype usb v4l x86 xine xml xml2 xmms xosd xosdxvid xv zlib"
Created attachment 46526 [details] daemonised version of portageq Quick'n'dirty hack to turn portageq into a daemon. In conjunction with my c replacement for portageq (thank you to beej for his sample code), this should work flawlessly, aside from the obvious problem of keeping portaged running. Proof of concept more than anything, and posted here to provide a decent way of testing this bug. See next attachment for portageq.c
Created attachment 46527 [details] c client for portageqd
Step 5 to reproduce should be 'emerge a package and observe the strace output'
offhand, this is quite interesting. Potentially overkill, but quite interesting none the less :) That said, sorry but I already nuked the cost of portageq calls already in cvs, via tieing it back into the 'originating' portage environment; basically, a set of pipes are left open so that ebuild.sh can query info from the parent portage process, accomplishing what you're doing (likely slightly quicker since there isn't any start up overhead). Your implementation looks a bit cleaner (certainly quite a bit more high level then what I added), but the pipes approach allows us to extend/add new hijacks for speeding things up w/out too much additional work. It's in portage's cvs, either available via our viewcvs, or in http://dev.gentoo.org/~ferringb/portage-cvs.tar.bz2 . Thank you for the contribution though- portageq in stable portage *is* slow, and it does rear it's head enough to be a pain in the ass. Most people either haven't noticed it, or never did anything about it :)
