Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 75159 - portageq executed for every line of output, slow
Summary: portageq executed for every line of output, slow
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
Depends on:
Blocks: 835380
  Show dependency tree
Reported: 2004-12-21 03:15 UTC by Daniel Lawson
Modified: 2023-05-20 07:45 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

daemonised version of portageq (portageqd,6.49 KB, text/plain)
2004-12-21 03:17 UTC, Daniel Lawson
c client for portageqd (portageq.c,1.77 KB, text/plain)
2004-12-21 03:18 UTC, Daniel Lawson

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Lawson 2004-12-21 03:15:45 UTC
I noticed that a portageq was taking a large amount of cpu time during emerges, and that emerges seemed to be slower than I remembered. I had a quick look at portageq, and reimplemented it as a python server backend with a C client. This saved a lot on the overhead of initiating a new instance of python for every call.

During some debugging, I was stracing my new daemon, and I noticed that (seemingly) for every line of output during the configure stage of a typical 'emerge' process, there was a portageq request for the envvar NOCOLOR.

Reproducing this on a 'vanilla' system is hard to prove, as linux doesnt have fine enough process monitoring to tell you how often a process is being executed. I don't know enough about the internals of emerge to track down where this is happening, but a grep across /usr/lib/portage/bin for NOCOLOR shows it appearing in one place only inside emerge.

Reproducible: Didn't try
Steps to Reproduce:
1. download and start portageqd (attached)
2. download and compile portageq (attached) (make portageq)
3. replace /usr/lib/portage/bin/portageq with the version you just compiled
4. Strace the portageqd process (strace -ff -p `pgrep portageqd`)

Actual Results:  
First of all, compiles run a lot quicker as the server / C client framework is
much faster than the original portageq written in python.

Second, you can see the high number of requests for 'envvar NOCOLOR'

Expected Results:  
This envvar should be retrieved at most once per ebuild. I could probably work
this out, but it's 1am and I feel like sleeping. I figure someone else will know
exactly where to fix this, given that the major problem seems to be the repeated
requests for 'envvar NOCOLOR' and that it only seems to happen during the
configure stage of an emerge.

Portage 2.0.51-r8 (default-linux/x86/2004.3, gcc-3.4.3, glibc-,
2.6.9-gentoo-r4 i686)
System uname: 2.6.9-gentoo-r4 i686 AMD Athlon(tm) XP 1800+
Gentoo Base System version 1.6.8
Python:              dev-lang/python-2.3.4,dev-lang/python-2.2.3-r5 [2.3.4 (#1,
Sep 30 2004, 17:42:12)]
dev-lang/python:     2.3.4, 2.2.3-r5
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.5, 1.8.5-r2, 1.6.3, 1.7.9, 1.4_p6, 1.9.3
sys-devel/libtool:   1.5.10-r2
virtual/os-headers:  2.4.19-r1, 2.4.22
CFLAGS="-mtune=athlon-xp -march=athlon-xp -O2 -pipe"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown
/usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config
/usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/
/usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/
/usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-mtune=athlon-xp -march=athlon-xp -O2 -pipe"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox sfperms"
USE="X a52 aalib acl acpi alsa apm arts audiofile avi berkdb bitmap-fonts bonobo
cdr crypt cscope cups curl directfb divx4linux dv dvd dvdr dvdread encode esd
evo f77 faad fam fame fbcon fftw flac foomaticdb fortran gb gd gdbm ggi gif
gnome gphoto2 gpm gstreamer gtk gtk2 gtkhtml guile imagemagick imap imlib ipv6
javascript jpeg junit ldap libg++ libwww lzo mad mikmod mmx mng motif mozilla
mpeg mpi mysql nas ncurses nls nocd offensive oggvorbis opengl oss pam pdflib
perl png pnp postgres python qt quicktime readline samba scanner sdl slang speex
spell sse ssl svga tcltk tcpd tetex theora tiff transcode truetype usb v4l x86
xine xml xml2 xmms xosd xosdxvid xv zlib"
Comment 1 Daniel Lawson 2004-12-21 03:17:55 UTC
Created attachment 46526 [details]
daemonised version of portageq

Quick'n'dirty hack to turn portageq into a daemon. In conjunction with my c
replacement for portageq (thank you to beej for his sample code), this should
work flawlessly, aside from the obvious problem of keeping portaged running. 

Proof of concept more than anything, and posted here to provide a decent way of
testing this bug.

See next attachment for portageq.c
Comment 2 Daniel Lawson 2004-12-21 03:18:26 UTC
Created attachment 46527 [details]
c client for portageqd
Comment 3 Daniel Lawson 2004-12-21 03:20:55 UTC
Step 5 to reproduce should be 'emerge a package and observe the strace output'
Comment 4 Brian Harring (RETIRED) gentoo-dev 2004-12-21 16:01:08 UTC
offhand, this is quite interesting.  Potentially overkill, but quite interesting none the less :)

That said, sorry but I already nuked the cost of portageq calls already in cvs, via tieing it back into the 'originating' portage environment; basically, a set of pipes are left open so that can query info from the parent portage process, accomplishing what you're doing (likely slightly quicker since there isn't any start up overhead).  Your implementation looks a bit cleaner (certainly quite a bit more high level then what I added), but the pipes approach allows us to extend/add new hijacks for speeding things up w/out too much additional work.

It's in portage's cvs, either available via our viewcvs, or in .

Thank you for the contribution though- portageq in stable portage *is* slow, and it does rear it's head enough to be a pain in the ass.  Most people either haven't noticed it, or never did anything about it :)