earlier versions of 2.1_pre were the expected speed. portage-2.1_pre10 repoman runs almost twice as slow. >220 minutes where it used to run in about 120 minutes. Portage 2.1_pre10 (default-linux/x86/2005.0, gcc-3.4.5, glibc-2.3.6-r3, 2.6.14.2 i686) ================================================================= System uname: 2.6.14.2 i686 AMD Athlon(tm) Processor Gentoo Base System version 1.6.14 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] dev-lang/python: 2.3.5-r2, 2.4.2-r1 dev-util/ccache: [Not Present] dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.12 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-mtune=athlon -O3 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /var/bind /var/qmail/control" CONFIG_PROTECT_MASK="/etc/X11/app-defaults /etc/X11/rstart /etc/X11/serverconfig /etc/X11/starthere /etc/X11/xdm /etc/bash_completion /etc/eselect/compiler /etc/gconf /etc/init.d /etc/pango /etc/sound/events /etc/ssmtp /etc/terminfo /usr/lib/X11/xkb /etc/env.d" CXXFLAGS="-mtune=athlon -O3 -pipe -Wno-deprecated" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache distlocks metadata-transfer sandbox sfperms strict" GENTOO_MIRRORS="http://www.ibiblio.org/pub/Linux/distributions/gentoo http://gentoo.ccccom.com" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/tmp/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 3dnow 3dnowext X aac alsa apm arts artworkextra avi bash-completion berkdb bitmap-fonts browserplugin bzip2 cli crypt cscope cups dri dvd eds emboss encode flac foomaticdb gdbm gif gimpprint gnome gpm gstreamer gtk gtk2 isdnlog java jpeg libg++ libwww live logrotate lzo mad matroska mikmod mmx mmxext mozilla moznocompose moznoirc moznomail mp3 mpeg musepack ncurses nomotif noreiserfs nsplugin ogg oggvorbis opengl oss pam pcre pdflib perl png ppds pppd python quicktime readline real reflection sdl session spell spl ssl svg tcpd theora tiff truetype truetype-fonts type1-fonts userlocales vorbis win32codecs xml2 xmms xorg xv zlib elibc_glibc kernel_linux userland_GNU" Unset: ASFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, MAKEOPTS
So, just to document my findings here, I ran category level scans for both sys-kernel and kde-misc and didn't find much difference between pre9 and pre10. The pre10 runs were both slightly longer, but not enough to make anything stand out above the noise level in the profiler runs. It seems that this problem is only really bad for certain types of ebuilds or categories (considering that Halcyon also experienced the performance problem when doing a full repo scan). So, it seems like it may be necessary to profile scans of the full repo. BTW, to profile repoman, here's how I did it: time python -m profile `which repoman` scan
http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler output
(In reply to comment #2) > http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler > output Thanks. Ideally though, the same exact scan needs to be performed with both pre9 and pre10 so that the stats can be directly compared against eachother (pre9 is the control sample and pre10 is the experimental sample). If I do `python -c 'import pstats; pstats.Stats("repoman-stats.pstats").print_stats()'` it shows me that this run was only about 1 hour which suggests that it's not a full repo scan. Is this from a specific category or something?
No idea how the portage profiling thing works but it actually ran for: real 2144m0.165s user 1944m33.336s sys 129m14.177s and sucked up 99%+ of the cpu the whole time. This was a repoman full for the whole tree.
(In reply to comment #3) > (In reply to comment #2) > > http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats is the python profiler > > output > > Thanks. Ideally though, the same exact scan needs to be performed with both > pre9 and pre10 so that the stats can be directly compared against eachother > (pre9 is the control sample and pre10 is the experimental sample). It was a full run- meet the python profiler, occasionally known to overflow the time count and go negative ;) Reverse the stats, and you'll see it.
Here is the profile run for pre9-r5: http://dev.gentoo.org/~mr_bones_/repoman-stats.pstats-2.1_pre9-r5
After I ran an `emerge --regen`, everything is fine for me.
I added two profile runs of the dev-util category for both pre9 and pre10 as requested by ferringb to http://dev.gentoo.org/~mr_bones_/
(In reply to comment #7) > After I ran an `emerge --regen`, everything is fine for me. Hrm. This bug looks like another manifestation of bug 124041. Mr. Bones, are you able to reproduce the problem if you make sure to run `emerge --regen` before the repoman scan?
I always run repoman for the whole tree like this: clear ; emerge --sync ; emerge --regen ; time ~/repoman.new full > ~/repoman.out
Last night I dedicated my desktop system to repoman benchmarks and got practically identical results for 2.1_pre9-r5 vs. 2.1_pre10-r3 when running `repoman full` on the entire repo: 2.1_pre9-r5 2.1_pre10-r3 real 132m19.175s 133m35.277s user 91m48.656s 92m45.168s sys 16m15.321s 15m57.328s Given these numbers (and Halcy0n's findings), Mr. Bones, it seems that something on your system is skewing your benchmarks. In order to ensure valid benchmark results in my test, I took several precautions: 1) ran `emerge --regen` to ensure that the cache was fully up to date 2) removed all load from the system (paused the cron daemon and didn't run anything that would put IO or cpu load on the system except repoman) 3) installed the version of portage that would be tested 4) ran `time repoman full > ~/repoman.out` 5) repeated steps 3 and 4 with a different version of portage, and the same exact portage tree / cache (no sync in between).
Then go back to versions before pre9. As I *just* mentioned in the previous comment, I always --regen before running.
(In reply to comment #12) > Then go back to versions before pre9. The performance difference is likely due to additional QA checks that have been added to repoman in the 2.1_pre series. We don't want to sacrifice QA for performance, so the only option would be to optimize the QA checks for better performance. I'm satisfied with the current performance level, but people are welcome to contribute repoman optimization patches. Actually, here's one that you might test (I haven't tried it myself): http://gentooexperimental.org/~ferringb/portage-on-steroids-again.patch
