Looks like there is regression in portage-2.2 - it's calculate dependencies in 3.5 times slower than portage-2.1. I've Intel Core i7-2600K overclocked at 4.6GHz and 8GB RAM. It's a shame `emerge -uDNpv world` spend 3 times more time (just to find out there is no available updates) than my kernel compiles! I've run `emerge -uDNpv world` several times, so everything from /usr/portage is cached in memory, and I've monitored conky to make sure there was no disk I/O while executing these benchmarks (also this is proved by `time` output with "real" same as "user"). While running, emerge use 100% CPU (one Core) all the time until it finish. Btw, with python-3.2 portage (both 2.2 and 2.1) works about 15% faster than with python-2.7. # eselect python list Available Python interpreters: [1] python2.7 [2] python3.2 * # qlist -I -v sys-apps/portage sys-apps/portage-2.2.1 # time emerge -uDNpv world These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB real 2m33.552s user 2m31.307s sys 0m1.923s # emerge -1av '<portage-2.2' These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild UD ] sys-apps/portage-2.1.12.2 [2.2.1] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru -pl%" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB Total: 1 package (1 downgrade), Size of downloads: 0 kB … # qlist -I -v sys-apps/portage sys-apps/portage-2.1.12.2 # time emerge -uDNpv world These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] sys-apps/portage-2.2.1 [2.1.12.2] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru (-pl%)" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB Total: 1 package (1 upgrade), Size of downloads: 0 kB real 0m42.091s user 0m41.371s sys 0m0.625s
For @system it's 2 times slower. # qlist -I -v sys-apps/portage sys-apps/portage-2.1.12.2 # time emerge -uDNpv system These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] sys-apps/portage-2.2.1 [2.1.12.2] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru (-pl%)" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB Total: 1 package (1 upgrade), Size of downloads: 0 kB real 0m31.177s user 0m30.820s sys 0m0.305s # qlist -I -v sys-apps/portage sys-apps/portage-2.2.1 # time emerge -uDNpv system These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB real 0m58.117s user 0m57.485s sys 0m0.541s Portage 2.2.1 (hardened/linux/amd64, gcc-4.6.3, glibc-2.15-r3, 3.10.1-hardened-r1 x86_64) ================================================================= System uname: Linux-3.10.1-hardened-r1-x86_64-Intel-R-_Core-TM-_i7-2600K_CPU_@_3.40GHz-with-gentoo-2.2 KiB Mem: 8163184 total, 1318412 free KiB Swap: 4200960 total, 4194960 free Timestamp of tree: Fri, 13 Sep 2013 15:15:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-java/java-config: 2.1.12-r1 dev-lang/python: 2.7.5-r2, 3.2.5-r2 dev-util/cmake: 2.8.10.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6, 1.12.6, 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo perl-experimental-snapshots gamerlay powerman local ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /opt/upsmon-usb/EXT/DownOS /opt/upsmon-usb/EXT/JSystem /service /usr/inferno/keydb /usr/inferno/lib /usr/inferno/services /usr/share/config /usr/share/gnupg/qualified.txt /var/log /var/qmail/alias /var/qmail/control" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage-distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask-write=y --backtrack=15" FCFLAGS="-march=native -O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr" FFLAGS="-march=native -O2 -pipe" GENTOO_MIRRORS="http://gentoo.kiev.ua/ftp/ http://gentoo.iteam.net.ua/ http://tux.rainside.sk/gentoo/ http://trumpetti.atm.tut.fi/gentoo/ http://gentoo.inode.at/" LANG="ru_RU.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j8" PKGDIR="/usr/portage-packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/perl-experimental-snapshots /var/lib/layman/gamerlay /var/lib/layman/powerman /usr/local/portage" SYNC="rsync://rsync3.ua.gentoo.org/gentoo-portage" USE="X a52 aac acl alac alsa amd64 avx bash-completion berkdb bzip2 caps cdda cddb cli cracklib crypt cxx dbus dri dts dvb dvd flac fontconfig gdbm gif gnutls gpg hardened iconv icu id3tag idn ipv6 jpeg jpeg2k justify libnotify mac mad matroska mbox mmx mng modules mp3 mpeg mudflap multilib musepack mysql ncurses network-cron nls nptl nsplugin ogg opengl openmp pam pax_kernel pcre perl png qt3support readline session spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 svg tcpd theora tiff truetype unicode urandom vdpau vim-syntax vorbis wavpack x264 xattr xosd xv xvid xvmc zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="log_config vhost_alias autoindex alias rewrite dir deflate filter mime negotiation auth_basic authn_file authz_host authz_user authz_groupfile cgi actions headers env setenvif" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en ru" NGINX_MODULES_HTTP="access auth_basic autoindex browser charset empty_gif fastcgi geo gzip limit_conn limit_req map memcached proxy referer rewrite scgi split_clients ssi upstream_ip_hash userid uwsgi fancyindex" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" QEMU_SOFTMMU_TARGETS="x86_64 i386" QEMU_USER_TARGETS="x86_64 i386" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia nv nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, USE_PYTHON
Hmm, I'll try to see if I can generate a new profiled image so we can compare it against the one I made a while ago so we can see in which path the regression is.
I've just updated portage on several servers, and noticed there is no slowdown - 2.2 works at about same speed as 2.1 (and it's much faster than on my home workstation, while server itself has much weaker CPU - like Core2 Duo E4500 @ 2.20GHz). I didn't updated kernel to 3.10.1 yet and didn't switched from PT_PAX to XATTR_PAX, but I think that's only differences except… servers are 32-bit while workstation is 64-bit - and I think this may be reason for that slowdown! # qlist -I -v sys-apps/portage sys-apps/portage-2.2.1 # time emerge -uDNpv system These are the packages that would be merged, in order: Calculating dependencies... done! Total: 0 packages, Size of downloads: 0 kB real 0m12.495s user 0m12.160s sys 0m0.200s Just in case, here is info for one of servers: Portage 2.2.1 (hardened/linux/x86, gcc-4.6.3, glibc-2.15-r3, 3.9.9-hardened i686) ================================================================= System uname: Linux-3.9.9-hardened-i686-Intel-R-_Core-TM-2_Duo_CPU_E4500_@_2.20GHz-with-gentoo-2.2 KiB Mem: 2056068 total, 183404 free KiB Swap: 6013948 total, 5947900 free Timestamp of tree: Fri, 13 Sep 2013 16:45:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r2, 3.2.5-r2 dev-util/cmake: 2.8.10.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6, 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4-r1 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo powerman perl-experimental-snapshots local ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="*" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /service /usr/inferno/keydb /usr/inferno/lib /usr/inferno/services /usr/share/gnupg/qualified.txt /var/log /var/qmail/alias /var/qmail/control /var/vpopmail/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe" DISTDIR="/usr/portage-distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask-write" FCFLAGS="-march=native -O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr" FFLAGS="-march=native -O2 -pipe" GENTOO_MIRRORS="http://mirror.qubenet.net/mirror/gentoo/ http://mirrors.linuxant.fr/distfiles.gentoo.org/ http://mirror.leaseweb.com/gentoo/ http://mirror.bytemark.co.uk/gentoo/ http://gentoo.modulix.net/gentoo/" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j3" PKGDIR="/usr/portage-packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/powerman /var/lib/layman/perl-experimental-snapshots /usr/local/portage" SYNC="rsync://rsync.uk.gentoo.org/gentoo-portage" USE="acl bash-completion berkdb bzip2 cli cracklib crypt cxx dri fontconfig gdbm gif gnutls gpg hardened iconv idn ipv6 jpeg jpeg2k mbox mmx mng modules mudflap mysql ncurses network-cron nls nptl openmp pam pax_kernel pcre perl pic png readline session spell sse sse2 sse3 ssl ssse3 svg tcpd tiff truetype unicode urandom vim-syntax x86 zlib" ABI_X86="32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="log_config vhost_alias autoindex alias rewrite dir deflate filter mime negotiation auth_basic authn_file authz_host authz_user authz_groupfile cgi actions headers env setenvif" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, USE_PYTHON
I've asked in maillist and my friends - looks like that guess about slowdown on 64-bit systems is correct, many people confirmed this. Hope that helps.
Portage (Old): http://i.imgur.com/A93CdNR.png Portage (2.2.1): http://i.imgur.com/UbxEUB2.png In my case, it takes much longer because it calls select_files twice instead of once; but looking at the code, that's normal and it seems to be due to the state my system is in. Perhaps the Portage tree is in some kind of situation that makes the dependency tree harder for some situations? Regardless of that, it is quite interesting to note that previously aux_get took ~10% of the time whereas now aux_get takes ~25% of the time; that almost sounds like a performance regression in looking up the db information.
Related to bug 468486
Things have changed substantially since then.
