Bug 484788 – =sys-apps/portage-2.2.1 - About 3.5 times slower calculate dependencies than 2.1.12.2.

• Product: Portage Development
• Status: RESOLVED OBSOLETE

[Description Alex Efros 2013-09-13 16:06:26 UTC]

Looks like there is regression in portage-2.2 - it's calculate dependencies in 3.5 times slower than portage-2.1.

I've Intel Core i7-2600K overclocked at 4.6GHz and 8GB RAM. It's a shame `emerge -uDNpv world` spend 3 times more time (just to find out there is no available updates) than my kernel compiles!

I've run `emerge -uDNpv world` several times, so everything from /usr/portage is cached in memory, and I've monitored conky to make sure there was no disk I/O while executing these benchmarks (also this is proved by `time` output with "real" same as "user").

While running, emerge use 100% CPU (one Core) all the time until it finish.
Btw, with python-3.2 portage (both 2.2 and 2.1) works about 15% faster than with python-2.7.

# eselect python list
Available Python interpreters:
  [1]   python2.7
  [2]   python3.2 *

# qlist -I -v sys-apps/portage
sys-apps/portage-2.2.1

# time emerge -uDNpv world

These are the packages that would be merged, in order:

Calculating dependencies... done!

Total: 0 packages, Size of downloads: 0 kB

real	2m33.552s
user	2m31.307s
sys	0m1.923s

# emerge -1av '<portage-2.2'

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     UD ] sys-apps/portage-2.1.12.2 [2.2.1] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru -pl%" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB

Total: 1 package (1 downgrade), Size of downloads: 0 kB
…

# qlist -I -v sys-apps/portage
sys-apps/portage-2.1.12.2

# time emerge -uDNpv world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] sys-apps/portage-2.2.1 [2.1.12.2] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru (-pl%)" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB

Total: 1 package (1 upgrade), Size of downloads: 0 kB

real	0m42.091s
user	0m41.371s
sys	0m0.625s

[Comment 1 Alex Efros 2013-09-13 16:15:16 UTC]

For @system it's 2 times slower.

# qlist -I -v sys-apps/portage
sys-apps/portage-2.1.12.2

# time emerge -uDNpv system

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] sys-apps/portage-2.2.1 [2.1.12.2] USE="(ipc) (xattr) -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="ru (-pl%)" PYTHON_TARGETS="python2_7 python3_2 (-pypy2_0) -python2_6 (-python3_3) (-python3_4)" 0 kB

Total: 1 package (1 upgrade), Size of downloads: 0 kB

real	0m31.177s
user	0m30.820s
sys	0m0.305s

# qlist -I -v sys-apps/portage
sys-apps/portage-2.2.1

# time emerge -uDNpv system

These are the packages that would be merged, in order:

Calculating dependencies... done!

Total: 0 packages, Size of downloads: 0 kB

real	0m58.117s
user	0m57.485s
sys	0m0.541s


Portage 2.2.1 (hardened/linux/amd64, gcc-4.6.3, glibc-2.15-r3, 3.10.1-hardened-r1 x86_64)
=================================================================
System uname: Linux-3.10.1-hardened-r1-x86_64-Intel-R-_Core-TM-_i7-2600K_CPU_@_3.40GHz-with-gentoo-2.2
KiB Mem:     8163184 total,   1318412 free
KiB Swap:    4200960 total,   4194960 free
Timestamp of tree: Fri, 13 Sep 2013 15:15:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p45
dev-java/java-config:     2.1.12-r1
dev-lang/python:          2.7.5-r2, 3.2.5-r2
dev-util/cmake:           2.8.10.2-r2
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.12.6, 1.13.4
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.6.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo perl-experimental-snapshots gamerlay powerman local
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /opt/upsmon-usb/EXT/DownOS /opt/upsmon-usb/EXT/JSystem /service /usr/inferno/keydb /usr/inferno/lib /usr/inferno/services /usr/share/config /usr/share/gnupg/qualified.txt /var/log /var/qmail/alias /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage-distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask-write=y --backtrack=15"
FCFLAGS="-march=native -O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr"
FFLAGS="-march=native -O2 -pipe"
GENTOO_MIRRORS="http://gentoo.kiev.ua/ftp/ http://gentoo.iteam.net.ua/ http://tux.rainside.sk/gentoo/ http://trumpetti.atm.tut.fi/gentoo/ http://gentoo.inode.at/"
LANG="ru_RU.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j8"
PKGDIR="/usr/portage-packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/perl-experimental-snapshots /var/lib/layman/gamerlay /var/lib/layman/powerman /usr/local/portage"
SYNC="rsync://rsync3.ua.gentoo.org/gentoo-portage"
USE="X a52 aac acl alac alsa amd64 avx bash-completion berkdb bzip2 caps cdda cddb cli cracklib crypt cxx dbus dri dts dvb dvd flac fontconfig gdbm gif gnutls gpg hardened iconv icu id3tag idn ipv6 jpeg jpeg2k justify libnotify mac mad matroska mbox mmx mng modules mp3 mpeg mudflap multilib musepack mysql ncurses network-cron nls nptl nsplugin ogg opengl openmp pam pax_kernel pcre perl png qt3support readline session spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 svg tcpd theora tiff truetype unicode urandom vdpau vim-syntax vorbis wavpack x264 xattr xosd xv xvid xvmc zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="log_config vhost_alias autoindex alias rewrite dir deflate filter mime negotiation auth_basic authn_file authz_host authz_user authz_groupfile cgi actions headers env setenvif" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en ru" NGINX_MODULES_HTTP="access auth_basic autoindex browser charset empty_gif fastcgi geo gzip limit_conn limit_req map memcached proxy referer rewrite scgi split_clients ssi upstream_ip_hash userid uwsgi fancyindex" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" QEMU_SOFTMMU_TARGETS="x86_64 i386" QEMU_USER_TARGETS="x86_64 i386" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia nv nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, USE_PYTHON

[Comment 2 Tom Wijsman (TomWij) (RETIRED) 2013-09-13 18:32:51 UTC]

Hmm, I'll try to see if I can generate a new profiled image so we can compare it against the one I made a while ago so we can see in which path the regression is.

[Comment 3 Alex Efros 2013-09-14 00:05:53 UTC]

I've just updated portage on several servers, and noticed there is no slowdown - 2.2 works at about same speed as 2.1 (and it's much faster than on my home workstation, while server itself has much weaker CPU - like Core2 Duo E4500 @ 2.20GHz).

I didn't updated kernel to 3.10.1 yet and didn't switched from PT_PAX to XATTR_PAX, but I think that's only differences except… servers are 32-bit while workstation is 64-bit - and I think this may be reason for that slowdown!

# qlist -I -v sys-apps/portage
sys-apps/portage-2.2.1

# time emerge -uDNpv system

These are the packages that would be merged, in order:

Calculating dependencies... done!

Total: 0 packages, Size of downloads: 0 kB

real    0m12.495s
user    0m12.160s
sys     0m0.200s


Just in case, here is info for one of servers:


Portage 2.2.1 (hardened/linux/x86, gcc-4.6.3, glibc-2.15-r3, 3.9.9-hardened i686)
=================================================================
System uname: Linux-3.9.9-hardened-i686-Intel-R-_Core-TM-2_Duo_CPU_E4500_@_2.20GHz-with-gentoo-2.2
KiB Mem:     2056068 total,    183404 free
KiB Swap:    6013948 total,   5947900 free
Timestamp of tree: Fri, 13 Sep 2013 16:45:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.5-r2, 3.2.5-r2
dev-util/cmake:           2.8.10.2-r2
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.13.4
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.6.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo powerman perl-experimental-snapshots local
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="*"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /service /usr/inferno/keydb /usr/inferno/lib /usr/inferno/services /usr/share/gnupg/qualified.txt /var/log /var/qmail/alias /var/qmail/control /var/vpopmail/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage-distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --autounmask-write"
FCFLAGS="-march=native -O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync webrsync-gpg xattr"
FFLAGS="-march=native -O2 -pipe"
GENTOO_MIRRORS="http://mirror.qubenet.net/mirror/gentoo/ http://mirrors.linuxant.fr/distfiles.gentoo.org/ http://mirror.leaseweb.com/gentoo/ http://mirror.bytemark.co.uk/gentoo/ http://gentoo.modulix.net/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/usr/portage-packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--exclude ChangeLog --delete-excluded"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/powerman /var/lib/layman/perl-experimental-snapshots /usr/local/portage"
SYNC="rsync://rsync.uk.gentoo.org/gentoo-portage"
USE="acl bash-completion berkdb bzip2 cli cracklib crypt cxx dri fontconfig gdbm gif gnutls gpg hardened iconv idn ipv6 jpeg jpeg2k mbox mmx mng modules mudflap mysql ncurses network-cron nls nptl openmp pam pax_kernel pcre perl pic png readline session spell sse sse2 sse3 ssl ssse3 svg tcpd tiff truetype unicode urandom vim-syntax x86 zlib" ABI_X86="32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="log_config vhost_alias autoindex alias rewrite dir deflate filter mime negotiation auth_basic authn_file authz_host authz_user authz_groupfile cgi actions headers env setenvif" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 intel mach64 mga nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa via vmware nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, USE_PYTHON

[Comment 4 Alex Efros 2013-09-15 12:44:27 UTC]

I've asked in maillist and my friends - looks like that guess about slowdown on 64-bit systems is correct, many people confirmed this. Hope that helps.

[Comment 5 Tom Wijsman (TomWij) (RETIRED) 2013-09-15 14:40:55 UTC]

Portage (Old): http://i.imgur.com/A93CdNR.png
Portage (2.2.1): http://i.imgur.com/UbxEUB2.png

In my case, it takes much longer because it calls select_files twice instead of once; but looking at the code, that's normal and it seems to be due to the state my system is in. Perhaps the Portage tree is in some kind of situation that makes the dependency tree harder for some situations?

Regardless of that, it is quite interesting to note that previously aux_get took ~10% of the time whereas now aux_get takes ~25% of the time; that almost sounds like a performance regression in looking up the db information.

[Comment 6 Xavier Miller (RETIRED) 2013-10-01 06:11:50 UTC]

Related to bug 468486

[Comment 7 Sam James 2023-08-29 05:17:59 UTC]

Things have changed substantially since then.