Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 498478

Summary: sys-apps/portage-2.2.7 with dev-lang/python-2.7.5-r3:2.7 - strange performance problem
Product: Portage Development Reporter: Frank Krömmelbein <kroemmelbein>
Component: UnclassifiedAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: antoni, floppym, nikoli, python
Priority: Normal    
Version: 2.2   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=498696
Whiteboard:
Package list:
Runtime testing required: ---
Attachments: Enforce python_targets on pyxattr

Description Frank Krömmelbein 2014-01-18 17:06:30 UTC
I just run into a strange performance problem with emerge gentoo-sources.
It took very long to re emerge the gentoo-sources with python 2.7.

time emerge gentoo-sources (Python 2.7)
real    9m11.195s
user    1m46.804s
sys     7m13.665s

time emerge gentoo-sources (Python 3.3)
real    1m19.685s
user    1m7.434s
sys     0m13.656s

So portage was 7 times slower with Python 2.7 as with 3.3!

But this not true for all other packages, in many other tests with various other packages, the results are close together.
The only other bigger difference i found was with:

time emerge -pv --update --newuse --deep world --with-bdeps=y

with Pyhon 2.7:
real    1m4.404s
user    1m3.746s
sys     0m0.624s

with Python 3.3:
real    0m55.784s
user    0m55.283s
sys     0m0.477s

System Infos:
emerge -pv1 dev-lang/python:2.7 dev-lang/python:3.3
[ebuild   R    ] dev-lang/python-2.7.5-r3:2.7  USE="gdbm ipv6 ncurses readline sqlite ssl threads (wide-unicode) xml -berkdb -build -doc -examples -hardened -tk -wininst" 0 kB
[ebuild   R    ] dev-lang/python-3.3.2-r2:3.3  USE="gdbm ipv6 ncurses readline sqlite ssl threads xml -build -doc -examples -hardened -tk -wininst" 0 kB

emerge portage
[ebuild   R    ] sys-apps/portage-2.2.7  USE="(ipc) xattr -build -doc -epydoc (-pypy2_0) -python2 -python3 (-selinux)" LINGUAS="-ru" PYTHON_TARGETS="python2_7 python3_3 (-pypy2_0) -python2_6 -python3_2 (-python3_4)" 0 kB

emerge --info
Portage 2.2.7 (default/linux/amd64/13.0/desktop/kde, gcc-4.7.3, glibc-2.17, 3.10.25-gentoo x86_64)
=================================================================
System uname: Linux-3.10.25-gentoo-x86_64-Intel-R-_Core-TM-_i7-2630QM_CPU_@_2.00GHz-with-gentoo-2.2
KiB Mem:    20579368 total,  14925104 free
KiB Swap:          0 total,         0 free
Timestamp of tree: Sat, 18 Jan 2014 16:30:01 +0000
ld GNU ld (GNU Binutils) 2.23.2
app-shells/bash:          4.2_p45
dev-java/java-config:     2.1.12-r1
dev-lang/python:          2.7.5-r3, 3.3.2-r2
dev-util/cmake:           2.8.11.2
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.12.4
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.11.6, 1.12.6, 1.13.4
sys-devel/binutils:       2.23.2
sys-devel/gcc:            4.7.3-r1
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo steam-overlay lokal
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=corei7 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=corei7 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/steam /usr/local/portage"
USE="X a52 aac aacs acl acpi alsa amd64 berkdb bluetooth bluray branding btrfs bzip2 cairo cdda cddb cdr cec clamav cli clucene consolekit corefonts cracklib crypt cryptsetup cups cxx dbus declarative device-mapper dmraid dri dts dv dvb dvd dvdr ebook emboss encode epub exif faac fam fat ffmpeg firefox flac fontconfig fontforge fortran gdbm gif gimp gphoto2 gpm gtk gudev iconv id3tag idn inotify ipv6 jfs jpeg jpeg2k kde kipi ladspa lame lcms ldap libkms libnotify libsamplerate libvisual lirc lvm lzma mad matroska mdadm mmx mng mod modplug modules mp3 mp4 mpeg mtp mudflap multilib musicbrainz mysql ncurses networkmanager nfs nls nptl ntfs ogg opencl openexr opengl openmp openrc opus pam pango pcmcia pcre pdf phonon pkcs11 plasma png policykit postscript ppds ppp pvr qt3support qt4 rar raw rdesktop readline reiserfs samba scanner sdl semantic-desktop session smartcard sna sndfile spell sse sse2 sse3 sse4 sse4_1 ssl ssse3 startup-notification svg system-cairo system-icu system-jpeg system-sqlite taglib tcpd tiff truetype udev udisks unicode upnp upower usb v4l vaapi vcd vdpau vlc vnc vorbis vpx wifi wxwidgets x264 xa xattr xcb xcomposite xfs xinerama xml xmp xpm xscreensaver xv xvid xvmc zip zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="*" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-32 efi-64 pc qemu emu" INPUT_DEVICES="evdev keyboard mouse joystick synaptics vmmouse lirc" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="nlpsolver presenter-minimizer wiki-publisher" LINGUAS="de en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" QEMU_SOFTMMU_TARGETS="x86_64 arm i386" QEMU_USER_TARGETS="x86_64 arm i386" RUBY_TARGETS="ruby19 ruby18" SANE_BACKENDS="*" USERLAND="GNU" VIDEO_CARDS="intel nvidia vmware virtualbox modesetting vesa nouveau i915 i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON



Reproducible: Always
Comment 1 Sebastian Luther (few) 2014-01-18 17:32:04 UTC
That's the time it takes to resolve dependencies + the time needed to actually install the package?

You should first separate the two. Add -p to get the dependency resolution time and then --nodeps to get the install time.
Comment 2 Frank Krömmelbein 2014-01-18 18:39:08 UTC
Hello Sebastian,

forget to memtion the problem occurs after the patching of the sources.

Filesystem root Partition is ext4 on a fast Samsung 840 Pro SSD.

Here are the times: 

time emerge -p gentoo-sources

Python 2.7
real    0m6.849s
user    0m6.706s
sys     0m0.138s

Python 3.3
real    0m6.265s
user    0m6.184s
sys     0m0.079s

So nearly no time difference for dependency resolution.

time emerge --nodeps gentoo-sources

Python 2.7
real    7m33.432s
user    1m39.035s
sys     5m44.095s


Python 3.3
real    1m13.797s
user    1m1.688s
sys     0m13.611s
Comment 3 Sebastian Luther (few) 2014-01-18 18:42:25 UTC
@python: Any idea why file system operations may be that much slower with 2.7 compared to 3.3?
Comment 4 Arfrever Frehtes Taifersar Arahesis 2014-01-18 19:04:38 UTC
Frank Krömmelbein: Show:
emerge -ptv dev-python/pyxattr
Comment 5 Frank Krömmelbein 2014-01-18 19:13:16 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #4)
> Frank Krömmelbein: Show:
> emerge -ptv dev-python/pyxattr

Not installed!

emerge -ptv dev-python/pyxattr

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild  N     ] dev-python/pyxattr-0.5.2  USE="{-test}" PYTHON_TARGETS="python2_7 python3_3 (-pypy2_0) -python2_6 -python3_2" 25 kB

Total: 1 package (1 new), Size of downloads: 25 kB

Should i install this?
Comment 6 Arfrever Frehtes Taifersar Arahesis 2014-01-18 19:16:44 UTC
Python >=3.3 has xattr support in standard library.
With older Python versions, Portage can use either dev-python/pyxattr (faster) or getfattr and setfattr from sys-apps/attr (slower).

Anyway why would you want to use Portage with Python 2?
Comment 7 Frank Krömmelbein 2014-01-18 19:33:06 UTC
(In reply to Arfrever Frehtes Taifersar Arahesis from comment #6)
> Python >=3.3 has xattr support in standard library.
> With older Python versions, Portage can use either dev-python/pyxattr
> (faster) or getfattr and setfattr from sys-apps/attr (slower).
> 

Thanks for the info!

I just installed dev-python/pyxattr-0.5.2 and now with Python 2.7

time emerge --nodeps gentoo-sources

real    1m14.420s
user    1m1.902s
sys     0m13.812s

So nearly identical with Python 3.3. Wow!


> Anyway why would you want to use Portage with Python 2?
In the past i had some problems with Python 3.x, so i switched on my new installation also back to the trused Python 2.x Version.
But i will try now Python 3.3.

Thanks!
Comment 8 Frank Krömmelbein 2014-01-18 19:45:51 UTC
Btw.
Are there any known errors with dev-python/pyxattr? 
If not would it not make sense, to add a "xattr" useflag for python:2,7 which pulls in dev-python/pyxattr?

I think i was not the only one who uses the older Python version, so this would be massiv speed improvement for this group.
Comment 9 Mike Gilbert gentoo-dev 2014-01-18 20:23:35 UTC
Created attachment 368112 [details, diff]
Enforce python_targets on pyxattr

This would resolve this issue for most python2 users; the exception being if they did not have python2_X in PYTHON_TARGETS but still invoke emerge with python2.X.
Comment 10 Mike Gilbert gentoo-dev 2014-01-20 17:43:32 UTC
+  20 Jan 2014; Mike Gilbert <floppym@gentoo.org> portage-2.2.8.ebuild,
+  portage-9999.ebuild:
+  Enforce python_targets on pyxattr dep, bug 498478.
Comment 11 Mike Gilbert gentoo-dev 2014-01-20 18:35:08 UTC
*** Bug 498696 has been marked as a duplicate of this bug. ***
Comment 12 Jeroen Roovers (RETIRED) gentoo-dev 2014-01-27 20:07:19 UTC
*** Bug 499488 has been marked as a duplicate of this bug. ***