emerge -Dvau1 --keep-going `qlist -IC` takes 30 seconds Portage 2.1.11.10 (hardened/linux/amd64, gcc-4.7.2, glibc-2.14.1-r3, 3.5.1-2 x86_64) ================================================================= System uname: Linux-3.5.1-2-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-gentoo-2.1 Timestamp of tree: Wed, 15 Aug 2012 00:45:01 +0000 ccache version 3.1.8 [enabled] app-shells/bash: 4.2_p37 dev-java/java-config: 2.1.12 dev-lang/python: 2.7.3-r2, 3.2.3-r1 dev-util/ccache: 3.1.8 dev-util/cmake: 2.8.8-r3 dev-util/pkgconfig: 0.27 sys-apps/baselayout: 2.1-r1 sys-apps/openrc: 0.9.8.4 sys-apps/sandbox: 2.6 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.6, 1.12.3 sys-devel/binutils: 2.22-r1, 2.22.90 sys-devel/gcc: 4.4.7, 4.5.4::x-portage, 4.6.4_pre9999::toolchain, 4.6.4::x-portage, 4.7.1, 4.7.2_pre9999::toolchain, 4.7.2::x-portage, 4.8.0_pre9999::toolchain sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r3 sys-kernel/linux-headers: 3.5 (virtual/os-headers) sys-libs/glibc: 2.14.1-r3 Repositories: gentoo qt toolchain x-portage ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA PUEL dlj-1.1 ETQW RTCW-ETEULA googleearth AdobeFlash-10.3 Oracle-BCLA-JavaSE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -O3 -pipe -march=core2 -mtune=core2 -fomit-frame-pointer -funroll-loops --param max-unrolled-insns=64" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/lib/hsqldb" CONFIG_PROTECT_MASK="${EPREFIX}/etc/gconf /etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -O3 -pipe -march=core2 -mtune=core2 -fomit-frame-pointer -funroll-loops --param max-unrolled-insns=64" DISTDIR="/mnt/md3/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg ccache compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles metadata-transfer news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms split-elog split-log splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv webrsync-gpg" FFLAGS="-O2 -O3 -pipe -march=core2 -mtune=core2 -fomit-frame-pointer -funroll-loops --param max-unrolled-insns=64" LANG="en_GB.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed" LINGUAS="en_GB" MAKEOPTS="-j2" PKGDIR="/mnt/md3/cache/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/mnt/md3/cache" PORTDIR="/mnt/md3/portage" PORTDIR_OVERLAY="/var/lib/layman/qt /var/lib/layman/toolchain /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acpi aio alsa amd64 amr apng berkdb bzip2 cairo caps chm cli consolekit cracklib crypt curl cxx dbus device-mapper dlz dri dts dvb eix enca exif extensions extras faad ffmpeg flac gcj gdbm glib glibc-omitfp glitz gnutls gold graphite gtk gudev handbook hardened iconv jit jpeg jpeg2k justify lcms lm_sensors logrotate lzma matroska mikmod minizip mmx mng modplug modules mp3 mpeg mudflap multilib multislot ncurses nodrm nptl nsplugin ogg openal opengl openmp pam pax_kernel pch pcntl pcre pcre16 pdf pic png pppd qt qt3support qt4 rar readline rtsp sandbox schroedinger session slang spell sqlite sqlite3 sse sse2 sse3 ssl ssse3 symlink theora threads tiff truetype unicode urandom usb userpriv v4l2 vlc vorbis vpx x264 xcb xcomposite xml xv xvid zlib" ALSA_CARDS="virtuoso" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_GB" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
22 seconds with Python 3 and emerge -vpu1 --keep-going `qlist -IC`
51 seconds with same as in comment 1, but after "echo 3 > drop_caches"
Are you saying that it's slow in comparison to some other version of portage, or just slower than you would expect given the task at hand?
It's slow compared to Debian APT.
Some of the performance difference for Portage is related to that fact that it's possible to modify ebuilds at any time, and Portage is expected to detect these changes automatically and account for them. The makes access to the available package database somewhat less efficient than it would be for something like Debian APT. You can try alternative package managers in order to compare their performance, such as sys-apps/pkgcore and sys-apps/paludis. I've heard that pkgcore is the fastest one.
emerge should get ebuild dependencies only from sqlite database and shouldn't scan ebuilds. ebuild digest should update database. emerge --sync shouldn't download all ebuilds emerge --sync should only download option 1: xz compressed sqlite database with dependencies among all ebuilds option 2: xz compressed files with SQL commands which update sqlite dependency database, one file per day ebuilds should be downloaded on demand, e.g. emerge sqlite will download sqlite-1.2.3.ebuild
I think the existing repository layout and database/cache formats are pretty optimal for development-oriented scenarios. If you we support both development-oriented and consumer-oriented scenarios simultaneously, then conflicting goals will lead to sacrifices that negatively impact both kinds of users. So, I think it would be optimal to introduce an entirely separate repository layout and database/cache format for the consumer-oriented scenario. This has been discussed previously as a proposed GSOC project: http://wiki.gentoo.org/wiki/Google_Summer_of_Code/2012/Ideas#Repository_of_self-contained_ebuild_source_packages
emerge -vpu1 --keep-going `qlist -IC` takes 100 seconds after emerge-delta-webrsync
The time is more a result of algorithmic complexity [1] than it is of how the information is obtained, note that there is already a cache present in /usr/portage/metadata/md5-cache/ which limits the amount of data that needs to be accessed; there is barely any benefit of putting this in sqlite, if you want to try how it works with that information cached in SQLite then go ahead [2] since the functionality is already there and hasn't found to be a remarkable improvement. So, back to the algorithmic complexity: There are a lot of visibility checks going on (highest version, USE flag conditional dependencies, masked USE flags, masked packages, keyworded packages, slots, subslots and so on...). It's where Portage is spending most of its time, evaluating all these checks [1]. So, if one would like to see a speed up one would need to write caches for some or all of these things. That on its own is a task that requires some time to fix. An alternative would be to come up with another algorithmic way of accomplishing this task, but is it really worth rewriting Portage for this? Some people said yes and wrote alternatives, but those all seem to run behind on Portage (not support all of its features, not supporting EAPI 5, ...). So, work is to be done; now we only need to find people interested in doing it. [1]: http://i.imgur.com/A93CdNR.png [2]: http://webcache.googleusercontent.com/search?q=cache:Sv5MJhN0eD0J:en.gentoo-wiki.com/wiki/Portage_SQLite_Cache
Related to bug 468486
It isn't big problem.
