I found that new 2.1.2 portage is really slow with the first 3-4 operations of the day, so it seems like a caching problem. With "slow" I mean minutes, especially when I try 'emerge -uDNpv world', but also with simple research like 'emerge -pv kdebase'. Here's my info. Portage 2.1.2_rc3-r9 (default-linux/x86/2006.1/desktop, gcc-4.1.1/vanilla, glibc-2.5-r0, 2.6.19-gentoo-r2 i686) ================================================================= System uname: 2.6.19-gentoo-r2 i686 Genuine Intel(R) CPU T2400 @ 1.83GHz Gentoo Base System version 1.12.8 Last Sync: Sat, 23 Dec 2006 01:30:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] ccache version 2.4 [enabled] app-admin/eselect-compiler: 2.0.0_rc2-r1 dev-java/java-config: 1.3.7, 2.0.31 dev-lang/python: 2.4.4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: 2.4-r6 sys-apps/sandbox: 1.2.18.1 sys-devel/autoconf: 2.13, 2.61 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.17 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium-m -pipe -fomit-frame-pointer -msse3" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/eselect/compiler /etc/gconf /etc/java-config/vms/ /etc/revdep-rebuild /etc/splash /etc/terminfo" CXXFLAGS="-O2 -march=pentium-m -pipe -fomit-frame-pointer -msse3" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache confcache digest distlocks metadata-transfer parallel-fetch sandbox sfperms strict" GENTOO_MIRRORS="ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/ http://distfiles.gentoo.org" LANG="it_IT.UTF-8" LC_ALL="it_IT.UTF-8" LINGUAS="it" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/portage/local/edo /usr/portage/local/xeffects/trunk /usr/portage/local/xeffects/experimental" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X acpi aiglx alsa alsa_cards_hda-intel alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_plugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoop alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter alsa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugins_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa_pcm_plugins_shm alsa_pcm_plugins_softvol apache2 arts bash-completion berkdb bitmap-fonts cairo cdr cli cracklib crypt cups dbus directfb dlloader dri dvb dvd dvdr dvdread eds elibc_glibc emboss encode esd fam fbcon firefox fortran gdbm gif gpm gstreamer gtk gtk2 hal iconv ieee1394 imagemagick imlib input_devices_evdev input_devices_keyboard input_devices_mouse input_devices_synaptics ipv6 isdnlog java jpeg kde kdeenablefinal kernel_linux ldap libg++ linguas_it mad mikmod mmx mmxext motif mp3 mpeg ncurses nls nptl nptlonly nsplugin nvidia ogg opengl oss pam pcmcia pcre pdf perl php png ppds pppd python qt qt3 qt4 quicktime readline reflection samba sdl session smp spell spl sse sse2 ssl tcpd threads truetype truetype-fonts type1-fonts udev unicode userland_GNU userlocales video_cards_nvidia vorbis win32codecs xcomposite xml xml2 xorg xsl xslt xv xvid zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS
The performance penalty in 2.1.2 relative to 2.1.1 is due to the addition of several important features: 1) reverse blocker detection (bug 128809) 2) complete dependency graph (bug 147766) 3) installed packages that do not have matching ebuilds can satisfy dependencies (bug 48195) The extra time taken for dependency calculations depends on the number of packages installed, so users with more packages will notice more of a performance difference. Given the improvements in dependency calculations, I think the performance level in 2.1.2_rc3-r9 is acceptable. I'll add an explanation of these issues to the release notes in order to help clarify the situation for users.
(In reply to comment #1) > Given the improvements in dependency calculations, I think the performance > level in 2.1.2_rc3-r9 is acceptable. I'll add an explanation of these issues > to the release notes in order to help clarify the situation for users. I think the general behaviour of portage 2.1.2_rc3-r9 is good, but in some situation the slowness is not completely acceptable. For me and many other users this is an important aspect of a package manager.
(In reply to comment #0) > I found that new 2.1.2 portage is really slow with the first 3-4 operations of > the day, so it seems like a caching problem. It must be some variation of bug 124041. I see that you have 3 overlays. Each time that you sync, it's possible that some of the eclasses have been modified. When ebuilds from the overlay are pulled into the dependency graph, their metadata cache needs to be regenerated if they inherit any modified eclasses.
(In reply to comment #3) You're right, without the overlays portage is a little bit faster, although there's not so much improvement on performances. Anyway I want my overlays to stay in portage tree, so you say that portage will remain so slow as it is now?
(In reply to comment #4) > without the overlays portage is a little bit faster If this is like bug 124041, then then it should only be slow when it's generating metadata cache. After the cache has been generated, there's no problem. Do you have any eclasses in your overlay? This type of thing is never an issue for people that have small overlays with no eclasses in them. They don't have to generate metadata cache because it's already generated on our master mirror and transferred when they sync. We can change portage to avoid triggering cache generation for ebuilds that match installed packages, but that means the the dependency metadata from live ebuilds will no longer be used for installed packages. That would probably be fine in most cases but portage has never behaved that way before. It's always used the live ebuild metadata (that was the root of bug 48195). Now it uses the live ebuild metadata and falls back to the installed metadata if there is no matching ebuild in the tree.
(In reply to comment #5) > If this is like bug 124041, then then it should only be slow when it's > generating metadata cache. After the cache has been generated, there's no > problem. Do you have any eclasses in your overlay? No I don't have eclasses in my overlays. And anyway they're small, two of them are the two from gentoo-xeffects and the third is a local overlay for ebuilds not in portage tree, and there's only 3 ebuilds in it. The general performances of my machine are good, but look, for example, at the first emerge of the day I do: # time emerge -p kdebase These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] kde-base/kdebase-3.5.5-r3 real 1m32.194s user 0m2.640s sys 0m1.000s Isn't it a lot of time? And the second time is much faster: # time emerge -p kdebase These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] kde-base/kdebase-3.5.5-r3 real 0m2.237s user 0m2.040s sys 0m0.108s Maybe the first time there's much more I/O operations?
In svn r5402 I've added a metadata cache for the installed packages. This greatly improves the performance for loading the metadata when the pagecache is cold.
In svn r5410 I've added a blocker cache for installed packages so that the dependency data doesn't have to be re-evaluated for every emerge invocation.
This has been released in 2.1.2_rc4-r2.
If you're interested in measuring the performance differences, you may want to know that emerge now creates 2 cache files when it's run as a superuser: /var/cache/edb/vdb_metadata.pickle /var/cache/edb/vdb_blockers.pickle Both files are disposable, so it's okay to remove them for testing purposes. The vdb_metadata.pickle is the one that will greatly improve performance for the first invocation of emerge. See http://linux-mm.org/Drop_Caches usage of /proc/sys/vm/drop_caches which is useful for testing it's effectiveness. Note that vdb_blockers.pickle will be invalidated every time that the set of installed packages or virtuals change, but it doesn't take very long to regenerate it. After an installation, this cache will be invalid, but any installation command run as a superuser (such as `emerge -p portage`) will cause it to regenerate and all users will be able to use it while it remains valid.
