For a while I am experiencing the following portage annoyance: After an "emerge sync" (for which the metadata/cache update takes about ten seconds), the first call to "emerge -NDuvp world" takes between five and ten minutes emerge -NDuvp world 321.05s user 77.90s system 94% cpu 7:02.77 total -- Portage 2.1_pre4-r1 (default-linux/x86/2005.1, gcc-3.4.5, glibc-2.3.6-r3, 2.6.15-gentoo-r5 i686) ================================================================= System uname: 2.6.15-gentoo-r5 i686 Intel(R) Pentium(R) M processor 1500MHz Gentoo Base System version 1.12.0_pre16 dev-lang/python: 2.4.2-r1 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1-r1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r3 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=pentium-m -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3/share/config /usr/share/X11/xkb /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/texmf/web2c /etc/env.d" CXXFLAGS="-march=pentium-m -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs autoconfig cvs digest distlocks fixpackages sandbox sfperms sign strict" GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo ftp://trumpetti.atm.tut.fi/gentoo http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo" LINGUAS="de" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/overlay/cvs /usr/local/overlay/gnome /usr/local/overlay/xgl" SYNC="rsync://194.97.4.250/gentoo-portage" USE="x86 X a52 aac acpi alsa asf avi beagle berkdb bitmap-fonts bonjour browserplugin bzip2 cairo cdr crypt cups dbus divx4linux dvd dvdr dvdread eds emboss encode exif fame firefox flac foomaticdb fortran galago gdbm gif glitz gnome gpm gstreamer gtk gtk2 hal howl imlib ipv6 java jpeg libg++ libnotify libsexy libwww lzo mad matroska mikmod mjpeg mmx mng mono moznocompose moznoirc moznomail mozsvg mp3 mpeg ncurses nls nomotif nptl nsplugin nvidia ogg oggvorbis opengl oss pam pam_console pcre pdflib perl png python quicktime readline spell sse sse2 ssl svg tcpd theora tiff truetype truetype-fonts type1-fonts unicode usb vorbis win32codecs wmf xml2 xv xvid zlib elibc_glibc input_devices_evdev input_devices_keyboard input_devices_mouse input_devices_synaptics kernel_linux linguas_de userland_GNU video_cards_nv video_cards_nvidia" Unset: ASFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, LANG, LC_ALL, LDFLAGS, MAKEOPTS
(In reply to comment #0) > After an "emerge sync" (for which the metadata/cache update takes about ten > seconds), the first call to "emerge -NDuvp world" takes between five and ten > minutes Generally, that type of behavior indicates that metadata is being generated due to invalid or nonexistent cache. Are you using the standard cache modules (anything special in /etc/portage/modules)? It seems like the post-sync metadata transfer isn't working correctly. Does this happen after every sync?
wopr-mobile ~ # cat /etc/portage/modules cat: /etc/portage/modules: No such file or directory And yes, this happens after every "emerge sync".
10 seconds seems a bit fast for metadata updates if your on a laptop - unless you've got a gig or more of ram perhaps. Doing anything interesting with rsync_excludes?
The 10 seconds should have been 30 to 90 seconds, sorry. I do not use rsync_excludes.
What's the average times after a reboot? Do you see any processes named "ebuild" during the first run?
After reboot (without an "emerge sync") the "emerge -NDuvp world" takes about 60 seconds. After an "emerge sync" I see ebuild.sh processes while "emerge -NDuvp world" is running. I have strace logfiles for both of the above cases that I can attach if you're interested.
Same here. I have two nearly identical machines running gentoo and mounting /usr/portage via nfs from a third machine. I do emerge --metadata before doing emerge -uDvNa world on both machines, emerge sync is done on the server. On one machine, emerge -uDvNa world is quite fast (20-30 seconds), on the other machine "it takes forever" when done for the first time and also 20-30 seconds when done for the second time.
The local dependency cache is stored at /var/cache/edb/dep. Anything interesting about that directory? Philipp, do you happen to be using an extensive overlay on the slower machine?
I've just one ebuild in my overlay. What do you mean with "interesting"? Same rights on both machines, same sizes, same filesystem (ext3), harddrives should be the same speed (is there a good and simple tool to test this?).
emerge --metadata on the "slower" machine takes as long as on the "faster" machine. emerge -uDvNp world on the "slower" machine prints --quote-- These are the packages that I would merge, in order: Calculating world dependencies \/usr/local/portage/eclass/flag-o-matic.eclass: line 276: ${#new_math}-2: substring expression < 0 true --/quote-- to stdout/stderr
Any change in absolute path or mtime of an ebuild or eclass will force regeneration. Here are a few things to check: 1) Do you have eclasses in your overlays? That will cause metadata regeneration for any ebuilds that use those eclasses (something like eutils.eclass in overlay would be very noticable). 2) The cache stores the absolute paths of ebuilds and eclasses, so if you move $PORTDIR you need to run `emerge --metadata` again to account for that. 3) The diff of the cache entries before and after regeneration should help isolate the problem. Here's an example: cd /var/cache/edb rm -rf dep/* emerge --metadata cp -a dep dep.rsync emerge -puD world diff -Nur dep.rsync dep > cache-regen.diff
I think I found the problem (and I feel really stupid about this, sorry): I have /usr/local/overlay/cvs in my PORTDIR_OVERLAY, that directory contains the following symlinks to my CVS checkout of the portage tree: drwxr-xr-x 2 sb users 72 Feb 11 17:00 dev-lang lrwxrwxrwx 1 sb users 29 Feb 11 17:00 dev-php -> /usr/local/src/gentoo/dev-php lrwxrwxrwx 1 sb users 30 Feb 11 17:00 dev-php4 -> /usr/local/src/gentoo/dev-php4 lrwxrwxrwx 1 sb users 30 Feb 11 17:00 dev-php5 -> /usr/local/src/gentoo/dev-php5 lrwxrwxrwx 1 sb users 28 Feb 11 17:00 eclass -> /usr/local/src/gentoo/eclass Am I right in assuming that once I start doing "cvs upd -dAP" in these directories before I do "emerge sync" the problem will go away? Thanks for bearing with me.
I removed the symlink to /usr/local/src/gentoo/eclass and created symlinks to the eclasses that I need.
(In reply to comment #12) > Am I right in assuming that once I start doing "cvs upd -dAP" in these > directories before I do "emerge sync" the problem will go away? Only under the condition that the eclasses have identical timestamps to those from rsync and that portage sees them in $PORTDIR/eclass/ rather than in the overlay. > Thanks for bearing with me. No problem, I experienced the same sort of confusion before I was familiar with the workings of the cache validation. We should probably document it better.
> 1) Do you have eclasses in your overlays? That will cause metadata > regeneration for any ebuilds that use those eclasses (something like > eutils.eclass in overlay would be very noticable). After removing this everything is fine. This solved my problem. I had /usr/local/portage/eclass linked to /usr/portage/eclass.
I'm also seeing this and it's very annnoying... It seems as if this is happening after *every* emerge --sync run, and then only for the first "emerge -Duvat world" run. alexander@blatt ~ $ cat /etc/portage/modules alexander@blatt ~ $ ls -la /etc/portage/modules -rw-rw-r-- 1 portage portage 0 17. Mar 20:29 /etc/portage/modules I also haven't set any rsync_excludes. I see "ebuild" processes during the long wait: alexander@blatt ~ $ ps awux|grep -i ebuild root 22673 0.0 0.0 1712 464 pts/9 S+ 07:20 0:00 [libgphoto2-2.1.6-r1] sandbox /usr/lib/portage/bin/ebuild.sh depend root 22674 0.0 0.2 4412 1716 pts/9 R+ 07:20 0:00 /bin/bash /usr/lib/portage/bin/ebuild.sh depend root 22675 0.0 0.0 4412 584 pts/9 R+ 07:20 0:00 /bin/bash /usr/lib/portage/bin/ebuild.sh depend 1000 22677 0.0 0.1 2892 824 pts/1 S+ 07:20 0:00 grep -i ebuild And a little later, I see different ebuild processes; ie. for different packages. Reg: /var/cache/edb/dep: It's on a reiserfs filesystem which is on a LVM lv. It's currently 86% filled. I don't have any eclasses in my overlays. $PORTDIR isn't moved. So, how can I find out, what's causing this overly long delay during
(In reply to comment #16) > I don't have any eclasses in my overlays. > > $PORTDIR isn't moved. If you've already checked everything mentioned in this bug, then please file a new bug. I don't want this one to get cluttered.
We've found that if $PORTDIR is a symlink then it can lead to cache problems with portage-2.1. See bug 137965.
*** Bug 153454 has been marked as a duplicate of this bug. ***
Zac, is there any reason to keep this one open?
The performance will improve quite a bit once the resolver has been rewritten so that it pulls metadata for a minimal number of packages rather than pulling it for every single version in visibility checks.
This is fixed in svn r13291 by adding support for metadata/layout.conf. Now, eclasses will only apply to their containing repository and any other repositories which reference their containing repository via layout.conf. This new approach is the first step toward "QA Overlay Layout support" as discussed on the gentoo-dev mailing list: http://archives.gentoo.org/gentoo-dev/msg_33c61550b4ed2b7b25dd5a4110e1ec81.xml
This is fixed in 2.2_rc29.
Great, but what's the syntax of this file and where exactly should be placed?
(In reply to comment #24) > Great, but what's the syntax of this file and where exactly should be placed? There's an example in the portage(5) man page which shows how the java-experimental overlay will use it. Inside that overlay they will have a metadata/layout.conf file containing "masters = gentoo java-overlay". If you want eclasses from overlays to be inherited by ebuilds in the main tree, then there's no way to configure that yet. I'll be adding it for bug #265747.
This is fixed in 2.1.6.12.
