$ time sudo FEATURES="-collision-protection -protect-owned" emerge -v sys-kernel/vanilla-sources > /dev/null real 10m18.040s user 1m37.181s sys 7m38.159s This is perf top while itβs doing that: ------------------------------------------------------------------------------ PerfTop: 23157 irqs/sec kernel:81.0% [100000 cycles], (all, 2 CPUs) ------------------------------------------------------------------------------ samples pcnt kernel function _______ _____ _______________ 100055.00 - 34.7% : copy_page_range 70068.00 - 24.3% : unmap_vmas 18438.00 - 6.4% : page_remove_rmap 14164.00 - 4.9% : release_pages 7897.00 - 2.7% : copy_page_c 7605.00 - 2.6% : page_fault 6090.00 - 2.1% : vm_normal_page 3969.00 - 1.4% : free_pages_and_swap_cache 2460.00 - 0.9% : _raw_spin_lock 2171.00 - 0.8% : clear_page_c 2143.00 - 0.7% : schedule 2018.00 - 0.7% : _raw_spin_lock_irqsave 1983.00 - 0.7% : flush_tlb_page 1870.00 - 0.6% : do_wp_page 1510.00 - 0.5% : __wake_up_bit Reproducible: Always Steps to Reproduce: 1. emerge the kernel sources 2. wait a long time 3. Actual Results: I have output of "strace -f" of that install, it's 850MB. I attach the output of "strace -f -c", maybe that's useful. Portage 2.1.7.16 (default/linux/amd64/10.0/desktop, gcc-4.4.2, glibc-2.11-r1, 2.6.33-rc5 x86_64) ================================================================= System uname: Linux-2.6.33-rc5-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E6550_@_2.33GHz-with-gentoo-2.0.1 Timestamp of tree: Fri, 22 Jan 2010 10:15:01 +0000 app-shells/bash: 4.0_p37 dev-java/java-config: 2.1.10 dev-lang/python: 2.6.4, 3.1.1-r1 dev-util/cmake: 2.8.0 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.6.0-r1 sys-apps/sandbox: 2.2 sys-devel/autoconf: 2.13, 2.65 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6b virtual/os-headers: 2.6.30-r1 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-g -O2 -pipe -march=core2 -momit-leaf-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-g -O2 -pipe -march=core2 -momit-leaf-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests distlocks fixpackages installsources news parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch" GENTOO_MIRRORS="http://mirror.isoc.org.il/pub/gentoo/ " LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_US" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/home/system/var_tmp_portage" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/gcc-porting /var/lib/layman/mozilla /var/lib/layman/java-overlay /var/lib/layman/toolchain /usr/local/portage" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 bash-completion bidi bluetooth branding bzip2 cairo cdr cli cracklib crypt cxx dbus dri dts dvd dvdr eds emboss encode evo fam firefox flac gif gpm gstreamer gtk hal iconv ipv6 jpeg kde libnotify lzma mad mikmod mmx mng modules mp3 mp4 mpeg mudflap multilib ncurses nls nptl nptlonly ogg opengl openmp pch pcre pdf perl png ppds pppd python qt3support qt4 quicktime readline reflection sdl session spell spl sqlite3 sse sse2 ssl startup-notification svg sysfs tcpd thunar tiff truetype unicode usb vorbis x264 xcb xcomposite xml xorg xulrunner xv xvid zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="stv0680" ELIBC="glibc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" QEMU_SOFTMMU_TARGETS="arm i386 ppc ppc64 sparc x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18" SANE_BACKENDS="niash stv680" USERLAND="GNU" VIDEO_CARDS="nv nvidia nouveau" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Created attachment 217190 [details] result of strace -f -c of emerge
Yes. It is a big source package, so it takes a longer time to download. Then it has to unpack, then it has to merge it to the live fs. As you can see, if your I/O is not that great (slow hard drive) then it will take a long time. This is not a fault of sys-apps/portage, nor something that can be "fixed" besides getting faster hard drives. (Or setting up tmpfs, etc). So, resolving as invalid. Not a bug.
"it takes a longer time to download" ??? No matter how bad my hard drive is, it just doesn't take 10 MINUTES to unpack the tarball. Here is how long it takes to unpack the tarball and patch the source: $ time tar xjf /usr/portage/distfiles/linux-2.6.32.tar.bz2 real 0m17.326s user 0m15.946s sys 0m1.597s $ cd linux-2.6.32/ $ time bzcat /usr/portage/distfiles/patch-2.6.32.4.bz2 | patch -p1 real 0m0.215s user 0m0.102s sys 0m0.066s The difference between 17 SECONDS of wall clock time and 10 MINUTES of wall clock time is the bug. The strace output shows (for those too lazy to download): % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 45.42 335.784690 10516 31932 635 wait4 28.21 208.520449 6736 30955 6 execve 26.25 194.052701 6204 31281 clone 0.03 0.230745 0 546458 91455 write The full strace log is only 25MB in lrzip, I'll mail it to you or upload it somewhere upon request.
Assigning to portage devs, although, I don't expect an action.
Zeev, one more thing. You must have something odd going on with your setup. %% time sudo emerge -v vanilla-sources > /dev/null real 1m42.407s user 0m30.964s sys 0m17.421s And that includes the wget time because I didn't have the sources.
I'll be glad of any help to find out what's exactly wrong with my setup. Or at least the steps to fix it, even without an explanation (though I'd really like to know).
If PORTAGE_TMPDIR is on a different partition than /usr/src/ then you'll have a significant performance penalty since the files have to be copied instead of renamed. You might want to try putting them on the same partition to see how much difference that makes.
This is with PORTAGE_TMPDIR="/var/tmp" real 9m4.759s user 1m20.650s sys 7m25.004s untar'ing on the partition where PORTAGE_TMPDIR was, patching there and then mv'ing the whole thing to /usr/src manually only adds seconds, too. The filesystems are EXT4 for root and resiserfs3 for /home (where PORTAGE_TMPDIR was). Does portage do mv or something clever, that might be slow in my setup? My filesystems don't have xattrs, for example.
(In reply to comment #8) > Does portage do mv or something clever, that might be slow in my setup? No, it should never call mv. In portage.movefile() there is a place where it will call mv if given a non-regular file, but that's not a typical case.
I've given this another go and found out that when portage merges into live filesystem it not only performs md5 on each file but also calls prelink --verify if I have sys-devel/prelink installed. No prelink: /usr/src $ sudo rm -rf linux-2.6.35-rc5/ $ time sudo emerge vanilla-sources > /dev/null real 2m29.873s user 1m14.472s sys 0m30.118s With prelink: /usr/src $ sudo rm -rf linux-2.6.35-rc5/ $ time sudo emerge vanilla-sources > /dev/null real 12m10.732s user 1m44.203s sys 9m29.295s Can portage call prelink only on executables and shared libraries? Or at least not on source and documentation files?
(In reply to comment #10) > I've given this another go and found out that when portage merges into live > filesystem it not only performs md5 on each file but also calls prelink > --verify if I have sys-devel/prelink installed. > > No prelink: > /usr/src $ sudo rm -rf linux-2.6.35-rc5/ > $ time sudo emerge vanilla-sources > /dev/null > real 2m29.873s > user 1m14.472s > sys 0m30.118s > > With prelink: > /usr/src $ sudo rm -rf linux-2.6.35-rc5/ > $ time sudo emerge vanilla-sources > /dev/null > real 12m10.732s > user 1m44.203s > sys 9m29.295s > > Can portage call prelink only on executables and shared libraries? Or at least > not on source and documentation files? Curious, why do you have unmerge-orphans off? I doubt portage does it, but FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that feature md5 doesn't matter, meaning the prelink reversal is uneeded). It'll be fun threading that down into perform_checksum I suspect, but it should be done (and portage should grow awareness of what all to actually do prelink checks on).
(In reply to comment #11) > Curious, why do you have unmerge-orphans off? I doubt portage does it, but > FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that > feature md5 doesn't matter, meaning the prelink reversal is uneeded). FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not during merge since the checksums are needed for recording in CONTENTS.
(In reply to comment #12) > (In reply to comment #11) > > Curious, why do you have unmerge-orphans off? I doubt portage does it, but > > FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that > > feature md5 doesn't matter, meaning the prelink reversal is uneeded). > > FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not > during merge since the checksums are needed for recording in CONTENTS. Narf. That said, it shouldn't be invoking prelink on the majority of these files, nor realistically should it be invoking prelink reversal during these cases of merging- if it's a first time merge w/ no conflicts, portage just needs to record the non-prelink checksums, meaning no need to do reversal.
Prelink is disabled by default now: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=7af9d8d0f879370130ffa5fa49de9d9c26ebef78 Enable FEATURES=prelink-checksums to get the old behavior.
(In reply to comment #14) > Prelink is disabled by default now: > Enable FEATURES=prelink-checksums to get the old behavior. This is included in portage-2.2_rc68 and later.
This is fixed in 2.1.9.