Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 301876

Summary: portage runs prelink on every file merged into live filesystem
Product: Gentoo Linux Reporter: Zeev Tarantov <zeev.tarantov>
Component: [OLD] UnspecifiedAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: esigra
Priority: High Keywords: InVCS
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 335925    
Attachments: result of strace -f -c of emerge

Description Zeev Tarantov 2010-01-22 22:59:20 UTC
$ time sudo FEATURES="-collision-protection -protect-owned" emerge -v sys-kernel/vanilla-sources > /dev/null

real    10m18.040s
user    1m37.181s
sys     7m38.159s


This is perf top while it’s doing that:

------------------------------------------------------------------------------
   PerfTop:   23157 irqs/sec  kernel:81.0% [100000 cycles],  (all, 2 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

           100055.00 - 34.7% : copy_page_range
            70068.00 - 24.3% : unmap_vmas
            18438.00 -  6.4% : page_remove_rmap
            14164.00 -  4.9% : release_pages
             7897.00 -  2.7% : copy_page_c
             7605.00 -  2.6% : page_fault
             6090.00 -  2.1% : vm_normal_page
             3969.00 -  1.4% : free_pages_and_swap_cache
             2460.00 -  0.9% : _raw_spin_lock
             2171.00 -  0.8% : clear_page_c
             2143.00 -  0.7% : schedule
             2018.00 -  0.7% : _raw_spin_lock_irqsave
             1983.00 -  0.7% : flush_tlb_page
             1870.00 -  0.6% : do_wp_page
             1510.00 -  0.5% : __wake_up_bit



Reproducible: Always

Steps to Reproduce:
1. emerge the kernel sources
2. wait a long time
3.

Actual Results:  
I have output of "strace -f" of that install, it's 850MB.
I attach the output of "strace -f -c", maybe that's useful.


Portage 2.1.7.16 (default/linux/amd64/10.0/desktop, gcc-4.4.2, glibc-2.11-r1, 2.6.33-rc5 x86_64)
=================================================================
System uname: Linux-2.6.33-rc5-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E6550_@_2.33GHz-with-gentoo-2.0.1
Timestamp of tree: Fri, 22 Jan 2010 10:15:01 +0000
app-shells/bash:     4.0_p37
dev-java/java-config: 2.1.10
dev-lang/python:     2.6.4, 3.1.1-r1
dev-util/cmake:      2.8.0
sys-apps/baselayout: 2.0.1
sys-apps/openrc:     0.6.0-r1
sys-apps/sandbox:    2.2
sys-devel/autoconf:  2.13, 2.65
sys-devel/automake:  1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.20
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-g -O2 -pipe -march=core2 -momit-leaf-frame-pointer"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-g -O2 -pipe -march=core2 -momit-leaf-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests distlocks fixpackages installsources news parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://mirror.isoc.org.il/pub/gentoo/ "
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/home/system/var_tmp_portage"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/gcc-porting /var/lib/layman/mozilla /var/lib/layman/java-overlay /var/lib/layman/toolchain /usr/local/portage"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa amd64 bash-completion bidi bluetooth branding bzip2 cairo cdr cli cracklib crypt cxx dbus dri dts dvd dvdr eds emboss encode evo fam firefox flac gif gpm gstreamer gtk hal iconv ipv6 jpeg kde libnotify lzma mad mikmod mmx mng modules mp3 mp4 mpeg mudflap multilib ncurses nls nptl nptlonly ogg opengl openmp pch pcre pdf perl png ppds pppd python qt3support qt4 quicktime readline reflection sdl session spell spl sqlite3 sse sse2 ssl startup-notification svg sysfs tcpd thunar tiff truetype unicode usb vorbis x264 xcb xcomposite xml xorg xulrunner xv xvid zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="stv0680" ELIBC="glibc" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" QEMU_SOFTMMU_TARGETS="arm i386 ppc ppc64 sparc x86_64" QEMU_USER_TARGETS="i386 x86_64" RUBY_TARGETS="ruby18" SANE_BACKENDS="niash stv680" USERLAND="GNU" VIDEO_CARDS="nv nvidia nouveau"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Zeev Tarantov 2010-01-22 22:59:59 UTC
Created attachment 217190 [details]
result of strace -f -c of emerge
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-01-22 23:15:48 UTC
Yes. It is a big source package, so it takes a longer time to download. Then it has to unpack, then it has to merge it to the live fs. As you can see, if your I/O is not that great (slow hard drive) then it will take a long time.

This is not a fault of sys-apps/portage, nor something that can be "fixed" besides getting faster hard drives. (Or setting up tmpfs, etc). So, resolving as invalid. Not a bug.
Comment 3 Zeev Tarantov 2010-01-22 23:31:47 UTC
"it takes a longer time to download" ???

No matter how bad my hard drive is, it just doesn't take 10 MINUTES to unpack the tarball. Here is how long it takes to unpack the tarball and patch the source:

$ time tar xjf /usr/portage/distfiles/linux-2.6.32.tar.bz2

real 0m17.326s
user 0m15.946s
sys 0m1.597s

$ cd linux-2.6.32/
$ time bzcat /usr/portage/distfiles/patch-2.6.32.4.bz2 | patch -p1
real 0m0.215s
user 0m0.102s
sys 0m0.066s

The difference between 17 SECONDS of wall clock time and 10 MINUTES of wall clock time is the bug. The strace output shows (for those too lazy to download):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 45.42  335.784690       10516     31932       635 wait4
 28.21  208.520449        6736     30955         6 execve
 26.25  194.052701        6204     31281           clone
  0.03    0.230745           0    546458     91455 write

The full strace log is only 25MB in lrzip, I'll mail it to you or upload it somewhere upon request.
Comment 4 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-01-22 23:50:17 UTC
Assigning to portage devs, although, I don't expect an action.
Comment 5 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-01-22 23:55:54 UTC
Zeev, one more thing. You must have something odd going on with your setup.

%% time sudo emerge -v vanilla-sources > /dev/null
real    1m42.407s
user    0m30.964s
sys     0m17.421s

And that includes the wget time because I didn't have the sources.
Comment 6 Zeev Tarantov 2010-01-23 00:07:05 UTC
I'll be glad of any help to find out what's exactly wrong with my setup. Or at least the steps to fix it, even without an explanation (though I'd really like to know).
Comment 7 Zac Medico gentoo-dev 2010-01-23 01:15:20 UTC
If PORTAGE_TMPDIR is on a different partition than /usr/src/ then you'll have a significant performance penalty since the files have to be copied instead of renamed. You might want to try putting them on the same partition to see how much difference that makes.
Comment 8 Zeev Tarantov 2010-01-23 02:46:03 UTC
This is with PORTAGE_TMPDIR="/var/tmp"

real    9m4.759s
user    1m20.650s
sys     7m25.004s

untar'ing on the partition where PORTAGE_TMPDIR was, patching there and then mv'ing the whole thing to /usr/src manually only adds seconds, too. The filesystems are EXT4 for root and resiserfs3 for /home (where PORTAGE_TMPDIR was). Does portage do mv or something clever, that might be slow in my setup? My filesystems don't have xattrs, for example.
Comment 9 Zac Medico gentoo-dev 2010-01-23 06:58:44 UTC
(In reply to comment #8)
> Does portage do mv or something clever, that might be slow in my setup?

No, it should never call mv. In portage.movefile() there is a place where it will call mv if given a non-regular file, but that's not a typical case.
Comment 10 Zeev Tarantov 2010-07-13 18:27:44 UTC
I've given this another go and found out that when portage merges into live filesystem it not only performs md5 on each file but also calls prelink --verify if I have sys-devel/prelink installed.

No prelink:
/usr/src $ sudo rm -rf linux-2.6.35-rc5/
$ time sudo emerge vanilla-sources > /dev/null
real    2m29.873s
user    1m14.472s
sys     0m30.118s

With prelink:
/usr/src $ sudo rm -rf linux-2.6.35-rc5/
$ time sudo emerge vanilla-sources > /dev/null
real    12m10.732s
user    1m44.203s
sys     9m29.295s

Can portage call prelink only on executables and shared libraries? Or at least not on source and documentation files?
Comment 11 Brian Harring (RETIRED) gentoo-dev 2010-07-14 10:51:32 UTC
(In reply to comment #10)
> I've given this another go and found out that when portage merges into live
> filesystem it not only performs md5 on each file but also calls prelink
> --verify if I have sys-devel/prelink installed.
> 
> No prelink:
> /usr/src $ sudo rm -rf linux-2.6.35-rc5/
> $ time sudo emerge vanilla-sources > /dev/null
> real    2m29.873s
> user    1m14.472s
> sys     0m30.118s
> 
> With prelink:
> /usr/src $ sudo rm -rf linux-2.6.35-rc5/
> $ time sudo emerge vanilla-sources > /dev/null
> real    12m10.732s
> user    1m44.203s
> sys     9m29.295s
> 
> Can portage call prelink only on executables and shared libraries? Or at least
> not on source and documentation files?

Curious, why do you have unmerge-orphans off?  I doubt portage does it, but FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that feature md5 doesn't matter, meaning the prelink reversal is uneeded).  It'll be fun threading that down into perform_checksum I suspect, but it should be done (and portage should grow awareness of what all to actually do prelink checks on).
Comment 12 Zac Medico gentoo-dev 2010-07-14 13:34:25 UTC
(In reply to comment #11)
> Curious, why do you have unmerge-orphans off?  I doubt portage does it, but
> FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that
> feature md5 doesn't matter, meaning the prelink reversal is uneeded).

FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not during merge since the checksums are needed for recording in CONTENTS.
Comment 13 Brian Harring (RETIRED) gentoo-dev 2010-07-14 17:28:33 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > Curious, why do you have unmerge-orphans off?  I doubt portage does it, but
> > FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that
> > feature md5 doesn't matter, meaning the prelink reversal is uneeded).
> 
> FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not
> during merge since the checksums are needed for recording in CONTENTS.

Narf.  That said, it shouldn't be invoking prelink on the majority of these files, nor realistically should it be invoking prelink reversal during these cases of merging- if it's a first time merge w/ no conflicts, portage just needs to record the non-prelink checksums, meaning no need to do reversal.

Comment 14 Zac Medico gentoo-dev 2010-09-03 16:40:42 UTC
Prelink is disabled by default now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=7af9d8d0f879370130ffa5fa49de9d9c26ebef78

Enable FEATURES=prelink-checksums to get the old behavior.
Comment 15 Zac Medico gentoo-dev 2010-09-03 16:43:40 UTC
(In reply to comment #14)
> Prelink is disabled by default now:
> Enable FEATURES=prelink-checksums to get the old behavior.

This is included in portage-2.2_rc68 and later.
Comment 16 Zac Medico gentoo-dev 2010-09-04 08:17:12 UTC
This is fixed in 2.1.9.