Summary: | portage runs prelink on every file merged into live filesystem | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Zeev Tarantov <zeev.tarantov> |
Component: | [OLD] Unspecified | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | esigra |
Priority: | High | Keywords: | InVCS |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 335925 | ||
Attachments: | result of strace -f -c of emerge |
Description
Zeev Tarantov
2010-01-22 22:59:20 UTC
Created attachment 217190 [details]
result of strace -f -c of emerge
Yes. It is a big source package, so it takes a longer time to download. Then it has to unpack, then it has to merge it to the live fs. As you can see, if your I/O is not that great (slow hard drive) then it will take a long time. This is not a fault of sys-apps/portage, nor something that can be "fixed" besides getting faster hard drives. (Or setting up tmpfs, etc). So, resolving as invalid. Not a bug. "it takes a longer time to download" ??? No matter how bad my hard drive is, it just doesn't take 10 MINUTES to unpack the tarball. Here is how long it takes to unpack the tarball and patch the source: $ time tar xjf /usr/portage/distfiles/linux-2.6.32.tar.bz2 real 0m17.326s user 0m15.946s sys 0m1.597s $ cd linux-2.6.32/ $ time bzcat /usr/portage/distfiles/patch-2.6.32.4.bz2 | patch -p1 real 0m0.215s user 0m0.102s sys 0m0.066s The difference between 17 SECONDS of wall clock time and 10 MINUTES of wall clock time is the bug. The strace output shows (for those too lazy to download): % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 45.42 335.784690 10516 31932 635 wait4 28.21 208.520449 6736 30955 6 execve 26.25 194.052701 6204 31281 clone 0.03 0.230745 0 546458 91455 write The full strace log is only 25MB in lrzip, I'll mail it to you or upload it somewhere upon request. Assigning to portage devs, although, I don't expect an action. Zeev, one more thing. You must have something odd going on with your setup. %% time sudo emerge -v vanilla-sources > /dev/null real 1m42.407s user 0m30.964s sys 0m17.421s And that includes the wget time because I didn't have the sources. I'll be glad of any help to find out what's exactly wrong with my setup. Or at least the steps to fix it, even without an explanation (though I'd really like to know). If PORTAGE_TMPDIR is on a different partition than /usr/src/ then you'll have a significant performance penalty since the files have to be copied instead of renamed. You might want to try putting them on the same partition to see how much difference that makes. This is with PORTAGE_TMPDIR="/var/tmp" real 9m4.759s user 1m20.650s sys 7m25.004s untar'ing on the partition where PORTAGE_TMPDIR was, patching there and then mv'ing the whole thing to /usr/src manually only adds seconds, too. The filesystems are EXT4 for root and resiserfs3 for /home (where PORTAGE_TMPDIR was). Does portage do mv or something clever, that might be slow in my setup? My filesystems don't have xattrs, for example. (In reply to comment #8) > Does portage do mv or something clever, that might be slow in my setup? No, it should never call mv. In portage.movefile() there is a place where it will call mv if given a non-regular file, but that's not a typical case. I've given this another go and found out that when portage merges into live filesystem it not only performs md5 on each file but also calls prelink --verify if I have sys-devel/prelink installed. No prelink: /usr/src $ sudo rm -rf linux-2.6.35-rc5/ $ time sudo emerge vanilla-sources > /dev/null real 2m29.873s user 1m14.472s sys 0m30.118s With prelink: /usr/src $ sudo rm -rf linux-2.6.35-rc5/ $ time sudo emerge vanilla-sources > /dev/null real 12m10.732s user 1m44.203s sys 9m29.295s Can portage call prelink only on executables and shared libraries? Or at least not on source and documentation files? (In reply to comment #10) > I've given this another go and found out that when portage merges into live > filesystem it not only performs md5 on each file but also calls prelink > --verify if I have sys-devel/prelink installed. > > No prelink: > /usr/src $ sudo rm -rf linux-2.6.35-rc5/ > $ time sudo emerge vanilla-sources > /dev/null > real 2m29.873s > user 1m14.472s > sys 0m30.118s > > With prelink: > /usr/src $ sudo rm -rf linux-2.6.35-rc5/ > $ time sudo emerge vanilla-sources > /dev/null > real 12m10.732s > user 1m44.203s > sys 9m29.295s > > Can portage call prelink only on executables and shared libraries? Or at least > not on source and documentation files? Curious, why do you have unmerge-orphans off? I doubt portage does it, but FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that feature md5 doesn't matter, meaning the prelink reversal is uneeded). It'll be fun threading that down into perform_checksum I suspect, but it should be done (and portage should grow awareness of what all to actually do prelink checks on). (In reply to comment #11) > Curious, why do you have unmerge-orphans off? I doubt portage does it, but > FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that > feature md5 doesn't matter, meaning the prelink reversal is uneeded). FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not during merge since the checksums are needed for recording in CONTENTS. (In reply to comment #12) > (In reply to comment #11) > > Curious, why do you have unmerge-orphans off? I doubt portage does it, but > > FEATURES=unmerge-orphans should disable the prelink --verify crap (w/ that > > feature md5 doesn't matter, meaning the prelink reversal is uneeded). > > FEATURES=unmerge-orphans causes it to skip checksums during unmerge, but not > during merge since the checksums are needed for recording in CONTENTS. Narf. That said, it shouldn't be invoking prelink on the majority of these files, nor realistically should it be invoking prelink reversal during these cases of merging- if it's a first time merge w/ no conflicts, portage just needs to record the non-prelink checksums, meaning no need to do reversal. Prelink is disabled by default now: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=7af9d8d0f879370130ffa5fa49de9d9c26ebef78 Enable FEATURES=prelink-checksums to get the old behavior. (In reply to comment #14) > Prelink is disabled by default now: > Enable FEATURES=prelink-checksums to get the old behavior. This is included in portage-2.2_rc68 and later. This is fixed in 2.1.9. |