With newer portage versions (tested up to 2.1_pre7-r3) the merging of files became extremely slow (1-2 files per second) on my machine. Downgrading to x86 portage solves the problem. For example, a linux kernel takes more than an hour to emerge, on a 2 GHz Turion (1.5 GB RAM) running in 32-bit mode with all CPU power available and zero swapping. Kernel is 2.6.16-archck1. FEATURES="fixpackages userpriv usersandbox ccache" Here is part of an strace: ========================================================== access("/usr/sbin/prelink", X_OK) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xa7f546f8) = 12377 waitpid(12377, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0) = 12377 *** here, about 0.5 to 1.0 second elapses *** --- SIGCHLD (Child exited) @ 0 (0) --- ========================================================== If a better strace log or more details are needed, please advise...
Ok, I found a workaround. Long merge times are caused by the following syscalls (as shown by strace): [pid 9481] close(20402) = -1 EBADF (Bad file descriptor) <0.000008> This is done for all possible file descriptors (ie: 0 to 65535), and for each file that's being merged ! Since I have set 65536 file descriptors in /etc/security/limits.conf, this takes about 0.000008 * 65536 = 0.52 seconds per merged file !! Workaround: override the max files settings by adding the following line at the beginning of limits.conf: portage - nofile 256 Now it's like about... 256 times faster, of course... Still, on my machine, merging a file (with linux's default 1024-file limit) wastes about 0.000008 * 1024 = about 1/100 second of CPU for each merged file. If a package contains 1000 files, this is 10 seconds of wasted CPU time !
Thanks for finding this. Actually, I had previously noticed the massive number of 'Bad file descriptor' exceptions when I was analyzing debug logs generated by the new python-trace feature. I'm sure there must be some alternative to scanning the whole range like that.
There'a process being spawned for every file merged?
Concerning my comment #1: to really have the different 'ulimit -n' value, one must relogin from a virtual console, and emerge stuff there. Doing 'su' is *not* enough, as I noticed.
*** Bug 128449 has been marked as a duplicate of this bug. ***
Created attachment 83671 [details, diff] saner fd closing example lifted from pkgcore, but easy to adapt to stable/2.1, this patch relies on /proc/$PID/fd/ listdir results to know what fds to close. It's linux specific, but it'll reduce the iteration to just what is an open fd in the fork (eg, it's bulletproof unless people change /proc/$PID/fd layout). Probably won't apply perfectly, but it's dead on for what you need.
Thanks for the patch! It's in svn r3054.
(In reply to comment #3) > There'a process being spawned for every file merged? shouldn't be on a sidenote... From what I know of movefile's code, there shouldn't be any shell calls, just shutil.copyfile (despite it's name, not shell inducing).
elif stat.S_ISREG(mymode): # we are merging a regular file mymd5=portage_checksum.perform_md5(mysrc,calc_prelink=1) For some reason the unprelinked md5 of every file _being merged_ is being calculated. As far as I know, files should never be prelinked during the compile/install phases so the "am-i-prelinked?" check seems pointless here.
In most cases, the prelink check is unneeded. However, tbz2 packages generated by quickpkg might contain prelinked binaries.
(In reply to comment #10) > In most cases, the prelink check is unneeded. However, tbz2 packages generated > by quickpkg might contain prelinked binaries. Sounds like quickpkg ought to be generating unprelinked binpkgs instead of dumping the cost on all merges...
Released in 2.1_pre7-r4.
I confirm it works great :-)
