Summary: | Merging slowness (1-2 files per second) with newer ~x86 portage | ||
---|---|---|---|
Product: | Portage Development | Reporter: | Joël <world.root> |
Component: | Core - Interface (emerge) | Assignee: | Portage team <dev-portage> |
Status: | RESOLVED FIXED | ||
Severity: | normal | Keywords: | InVCS |
Priority: | High | ||
Version: | 2.1 | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 835380, 115839 | ||
Attachments: | saner fd closing example |
Description
Joël
2006-03-31 11:42:39 UTC
Ok, I found a workaround. Long merge times are caused by the following syscalls (as shown by strace): [pid 9481] close(20402) = -1 EBADF (Bad file descriptor) <0.000008> This is done for all possible file descriptors (ie: 0 to 65535), and for each file that's being merged ! Since I have set 65536 file descriptors in /etc/security/limits.conf, this takes about 0.000008 * 65536 = 0.52 seconds per merged file !! Workaround: override the max files settings by adding the following line at the beginning of limits.conf: portage - nofile 256 Now it's like about... 256 times faster, of course... Still, on my machine, merging a file (with linux's default 1024-file limit) wastes about 0.000008 * 1024 = about 1/100 second of CPU for each merged file. If a package contains 1000 files, this is 10 seconds of wasted CPU time ! Thanks for finding this. Actually, I had previously noticed the massive number of 'Bad file descriptor' exceptions when I was analyzing debug logs generated by the new python-trace feature. I'm sure there must be some alternative to scanning the whole range like that. There'a process being spawned for every file merged? Concerning my comment #1: to really have the different 'ulimit -n' value, one must relogin from a virtual console, and emerge stuff there. Doing 'su' is *not* enough, as I noticed. *** Bug 128449 has been marked as a duplicate of this bug. *** Created attachment 83671 [details, diff]
saner fd closing example
lifted from pkgcore, but easy to adapt to stable/2.1, this patch relies on /proc/$PID/fd/ listdir results to know what fds to close.
It's linux specific, but it'll reduce the iteration to just what is an open fd in the fork (eg, it's bulletproof unless people change /proc/$PID/fd layout).
Probably won't apply perfectly, but it's dead on for what you need.
Thanks for the patch! It's in svn r3054. (In reply to comment #3) > There'a process being spawned for every file merged? shouldn't be on a sidenote... From what I know of movefile's code, there shouldn't be any shell calls, just shutil.copyfile (despite it's name, not shell inducing). elif stat.S_ISREG(mymode): # we are merging a regular file mymd5=portage_checksum.perform_md5(mysrc,calc_prelink=1) For some reason the unprelinked md5 of every file _being merged_ is being calculated. As far as I know, files should never be prelinked during the compile/install phases so the "am-i-prelinked?" check seems pointless here. In most cases, the prelink check is unneeded. However, tbz2 packages generated by quickpkg might contain prelinked binaries. (In reply to comment #10) > In most cases, the prelink check is unneeded. However, tbz2 packages generated > by quickpkg might contain prelinked binaries. Sounds like quickpkg ought to be generating unprelinked binpkgs instead of dumping the cost on all merges... Released in 2.1_pre7-r4. I confirm it works great :-) |