Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 128284 - Merging slowness (1-2 files per second) with newer ~x86 portage
Summary: Merging slowness (1-2 files per second) with newer ~x86 portage
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Interface (emerge) (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
Keywords: InVCS
Depends on:
Blocks: 835380 115839
  Show dependency tree
Reported: 2006-03-31 11:42 UTC by Joël
Modified: 2023-05-20 07:45 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---

saner fd closing example (patch,1017 bytes, patch)
2006-04-01 19:14 UTC, Brian Harring (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joël 2006-03-31 11:42:39 UTC
With newer portage versions (tested up to 2.1_pre7-r3) the merging of files became extremely slow (1-2 files per second) on my machine. Downgrading to x86 portage solves the problem.

For example, a linux kernel takes more than an hour to emerge, on a 2 GHz Turion (1.5 GB RAM) running in 32-bit mode with all CPU power available and zero swapping. Kernel is 2.6.16-archck1.

FEATURES="fixpackages userpriv usersandbox ccache"

Here is part of an strace:

access("/usr/sbin/prelink", X_OK)       = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xa7f546f8) = 12377
waitpid(12377, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0) = 12377
*** here, about 0.5 to 1.0 second elapses ***
--- SIGCHLD (Child exited) @ 0 (0) ---

If a better strace log or more details are needed, please advise...
Comment 1 Joël 2006-03-31 13:26:39 UTC
Ok, I found a workaround.

Long merge times are caused by the following syscalls (as shown by strace):

[pid  9481] close(20402)                = -1 EBADF (Bad file descriptor) <0.000008>

This is done for all possible file descriptors (ie: 0 to 65535), and for each file that's being merged !

Since I have set 65536 file descriptors in /etc/security/limits.conf, this takes about 0.000008 * 65536 = 0.52 seconds per merged file !!

Workaround: override the max files settings by adding the following line at the beginning of limits.conf:

portage          -       nofile          256

Now it's like about... 256 times faster, of course...

Still, on my machine, merging a file (with linux's default 1024-file limit) wastes about 0.000008 * 1024 = about 1/100 second of CPU for each merged file. If a package contains 1000 files, this is 10 seconds of wasted CPU time !
Comment 2 Zac Medico gentoo-dev 2006-03-31 16:45:40 UTC
Thanks for finding this.  Actually, I had previously noticed the massive number of 'Bad file descriptor' exceptions when I was analyzing debug logs generated by the new python-trace feature.  I'm sure there must be some alternative to scanning the whole range like that.
Comment 3 Jason Stubbs (RETIRED) gentoo-dev 2006-03-31 17:49:19 UTC
There'a process being spawned for every file merged?
Comment 4 Joël 2006-04-01 03:33:18 UTC
Concerning my comment #1: to really have the different 'ulimit -n' value, one must relogin from a virtual console, and emerge stuff there.

Doing 'su' is *not* enough, as I noticed.
Comment 5 Zac Medico gentoo-dev 2006-04-01 17:05:36 UTC
*** Bug 128449 has been marked as a duplicate of this bug. ***
Comment 6 Brian Harring (RETIRED) gentoo-dev 2006-04-01 19:14:56 UTC
Created attachment 83671 [details, diff]
saner fd closing example

lifted from pkgcore, but easy to adapt to stable/2.1, this patch relies on /proc/$PID/fd/ listdir results to know what fds to close.

It's linux specific, but it'll reduce the iteration to just what is an open fd in the fork (eg, it's bulletproof unless people change /proc/$PID/fd layout).

Probably won't apply perfectly, but it's dead on for what you need.
Comment 7 Zac Medico gentoo-dev 2006-04-01 22:02:19 UTC
Thanks for the patch! It's in svn r3054.
Comment 8 Brian Harring (RETIRED) gentoo-dev 2006-04-01 23:44:44 UTC
(In reply to comment #3)
> There'a process being spawned for every file merged?

shouldn't be on a sidenote...
From what I know of movefile's code, there shouldn't be any shell calls, just shutil.copyfile (despite it's name, not shell inducing).
Comment 9 Jason Stubbs (RETIRED) gentoo-dev 2006-04-01 23:56:53 UTC
    elif stat.S_ISREG(mymode):
        # we are merging a regular file

For some reason the unprelinked md5 of every file _being merged_ is being calculated. As far as I know, files should never be prelinked during the compile/install phases so the "am-i-prelinked?" check seems pointless here.
Comment 10 Zac Medico gentoo-dev 2006-04-02 00:49:50 UTC
In most cases, the prelink check is unneeded.  However, tbz2 packages generated by quickpkg might contain prelinked binaries.
Comment 11 Brian Harring (RETIRED) gentoo-dev 2006-04-02 01:02:41 UTC
(In reply to comment #10)
> In most cases, the prelink check is unneeded.  However, tbz2 packages generated
> by quickpkg might contain prelinked binaries.

Sounds like quickpkg ought to be generating unprelinked binpkgs instead of dumping the cost on all merges...
Comment 12 Zac Medico gentoo-dev 2006-04-02 23:53:59 UTC
Released in 2.1_pre7-r4.
Comment 13 Joël 2006-04-03 03:34:36 UTC
I confirm it works great :-)