Bug 648432 – sys-apps/portage: File merging is ultra-slow on FreeBSD

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Michał Górny 2018-02-21 18:46:38 UTC]

Let's say I'm installing sys-apps/portage which has over 8000 files. On FreeBSD, merging those files to the live filesystem takes over 10 minutes.

When trying to figure it out via truss, the first thing I notice is that for every file Portage forks and then closes fds from get_open_fds(). Since Portage doesn't have any implementation working for FreeBSD, it ends up closing over 300000 descriptors every time.

This can't be solved via mounting procfs or linprocfs since neither provides /proc/*/fd. FWICS, the BSD interface to getting open file descriptor list may be using libprocstat.

[Comment 1 Mike Gilbert 2018-02-21 19:05:34 UTC]

What does /dev/fd/ look like on FreeBSD?

[Comment 2 Mike Gilbert 2018-02-21 19:08:15 UTC]

Nevermind, that's been thought of before:

> # /dev/fd does not work on FreeBSD, see bug #478446
> if platform.system() in ('FreeBSD',) and _fd_dir == '/dev/fd':
>     _fd_dir = None

[Comment 3 Zac Medico 2018-02-21 19:13:37 UTC]

We could make close_fds default to False for python3.4 and later, since most file descriptors are not inherited by default:

    https://www.python.org/dev/peps/pep-0446/

[Comment 4 Zac Medico 2018-02-21 23:31:32 UTC]

Patch posted for review:

https://archives.gentoo.org/gentoo-portage-dev/message/2360a4c40ab4f89d6a51ebc3f9da8f7c
https://github.com/gentoo/portage/pull/258

[Comment 5 Larry the Git Cow 2018-02-22 17:32:28 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c01fdd27473a76d1c8b6edb1b9dfb2c29645b1c2

commit c01fdd27473a76d1c8b6edb1b9dfb2c29645b1c2
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-02-22 02:44:06 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-02-22 17:30:27 +0000

    emerge/ebuild: sanitize file descriptors on startup
    
    In order to ensure that any unintentionally inherited file descriptors
    will not be inherited by child processes, set the inheritable flag to
    False on startup, except for those corresponding to stdin, stdout, and
    stderr. This mitigates potential problems that might result from
    making the portage.process.spawn close_fds parameter default to False
    for versions of python with PEP 446 support.
    
    Bug: https://bugs.gentoo.org/648432

 bin/ebuild             |  2 ++
 bin/emerge             |  1 +
 pym/portage/process.py | 24 ++++++++++++++++++++++++
 3 files changed, 27 insertions(+)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d3778a92be0ac4a22eb61e3affdc85f99337847a

commit d3778a92be0ac4a22eb61e3affdc85f99337847a
Author:     Zac Medico <[e-mail]>
AuthorDate: 2018-02-21 23:14:44 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2018-02-22 17:30:26 +0000

    portage.process.spawn: default close_fds=False (bug 648432)
    
    For python3.4 and later, default to close_fds=False, since file
    descriptors are non-inheritable by default due to PEP 446. This solves
    a performance problem on systems like FreeBSD, where our get_open_fds
    function returns all possible file descriptor values (including those
    that are not open).
    
    Bug: https://bugs.gentoo.org/648432
    See: https://www.python.org/dev/peps/pep-0446/

 pym/portage/process.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)}

[Comment 6 Zac Medico 2018-02-22 17:35:19 UTC]

We also have a patch to use os.chflags:

https://archives.gentoo.org/gentoo-portage-dev/message/f8f45bf3529dc4ecc3c885a26e2a28f9

[Comment 7 Larry the Git Cow 2018-02-22 19:13:50 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=39c797992bcdf8403521d8b61bb3e592135b3307

commit 39c797992bcdf8403521d8b61bb3e592135b3307
Author:     Michał Górny <[e-mail]>
AuthorDate: 2018-02-22 15:04:52 +0000
Commit:     Michał Górny <[e-mail]>
CommitDate: 2018-02-22 19:13:43 +0000

    FreeBSD: use os.*chflags() instead of calling external tool
    
    Use os.chflags() and os.lchflags() built-in functions instead of calling
    external 'chflags' tool on FreeBSD. This fixes major performance
    problems Portage has on FreeBSD.
    
    Bug: https://bugs.gentoo.org/648432
    Reviewed-by: Zac Medico <[e-mail]>

 pym/portage/__init__.py | 50 +++----------------------------------------------
 1 file changed, 3 insertions(+), 47 deletions(-)}

[Comment 8 Zac Medico 2018-07-02 18:44:35 UTC]

Fixed in portage-2.3.40-r1.