Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 648432 - sys-apps/portage: File merging is ultra-slow on FreeBSD
Summary: sys-apps/portage: File merging is ultra-slow on FreeBSD
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
Keywords: InVCS
Depends on:
Blocks: 835380 651804
  Show dependency tree
Reported: 2018-02-21 18:46 UTC by Michał Górny
Modified: 2023-05-20 07:42 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2018-02-21 18:46:38 UTC
Let's say I'm installing sys-apps/portage which has over 8000 files. On FreeBSD, merging those files to the live filesystem takes over 10 minutes.

When trying to figure it out via truss, the first thing I notice is that for every file Portage forks and then closes fds from get_open_fds(). Since Portage doesn't have any implementation working for FreeBSD, it ends up closing over 300000 descriptors every time.

This can't be solved via mounting procfs or linprocfs since neither provides /proc/*/fd. FWICS, the BSD interface to getting open file descriptor list may be using libprocstat.
Comment 1 Mike Gilbert gentoo-dev 2018-02-21 19:05:34 UTC
What does /dev/fd/ look like on FreeBSD?
Comment 2 Mike Gilbert gentoo-dev 2018-02-21 19:08:15 UTC
Nevermind, that's been thought of before:

> # /dev/fd does not work on FreeBSD, see bug #478446
> if platform.system() in ('FreeBSD',) and _fd_dir == '/dev/fd':
>     _fd_dir = None
Comment 3 Zac Medico gentoo-dev 2018-02-21 19:13:37 UTC
We could make close_fds default to False for python3.4 and later, since most file descriptors are not inherited by default:
Comment 5 Larry the Git Cow gentoo-dev 2018-02-22 17:32:28 UTC
The bug has been referenced in the following commit(s):

commit c01fdd27473a76d1c8b6edb1b9dfb2c29645b1c2
Author:     Zac Medico <>
AuthorDate: 2018-02-22 02:44:06 +0000
Commit:     Zac Medico <>
CommitDate: 2018-02-22 17:30:27 +0000

    emerge/ebuild: sanitize file descriptors on startup
    In order to ensure that any unintentionally inherited file descriptors
    will not be inherited by child processes, set the inheritable flag to
    False on startup, except for those corresponding to stdin, stdout, and
    stderr. This mitigates potential problems that might result from
    making the portage.process.spawn close_fds parameter default to False
    for versions of python with PEP 446 support.

 bin/ebuild             |  2 ++
 bin/emerge             |  1 +
 pym/portage/ | 24 ++++++++++++++++++++++++
 3 files changed, 27 insertions(+)

commit d3778a92be0ac4a22eb61e3affdc85f99337847a
Author:     Zac Medico <>
AuthorDate: 2018-02-21 23:14:44 +0000
Commit:     Zac Medico <>
CommitDate: 2018-02-22 17:30:26 +0000

    portage.process.spawn: default close_fds=False (bug 648432)
    For python3.4 and later, default to close_fds=False, since file
    descriptors are non-inheritable by default due to PEP 446. This solves
    a performance problem on systems like FreeBSD, where our get_open_fds
    function returns all possible file descriptor values (including those
    that are not open).

 pym/portage/ | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)}
Comment 6 Zac Medico gentoo-dev 2018-02-22 17:35:19 UTC
We also have a patch to use os.chflags:
Comment 7 Larry the Git Cow gentoo-dev 2018-02-22 19:13:50 UTC
The bug has been referenced in the following commit(s):

commit 39c797992bcdf8403521d8b61bb3e592135b3307
Author:     Michał Górny <>
AuthorDate: 2018-02-22 15:04:52 +0000
Commit:     Michał Górny <>
CommitDate: 2018-02-22 19:13:43 +0000

    FreeBSD: use os.*chflags() instead of calling external tool
    Use os.chflags() and os.lchflags() built-in functions instead of calling
    external 'chflags' tool on FreeBSD. This fixes major performance
    problems Portage has on FreeBSD.
    Reviewed-by: Zac Medico <>

 pym/portage/ | 50 +++----------------------------------------------
 1 file changed, 3 insertions(+), 47 deletions(-)}
Comment 8 Zac Medico gentoo-dev 2018-07-02 18:44:35 UTC
Fixed in portage-2.3.40-r1.