Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 648432 - sys-apps/portage: File merging is ultra-slow on FreeBSD
Summary: sys-apps/portage: File merging is ultra-slow on FreeBSD
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 651804
  Show dependency tree
 
Reported: 2018-02-21 18:46 UTC by Michał Górny
Modified: 2018-07-02 18:44 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2018-02-21 18:46:38 UTC
Let's say I'm installing sys-apps/portage which has over 8000 files. On FreeBSD, merging those files to the live filesystem takes over 10 minutes.

When trying to figure it out via truss, the first thing I notice is that for every file Portage forks and then closes fds from get_open_fds(). Since Portage doesn't have any implementation working for FreeBSD, it ends up closing over 300000 descriptors every time.

This can't be solved via mounting procfs or linprocfs since neither provides /proc/*/fd. FWICS, the BSD interface to getting open file descriptor list may be using libprocstat.
Comment 1 Mike Gilbert gentoo-dev 2018-02-21 19:05:34 UTC
What does /dev/fd/ look like on FreeBSD?
Comment 2 Mike Gilbert gentoo-dev 2018-02-21 19:08:15 UTC
Nevermind, that's been thought of before:

> # /dev/fd does not work on FreeBSD, see bug #478446
> if platform.system() in ('FreeBSD',) and _fd_dir == '/dev/fd':
>     _fd_dir = None
Comment 3 Zac Medico gentoo-dev 2018-02-21 19:13:37 UTC
We could make close_fds default to False for python3.4 and later, since most file descriptors are not inherited by default:

    https://www.python.org/dev/peps/pep-0446/
Comment 5 Larry the Git Cow gentoo-dev 2018-02-22 17:32:28 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c01fdd27473a76d1c8b6edb1b9dfb2c29645b1c2

commit c01fdd27473a76d1c8b6edb1b9dfb2c29645b1c2
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-02-22 02:44:06 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-02-22 17:30:27 +0000

    emerge/ebuild: sanitize file descriptors on startup
    
    In order to ensure that any unintentionally inherited file descriptors
    will not be inherited by child processes, set the inheritable flag to
    False on startup, except for those corresponding to stdin, stdout, and
    stderr. This mitigates potential problems that might result from
    making the portage.process.spawn close_fds parameter default to False
    for versions of python with PEP 446 support.
    
    Bug: https://bugs.gentoo.org/648432

 bin/ebuild             |  2 ++
 bin/emerge             |  1 +
 pym/portage/process.py | 24 ++++++++++++++++++++++++
 3 files changed, 27 insertions(+)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d3778a92be0ac4a22eb61e3affdc85f99337847a

commit d3778a92be0ac4a22eb61e3affdc85f99337847a
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2018-02-21 23:14:44 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2018-02-22 17:30:26 +0000

    portage.process.spawn: default close_fds=False (bug 648432)
    
    For python3.4 and later, default to close_fds=False, since file
    descriptors are non-inheritable by default due to PEP 446. This solves
    a performance problem on systems like FreeBSD, where our get_open_fds
    function returns all possible file descriptor values (including those
    that are not open).
    
    Bug: https://bugs.gentoo.org/648432
    See: https://www.python.org/dev/peps/pep-0446/

 pym/portage/process.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)}
Comment 6 Zac Medico gentoo-dev 2018-02-22 17:35:19 UTC
We also have a patch to use os.chflags:

https://archives.gentoo.org/gentoo-portage-dev/message/f8f45bf3529dc4ecc3c885a26e2a28f9
Comment 7 Larry the Git Cow gentoo-dev 2018-02-22 19:13:50 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=39c797992bcdf8403521d8b61bb3e592135b3307

commit 39c797992bcdf8403521d8b61bb3e592135b3307
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2018-02-22 15:04:52 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2018-02-22 19:13:43 +0000

    FreeBSD: use os.*chflags() instead of calling external tool
    
    Use os.chflags() and os.lchflags() built-in functions instead of calling
    external 'chflags' tool on FreeBSD. This fixes major performance
    problems Portage has on FreeBSD.
    
    Bug: https://bugs.gentoo.org/648432
    Reviewed-by: Zac Medico <zmedico@gentoo.org>

 pym/portage/__init__.py | 50 +++----------------------------------------------
 1 file changed, 3 insertions(+), 47 deletions(-)}
Comment 8 Zac Medico gentoo-dev 2018-07-02 18:44:35 UTC
Fixed in portage-2.3.40-r1.