Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 663324 - sys-apps/portage: emerge --jobs triggers race conditions in dev-python/setuptools when installing packages like dev-python/setuptools-scm, dev-python/setuptools-git, dev-python/pbr
Summary: sys-apps/portage: emerge --jobs triggers race conditions in dev-python/setupt...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 921380
Blocks: 184128
  Show dependency tree
 
Reported: 2018-08-10 19:44 UTC by Zac Medico
Modified: 2024-06-16 03:51 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2018-08-10 19:44:03 UTC
I've seen this race triggered when dev-python/setuptools-git or dev-python/pbr is merged while a package is being built with setuptools:

> Traceback (most recent call last):
>   File "setup.py", line 114, in <module>
>     include_package_data=False,
>   File "/usr/lib64/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
>     return distutils.core.setup(**attrs)
>   File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
>     dist.run_commands()
>   File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
>     self.run_command(cmd)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/install.py", line 61, in run
>     return orig.install.run(self)
>   File "/usr/lib64/python3.6/distutils/command/install.py", line 557, in run
>     self.run_command(cmd_name)
>   File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/install_egg_info.py", line 34, in run
>     self.run_command('egg_info')
>   File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/egg_info.py", line 272, in run
>     writer = ep.resolve()
>   File "/usr/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 2428, in resolve
>     module = __import__(self.module_name, fromlist=['__name__'], level=0)
> ModuleNotFoundError: No module named 'pbr'
> 
> Traceback (most recent call last):
>   File "setup.py", line 142, in <module>
>     keywords=['interface', 'components', 'plugins'],
>   File "/usr/lib64/python3.4/site-packages/setuptools/__init__.py", line 129, in setup
>     return distutils.core.setup(**attrs)
>   File "/usr/lib64/python3.4/distutils/core.py", line 148, in setup
>     dist.run_commands()
>   File "/usr/lib64/python3.4/distutils/dist.py", line 955, in run_commands
>     self.run_command(cmd)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/install.py", line 61, in run
>     return orig.install.run(self)
>   File "/usr/lib64/python3.4/distutils/command/install.py", line 539, in run
>     self.run_command('build')
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/distutils/command/build.py", line 126, in run
>     self.run_command(cmd_name)
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 53, in run
>     self.build_package_data()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 118, in build_package_data
>     for package, src_dir, build_dir, filenames in self.data_files:
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 66, in __getattr__
>     self.data_files = self._get_data_files()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 82, in _get_data_files
>     self.analyze_manifest()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 138, in analyze_manifest
>     self.run_command('egg_info')
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 280, in run
>     self.find_sources()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 295, in find_sources
>     mm.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 526, in run
>     self.add_defaults()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 565, in add_defaults
>     rcfiles = list(walk_revctrl())
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl
>     for item in ep.load()(dirname):
>   File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2422, in load
>     return self.resolve()
>   File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2428, in resolve
>     module = __import__(self.module_name, fromlist=['__name__'], level=0)
> ImportError: No module named 'setuptools_git'
Comment 1 Zac Medico gentoo-dev 2018-08-10 19:58:22 UTC
A potential way to model this relationship would be to introduce a new kind of blocker that indicates that merging a package will temporarily interfere with the operation of another package. In the case described in comment #0, merging dev-python/setuptools-git or dev-python/pbr temporarily interferes with the operation of dev-python/setuptools.
Comment 2 Zac Medico gentoo-dev 2018-08-10 23:53:56 UTC
A "more atomic" package merge approach would minimize interference with setuptools, for example, these libraries use directory renames when possible so that the new package files are exposed as atomically as possible:

https://github.com/mgorny/atomic-install
https://github.com/mgorny/atomic-install-py
Comment 3 Zac Medico gentoo-dev 2018-08-11 00:12:29 UTC
Also, it might be possible to make setuptools do a retry loop, using timestamp comparisons to detect file/directory changes that have occurred during the last few seconds.
Comment 4 Zac Medico gentoo-dev 2018-08-12 05:14:08 UTC
If the package manager has access to snapshot capabilities for the root filesystem (bug 40127), it can run each build in a chroot environment with an immutable snapshot of the root filesystem, effectively implementing read-copy-update (RCU) for the root filesystem.

I imagine that many libraries with plugin systems might be vulnerable to the sort of race condition that setuptools / pkg_resources has. It seems like it's practically impossible to handle this sort of thing at the individual library level, since there could be a vast number of consumers of the library at any given time, so there might never be a safe time to install a plugin on a running system.
Comment 5 Larry the Git Cow gentoo-dev 2023-09-01 00:56:21 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b3e9f8fd458a77e067ccb50f8a7f0cd29587c18c

commit b3e9f8fd458a77e067ccb50f8a7f0cd29587c18c
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-09-01 00:55:20 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-09-01 00:55:20 +0000

    dev-python/setuptools: restore comment + add bug ref for setuptools_scm PDEPEND
    
    See e96c1d113f47ac076ee71b951afebe3515850918.
    
    Eli noticed that the comment had got lost here & it wasn't particularly
    descriptive, so restore the comment and add a bug reference.
    
    Bug: https://bugs.gentoo.org/663324
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-python/setuptools/setuptools-68.0.0-r1.ebuild | 2 ++
 dev-python/setuptools/setuptools-68.1.2.ebuild    | 2 ++
 2 files changed, 4 insertions(+)
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-09-01 01:02:05 UTC
(In reply to Zac Medico from comment #1)
> A potential way to model this relationship would be to introduce a new kind
> of blocker that indicates that merging a package will temporarily interfere
> with the operation of another package. In the case described in comment #0,
> merging dev-python/setuptools-git or dev-python/pbr temporarily interferes
> with the operation of dev-python/setuptools.

See also:
* bug 906809 (bit different, but related)
* Exherbo's https://www.exherbolinux.org/docs/eapi/exheres-for-smarties.html#package_dependencies and https://www.exherbolinux.org/docs/eapi/exheres-for-smarties.html#annotations which has 

"""
resolution, for blockers only:
    manual
    uninstall-blocked-after suggests uninstalling the blocked package after we are merged (you’re allowed to file collide if you do this).
    uninstall-blocked-before suggests uninstalling the blocked package before we are merged (only use if you really really have to, and most definitely not just for collisions).
    upgrade-blocked-before suggests upgrading the blocked package before we are merged
"""
Comment 7 Eli Schwartz gentoo-dev 2023-09-01 01:34:51 UTC
This is a bit of a funny issue and an unpleasant solution to have to feel compelled into using. I think it deserves a more in depth rethink.

In general, (deleting one version of a package and/or) installing a package is an unsafe time of transition. Any program started in the interim period may manifest random issues, but the best that all us package managers can hope for is that the user won't start new programs while running installations in another shell, or cross our fingers and hope the timing works out.

The problem here is just another example of that, but one that portage specifically is prone to triggering by use of "good" timing, since it starts a bunch of programs around the same time.

I think the intuitive solution here is that portage should be able to merge images in parallel, and it should be able to perform compiles in parallel, but it shouldn't try to merge an image in parallel with doing a compile -- this is inherently unsafe.

Portage could/should add locking to pause one package after creating the image but before merging it, if another compile is running. This sacrifices a bit of parallelism but IMHO people will be thankful for that.
Comment 8 Zac Medico gentoo-dev 2024-01-14 22:34:52 UTC
(In reply to Eli Schwartz from comment #7)
> Portage could/should add locking to pause one package after creating the
> image but before merging it, if another compile is running. This sacrifices
> a bit of parallelism but IMHO people will be thankful for that.

Yeah, we have a _merge_wait_queue that implements similar behavior to handle system packages since bug 256616: 

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e589ece0600c449238d70f543eaba2f32a93ef5f

commit e589ece0600c449238d70f543eaba2f32a93ef5f
Author: Zac Medico <zmedico@gentoo.org>
Date:   2009-02-02 06:11:06 +0000

    Bug #256616 - Since dependencies on system packages are frequently unspecified,
    merge them only when no builds are executing. When a system package finishes
    building, it's added to a wait queue that is only processed when the number
    of running builds drops to zero. All pending merges are then processed before
    any new builds are allowed to start.
    
    svn path=/main/trunk/; revision=12569
Comment 9 Eli Schwartz gentoo-dev 2024-01-16 04:13:44 UTC
That bug has a commenter making a bit of a stink about always using binpkgs, which I think is unnecessary. But I do think that using the system set for this may have been optimistic. :)
Comment 10 Larry the Git Cow gentoo-dev 2024-01-16 05:21:10 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=825db01b91a37dcd9890ee5bf9f462ea524ac5cc

commit 825db01b91a37dcd9890ee5bf9f462ea524ac5cc
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-01-16 05:06:36 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-01-16 05:20:37 +0000

    Add merge-wait FEATURES setting enabled by default
    
    Add a new merge-wait FEATURES setting, enabled by default,
    which prevents packages from being merged while other packages
    are building. Previously, this behavior was already implemented
    for packages that satisfy direct or indirect dependencies of
    the system set. This feature can be disabled only for other
    packages which do not satisfy direct or indirect dependencies
    of the system set, in order to trade the possibility of random
    build failures for greater parallelism.
    
    Currently, it is known that having merge-wait disabled can
    cause "random" build failures for builds using setuptools when
    setuptools plugins are merged during the build.
    
    Bug: https://bugs.gentoo.org/663324
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 cnf/make.globals         |  4 ++--
 lib/_emerge/Scheduler.py | 14 ++++++++------
 lib/portage/const.py     |  3 ++-
 man/make.conf.5          | 16 +++++++++++++---
 4 files changed, 25 insertions(+), 12 deletions(-)
Comment 11 Larry the Git Cow gentoo-dev 2024-02-22 07:24:05 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77c44c46194922509bc4f2b5cfc099412a560a69

commit 77c44c46194922509bc4f2b5cfc099412a560a69
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-02-22 07:23:40 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-22 07:23:50 +0000

    sys-apps/portage: add 3.0.62
    
    Closes: https://bugs.gentoo.org/663324
    Closes: https://bugs.gentoo.org/728046
    Closes: https://bugs.gentoo.org/891137
    Closes: https://bugs.gentoo.org/906368
    Closes: https://bugs.gentoo.org/916566
    Closes: https://bugs.gentoo.org/921170
    Closes: https://bugs.gentoo.org/921208
    Closes: https://bugs.gentoo.org/921400
    Closes: https://bugs.gentoo.org/922038
    Closes: https://bugs.gentoo.org/922142
    Closes: https://bugs.gentoo.org/923368
    Closes: https://bugs.gentoo.org/923750
    Closes: https://bugs.gentoo.org/923841
    Closes: https://bugs.gentoo.org/923852
    Closes: https://bugs.gentoo.org/923854
    Closes: https://bugs.gentoo.org/924192
    Closes: https://bugs.gentoo.org/924273
    Closes: https://bugs.gentoo.org/924585
    Closes: https://bugs.gentoo.org/921380
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.62.ebuild | 246 +++++++++++++++++++++++++++++++++
 2 files changed, 247 insertions(+)