Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 663324 - sys-apps/portage: emerge --jobs triggers race conditions in dev-python/setuptools when installing packages like dev-python/setuptools-git and dev-python/pbr
Summary: sys-apps/portage: emerge --jobs triggers race conditions in dev-python/setupt...
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-10 19:44 UTC by Zac Medico
Modified: 2018-08-12 05:14 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2018-08-10 19:44:03 UTC
I've seen this race triggered when dev-python/setuptools-git or dev-python/pbr is merged while a package is being built with setuptools:

> Traceback (most recent call last):
>   File "setup.py", line 114, in <module>
>     include_package_data=False,
>   File "/usr/lib64/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
>     return distutils.core.setup(**attrs)
>   File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
>     dist.run_commands()
>   File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
>     self.run_command(cmd)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/install.py", line 61, in run
>     return orig.install.run(self)
>   File "/usr/lib64/python3.6/distutils/command/install.py", line 557, in run
>     self.run_command(cmd_name)
>   File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/install_egg_info.py", line 34, in run
>     self.run_command('egg_info')
>   File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.6/site-packages/setuptools/command/egg_info.py", line 272, in run
>     writer = ep.resolve()
>   File "/usr/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 2428, in resolve
>     module = __import__(self.module_name, fromlist=['__name__'], level=0)
> ModuleNotFoundError: No module named 'pbr'
> 
> Traceback (most recent call last):
>   File "setup.py", line 142, in <module>
>     keywords=['interface', 'components', 'plugins'],
>   File "/usr/lib64/python3.4/site-packages/setuptools/__init__.py", line 129, in setup
>     return distutils.core.setup(**attrs)
>   File "/usr/lib64/python3.4/distutils/core.py", line 148, in setup
>     dist.run_commands()
>   File "/usr/lib64/python3.4/distutils/dist.py", line 955, in run_commands
>     self.run_command(cmd)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/install.py", line 61, in run
>     return orig.install.run(self)
>   File "/usr/lib64/python3.4/distutils/command/install.py", line 539, in run
>     self.run_command('build')
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/distutils/command/build.py", line 126, in run
>     self.run_command(cmd_name)
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 53, in run
>     self.build_package_data()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 118, in build_package_data
>     for package, src_dir, build_dir, filenames in self.data_files:
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 66, in __getattr__
>     self.data_files = self._get_data_files()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 82, in _get_data_files
>     self.analyze_manifest()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 138, in analyze_manifest
>     self.run_command('egg_info')
>   File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command
>     self.distribution.run_command(command)
>   File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
>     cmd_obj.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 280, in run
>     self.find_sources()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 295, in find_sources
>     mm.run()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 526, in run
>     self.add_defaults()
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 565, in add_defaults
>     rcfiles = list(walk_revctrl())
>   File "/usr/lib64/python3.4/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl
>     for item in ep.load()(dirname):
>   File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2422, in load
>     return self.resolve()
>   File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2428, in resolve
>     module = __import__(self.module_name, fromlist=['__name__'], level=0)
> ImportError: No module named 'setuptools_git'
Comment 1 Zac Medico gentoo-dev 2018-08-10 19:58:22 UTC
A potential way to model this relationship would be to introduce a new kind of blocker that indicates that merging a package will temporarily interfere with the operation of another package. In the case described in comment #0, merging dev-python/setuptools-git or dev-python/pbr temporarily interferes with the operation of dev-python/setuptools.
Comment 2 Zac Medico gentoo-dev 2018-08-10 23:53:56 UTC
A "more atomic" package merge approach would minimize interference with setuptools, for example, these libraries use directory renames when possible so that the new package files are exposed as atomically as possible:

https://github.com/mgorny/atomic-install
https://github.com/mgorny/atomic-install-py
Comment 3 Zac Medico gentoo-dev 2018-08-11 00:12:29 UTC
Also, it might be possible to make setuptools do a retry loop, using timestamp comparisons to detect file/directory changes that have occurred during the last few seconds.
Comment 4 Zac Medico gentoo-dev 2018-08-12 05:14:08 UTC
If the package manager has access to snapshot capabilities for the root filesystem (bug 40127), it can run each build in a chroot environment with an immutable snapshot of the root filesystem, effectively implementing read-copy-update (RCU) for the root filesystem.

I imagine that many libraries with plugin systems might be vulnerable to the sort of race condition that setuptools / pkg_resources has. It seems like it's practically impossible to handle this sort of thing at the individual library level, since there could be a vast number of consumers of the library at any given time, so there might never be a safe time to install a plugin on a running system.