I've seen this race triggered when dev-python/setuptools-git or dev-python/pbr is merged while a package is being built with setuptools: > Traceback (most recent call last): > File "setup.py", line 114, in <module> > include_package_data=False, > File "/usr/lib64/python3.6/site-packages/setuptools/__init__.py", line 129, in setup > return distutils.core.setup(**attrs) > File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup > dist.run_commands() > File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands > self.run_command(cmd) > File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.6/site-packages/setuptools/command/install.py", line 61, in run > return orig.install.run(self) > File "/usr/lib64/python3.6/distutils/command/install.py", line 557, in run > self.run_command(cmd_name) > File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.6/site-packages/setuptools/command/install_egg_info.py", line 34, in run > self.run_command('egg_info') > File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.6/site-packages/setuptools/command/egg_info.py", line 272, in run > writer = ep.resolve() > File "/usr/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 2428, in resolve > module = __import__(self.module_name, fromlist=['__name__'], level=0) > ModuleNotFoundError: No module named 'pbr' > > Traceback (most recent call last): > File "setup.py", line 142, in <module> > keywords=['interface', 'components', 'plugins'], > File "/usr/lib64/python3.4/site-packages/setuptools/__init__.py", line 129, in setup > return distutils.core.setup(**attrs) > File "/usr/lib64/python3.4/distutils/core.py", line 148, in setup > dist.run_commands() > File "/usr/lib64/python3.4/distutils/dist.py", line 955, in run_commands > self.run_command(cmd) > File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.4/site-packages/setuptools/command/install.py", line 61, in run > return orig.install.run(self) > File "/usr/lib64/python3.4/distutils/command/install.py", line 539, in run > self.run_command('build') > File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.4/distutils/command/build.py", line 126, in run > self.run_command(cmd_name) > File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 53, in run > self.build_package_data() > File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 118, in build_package_data > for package, src_dir, build_dir, filenames in self.data_files: > File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 66, in __getattr__ > self.data_files = self._get_data_files() > File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 82, in _get_data_files > self.analyze_manifest() > File "/usr/lib64/python3.4/site-packages/setuptools/command/build_py.py", line 138, in analyze_manifest > self.run_command('egg_info') > File "/usr/lib64/python3.4/distutils/cmd.py", line 313, in run_command > self.distribution.run_command(command) > File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command > cmd_obj.run() > File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 280, in run > self.find_sources() > File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 295, in find_sources > mm.run() > File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 526, in run > self.add_defaults() > File "/usr/lib64/python3.4/site-packages/setuptools/command/egg_info.py", line 565, in add_defaults > rcfiles = list(walk_revctrl()) > File "/usr/lib64/python3.4/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl > for item in ep.load()(dirname): > File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2422, in load > return self.resolve() > File "/usr/lib64/python3.4/site-packages/pkg_resources/__init__.py", line 2428, in resolve > module = __import__(self.module_name, fromlist=['__name__'], level=0) > ImportError: No module named 'setuptools_git'
A potential way to model this relationship would be to introduce a new kind of blocker that indicates that merging a package will temporarily interfere with the operation of another package. In the case described in comment #0, merging dev-python/setuptools-git or dev-python/pbr temporarily interferes with the operation of dev-python/setuptools.
A "more atomic" package merge approach would minimize interference with setuptools, for example, these libraries use directory renames when possible so that the new package files are exposed as atomically as possible: https://github.com/mgorny/atomic-install https://github.com/mgorny/atomic-install-py
Also, it might be possible to make setuptools do a retry loop, using timestamp comparisons to detect file/directory changes that have occurred during the last few seconds.
If the package manager has access to snapshot capabilities for the root filesystem (bug 40127), it can run each build in a chroot environment with an immutable snapshot of the root filesystem, effectively implementing read-copy-update (RCU) for the root filesystem. I imagine that many libraries with plugin systems might be vulnerable to the sort of race condition that setuptools / pkg_resources has. It seems like it's practically impossible to handle this sort of thing at the individual library level, since there could be a vast number of consumers of the library at any given time, so there might never be a safe time to install a plugin on a running system.
