Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 529120 - portage.cache.fs_template._ensure_dirs is not parallel safe, os.mkdir may throw EEXIST
Summary: portage.cache.fs_template._ensure_dirs is not parallel safe, os.mkdir may thr...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS, PATCH
Depends on:
Blocks: 484436
  Show dependency tree
 
Reported: 2014-11-13 07:34 UTC by Michael Haubenwallner
Modified: 2017-08-11 20:17 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
_ensure_dirs: ignore EEXIST from mkdir (portage-ensure_dirs-parallel.patch,591 bytes, patch)
2014-11-13 07:34 UTC, Michael Haubenwallner
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Haubenwallner gentoo-dev 2014-11-13 07:34:16 UTC
Created attachment 389218 [details, diff]
_ensure_dirs: ignore EEXIST from mkdir

Creating a distfiles tarball snapshot (containing distfiles for all ebuilds with stable keywords in my Prefix-based distro, for offline install) using the command 'ebuild path-to/stable/stable-PVR.ebuild fetch' in parallel may fail with this backtrace with prefix-portage-2.2.10.1, leading me to attached patch:

Thanks!

Traceback (most recent call last):
  File "/opt/toolsbox/toolsbase-2015/usr/bin/ebuild", line 147, in <module>
    ebuild_portdir not in portage.portdb.porttrees:
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/proxy/objectproxy.py", line 22, in __getattribute__
    result = object.__getattribute__(self, '_get_target')()
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/__init__.py", line 689, in _get_target
    return _get_legacy_global(name)
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/_legacy_globals.py", line 14, in _get_legacy_global
    portage.portdb = portage.db[portage.root]["porttree"].dbapi
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/util/__init__.py", line 1472, in __getitem__
    result = lazy_item.func(*pargs, **kwargs)
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/dbapi/porttree.py", line 1089, in __init__
    self.dbapi = portdbapi(mysettings=settings)
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/dbapi/porttree.py", line 240, in __init__
    self.depcachedir, x, self._known_keys, **cache_kwargs)
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/cache/flat_hash.py", line 37, in __init__
    self._ensure_dirs()
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/cache/fs_template.py", line 72, in _ensure_dirs
    os.mkdir(base, perms)
  File "/opt/toolsbox/toolsbase-2015/usr/lib/portage/pym/portage/__init__.py", line 260, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 17] File exists: '/opt/toolsbox/toolsbase-2015/var/cache/edb/dep/ftp/pub/portage/toolsbox-tree-20141112'
Comment 1 Zac Medico gentoo-dev 2014-11-13 19:49:19 UTC
In the following branch I have an alternative patch which uses the util.ensure_dirs function to handle EEXIST:

	https://github.com/zmedico/portage/tree/bug_529120

I've posted it for review here:

	http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4782
Comment 3 Zac Medico gentoo-dev 2017-08-11 20:17:27 UTC
Fixed in 2.2.15.