Bug 902713 – Portage might create a directory during merge then pause before installing files (dev-python/shapely-2.0.1 fails to compile: AttributeError: module pythran has no attribute __version__)

• Status: CONFIRMED
• Product: Portage Development

[Description Agostino Sarubbo 2023-03-22 07:54:34 UTC]

https://blogs.gentoo.org/ago/2020/07/04/gentoo-tinderbox/

Issue: dev-python/shapely-2.0.1 fails to compile.
Discovered on: amd64 (internal ref: ci)

[Comment 1 Agostino Sarubbo 2023-03-22 07:54:36 UTC]

Created attachment 858585 [details]
build.log

build log and emerge --info

[Comment 2 Sam James 2023-03-22 08:03:04 UTC]

dev-python/cython-0.29.33:0 -doc -emacs python_targets_pypy3 python_targets_python3_10 python_targets_python3_11 python_targets_python3_9 -test

python3.10 setup.py build_ext -j 32
Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/shapely-2.0.1/work/shapely-2.0.1/setup.py", line 199, in <module>
    ext_modules += cythonize(
  File "/usr/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 968, in cythonize
    ctx = c_options.create_context()
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 600, in create_context
    return Context(self.include_path, self.compiler_directives,
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 80, in __init__
    from . import Builtin, CythonScope
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
    from .UtilityCode import CythonUtilityCode
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
    from .TreeFragment import parse_from_strings, StringParseContext
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
    from .Visitor import VisitorTransform
  File "Cython/Compiler/Visitor.py", line 16, in init Cython.Compiler.Visitor
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/Nodes.py", line 31, in <module>
    from .Pythran import has_np_pythran, pythran_type, is_pythran_buffer
  File "/usr/lib/python3.10/site-packages/Cython/Compiler/Pythran.py", line 11, in <module>
    pythran_is_pre_0_9 = tuple(map(int, pythran.__version__.split('.')[0:2])) < (0, 9)
AttributeError: module 'pythran' has no attribute '__version__'
 * ERROR: dev-python/shapely-2.0.1::gentoo failed (compile phase):
 *   (no error message)
 *

[Comment 3 Sam James 2023-03-22 08:06:36 UTC]

It's imported pythran but pythran has no __version__?

[Comment 4 Sam James 2023-03-24 04:12:48 UTC]

Created attachment 858783 [details]
emerge output w/ merge order

I hit this too. It's a very narrow window where pythran is importable but.. not actually fully installed.

[Comment 5 Larry the Git Cow 2023-03-24 05:12:06 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9e9a93ca1c92f22f4ad15ae2e814c81deb1145f8

commit 9e9a93ca1c92f22f4ad15ae2e814c81deb1145f8
Author:     Sam James <[e-mail]>
AuthorDate: 2023-03-24 05:03:59 +0000
Commit:     Sam James <[e-mail]>
CommitDate: 2023-03-24 05:09:07 +0000

    dev-python/cython: workaround parallel install race for importing pythran
    
    An empty directory in site-packages is a valid Python module, unfortunately, because
    of namespaces. If installing packages in parallel, the pythran module might "exist"
    but be empty (hence no __version__ attribute). Catch AttributeError to avoid a narrow
    race.
    
    This might still be a Portage issue if there's a substantial delay between creating
    the directory and installing the remaining files though.
    
    Thanks to Eli Schwartz for helping out.
    
    Bug: https://bugs.gentoo.org/902713
    Signed-off-by: Sam James <[e-mail]>

 dev-python/cython/cython-0.29.33-r1.ebuild         | 91 ++++++++++++++++++++++
 ...0_beta1.ebuild => cython-3.0.0_beta1-r1.ebuild} |  0
 .../cython-0.29.23-pythran-parallel-install.patch  | 20 +++++
 3 files changed, 111 insertions(+)

[Comment 6 Sam James 2023-03-24 05:12:58 UTC]

Let's keep this open for now. It's suspicious that both ago and I hit what should be a very narrow window indeed.

We're wondering if there's some expensive step portage does between merging directories vs merging files.