Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 802150 - dev-python/numpy: incorrectly handles compiler arguments (was: sys-cluster/ceph-16.2.5 error: ##: No such file or directory)
Summary: dev-python/numpy: incorrectly handles compiler arguments (was: sys-cluster/ce...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Science Related Packages
URL:
Whiteboard:
Keywords: PATCH
: 821580 823032 (view as bug list)
Depends on: 824478
Blocks:
  Show dependency tree
 
Reported: 2021-07-14 12:09 UTC by Alessandro Barbieri
Modified: 2021-12-15 04:03 UTC (History)
9 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge_info (emerge_info,36.14 KB, text/plain)
2021-07-14 12:09 UTC, Alessandro Barbieri
Details
ceph-16.2.5:20210714-080216.log.xz (ceph-16.2.5:20210714-080216.log.xz,68.32 KB, application/x-xz)
2021-07-14 12:10 UTC, Alessandro Barbieri
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alessandro Barbieri 2021-07-14 12:09:45 UTC
This package fails to build on an unstable system. Attached relevant logs and emerge --info
Comment 1 Alessandro Barbieri 2021-07-14 12:09:59 UTC
Created attachment 723853 [details]
emerge_info

emerge --info
Comment 2 Alessandro Barbieri 2021-07-14 12:10:01 UTC
Created attachment 723856 [details]
ceph-16.2.5:20210714-080216.log.xz

build log (compressed because it exceeds attachment limit, use 'xzless' to read it)
Comment 3 Alessandro Barbieri 2021-07-14 12:11:47 UTC
What's happening here? There is a ## 0 in the arguments of gcc


[1/1] Cythonizing rgw.pyx                                                                                             
running build                                                                                                         
running build_ext                                          
building 'rgw' extension                                                                                              
C compiler: /usr/bin/x86_64-pc-linux-gnu-gcc -iquote/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Os -pipe -march=native -fdi
agnostics-color=always -fexceptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -D_GLIBCXX_ASSERTIONS -fPIC

creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind
creating /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw
compile options: '-I/usr/include/python3.9 -I/usr/include/python3.9 -c'                                               
extra options: '-iquote/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Os -pipe -march=native -fdiagnostics-color=always -fexce
ptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -D_GLIBCXX_ASSERTIONS -Wno-unused-result -Wsign-compare -DNDEBUG -Os -pipe -march=native -fdiagnostics-color=always -fexce
ptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -fwrapv'
x86_64-pc-linux-gnu-gcc: /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/rgw.c 
x86_64-pc-linux-gnu-gcc: error: ##: No such file or directory                                                         
x86_64-pc-linux-gnu-gcc: error: 0: No such file or directory                                                          
error: Command "/usr/bin/x86_64-pc-linux-gnu-gcc -iquote/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Os -pipe -march=native 
-fdiagnostics-color=always -fexceptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -D_GLIBCXX_ASSERTIONS -fPIC -I/usr/include/python3.9 -I/usr/include/python3.9 -c /var/tmp
/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/rgw.c -o /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/lib/cython_modules/temp.linux-x86_64-3.9/var/tmp/notmpfs/portage/sys-cluster
/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/rgw.o -iquote/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Os -pipe -march
=native -fdiagnostics-color=always -fexceptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -D_GLIBCXX_ASSERTIONS -Wno-unused-result -Wsign-compare -DNDEBUG -Os -pipe -march
=native -fdiagnostics-color=always -fexceptions -Wformat -fstack-clash-protection -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -fwrapv" failed with exit status 1
src/pybind/rgw/CMakeFiles/cython_rgw.dir/build.make:73: *** [lib/cython_modules/lib.3/rgw.cpython-39-x86_64-linux-gnu.so] error 1
                                                                                                                      
#0  lib/cython_modules/lib.3/rgw.cpython-39-x86_64-linux-gnu.so at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/CMakeFiles/cython_rgw.dir/build.make:72
#1  cython_rgw at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/CMakeFiles/cython_rgw.dir/build.make:78
#2  src/pybind/rgw/CMakeFiles/cython_rgw.dir/build at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/src/pybind/rgw/CMakeFiles/cython_rgw.dir/build.make:82
remake[2]: Leaving directory '/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build'
CMakeFiles/Makefile2:5240: *** [src/pybind/rgw/CMakeFiles/cython_rgw.dir/all] error 2                                 
                                                           
#0  src/pybind/rgw/CMakeFiles/cython_rgw.dir/all at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/CMakeFiles/Makefile2:5238
#1  src/pybind/CMakeFiles/cython_modules.dir/all at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/CMakeFiles/Makefile2:5134
#2  src/pybind/all at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/CMakeFiles/Makefile2:1284                                                                                                                    
#3  src/all at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/CMakeFiles/Makefile2:278
#4  all at /var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build/CMakeFiles/Makefile2:75
remake[1]: Leaving directory '/var/tmp/notmpfs/portage/sys-cluster/ceph-16.2.5/work/ceph-16.2.5_build'
Makefile:146: *** [all] error 2
Comment 4 Andrej Filipcic 2021-10-04 08:52:20 UTC
Same is happening with 16.2.6 with recently updated packages, since 2-3 weeks. 

Try to build in a minimal ~amd64 container, with
[ebuild   R    ] sys-cluster/ceph-16.2.6::gentoo  USE="cephfs fuse kerberos mgr numa radosgw sqlite ssl systemd tcmalloc -babeltrace -custom-cflags -diskprediction (-dpdk) -grafana -jaeger -jemalloc -kafka -ldap -lttng -pmdk -rabbitmq -rbd-rwl -rbd-ssd -rgw-lua -spdk (-system-boost) -test -uring -xfs -zbd -zfs" CPU_FLAGS_X86="sse sse2 sse3 -avx2 -avx512f -pclmul -sse4_1 -sse4_2 -ssse3" LUA_SINGLE_TARGET="lua5-3" PYTHON_TARGETS="python3_9 -python3_8" 0 KiB

one of the recent dependent packages must be causing this issue.

Cheers,
Andrej
Comment 5 Hector Martin 2021-10-14 06:14:03 UTC
Well this is a funny one. Turns out the Ceph build is ending up pulling in distutils customizations from numpy, and those customizations are broken when the C compiler has arguments with spaces in them, which Cython uses.

Here's the call stack where that happens:

Traceback (most recent call last):
  File "/var/tmp/portage/sys-cluster/ceph-16.2.6/work/ceph-16.2.6/src/pybind/rados/setup.py", line 194, in <module>
    ext_modules=cythonize(
  File "/usr/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 963, in cythonize
    ctx = c_options.create_context()
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Main.py", line 596, in create_context
    return Context(self.include_path, self.compiler_directives,
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Main.py", line 80, in __init__
    from . import Builtin, CythonScope
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/CythonScope.py", line 5, in <module>
    from .UtilityCode import CythonUtilityCode
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/UtilityCode.py", line 3, in <module>
    from .TreeFragment import parse_from_strings, StringParseContext
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/TreeFragment.py", line 17, in <module>
    from .Visitor import VisitorTransform
  File "Cython/Compiler/Visitor.py", line 16, in init Cython.Compiler.Visitor
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Nodes.py", line 31, in <module>
    from .Pythran import has_np_pythran, pythran_type, is_pythran_buffer
  File "/usr/lib/python3.9/site-packages/Cython/Compiler/Pythran.py", line 10, in <module>
    import pythran
  File "/usr/lib/python3.9/site-packages/pythran/__init__.py", line 40, in <module>
    from pythran.config import get_include
  File "/usr/lib/python3.9/site-packages/pythran/config.py", line 8, in <module>
    import numpy.distutils.system_info as numpy_sys
  File "/usr/lib/python3.9/site-packages/numpy/distutils/__init__.py", line 24, in <module>
    from . import ccompiler
  File "/usr/lib/python3.9/site-packages/numpy/distutils/ccompiler.py", line 35, in <module>

So this is ceph -> Cython -> pythran -> numpy. 

The bug is in /usr/lib/python3.9/site-packages/numpy/distutils/unixccompiler.py:

        ccomp_s = " ".join(self.compiler_so)
        if opt not in ccomp_s:
            ccomp_s = ccomp_s.replace(gcv_opt, opt)
            self.compiler_so = ccomp_s.split()

This breaks when self.compiler_so has arguments with spaces.

I submitted a fix to numpy here: https://github.com/numpy/numpy/pull/20116 . Applying that patch to dev-python/numpy fixes the ceph build.
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-10-14 06:41:04 UTC
(In reply to Hector Martin from comment #5)
> I submitted a fix to numpy here: https://github.com/numpy/numpy/pull/20116 .
> Applying that patch to dev-python/numpy fixes the ceph build.

Big thanks. I'll wait a wee bit to see if anyone comments upstream but pull it in soon anyway.
Comment 7 Patrick McLean gentoo-dev 2021-11-08 19:02:13 UTC
*** Bug 821580 has been marked as a duplicate of this bug. ***
Comment 8 Toralf Förster gentoo-dev 2021-11-17 16:57:29 UTC
*** Bug 823032 has been marked as a duplicate of this bug. ***
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-11-17 23:02:30 UTC
I'd missed (somehow) that the patch was merged a few days ago. On it.
Comment 10 Larry the Git Cow gentoo-dev 2021-11-17 23:13:12 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=aeb4bf89ff38ff18c17f51d8c9d0d19cae1c349d

commit aeb4bf89ff38ff18c17f51d8c9d0d19cae1c349d
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-11-17 23:12:39 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-11-17 23:13:04 +0000

    dev-python/numpy: add upstream patch for compiler args (fix Ceph build failure)
    
    Bug: https://bugs.gentoo.org/802150
    Thanks-to: Hector Martin <marcan@marcan.st>
    Signed-off-by: Sam James <sam@gentoo.org>

 .../numpy-1.21.4-build-compiler-args-ceph.patch    |  49 +++++++
 dev-python/numpy/numpy-1.21.4-r1.ebuild            | 144 +++++++++++++++++++++
 2 files changed, 193 insertions(+)
Comment 11 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-15 04:03:14 UTC
Fixed version is stabled.