Summary: | dev-python/scipy-1.6.1: compilation failed "RuntimeError: NumPy was built with baseline optimizations: (SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support: (POPCNT)" | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Marcin Mirosław <bug> |
Component: | Current packages | Assignee: | Gentoo Science Related Packages <sci> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | esigra, ionen, jstein, kingjon3377, krzysztof, leonchik1976, mgorny, michal, mplichta, python, sam, sboub88 |
Priority: | Normal | Keywords: | PATCH |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: |
https://github.com/numpy/numpy/issues/19067 https://bugs.gentoo.org/show_bug.cgi?id=794187 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 788712 | ||
Attachments: |
numpy-build.log
fix the compile time test for working popcnt Actually fix the popcnt test |
Description
Marcin Mirosław
2021-05-04 17:53:33 UTC
# cat build.log * Package: dev-python/scipy-1.6.1 * Repository: gentoo * Maintainer: sci@gentoo.org python@gentoo.org * USE: abi_x86_64 amd64 elibc_glibc kernel_linux python_targets_python3_8 userland_GNU * FEATURES: ccache compressdebug network-sandbox preserve-libs sandbox splitdebug userpriv usersandbox * Using following Fortran compiler: * F77: x86_64-pc-linux-gnu-gfortran * FC: x86_64-pc-linux-gnu-gfortran >>> Unpacking source... >>> Unpacking scipy-1.6.1.tar.gz to /var/tmp/portage/dev-python/scipy-1.6.1/work >>> Source unpacked in /var/tmp/portage/dev-python/scipy-1.6.1/work >>> Preparing source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ... * Will copy sources from /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 * python3_8: copying to /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8 >>> Source prepared. >>> Configuring source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ... * Using python3.8 in global scope * python3_8: running distutils-r1_run_phase python_configure_all >>> Source configured. >>> Compiling source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ... * python3_8: running distutils-r1_run_phase python_compile scipy/linalg/_generate_pyx.py: all files up-to-date Traceback (most recent call last): File "scipy/special/_generate_pyx.py", line 233, in <module> import numpy File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 145, in <module> from . import core File "/usr/lib/python3.8/site-packages/numpy/core/__init__.py", line 22, in <module> from . import multiarray File "/usr/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module> from . import overrides File "/usr/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module> from numpy.core._multiarray_umath import ( RuntimeError: NumPy was built with baseline optimizations: (SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support: (POPCNT). Running scipy/special/_generate_pyx.py Running scipy/linalg/_generate_pyx.py Traceback (most recent call last): File "tools/cythonize.py", line 311, in <module> main() File "tools/cythonize.py", line 307, in main find_process_files(root_dir) File "tools/cythonize.py", line 273, in find_process_files for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs): File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next raise value File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "tools/cythonize.py", line 273, in <lambda> for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs): File "tools/cythonize.py", line 244, in process_generate_pyx raise RuntimeError("Running {} failed".format(path)) RuntimeError: Running scipy/special/_generate_pyx.py failed * ERROR: dev-python/scipy-1.6.1::gentoo failed (compile phase): * (no error message) * * Call stack: * ebuild.sh, line 125: Called src_compile * environment, line 3534: Called distutils-r1_src_compile * environment, line 1514: Called _distutils-r1_run_foreach_impl 'python_compile' * environment, line 466: Called python_foreach_impl 'distutils-r1_run_phase' 'python_compile' * environment, line 3100: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile' * environment, line 2584: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile' * environment, line 2582: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_compile' * environment, line 954: Called distutils-r1_run_phase 'python_compile' * environment, line 1507: Called python_compile * environment, line 2870: Called die * The specific snippet of code: * ${EPYTHON} tools/cythonize.py || die; * * If you need support, post the output of `emerge --info '=dev-python/scipy-1.6.1::gentoo'`, * the complete build log and the output of `emerge -pqv '=dev-python/scipy-1.6.1::gentoo'`. * The complete build log is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/environment'. * Working directory: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8' * S: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1' Thank you for the report. Please * attach the logs and * paste the emerge info as described on https://wiki.gentoo.org/wiki/Attach_the_logs_to_the_bug_ticket next time. It is more readable then. Do you have a numpy log handy? Created attachment 706377 [details]
numpy-build.log
*** Bug 788673 has been marked as a duplicate of this bug. *** *** Bug 789072 has been marked as a duplicate of this bug. *** *** Bug 788769 has been marked as a duplicate of this bug. *** I thought I posted this as a comment but maybe not. It looks like Numpy autodetects based on CFLAGS. We need to disable that. (In reply to Sam James from comment #8) > I thought I posted this as a comment but maybe not. It looks like Numpy > autodetects based on CFLAGS. We need to disable that. https://github.com/numpy/numpy/blob/623bc1fae1d47df24e7f1e29321d0c0ba2771ce0/doc/source/reference/simd/simd-optimizations.rst#id1 (In reply to Sam James from comment #8) > I thought I posted this as a comment but maybe not. It looks like Numpy > autodetects based on CFLAGS. We need to disable that. I disagree. If your CFLAGS indicate POPCNT but you don't have POPCNT, then it's a wrong CFLAGS problem. OTOH, if this is cross-building, then I suppose numpy shouldn't check CPU on build host. (In reply to Michał Górny from comment #10) > (In reply to Sam James from comment #8) > > I thought I posted this as a comment but maybe not. It looks like Numpy > > autodetects based on CFLAGS. We need to disable that. > > I disagree. If your CFLAGS indicate POPCNT but you don't have POPCNT, then > it's a wrong CFLAGS problem. > > OTOH, if this is cross-building, then I suppose numpy shouldn't check CPU on > build host. Sorry, I was a bit unclear because I was trying to quickly post a comment which I thought I'd already done... It looks like it auto-detects based on -march=native among other things which is problematic for cross or binary packages like you say (this is a big thing for releng, for example). But of course, there's a real actual bug here in why-the-heck it is assuming POPCNT when people don't have it. (In reply to Sam James from comment #11) > It looks like it auto-detects based on -march=native among other things > which is problematic for cross or binary packages like you say (this is a > big thing for releng, for example). If OP's using -march=native, then everything relies on host CPU anyway, so there is nothing wrong with this. Just to be clear, I'm saying that it's fine for numpy to detect based on current CFLAGS, even if they're -march=native. However, it's not fine if the detection misfires. I get this error with sci-libs/gdal-3.2.2. I upgraded to dev-python/numpy-1.20.3 but gdal still fails after that. I'm wondering if the test for popcnt actually works? If I just compile the test program (cpu_popcnt.c) by hand and execute it, it seems to run ok. I edited it to return 0 instead of the 1 or 2 that its supposed to, and it doesn't seem to affect anything in the build. To have the check for popcnt actually fail I put a syntax error into it so the compiler throws a fit and the test fails. Its puzzling to me that the runtime test for POPCNT and compile time test are checking very different things. The compile-time test checks to see if code like: a = __builtin_popcountll(1); executes, and this does seem to execute properly on this CPU (Core 2 Due E8400) But the run-time code I think checks for a particular bit in a cpu register. Perhaps the compiler is out-smarting the test? I think the test is borked. If I compile that test with gcc -S -mpopcnt no popcnt instruction appears in the assembly. It looks like the compiler short circuits the test. If however I change this: #ifdef __x86_64__ a = __builtin_popcountll(1); #endif b = __builtin_popcount(1) to #ifdef __x86_64__ a = __builtin_popcountll(1); #endif b = __builtin_popcount(a) then the compiler includes: popcntl %eax, %eax which crashes the program with "Illegal Instruction" That's with gcc 10.2.0 and no optimization. If I include -O2, then the popcntl disappears even with this modification. Probably a random number should be chosen, or the pid, or something that's not known at compile time so the popcount can't be optimized away. Or, the runtime test could be used. I'll file a bug upstream. Upstream issue at: https://github.com/numpy/numpy/issues/19067 Created attachment 710367 [details, diff]
fix the compile time test for working popcnt
This patches cpu_popcnt.c so that it actually tests whether the popcnt instruction can be executed.
sorry - my first patch causes other problems. New one coming. Created attachment 710691 [details, diff]
Actually fix the popcnt test
This is based on a suggestion of seiko2plus at numpy.
That second patch has been merged upstream. I apply chnages from patch and it helped - matpotlib merged (In reply to Michal Plichta from comment #23) > I apply chnages from patch and it helped - matpotlib merged I ment opencv... and not changing /usr/lib/python3.x/site-packages/numpy/distutils/checks/cpu_pop.cnt.c didn't helped.... The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ca58a4b159282f564f046e035a17f7ce0bd30f01 commit ca58a4b159282f564f046e035a17f7ce0bd30f01 Author: Sam James <sam@gentoo.org> AuthorDate: 2021-05-24 22:08:17 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-05-24 22:08:55 +0000 dev-python/numpy: add popcnt patch Closes: https://bugs.gentoo.org/788184 Signed-off-by: Sam James <sam@gentoo.org> .../files/numpy-1.20.2-fix-popcnt-detection.patch | 103 +++++++++++++++++++++ ...{numpy-1.20.2.ebuild => numpy-1.20.2-r1.ebuild} | 1 + dev-python/numpy/numpy-1.20.3.ebuild | 1 + 3 files changed, 105 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bb670c5ea6d2257ff88e3cb513b7593794f1974a commit bb670c5ea6d2257ff88e3cb513b7593794f1974a Author: Sam James <sam@gentoo.org> AuthorDate: 2021-05-24 22:11:23 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-05-24 22:11:23 +0000 dev-python/numpy: revbump ~arch version (1.20.3) for popcnt fix too Bug: https://bugs.gentoo.org/788184 Fixes: ca58a4b159282f564f046e035a17f7ce0bd30f01 Signed-off-by: Sam James <sam@gentoo.org> dev-python/numpy/{numpy-1.20.3.ebuild => numpy-1.20.3-r1.ebuild} | 0 1 file changed, 0 insertions(+), 0 deletions(-) |