Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 788184 - dev-python/scipy-1.6.1: compilation failed "RuntimeError: NumPy was built with baseline optimizations: (SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support: (POPCNT)"
Summary: dev-python/scipy-1.6.1: compilation failed "RuntimeError: NumPy was built wit...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 2 votes (vote)
Assignee: Gentoo Science Related Packages
URL:
Whiteboard:
Keywords: PATCH
: 788673 788769 789072 (view as bug list)
Depends on:
Blocks: 788712
  Show dependency tree
 
Reported: 2021-05-04 17:53 UTC by Marcin Mirosław
Modified: 2021-06-04 16:54 UTC (History)
12 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
numpy-build.log (build.log,397.55 KB, text/plain)
2021-05-06 16:52 UTC, Marcin Mirosław
Details
fix the compile time test for working popcnt (fix_popcnt_test.patch,851 bytes, patch)
2021-05-23 03:37 UTC, Carl Michal
Details | Diff
Actually fix the popcnt test (fix_popcnt_test2.patch,924 bytes, patch)
2021-05-23 15:35 UTC, Carl Michal
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Marcin Mirosław 2021-05-04 17:53:33 UTC
>>> Emerging (1 of 1) dev-python/scipy-1.6.1::gentoo
 * scipy-1.6.1.tar.gz BLAKE2B SHA512 size ;-) ...                                                                                                                                                      [ ok ]
 * Using following Fortran compiler:
 *   F77: x86_64-pc-linux-gnu-gfortran
 *   FC:  x86_64-pc-linux-gnu-gfortran
>>> Unpacking source...
>>> Unpacking scipy-1.6.1.tar.gz to /var/tmp/portage/dev-python/scipy-1.6.1/work
>>> Source unpacked in /var/tmp/portage/dev-python/scipy-1.6.1/work
>>> Preparing source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * Will copy sources from /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1
 * python3_8: copying to /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * Using python3.8 in global scope
 * python3_8: running distutils-r1_run_phase python_configure_all
>>> Source configured.
>>> Compiling source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * python3_8: running distutils-r1_run_phase python_compile
scipy/linalg/_generate_pyx.py: all files up-to-date
Traceback (most recent call last):
  File "scipy/special/_generate_pyx.py", line 233, in <module>
    import numpy
  File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "/usr/lib/python3.8/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/usr/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
RuntimeError: NumPy was built with baseline optimizations:
(SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support:
(POPCNT).
Running scipy/special/_generate_pyx.py
Running scipy/linalg/_generate_pyx.py
Traceback (most recent call last):
  File "tools/cythonize.py", line 311, in <module>
    main()
  File "tools/cythonize.py", line 307, in main
    find_process_files(root_dir)
  File "tools/cythonize.py", line 273, in find_process_files
    for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "tools/cythonize.py", line 273, in <lambda>
    for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs):
  File "tools/cythonize.py", line 244, in process_generate_pyx
    raise RuntimeError("Running {} failed".format(path))
RuntimeError: Running scipy/special/_generate_pyx.py failed
 * ERROR: dev-python/scipy-1.6.1::gentoo failed (compile phase):
 *   (no error message)
 *
 * Call stack:
 *     ebuild.sh, line  125:  Called src_compile
 *   environment, line 3534:  Called distutils-r1_src_compile
 *   environment, line 1514:  Called _distutils-r1_run_foreach_impl 'python_compile'
 *   environment, line  466:  Called python_foreach_impl 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 3100:  Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 2584:  Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 2582:  Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_compile'
 *   environment, line  954:  Called distutils-r1_run_phase 'python_compile'
 *   environment, line 1507:  Called python_compile
 *   environment, line 2870:  Called die
 * The specific snippet of code:
 *       ${EPYTHON} tools/cythonize.py || die;
 *
 * If you need support, post the output of `emerge --info '=dev-python/scipy-1.6.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-python/scipy-1.6.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8'
 * S: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1'



Reproducible: Always




# emerge --inf
/usr/lib/python3.8/site-packages/portage/package/ebuild/_config/KeywordsManager.py:70: UserWarning: /etc/portage/package.keywords is deprecated, use /etc/portage/package.accept_keywords instead
  warnings.warn(_("%s is deprecated, use %s instead") %
Portage 3.0.18 (python 3.8.8-final-0, default/linux/amd64/17.1, gcc-10.2.0, glibc-2.32-r7, 5.11.0-00700-g568238cf3534 x86_64)
=================================================================
System uname: Linux-5.11.0-00700-g568238cf3534-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E8400_@_3.00GHz-with-glibc2.2.5
KiB Mem:     3561968 total,    235880 free
KiB Swap:    2572284 total,   2161148 free
Timestamp of repository gentoo: Tue, 04 May 2021 05:35:38 +0000
Head commit of repository gentoo: dba937842ed94f913763f2fcdd59f1d62c26001b

Timestamp of repository monero: Sat, 24 Apr 2021 14:34:57 +0000                                                                                                                                               Head commit of repository monero: aff1d28621d7c1b24f80acb2f028af5e8f2cae67                                                                                                                                    
sh bash 5.0_p18
ld GNU gold (Gentoo 2.35.2 p1 2.35.2) 1.16
ccache version 4.2.1 [enabled]
app-shells/bash:          5.0_p18::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          3.8.8_p1::gentoo, 3.9.2_p1::gentoo
dev-lang/rust-bin:        1.51.0::gentoo
dev-util/ccache:          4.2.1::gentoo
dev-util/cmake:           3.18.5::gentoo                                                                                                                                                                      dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/openrc:          0.42.1-r1::gentoo                                                                                                                                                                   sys-apps/sandbox:         2.22::gentoo                                                                                                                                                                        sys-devel/autoconf:       2.69-r5::gentoo                                                                                                                                                                     sys-devel/automake:       1.16.2-r1::gentoo                                                                                                                                                                   sys-devel/binutils:       2.35.2::gentoo                                                                                                                                                                      sys-devel/gcc:            10.2.0-r5::gentoo                                                                                                                                                                   sys-devel/gcc-config:     2.4::gentoo                                                                                                                                                                         sys-devel/libtool:        2.4.6-r6::gentoo                                                                                                                                                                    sys-devel/make:           4.3::gentoo                                                                                                                                                                         sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers)
sys-libs/glibc:           2.32-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    sync-user: portage:portage
    priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native "
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native "
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache cgroup collision-protect compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="pl_PL.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en pl"
MAKEOPTS="-j2"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl acpi aio amd64 bash-completion bzip2 caps cli dri fortran hardened iconv idn ipv6 libglvnd libtirpc mmx mmxext multilib ncurses nls nptl openmp pam pcre readline recursion-limit seccomp split-usr sse sse2 sse3 ssl ssse3 threads threadsafe unicode vim-syntax xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3 sse4_1" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_8" PYTHON_TARGETS="python3_8" RUBY_TARGETS="ruby26" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS


# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   36 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           23
Model name:                      Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
Stepping:                        10
CPU MHz:                         2001.731
CPU max MHz:                     3000.0000
CPU min MHz:                     2000.0000
BogoMIPS:                        6000.39
Virtualization:                  VT-x
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        6 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX EPT disabled
Vulnerability Mds:               Vulnerable; SMT disabled
Vulnerability Meltdown:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, STIBP: disabled
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_goo
                                 d nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority vpid dtherm
Comment 1 Marcin Mirosław 2021-05-04 17:59:37 UTC
# cat build.log
 * Package:    dev-python/scipy-1.6.1
 * Repository: gentoo
 * Maintainer: sci@gentoo.org python@gentoo.org
 * USE:        abi_x86_64 amd64 elibc_glibc kernel_linux python_targets_python3_8 userland_GNU
 * FEATURES:   ccache compressdebug network-sandbox preserve-libs sandbox splitdebug userpriv usersandbox
 * Using following Fortran compiler:
 *   F77: x86_64-pc-linux-gnu-gfortran
 *   FC:  x86_64-pc-linux-gnu-gfortran
>>> Unpacking source...
>>> Unpacking scipy-1.6.1.tar.gz to /var/tmp/portage/dev-python/scipy-1.6.1/work
>>> Source unpacked in /var/tmp/portage/dev-python/scipy-1.6.1/work
>>> Preparing source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * Will copy sources from /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1
 * python3_8: copying to /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * Using python3.8 in global scope
 * python3_8: running distutils-r1_run_phase python_configure_all
>>> Source configured.
>>> Compiling source in /var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1 ...
 * python3_8: running distutils-r1_run_phase python_compile
scipy/linalg/_generate_pyx.py: all files up-to-date
Traceback (most recent call last):
  File "scipy/special/_generate_pyx.py", line 233, in <module>
    import numpy
  File "/usr/lib/python3.8/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "/usr/lib/python3.8/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/usr/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
RuntimeError: NumPy was built with baseline optimizations:
(SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42) but your machine doesn't support:
(POPCNT).
Running scipy/special/_generate_pyx.py
Running scipy/linalg/_generate_pyx.py
Traceback (most recent call last):
  File "tools/cythonize.py", line 311, in <module>
    main()
  File "tools/cythonize.py", line 307, in main
    find_process_files(root_dir)
  File "tools/cythonize.py", line 273, in find_process_files
    for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "tools/cythonize.py", line 273, in <lambda>
    for result in pool.imap_unordered(lambda fn: process_generate_pyx(fn, lock), jobs):
  File "tools/cythonize.py", line 244, in process_generate_pyx
    raise RuntimeError("Running {} failed".format(path))
RuntimeError: Running scipy/special/_generate_pyx.py failed
 * ERROR: dev-python/scipy-1.6.1::gentoo failed (compile phase):
 *   (no error message)
 *
 * Call stack:
 *     ebuild.sh, line  125:  Called src_compile
 *   environment, line 3534:  Called distutils-r1_src_compile
 *   environment, line 1514:  Called _distutils-r1_run_foreach_impl 'python_compile'
 *   environment, line  466:  Called python_foreach_impl 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 3100:  Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 2584:  Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'python_compile'
 *   environment, line 2582:  Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'python_compile'
 *   environment, line  954:  Called distutils-r1_run_phase 'python_compile'
 *   environment, line 1507:  Called python_compile
 *   environment, line 2870:  Called die
 * The specific snippet of code:
 *       ${EPYTHON} tools/cythonize.py || die;
 *
 * If you need support, post the output of `emerge --info '=dev-python/scipy-1.6.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-python/scipy-1.6.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-python/scipy-1.6.1/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1-python3_8'
 * S: '/var/tmp/portage/dev-python/scipy-1.6.1/work/scipy-1.6.1'
Comment 2 Jonas Stein gentoo-dev 2021-05-04 21:19:37 UTC
Thank you for the report. Please 
* attach the logs and 
* paste the emerge info 
as described on https://wiki.gentoo.org/wiki/Attach_the_logs_to_the_bug_ticket
next time. It is more readable then.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-05-06 06:56:16 UTC
Do you have a numpy log handy?
Comment 4 Marcin Mirosław 2021-05-06 16:52:20 UTC
Created attachment 706377 [details]
numpy-build.log
Comment 5 Sam James archtester gentoo-dev Security 2021-05-07 05:03:57 UTC
*** Bug 788673 has been marked as a duplicate of this bug. ***
Comment 6 Ionen Wolkens gentoo-dev 2021-05-09 13:16:20 UTC
*** Bug 789072 has been marked as a duplicate of this bug. ***
Comment 7 Ionen Wolkens gentoo-dev 2021-05-09 13:16:57 UTC
*** Bug 788769 has been marked as a duplicate of this bug. ***
Comment 8 Sam James archtester gentoo-dev Security 2021-05-11 22:19:25 UTC
I thought I posted this as a comment but maybe not. It looks like Numpy autodetects based on CFLAGS. We need to disable that.
Comment 9 Sam James archtester gentoo-dev Security 2021-05-11 22:19:37 UTC
(In reply to Sam James from comment #8)
> I thought I posted this as a comment but maybe not. It looks like Numpy
> autodetects based on CFLAGS. We need to disable that.

https://github.com/numpy/numpy/blob/623bc1fae1d47df24e7f1e29321d0c0ba2771ce0/doc/source/reference/simd/simd-optimizations.rst#id1
Comment 10 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-05-11 22:23:26 UTC
(In reply to Sam James from comment #8)
> I thought I posted this as a comment but maybe not. It looks like Numpy
> autodetects based on CFLAGS. We need to disable that.

I disagree.  If your CFLAGS indicate POPCNT but you don't have POPCNT, then it's a wrong CFLAGS problem.

OTOH, if this is cross-building, then I suppose numpy shouldn't check CPU on build host.
Comment 11 Sam James archtester gentoo-dev Security 2021-05-11 23:02:03 UTC
(In reply to Michał Górny from comment #10)
> (In reply to Sam James from comment #8)
> > I thought I posted this as a comment but maybe not. It looks like Numpy
> > autodetects based on CFLAGS. We need to disable that.
> 
> I disagree.  If your CFLAGS indicate POPCNT but you don't have POPCNT, then
> it's a wrong CFLAGS problem.
> 
> OTOH, if this is cross-building, then I suppose numpy shouldn't check CPU on
> build host.

Sorry, I was a bit unclear because I was trying to quickly post a comment which I thought I'd already done...

It looks like it auto-detects based on -march=native among other things which is problematic for cross or binary packages like you say (this is a big thing for releng, for example).

But of course, there's a real actual bug here in why-the-heck it is assuming POPCNT when people don't have it.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-05-12 07:42:56 UTC
(In reply to Sam James from comment #11)
> It looks like it auto-detects based on -march=native among other things
> which is problematic for cross or binary packages like you say (this is a
> big thing for releng, for example).

If OP's using -march=native, then everything relies on host CPU anyway, so there is nothing wrong with this.
Comment 13 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-05-12 08:09:23 UTC
Just to be clear, I'm saying that it's fine for numpy to detect based on current CFLAGS, even if they're -march=native.  However, it's not fine if the detection misfires.
Comment 14 Erik 2021-05-20 07:07:16 UTC
I get this error with sci-libs/gdal-3.2.2.
Comment 15 Erik 2021-05-20 11:48:15 UTC
I upgraded to dev-python/numpy-1.20.3 but gdal still fails after that.
Comment 16 Carl Michal 2021-05-22 00:28:44 UTC
I'm wondering if the test for popcnt actually works? If I just compile the test program (cpu_popcnt.c) by hand and execute it, it seems to run ok. I edited it to return 0 instead of the 1 or 2 that its supposed to, and it doesn't seem to affect anything in the build.

To have the check for popcnt actually fail I put a syntax error into it so the compiler throws a fit and the test fails.

Its puzzling to me that the runtime test for POPCNT and compile time test are checking very different things. The compile-time test checks to see if code like:   
a = __builtin_popcountll(1);
executes, and this does seem to execute properly on this CPU (Core 2 Due E8400)
But the run-time code I think checks for a particular bit in a cpu register. 

Perhaps the compiler is out-smarting the test?
Comment 17 Carl Michal 2021-05-22 19:02:37 UTC
I think the test is borked. If I compile that test with gcc -S -mpopcnt
no popcnt instruction appears in the assembly. It looks like the compiler short circuits the test. If however I change this:
    #ifdef __x86_64__
    a = __builtin_popcountll(1);
    #endif
    b = __builtin_popcount(1)

to
    #ifdef __x86_64__
    a = __builtin_popcountll(1);
    #endif
    b = __builtin_popcount(a)

then the compiler includes:
	popcntl	%eax, %eax
which crashes the program with "Illegal Instruction" That's with gcc 10.2.0 and no optimization. If I include -O2, then the popcntl disappears even with this modification. Probably a random number should be chosen, or the pid, or something that's not known at compile time so the popcount can't be optimized away.

Or, the runtime test could be used. I'll file a bug upstream.
Comment 18 Carl Michal 2021-05-22 19:11:52 UTC
Upstream issue at: https://github.com/numpy/numpy/issues/19067
Comment 19 Carl Michal 2021-05-23 03:37:17 UTC
Created attachment 710367 [details, diff]
fix the compile time test for working popcnt

This patches cpu_popcnt.c so that it actually tests whether the popcnt instruction can be executed.
Comment 20 Carl Michal 2021-05-23 15:23:00 UTC
sorry - my first patch causes other problems. New one coming.
Comment 21 Carl Michal 2021-05-23 15:35:06 UTC
Created attachment 710691 [details, diff]
Actually fix the popcnt test

This is based on a suggestion of seiko2plus at numpy.
Comment 22 Carl Michal 2021-05-23 18:50:41 UTC
That second patch has been merged upstream.
Comment 23 Michal Plichta 2021-05-23 19:51:14 UTC
I apply chnages from patch and it helped - matpotlib merged
Comment 24 Michal Plichta 2021-05-23 20:06:34 UTC
(In reply to Michal Plichta from comment #23)
> I apply chnages from patch and it helped - matpotlib merged

I ment opencv... and not changing /usr/lib/python3.x/site-packages/numpy/distutils/checks/cpu_pop.cnt.c didn't helped....
Comment 25 Larry the Git Cow gentoo-dev 2021-05-24 22:09:03 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ca58a4b159282f564f046e035a17f7ce0bd30f01

commit ca58a4b159282f564f046e035a17f7ce0bd30f01
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-05-24 22:08:17 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-05-24 22:08:55 +0000

    dev-python/numpy: add popcnt patch
    
    Closes: https://bugs.gentoo.org/788184
    Signed-off-by: Sam James <sam@gentoo.org>

 .../files/numpy-1.20.2-fix-popcnt-detection.patch  | 103 +++++++++++++++++++++
 ...{numpy-1.20.2.ebuild => numpy-1.20.2-r1.ebuild} |   1 +
 dev-python/numpy/numpy-1.20.3.ebuild               |   1 +
 3 files changed, 105 insertions(+)
Comment 26 Larry the Git Cow gentoo-dev 2021-05-24 22:11:49 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bb670c5ea6d2257ff88e3cb513b7593794f1974a

commit bb670c5ea6d2257ff88e3cb513b7593794f1974a
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-05-24 22:11:23 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-05-24 22:11:23 +0000

    dev-python/numpy: revbump ~arch version (1.20.3) for popcnt fix too
    
    Bug: https://bugs.gentoo.org/788184
    Fixes: ca58a4b159282f564f046e035a17f7ce0bd30f01
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-python/numpy/{numpy-1.20.3.ebuild => numpy-1.20.3-r1.ebuild} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)