Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 700848 - dev-python/numpy-1.17.4 fails to build
Summary: dev-python/numpy-1.17.4 fails to build
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 3 votes (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
: 519984 702328 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-11-21 06:58 UTC by Patrick Lauer
Modified: 2020-04-27 10:20 UTC (History)
17 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Lauer gentoo-dev 2019-11-21 06:58:20 UTC
* python3_6: running distutils-r1_run_phase python_compile
python3.6 setup.py build -j 1 config_fc --noopt --noarch --fcompiler=gnu95
blas_opt_info:
system_info:
  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

blas_info:
customize UnixCCompiler
customize UnixCCompiler
C compiler: x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -flto=4 -fuse-linker-plugin -O2 -pipe -march=native -flto=4 -fuse-linker-plugin -fno-strict-aliasing -fPIC

creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage/dev-python
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage/dev-python/numpy-1.17.4
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage/dev-python/numpy-1.17.4/temp
creating /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua
compile options: '-I -I/usr/include -c'
x86_64-pc-linux-gnu-gcc: /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/source.c
x86_64-pc-linux-gnu-gcc -O2 -pipe -march=native -flto=4 -fuse-linker-plugin /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/source.o -L/usr/lib64 -lblas -lcblas -o /var/tmp/portage/dev-python/numpy-1.17.4/temp/tmpeyz2xqua/a.out
  FOUND:
    libraries = [['blas', 'cblas']]
    library_dirs = ['/usr/lib64']
    include_dirs = ['', '/usr/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = [['blas', 'cblas']]
    library_dirs = ['/usr/lib64']
    include_dirs = ['', '/usr/include']
    language = c

/bin/sh: svnversion: command not found
lapack_opt_info:
flame_info:
customize UnixCCompiler
  libraries flame not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

lapack_info:
customize UnixCCompiler
  FOUND:
    libraries = ['lapack']
    library_dirs = ['/usr/lib64']
    language = f77

  FOUND:
    libraries = ['lapack', ['blas', 'cblas']]
    library_dirs = ['/usr/lib64']
    language = c
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    include_dirs = ['', '/usr/include']

running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
Running from numpy source directory.
/var/tmp/portage/dev-python/numpy-1.17.4/work/numpy-1.17.4/numpy/distutils/system_info.py:669: UserWarning:
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  self.calc_info()
/usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
error: second element of each tuple in 'libraries' must be a dictionary (build info)
 * ERROR: dev-python/numpy-1.17.4::gentoo failed (compile phase):
 *   (no error message)
Comment 1 Patrick Lauer gentoo-dev 2019-11-21 06:59:09 UTC
Portage 2.3.79 (python 3.6.9-final-0, default/linux/amd64/17.0, gcc-9.2.0, glibc-2.30-r2, 5.2.13-gentoo x86_64)
=================================================================
System uname: Linux-5.2.13-gentoo-x86_64-AMD_FX-tm-8350_Eight-Core_Processor-with-gentoo-2.6
KiB Mem:    32928272 total,  12761096 free
KiB Swap:          0 total,         0 free
sh bash 5.0_p11
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          5.0_p11::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.17::gentoo, 3.6.9::gentoo
dev-util/cmake:           3.15.5::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r2::gentoo
sys-devel/binutils:       2.32-r1::gentoo, 2.33.1::gentoo
sys-devel/gcc:            9.2.0-r2::gentoo
sys-devel/gcc-config:     2.1::gentoo
sys-devel/libtool:        2.4.6-r5::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.3-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.30-r2::gentoo
Repositories:

gentoo
    location: /home/me/code/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts:
    sync-rsync-verify-metamanifest: no

adjust
    location: /home/me/code/adjust/gentoo-overlay
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -flto=4 -fuse-linker-plugin"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -flto=4 -fuse-linker-plugin"
DISTDIR="/chroot/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="/etc/X11/xinit/xinitrc.d/80-dbus /usr/bin/dbus-launch"
LANG="en_US.utf8"
LDFLAGS="-O2 -flto=4"
MAKEOPTS="-j8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X acl alsa amd64 apng berkdb bzip2 cli crypt cups cxx dri ffmpeg fortran gdbm glamor gmp gpo iconv icu ipv6 lapack libtirpc minizip multilib ncurses nls nptl opencl opengl openmp pam pcre pgo readline seccomp split-usr sqlite ssl tcpd truetype udev unicode vdpau vulkan xattr xvmc zlib" ABI_X86="64 32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx f16c fma3 fma4 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="radeon r600 radeonsi amdgpu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 MaxKarlErnstPlanck 2019-11-29 17:32:20 UTC
Message -- error: second element of each tuple in 'libraries' must be a dictionary (build info) -- comes from
/usr/lib/python3.*/distutils/command/build_clib.py
from "def check_library_list(self, libraries)"
from loop "for lib in libraries:"

On my system "libraries" is list of 3 elem:
[('npymath', {'include_dirs': [], ... 'depends': []}),
 ('npysort', {'include_dirs': [], ... 'depends': []}),
 ('blas', 'cblas')]

last string -- 'cblas' -- is obviously not dictionary

In my desperate daring I patched distutils/command/build_clib.py to replace 'cblas' with {'include_dirs': [], 'depends': []}  and it worked.
Comment 3 MaxKarlErnstPlanck 2019-11-29 17:38:46 UTC
And now it's
/usr/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: cblas_dsyrk
Comment 4 Michelangelo Scopelliti 2019-12-06 20:59:07 UTC
I was able to build numpy-1.17.4 reverting the no-hardcoded-blas patch:

# ebuild `equery w numpy` clean prepare
# patch -RNp1 -i /var/db/repos/gentoo/dev-python/numpy/files/numpy-1.17.4-no-hardcode-blas.patch -d /var/tmp/portage/dev-python/numpy-1.17.4/work/numpy-1.17.4/
# ebuild `equery w numpy` merge clean

On a side note: on my system(s), the generated site.cfg looks bizarre (to me) because of the presence of colons:

cat site.cfg
[blas]
include_dirs = :/usr/include
library_dirs = :/usr/lib64
blas_libs = blas,cblas
[lapack]
library_dirs = :/usr/lib64
lapack_libs = lapack

Is that the expected result?

Additional info
On my systems, sci-libs/lapack-3.8.0:0, sci-libs/openblas-0.3.7:0, virtual/blas-3.8:0, virtual/cblas-3.8:0 and virtual/lapack-3.8 are built with USE=eselect-ldso
Comment 5 LE GARREC Vincent 2019-12-07 03:27:00 UTC
Right, the patch is maybe buggy.
Comment 6 Paul Zander 2019-12-10 01:41:46 UTC
The last segment of the patch can be removed, the compile will then succeed.

The call to pc_incdir() should be removed, as $(pkg-config --cflags-only-I) is empty for blas, cblas and lapack. 
The call to pc_libdir() should be removed, as $(pkg-config --libs-only-L) is empty for blas, cblas and lapack.
Comment 7 Patrick Lauer gentoo-dev 2019-12-29 16:11:51 UTC
Fixed in 817df86ab2fc967c40f1d2abe50a43bab24917df
Comment 8 David Seifert gentoo-dev 2019-12-30 17:26:00 UTC
*** Bug 702328 has been marked as a duplicate of this bug. ***
Comment 9 Benda Xu gentoo-dev 2020-01-20 08:13:05 UTC
(In reply to Patrick Lauer from comment #7)
> Fixed in 817df86ab2fc967c40f1d2abe50a43bab24917df

Patrick, what was your motivation making the following change along with the patch update?   There are needed for numpy on Prefix to locate the correct locations of blas and lapack.

@@ -74,11 +74,8 @@
                local libdir="${EPREFIX}"/usr/$(get_libdir)
                cat >> site.cfg <<-EOF || die
                        [blas]
-                       include_dirs = $(pc_incdir cblas):${incdir}
-                       library_dirs = $(pc_libdir cblas blas):${libdir}
                        blas_libs = $(pc_libs cblas blas)
                        [lapack]
-                       library_dirs = $(pc_libdir lapack):${libdir}
                        lapack_libs = $(pc_libs lapack)
                EOF
        else
Comment 10 Larry the Git Cow gentoo-dev 2020-01-25 03:25:06 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=261f747b27a57db8c9da8fc9f2d1a991a5db350e

commit 261f747b27a57db8c9da8fc9f2d1a991a5db350e
Author:     Benda Xu <heroxbd@gentoo.org>
AuthorDate: 2020-01-25 03:20:43 +0000
Commit:     Benda Xu <heroxbd@gentoo.org>
CommitDate: 2020-01-25 03:24:56 +0000

    dev-python/numpy: replace pc_incdir() with plain names.
    
      Same goes to pc_libs() and pc_libdir(). pkg-config mechanism is not
      necessary any more with the introduction of eselect-ldso. All the
      compile and linking time operation are done against standard blas
      and lapack headers and libraries.
    
    Closes: https://bugs.gentoo.org/700848
    Package-Manager: Portage-2.3.79, Repoman-2.3.18
    Signed-off-by: Benda Xu <heroxbd@gentoo.org>

 dev-python/numpy/numpy-1.17.4-r2.ebuild | 139 ++++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)
Comment 11 Benda Xu gentoo-dev 2020-04-27 10:20:08 UTC
*** Bug 519984 has been marked as a duplicate of this bug. ***