Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 660754 - >=dev-python/numpy-1.10.4: parallel build issue
Summary: >=dev-python/numpy-1.10.4: parallel build issue
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Science Related Packages
URL: https://github.com/numpy/numpy/issues...
Whiteboard:
Keywords:
: 623790 679298 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-07-08 21:45 UTC by Thomas Deutschmann (RETIRED)
Modified: 2023-06-04 20:01 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---

Attachments
build.log (build.log,322.82 KB, text/plain)
2018-07-08 21:45 UTC, Thomas Deutschmann (RETIRED) 		Details
failed build.log of numpy-1.18.3 with MAKEOPTS=-j2 (build-parallel.log,199.86 KB, text/x-log)
2020-04-27 09:24 UTC, Karl-Johan Karlsson 		Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Deutschmann (RETIRED) gentoo-dev 2018-07-08 21:45:26 UTC 
Created attachment 538880 [details]
build.log

See build.log for details.


Portage 2.3.40 (python 3.6.5-final-0, default/linux/x86/17.0, gcc-7.3.0, glibc-2.26-r7, 4.14.52-gentoo i686)
=================================================================
System uname: Linux-4.14.52-gentoo-i686-Intel-R-_Core-TM-_i7-3770K_CPU_@_3.50GHz-with-gentoo-2.4.1
KiB Mem:     3107600 total,    778568 free
KiB Swap:     488276 total,    488276 free
Timestamp of repository gentoo: Sun, 08 Jul 2018 19:44:24 +0000
Head commit of repository gentoo: 3e71460c2f988d8821b6d4c9efae2cc5999a4820

sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.26-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000

ABI="x86"
ABI_X86="32"
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
ACCEPT_PROPERTIES="*"
ACCEPT_RESTRICT="*"
ARCH="x86"
BROOT=""
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=pentium4m -mtune=pentium4m"
CHOST="i686-pc-linux-gnu"
CHOST_x86="i686-pc-linux-gnu"
COLLISION_IGNORE="/lib/modules/* *.py[co] *$py.class */dropin.cache"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CPU_FLAGS_X86="mmx mmxext sse sse2"
CXXFLAGS="-O2 -pipe -march=pentium4m -mtune=pentium4m"
DEFAULT_ABI="x86"
EDITOR="/usr/bin/mcedit"
ELIBC="glibc"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
EPREFIX=""
EROOT="/"
ESYSROOT="/"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-logs cgroup config-protect-if-modified distlocks downgrade-backup ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=i686 -pipe"
GCC_SPECS=""
GRUB_PLATFORMS="efi-32 pc"
GSETTINGS_BACKEND="dconf"
HOME="/root"
INFOPATH="/usr/share/gcc-data/i686-pc-linux-gnu/7.3.0/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.30/info:/usr/share/info"
INPUT_DEVICES="libinput keyboard mouse"
IUSE_IMPLICIT="abi_x86_32 prefix prefix-chain prefix-guest"
KERNEL="linux"
L10N="en en-US de de-DE"
LANG="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
LC_MESSAGES="C"
LC_PAPER="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LIBDIR_x86="lib"
LINGUAS="en de"
LOGNAME="root"
MAIL="/var/mail/root"
MAKEOPTS="--jobs 5 --load-average 7.95"
MULTILIB_ABIS="x86"
NOCOLOR="true"
OFFICE_IMPLEMENTATION="libreoffice"
OLDPWD="/root"
OPENCL_PROFILE="mesa"
OPENGL_PROFILE="xorg-x11"
PAGER="/usr/bin/less"
PATH="/usr/i686-pc-linux-gnu/gcc-bin/7.3.0:/usr/lib/llvm/5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin"
PHP_TARGETS="php5-6 php7-0"
POSTGRES_TARGETS="postgres9_5 postgres10"
PWD="/tmp/stable"
PYTHONDONTWRITEBYTECODE="1"
PYTHON_SINGLE_TARGET="python3_6"
PYTHON_TARGETS="python2_7 python3_6"
QT_GRAPHICSSYSTEM="raster"
ROOT="/"
ROOTPATH="/usr/i686-pc-linux-gnu/gcc-bin/7.3.0:/usr/lib/llvm/5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin"
RUBY_TARGETS="ruby22 ruby23"
SHELL="/bin/bash"
SHLVL="2"
SSH_TTY="/dev/pts/0"
SYSROOT="/"
TERM="tmux-256color"
TMUX="/tmp/tmux-0/default,5402,0"
TMUX_PANE="%2"
TWISTED_DISABLE_WRITING_OF_PLUGIN_CACHE="1"
USER="root"
USERLAND="GNU"
VIDEO_CARDS="vmware"
Comment 1 François Bissey 2018-07-08 21:48:43 UTC 
Is it after a recent upgrade of gcc?
Comment 2 Thomas Deutschmann (RETIRED) gentoo-dev 2018-07-08 21:55:21 UTC 
Mh, don't think so:

     Mon Jun 25 17:41:35 2018 >>> dev-python/numpy-1.10.4
     Sun Jul  1 21:25:20 2018 >>> dev-python/numpy-1.10.4
     Wed Jul  4 01:34:20 2018 >>> dev-python/numpy-1.10.4
     Wed Jul  4 01:42:25 2018 >>> dev-python/numpy-1.10.4
     Fri Jul  6 23:17:27 2018 >>> dev-python/numpy-1.10.4


     Wed Dec 20 23:40:20 2017 >>> sys-devel/gcc-6.4.0
     Fri Jan 12 16:51:32 2018 >>> sys-devel/gcc-6.4.0-r1
     Sat Mar  3 23:53:15 2018 >>> sys-devel/gcc-6.4.0-r1
     Tue Jun 19 13:34:22 2018 >>> sys-devel/gcc-7.3.0-r3
Comment 3 François Bissey 2018-07-08 22:01:17 UTC 
(In reply to Thomas Deutschmann from comment #2)
> Mh, don't think so:
> 
>      Mon Jun 25 17:41:35 2018 >>> dev-python/numpy-1.10.4
>      Sun Jul  1 21:25:20 2018 >>> dev-python/numpy-1.10.4
>      Wed Jul  4 01:34:20 2018 >>> dev-python/numpy-1.10.4
>      Wed Jul  4 01:42:25 2018 >>> dev-python/numpy-1.10.4
>      Fri Jul  6 23:17:27 2018 >>> dev-python/numpy-1.10.4
> 
> 
>      Wed Dec 20 23:40:20 2017 >>> sys-devel/gcc-6.4.0
>      Fri Jan 12 16:51:32 2018 >>> sys-devel/gcc-6.4.0-r1
>      Sat Mar  3 23:53:15 2018 >>> sys-devel/gcc-6.4.0-r1
>      Tue Jun 19 13:34:22 2018 >>> sys-devel/gcc-7.3.0-r3

OK so that's not change of fortran soname. That would have been so easy. Instead it looks more like the parallel building bug that has been plaguing scipy for a while
/var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-i686-3.6/numpy/linalg/lapack_lite/python_xerbla.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
/var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-i686-3.6/numpy/linalg/lapack_lite/python_xerbla.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status

Try again at "-j1".
Comment 4 Thomas Deutschmann (RETIRED) gentoo-dev 2018-07-08 22:47:51 UTC 
Yup, looks like a parallel build issue: Building with MAKEOPTS=-j2 successful.
Comment 5 Alexander Tsoy 2018-08-22 11:23:13 UTC 
(In reply to Thomas Deutschmann from comment #4)
> Building with MAKEOPTS=-j2 successful.

I can build only with MAKEOPTS=-j1. Got the same error as you with MAKEOPTS=-j2:

/usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'
  warnings.warn(msg)
/usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
error: Command "x86_64-pc-linux-gnu-gcc -O0 -fno-omit-frame-pointer -ggdb -pipe -shared -Wl,-O1 -Wl,--as-needed -shared -O0 -fno-omit-frame-pointer -ggdb -pipe -fno-strict-aliasing /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_litemodule.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/python_xerbla.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/zlapack_lite.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/dlapack_lite.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/blas_lite.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/dlamch.o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/f2c_lite.o -L/usr/lib64 -L/var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/temp.linux-x86_64-3.6 -lpython3.6m -o /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/lib/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so" failed with exit status 1
Comment 6 Tomáš Mózes 2018-09-20 12:43:06 UTC 
(In reply to Alexander Tsoy from comment #5)
> (In reply to Thomas Deutschmann from comment #4)
> > Building with MAKEOPTS=-j2 successful.
> 
> I can build only with MAKEOPTS=-j1. Got the same error as you with
> MAKEOPTS=-j2:
> 
> /usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown
> distribution option: 'test_suite'
>   warnings.warn(msg)
> /usr/lib64/python3.6/distutils/dist.py:261: UserWarning: Unknown
> distribution option: 'define_macros'
>   warnings.warn(msg)
> error: Command "x86_64-pc-linux-gnu-gcc -O0 -fno-omit-frame-pointer -ggdb
> -pipe -shared -Wl,-O1 -Wl,--as-needed -shared -O0 -fno-omit-frame-pointer
> -ggdb -pipe -fno-strict-aliasing
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_litemodule.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/python_xerbla.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/zlapack_lite.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/dlapack_lite.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/blas_lite.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/dlamch.o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6/numpy/linalg/lapack_lite/f2c_lite.o -L/usr/lib64
> -L/var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> temp.linux-x86_64-3.6 -lpython3.6m -o
> /var/tmp/portage/dev-python/numpy-1.10.4/work/numpy-1.10.4-python3_6/build/
> lib/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so" failed with
> exit status 1

Got the same error on ~amd64 with numpy 1.14.5. MAKEOPTS="-j1" emerge numpy works.
Comment 7 Larry the Git Cow gentoo-dev 2018-09-20 12:58:57 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0b5b4b4c8770a61a9eb28ca5fc45ceca65a7ee6d

commit 0b5b4b4c8770a61a9eb28ca5fc45ceca65a7ee6d
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2018-09-20 12:50:41 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2018-09-20 12:58:48 +0000

    dev-python/numpy: force MAKEOPTS=-j1
    
    Looks like nobody cares. To allow users to build that package in
    the meanwhile, let's force MAKEOPTS=-j1.
    
    Bug: https://bugs.gentoo.org/660754
    Package-Manager: Portage-2.3.49, Repoman-2.3.10

 dev-python/numpy/numpy-1.10.4.ebuild | 2 ++
 dev-python/numpy/numpy-1.13.3.ebuild | 2 ++
 dev-python/numpy/numpy-1.14.5.ebuild | 2 ++
 dev-python/numpy/numpy-9999.ebuild   | 2 ++
 4 files changed, 8 insertions(+)
Comment 8 Larry the Git Cow gentoo-dev 2019-03-02 22:57:39 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=885d5f81e25739a1b088e0d69369d2db992454cc

commit 885d5f81e25739a1b088e0d69369d2db992454cc
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2019-03-02 22:57:21 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2019-03-02 22:57:21 +0000

    dev-python/numpy: drop parallel build workaround
    
    Cannot reproduce with >numpy-1.13.3 anymore.
    
    Bug: https://bugs.gentoo.org/660754
    Package-Manager: Portage-2.3.62, Repoman-2.3.12
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 dev-python/numpy/numpy-1.14.5.ebuild | 4 +---
 dev-python/numpy/numpy-1.15.4.ebuild | 2 --
 dev-python/numpy/numpy-9999.ebuild   | 2 --
 3 files changed, 1 insertion(+), 7 deletions(-)
Comment 9 Thomas Deutschmann (RETIRED) gentoo-dev 2019-03-03 16:15:48 UTC 
*** Bug 679298 has been marked as a duplicate of this bug. ***
Comment 10 Thomas Deutschmann (RETIRED) gentoo-dev 2019-03-03 16:22:21 UTC 
Looks like this is still happening (bug 679298). But it looks like it is only happening for USE=-lapack. I'll push an updated ebuild shortly which will force MAKEOPTS=-j1 only for USE=-lapack... please test and report.
Comment 11 Larry the Git Cow gentoo-dev 2019-03-03 16:25:40 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=09f3824f30a37092664909b512043b0321e3d14d

commit 09f3824f30a37092664909b512043b0321e3d14d
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2019-03-03 16:23:05 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2019-03-03 16:25:31 +0000

    dev-python/numpy: re-add parallel make workaround for USE=-lapack
    
    Bug: https://bugs.gentoo.org/660754
    Package-Manager: Portage-2.3.62, Repoman-2.3.12
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 dev-python/numpy/numpy-1.13.3.ebuild | 4 ++--
 dev-python/numpy/numpy-1.14.5.ebuild | 2 ++
 dev-python/numpy/numpy-1.15.4.ebuild | 2 ++
 dev-python/numpy/numpy-1.16.1.ebuild | 2 ++
 dev-python/numpy/numpy-9999.ebuild   | 2 ++
 5 files changed, 10 insertions(+), 2 deletions(-)
Comment 12 Larry the Git Cow gentoo-dev 2019-03-03 16:29:31 UTC 
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d5337dedc68adc4dfcbfe8bb045c96554512aabf

commit d5337dedc68adc4dfcbfe8bb045c96554512aabf
Author:     Thomas Deutschmann <whissi@gentoo.org>
AuthorDate: 2019-03-03 16:29:22 +0000
Commit:     Thomas Deutschmann <whissi@gentoo.org>
CommitDate: 2019-03-03 16:29:22 +0000

    dev-python/numpy: force MAKEOPTS=-j1 for all USE configurations
    
    ...even failing for USE=lapack on some systems.
    
    Bug: https://bugs.gentoo.org/660754
    Package-Manager: Portage-2.3.62, Repoman-2.3.12
    Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>

 dev-python/numpy/numpy-1.13.3.ebuild | 2 +-
 dev-python/numpy/numpy-1.14.5.ebuild | 2 +-
 dev-python/numpy/numpy-1.15.4.ebuild | 2 +-
 dev-python/numpy/numpy-1.16.1.ebuild | 2 +-
 dev-python/numpy/numpy-9999.ebuild   | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)
Comment 13 Karl-Johan Karlsson 2020-04-27 07:56:55 UTC 
It seems the dev-python/numpy-1.18.3 ebuild removed the forcing of MAKEOPTS=-j1, so now I'm seeing the same build failures again. When manually specifying MAKEOPTS=-j1 to 1.18.3 it builds successfully.
Comment 14 Benda Xu gentoo-dev 2020-04-27 09:15:09 UTC 
*** Bug 623790 has been marked as a duplicate of this bug. ***
Comment 15 Karl-Johan Karlsson 2020-04-27 09:24:51 UTC 
Created attachment 634836 [details]
failed build.log of numpy-1.18.3 with MAKEOPTS=-j2
Comment 16 Larry the Git Cow gentoo-dev 2020-04-27 10:08:30 UTC 
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0d784579f7d2fa66ab44c80e2ff0fba353f08dbf

commit 0d784579f7d2fa66ab44c80e2ff0fba353f08dbf
Author:     Benda Xu <heroxbd@gentoo.org>
AuthorDate: 2020-04-27 10:03:44 +0000
Commit:     Benda Xu <heroxbd@gentoo.org>
CommitDate: 2020-04-27 10:06:39 +0000

    dev-python/numpy: fix regressions introduced by 48dd16c7839fd.
    
    Bug: https://github.com/gentoo/gentoo/pull/15315
    Closes: https://bugs.gentoo.org/660754
    Package-Manager: Portage-2.3.88, Repoman-2.3.18
    Signed-off-by: Benda Xu <heroxbd@gentoo.org>

 dev-python/numpy/numpy-1.18.3.ebuild   | 17 +++++------------
 profiles/arch/powerpc/package.use.mask |  4 ++++
 profiles/arch/sparc/package.use.mask   |  4 ++++
 3 files changed, 13 insertions(+), 12 deletions(-)
Comment 17 cJ 2022-04-01 19:42:40 UTC 
Scipy is in process of switching to meson:
 https://github.com/scipy/scipy/issues/7112
 https://github.com/scipy/scipy/issues/13615
and numpy & al will probably follow suit.

With some luck, in a few years this limitation will be gone!