Using lapack-reference and gslcblas to build numpy-1.6.0 and then building scipy results in undefined symbols in scipy/linalg/clapack.so. This issue surfaced on gentoo-science. See http://archives.gentoo.org/gentoo-science/msg_83c0a72f758099201944bdccd401603a.xml The issue isn't present if numpy-1.5.1 is used. Reproducible: Always Steps to Reproduce: 1.eselect lapack set reference 2.eselect blas set reference 3.eselect cblas set gsl 4.emerge =dev-python/numpy-1.6.0 && emerge =sci-libs/scipy-0.9.0 Actual Results: ldd -r /storage/strogdon/gentoo/usr/lib(64)/python2.7/site packages/scipy/linalg/clapack.so gives: linux-vdso.so.1 => (0x00007fffbb1c7000) libpython2.7.so.1.0 => /storage/strogdon/gentoo/usr/lib/libpython2.7.so.1.0 ( 0x00007f1fe376d000) libc.so.6 => /lib/libc.so.6 (0x00007f1fe33f9000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f1fe31dd000) libdl.so.2 => /lib/libdl.so.2 (0x00007f1fe2fd9000) libutil.so.1 => /lib/libutil.so.1 (0x00007f1fe2dd5000) libm.so.6 => /lib/libm.so.6 (0x00007f1fe2b53000) /lib64/ld-linux-x86-64.so.2 (0x00007f1fe3d3d000) undefined symbol: clapack_sgesv (/storage/strogdon/gentoo/usr/lib/python2.7/site-pack ages/scipy/linalg/clapack.so) undefined symbol: clapack_dgesv (/storage/strogdon/gentoo/usr/lib/python2.7/site-pack ages/scipy/linalg/clapack.so) ... Expected Results: With numpy-1.5.1 the result is linux-vdso.so.1 => (0x00007fffa1db1000) libpython2.7.so.1.0 => /storage/strogdon/gentoo/usr/lib/libpython2.7.so.1.0 ( 0x00007fd3fe01b000) libc.so.6 => /lib/libc.so.6 (0x00007fd3fdca7000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007fd3fda8b000) libdl.so.2 => /lib/libdl.so.2 (0x00007fd3fd887000) libutil.so.1 => /lib/libutil.so.1 (0x00007fd3fd683000) libm.so.6 => /lib/libm.so.6 (0x00007fd3fd401000) /lib64/ld-linux-x86-64.so.2 (0x00007fd3fe5d2000) By comparing logs of building scipy with numpy-1.5.1 and with numpy-1.6.0, we get a sense of where things go awry. With numpy-1.5.1 the clapack Block is "empty": Post-processing... Block: clapack Block: empty_module while with numpy-1.6.0 we get: Post-processing... Block: clapack Block: sgesv Block: dgesv Block: cgesv The same is true relative to the cblas Block. With numpy-1.5.1 this Block is "empty" Post-processing... Block: cblas Block: empty_module Post-processing (stage 2)... while with numpy-1.6.0 we get Post-processing... Block: cblas Block: saxpy Block: daxpy Block: caxpy Now there is some indication that the problem is somewhere in the distutils that's shipped with numpy-1.6.0. If I unpack numpy-1.6.0, replace the numpy/distutils folder with the distutils that's shipped with numpy-1.5.1, merge the "new" numpy-1.6.0 into the live system, then upon building scipy there are no undefined symbols in clapack.so - the results being the same as if numpy-1.5.1 had been used. The build log for scipy looks very similar to that obtained when building scipy with numpy-1.5.1 - we have the "empty" Blocks for clapack and cblas. I'll post the logs I have.
Created attachment 276603 [details] build log of scipy-0.9.0 using numpy-1.5.1 build log of scipy-0.9.0 using numpy-1.5.1
Created attachment 276605 [details] build log of scipy-0.9.0 using numpy-1.6.0 build log of scipy-0.9.0 using numpy-1.6.0
I can confirm this bug. Downgrading to numpy-1.5.1 solves the issue. It showed up after I rebuilt scipy-0.9.0-r1.
i, too, have run into this same problem. i added... >=dev-python/numpy-1.6.0 ...to /etc/portage/package.mask which downgraded numpy on re-emerge. i also had to rebuild matplotlib. after that i was able to use scipy.stats, whose import was causing the issue. interestingly, i don't think this problem exist on my other desktop box. the one that has an issues is an x86 while the one that does not is an amd64.
That's interesting I don't believe I had the problem on amd64 either. However there may be another factor. Do you use bicatali's overlay for blas/cblas/lapack on either box?
(In reply to comment #5) > That's interesting I don't believe I had the problem on amd64 either. However > there may be another factor. Do you use bicatali's overlay for > blas/cblas/lapack on either box? nope, i'm not using bicatali's overlay on either x86 or amd64.
(In reply to comment #5) > That's interesting I don't believe I had the problem on amd64 either. However > there may be another factor. Do you use bicatali's overlay for > blas/cblas/lapack on either box? Now the logs I posted were on amd64, in prefix and with portage provided lapack/blas. The problem still persists here on another amd64, normal gentoo install but with bicatali's overlay.
(In reply to comment #4) > i, too, have run into this same problem. i added... > > >=dev-python/numpy-1.6.0 > > ...to /etc/portage/package.mask which downgraded numpy on re-emerge. i also > had to rebuild matplotlib. after that i was able to use scipy.stats, whose > import was causing the issue. > > interestingly, i don't think this problem exist on my other desktop box. the > one that has an issues is an x86 while the one that does not is an amd64. Ok given the other comments I think a good starting point would be the emerge --info on both machines.
Portage 2.1.9.42 (default/linux/amd64/10.0/desktop, gcc-4.4.5, libc-0-r0, emerge --info (amd64, native gentoo install, bicatali overlay) 2.6.36-gentoo-r5 x86_64) ================================================================= System uname: Linux-2.6.36-gentoo-r5-x86_64-Dual-Core_AMD_Opteron-tm-_Processor_2216-with-gentoo-2.0.2 Timestamp of tree: Fri, 01 Jul 2011 03:15:01 +0000 ccache version 2.4 [enabled] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.5.4-r4, 2.6.6-r2::sage-on-gentoo, 2.7.1-r1::sage-on-gentoo, 3.1.3-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.8.4-r1 dev-util/pkgconfig: 0.25-r2 sys-apps/baselayout: 2.0.2 sys-apps/openrc: 0.8.2-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13, 2.65-r1 sys-devel/automake: 1.5::<unknown repository>, 1.6.3::<unknown repository>, 1.7.9-r1, 1.8.5-r3::<unknown repository>, 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.1.2, 4.3.4, 4.4.5 sys-devel/gcc-config: 1.4.1-r1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.82 sys-kernel/linux-headers: 2.6.36.1 sys-libs/glibc: 2.12.2 virtual/os-headers: 0 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA googleearth AdobeFlash-10 AdobeFlash-10.1" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=opteron -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=opteron -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache collision-protect distlocks fixlafiles fixpackages multilib-strict news parallel-fetch protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--delete-after --timeout=500" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/var/lib/layman/desktop-effects /var/lib/layman/science /var/lib/layman/sunrise /var/lib/layman/x11 /var/lib/layman/sage-on-gentoo /var/lib/layman/bicatali /usr/local/portage" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="3dnow X Xaw3d a52 aac aalib acl acpi alsa amd64 apache2 audiofile bash-completion berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gdu gif gphoto2 gpm gstreamer gtk iconv ieee1394 imlib ipv6 java jpeg lcms ldap lesstif libnotify lm_sensors mad midi mmx mng modules motif mozilla mp3 mp4 mpeg mudflap multilib ncurses nls nptl nptlonly nsplugin ogg opengl openmp oss pam pango pcre pdf perl png policykit ppds pppd python qt3support qt4 radius readline sdl session spell sse sse2 ssl startup-notification svg sysfs tcl tcpd tiff tk truetype udev unicode usb vorbis x264 xcb xml xorg xulrunner xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia fbdev vesa vga nv" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
emerge --info (amd64, prefix, portage provided lapack/blas) Portage 2.2.01.18826-prefix (prefix/linux/amd64, gcc-4.5.2, unavailable, 2.6.32-5-amd64 x86_64) ================================================================= System uname: Linux-2.6.32-5-amd64-x86_64-Six-Core_AMD_Opteron-tm-_Processor_2427-with-debian-6.0.2 Timestamp of tree: Fri, 01 Jul 2011 03:36:45 +0000 ccache version 3.1.4 [enabled] app-shells/bash: 4.2_p10 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.6.6-r2::sage-on-gentoo, 2.7.2::sage-on-gentoo dev-util/ccache: 3.1.4 dev-util/cmake: 2.8.4 dev-util/pkgconfig: 0.25-r2 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13, 2.68 sys-devel/automake: 1.11.1 sys-devel/binutils: 2.21.51.0.7 sys-devel/gcc: 4.2.4-r01.2, 4.4.5, 4.5.2-r00.1 sys-devel/gcc-config: 1.4.1-r00.2 sys-devel/libtool: 2.4-r01.1 sys-devel/make: 3.82 sys-kernel/linux-headers: 2.6.38 Repositories: gentoo_prefix local-overlay sage-on-gentoo sage-on-prefix Installed sets: ACCEPT_KEYWORDS="~amd64-linux" ACCEPT_LICENSE="* -@EULA dlj-1.1" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=opteron -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/portage /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=opteron -O2 -pipe" DISTDIR="/storage/strogdon/gentoo/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache collision-protect distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j13" PKGDIR="/storage/strogdon/gentoo/usr/portage/packages" PORTAGE_CONFIGROOT="/storage/strogdon/gentoo/" PORTAGE_RSYNC_EXTRA_OPTS="--delete-after --timeout=500" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/storage/strogdon/gentoo/var/tmp" PORTDIR="/storage/strogdon/gentoo/usr/portage" PORTDIR_OVERLAY="/storage/strogdon/gentoo/usr/local/portage /storage/strogdon/gentoo/usr/local/portage/sage-on-gentoo /storage/strogdon/gentoo/usr/local/portage/sage-on-prefix" SYNC="rsync://rsync.prefix.freens.org/gentoo-portage-prefix" USE="amd64 berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 mmx modules mudflap ncurses nls nptl nptlonly openmp pcre perl pppd prefix python readline session sse sse2 ssl sysfs tcpd unicode xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS
emerge --info from my amd64 machine, which has scipy-0.9.0-r1, numpy-1.6.0, and blas-reference-20070226... Portage 2.1.9.42 (default/linux/amd64/10.0/desktop, gcc-4.4.5, libc-0-r0, 2.6.38-gentoo-r6 x86_64) ================================================================= System uname: Linux-2.6.38-gentoo-r6-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4200+-with-gentoo-2.0.2 Timestamp of tree: Sun, 26 Jun 2011 05:15:01 +0000 distcc 3.1 x86_64-pc-linux-gnu [enabled] ccache version 2.4 [enabled] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.1-r1, 3.1.3-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.8.4-r1 sys-apps/baselayout: 2.0.2 sys-apps/openrc: 0.8.2-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13::<unknown repository>, 2.65-r1 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5 sys-devel/gcc-config: 1.4.1-r1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.82 sys-kernel/linux-headers: 2.6.36.1 sys-libs/glibc: 2.12.2 virtual/os-headers: 0 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=athlon64 -O2 -pipe -msse3" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=athlon64 -O2 -pipe -msse3" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache collision-protect distcc distlocks fixlafiles fixpackages metadata-transfer news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://gentoo.arcticnetwork.ca/ http://gentoo.wetzlmayr.com/ http://gentoo.netnitco.net" LANG="en_US.utf8" LC_ALL="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en ja" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage /var/lib/layman/wschlich-testing /var/lib/layman/pd-overlay /var/lib/layman/arduino /var/lib/layman/gnustep" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="3dnowext X Xaw3d a52 aac acpi alsa amd64 apache2 bash-completion bcmath berkdb branding bzip2 cairo cddb cdr cjk cleartype cli consolekit corefonts cracklib crypt cups cxx dbus djvu doc dri dts dvb dvd dvdr emacs emboss encode examples exif faac fam firefox flac fontconfig fortran gcj gd gdbm gdu gif gimp gmp gnome gnustep gnutls graphviz gtk iconv id3tag imap ipv6 jadetex java jbig jpeg jpeg2k kde lapack latex lcms libedit libnotify mad mbox midi mmx mng modules motif mp3 mp4 mpeg mudflap multilib musepack nas nautilus ncurses nls nptl nptlonly nsplugin nvidia ofx ogg openexr opengl openmp pam pango pcre pdf perl php plotutils png policykit pop ppds pppd python qt3 qt3support qt4 readline sasl sdl session skey spell sqlite sqlite3 sse sse2 ssl startup-notification subversion svg sysfs taglib tcl tcpd theora threads tiff tk truetype twolame type1 udev unicode usb vdpau vim-syntax vorbis wmf wxwidgets x264 xcb xml xorg xpm xulrunner xv xvid xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en ja" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" SANE_BACKENDS="epson epson2" USERLAND="GNU" VIDEO_CARDS="nv nvidia vesa vga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
emerge --info from my x86 machine, which has scipy-0.9.0-r1, numpy-1.5.1, and blas-reference-20070226... [btw, after going through this is a greater level of detail here, i noted that this bug report is specifically for =scipy-0.9.0 while i'm using scipy-0.9.0-r1. nonetheless, the same problem exists and, in my case, only on my x86 box. just wanted to point this out as i didn't realize the '-r1' version difference.] Portage 2.1.9.42 (default/linux/x86/10.0/desktop, gcc-4.4.5, libc-0-r0, 2.6.38-gentoo-r6 i686) ================================================================= System uname: Linux-2.6.38-gentoo-r6-i686-Intel-R-_Pentium-R-_4_CPU_3.00GHz-with-gentoo-2.0.2 Timestamp of tree: Sun, 26 Jun 2011 05:15:01 +0000 distcc 3.1 i686-pc-linux-gnu [enabled] ccache version 2.4 [enabled] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.1-r1, 3.1.3-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.8.4-r1 sys-apps/baselayout: 2.0.2 sys-apps/openrc: 0.8.2-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13::<unknown repository>, 2.65-r1 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5 sys-devel/gcc-config: 1.4.1-r1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.82 sys-kernel/linux-headers: 2.6.36.1 sys-libs/glibc: 2.12.2 virtual/os-headers: 0 ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=pentium4 -O2 -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=pentium4 -O2 -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs ccache collision-protect distcc distlocks fixlafiles fixpackages metadata-transfer news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://gentoo.gossamerhost.com http://gentoo.arcticnetwork.ca/ http://gentoo.mirrors.tera-byte.com/" LANG="en_US.utf8" LC_ALL="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en ja" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage /var/lib/layman/pd-overlay /var/lib/layman/wschlich-testing /var/lib/layman/arduino" SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" USE="X Xaw3d a52 aac acpi alsa apache2 bash-completion bcmath berkdb branding bzip2 cairo cdr cjk cleartype cli consolekit corefonts cracklib crypt cups cxx dbus djvu doc dri dts dvd dvdr emacs emboss encode examples exif faac fam firefox flac fontconfig fortran gcj gd gdbm gdu gif gimp gmp gnome gnustep gnutls graphviz gtk iconv imap ipv6 jadetex java jbig jpeg jpeg2k kde lapack latex lcms libedit libnotify mad mbox midi mmx mng modules motif mp3 mp4 mpeg mudflap nas nautilus ncurses nls nptl nptlonly nsplugin nvidia ogg openexr opengl openmp pam pango pcre pdf perl php plotutils png policykit pop ppds pppd python qt3support qt4 readline sasl sdl session skey spell sqlite sqlite3 ssl startup-notification subversion svg sysfs tcl tcpd threads tiff tk truetype type1 udev unicode usb vim-syntax vorbis wmf wxwidgets x264 x86 xcb xml xorg xpm xulrunner xv xvid xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en ja" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nv nvidia vesa vga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
(In reply to comment #12) > emerge --info from my x86 machine, which has scipy-0.9.0-r1, numpy-1.5.1, and > blas-reference-20070226... > > [btw, after going through this is a greater level of detail here, i noted that > this bug report is specifically for =scipy-0.9.0 while i'm using > scipy-0.9.0-r1. nonetheless, the same problem exists and, in my case, only on > my x86 box. just wanted to point this out as i didn't realize the '-r1' > version difference.] > I'm curious. On your amd64 box with the installed numpy-1.6.0, did you rebuild scipy after the numpy upgrade? Here if I install numpy-1.6.0 without rebuilding scipy, I get no undefined symbols in clapack.so. It is only after scipy is rebuilt that things get messed up. I don't think the -r1 version will make a difference. It is scipy-0.9.0-r1 that was used when the problem surfaced on gentoo-science, http://archives.gentoo.org/gentoo-science/msg_83c0a72f758099201944bdccd401603a.xml
i'm not entirely sure. this is all the result of a recent world update. i was likely ~30 out of date, then updated both of these boxes at the same time. i don't know what my version of scipy & numpy prior to update, and i also don't know the order of the update (numpy before scipy or vise versa). i am fairly certain, however, that i didn't have these packages on my x86 box (the one with the issue) --- i took them out during a previous world update, perhaps due to conflicts (and i didn't need them at the time). i emerged them by hand on the x86 box. is there something i can try on my x86 box? re-emerge scipy then numpy (or vise versa)...
(In reply to comment #14) > i'm not entirely sure. this is all the result of a recent world update. i was > likely ~30 out of date, then updated both of these boxes at the same time. i > don't know what my version of scipy & numpy prior to update, and i also don't > know the order of the update (numpy before scipy or vise versa). i am fairly > certain, however, that i didn't have these packages on my x86 box (the one with > the issue) --- i took them out during a previous world update, perhaps due to > conflicts (and i didn't need them at the time). i emerged them by hand on the > x86 box. > > is there something i can try on my x86 box? re-emerge scipy then numpy (or > vise versa)... As I see it you have numpy-1.6.0 installed on your amd64 box and there doesn't seem to be a problem. This the architecture that appears to have equivocal results. You could rebuild just scipy on that box and see if things are still OK. Here numpy is pulled in before scipy.
(In reply to comment #15) > (In reply to comment #14) > > i'm not entirely sure. this is all the result of a recent world update. i was > > likely ~30 out of date, then updated both of these boxes at the same time. i > > don't know what my version of scipy & numpy prior to update, and i also don't > > know the order of the update (numpy before scipy or vise versa). i am fairly > > certain, however, that i didn't have these packages on my x86 box (the one with > > the issue) --- i took them out during a previous world update, perhaps due to > > conflicts (and i didn't need them at the time). i emerged them by hand on the > > x86 box. > > > > is there something i can try on my x86 box? re-emerge scipy then numpy (or > > vise versa)... > > As I see it you have numpy-1.6.0 installed on your amd64 box and there doesn't > seem to be a problem. This the architecture that appears to have equivocal > results. You could rebuild just scipy on that box and see if things are still > OK. Here numpy is pulled in before scipy. fail. i re-emerged scipy on my amd64 box, which previously worked, but now the error appears: ImportError: /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv
I also don't use an overlay. I am on an amd64 machine. [emerge --info] app-shells/bash: 4.1_p9 dev-java/java-config: 2.1.11-r3 dev-lang/python: 2.7.1-r1, 3.1.3-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.8.4-r1 dev-util/pkgconfig: 0.25-r2 sys-apps/baselayout: 2.0.2 sys-apps/openrc: 0.8.2-r1 sys-apps/sandbox: 2.4 sys-devel/autoconf: 2.13, 2.65-r1 sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1 sys-devel/binutils: 2.20.1-r1 sys-devel/gcc: 4.4.5 sys-devel/gcc-config: 1.4.1-r1 sys-devel/libtool: 2.2.10 sys-devel/make: 3.82 sys-kernel/linux-headers: 2.6.36.1 (virtual/os-headers) sys-libs/glibc: 2.12.2 Repositories: gentoo local layman Installed sets: @kdeartwork-4.6, @kdebase-4.6, @system ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA dlj-1.1 skype-eula AdobeFlash-10.1" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=native -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -march=native -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch" FFLAGS="" GENTOO_MIRRORS="http://gentoo.tiscali.nl/ ftp://gentoo.tiscali.nl/pub/mirror/gentoo/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_US en_GB nl" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage /var/lib/portage" SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage" Portage 2.2.0_alpha42 (default/linux/amd64/10.0/desktop/kde, gcc-4.4.5, glibc-2.12.2-r0, 2.6.38-gentoo-r6 x86_64) ================================================================= System uname: Linux-2.6.38-gentoo-r6-x86_64-AMD_Phenom-tm-_II_X4_940_Processor-with-gentoo-2.0.2 Timestamp of tree: Fri, 01 Jul 2011 10:45:01 +0000 ccache version 2.4 [disabled] USE="3dnow 3dnowext 64bit X \ a52 aac acl acpi alsa amd64 bash-completion berkdb bitmap-fonts blas bluetooth bmp branding bzip2 cairo caps cdda cdr cli consolekit cracklib crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam fbcon ffmpeg firefox flac fortran gcj gd gdbm gdu gif gpm gstreamer iconv ieee1394 imagemagick ipv6 java java6 jpeg kde kipi kpathsea lame lapack latex lcms libnotify lm_sensors logrotate mad matroska mmx mng modules moonlight mozdevelop mp3 mp4 mpeg mudflap multilib musepack ncurses nls nptl nptlonly nsplugin ogg ogm opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pppd python python3 qt3 qt3support qt4 readline samba sdl semantic-desktop session sox spell sqlite sqlite3 srt sse sse2 sse3 sse4a ssl startup-notification svg sysfs tcpd theora threads tiff truetype truetype-fonts type1-fonts udev umfpack unicode usb vim vim-syntax vorbis wav webkit x264 xcb xcomposite xetex xine xinerama xinetd xml xorg xscreensaver xulrunner xv xvid zlib zsh-completion" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US en_GB nl" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="radeonhd radeon vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS I tried re-emerging several times. Got it working by using atlas instead of reference.
Still in numpy-1.6.1. From the previous post it looks like it may detect atlas whether you have it eselected or not. Jacob, did you have atlas and reference both installed at the same time, with reference being the eselected one? Or you didn't have atlas at all?
OK I now believe the problem may be caused by the atlas section of site.cfg in the numpy ebuild. I am not sure you can get rid of it in scipy in case numpy was installed without lapack (is it desirable?). It looks like other modifications may be needed to site.cfg. If my tests are positive and solve the issues here I'll attach numpy and scipy ebuilds for people to test.
Created attachment 282493 [details] numpy ebuild that doesn't exhibit problems I'll attach a matching scipy ebuild shortly but my test are successful and I got rid of the problem. More comments in the scipy ebuild attachment.
Created attachment 282497 [details] scipy ebuild matching the numpy one OK so in this ebuild I removed site.cfg. As it is the setup.py looks to numpy/distutils/site.cfg exclusively as far as I can tell. Putting a site.cfg in scipy doesn't seem to have an effect. Granted I haven't tried to match the final configuration that worked from numpy. Someone may want to try that. Before it worked several components had to be in place that I believe were essential: 1) removal of the [atlas] section. I think that was the trigger. 2) rebranding the sections [blas_opt] and [lapack_opt]. In theory numpy documentation in system_info.py says we should use _opt_info in practice scipy only looks for blas and lapack. 3) replacing libraries {blas,lapack}_libs which seems to be the way to set a library(ies) with a different name. This is critical using the new blas/lapack structure. So I removed site.cfg from scipy. I am not sure it is queried at all. It never appeared to be queried in my failed test build. It may just be that it was ignored because apart from removing the [atlas] section I didn't take the time to reproduce the final file I created for the numpy ebuild. It is probably the case otherwise you couldn't build scipy with a lapack-less numpy. This traditionally works.
Another thought. If you build numpy without lapack you may not see this mess. I cannot completely affirm it however.
(In reply to comment #18) > Still in numpy-1.6.1. From the previous post it looks like it may detect atlas > whether you have it eselected or not. > > Jacob, did you have atlas and reference both installed at the same time, with > reference being the eselected one? Or you didn't have atlas at all? I didn't have atlas at all. I installed it after this problem to 'solve' it.
(In reply to comment #23) > (In reply to comment #18) > > Still in numpy-1.6.1. From the previous post it looks like it may detect atlas > > whether you have it eselected or not. > > > > Jacob, did you have atlas and reference both installed at the same time, with > > reference being the eselected one? Or you didn't have atlas at all? > > I didn't have atlas at all. I installed it after this problem to 'solve' it. Thanks for the answer. I should have pointed out that I figured out that whether or not atlas was installed didn't matter. When after removing atlas and sci-libs/clapack from my system I still had the problem, I started to suspect something was smelly in the numpy/scipy configuration that we are passing to numpy/scipy.
(In reply to comment #21) > Created attachment 282497 [details] > scipy ebuild matching the numpy one > > OK so in this ebuild I removed site.cfg. As it is the setup.py looks to > numpy/distutils/site.cfg exclusively as far as I can tell. Putting a site.cfg > in scipy doesn't seem to have an effect. Granted I haven't tried to match the > final configuration that worked from numpy. Someone may want to try that. > > Before it worked several components had to be in place that I believe were > essential: > 1) removal of the [atlas] section. I think that was the trigger. > 2) rebranding the sections [blas_opt] and [lapack_opt]. In theory numpy > documentation in system_info.py says we should use _opt_info in practice scipy > only looks for blas and lapack. > 3) replacing libraries {blas,lapack}_libs which seems to be the way to set a > library(ies) with a different name. This is critical using the new blas/lapack > structure. > > So I removed site.cfg from scipy. I am not sure it is queried at all. It never > appeared to be queried in my failed test build. It may just be that it was > ignored because apart from removing the [atlas] section I didn't take the time > to reproduce the final file I created for the numpy ebuild. It is probably the > case otherwise you couldn't build scipy with a lapack-less numpy. This > traditionally works. François, I believe you have stumbled on the problem - ATLAS. But why should this [atlas] block in site.cfg only start to affect things beginning with numpy-1.6.0? I think that perhaps numpy is trying to be overly clever in how it deals with ATLAS and this may well conflict with how Gentoo does things. In system_info.py a get_atlas_version() function is defined: def get_atlas_version(**config): This function appears in numpy-1.5.1 but is first called in numpy-1.6.0 and also in numpy-1.6.1. I'll attach a patch that deletes these calls. It is to go in src_prepare(). By simply adding the patch, scipy when built against numpy-1.6.0/1, builds correctly. That is, there are no undefined symbols in scipy/linalg/clapack.so. I'm not sure which is the more consistent approach; to patch or to remove the [atlas] block? And I believe you are correct, the site.cfg file is not needed in scipy. Scipy uses the site.cfg from the numpy distutils.
Created attachment 282595 [details] patch to remove calls to get_atlas_version patch to remove calls to get_atlas_version
fyi, originally in the older numpy/scipy ebuilds, i added the [atlas] instead of [blas_opt] or [blas] in the site.cfg to make sure the _dotblas.so would use any cblas implementation instead of assuming that only atlas would give an optimized cblas as numpy.distutils did. upstream may have fixed that, i have not checked. numpy.distutils is spaghetti code and overly clever as steve said. now setting [atlas] in the site.cfg may have triggered a flag in numpy.distutils in the later numpy versions, and scipy would read this flag to trigger the use of the clapack_* routines. since only sci-libs/lapack-atlas (or sci-libs/atlas from the science overlay) provides the clapack_* routines, if one has another lapack implementation, it would result in undefined symbols.
According to the documentation of numpy/distutils/system_info.py (the file that changed between 1.5.1 and 1.6.0 and therefore correctly patched by Steve) we should use blas_opt and lapack_opt, atlas shouldn't be used and numpy should be smart enough to figure out whether it is using atlas or not. In practice blas, blas_opt blas_info and blas_opt_info are all valid but scipy specifically calls for blas (and lapack) and stops if it doesn't have these sections (or an atlas section). There are several things to check and to do: 1) does numpy correctly identify atlas without an atlas section with the site.cfg I provided? 2) what happen to _dotblas.so as Sébastien pointed out. I am not sure of the mechanics at play here just the side effects on scipy. 3) site.cfg is only created in this form if we enable the lapack use flag in numpy. However that hasn't prevented scipy to build before. So can we build scipy with numpy[-lapack] and this set up. If not does adding a site.cfg section in scipy mirroring the one I made for numpy work? 4) should we use Steve's patch instead? Basically he reverts to the behaviour of 1.5.1, That's an option but I think we should try to adapt site.cfg if possible rather than patching, i.e. go with the least invasive route. If we need dotblas to be in a special shape we may have to patch. Just looked at the system_info.py from numpy-1.6.1 to check some of my facts... in get_info line 251 and after: cl = {'atlas':atlas_info, # use lapack_opt or blas_opt instead 'atlas_threads':atlas_threads_info, # ditto 'atlas_blas':atlas_blas_info, 'atlas_blas_threads':atlas_blas_threads_info, 'lapack':lapack_atlas_info, # use lapack_opt instead 'lapack_atlas_threads':lapack_atlas_threads_info, # ditto 'mkl':mkl_info, 'lapack_mkl':lapack_mkl_info, # use lapack_opt instead 'blas_mkl':blas_mkl_info, # use blas_opt instead 'x11':x11_info, 'fft_opt':fft_opt_info, 'fftw':fftw_info, 'fftw2':fftw2_info, 'fftw3':fftw3_info, 'dfftw':dfftw_info, 'sfftw':sfftw_info, 'fftw_threads':fftw_threads_info, 'dfftw_threads':dfftw_threads_info, 'sfftw_threads':sfftw_threads_info, 'djbfft':djbfft_info, 'blas':blas_info, # use blas_opt instead 'lapack':lapack_info, # use lapack_opt instead 'lapack_src':lapack_src_info, 'blas_src':blas_src_info, 'numpy':numpy_info, 'f2py':f2py_info, 'Numeric':Numeric_info, 'numeric':Numeric_info, 'numarray':numarray_info, 'numerix':numerix_info, 'lapack_opt':lapack_opt_info, 'blas_opt':blas_opt_info, 'boost_python':boost_python_info, 'agg2':agg2_info, 'wx':wx_info, 'gdk_pixbuf_xlib_2':gdk_pixbuf_xlib_2_info, 'gdk-pixbuf-xlib-2.0':gdk_pixbuf_xlib_2_info, 'gdk_pixbuf_2':gdk_pixbuf_2_info, 'gdk-pixbuf-2.0':gdk_pixbuf_2_info, 'gdk':gdk_info, 'gdk_2':gdk_2_info, 'gdk-2.0':gdk_2_info, 'gdk_x11_2':gdk_x11_2_info, 'gdk-x11-2.0':gdk_x11_2_info, 'gtkp_x11_2':gtkp_x11_2_info, 'gtk+-x11-2.0':gtkp_x11_2_info, 'gtkp_2':gtkp_2_info, 'gtk+-2.0':gtkp_2_info, 'xft':xft_info, 'freetype2':freetype2_info, 'umfpack':umfpack_info, 'amd':amd_info, }.get(name.lower(),system_info) return cl().get_info(notfound_action) Note that there are two entries for lapack... one that says it should go to lapack_atlas and one to lapack_opt, which one is used? I would say the first one lapack_atlas.
*** Bug 376771 has been marked as a duplicate of this bug. ***
Tested point 3 of my previous post. site.cfg in the scipy ebuild is useless as it is never read. So starting with numpy-1.6.0 if you want to install scipy, numpy has to be installed with the lapack use flag as the site.cfg provided by numpy is the only one that will be used. I am fairly sure that the patch Steve attached doesn't do anything about that. The next question is: is it a feature or a bug according to upstream? The instruction shipped with scipy-0.9 don't mention the use of a local site.cfg. Point 2 about _dotblas.so: I think that part works fine without the atlas section. ldd /usr/lib64/python2.7/site-packages/numpy/core/_dotblas.so linux-vdso.so.1 => (0x00007fffdf391000) libopenblas_nehalem-r0.1alpha2.2.so => /usr/lib64/libopenblas_nehalem-r0.1alpha2.2.so (0x00007fad5c3cd000) libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007fad5c01c000) libc.so.6 => /lib64/libc.so.6 (0x00007fad5bc97000) libm.so.6 => /lib64/libm.so.6 (0x00007fad5ba13000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fad5b7f4000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fad5b5f0000) libutil.so.1 => /lib64/libutil.so.1 (0x00007fad5b3ed000) /lib64/ld-linux-x86-64.so.2 (0x00007fad5cb27000) with numpy-1.6.1 built with my ebuild.
More tests reveal that scipy will build against numpy without lapack if site.cfg has an atlas section. Which brings us to square one if blas/lapack is not atlas. There must be something that the scipy's site.cfg will accept for blas/lapack other than atlas.
OK beats me why it didn't work in my last round of testing. I must have done something slightly different. This: cat >> site.cfg <<-EOF [blas] include_dirs = $(pkg-config --cflags-only-I \ cblas | sed -e 's/^-I//' -e 's/ -I/:/g') library_dirs = $(pkg-config --libs-only-L \ cblas blas | sed -e 's/^-L//' -e 's/ -L/:/g' -e 's/ //g'):${libdir} blas_libs = $(pkg-config --libs-only-l \ cblas blas | sed -e 's/^-l//' -e 's/ -l/, /g' -e 's/,.pthread//g') [lapack] library_dirs = $(pkg-config --libs-only-L \ lapack | sed -e 's/^-L//' -e 's/ -L/:/g' -e 's/ //g'):${libdir} lapack_libs = $(pkg-config --libs-only-l \ lapack | sed -e 's/^-l//' -e 's/ -l/, /g' -e 's/,.pthread//g') EOF seems to work in scipy even if numpy is build without lapack. And it has the right behavior: ldd -r /scratch/portage/sci-libs/scipy-0.9.0-r1/image/usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so linux-vdso.so.1 => (0x00007fff58b0a000) libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007ffafa295000) libc.so.6 => /lib64/libc.so.6 (0x00007ffaf9f10000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffaf9cf1000) libdl.so.2 => /lib64/libdl.so.2 (0x00007ffaf9aed000) libutil.so.1 => /lib64/libutil.so.1 (0x00007ffaf98ea000) libm.so.6 => /lib64/libm.so.6 (0x00007ffaf9665000) /lib64/ld-linux-x86-64.so.2 (0x00007ffafa886000) No undefined symbols this time. I will attach a scipy ebuild shortly.
Created attachment 283705 [details] corrected scipy ebuild Updating the scipy ebuild. This one works with numpy[-lapack].
(In reply to comment #30) > Tested point 3 of my previous post. site.cfg in the scipy ebuild is useless as > it is never read. So starting with numpy-1.6.0 if you want to install scipy, > numpy has to be installed with the lapack use flag as the site.cfg provided by > numpy is the only one that will be used. I am fairly sure that the patch Steve > attached doesn't do anything about that. > > The next question is: is it a feature or a bug according to upstream? The > instruction shipped with scipy-0.9 don't mention the use of a local site.cfg. > Actually, scipy-0.9 built against either numpy-1.6.0[lapack] or numpy-1.6.0[-lapack] with the patch to remove calls to get_atlas_version doesn't build quite correctly. There are no undefined symbols in scipy/linalg/clapack.so. However, there are undefined symbols in scipy/linalg/atlas_version.so relative to function locations ldd -r /storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/linalg/atlas_version.so linux-vdso.so.1 => (0x00007ffff97d4000) libpython2.7.so.1.0 => /storage/strogdon/gentoo/usr/lib/libpython2.7.so.1.0 (0x00007f12c7cc4000) libc.so.6 => /lib/libc.so.6 (0x00007f12c7950000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f12c7734000) libdl.so.2 => /lib/libdl.so.2 (0x00007f12c7530000) libutil.so.1 => /lib/libutil.so.1 (0x00007f12c732c000) libm.so.6 => /lib/libm.so.6 (0x00007f12c70aa000) /lib64/ld-linux-x86-64.so.2 (0x00007f12c8286000) undefined symbol: ATL_buildinfo (/storage/strogdon/gentoo/usr/lib/python2.7/site-packages/scipy/linalg/atlas_version.so) which do not appear when scipy is built correctly. Now if I build numpy-1.5.1 using the site.cfg portion of the above numpy ebuild (only blas/lapack sections) then the above revised scipy builds correctly even though numpy-1.5.1 doesn't have calls to the get_atlas_version function. At least, I can't find any place where the function is called. So, tests here would indicate that only the blas/lapack sections are needed in the site.cfg portion of the ebuilds. This perhaps explains why the documentation instructions in numpy/distutils/system_info.py hasn't changed since numpy-1.5.1. Also, this revised numpy-1.5.1 built using openblas seems to build dotblas correctly, ldd -r /usr/lib/python2.7/site-packages/numpy/core/_dotblas.so linux-vdso.so.1 => (0x00007fffbcfff000) libopenblas_opteron-r0.1alpha2.2.so => /usr/lib64/libopenblas_opteron-r0.1alpha2.2.so (0x00007f4c91f07000) libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007f4c91b5d000) libc.so.6 => /lib64/libc.so.6 (0x00007f4c917fa000) libm.so.6 => /lib64/libm.so.6 (0x00007f4c9157a000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4c9135d000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f4c91159000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f4c90f56000) /lib64/ld-linux-x86-64.so.2 (0x00007f4c926b1000)
Thanks for the extra info Steve. So it looks like my numpy/scipy ebuild modification has your seal of approval then. I think it is time for someone to put this in the tree and end this little story.
Have you guys considered trying to get this stuff upstream since it seems the correct place to fix these issues. In any case, I will try to get this all tested and in the tree soon.
(In reply to comment #36) > Have you guys considered trying to get this stuff upstream since it seems the > correct place to fix these issues. In any case, I will try to get this all > tested and in the tree soon. I am not sure it is appropriate for upstream since the problem is mainly due to how we currently configure numpy and scipy in the tree. The configuration was working well until recently when upstream actually got its setup to be smarter in some way and dumber in other. Like bicatali said earlier in the bug we had an [atlas] entry because for a reason that is now obsolete, not only obsolete but leading to a buggy behavior. Also syntax of the configuration has changed slightly and we definitely want the format I put in for the new blas/lapack framework now in the science overlay - because of the new naming scheme.
I'm wondering whether the scipy build problem relative to site.cfg isn't an upstream issue. I haven't scoured the documentation completely but if François is correct there is need for a documented use of a local scipy site.cfg file. At least there is when a numpy site.cfg isn't present. Or is building numpy with USE=-lapack just a gentoo possibility?
Any news for this bug?
+*numpy-1.6.1-r1 (07 Oct 2011) + + 07 Oct 2011; Justin Lecher <jlec@gentoo.org> +numpy-1.6.1-r1.ebuild, + +files/numpy-1.6.1-atlas.patch, metadata.xml: + Fix for missing clapack_* symbols in scipy, #371099 + +*scipy-0.9.0-r2 (07 Oct 2011) + + 07 Oct 2011; Justin Lecher <jlec@gentoo.org> +scipy-0.9.0-r2.ebuild: + Fix for missing clapack_* symbols in scipy, #371099; thanks Francois Bissey + for the patches +
