Bugzilla Bug 195619

After a system upgrade, I get the following error when using scipy:

    from scipy.linalg import clapack
ImportError: /usr/lib64/python2.4/site-packages/scipy/linalg/clapack.so:
undefined symbol: clapack_sgesv

This is with MKL. I tried re-ermerging MKL and scipy, and making sure that all
the relevant eselects were done, but to no avail. 

I tried other lapack implementations, but that didn't help.


Reproducible: Always




Portage 2.1.3.9 (default-linux/amd64/2007.0/desktop, gcc-4.1.2, glibc-2.5-r4,
2.6.22-gentoo-r2 x86_64)
=================================================================
System uname: 2.6.22-gentoo-r2 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz
Timestamp of tree: Fri, 12 Oct 2007 01:31:01 +0000
app-shells/bash:     3.2_p17
dev-java/java-config: 1.3.7, 2.0.33-r1
dev-lang/python:     2.4.4-r5
dev-python/pycrypto: 2.0.1-r6
sys-apps/baselayout: 1.12.9-r2
sys-apps/sandbox:    1.2.17
sys-devel/autoconf:  2.13, 2.61-r1
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.17-r1
sys-devel/gcc-config: 1.3.16
sys-devel/libtool:   1.5.24
virtual/os-headers:  2.6.21
ACCEPT_KEYWORDS="amd64"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=nocona -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/revdep-rebuild
/etc/terminfo /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="distlocks metadata-transfer sandbox sfperms strict unmerge-orphans
userfetch"
GENTOO_MIRRORS="ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/
ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --delete-after --stats --timeout=180
--exclude=/distfiles --exclude=/local --exclude=/packages
--filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X aalib acl acpi alsa amd64 arts atlas avi bash-completion berkdb
bitmap-fonts cairo cdr cli cracklib crypt cups cvs dbus dri dvd dvdr dvdread
eds emboss encode esd evo f77 fam firefox flac fortran gdbm gif gphoto2 gpm
gstreamer gtk hal iconv imap imlib ipv6 isdnlog jpeg kde kerberos ldap mad midi
mikmod mmx mp3 mpeg mudflap ncurses nls nptl nptlonly nsplugin ntplonly nvidia
ogg oggvorbis opengl openmp oss pam pcre pda pdf pdflib perl pic png pppd
prelink python qt3 qt3support qt4 qtmt quicktime readline reflection samba sdl
session smime snmp spell spl sse sse2 ssl subversion svg tcltk tcpd tetex tiff
tk truetype truetype-fonts type1-fonts unicode vorbis xml xorg xv"
ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x
ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3
trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw
asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa
lfloat linear meter mulaw multi null plug rate route share shm softvol"
ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad
cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU"
VIDEO_CARDS="nvidia"
Unset:  CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS,
LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS,
PORTDIR_OVERLAY

Hi Peter,

Haven't really test it with MKL yet. I thought mkl had clapack bindings. I will
look into it asap.

Sébastien

I could not reproduce it with a similar system. 
The only thing I could think is the eselect bug again.

Remove /etc/env.d/{blas,cblas,lapack}/lib*/config, and re-eselect mkl to all,
making sure all libs points to  then re-emerge scipy.

I tried your suggestion, but to no avail...

I can now reproduce this on a second (non 64-bit) system as well.

One thing I noticed, though: if I try importing scipy a second time from the
Python prompt, I get a different error:

>>> from scipy import *
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.4/site-packages/scipy/linalg/__init__.py", line 8, in
?
    from basic import *
  File "/usr/lib/python2.4/site-packages/scipy/linalg/basic.py", line 17, in ?
    from lapack import get_lapack_funcs
  File "/usr/lib/python2.4/site-packages/scipy/linalg/lapack.py", line 17, in ?
    from scipy.linalg import flapack
ImportError: cannot import name flapack

Not sure if this is relevant, though, could be a leftover from the first failed
import.

I can reproduce this on an amd64 box with lapack-reference-3.1.1-r1.
I'll have a look at it.

Markus

The main problem here is the fact the scipy names the lapack 
symbols improperly. It seems that setup.py is trying to do a lot of 
autodetection of various blas/lapack packages. On my box it
eventually also picks generic_clapack.pyf which names various
symbols as clapack_foo. This of course won't fly since at least
lapack-reference and lapack-atlas export them as foo_ (not
sure about MKL). I'll post a patch that fixes the issues for me.
Please give it a spin and see if it works for you. 

cheers,
Markus 

Created an attachment (id=133332) [edit]
patch to fix improperly named symbols

Success! (at least on 1 of my machines, the other is at work)

Thanks for the quick fix,

Peter

I've committed -r2 which contains this patch.

Thanks,
Markus

Looks like this one is back with scipy-0.6.0-r3 and mkl-9.1.023...

Portage 2.1.3.19 (default-linux/x86/2007.0/desktop, gcc-4.1.2, glibc-2.6.1-r0,
2.6.22-gentoo-r9 i686)
=================================================================
System uname: 2.6.22-gentoo-r9 i686 Intel(R) Pentium(R) M processor 2.00GHz
Timestamp of tree: Fri, 18 Jan 2008 02:30:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632)
[disabled]
app-shells/bash:     3.2_p17-r1
dev-java/java-config: 1.3.7, 2.0.33-r1
dev-lang/python:     2.4.4-r6
dev-python/pycrypto: 2.0.1-r6
sys-apps/baselayout: 1.12.10-r5
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.61-r1
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.18-r1
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.24
virtual/os-headers:  2.6.23-r3
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=pentium-m -O3 -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/lib/mozilla/defaults/pref /usr/share/config"
CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf
/etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c
/etc/udev/rules.d"
CXXFLAGS="-O2 -mcpu=i686 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="cvs distlocks metadata-transfer sandbox sfperms strict
unmerge-orphans userfetch"
GENTOO_MIRRORS="http://cesium.di.uminho.pt/pub/gentoo/
http://fido.online.kz/gentoo ftp://mirror.isp.net.au/pub/gentoo/"
LANG="en_US.utf8"
LC_ALL="en_US.utf8"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress
--force --whole-file --delete --delete-after --stats --timeout=180
--exclude=/distfiles --exclude=/local --exclude=/packages
--filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac accessibility acl acpi alsa amarok amd arts asf automount
bash-completion batch berkdb bidi bitmap-fonts bittorrent bl cairo cdparanoia
cdr cgi cli cracklib crypt css cups curl dbus dpms dri dvd dvdr dvdread eds
emboss encode esd ethereal evo exif fam fasttrack fat ffmpeg fftw firefox flac
font-server fortran gdb gdbm gif gnutella gphoto2 gpm gsm gstreamer gtk gtk2
hal hdf5 iconv imap immqt-bc innodb ipv6 isdnlog ivtv jpeg kde kdepim kerberos
kqemu latex ldap lirc lm_sensors logrotate mad magic midi mikmod mjpeg mmx mp3
mp4live mpeg mpeg2 mplayer msn mudflap musepack musicbrainz mysql mythtv
ncurses nls nptl nptlonly nsplugin nvidia ofx ogg opengl openmp openssh openssl
oss pam pcre pda pdf perl png pppd python qt3 qt3support qt4 quicktime readline
real reflection samba sdl server session spell spl sqlite sqlite3 sse
sse-filters sse2 ssl subversion svg svga tcl tcpd threads tiff tk truetype
truetype-fonts type1-fonts udev unicode v4l v4l2 vcd vfat videos vidix
visualization vorbis webdav wifi win32codecs wma x86 xinerama xml xorg xv xvid
xvmc zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci
emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0
intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci"
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file
hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route
share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest
authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default
authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs
dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter
headers ident imagemap include info log_config logio mem_cache mime mime_magic
negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite
setenvif so speling status unique_id userdir usertrack vhost_alias"
ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad
cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text"
LIRC_DEVICES="pctv" USERLAND="GNU" VIDEO_CARDS="nvidia"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, LINGUAS,
PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Hi Peter,

Unfortunately Markus patch could not be applied (see bug #197527). So I tried
several blas/lapack combination, could reproduce the behaviour and updated
scipy and numpy.
The import clapack command should now work. However note that of all lapack
implementations, only lapack-atlas bring clapack_* routines, and is incomplete
relative to the f77 implementation. So when you import scipy.linalg.clapack
with scipy built on mkl, it will not crash but give you an empty module. Only
building scipy on lapack-atlas will produce the clapack module. This is the
intended behaviour upstream. I am not sure it is worth a use flag though.
Let me know if you see any issue with it.

> relative to the f77 implementation. So when you import scipy.linalg.clapack
> with scipy built on mkl, it will not crash but give you an empty module. Only
> building scipy on lapack-atlas will produce the clapack module.

Since it is the upstream intended behaviour, I think this one could be closed.