Home | Docs | Forums | Lists | Bugs | Planet | Store | GMN | Get Gentoo!
Not eligible to see or edit group visibility for this bug.
View Bug Activity | Format For Printing | XML | Clone This Bug
Using eselect, I have chosen atlas-threads for my blas setting, and atlas for my lapack setting. When I attempt to emerge numpy, I get the following report: ------------------- blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE atlas_blas_info: NOT AVAILABLE /var/tmp/portage/dev-python/numpy-1.1.1/work/numpy-1.1.1/numpy/distutils/system_info.py:1340: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE /var/tmp/portage/dev-python/numpy-1.1.1/work/numpy-1.1.1/numpy/distutils/system_info.py:1247: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 ------------------- Notice also that the cblas libraries are also not identified. Both atlas and cblas are present on my system: # slocate libatlas /usr/lib64/libatlas.so.0.0.0 /usr/lib64/libatlas.so.0 /usr/lib64/libatlas.a /usr/lib64/libatlas.la /usr/lib64/libatlas.so # slocate libcblas /usr/lib64/blas/atlas/libcblas.so.0.0.0 /usr/lib64/blas/atlas/libcblas.la /usr/lib64/blas/atlas/libcblas.so /usr/lib64/blas/atlas/libcblas.so.0 /usr/lib64/blas/atlas/libcblas.a /usr/lib64/blas/threaded-atlas/libcblas.so.0.0.0 /usr/lib64/blas/threaded-atlas/libcblas.la /usr/lib64/blas/threaded-atlas/libcblas.so /usr/lib64/blas/threaded-atlas/libcblas.so.0 /usr/lib64/blas/threaded-atlas/libcblas.a /usr/lib64/libcblas.so /usr/lib64/libcblas.so.0 /usr/lib64/libcblas.a Here is the site.cfg generated by the numpy ebuilds: [atlas] include_dirs = library_dirs = atlas_libs = cblas, atlas lapack_libs = lapack, blas, cblas, atlas Note that the include_dirs is empty, I might have expected it to point to /usr/include/atlas. When I checked /usr/include, I discovered that cblas.h was a broken link, while clapack.h was ok and pointed to atlas/clapack.h. In numpy-1.1.1, site.cfg.example indicates that the [atlas] section is deprecated and should not be used. I think the site configuration needs to be revisited in the numpy ebuilds. The ebuilds used to modify numpy/distutils/system_info.py to change instances of 'f77blas' in _lib_names to 'blas'. I don't know why this was removed from the numpy ebuilds. When this fix is reintroduced, and /usr/include/atlas is added to the list of include dirs in [default], numpy's distutils generates the following report: -------------------- blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: FOUND: libraries = ['blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize LaheyFCompiler Could not locate executable lf95 customize PGroupFCompiler Could not locate executable pgf90 Could not locate executable pgf77 customize AbsoftFCompiler Could not locate executable f90 customize NAGFCompiler Could not locate executable f95 customize VastFCompiler customize GnuFCompiler customize CompaqFCompiler Could not locate executable fort customize IntelItaniumFCompiler Could not locate executable efort Could not locate executable efc customize IntelEM64TFCompiler customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: x86_64-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -O2 -march=k8 -mtune=k8 -pipe -fomit-frame-pointer -fPIC compile options: '-c' x86_64-pc-linux-gnu-gcc: _configtest.c x86_64-pc-linux-gnu-gcc -pthread _configtest.o -L/usr/lib -lblas -lcblas -latlas -o _configtest ATLAS version 3.9.3 built by root on Mon Aug 18 09:02:22 EDT 2008: UNAME : Linux dale 2.6.26-gentoo #1 SMP Thu Jul 31 11:13:19 EDT 2008 x86_64 Dual Core AMD Opteron(tm) Processor 275 AuthenticAMD GNU/Linux INSTFLG : -1 0 -a 1 -l 0 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_HAMMER -DATL_CPUMHZ=2210 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_3DNow -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (Gentoo 4.3.1-r1 p1.1) 4.3.1 F77FLAGS : -O2 -m64 SMC : gcc, version gcc (Gentoo 4.3.1-r1 p1.1) 4.3.1 SMCFLAGS : -O2 -march=k8 -mtune=k8 -pipe -fomit-frame-pointer SKC : gcc, version gcc (Gentoo 4.3.1-r1 p1.1) 4.3.1 SKCFLAGS : -O2 -march=k8 -mtune=k8 -pipe -fomit-frame-pointer success! removing: _configtest.c _configtest.o _configtest FOUND: libraries = ['blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c define_macros = [('ATLAS_INFO', '"\\"3.9.3\\""')] lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info FOUND: libraries = ['lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 -------------------- Note that the atlas libraries are now found, as is cblas. They are not identified as the threaded version of the libraries, although they should be, since /usr/lib/libblas.so is a soft link to /usr/lib/blas/threaded-atlas/libblas.so. I am concerned, but am not certain, that numpy is going to be confused by gentoo's eselect system of having libblas* link to whatever actual library we want to use. For this eselect system to work, is it really necessary to rename the threaded atlas libraries in /usr/lib/blas/threaded-atlas? Any package, like numpy, which relies on the original naming convention gets confused. For example, numpy will identify the threaded atlas libraries if they are called ptf77blas and ptcblas. If gentoo installed libraries into, for example, /usr/lib/blas/threaded-atlas using the standard naming convention, we could add that directory to the list of library dirs in [default] and not have to worry about editing numpy/distutils/system_info.py to adapt to gentoo's unusual setup. respectfully, Darren Reproducible: Always Portage 2.2_rc8 (default/linux/amd64/2008.0/desktop, gcc-4.3.1, glibc-2.8_p20080602-r0, 2.6.26-gentoo-r1 x86_64) ================================================================= System uname: Linux-2.6.26-gentoo-r1-x86_64-Dual_Core_AMD_Opteron-tm-_Processor_275-with-glibc2.2.5 Timestamp of tree: Mon, 15 Sep 2008 12:30:01 +0000 app-shells/bash: 3.2_p39 dev-java/java-config: 1.3.7, 2.1.6-r1 dev-lang/python: 2.5.2-r7 sys-apps/baselayout: 2.0.0 sys-apps/openrc: 0.2.5 sys-apps/sandbox: 1.2.18.1-r3 sys-devel/autoconf: 2.13, 2.62-r1 sys-devel/automake: 1.5, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 2.2.4 virtual/os-headers: 2.6.26 ACCEPT_KEYWORDS="amd64 ~amd64" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -march=k8 -mtune=k8 -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/kde/4.0/env /usr/kde/4.0/share/config /usr/kde/4.0/shutdown /usr/share/config /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -march=k8 -mtune=k8 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="collision-protect distlocks fixpackages parallel-fetch preserve-libs sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" LDFLAGS="-Wl,-O1" LINGUAS="en en_US" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/layman/soor-overlay /usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dnow X aac acl acpi alsa amd64 apache2 atlas avahi bash-completion berkdb blas bluetooth bookmarks branding bzip2 cairo cblas cdr cli cracklib crypt ctype cups dbus doc dri dvd dvdr dvdread eds emacs emboss encode esd evo examples f77 fam fftw firefox foomativdb fortran gdbm gfortran gif gimpprint glitz gpm gstreamer gtk hal iconv imagemagick imap isdnlog ivman java jpeg jpeg2k kde kdrive kerberos lapack latex ldap libnotify lzo mad mdnsresponder-compat midi mikmod mime mmap mmx mozbranding mozilla mozsvg mp3 mpeg mplayer mudflap multilib multislot mysql ncurses newspr nptl nptlonly nsplugin ogg opengl openmp oss pam pcre pdf perl pertty pic png postgres ppds pppd python qt3 qt3support qt4 quicktime readline reflection rss samba sdl session spell spl sse sse2 ssl startup-notification subversion svg symlink sysfs tcltk tcpd tetex threads tiff tk transparency truetype ubuntu umfpack unicode usb v4l v4l2 vorbis webdav winbind xcomposite xfs xft xinerama xml xorg xulrunner xv zeroconf zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" USERLAND="GNU" VIDEO_CARDS="nvidia vga nv" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Hi, Thanks for your report. It is fairly hard to statisfy numpy with all our blas/lapack implementations. numpy distutils is an overly complex automagic/autodetect piece of software (which really needs rewrite). It seems some things have changed upstream in 1.1.1 where the [atlas] trick to force _dotblas.so building does not work anymore. As I understand, numpy assumes atlas is the only cblas implementation and build the dotblas library when atlas is detected. I've made some fixes in the numpy ebuild to use [blas_opt] and force building of _dotblas.so when lapack flag is set. Also the broken link in atlas/cblas has been fixed in the blas-atlas ebuild. To avoid rebuilding simply change cblas/cblas.h to atlas/cblas.h in the file /etc/env.d/lib*/cblas/atlas-threads, re-run "eselect cblas set atlas-threads". Let me know if it's ok.
(In reply to comment #1) > Thanks for your report. It is fairly hard to statisfy numpy with all our > blas/lapack implementations. numpy distutils is an overly complex > automagic/autodetect piece of software (which really needs rewrite). It seems > some things have changed upstream in 1.1.1 where the [atlas] trick to force > _dotblas.so building does not work anymore. As I understand, numpy assumes > atlas is the only cblas implementation and build the dotblas library when atlas > is detected. > > I've made some fixes in the numpy ebuild to use [blas_opt] and force building > of _dotblas.so when lapack flag is set. > > Also the broken link in atlas/cblas has been fixed in the blas-atlas ebuild. To > avoid rebuilding simply change cblas/cblas.h to atlas/cblas.h in the file > /etc/env.d/lib*/cblas/atlas-threads, re-run "eselect cblas set atlas-threads". Thanks for the pointer. > Let me know if it's ok. I don't think so. Numpy still needs to be patched to look for "blas" instead of "f77blas": $ sudo emerge "<numpy-1.1.99" Calculating dependencies... done! >>> Verifying ebuild manifests >>> Emerging (1 of 1) dev-python/numpy-1.1.1 * numpy-1.1.1.tar.gz RMD160 SHA1 SHA256 size ;-) ... [ ok ] * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ... [ ok ] * checking miscfile checksums ;-) ... [ ok ] * You need one of these Fortran Compilers: gfortran g77 ifc * Installed are: gfortran >>> Unpacking source... >>> Unpacking numpy-1.1.1.tar.gz to /var/tmp/portage/dev-python/numpy-1.1.1/work * Applying numpy-1.1.0-f2py.patch ... [ ok ] >>> Source unpacked. >>> Compiling source in /var/tmp/portage/dev-python/numpy-1.1.1/work/numpy-1.1.1 ... Running from numpy source directory. F2PY Version 2_5585 blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib NOT AVAILABLE /var/tmp/portage/dev-python/numpy-1.1.1/work/numpy-1.1.1/numpy/distutils/system_info.py:1340: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: libraries blas not found in /usr/local/lib FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries ptf77blas,ptcblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: libraries f77blas,cblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries f77blas,cblas,atlas not found in /usr/lib libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_info NOT AVAILABLE /var/tmp/portage/dev-python/numpy-1.1.1/work/numpy-1.1.1/numpy/distutils/system_info.py:1247: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) lapack_info: libraries lapack not found in /usr/local/lib FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77
(In reply to comment #2) > I don't think so. Numpy still needs to be patched to look for "blas" instead of > "f77blas": > Even if the config part of the build doesn't show atlas is detected, it does not matter. The sed line in the ebuild forces the dotblas build whenever cblas+lapack are installed. As far as I can see atlas is only used to build _dotblas.so. I could re-patch f77blas to blas and friends in the distutils source, but it will change only the printing during the config stage and the installed numpy would be identical.
I see, thanks for the explanation. perhaps some information could be passed by the ebuild so future people with just enough knowledge of numpy will not make the same conclusion I did. Perhaps some information about the numpy's ability to use atlas-threads as well, which I am still unclear on.
(In reply to comment #4) > I see, thanks for the explanation. perhaps some information could be passed by > the ebuild so future people with just enough knowledge of numpy will not make > the same conclusion I did. Perhaps some information about the numpy's ability > to use atlas-threads as well, which I am still unclear on. > OK to remove ambiguity I re-added a small patch for the gentoo atlas library names, applied in the just bumped numpy-1.2.0. Thanks for your remarks. Closing now, re-open if you still see a related issue.
