Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 456280 - =sci-libs/scipy-0.11.0-r1 does not find blas with >=dev-util/pkgconfig-0.28
Summary: =sci-libs/scipy-0.11.0-r1 does not find blas with >=dev-util/pkgconfig-0.28
Status: RESOLVED DUPLICATE of bug 386269
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Gentoo Science Related Packages
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-09 11:09 UTC by Thomas Kahle (RETIRED)
Modified: 2014-05-08 11:21 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
/var/tmp/portage/sci-libs/scipy-0.11.0-r1/temp/build.log (build.log,6.46 KB, text/plain)
2013-02-11 11:40 UTC, Juergen Rose
Details
This patch allow for duplicate libraries to be passed to system_info.py without (numpy-1.6.2-duplicatelibs.patch,719 bytes, patch)
2013-02-12 21:16 UTC, François Bissey
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Kahle (RETIRED) gentoo-dev 2013-02-09 11:09:09 UTC
Hi, I use sci-libs/atlas from the science overlay and it works fine with other lapack/blas dependent programs.

$ pkg-config --libs blas
-lf77blas -lm -latlas

However, scipy does not find my blas installation and fails:

>>> Compiling source in /var/tmp/portage/sci-libs/scipy-0.11.0-r1/work/scipy-0.11.0 ...
 * python2.7: running distutils-r1_run_phase python_compile
/usr/bin/python2.7 setup.py build config_fc --noopt --noarch
Running from scipy source directory.
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib']
  NOT AVAILABLE

system_info:
  NOT AVAILABLE

/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1488: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:558: UserWarning: Specified path  is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
  libraries atlcblas,atlas,f77blas,atlas not found in ['/usr/lib64']
  NOT AVAILABLE

[ ... ]

 * ERROR: sci-libs/scipy-0.11.0-r1 failed (compile phase):
 *   (n

Reproducible: Always




# emerge --info
Portage 2.1.11.50 (default/linux/amd64/10.0/desktop/kde, gcc-4.6.3, glibc-2.15-r3, 3.4.24-gentoo x86_64)
=================================================================
System uname: Linux-3.4.24-gentoo-x86_64-Intel-R-_Core-TM-_i5-2520M_CPU_@_2.50GHz-with-gentoo-2.1
KiB Mem:     8065804 total,    842316 free
KiB Swap:    8191996 total,   8163620 free
Timestamp of tree: Sat, 09 Feb 2013 10:30:01 +0000
ld GNU ld (GNU Binutils) 2.22
ccache version 3.1.8 [enabled]
app-shells/bash:          4.2_p37
dev-java/java-config:     2.1.12-r1
dev-lang/python:          2.7.3-r2, 3.2.3
dev-util/ccache:          3.1.8
dev-util/cmake:           2.8.9
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.1-r1
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.69
sys-devel/automake:       1.9.6-r3, 1.10.3, 1.11.6
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.6.3
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.6 (virtual/os-headers)
sys-libs/glibc:           2.15-r3
Repositories: gentoo science sunrise sage-on-gentoo emacs java-overlay gentoo-haskell owncloud-client TomOverlay proxy-maint
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA Intel-SDP nauty PUEL googleearth google-talkplugin AdobeFlash-10.1 AdobeFlash-10.3 dlj-1.1 skype-eula skype-4.0.0.7-copyright IBM-J1.6 google-chrome Oracle-BCLA-JavaSE Google-TOS"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa /var/lib/hsqldb"
CONFIG_PROTECT_MASK="${EPREFIX}/etc/gconf /etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--autounmask-write --quiet-build=n"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs ccache collision-protect config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch protect-owned sandbox sfperms sign splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
MAKEOPTS="-j5 -l5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/science /var/lib/layman/sunrise /var/lib/layman/sage-on-gentoo /var/lib/layman/emacs /var/lib/layman/java-overlay /var/lib/layman/haskell /var/lib/layman/owncloud-client /home/tom/tom-overlay /home/tom/gentoo/dlan-proxy"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa amd64 anthy apache2 avahi bash-completion berkdb branding bzip2 cairo cdda cddb cdparanoia cdr cjk cli consolekit cracklib crypt ctype cups curl cxx daap dbus declarative djvu dri dts dvb dvd dvdr emacs encode exif expat fam fbcon ffmpeg firefox flac fortran ftp gd gdbm gif gimp glut gmp gnome gnome-keyring gnutls gpm graphviz gstreamer gtk guile hdaps iconv icu ieee1394 imagemagick imap imlib ipod ipv6 java javascript jit jpeg kde kipi latex lcms ldap leim libnotify lm_sensors lzma m17n-lib mad migemo mime mmx mng modules mp3 mp4 mpeg mplayer mtp mudflap mule multilib musicbrainz mysql mysqli ncurses nls nptl nsplugin obex ocaml ogg oggvorbis openal opengl openmp pam pango pcmcia pcre pdf perl phonon php plasma plotutils png policykit ppds pulseaudio python qt3support qt4 readline sasl sdl semantic-desktop session slang smartcard spell sqlite sqlite3 sse sse2 ssl ssse3 startup-notification svg tcpd texlive theora threads tiff tk truetype udev udisks uim unicode upower usb v4l v4l2 vaapi visualization vorbis wicd wifi wxwidgets wxwindows x264 xcb xcomposite xft xine xinerama xml xscreensaver xulrunner xv xvid zeroconf zlib zsh-completion" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US de" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="intel vesa fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 1 Justin Lecher (RETIRED) gentoo-dev 2013-02-10 17:37:20 UTC
build.log please.
Comment 2 Justin Lecher (RETIRED) gentoo-dev 2013-02-10 17:39:08 UTC
Cannot reproduce it here.
Comment 3 François Bissey 2013-02-10 20:49:29 UTC
Hi Thomas,

just out of curiosity can you retry after downgrading pkgconfig to 0.27.1? My previous experiments seemed to show that numpy/scipy were unaffected by bug #454254 but this is a new scipy ebuild and I haven't checked it.
Comment 4 Thomas Kahle (RETIRED) gentoo-dev 2013-02-11 10:22:51 UTC
(In reply to comment #3)
> Hi Thomas,
> 
> just out of curiosity can you retry after downgrading pkgconfig to 0.27.1?
> My previous experiments seemed to show that numpy/scipy were unaffected by
> bug #454254 but this is a new scipy ebuild and I haven't checked it.

Yes, works with pkconfig-0.27.1
Comment 5 Justin Lecher (RETIRED) gentoo-dev 2013-02-11 10:43:13 UTC
I am using pkgconfig-0.28 but cannot reproduce it at all.
Comment 6 Thomas Kahle (RETIRED) gentoo-dev 2013-02-11 10:52:09 UTC
(In reply to comment #5)
> I am using pkgconfig-0.28 but cannot reproduce it at all.

Maybe related to atlas?  Is there a way to produce all information that pkg-config would ever output for a package so that we could compare the two outputs?
Comment 7 Juergen Rose 2013-02-11 11:39:13 UTC
I have the same issue, 'emerge scipy' fails with:
...
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1488: UserWarning: 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:558: UserWarning: Specified path  is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
  libraries atlcblas,atlas,f77blas,atlas not found in ['/usr/lib64']
  NOT AVAILABLE

/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1497: UserWarning: 
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
blas_src_info:
  NOT AVAILABLE

/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1500: UserWarning: 
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
...
root@leopard:/root(361)# eselect blas list
Available providers for blas:
  [1]   atlas *
  [2]   atlas-threads
  [3]   eigen

root@leopard:/root(362)# pkg-config --libs blas
-lf77blas -lm -latlas 

root@leopard:/root(363)# qlist -Iv pkgconfig
dev-perl/extutils-pkgconfig-1.140.0
dev-util/pkgconfig-0.28
virtual/pkgconfig-0
Comment 8 Juergen Rose 2013-02-11 11:40:30 UTC
Created attachment 338582 [details]
/var/tmp/portage/sci-libs/scipy-0.11.0-r1/temp/build.log
Comment 9 François Bissey 2013-02-11 19:58:22 UTC
What I'd like you to post is the site.cfg file generated by the ebuild. Furthermore you are both consumer of sage-on-gentoo, what exact version of numpy are you using and did you enable the lapack useflag with it?
Comment 10 Thomas Kahle (RETIRED) gentoo-dev 2013-02-11 20:28:49 UTC
(In reply to comment #9)
> What I'd like you to post is the site.cfg file generated by the ebuild.
> Furthermore you are both consumer of sage-on-gentoo, what exact version of
> numpy are you using and did you enable the lapack useflag with it?

I now have installed

=dev-python/numpy-1.7.0_pre20121217-r2 with USE="lapack" enabled and 

# cat /usr/lib64/python2.7/site-packages/numpy/distutils/site.cfg
[blas]
include_dirs = /usr/include/atlas 
library_dirs = :/usr/lib64
blas_libs = atlcblas,atlas,f77blas,atlas 
[lapack]
library_dirs = :/usr/lib64
lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas

and in this state scipy fails to build exactly if >=pkgconfig-0.28 is installed.
Comment 11 François Bissey 2013-02-11 20:37:06 UTC
Apart from syncing the overlay and move to a better numpy ebuild can you provide scipy's site.cfg
Comment 12 Thomas Kahle (RETIRED) gentoo-dev 2013-02-11 21:09:17 UTC
(In reply to comment #11)
> Apart from syncing the overlay and move to a better numpy ebuild can you
> provide scipy's site.cfg

Yeah, I updated my numpy now.  And here is the failing site.cfg, look how 
1) -lm is missing (I guess that is no problem if it is on the ldflags somewhere else?)
2) There is some duplication

# cat ./work/scipy-0.11.0/site.cfg
[blas]
include_dirs = /usr/include/atlas 
library_dirs = :/usr/lib64
blas_libs = atlcblas,atlas,f77blas,atlas 
[lapack]
library_dirs = :/usr/lib64
lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas

$ pkg-config --libs blas
-lf77blas -lm -latlas 
$ pkg-config --libs lapack
-latllapack -lm -latlas -lf77blas -lm -latlas -latlcblas -lm -latlas

While with pkgconfig-0.27 it looks like this:

$ pkg-config --libs blas 
-lf77blas -lm -latlas
$ pkg-config --libs lapack
-latllapack -lf77blas -latlcblas -lm -latlas
Comment 13 François Bissey 2013-02-11 22:23:11 UTC
pkgconfig 0.28 handles rquires slightly differently, your diff between 0.27.1 and 0.28 - underscore for trailing whitespace added by me:
--- site.cfg-27	2013-02-12 10:14:29.000000000 +1300
+++ site.cfg-28	2013-02-12 10:16:28.000000000 +1300
@@ -1,7 +1,7 @@
 [blas]
-include_dirs = /usr/include/atlas__
-library_dirs =  :/usr/lib64
-blas_libs = atlcblas,f77blas,atlas__
+include_dirs = /usr/include/atlas_
+library_dirs = :/usr/lib64
+blas_libs = atlcblas,atlas,f77blas,atlas_
 [lapack]
-library_dirs =  :/usr/lib64
-lapack_libs = atllapack,f77blas,atlcblas,atlas__
+library_dirs = :/usr/lib64
+lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas_

So with 0.28 there is only one trailing white space for *_libs instead of two and this is the stuff of #454254, Note that it also means library_dirs have one space instead of two before ":" but that is absolutely irrelevant.
The next interesting thing would be to know if it is specific to atlas or if it can be reproduced for reference. In #454254 I noted that I didn't see that for numpy/scipy and that's still the case and I am defaulting to openblas/lapack-reference.

A quick check shows that using lapack-atlas with openblas-blas and openblas-cblas works.

Next I will see if doctoring site.cfg for atlas does improve things.
Comment 14 François Bissey 2013-02-11 22:34:05 UTC
Multiple atlas in *_libs is the root cause of the problem (it should also have an impact on numpy[lapack]). The following site.cfg works fine:
[blas]
include_dirs = /usr/include/atlas
library_dirs = :/usr/lib64
blas_libs = atlcblas,f77blas,atlas
[lapack]
library_dirs = :/usr/lib64
lapack_libs = atllapack,f77blas,atlcblas,atlas

Adding "atlas" in any of the other original locations will results in things going boom. That's definitely a change due to how pkgconfig handles "require" in .pc files. The problem is not apparent with other implementation because they don't have an auxiliary library littering the place. 

In this instance we need to remove duplications I think.
Comment 15 Thomas Kahle (RETIRED) gentoo-dev 2013-02-12 09:25:21 UTC
(In reply to comment #14)
> Multiple atlas in *_libs is the root cause of the problem (it should also
> have an impact on numpy[lapack]). The following site.cfg works fine:
> [blas]
> include_dirs = /usr/include/atlas
> library_dirs = :/usr/lib64
> blas_libs = atlcblas,f77blas,atlas
> [lapack]
> library_dirs = :/usr/lib64
> lapack_libs = atllapack,f77blas,atlcblas,atlas
> 
> Adding "atlas" in any of the other original locations will results in things
> going boom. That's definitely a change due to how pkgconfig handles
> "require" in .pc files. The problem is not apparent with other
> implementation because they don't have an auxiliary library littering the
> place. 
> 
> In this instance we need to remove duplications I think.

Let's play the blame game!  Is it a bug in pkgconfig or is it a bad pkgconfig file that atlas provides?
Comment 16 Justin Lecher (RETIRED) gentoo-dev 2013-02-12 09:28:37 UTC
$ cat /usr/lib64/pkgconfig/blas.pc 
prefix=/usr
libdir=${prefix}/lib64
includedir=${prefix}/include
Name: atlas-blas
Description: atlas atlas-blas
Version: 3.10.1
URL: http://math-atlas.sourceforge.net/
Libs: -L${libdir} -lf77blas -lm -latlas
Cflags: -I${includedir}/atlas

$ cat /usr/lib64/pkgconfig/lapack.pc 
prefix=/usr
libdir=${prefix}/lib64
includedir=${prefix}/include
Name: atlas-lapack
Description: atlas atlas-lapack
Version: 3.10.1
URL: http://math-atlas.sourceforge.net/
Libs: -L${libdir} -latllapack -lm -latlas
Cflags: -I${includedir}/atlas
Requires: blas cblas
Comment 17 Justin Lecher (RETIRED) gentoo-dev 2013-02-12 09:29:25 UTC
So, my pc files look normal.
Are you sure you selected lapck, blas and cblas to atlas?
Comment 18 François Bissey 2013-02-12 09:38:11 UTC
Sure the pc file look normal but pkgconfig changed the way it handles duplicates, from http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html
The second is that aggressive
  removal of all duplicate arguments has been scaled back so that just
  consecutive duplicate arguments are removed. One result of this change
  is that some flags could be repeated in the final output, especially
  flags from non-pkg-config packages like -lm. Since pkg-config rarely
  has enough knowledge here about the right thing to do, we throw the
  duplicate arguments at the compiler/linker and trust it will do the
  right thing.

Now pkgconfig is not really handling the use case we have at hands which is why we have to do so much parsing of its output.
Comment 19 Thomas Kahle (RETIRED) gentoo-dev 2013-02-12 09:49:31 UTC
(In reply to comment #18)
> Sure the pc file look normal but pkgconfig changed the way it handles
> duplicates, from
> http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html
> The second is that aggressive
>   removal of all duplicate arguments has been scaled back so that just
>   consecutive duplicate arguments are removed. One result of this change
>   is that some flags could be repeated in the final output, especially
>   flags from non-pkg-config packages like -lm. Since pkg-config rarely
>   has enough knowledge here about the right thing to do, we throw the
>   duplicate arguments at the compiler/linker and trust it will do the
>   right thing.
> 
> Now pkgconfig is not really handling the use case we have at hands which is
> why we have to do so much parsing of its output.

so scipy is to blame for doing something like 

LDADD = $(pkgconfig --libs atlas)$(pkgconfig --libs blas)  $(pkgconfig --libs atlas)

?
Comment 20 Thomas Kahle (RETIRED) gentoo-dev 2013-02-12 09:51:07 UTC
(In reply to comment #19)
> (In reply to comment #18)
> > Sure the pc file look normal but pkgconfig changed the way it handles
> > duplicates, from
> > http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html
> > The second is that aggressive
> >   removal of all duplicate arguments has been scaled back so that just
> >   consecutive duplicate arguments are removed. One result of this change
> >   is that some flags could be repeated in the final output, especially
> >   flags from non-pkg-config packages like -lm. Since pkg-config rarely
> >   has enough knowledge here about the right thing to do, we throw the
> >   duplicate arguments at the compiler/linker and trust it will do the
> >   right thing.
> > 
> > Now pkgconfig is not really handling the use case we have at hands which is
> > why we have to do so much parsing of its output.
> 
> so scipy is to blame for doing something like 
> 
> LDADD = $(pkgconfig --libs atlas)$(pkgconfig --libs blas)  $(pkgconfig
> --libs atlas)

Argh, I mean 

LDADD = $(pkconfig --libs --libs-only-L lapack)

I think that is what they call and then I get this here:

$ pkg-config --libs --libs-only-L lapack
-latllapack -lm -latlas -lf77blas -lm -latlas -latlcblas -lm -latlas
Comment 21 François Bissey 2013-02-12 09:58:44 UTC
No. We as _gentoo_ take the output of pkg-config, parse it to remove "-l", "-lm" "-lpthreads" and some spaces, make it into a comma separated list. Then we write that in a site.cfg that is how python build system like and numpy's in particular likes it. And scipy fails to digest the list properly when there are duplicates.

I'll check that statement with something else than atlas to make sure.

I would presume the same problem happens with numpy[lapack].
Comment 22 François Bissey 2013-02-12 10:15:47 UTC
yep, if I use openblas and repeat openblas in site.cfg, boom. scipy for some reason doesn't handle the duplication and throw a tantrum. Make that numpy throw a tantrum because it is numpy distutils that are used.
Comment 23 Julian Ospald 2013-02-12 13:45:40 UTC
maybe we should try to unhack this, but I know that blas is a pita...
Comment 24 François Bissey 2013-02-12 20:38:37 UTC
The problem is only with atlas so far. I believe it can be fixed in upstream numpy, I'll open an issue with them (I have some contacts already because of my work on sage).
In the mean time for completeness, I have mentioned that numpy[lapack] should fail also. But because lapack support is optional it is probably a silent failure and you end up with a numpy without lapack.

A little bit of debugging shows that numpy code in _check_libs (distutils/system_info.py) actually construct a list that excludes duplicates but if the resulting list is not the same length than the original one it decides it is not a valid list.
Comment 25 François Bissey 2013-02-12 21:16:01 UTC
Created attachment 338728 [details, diff]
This patch allow for duplicate libraries to be passed to system_info.py without

I have an ugly patch for numpy-1.6.2 in attachment. Patch for upcoming 1.7.0 will be slightly different as _check_libs has changed quite a bit. I'll update the sage-on-gentoo overlay with a matching patch for 1.7.0.

I believe the code logic would cause the failure with both numpy 1.6.2 and numpy 1.7.0 but I haven't checked this however I believe all the people that have seen this had numpy-1.7.0_rc from the sage-on-gentoo overlay so a check with numpy 1.6.2 would be welcome.
Comment 26 Steven Trogdon 2013-02-12 21:22:48 UTC
(In reply to comment #14)
> Multiple atlas in *_libs is the root cause of the problem (it should also
> have an impact on numpy[lapack]). The following site.cfg works fine:
> [blas]
> include_dirs = /usr/include/atlas
> library_dirs = :/usr/lib64
> blas_libs = atlcblas,f77blas,atlas
> [lapack]
> library_dirs = :/usr/lib64
> lapack_libs = atllapack,f77blas,atlcblas,atlas
> 
> Adding "atlas" in any of the other original locations will results in things
> going boom. That's definitely a change due to how pkgconfig handles
> "require" in .pc files. The problem is not apparent with other
> implementation because they don't have an auxiliary library littering the
> place. 
> 
> In this instance we need to remove duplications I think.

Can't the output of pkgconfig be "sanitized" to produce this for both numpy and scipy

@@ -66,7 +66,7 @@
 pc_libs() {
        pkg-config --libs-only-l $@ | \
                sed -e 's/[ ]-l*\(pthread\|m\)[ ]*//g' \
-               -e 's/^-l//' -e 's/[ ]*-l/,/g'
+               -e 's/^-l//' -e 's/[ ]*-l/,/g' -e 's/atlas,//g' -e 's/[ ]*$//g'
 }

as a temporary fix.
Comment 27 Sébastien Fabbro (RETIRED) gentoo-dev 2013-02-12 21:27:46 UTC
(In reply to comment #16)
> $ cat /usr/lib64/pkgconfig/blas.pc 
...
> Libs: -L${libdir} -lf77blas -lm -latlas
...
> $ cat /usr/lib64/pkgconfig/lapack.pc 
...
> Libs: -L${libdir} -latllapack -lm -latlas

i think -lm -latlas would actually be better placed in Libs.private, which would also resolve the problem. could anyone with the issue edit their atlas.pc and atlas-threads.pc to see whether it would break other packages?
Comment 28 François Bissey 2013-02-12 21:36:39 UTC
(In reply to comment #27)
> (In reply to comment #16)
> > $ cat /usr/lib64/pkgconfig/blas.pc 
> ...
> > Libs: -L${libdir} -lf77blas -lm -latlas
> ...
> > $ cat /usr/lib64/pkgconfig/lapack.pc 
> ...
> > Libs: -L${libdir} -latllapack -lm -latlas
> 
> i think -lm -latlas would actually be better placed in Libs.private, which
> would also resolve the problem. could anyone with the issue edit their
> atlas.pc and atlas-threads.pc to see whether it would break other packages?

Would that work properly on a prefix? I am not sure it would fly on macos or aix but I wouldn't recommend trying the atlas ebuild out on these either.
Comment 29 François Bissey 2013-02-12 21:57:54 UTC
(In reply to comment #27)
> (In reply to comment #16)
> > $ cat /usr/lib64/pkgconfig/blas.pc 
> ...
> > Libs: -L${libdir} -lf77blas -lm -latlas
> ...
> > $ cat /usr/lib64/pkgconfig/lapack.pc 
> ...
> > Libs: -L${libdir} -latllapack -lm -latlas
> 
> i think -lm -latlas would actually be better placed in Libs.private, which
> would also resolve the problem. could anyone with the issue edit their
> atlas.pc and atlas-threads.pc to see whether it would break other packages?

Seems to be linking stuff properly but my own atlas install seems to be foobar so no real testing aside from "ldd -r".
Comment 30 Steven Trogdon 2013-02-13 00:13:33 UTC
(In reply to comment #29)
> (In reply to comment #27)
> > (In reply to comment #16)
> > > $ cat /usr/lib64/pkgconfig/blas.pc 
> > ...
> > > Libs: -L${libdir} -lf77blas -lm -latlas
> > ...
> > > $ cat /usr/lib64/pkgconfig/lapack.pc 
> > ...
> > > Libs: -L${libdir} -latllapack -lm -latlas
> > 
> > i think -lm -latlas would actually be better placed in Libs.private, which
> > would also resolve the problem. could anyone with the issue edit their
> > atlas.pc and atlas-threads.pc to see whether it would break other packages?
> 
> Seems to be linking stuff properly but my own atlas install seems to be
> foobar so no real testing aside from "ldd -r".

I can confirm that the results of "ldd -r" appear correct here after doing this - achieved though editing the atlas ebuild. Numpy and Scipy build just fine. Build done in prefix.
Comment 31 Juergen Rose 2013-02-21 00:56:27 UTC
BTW, if I mask >=dev-python/numpy-1.7.0_rc2, what means that numpy-1.6.2-r2 will be installed I can emerge scipy-0.11.0-r1.
Comment 32 François Bissey 2013-02-21 01:05:32 UTC
(In reply to comment #31)
> BTW, if I mask >=dev-python/numpy-1.7.0_rc2, what means that numpy-1.6.2-r2
> will be installed I can emerge scipy-0.11.0-r1.

Oh, someone doing what I asked in comment 25. So you are saying this is specific to numpy-1.7.0_rc2 (which ebuild wise is cloned from 1.6.2-r2). Looking at the code I would have thought the problem would have been in both.

Bicatali's solution is probably the best in the log run but I can fix numpy-1.7.0_rc2 in the s-o-g overlay.
Comment 33 Jeroen Roovers (RETIRED) gentoo-dev 2014-05-08 11:21:53 UTC

*** This bug has been marked as a duplicate of bug 386269 ***