| Summary: | =sci-libs/scipy-0.11.0-r1 does not find blas with >=dev-util/pkgconfig-0.28 | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Thomas Kahle (RETIRED) <tomka> |
| Component: | Current packages | Assignee: | Gentoo Science Related Packages <sci> |
| Status: | RESOLVED DUPLICATE | ||
| Severity: | normal | CC: | alito, jeanfrancis, rose, strogdon |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
/var/tmp/portage/sci-libs/scipy-0.11.0-r1/temp/build.log
This patch allow for duplicate libraries to be passed to system_info.py without |
||
|
Description
Thomas Kahle (RETIRED)
2013-02-09 11:09:09 UTC
build.log please. Cannot reproduce it here. Hi Thomas, just out of curiosity can you retry after downgrading pkgconfig to 0.27.1? My previous experiments seemed to show that numpy/scipy were unaffected by bug #454254 but this is a new scipy ebuild and I haven't checked it. (In reply to comment #3) > Hi Thomas, > > just out of curiosity can you retry after downgrading pkgconfig to 0.27.1? > My previous experiments seemed to show that numpy/scipy were unaffected by > bug #454254 but this is a new scipy ebuild and I haven't checked it. Yes, works with pkconfig-0.27.1 I am using pkgconfig-0.28 but cannot reproduce it at all. (In reply to comment #5) > I am using pkgconfig-0.28 but cannot reproduce it at all. Maybe related to atlas? Is there a way to produce all information that pkg-config would ever output for a package so that we could compare the two outputs? I have the same issue, 'emerge scipy' fails with:
...
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1488: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:558: UserWarning: Specified path is invalid.
warnings.warn('Specified path %s is invalid.' % d)
libraries atlcblas,atlas,f77blas,atlas not found in ['/usr/lib64']
NOT AVAILABLE
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1497: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
warnings.warn(BlasNotFoundError.__doc__)
blas_src_info:
NOT AVAILABLE
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:1500: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
...
root@leopard:/root(361)# eselect blas list
Available providers for blas:
[1] atlas *
[2] atlas-threads
[3] eigen
root@leopard:/root(362)# pkg-config --libs blas
-lf77blas -lm -latlas
root@leopard:/root(363)# qlist -Iv pkgconfig
dev-perl/extutils-pkgconfig-1.140.0
dev-util/pkgconfig-0.28
virtual/pkgconfig-0
Created attachment 338582 [details]
/var/tmp/portage/sci-libs/scipy-0.11.0-r1/temp/build.log
What I'd like you to post is the site.cfg file generated by the ebuild. Furthermore you are both consumer of sage-on-gentoo, what exact version of numpy are you using and did you enable the lapack useflag with it? (In reply to comment #9) > What I'd like you to post is the site.cfg file generated by the ebuild. > Furthermore you are both consumer of sage-on-gentoo, what exact version of > numpy are you using and did you enable the lapack useflag with it? I now have installed =dev-python/numpy-1.7.0_pre20121217-r2 with USE="lapack" enabled and # cat /usr/lib64/python2.7/site-packages/numpy/distutils/site.cfg [blas] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 blas_libs = atlcblas,atlas,f77blas,atlas [lapack] library_dirs = :/usr/lib64 lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas and in this state scipy fails to build exactly if >=pkgconfig-0.28 is installed. Apart from syncing the overlay and move to a better numpy ebuild can you provide scipy's site.cfg (In reply to comment #11) > Apart from syncing the overlay and move to a better numpy ebuild can you > provide scipy's site.cfg Yeah, I updated my numpy now. And here is the failing site.cfg, look how 1) -lm is missing (I guess that is no problem if it is on the ldflags somewhere else?) 2) There is some duplication # cat ./work/scipy-0.11.0/site.cfg [blas] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 blas_libs = atlcblas,atlas,f77blas,atlas [lapack] library_dirs = :/usr/lib64 lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas $ pkg-config --libs blas -lf77blas -lm -latlas $ pkg-config --libs lapack -latllapack -lm -latlas -lf77blas -lm -latlas -latlcblas -lm -latlas While with pkgconfig-0.27 it looks like this: $ pkg-config --libs blas -lf77blas -lm -latlas $ pkg-config --libs lapack -latllapack -lf77blas -latlcblas -lm -latlas pkgconfig 0.28 handles rquires slightly differently, your diff between 0.27.1 and 0.28 - underscore for trailing whitespace added by me: --- site.cfg-27 2013-02-12 10:14:29.000000000 +1300 +++ site.cfg-28 2013-02-12 10:16:28.000000000 +1300 @@ -1,7 +1,7 @@ [blas] -include_dirs = /usr/include/atlas__ -library_dirs = :/usr/lib64 -blas_libs = atlcblas,f77blas,atlas__ +include_dirs = /usr/include/atlas_ +library_dirs = :/usr/lib64 +blas_libs = atlcblas,atlas,f77blas,atlas_ [lapack] -library_dirs = :/usr/lib64 -lapack_libs = atllapack,f77blas,atlcblas,atlas__ +library_dirs = :/usr/lib64 +lapack_libs = atllapack,atlas,f77blas,atlas,atlcblas,atlas_ So with 0.28 there is only one trailing white space for *_libs instead of two and this is the stuff of #454254, Note that it also means library_dirs have one space instead of two before ":" but that is absolutely irrelevant. The next interesting thing would be to know if it is specific to atlas or if it can be reproduced for reference. In #454254 I noted that I didn't see that for numpy/scipy and that's still the case and I am defaulting to openblas/lapack-reference. A quick check shows that using lapack-atlas with openblas-blas and openblas-cblas works. Next I will see if doctoring site.cfg for atlas does improve things. Multiple atlas in *_libs is the root cause of the problem (it should also have an impact on numpy[lapack]). The following site.cfg works fine: [blas] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 blas_libs = atlcblas,f77blas,atlas [lapack] library_dirs = :/usr/lib64 lapack_libs = atllapack,f77blas,atlcblas,atlas Adding "atlas" in any of the other original locations will results in things going boom. That's definitely a change due to how pkgconfig handles "require" in .pc files. The problem is not apparent with other implementation because they don't have an auxiliary library littering the place. In this instance we need to remove duplications I think. (In reply to comment #14) > Multiple atlas in *_libs is the root cause of the problem (it should also > have an impact on numpy[lapack]). The following site.cfg works fine: > [blas] > include_dirs = /usr/include/atlas > library_dirs = :/usr/lib64 > blas_libs = atlcblas,f77blas,atlas > [lapack] > library_dirs = :/usr/lib64 > lapack_libs = atllapack,f77blas,atlcblas,atlas > > Adding "atlas" in any of the other original locations will results in things > going boom. That's definitely a change due to how pkgconfig handles > "require" in .pc files. The problem is not apparent with other > implementation because they don't have an auxiliary library littering the > place. > > In this instance we need to remove duplications I think. Let's play the blame game! Is it a bug in pkgconfig or is it a bad pkgconfig file that atlas provides? $ cat /usr/lib64/pkgconfig/blas.pc
prefix=/usr
libdir=${prefix}/lib64
includedir=${prefix}/include
Name: atlas-blas
Description: atlas atlas-blas
Version: 3.10.1
URL: http://math-atlas.sourceforge.net/
Libs: -L${libdir} -lf77blas -lm -latlas
Cflags: -I${includedir}/atlas
$ cat /usr/lib64/pkgconfig/lapack.pc
prefix=/usr
libdir=${prefix}/lib64
includedir=${prefix}/include
Name: atlas-lapack
Description: atlas atlas-lapack
Version: 3.10.1
URL: http://math-atlas.sourceforge.net/
Libs: -L${libdir} -latllapack -lm -latlas
Cflags: -I${includedir}/atlas
Requires: blas cblas
So, my pc files look normal. Are you sure you selected lapck, blas and cblas to atlas? Sure the pc file look normal but pkgconfig changed the way it handles duplicates, from http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html The second is that aggressive removal of all duplicate arguments has been scaled back so that just consecutive duplicate arguments are removed. One result of this change is that some flags could be repeated in the final output, especially flags from non-pkg-config packages like -lm. Since pkg-config rarely has enough knowledge here about the right thing to do, we throw the duplicate arguments at the compiler/linker and trust it will do the right thing. Now pkgconfig is not really handling the use case we have at hands which is why we have to do so much parsing of its output. (In reply to comment #18) > Sure the pc file look normal but pkgconfig changed the way it handles > duplicates, from > http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html > The second is that aggressive > removal of all duplicate arguments has been scaled back so that just > consecutive duplicate arguments are removed. One result of this change > is that some flags could be repeated in the final output, especially > flags from non-pkg-config packages like -lm. Since pkg-config rarely > has enough knowledge here about the right thing to do, we throw the > duplicate arguments at the compiler/linker and trust it will do the > right thing. > > Now pkgconfig is not really handling the use case we have at hands which is > why we have to do so much parsing of its output. so scipy is to blame for doing something like LDADD = $(pkgconfig --libs atlas)$(pkgconfig --libs blas) $(pkgconfig --libs atlas) ? (In reply to comment #19) > (In reply to comment #18) > > Sure the pc file look normal but pkgconfig changed the way it handles > > duplicates, from > > http://lists.freedesktop.org/archives/pkg-config/2013-January/000912.html > > The second is that aggressive > > removal of all duplicate arguments has been scaled back so that just > > consecutive duplicate arguments are removed. One result of this change > > is that some flags could be repeated in the final output, especially > > flags from non-pkg-config packages like -lm. Since pkg-config rarely > > has enough knowledge here about the right thing to do, we throw the > > duplicate arguments at the compiler/linker and trust it will do the > > right thing. > > > > Now pkgconfig is not really handling the use case we have at hands which is > > why we have to do so much parsing of its output. > > so scipy is to blame for doing something like > > LDADD = $(pkgconfig --libs atlas)$(pkgconfig --libs blas) $(pkgconfig > --libs atlas) Argh, I mean LDADD = $(pkconfig --libs --libs-only-L lapack) I think that is what they call and then I get this here: $ pkg-config --libs --libs-only-L lapack -latllapack -lm -latlas -lf77blas -lm -latlas -latlcblas -lm -latlas No. We as _gentoo_ take the output of pkg-config, parse it to remove "-l", "-lm" "-lpthreads" and some spaces, make it into a comma separated list. Then we write that in a site.cfg that is how python build system like and numpy's in particular likes it. And scipy fails to digest the list properly when there are duplicates. I'll check that statement with something else than atlas to make sure. I would presume the same problem happens with numpy[lapack]. yep, if I use openblas and repeat openblas in site.cfg, boom. scipy for some reason doesn't handle the duplication and throw a tantrum. Make that numpy throw a tantrum because it is numpy distutils that are used. maybe we should try to unhack this, but I know that blas is a pita... The problem is only with atlas so far. I believe it can be fixed in upstream numpy, I'll open an issue with them (I have some contacts already because of my work on sage). In the mean time for completeness, I have mentioned that numpy[lapack] should fail also. But because lapack support is optional it is probably a silent failure and you end up with a numpy without lapack. A little bit of debugging shows that numpy code in _check_libs (distutils/system_info.py) actually construct a list that excludes duplicates but if the resulting list is not the same length than the original one it decides it is not a valid list. Created attachment 338728 [details, diff]
This patch allow for duplicate libraries to be passed to system_info.py without
I have an ugly patch for numpy-1.6.2 in attachment. Patch for upcoming 1.7.0 will be slightly different as _check_libs has changed quite a bit. I'll update the sage-on-gentoo overlay with a matching patch for 1.7.0.
I believe the code logic would cause the failure with both numpy 1.6.2 and numpy 1.7.0 but I haven't checked this however I believe all the people that have seen this had numpy-1.7.0_rc from the sage-on-gentoo overlay so a check with numpy 1.6.2 would be welcome.
(In reply to comment #14) > Multiple atlas in *_libs is the root cause of the problem (it should also > have an impact on numpy[lapack]). The following site.cfg works fine: > [blas] > include_dirs = /usr/include/atlas > library_dirs = :/usr/lib64 > blas_libs = atlcblas,f77blas,atlas > [lapack] > library_dirs = :/usr/lib64 > lapack_libs = atllapack,f77blas,atlcblas,atlas > > Adding "atlas" in any of the other original locations will results in things > going boom. That's definitely a change due to how pkgconfig handles > "require" in .pc files. The problem is not apparent with other > implementation because they don't have an auxiliary library littering the > place. > > In this instance we need to remove duplications I think. Can't the output of pkgconfig be "sanitized" to produce this for both numpy and scipy @@ -66,7 +66,7 @@ pc_libs() { pkg-config --libs-only-l $@ | \ sed -e 's/[ ]-l*\(pthread\|m\)[ ]*//g' \ - -e 's/^-l//' -e 's/[ ]*-l/,/g' + -e 's/^-l//' -e 's/[ ]*-l/,/g' -e 's/atlas,//g' -e 's/[ ]*$//g' } as a temporary fix. (In reply to comment #16) > $ cat /usr/lib64/pkgconfig/blas.pc ... > Libs: -L${libdir} -lf77blas -lm -latlas ... > $ cat /usr/lib64/pkgconfig/lapack.pc ... > Libs: -L${libdir} -latllapack -lm -latlas i think -lm -latlas would actually be better placed in Libs.private, which would also resolve the problem. could anyone with the issue edit their atlas.pc and atlas-threads.pc to see whether it would break other packages? (In reply to comment #27) > (In reply to comment #16) > > $ cat /usr/lib64/pkgconfig/blas.pc > ... > > Libs: -L${libdir} -lf77blas -lm -latlas > ... > > $ cat /usr/lib64/pkgconfig/lapack.pc > ... > > Libs: -L${libdir} -latllapack -lm -latlas > > i think -lm -latlas would actually be better placed in Libs.private, which > would also resolve the problem. could anyone with the issue edit their > atlas.pc and atlas-threads.pc to see whether it would break other packages? Would that work properly on a prefix? I am not sure it would fly on macos or aix but I wouldn't recommend trying the atlas ebuild out on these either. (In reply to comment #27) > (In reply to comment #16) > > $ cat /usr/lib64/pkgconfig/blas.pc > ... > > Libs: -L${libdir} -lf77blas -lm -latlas > ... > > $ cat /usr/lib64/pkgconfig/lapack.pc > ... > > Libs: -L${libdir} -latllapack -lm -latlas > > i think -lm -latlas would actually be better placed in Libs.private, which > would also resolve the problem. could anyone with the issue edit their > atlas.pc and atlas-threads.pc to see whether it would break other packages? Seems to be linking stuff properly but my own atlas install seems to be foobar so no real testing aside from "ldd -r". (In reply to comment #29) > (In reply to comment #27) > > (In reply to comment #16) > > > $ cat /usr/lib64/pkgconfig/blas.pc > > ... > > > Libs: -L${libdir} -lf77blas -lm -latlas > > ... > > > $ cat /usr/lib64/pkgconfig/lapack.pc > > ... > > > Libs: -L${libdir} -latllapack -lm -latlas > > > > i think -lm -latlas would actually be better placed in Libs.private, which > > would also resolve the problem. could anyone with the issue edit their > > atlas.pc and atlas-threads.pc to see whether it would break other packages? > > Seems to be linking stuff properly but my own atlas install seems to be > foobar so no real testing aside from "ldd -r". I can confirm that the results of "ldd -r" appear correct here after doing this - achieved though editing the atlas ebuild. Numpy and Scipy build just fine. Build done in prefix. BTW, if I mask >=dev-python/numpy-1.7.0_rc2, what means that numpy-1.6.2-r2 will be installed I can emerge scipy-0.11.0-r1. (In reply to comment #31) > BTW, if I mask >=dev-python/numpy-1.7.0_rc2, what means that numpy-1.6.2-r2 > will be installed I can emerge scipy-0.11.0-r1. Oh, someone doing what I asked in comment 25. So you are saying this is specific to numpy-1.7.0_rc2 (which ebuild wise is cloned from 1.6.2-r2). Looking at the code I would have thought the problem would have been in both. Bicatali's solution is probably the best in the log run but I can fix numpy-1.7.0_rc2 in the s-o-g overlay. *** This bug has been marked as a duplicate of bug 386269 *** |