Summary: | sci-libs/scipy-0.9.0-r2 - numpy.distutils.system_info.BlasNotFoundError: Blas (http://www.netlib.org/blas/) libraries not found | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Juergen Rose <rose> |
Component: | [OLD] Unspecified | Assignee: | Gentoo Science Related Packages <sci> |
Status: | RESOLVED TEST-REQUEST | ||
Severity: | normal | CC: | javran.c, tomka |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: | updated numpy-15.1 ebuild |
Description
Juergen Rose
2011-10-08 12:56:28 UTC
It seems me, that this bug is connected with numpy-1.5.1. At systems where numpy-1.6.1-r1 is installed, I can also install scipy-0.9.0-r2. At systems where the installation of >numpy-1.5.1 is blocked by sage-4.7.1-r2 'emerge scipy' fails. OK, taking over from the gentoo-science mailing list. Output of: eselect blas list eselect cblas list eselect lapack list And what does the "site.cfg" produced by scipy in src_prepare looks like? Hi Francois, I have: root@caiman:/root(31)# eselect blas list Available providers for blas: [1] atlas * [2] atlas-threads root@caiman:/root(32)# eselect cblas list Available providers for cblas: [1] atlas * [2] atlas-threads [3] gsl root@caiman:/root(33)# eselect lapack list Available providers for lapack: [1] atlas * [2] atlas-threads I find the following site.cfg under /var/tmp/portage: root@caiman:/root(38)# find /var/tmp/portage/ -name site.cfg -ls 23145063 4 -rw-r--r-- 1 root root 376 Aug 25 09:37 /var/tmp/portage/sci-libs/scipy-0.9.0-r1/work/scipy-0.9.0/build/py3k/site.cfg 23128011 4 -rw-r--r-- 1 root root 376 Aug 25 09:32 /var/tmp/portage/sci-libs/scipy-0.9.0-r1/work/scipy-0.9.0/site.cfg 27591833 4 -rw-r--r-- 1 root root 203 Oct 11 07:57 /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/site.cfg 27560760 4 -rw-r--r-- 1 root root 203 Oct 11 07:52 /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/site.cfg The site.cfg for in work/scipy-0.9.0 and scipy-0.9.0/build/py3k are identical. The content of /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/site.cfg is: root@caiman:/root(41)# cat /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/site.cfg [blas] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 blas_libs = atlcblas, f77blas, m, atlas [lapack] library_dirs = :/usr/lib64 lapack_libs = atllapack, f77blas, atlcblas, m, atlas Then I still find /usr/lib64/python2.7/site-packages/numpy/distutils/site.cfg /usr/lib64/python3.2/site-packages/numpy/distutils/site.cfg which are also identical. root@caiman:/root(45)# cat /usr/lib64/python2.7/site-packages/numpy/distutils/site.cfg [atlas] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 atlas_libs = atlcblas, f77blas, m, atlas lapack_libs = atllapack, f77blas, atlcblas, m, atlas [blas_opt] include_dirs = /usr/include/atlas library_dirs = :/usr/lib64 libraries = atlcblas, f77blas, m, atlas [lapack_opt] library_dirs = :/usr/lib64 libraries = atllapack, f77blas, atlcblas, m, atlas Regards Juergen OK. I suddenly realise that your build seems to break in python-3.2.2. Could you confirm if it does work with just python-2.7? USE_PYTHON="2.7" emerge -uv scipy Hi Francois, USE_PYTHON="2.7" emerge -uv scipy works without problems. Juergen OK, so let's put some things straight. scipy doesn't look for site.cfg in its own build directory if it finds one in numpy. It only uses its own if there is none in numpy, this only happens if you build numpy without lapack. If we force numpy to be built with lapack for scipy then we can drop site.cfg from src_prepare altogether. I guess this fact may have been lost in the noise in the previous bug. My suspicions here is that the site.cfg from numpy-1.5.1 which has the old syntax doesn't play nice with the combo scipy-0.9.0-r2/python-3.2.2. Juergen could you replace site.cfg in /usr/lib64/python3.2/site-packages/numpy/distutils/site.cfg by the one generated by scipy (or alternatively just remove it) and then try again. Hi Francois, I now have no access to caiman where the error has originally happened. Now I am sitting at lynx with the same error. /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/site.cfg disappeared, because 'USE_PYTHON="2.7" emerge -uv scipy' was succesfull, any other site.cfg belonging to scipy disappeared, because I did 'emerge -C scipy' to do again 'emerge -uvDN world' for emerging scipy-0.9.0-r2. Thus I can't replace /usr/lib64/python3.2/site-packages/numpy/distutils/site.cfg by the one generated by scipy. That's why I renamed it: root@lynx:/root(15)# mv /usr/lib64/python3.2/site-packages/numpy/distutils/site.cfg /usr/lib64/python3.2/site-packages/numpy/distutils/site.cfg_orig The following 'emerge -uvDN world' fails with the old error: root@lynx:/root(16)# emerge -vuND world ... RefactoringTool: /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/integrate/setup.py /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/io/__init__.py ... /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/integrate/quadpack.py blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib64 libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib64 libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries blas,cblas,atlas not found in /usr/local/lib64 libraries blas,cblas,atlas not found in /usr/local/lib libraries blas,cblas,atlas not found in /usr/lib64 libraries blas,cblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries blas,cblas,atlas not found in /usr/local/lib64 libraries blas,cblas,atlas not found in /usr/local/lib libraries blas,cblas,atlas not found in /usr/lib64 libraries blas,cblas,atlas not found in /usr/lib NOT AVAILABLE Warning: No configuration returned, assuming unavailable./usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1399: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:527: UserWarning: Specified path is invalid. warnings.warn('Specified path %s is invalid.' % d) libraries atlcblas,f77blas,m,atlas not found in /usr/lib64 NOT AVAILABLE /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1408: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) blas_src_info: NOT AVAILABLE /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1411: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) Traceback (most recent call last): File "setup.py", line 181, in <module> setup_package() File "setup.py", line 173, in setup_package configuration=configuration ) File "/usr/lib64/python3.2/site-packages/numpy/distutils/core.py", line 152, in setup config = configuration() File "setup.py", line 122, in configuration config.add_subpackage('scipy') File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/setup.py", line 8, in configuration config.add_subpackage('integrate') File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/integrate/setup.py", line 10, in configuration blas_opt = get_info('blas_opt',notfound_action=2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py", line 310, in get_info return cl().get_info(notfound_action) File "/usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py", line 461, in get_info raise self.notfounderror(self.notfounderror.__doc__) numpy.distutils.system_info.BlasNotFoundError: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. * ERROR: sci-libs/scipy-0.9.0-r2 failed (compile phase): * Building failed with CPython 3.2 in distutils_building() function Next attempt, I copied the site.cfg from scipy to numpy: root@lynx:/root(24)# cp -a /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/site.cfg /usr/lib64/python3.2/site-packages/numpy/distutils/ 'emerge -vuND world' fails with: ... /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1411: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) Traceback (most recent call last): ... * ERROR: sci-libs/scipy-0.9.0-r2 failed (compile phase): * Building failed with CPython 3.2 in distutils_building() function Next test: I delete sage and try once more: root@lynx:/root(27)# emerge -C sage root@lynx:/root(28)# emerge -pvuND world These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild U ] dev-python/pexpect-2.4-r1 [2.0] USE="doc%* examples%*" 111 kB [1] [ebuild U ] dev-python/numpy-1.6.1-r1 [1.5.1] USE="doc lapack -test" 18,029 kB [0] [ebuild N ] sci-libs/scipy-0.9.0-r2 USE="doc -umfpack" 0 kB [0] root@lynx:/root(29)# emerge -vuND world works like a charm. I'll have to scratch my head a little bit more on that one. First off it looks like a python-3.2.2 problem, I cannot reproduce it with python-3.1. site.cfg is at fault but there has to be something else. Created attachment 290207 [details]
updated numpy-15.1 ebuild
OK the attached ebuild migrate numpy-1.5.1 to the new style style.cfg. It works here but of course I didn't check with python-3.2.2.
Could you try this to build numpy-1.5.1 and then scipy?
I created a local overlay /usr/local/portage/dev-python/numpy, put the new numpy-1.5.1.ebuild to that overlay, generated a new manifest, masked again >=dev-python/numpy-1.6.0 in /etc/portage/packages.mask. The next 'emerge -v1 numpy' downgrades succesfully numpy to numpy-1.5.1: root@lynx:/usr/local/portage/dev-python/numpy(30)# ebuild numpy-1.5.1.ebuild digest >>> Creating Manifest for /usr/local/portage/dev-python/numpy root@lynx:/usr/local/portage/dev-python/numpy(31)# emerge -v1 numpy These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild UD ] dev-python/numpy-1.5.1 [1.6.1-r1] USE="doc lapack -test" 18,067 kB [0=>1] Total: 1 package (1 downgrade), Size of downloads: 18,067 kB Portage tree and overlays: [0] /usr/portage [1] /usr/local/portage ... >>> dev-python/numpy-1.5.1 merged. >>> Auto-cleaning packages... >>> No outdated packages were found on your system. * GNU info directory index is up-to-date. The subsequent 'emerge -v1 scipy' fails again as far as I see with the same error: root@lynx:/usr/local/portage/dev-python/numpy(32)# emerge -v1 scipy These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] sci-libs/scipy-0.9.0-r2 USE="doc -umfpack" 0 kB Total: 1 package (1 reinstall), Size of downloads: 0 kB >>> Verifying ebuild manifests >>> Emerging (1 of 1) sci-libs/scipy-0.9.0-r2 ... RefactoringTool: /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/maxentropy/examples/bergerexample.py RefactoringTool: Warnings/messages while refactoring: RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/stats/distributions.py ### RefactoringTool: Line 479: You should use 'operator.mul([value],product(shape,axis=0),axis=0)' here. RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/weave/blitz_tools.py ### RefactoringTool: Line 36: could not convert: raise "inputs failed to pass size check." RefactoringTool: Python 3 does not support string exceptions RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/weave/ast_tools.py ### RefactoringTool: Line 183: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Line 183: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/weave/bytecodecompiler.py ### RefactoringTool: Line 1157: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Line 1157: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Line 150: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Line 150: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/weave/bytecodecompiler.py ### RefactoringTool: Line 242: could not convert: raise "Executing code failed." RefactoringTool: Python 3 does not support string exceptions RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/weave/c_spec.py ### RefactoringTool: Line 380: absolute and local imports together RefactoringTool: ### In file /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/interpolate/fitpack.py ### RefactoringTool: Line 1173: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Line 1173: cannot convert map(None, ...) with multiple arguments because map() now truncates to the shortest sequence RefactoringTool: Skipping implicit fixer: buffer ... /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/integrate/ode.py /var/tmp/portage/sci-libs/scipy-0.9.0-r2/work/scipy-0.9.0/build/py3k/scipy/integrate/quadpack.py blas_opt_info: blas_mkl_info: libraries mkl,vml,guide not found in /usr/local/lib64 libraries mkl,vml,guide not found in /usr/local/lib libraries mkl,vml,guide not found in /usr/lib64 libraries mkl,vml,guide not found in /usr/lib NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS libraries blas,cblas,atlas not found in /usr/local/lib64 libraries blas,cblas,atlas not found in /usr/local/lib libraries blas,cblas,atlas not found in /usr/lib64 libraries blas,cblas,atlas not found in /usr/lib NOT AVAILABLE atlas_blas_info: libraries blas,cblas,atlas not found in /usr/local/lib64 libraries blas,cblas,atlas not found in /usr/local/lib libraries blas,cblas,atlas not found in /usr/lib64 libraries blas,cblas,atlas not found in /usr/lib NOT AVAILABLE Warning: No configuration returned, assuming unavailable./usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1399: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) blas_info: /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:527: UserWarning: Specified path is invalid. warnings.warn('Specified path %s is invalid.' % d) libraries atlcblas,f77blas,m,atlas not found in /usr/lib64 NOT AVAILABLE /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1408: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) blas_src_info: NOT AVAILABLE /usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py:1411: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) Traceback (most recent call last): File "setup.py", line 181, in <module> setup_package() File "setup.py", line 173, in setup_package configuration=configuration ) File "/usr/lib64/python3.2/site-packages/numpy/distutils/core.py", line 152, in setup config = configuration() File "setup.py", line 122, in configuration config.add_subpackage('scipy') File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/setup.py", line 8, in configuration config.add_subpackage('integrate') File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 972, in add_subpackage caller_level = 2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 941, in get_subpackage caller_level = caller_level + 1) File "/usr/lib64/python3.2/site-packages/numpy/distutils/misc_util.py", line 878, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "scipy/integrate/setup.py", line 10, in configuration blas_opt = get_info('blas_opt',notfound_action=2) File "/usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py", line 310, in get_info return cl().get_info(notfound_action) File "/usr/lib64/python3.2/site-packages/numpy/distutils/system_info.py", line 461, in get_info raise self.notfounderror(self.notfounderror.__doc__) numpy.distutils.system_info.BlasNotFoundError: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. * ERROR: sci-libs/scipy-0.9.0-r2 failed (compile phase): * Building failed with CPython 3.2 in distutils_building() function * * Call stack: * ebuild.sh, line 91: Called src_compile * environment, line 6032: Called distutils_src_compile 'config_fc' '--noopt' '--noarch' * environment, line 1820: Called python_execute_function 'distutils_building' 'config_fc' '--noopt' '--noarch' * environment, line 4253: Called die * The specific snippet of code: * die "${failure_message}"; Thanks for trying. Back to head scratching. This is specific to python-3.2 but cannot fathom what yet. May be some incompatibilities between numpy-1.5.1 distutils and python-3.2.2 (how did it install in the first place then?). Actually can you attach a build log of numpy-1.5.1 in case there is some interesting info in there. I also ran into this, but it seems to be working with scipy-0.10.1. Maybe this bug can be closed as fixed upstream. (In reply to comment #13) > I also ran into this, but it seems to be working with scipy-0.10.1. > Maybe this bug can be closed as fixed upstream. I take that back. Just ran into it again on scipy-0.10.1 Completly mysterious why it works sometimes and fails on a revdep rebuild run later... Just a thought: I noticed today, that neither scipy-0.9 nor scipy-0.12 want to compile on my system. Yet, it doesn't depend on the python version. The problem seems to be the generated site.cfg: [blas] include_dirs = /opt/intel/composerxe-2013_update4.5.192/mkl/include library_dirs = /opt/intel/composerxe-2013_update4.5.192/mkl/lib/intel64 :/usr/lib64 blas_libs = mkl_rt,mkl_rt [lapack] library_dirs = /opt/intel/composerxe-2013_update4.5.192/mkl/lib/intel64 :/usr/lib64 lapack_libs = mkl_rt note the space in library_dirs. Removing it manually, and calling /usr/bin/python3.2 setup.py build --build-base /var/tmp/portage/sci-libs/scipy-0.12.0/work/scipy-0.12.0-python3_2/build --build-platlib /var/tmp/portage/sci-libs/scipy-0.12.0/work/scipy-0.12.0-python3_2/build/lib --build-purelib /var/tmp/portage/sci-libs/scipy-0.12.0/work/scipy-0.12.0-python3_2/build/lib --build-scripts /var/tmp/portage/sci-libs/scipy-0.12.0/work/scipy-0.12.0-python3_2/build/scripts config_fc --noopt --noarc setup works just fine. So I'm wondering how to fix this in the ebuild. echo "$(pkg-config --libs-only-L cblas blas):hhh" yields: -L/opt/intel/composerxe-2013_update4.5.192/mkl/lib/intel64 :hhh So the space is there already. I'm wondering if :${libdir} from library_dirs = $(pc_libdir cblas blas):${libdir} as written in the ebuild is needed. If $libdir is always /usr/lib64 it should be really unnecessary. I'd either remove ${libdir}, or trim the whitespace of the pc_libdir command, maybe by writing $(pc_libdir cblas blas | sed -e 's/ *$//g') -- which works for me *** Bug 509816 has been marked as a duplicate of this bug. *** *** Bug 456280 has been marked as a duplicate of this bug. *** Please see whether this is still valid for scipy 0.14/0.15. |