Summary: | media-libs/opencv - ld: lib/libopencv_core.so.3.4.1: undefined reference to `cblas_zgemm' | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Thomas Beutin <tb> |
Component: | Current packages | Assignee: | Amy Liffey <amynka> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | axl, cafaia, frp.bissey, leonchik1976, lumin, mail, rossbridger.cc, sam, sci |
Priority: | Normal | Keywords: | PullRequest |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
URL: | https://wiki.gentoo.org/wiki/Blas-lapack-switch | ||
See Also: | https://github.com/gentoo/gentoo/pull/17964 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 746596 | ||
Attachments: | build.log (gzipped) |
Description
Thomas Beutin
2019-11-15 13:55:58 UTC
Created attachment 596234 [details]
build.log (gzipped)
I have pinned down the bug. Please run: `eselect blas add lib64 /usr/lib64/blas/reference reference` manually. @lumin, the upgrade path from sci-libs/blas-reference to sci-libs/lapack-3.8 is not smooth. Somehow the above command in pkg_postinst() does not succeed leaving no viable blas candidates. Uninstallation of sci-libs/blas-reference removes /etc/env.d/blas/lib64/reference, and uninstallation of sci-libs/lapack-reference removes /etc/env.d/lapack/lib64/reference. Unfortunately they are uninstalled after sci-libs/lapack is installed, and /etc/env.d/blas/lib64/reference is not in the package file list and is not protected. (In reply to Benda Xu from comment #2) > I have pinned down the bug. > > Please run: > > `eselect blas add lib64 /usr/lib64/blas/reference reference` > > manually. i did it, but it doesn't helps - the issue persists. I am more inclined to think there is a flaw in the build system. The run of cmake on the 64bit ABI arch finds /usr/lib64/liblapack.so, however the error is on a cblas symbol. I think libopencv_core.so.3.4.1 expect to be compiled with a lapack that includes cblas symbols as well. [90/680] : && /usr/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=sandybridge -fomit-frame-pointer -O2 -pipe -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -Wno-implicit-fallthrough -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -Wl,-O1 -Wl,--as-needed -shared -Wl,-soname,libopencv_core.so.3.4 -o lib/libopencv_core.so.3.4.1 modules/core/CMakeFiles/opencv_core.dir/src/algorithm.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/alloc.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/arithm.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/array.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/batch_distance.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/channels.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/command_line_parser.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/conjugate_gradient.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/convert.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/convert.sse4_1.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/convert_c.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/convert_scale.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/copy.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/count_non_zero.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/cuda_gpu_mat.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/cuda_host_mem.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/cuda_info.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/cuda_stream.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/datastructs.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/directx.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/downhill_simplex.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/dxt.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/gl_core_3_1.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/glob.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/hal_internal.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/kmeans.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/lapack.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/lda.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/logger.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/lpsolver.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/lut.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/mathfuncs.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/mathfuncs_core.dispatch.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matmul.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_c.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_decomp.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_expressions.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_iterator.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_operations.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_sparse.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/matrix_wrap.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/mean.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/merge.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/minmax.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/norm.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/ocl.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/opencl/runtime/opencl_clamdblas.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/opencl/runtime/opencl_clamdfft.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/opencl/runtime/opencl_core.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/opengl.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/out.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/ovx.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/parallel.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/parallel_impl.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/pca.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_base64.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_c.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_cpp.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_json.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_types.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_xml.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/persistence_yml.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/rand.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/softfloat.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/split.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/stat.dispatch.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/stat_c.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/stl.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/sum.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/system.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/tables.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/trace.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/types.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/umatrix.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/utils/filesystem.cpp.o modules/core/CMakeFiles/opencv_core.dir/src/va_intel.cpp.o modules/core/CMakeFiles/opencv_core.dir/opencl_kernels_core.cpp.o -ldl -lm -lpthread -lrt -lGL -lGLU /usr/lib64/libtbb.so -lz -lva -lva-drm -llapack -L/usr/lib64 && : If -llapack doesn't provide cblas and lib/libopencv_core.so.3.4.1 does use cblas directly, then of course it is underlinked. Using openblas instead of reference would probably fix the issue. A more general fix would be pass -lcblas as well. A deep look at cmake/OpenCVFindLAPACK.cmake in opencv's source shows that for mkl, openblas and accelerate they are actually after * lapacke (headers) * cblas (headers) * lapack (libraries) stuff. The compilation check for the libraries only look for lapack functions. opencv also seem to provide its alternate lapacke headers in any case. I don't know that lapacke is actually required. But cblas definitely is. cmake doesn't really have a FindCBLAS module, just BLAS and LAPACK. My recommendation would be for the ebuild to pass -DLAPACK_LIBRARIES="-llapack -lcblas" to cmake when lapack is required. Probably should add virtual/cblas to the dependencies as well. (In reply to Thomas Beutin from comment #4) > (In reply to Benda Xu from comment #2) > > I have pinned down the bug. > > > > Please run: > > > > `eselect blas add lib64 /usr/lib64/blas/reference reference` > > > > manually. > > i did it, but it doesn't helps - the issue persists. How about reinstalling sci-libs/lapack? (In reply to François Bissey from comment #6) > A deep look at cmake/OpenCVFindLAPACK.cmake in opencv's source shows that > for mkl, openblas and accelerate they are actually after > * lapacke (headers) > * cblas (headers) > * lapack (libraries) > stuff. The compilation check for the libraries only look for lapack > functions. opencv also seem to provide its alternate lapacke headers in any > case. I don't know that lapacke is actually required. But cblas definitely > is. cmake doesn't really have a FindCBLAS module, just BLAS and LAPACK. > > My recommendation would be for the ebuild to pass > -DLAPACK_LIBRARIES="-llapack -lcblas" > to cmake when lapack is required. Probably should add virtual/cblas to the > dependencies as well. I don't understand. Lapack is a fortran library. Why does it depend on a c library? Lapack is indeed a fortran library. There is also lapacke (notice the final "e") which a c interface to it or a c version. Lapack depends on blas which also has c version called cblas. The point is, big libraries like openblas or MKL present one library that includes: lapack lapacke blas cblas and sometimes more. All in one single library I repeat. The issue here is that opencv actually depends not only on lapack but on cblas. But it expect to get everything from one single library. And the *-reference libraries in Gentoo are definitely separate libraries. I was a bit hasty and may be not clear. opencv calls functions from both lapack and cblas so needs to be linked with both. opencv is actually in C so it just wraps lapack's fortran calls but it is using cblas directly. (In reply to François Bissey from comment #10) > I was a bit hasty and may be not clear. opencv calls functions from both > lapack and cblas so needs to be linked with both. opencv is actually in C so > it just wraps lapack's fortran calls but it is using cblas directly. Thanks François, that makes a lot of sense to me now. It is not a bug of lapack we are providing. Let's call opencv maintainer for help, hopefully adding cblas linking explicitly. *** Bug 741626 has been marked as a duplicate of this bug. *** *** Bug 701940 has been marked as a duplicate of this bug. *** The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5c21b91ad2464439514aae406a0c0d5f310aca1d commit 5c21b91ad2464439514aae406a0c0d5f310aca1d Author: Ross Charles Campbell <rossbridger.cc@gmail.com> AuthorDate: 2020-10-14 02:17:02 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2020-10-16 21:47:50 +0000 media-libs/opencv: version bump to 4.5.0 Bug: https://bugs.gentoo.org/701790 Bug: https://bugs.gentoo.org/700176 Bug: https://bugs.gentoo.org/717812 Bug: https://bugs.gentoo.org/703658 Bug: https://bugs.gentoo.org/689838 Closes: https://bugs.gentoo.org/666716 Closes: https://bugs.gentoo.org/747163 Closes: https://bugs.gentoo.org/729190 Closes: https://bugs.gentoo.org/699730 Closes: https://bugs.gentoo.org/672840 Package-Manager: Portage-3.0.8, Repoman-3.0.1 Signed-off-by: Ross Charles Campbell <rossbridger.cc@gmail.com> Closes: https://github.com/gentoo/gentoo/pull/17899 Signed-off-by: Sam James <sam@gentoo.org> media-libs/opencv/Manifest | 2 + media-libs/opencv/opencv-4.5.0.ebuild | 555 ++++++++++++++++++++++++++++++++++ 2 files changed, 557 insertions(+) Still a thing in media-libs/opencv-4.5.0[lapack]: [108/488] : && /usr/bin/x86_64-pc-linux-gnu-g++ -O3 -pipe -march=znver1 -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -flto -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mf16c -mfma -mavx -mavx2 -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -Wl,-O1 -Wl,--as-needed -Wl,--gc-sections -Wl,--as-needed apps/version/CMakeFiles/opencv_version.dir/opencv_version.cpp.o -o bin/opencv_version -ldl -lm -lpthread -lrt /usr/lib64/libGL.so /usr/lib64/libGLU.so lib/libopencv_core.so.4.5.0 -Wl,-rpath-link,"\$ORIGIN" && : FAILED: bin/opencv_version : && /usr/bin/x86_64-pc-linux-gnu-g++ -O3 -pipe -march=znver1 -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -flto -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mf16c -mfma -mavx -mavx2 -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -Wl,-O1 -Wl,--as-needed -Wl,--gc-sections -Wl,--as-needed apps/version/CMakeFiles/opencv_version.dir/opencv_version.cpp.o -o bin/opencv_version -ldl -lm -lpthread -lrt /usr/lib64/libGL.so /usr/lib64/libGLU.so lib/libopencv_core.so.4.5.0 -Wl,-rpath-link,"\$ORIGIN" && : /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: lib/libopencv_core.so.4.5.0: undefined reference to `cblas_zgemm' /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: lib/libopencv_core.so.4.5.0: undefined reference to `cblas_sgemm' /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: lib/libopencv_core.so.4.5.0: undefined reference to `cblas_dgemm' /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: lib/libopencv_core.so.4.5.0: undefined reference to `cblas_cgemm' collect2: error: ld returned 1 exit status This is with sci-libs/lapack-3.9.0 USE="lapacke -doc -eselect-ldso -test", currently the latest one in the tree. Nothing changed, still not using -cblas when linking. I suggested a work around during src_config in an earlier comment. I didn't enable lapacke USE for lapack, and opencv compiles fine, but after I rebuilt lapack with that USE, I was able to reproduce the same error. That's an interesting comment. It suggests that some internal functions are used when a system lapacke is not found. What does eselect lapack show say for you? (In reply to François Bissey from comment #18) > That's an interesting comment. It suggests that some internal functions are > used when a system lapacke is not found. > What does > eselect lapack show > say for you? # eselect lapack show !!! Error: Can't load module lapack exiting (In reply to François Bissey from comment #18) > That's an interesting comment. It suggests that some internal functions are > used when a system lapacke is not found. > What does > eselect lapack show > say for you? I'm running with USE="-eselect-ldso" (which is the default), so there is no lapack eselect module at all. OK, I didn't realise that. So just netlib lapack/cblas/blas/lapacke. Which are all in separate libraries and not one super library like openblas or mkl. The situation is getting more elusive: it turns out opencv doesn't even link with lapack if lapacke USE is not enabled (In reply to Ross Charles C. from comment #22) > The situation is getting more elusive: it turns out opencv doesn't even link > with lapack if lapacke USE is not enabled Not really, I suspected as much from Maciej's comments earlier. It looks like a case where the real dependency should be lapacke. And I suspect an internal copy is used if no external library is provided. So as I see it, this bug contains two levels: 1. OpenCV requires a lapacke implementation (virtual/lapacke) or lapack support will be disabled implicitly by its cmake script. 2. Linking with cblas library is required when sci-libs/lapack is supplemented as the implementation for virtual/lapacke. Correction: opencv probably doesn't actually use lapacke API per se, but it still detects the existence of lapacke.h or implicitly disables lapack support if it can't find one when sci-libs/lapack is supplemented as the lapack implementation. The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c92eaa6a7d93400503334e9c65df2b8e400ab90f commit c92eaa6a7d93400503334e9c65df2b8e400ab90f Author: Ross Charles Campbell <rossbridger.cc@gmail.com> AuthorDate: 2020-10-18 14:59:46 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2020-11-03 07:18:26 +0000 media-libs/opencv: various fixes - Explicitly depends on virtual/lapacke to prevent its cmake build scripts from implicitly disabling lapack support when it can't find lapacke.h. - Link with cblas when sci-libs/lapack is supplimented as the default lapack implementation. - examples USE requires contribdnn USE to be enabled to prevent build failure. Closes: https://bugs.gentoo.org/700176 Closes: https://bugs.gentoo.org/749681 Closes: https://bugs.gentoo.org/717812 Closes: https://bugs.gentoo.org/701790 Closes: https://bugs.gentoo.org/704518 Package-Manager: Portage-3.0.8, Repoman-3.0.1 Signed-off-by: Ross Charles Campbell <rossbridger.cc@gmail.com> Closes: https://github.com/gentoo/gentoo/pull/17964 Signed-off-by: Sam James <sam@gentoo.org> .../opencv-4.5.0-link-with-cblas-for-lapack.patch | 18 ++++++++++++++++++ media-libs/opencv/opencv-4.5.0.ebuild | 8 +++++++- 2 files changed, 25 insertions(+), 1 deletion(-) Additionally, it has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=41748611705de3c85338e2bfc6c61aabf4aed2fb commit 41748611705de3c85338e2bfc6c61aabf4aed2fb Author: Ross Charles Campbell <rossbridger.cc@gmail.com> AuthorDate: 2020-10-18 14:45:00 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2020-11-03 07:18:26 +0000 virtual/lapacke: add sci-libs/openblas as an alternative The current sci-libs/openblas versions come with an implementation for lapacke API, therefore adding it as a conditional dep for virtual/lapacke. Bug: https://bugs.gentoo.org/700176 Bug: https://bugs.gentoo.org/749921 Package-Manager: Portage-3.0.8, Repoman-3.0.1 Signed-off-by: Ross Charles Campbell <rossbridger.cc@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> virtual/lapacke/lapacke-3.8.ebuild | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) |