I noticed an issue when building pkgcheck on macOS 11 (Prefix) [0]: >OSError: dlopen(/Users/sam/Gentoo/usr/lib/python3.8/site-packages/pkgcheck/_bash-lang.so, 6): Symbol not found: >__ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE4copyEPcmm > Referenced from: /Users/sam/Gentoo/usr/lib/python3.8/site-packages/pkgcheck/_bash-lang.so > Expected in: flat namespace > in /Users/sam/Gentoo/usr/lib/python3.8/site-packages/pkgcheck/_bash-lang.so The relevant code from pkgcheck tries to find which c++ stdlib to use: > compiler = new_compiler() > if cpp: > if find_library("stdc++"): > compiler.add_library("stdc++") > elif find_library("c++"): > compiler.add_library("c++") > Now: 1) Python 3.8 gave nothing(!) 2) Python 3.9 gave c++ (libcxx - LLVM/Clang). It turned out that: 1) This is because Python 3.8 lacks support for macOS 11's dyld cache and the fixes haven't been backported from Python 3.9. 2) While Python 3.9 contains the above fixes, it has the wrong library and framework search dirs: >Lib/ctypes/macholib/dyld.py: > > DEFAULT_FRAMEWORK_FALLBACK = [ > os.path.expanduser("~/Library/Frameworks"), > "/Library/Frameworks", > "/Network/Library/Frameworks", > [...] > > DEFAULT_LIBRARY_FALLBACK = [ > os.path.expanduser("~/lib"), > "/usr/local/lib", > "/lib", So, while Python 3.9 has been fixed upstream, it's not Prefix-adapted. [0] https://github.com/pkgcore/pkgcheck/issues/299#issuecomment-791887928
Created attachment 691203 [details, diff] 0001-Lib-ctypes-macholib-dyld.py-tweak-search-paths-for-P.patch Using >sed -i -e "s:\${EPREFIX}:${EPREFIX}:" Lib/ctypes/macholib/dyld.py || die in the ebuild seems fine. We should probably guard it, just in case though.
Can you use @GENTOO_PORTAGE_EPREFIX@ instead of ${EPREFIX} in the code, and then call eprefixify on the file?
(In reply to Fabian Groffen from comment #2) > Can you use @GENTOO_PORTAGE_EPREFIX@ instead of ${EPREFIX} in the code, and > then call eprefixify on the file? Of course - I'll do that later. Didn't think about that!
Created attachment 691260 [details, diff] 0001-Lib-ctypes-macholib-dyld.py-tweak-search-paths-for-P.patch
Comment on attachment 691203 [details, diff] 0001-Lib-ctypes-macholib-dyld.py-tweak-search-paths-for-P.patch >From c49d80f2001524c38313582ec1974a9e4f703489 Mon Sep 17 00:00:00 2001 >From: Sam James <sam@gentoo.org> >Date: Sat, 13 Mar 2021 17:27:48 +0000 >Subject: [PATCH] Lib/ctypes/macholib/dyld.py: tweak search paths for Prefix > >We need to tweak: >* DEFAULT_FRAMEWORK_FALLBACK >* DEFAULT_LIBRARY_FALLBACK >for macOS within Gentoo Prefix. > >These paths ignore the prefix and search the system for >frameworks and compiled libraries which may not be compatible >with versions inside the Prefix. > >I've opted to keep the system versions but give priority >to Prefix locations. It's expected that we sed ${EPREFIX} >in the ebuild with the true value. > >Closes: https://bugs.gentoo.org/775839 >Signed-off-by: Sam James <sam@gentoo.org> >--- > Lib/ctypes/macholib/dyld.py | 4 ++++ > 1 file changed, 4 insertions(+) > >diff --git a/Lib/ctypes/macholib/dyld.py b/Lib/ctypes/macholib/dyld.py >index 1c3f8fd..5599e12 100644 >--- a/Lib/ctypes/macholib/dyld.py >+++ b/Lib/ctypes/macholib/dyld.py >@@ -20,6 +20,7 @@ __all__ = [ > # These are the defaults as per man dyld(1) > # > DEFAULT_FRAMEWORK_FALLBACK = [ >+ "${EPREFIX}/MacOSX.sdk/System/Library/Frameworks", > os.path.expanduser("~/Library/Frameworks"), > "/Library/Frameworks", > "/Network/Library/Frameworks", >@@ -27,6 +28,9 @@ DEFAULT_FRAMEWORK_FALLBACK = [ > ] > > DEFAULT_LIBRARY_FALLBACK = [ >+ "${EPREFIX}/lib", >+ "${EPREFIX}/usr/lib", >+ "${EPREFIX}/usr/local/lib", > os.path.expanduser("~/lib"), > "/usr/local/lib", > "/lib", >-- >2.30.2
Created attachment 691263 [details] python-3.9.1_p2-r1.ebuild.patch For completeness.
Actually, I need a bit of help. We need some way of adding detection for libstdc++ which Python doesn't cater for. It's in ${EPREFIX}/usr/x86_64-apple-darwin20/lib/gcc/libstdc++.6.dylib and I don't really want to hardcode anything GCC, just reference it if we can. I suppose there won't be anything magic here, just search in ${EPREFIX}/usr/${CHOST}/lib/gcc/* if it exists...?
(In reply to Sam James from comment #7) > Actually, I need a bit of help. We need some way of adding detection for > libstdc++ which Python doesn't cater for. > > It's in ${EPREFIX}/usr/x86_64-apple-darwin20/lib/gcc/libstdc++.6.dylib and I > don't really want to hardcode anything GCC, just reference it if we can. > > I suppose there won't be anything magic here, just search in > ${EPREFIX}/usr/${CHOST}/lib/gcc/* if it exists...? ${EPREFIX}/usr/lib/gcc/${CHOST}/10.2.0/ of course for the general lib* (forgot /lib/)
Ok, so a dlopen of some lib is done, that depends on c++. The lib in question, is that a module/bundle or a shared library? In case of the latter, does that shared library depend on the c++ runtime?
if I understand this well, the reproducer for this is: from ctypes.util import find_library find_library("stdc++")
https://docs.python.org/3/library/ctypes.html#finding-shared-libraries documents that on Linux, a search is done using external tools (like ld), while on macOS it sticks to the paths that you found, and adapted. This explains why find_library finds our prefix libs just fine without help on e.g. Solaris, and we have this particular problem on macOS. To make it find our GCC libs, we probably have to see if we can actually adapt the implementation to also do a runtime search.
what I think is missing in this patch is the path to the GCC libs, e.g. $EPREFIX /usr/lib/gcc/arm64-apple-darwin21/12.1.0, for that's where one can find the correct libstdc++.dylib. I think we need to add Gentoo-specific magic here to load at runtime the current list of paths, we could simply read $EPREFIX/etc/ld.so.conf, which contains the paths as they should be. DEFAULT_LIBRARY_FALLBACK should be populated with ld.so.conf material in front DEFAULT_FRAMEWORK_FALLBACK should use $EPREFIX/Frameworks followed by the sdk one.
