The build of dev-lang/python-3.5.1 and of dev-lang/python-3.4.3-r5 fails the bootstrap-prefix.sh script, with and without LATEST_TREE_YES=1. The error messages are shown below in section "Actual Results". And also in the attached file linking_with_-lcrypt_-lnsl_necessary.stage3.log.diff.part. Some of these messages are similar to the one in bug #565558 (is out of prefix), but the fix may be different. The host OS is an unpatched CentOS box running kernel 2.6.32-279.11.1.el6.x86_64. Diffing two buildlogs, one with and the other without these error messages, I found a culprit: The function build_extension(), defined in dev-lang/python-3.5.1/work/Python-3.5.1/Lib/distutils/command/build_ext.py, sometimes forgets to put -lcrypt and -lnsl before -lpython3.5. I patched build_extension() (with os.system("declare -x;sleep 99999"), see patch patch.python-build_ext.py) so that I have a chance to add -lcrypt and -lnsl just after the messages "building '_crypt' extension" and "building 'nis' extension". I attached the resulting changes in the build log : dev-lang/python-3.5.1 now builds correctly. Reproducible: Always Steps to Reproduce: 1. http://rsync.prefix.bitzolder.nl/scripts/bootstrap-prefix.sh 2. chmod 755 bootstrap-prefix.sh 3. LATEST_TREE_YES=1 ./bootstrap-prefix.sh Actual Results: (see diff with section "Expected results" below in attached file linking_with_-lcrypt_-lnsl_necessary.stage3.log.diff.part) *** WARNING: renaming "_crypt" since importing it failed: build/lib.linux-x86_64-3.5/_crypt.cpython-35-x86_64-linux-gnu.so: undefined symbol: crypt *** WARNING: renaming "nis" since importing it failed: build/lib.linux-x86_64-3.5/nis.cpython-35-x86_64-linux-gnu.so: undefined symbol: yp_master Python build finished successfully! [...] Following modules built successfully but were removed because they could not be imported: _crypt nis [...] renaming build/scripts-3.5/pyvenv to build/scripts-3.5/pyvenv-3.5 Makefile:610: recipe for target 'sharedmods' failed make: *** [sharedmods] Error 1 Expected Results: (see diff with above in attached file linking_with_-lcrypt_-lnsl_necessary.stage3.log.diff.part) Python build finished successfully! [...] running build_scripts creating build/scripts-3.5 copying and adjusting $EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/work/Python-3.5.1/Tools/scripts/pydoc3 -> build/scripts-3.5 [...] running build_scripts copying and adjusting $EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/work/Python-3.5.1/Tools/scripts/pydoc3 -> build/scripts-3.5 [...] renaming build/scripts-3.5/pyvenv to build/scripts-3.5/pyvenv-3.5 python$EXE ../../Tools/scripts/h2py.py -i '(u_long)' /usr/include/sys/types.h /usr/include/netinet/in.h /usr/include/dlfcn.h /usr/include/linux/cdrom.h Skipping: _POSIX_C_SOURCE = 200809L Skipping: _POSIX_C_SOURCE = 199506L Skipping: _POSIX_C_SOURCE = 200112L [...] LD_LIBRARY_PATH=$EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/work/x86_64-pc-linux-gnu ./python -E -m ensurepip \ $ensurepip --root=$EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/image// ; \ fi >>> Completed installing python-3.5.1 into $EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/image/ Of course, the patch patch.python-build_ext.py is a hack. The real error is in an undentified file of python-3.5.1/work/Python-3.5.1/, good luck. The workaround would be to mask python-3 during bootstrap, but I do not know how to achieve that.
Created attachment 423598 [details, diff] diff of build log with and without the bug. The relevant error messages are prefixed by '-'. I compared the build log (with -j1 with the error) to the build log with my workaround. This workaround is based on: - next attached file, - manual execution of the two commands prefixed by '+x86_64-pc-linux-gnu-gcc' by hand, in another shell with a copy of the original environment.
Created attachment 423600 [details] Patch to dev-lang/python-3.5.1/work/P*/Lib/distutils/command/build_ext.py allowing manual execution of the correct x86_64-pc-linux-gnu-gcc command in the correct environment Steps to complete the bootstrap-prefix.sh: - during configure of python-3.*, suspend the emerge command with ^Z then apply this attachment (it contains sleep 99999) - then type fg to let it go until the command sleep 99999 - Use these 99999 seconds to: * take a new bash shell, go in the directory $EPREFIX/tmp/var/tmp/portage/dev-lang/python-3.5.1/work/x86_64-pc-linux-gnu * get the right environment by pasting all the declare commands shown below "time to hack" * reexecute the last shown x86_64-pc-linux-gnu-gcc command after having inserted -lcrypt respectively -lnsl just before " -lpython3.5" * kill the sleep 99999 - Then, python-3.* emerge successfully. When next python-3.* is configured, redo all that once.
Created attachment 423602 [details] Full version of the diff in the attachment https://bugs.gentoo.org/show_bug.cgi?id=572600#c1 Full version of the diff in the attachment https://bugs.gentoo.org/show_bug.cgi?id=572600#c1 As it contained too much new lines, I have suppressed some of them to create the attachment https://bugs.gentoo.org/show_bug.cgi?id=572600#c1 This third attachment is not the *real* diff between changelog, because I used $EPREFIX to hide my real name, and I suppressed some diffs.
The manual workaround also allowed me to dev-lang/python-3.4.3-r5.
Guilherme Amadio provided a better workaround in https://bugs.gentoo.org/show_bug.cgi?id=572726#c12 gentoo-dev 2016-02-02 11:03:20 UTC : "Note: on a boostrap on CentOS 7, I also needed to add append-ldflags -lcrypt -lnss_nis" My question are: (1) in what file do you add these flags ? (2) why -lnss_nis is better than -lnsl ?
(3) (continued) I didn't have this bug on 2.7.11
(In reply to Daa Jaa from comment #5) > Guilherme Amadio provided a better workaround in > https://bugs.gentoo.org/show_bug.cgi?id=572726#c12 gentoo-dev 2016-02-02 > 11:03:20 UTC : > > "Note: on a boostrap on CentOS 7, I also needed to add > > append-ldflags -lcrypt -lnss_nis" > > My question are: > > (1) in what file do you add these flags ? That needs to go into the ebuild for python, but since it's a system-dependent thing, I'm a bit reluctant to do it. It might be better to add it in the bootstrap-prefix.sh script when CentOS is detected by setting LDFLAGS in the environment. > (2) why -lnss_nis is better than -lnsl ? I just looked where the missing symbol was defined and added that.
Thanks for you answer. On an older centos box, python-3.[45] uses automatically -lnsl, this is why I prefered -lnsl. Is this a bug of autotools which does not recognize correctly the host Libc ?
Hey guys, I ran into this last year. IRC chatter as I discovered this on #gentoo-prefix: javaJake Hmm. I am getting an error. javaJake build/lib.linux-x86_64-3.5/_crypt.cpython-35-x86_64-linux-gnu.so: undefined symbol: crypt [...later...] 2015-12-15 12:29:52 javaJake For some reason _crypt...so is not linked against crypt 2015-12-15 13:17:54 javaJake Ah. 64-bit. 2015-12-15 13:18:16 javaJake setup.py is only scanning all forms of lib and usr/lib, but not lib64 2015-12-15 13:26:17 javaJake Oh. Prefix assumes I have a lib symlink? 2015-12-15 13:26:24 javaJake Am I supposed to have one? 2015-12-15 13:26:53 javaJake I notice now there is a prefix patch which removes the call to add_gcc_paths which would've invoked the 64-bit gcc and parsed the correct paths from it. 2015-12-15 13:28:24 javaJake Or, if that's not the case, then I am missing /usr/lib/libcrypt.so which Gentoo has but CentOS does not 2015-12-15 13:28:54 javaJake /usr/lib/libcrypt.so is 32-bit anyway 2015-12-15 13:28:56 javaJake If it would exist. 2015-12-15 14:01:39 javaJake ...and that's because multilib.eclass' get_libdir returns 'lib' when the host system uses 'lib64' 2015-12-15 14:05:01 javaJake And built. Finally! For my system, the issue is that my host system has completely distinct /usr/lib and /usr/lib64 directories, and libcrypt.so only exists in /usr/lib64.
To be clear, I don't have an answer. :) I just know that Gentoo is trying to be smart about which /usr/lib{,64} directory to use and it ends up picking the wrong one.
../patches/03_all_libdir.patch replaces: - '/lib64', '/usr/lib64', - '/lib', '/usr/lib', + '/@@GENTOO_LIBDIR@@', '/usr/@@GENTOO_LIBDIR@@', ...which implies that Python is going to use the system libdir? Shouldn't it link against Prefix's libdir?
1) in Prefix we always use "lib" (we don't want to do multilib) 2) 03_all_libdir.patch is later amended by a prefix patch that adds EPREFIX in front of it
perhaps we have to add more crap to the build to figure out where the native host libs are so we can tell Python about that, since it is lovely enough not to rely on standard measures
(In reply to Fabian Groffen from comment #12) > 1) in Prefix we always use "lib" (we don't want to do multilib) > 2) 03_all_libdir.patch is later amended by a prefix patch that adds EPREFIX > in front of it I didn't see EPREFIX after doing an 'ebuild ... clean prepare'. Perhaps that step is failing.
I think I might have the same issue on a Debian Jessie host system, where the libraries aren't even in either of /usr/lib{,64} but in /usr/lib/x86_64-linux-gnu/ instead...
I think this issue has been fixed, but at the very least we don't use these python versions any more
