Bug 705970

Summary:	dev-lang/python: cross compilation using different libc is broken
Product:	Gentoo Linux	Reporter:	Andrew Aladjev <aladjev.andrew>
Component:	Current packages	Assignee:	Cross compilation support <cross>
Status:	RESOLVED FIXED
Severity:	minor	CC:	aladjev.andrew, jstein, python
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
URL:	https://bugs.python.org/issue39399
See Also:	https://bugs.gentoo.org/show_bug.cgi?id=864911
Whiteboard:
Package list:		Runtime testing required:	---
Attachments:	epython for build patch x86_64-pc-linux-musl-python3.6

Description Andrew Aladjev 2020-01-20 19:24:48 UTC

Hello. I am implementing python cross compilation using "x86_64-pc-linux-musl" toolchain: "x86_64-pc-linux-musl-emerge -v1 python:3.6". Please see the following build log https://gist.github.com/andrew-aladev/e10fa5a8151ffb3c5782edd64ae08b28.

We can see the following part:

Traceback (most recent call last):
File "/usr/x86_64-pc-linux-musl/tmp/portage/dev-lang/python-3.6.9/image//usr/lib/python3.6/compileall.py", line 17, in <module>
import struct
File "/usr/x86_64-pc-linux-musl/tmp/portage/dev-lang/python-3.6.9/image/usr/lib/python3.6/struct.py", line 13, in <module>
from _struct import *
ImportError: libc.so: cannot open shared object file: No such file or directory

It means that cross compilation of python is not reliable today by design. Python is trying to use PYTHON_FOR_BUILD for loading cross compiled modules. It is not possible in general case. I will create a bug in upstream bugtracker.

I am not sure that this bug will be fixed in the nearest 5 years, so we need a good reliable workaround for today. I want to introduce double recompilation method:

1. x86_64-pc-linux-musl-emerge -v1 python:3.6
2. EPYTHON_FOR_BUILD="x86_64-pc-linux-musl-python3.6" x86_64-pc-linux-musl-emerge -v1 python:3.6

First step is to cross compile python with default broken "PYTHON_FOR_BUILD". Second step is to provide cross compiled python as "EPYTHON_FOR_BUILD". This method works perfect with both native execution (i686, amd64) and qemu user emulation (arm, mips, etc).

Unfortunately today python doesn't support "EPYTHON_FOR_BUILD". I will attach patch and "x86_64-pc-linux-musl-python3.6" file here.

PS the success of double recompilation method depends on "elf_mismatch_is_not_fatal.glibc-2.29.patch". It is provided here https://sourceware.org/bugzilla/show_bug.cgi?id=25341.

Please see here too https://github.com/andrew-aladev/test-images/blob/master/native/x86_64-pc-linux-musl/0-crossdev/buildah.sh. This image is not complete today, because it depends on other musl issues. But you can use it as complete example.

Reproducible: Always

Comment 1 Andrew Aladjev 2020-01-20 19:27:33 UTC

Created attachment 603822 [details, diff]
epython for build patch

Comment 2 Andrew Aladjev 2020-01-20 19:29:11 UTC

Created attachment 603824 [details]
x86_64-pc-linux-musl-python3.6

Comment 3 Sam James archtester

2022-12-06 17:27:40 UTC

This was fixed by bug 864911.