Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 413863 - dev-lang/python-3.2.3 - improve cross compilation
Summary: dev-lang/python-3.2.3 - improve cross compilation
Status: RESOLVED DUPLICATE of bug 268887
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Development (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-04-28 12:29 UTC by Marko Durkovic
Modified: 2013-03-19 05:22 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
python-3.2.3.ebuild (python-3.2.3.ebuild,10.45 KB, text/plain)
2012-04-28 12:29 UTC, Marko Durkovic
Details
fix-cross-compile.patch (fix-cross-compile.patch,1.95 KB, patch)
2012-04-28 12:30 UTC, Marko Durkovic
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Marko Durkovic 2012-04-28 12:29:59 UTC
Created attachment 310339 [details]
python-3.2.3.ebuild

The cross-compilation of python has some issues with the most current python versions. I have tested this with an armv6j-hardfloat-linux-gnueabi and a mips-unknown-linux-gnu target.

Steps to reproduce:
1. Start with a fresh x86_64 stage3.
2. Install crossdev and a toolchain with crossdev -t ${TARGET}.
3. Change make.profile symlink to the corresponding 10.0 profile.
4. Set MARCH_TUNE and USE flags in /usr/$TARGET/etc/portage/make.conf.
5. ${TARGET}-emerge python:3.2 fails

Please find attached a modified ebuild and patch for python-3.2.3. I've tried to keep the changes as unintrusive as possible. A few comments on what had to be changed:

1. When building current versions of the python3 series CFLAGS are read in the make phase. Hostpython cannot be build with CFLAGS that contain optimizations for the target platform. This is fixed by clearing CFLAGS when building hostpython.
2. In the original ebuild no modules that are written in C are compiled for hostpython. Hostpython then uses modules from /usr/lib/pythonX.X if available. This can cause errors during build as hostpython is mixing binary modules from the outer gentoo system with .py modules of the python source, which are to my understanding not necessarily compatible. This is can be fixed by making hostpyhon self-contained and building the modules that are required for the later stages of the ebuild.
Note: When building the modules for hostpython, make invokes "python setup.py build" which already needs some binary python modules and tries to find them in the system (_struct.so and others). After creating an empty module directory, hostpython does not try to find modules under /usr/lib anymore and setup.py automagically creates the required modules.
3. setup.py adds /usr/include to the CFLAGS for the compilation of the modules for the target python. GCC always searches the include directories under its --sysroot last and thus /usr/include is searched before the correct /usr/${TARGET}/usr/include. The patch for setup.py disables this behavior.
4. Two small fixes for calling compileall and regen with hostpython. Regen also has to use the headers under ${SYSROOT}. 

Python-2.7 cross compilation is also fixed by these modifications.

I hope this is helpful!
Comment 1 Marko Durkovic 2012-04-28 12:30:59 UTC
Created attachment 310341 [details, diff]
fix-cross-compile.patch
Comment 2 Jeroen Roovers (RETIRED) gentoo-dev 2012-05-01 22:20:40 UTC
Comment on attachment 310339 [details]
python-3.2.3.ebuild

--- python-3.2.3.ebuild 2012-05-01 18:34:18.000000000 +0200
+++ -   2012-05-02 00:20:27.410235222 +0200
@@ -1,6 +1,6 @@
 # Copyright 1999-2012 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
-# $Header: /var/cvsroot/gentoo-x86/dev-lang/python/python-3.2.3.ebuild,v 1.7 2012/04/28 05:02:04 floppym Exp $
+# $Header: $
 
 EAPI="3"
 WANT_AUTOMAKE="none"
@@ -42,6 +42,7 @@
                )"
 DEPEND="${RDEPEND}
                dev-util/pkgconfig
+               sys-apps/paxctl
                >=sys-devel/autoconf-2.65
                !sys-devel/gcc[libffi]"
 RDEPEND+=" !build? ( app-misc/mime-types )
@@ -72,6 +73,20 @@

        EPATCH_EXCLUDE="${excluded_patches}" EPATCH_SUFFIX="patch" \
                epatch "${WORKDIR}/${PV}-${PATCHSET_REVISION}"
+
+       if tc-is-cross-compiler; then
+               # Patch contains three fixes:
+               # 1. setup.py does not append "-I/usr/include" to the CFLAGS when
+               # compiling the python modules for the target system.  Otherwise the
+               # compilation can fail, e.g. with weird asm errors, due to wrong system
+               # headers being used.  It does not suffice to have the correct
+               # "-I${SYSROOT}/usr/include" in the CFLAGS variable in make.conf as the
+               # gcc include directory search order will always look in those
+               # directories last.
+               # 2. Use hostpython and header files from ${SYSROOT} for regen
+               # 3. Makefile uses hostpython for compileall
+               epatch "${FILESDIR}"/fix-cross-compile.patch
+       fi
        
        sed -i -e "s:@@GENTOO_LIBDIR@@:$(get_libdir):g" \
                Lib/distutils/command/install.py \
@@ -135,10 +150,38 @@
        if tc-is-cross-compiler; then
                OPT="-O1" CFLAGS="" LDFLAGS="" CC="" \
                ./configure --{build,host}=${CBUILD} || die "cross-configure failed"
+               # Ensure that CFLAGS are cleared during the build of hostpython, as
+               # the CFLAGS environment variable is also evaluated during make.
+               OPT="-O1" CFLAGS="" LDFLAGS="" CC="" \
                emake python Parser/pgen || die "cross-make failed"
                mv python hostpython
                mv Parser/pgen Parser/hostpgen
+               # If this directory does not exist, hostpython fallbacks to
+               # /usr/lib/pythonX.X/lib-dynload when searching for compiled modules.
+               # See './hostpython -c "import sys; print(sys.path)"'.
+               mkdir -p "lib/python${SLOT}/lib-dynload"
+               # To make hostpython self-contained build the binary python modules.
+               # Disable all modules not required for building the target python
+               # distribution to speed up compiling.
+               PYTHON_DISABLE_MODULES="array audioop bz2 cmath _codecs_cn \
+               _codecs_hk _codecs_iso2022 _codecs_jp _codecs_kr _codecs_tw crypt _csv \
+               _ctypes _ctypes_test _curses _curses_panel _datetime _dbm _elementtree \
+               fcntl _gdbm grp _hashlib _heapq _json _lsprof math _md5 mmap \
+               _multibytecodec _multiprocessing nis ossaudiodev parser _pickle \
+               _posixsubprocess pyexpat _random readline resource select _sha1 \
+               _sha256 _sha512 _socket spwd _sqlite3 _ssl syslog termios _testcapi \
+               _tkinter zlib" \
+               OPT="-O1" CFLAGS="" LDFLAGS="" CC="" \
+               emake sharedmods || die "cross-make failed"
+               # "make distclean" will remove all .so files. Therefore move the modules
+               # to updir and move them after "make distclean" into Lib. Moving into
+               # lib/python${SLOT}/lib-dynload is not possible as the search order for
+               # binary modules is Lib -> build/lib.linux-${ARCH}-${SLOT}/ ->
+               # lib/python${SLOT}/lib-dynload and the second one is filled is filled
+               # with the built modules for the target host.
+               mv build ..
                make distclean
+               mv ../build/*/*.so Lib/
                sed -i \
                        -e "/^HOSTPYTHON/s:=.*:=./hostpython:" \
                        -e "/^HOSTPGEN/s:=.*:=./Parser/hostpgen:" \
@@ -177,7 +220,6 @@
 src_compile() {
        emake CPPFLAGS="" CFLAGS="" LDFLAGS="" || die "emake failed"

-       # Work around bug 329499. See also bug 413751.
        pax-mark m python
 }
Comment 3 Dirkjan Ochtman (RETIRED) gentoo-dev 2012-05-02 07:02:49 UTC
Thanks for looking into this! To make sure this keeps working well with new upstream releases, it would be great if you file an upstream bug for the changes to Python; we prefer to not take patches that aren't upstream, because we end up maintaining them potentially for a long time.

Are there any ebuild changes that would help even without patching python itself?
Comment 4 Marko Durkovic 2012-05-02 13:39:05 UTC
Upstream is working on patches for python-trunk that should enable some cross-compilation capabilities. One major showstopper for properly implementing cross building is the distutils package as it is considered feature frozen (changes to distutils often broke 3rd party packages in the past). Python 3.3 will ship with distutils2, aka 'packages', included in its standard library. As far as I can see upstream is focusing its cross-compilation efforts on python-trunk. As for python-2.7 (also considered feature frozen) and python-3.2 I'm afraid we are stuck with what is provided by upstream at the moment.

The patch I proposed changes three files of the python build system, namely Makefile.pre.in, setup.py and Lib/plat-linux2/regen.

Makefile.pre.in and setup.py are patched by 41_all_crosscompile.patch during a build of dev-lang/python anyways. My modifications to setup.py could be replaced with an ugly hack that sets some CFLAGS in the ebuild, but this could bring some unforeseen problems with user set CFLAGS.

The patch to the regen script is fairly gentoo specific as is depends on the ${SYSROOT} environment variable which is set on cross-emerges. The modifications are possibly also interesting to Gentoo/Alt, as the original regen is hardcoded to use the header files that are installed in /usr/include as opposed to those corresponding to the toolchain that is is used for building python.
Comment 5 Dennis Schridde 2012-09-14 16:30:58 UTC
Is this a duplicate of bug #268887?
Comment 6 SpanKY gentoo-dev 2013-03-19 05:22:40 UTC

*** This bug has been marked as a duplicate of bug 268887 ***