Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 268887 - dev-lang/python cross compilation aware
Summary: dev-lang/python cross compilation aware
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Development (show other bugs)
Hardware: All Linux
: High normal with 2 votes (vote)
Assignee: Embedded Gentoo Team
: 413863 511476 534706 (view as bug list)
Depends on:
Reported: 2009-05-07 00:07 UTC by Maksim 'max_posedon' Melnikau
Modified: 2018-12-30 09:46 UTC (History)
13 users (show)

See Also:
Package list:
Runtime testing required: ---

python 2.5.2-r8 ebuild patch (python-2.5.2-r8.ebuild.patch,1.46 KB, patch)
2009-05-07 00:10 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.4-r2 ebuild patch (python-2.5.4-r2.ebuild.patch,1.08 KB, patch)
2009-05-07 00:12 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.6.2 ebuild patch (python-2.6.2.ebuild.patch,1.42 KB, patch)
2009-05-07 00:12 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 cross enable ctypes (python-2.5.2-cross-enable-ctypes-module.patch,3.74 KB, patch)
2009-05-07 00:14 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 cross setup sysroot (python-2.5.2-cross-setup-sysroot.patch,11.09 KB, patch)
2009-05-07 00:15 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 cross patch (,487 bytes, patch)
2009-05-07 00:17 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 cross distutils (python-2.5.2-cross-distutils-build_ext.patch,780 bytes, patch)
2009-05-07 00:18 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 cross distuils sysconfig (python-2.5.2-cross-distutils-sysconfig.patch,2.16 KB, patch)
2009-05-07 00:19 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.5.2 crpss distutils install patch (python-2.5.2-cross-distutils-install.patch,1.56 KB, patch)
2009-05-07 00:20 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.6 cross setup sysroot patch (python-2.6-cross-setup-sysroot.patch,10.16 KB, patch)
2009-05-07 00:22 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.6 enable ctypes cross build (python-2.6-enable-ctypes-cross-build.patch,1.38 KB, patch)
2009-05-07 00:23 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.6 cross distutils sysconfig (python-2.6-cross-distutils-sysconfig.patch,1.76 KB, patch)
2009-05-07 00:24 UTC, Maksim 'max_posedon' Melnikau
Details | Diff
python 2.6.2 build log cross compilation fails (,416.54 KB, text/plain)
2009-05-07 19:12 UTC, Maksim 'max_posedon' Melnikau
null (null,4 bytes, text/plain)
2009-05-09 09:23 UTC, Maksim 'max_posedon' Melnikau
remove obsolete patches, because of separate tickets for them (null,6 bytes, text/plain)
2009-05-28 18:16 UTC, Maksim 'max_posedon' Melnikau
python 2.7.3-r2 build log (build.log,68.01 KB, text/plain)
2012-05-13 20:01 UTC, Vladimir Smirnov (RETIRED)
python 3.2.2-r1 build log (build.log,136.56 KB, text/plain)
2012-05-13 20:03 UTC, Vladimir Smirnov (RETIRED)

Note You need to log in before you can comment on or make changes to this bug.
Description Maksim 'max_posedon' Melnikau 2009-05-07 00:07:32 UTC
python cross compilation aware ebuilds

Reproducible: Always

Steps to Reproduce:
1. emerge-cross python
2. emerge-cross dev-lang/<any-package>

Actual Results:  
python doesn't cross compile (2.6)
or python can't cross compile any c extension (python distutils bugs)

Expected Results:  
python should cross compile, and successfully cross compile any others modules
Comment 1 Maksim 'max_posedon' Melnikau 2009-05-07 00:10:41 UTC
Created attachment 190575 [details, diff]
python 2.5.2-r8 ebuild patch
Comment 2 Maksim 'max_posedon' Melnikau 2009-05-07 00:12:21 UTC
Created attachment 190577 [details, diff]
python 2.5.4-r2 ebuild patch
Comment 3 Maksim 'max_posedon' Melnikau 2009-05-07 00:12:59 UTC
Created attachment 190579 [details, diff]
python 2.6.2 ebuild patch
Comment 4 Maksim 'max_posedon' Melnikau 2009-05-07 00:14:39 UTC
Created attachment 190580 [details, diff]
python 2.5.2 cross enable ctypes

Compilation of the ctypes extension fails for archs that do not have FFI_CLOSURES.
This fixes the compilation and makes ctypes also available to those archs.
Comment 5 Maksim 'max_posedon' Melnikau 2009-05-07 00:15:47 UTC
Created attachment 190582 [details, diff]
python 2.5.2 cross setup sysroot

Change to respect the SYSROOT environment variable
Comment 6 Maksim 'max_posedon' Melnikau 2009-05-07 00:17:07 UTC
Created attachment 190583 [details, diff]
python 2.5.2 cross patch

Add a libdirname to enable python cross-compiling and add a user specifyable libdir
Comment 7 Maksim 'max_posedon' Melnikau 2009-05-07 00:18:04 UTC
Created attachment 190585 [details, diff]
python 2.5.2 cross distutils

Make distutils cross-compile aware (1/3)
Comment 8 Maksim 'max_posedon' Melnikau 2009-05-07 00:19:08 UTC
Created attachment 190587 [details, diff]
python 2.5.2 cross distuils sysconfig

Make distutils cross-compile aware (2/3)
Comment 9 Maksim 'max_posedon' Melnikau 2009-05-07 00:20:02 UTC
Created attachment 190588 [details, diff]
python 2.5.2 crpss distutils install patch

Make distutils cross-compile aware (3/3)
Comment 10 Maksim 'max_posedon' Melnikau 2009-05-07 00:22:23 UTC
Created attachment 190590 [details, diff]
python 2.6 cross setup sysroot patch

like "python 2.5.2 cross setup sysroot", but reworked for 2.6
Comment 11 Maksim 'max_posedon' Melnikau 2009-05-07 00:23:31 UTC
Created attachment 190591 [details, diff]
python 2.6 enable ctypes cross build

like "python 2.5.2 cross enable ctypes", but reworked for 2.6
Comment 12 Maksim 'max_posedon' Melnikau 2009-05-07 00:24:29 UTC
Created attachment 190592 [details, diff]
python 2.6 cross distutils sysconfig

like "python 2.5.2 cross distuils sysconfig", but reworked for 2.6
Comment 13 Maksim 'max_posedon' Melnikau 2009-05-07 00:28:41 UTC
Mainly all this patches created by sleipnir(OdinHorse), I just reworked them for python 2.6.

All this 3 python versions was tested by Gentoo Openmoko, and work well, plz say us, if patches should be reworked before go upstream. 

If you have any questions, plz found me (max_posedon) or sleipnir(OdinHorse) in (I also will be on #gentoo-python for any discussions)
Comment 14 SpanKY gentoo-dev 2009-05-07 18:39:17 UTC
python-2.6 cross-compiled fine for me (ignoring the normal module issues that all versions of python have)
Comment 15 Maksim 'max_posedon' Melnikau 2009-05-07 19:12:37 UTC
Created attachment 190639 [details]
python 2.6.2 build log cross compilation fails

(In reply to comment #14)
> python-2.6 cross-compiled fine for me (ignoring the normal module issues that
> all versions of python have)
fails for me during install, with (from build.log)
Compiling /mnt/floppy/data/gentoo/tmp/portage/dev-lang/python-2.6.2/image//usr/lib/python2.6/json/tests/ ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/usr/lib/python2.6/json/tests/', 8, None, "u = u'\\N{GREEK SMALL LETTER ALPHA}\\N{GREEK CAPITAL LETTER OMEGA}'\n"))

Which is fixed by,
# Ugly fix, hostpython require few modules to pass test
cp -v /usr/lib/python2.6/lib-dynload/ Modules/.
cp -v /usr/lib/python2.6/lib-dynload/ Modules/.
cp -v /usr/lib/python2.6/lib-dynload/ Modules/.
but of course I'll be lucky know another way to fix it.

One more point, now I understand that I shouldn't create such big ticket, it should be splitted in many, because of many different topics in python should be fixed. 

And I don't agree that module issue is "normal"...
Comment 16 Jesus Rivero (RETIRED) gentoo-dev 2009-05-09 03:28:45 UTC

   Maksim, are you having this issues with py2.5 as well? or just 2.6?. Can you post your emerge --info and tell me what locale are you using?

   Can somebody else from embedded confirm this bug? I have heard solar sometime ago telling us python2.6 was cross compiling without any issue. 

    Thanks and best regards, 
Comment 17 Sven 'sleipnir' Rebhan 2009-05-09 05:53:50 UTC
Jesus, the problem is _not_ to cross-compile python itself (in that way the topic is misleading). The problem addressed here is to cross-compile the modules and extensions. I (as one of the patch authors) will spilt this bug up into nice topic chunks and open separate bugs. Should I link them here, to have a single place tracking this?

And btw. this problem exists for 2.5 and 2.6 as well. We tested this now for quite some time in the openmoko overlay and it works well. There is one open issue I discussed with solar, but I will mention it on the right ticket.
Comment 18 Sven 'sleipnir' Rebhan 2009-05-09 08:35:14 UTC
The ctypes enable patches now have their own bug #269098 as it is a completely different story. Please mark them obsolete here.
Comment 19 Maksim 'max_posedon' Melnikau 2009-05-09 09:23:40 UTC
Created attachment 190742 [details]
Comment 20 Sven 'sleipnir' Rebhan 2009-05-09 11:33:00 UTC
The patches to make the python modules cross-compile now have their own bug #269111. Please mark them obsolete here (python-*-cross-setup-sysroot).
Comment 21 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-05-28 15:27:17 UTC
Please report it to Python upstream [1] and we will apply the version of the
patch accepted by upstream.

Comment 22 Maksim 'max_posedon' Melnikau 2009-05-28 18:16:07 UTC
Created attachment 192741 [details]
remove obsolete patches, because of separate tickets for them

bug #269111
Comment 23 Dirkjan Ochtman gentoo-dev 2012-02-16 13:20:08 UTC
This stuff should go upstream first, closing to that effect.
Comment 24 solar (RETIRED) gentoo-dev 2012-02-17 03:07:39 UTC
djc, Please do not close bugs which you are not a member of that team that are marked with an importance of High. A lot of valuable time has been spent  on these patches and have made attempts to work with upstream. This place holder should remain open till the python and it's submodules correctly work @ gentoo.
Comment 25 Dirkjan Ochtman gentoo-dev 2012-02-17 07:50:41 UTC
Ah, sorry.
Comment 26 Vladimir Smirnov (RETIRED) gentoo-dev 2012-05-13 20:01:05 UTC
Created attachment 311657 [details]
python 2.7.3-r2 build log

Just not to create duplicate bug:

I'm trying to cross-compile python 2.7 and 3.2 for armv6z target and it fails.

emerge-armv6z-hardfloat-linux-gnueabi --info:

Portage 2.2.0_alpha102_p5 (default/linux/arm/10.0/armv6j, gcc-4.5.3, unavailable, 3.3.2-gentoo x86_64)
System uname: Linux-3.3.2-gentoo-x86_64-Intel-R-_Pentium-R-_CPU_G630T_@_2.30GHz-with-gentoo-2.1
Timestamp of tree: Sun, 13 May 2012 17:45:01 +0000
ccache version 3.1.7 [disabled]
app-shells/bash:          4.2_p24
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.6.7-r2, 2.7.2-r3, 3.1.4-r3, 3.2.2
dev-util/ccache:          3.1.7
dev-util/cmake:           2.8.7-r5
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.9.6-r3, 1.11.3
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.4.4-r2, 4.5.3-r2
sys-devel/gcc-config:     1.5.1-r1
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.2-r1 (virtual/os-headers)
sys-libs/glibc:           2.14.1-r2
Repositories: gentoo bircoph alexxy betagarden piczu xhub rion slyfox-gentoo nrndda Local
Installed sets: 
CFLAGS="-Os -pipe  -fomit-frame-pointer -I/usr/armv6z-hardfloat-linux-gnueabi/usr/include/ -I/usr/armv6z-hardfloat-linux-gnueabi/include/"
CONFIG_PROTECT="/etc /var/bind /var/lib/redmine/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /var/lib/redmine/config/locales /var/lib/redmine/config/settings.yml"
CXXFLAGS="-Os -pipe  -fomit-frame-pointer -I/usr/armv6z-hardfloat-linux-gnueabi/usr/include/ -I/usr/armv6z-hardfloat-linux-gnueabi/include/"
FEATURES="assume-digests binpkg-logs buildpkg distlocks ebuild-locks fixlafiles news nodoc noinfo noman parallel-fetch parse-eapi-ebuild-head preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
LDFLAGS="-L/usr/armv6z-hardfloat-linux-gnueabi/lib -L/usr/armv6z-hardfloat-linux-gnueabi/usr/lib"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTDIR_OVERLAY="/var/lib/layman/bircoph /var/lib/layman/alexxy /var/lib/layman/betagarden /var/lib/layman/piczu /var/lib/layman/xhub /var/lib/layman/rion /var/lib/layman/slyfox /usr/local/nrndda-overlay /usr/local/portage"
USE="arm bzip2 cli cracklib crypt cups cxx fortran gdbm gpm iconv ipv6 modules mudflap ncurses nls nptl openmp pam pcre pppd readline session ssl tcpd unicode xorg zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev omapfb dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Comment 27 Vladimir Smirnov (RETIRED) gentoo-dev 2012-05-13 20:03:26 UTC
Created attachment 311659 [details]
python 3.2.2-r1 build log

Build log for python-3.2.3-r1.
Comment 28 Vladimir Smirnov (RETIRED) gentoo-dev 2012-05-13 20:13:56 UTC
I've managed to get past first errors with python-2.7.3 with ugly hack:
   sed -i 's#\(\#endif\)$#\1\n\#define PY_FORMAT_LONG_LONG#' "${S}"/pyconfig.h

Then I'm got next couple of errors: accidently -I/usr/include is passed for armv6z-hardfloat-linux-gnueabi-gcc. I've worked it around with another ugly hack (for fgrep <...>; do sed <...>), then I've got another error - hostpython segfaults, but runs flawlessly under gdb, coredump is unreadable.
Comment 29 James Le Cuirot gentoo-dev 2012-05-13 20:34:34 UTC
(In reply to comment #28)
> I've managed to get past first errors with python-2.7.3 with ugly hack:
>    sed -i 's#\(\#endif\)$#\1\n\#define PY_FORMAT_LONG_LONG#'
> "${S}"/pyconfig.h

This is an easier way around it...

ac_cv_have_long_long_format=yes emerge python
Comment 30 Sergei Trofimovich gentoo-dev 2012-05-27 18:31:30 UTC
So, nobody bothered to report latest SIGSEGV upstream, right?
Comment 31 SpanKY gentoo-dev 2012-06-01 05:41:21 UTC
(In reply to comment #30)

this is about making python cross-compile, not it crashing
Comment 32 Dennis Schridde 2012-09-14 16:30:27 UTC
Is there a relation to bug #413863?
Comment 33 SpanKY gentoo-dev 2013-03-19 05:22:40 UTC
*** Bug 413863 has been marked as a duplicate of this bug. ***
Comment 34 Jeroen Roovers (RETIRED) gentoo-dev 2014-05-26 00:00:39 UTC
*** Bug 511476 has been marked as a duplicate of this bug. ***
Comment 35 James Le Cuirot gentoo-dev 2014-05-26 08:22:12 UTC
It's worth pointing out that this is almost resolved now. The main thing missing is one or two entries like ac_cv_file__dev_ptmx being added to crossdev. Recent version of both Python 2 and 3 cross-compile out of the box.

The other remaining issue is a trickier one. A copy of Python is needed on /. If pretty much any version worked then we could just imply the dependency through @system. Unfortunately it's a little fussier than that. Python 3.4 needs at least 3.3, for example. The check is made very early in the configure script. It doesn't matter which version you have eselected. This is tricky because it's a job for HDEPEND, which is going nowhere fast in bug #317337.
Comment 36 SpanKY gentoo-dev 2015-05-18 16:46:11 UTC
*** Bug 534706 has been marked as a duplicate of this bug. ***
Comment 37 James Le Cuirot gentoo-dev 2018-12-30 09:46:50 UTC
Apart from a new issue in 3.7.0 that I will address separately, I believe this has been resolved for some time now.