Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 354877 - dev-lang/python-3.3.2 - error: [...]/work/Python-3.3.2/Modules/_ctypes/libffi: No such file or directory
Summary: dev-lang/python-3.3.2 - error: [...]/work/Python-3.3.2/Modules/_ctypes/libffi...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: MIPS Linux
: High normal (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard: python-overlay
Keywords: InOverlay, InVCS
: 452758 471228 (view as bug list)
Depends on:
Blocks: 380025 348650
  Show dependency tree
 
Reported: 2011-02-14 14:56 UTC by Matt Turner
Modified: 2014-03-25 00:51 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log showing failure (build.log,109.83 KB, text/plain)
2011-02-14 14:57 UTC, Matt Turner
Details
environment (environment,221.03 KB, text/plain)
2011-02-14 14:57 UTC, Matt Turner
Details
patch to add lib32 directories to setup.py (setup.py.diff,431 bytes, patch)
2011-02-14 19:02 UTC, Matt Turner
Details | Diff
build.log (build.log,130.65 KB, text/x-log)
2013-06-29 09:35 UTC, Markos Chandras (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Turner gentoo-dev 2011-02-14 14:56:44 UTC
Relevant error:
error: /var/tmp/portage/dev-lang/python-2.6.6-r1/work/Python-2.6.6/Modules/_ctypes/libffi: No such file or directory

The only relevant hit I see on google is a gentoo/prefix mailing list post, http://archives.gentoo.org/gentoo-alt/msg_0e371693bff5e4835422d0250da7726e.xml

CC'ing them to see if they know what's going on.
Comment 1 Matt Turner gentoo-dev 2011-02-14 14:57:05 UTC
Created attachment 262479 [details]
build.log showing failure
Comment 2 Matt Turner gentoo-dev 2011-02-14 14:57:26 UTC
Created attachment 262481 [details]
environment
Comment 3 Fabian Groffen gentoo-dev 2011-02-14 14:59:38 UTC
Obvious question: do you have libffi installed at the point you build python?
Comment 4 Matt Turner gentoo-dev 2011-02-14 17:17:30 UTC
(In reply to comment #3)
> Obvious question: do you have libffi installed at the point you build python?

Yes, =dev-libs/libffi-3.0.10_rc4 and virtual/libffi are installed.
Comment 5 Fabian Groffen gentoo-dev 2011-02-14 17:47:03 UTC
then my bet is that setup.py does some inventing and lame assuming for your system, as it does for Darwin boxes (which was the cause, last time I saw an incarnation of this problem)
Comment 6 Matt Turner gentoo-dev 2011-02-14 18:15:45 UTC
Arfrever suggested I try 2.7.1 (I had it package.mask'd before, for some reason I've now forgotten), but it fails too in the exact same way.
Comment 7 Matt Turner gentoo-dev 2011-02-14 18:53:34 UTC
So the problem is that lib_dirs (printed in detect_ctypes()) is detected as ['/usr/local/lib', '/lib64', '/usr/lib64', '/lib', '/usr/lib'] but for N32 MIPS, the libdir is lib32.

On MIPS, we've got a clusterfuck of ABIs:
lib is for o32 binaries
lib32 is for n32 binaries
lib64 is for n64 binaries

Looks like we need to add lib32 directories to lib_dirs.
Comment 8 Matt Turner gentoo-dev 2011-02-14 19:02:27 UTC
Created attachment 262513 [details, diff]
patch to add lib32 directories to setup.py

This patch fixes the problem on 2.7.1, and I can't see any reason why it wouldn't for 2.6.6-r1 and 3.1.x.

Does this look reasonable?
Comment 9 Fabian Groffen gentoo-dev 2011-02-14 22:00:59 UTC
I guess adding this in the ebuild (like we do in Prefix) should work for you then too:

append-flags -L${EPREFIX}/$(get_libdir)
append-flags -L${EPREFIX}/usr/$(get_libdir)
Comment 10 Matt Turner gentoo-dev 2011-06-08 01:09:16 UTC
Could we please commit one fix or the other?

lib32 isn't always a symlink and should be checked just like lib and lib64.
Comment 11 Matt Turner gentoo-dev 2011-06-09 16:47:14 UTC
Fabian's change affects python's installed Makefile, but my setup.py patch does not.
Comment 12 Matt Turner gentoo-dev 2011-06-09 17:34:51 UTC
(In reply to comment #11)
> Fabian's change affects python's installed Makefile, but my setup.py patch does
> not.

Hm, also, Fabian's change doesn't fix the problem I'm seeing.
Comment 13 Matt Turner gentoo-dev 2011-06-09 23:54:04 UTC
marienz helped me on irc and discovered that if we use Fabian's change, but use append-ldflags instead, and modify setup.py:431 to unconditionally execute that block then it fixes the bug.

append-flags -L${EPREFIX}/$(get_libdir)
append-flags -L${EPREFIX}/usr/$(get_libdir)

I'm really tired of this bug.
Comment 14 Fabian Groffen gentoo-dev 2011-06-10 18:20:13 UTC
Hmm, the append-ldflags change sounds pretty much right to me.  I'll have to investigate why it work(s/ed) for me.

> I'm really tired of this bug.

Since this bug got a Prefix flavour now, you can expect it to be botted/ignored/thwarted for the forseeable future.
Comment 15 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-06-10 18:28:50 UTC
I think that the safest solution would be to apply patch from comment #8 only on mips (42_mips_libdirs.patch). I will include this patch in next snapshots of Python.
Comment 16 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-06-12 19:41:37 UTC
I have implemented a better solution.

Please test Python 2.7.3_pre20110612, 3.2.1_pre20110612 and 3.3_pre20110612 from python overlay.
Comment 17 Matt Turner gentoo-dev 2011-06-12 21:45:19 UTC
I don't really know what the fix is, but it works. Tested with python-2.7.3_pre20110612.
Comment 18 Dirkjan Ochtman gentoo-dev 2011-06-13 07:20:20 UTC
Arfrever, can you please point to the diff or patch that solved this issue?
Comment 20 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-06-27 14:49:57 UTC
Fixed also in 2.6.7, 2.7.2 and 3.1.4.
Comment 21 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-06-27 16:39:27 UTC
Not yet fixed in 3.2 slot. Release of Python 3.2.1 is delayed...
Comment 22 Matt Turner gentoo-dev 2011-06-27 17:13:42 UTC
(In reply to comment #20)
> Fixed also in 2.6.7, 2.7.2 and 3.1.4.

I just tried 2.7.2, and it didn't work. Same error as before, which I don't understand because when I tested the patch in the overlay (python-2.7.3_pre20110612) it worked.

Thanks for your help, Arfrever.
Comment 23 Matt Turner gentoo-dev 2011-06-28 15:57:27 UTC
Sorry for the noise. Ignore that last report. Something was wrong on my end.

It works in my stage builds. Thanks a lot for your work, Arfrever!
Comment 24 Matt Turner gentoo-dev 2011-07-30 04:29:21 UTC
So, python team, this has been fixed for everything but -3.2. Any ETA?
Comment 25 Matt Turner gentoo-dev 2011-08-15 18:09:07 UTC
(In reply to comment #24)
> So, python team, this has been fixed for everything but -3.2. Any ETA?

OK... well, I'll just package.mask >=python-3.2 on mips until something changes.
Comment 26 Matt Turner gentoo-dev 2011-10-11 21:07:32 UTC
I noticed that 3.2.2 has sed GENTOO_LIBDIR like the other ebuilds, so I thought it might have been fixed, but no.

error: /var/tmp/portage/dev-lang/python-3.2.2/work/Python-3.2.2/Modules/_ctypes/libffi: No such file or directory

Since they retired Arfrever, no one on the python team has acknowledged this bug...
Comment 27 Dirkjan Ochtman gentoo-dev 2011-10-12 07:34:56 UTC
You're looking at 3.2.2, right? It could easily be I copied something with the sed without paying much attention. If you point me in the right direction with what you need for mips to work with python-3.2.2, I'd be happy to include it in the next revision.
Comment 28 Matt Turner gentoo-dev 2011-10-14 00:53:59 UTC
The 04_all_libdir.patch patch and sed @@GENTOO_LIBDIR@@ are the fix.

I'm guessing that there are some places that 04_all_libdir.patch should have patched in 3.2 that were missed, but this is just a guess.
Comment 29 Jim Faulkner 2012-03-07 15:30:22 UTC
FYI, dev-lang/python-3.2.2 builds fine for me on my multilib mips system with gcc 4.6.2.  I was also able to uninstall python-3.1 and run python-updater without any problems.

I think that the python-3.2 mips mask can be removed.

flummox ~ # equery list python
 * Searching for python ...
[IP-] [  ] dev-lang/python-2.7.2-r3:2.7
[IP-] [M ] dev-lang/python-3.2.2:3.2
flummox ~ # python3
Python 3.2.2 (default, Mar  6 2012, 15:52:28) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
flummox ~ # emerge --info
Portage 2.1.10.49 (default/linux/mips/10.0/multilib/n32, gcc-4.6.2, glibc-2.13-r4, 3.2.9 mips64)
=================================================================
System uname: Linux-3.2.9-mips64-R5000_V2.1_FPU_V1.0-with-gentoo-2.1
Timestamp of tree: Wed, 07 Mar 2012 14:15:01 +0000
app-shells/bash:          4.2_p20
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.9.2
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.68
sys-devel/automake:       1.11.3
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.6.2
sys-devel/gcc-config:     1.5.1
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.2-r1 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo
ACCEPT_KEYWORDS="mips ~mips"
ACCEPT_LICENSE="* -@EULA"
CBUILD="mips64-unknown-linux-gnu"
CFLAGS="-Os -march=r5000 -fomit-frame-pointer"
CHOST="mips64-unknown-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-Os -march=r5000 -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-Os -march=r5000 -fomit-frame-pointer"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://jove.eng.yale.edu/gentoo-portage"
USE="berkdb bzip2 cli cracklib crypt cups cxx device-mapper gdbm gpm iconv ipv6 memlimit mips modules mudflap multilib ncurses nls nptl nptlonly offensive pam pcre pppd readline session ssl sysfs tcpd unicode xorg zlib" ALSA_CARDS="au1x00" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev impact newport dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

flummox ~ #
Comment 30 Matt Turner gentoo-dev 2012-09-23 04:18:38 UTC
Thanks Jim. I just unpacked my mips box and confirm that 3.2.3 builds fine here. I'm not sure the reason and I'm not sure I care enough to find out. :)

I'll remove the mask. Rather disappointed in the python team...
Comment 31 Arfrever Frehtes Taifersar Arahesis 2012-09-23 08:28:43 UTC
This bug was fixed in this commit:

*python-3.1.5 (26 Apr 2012)
*python-2.7.3-r1 (26 Apr 2012)
*python-3.2.3 (26 Apr 2012)
*python-2.6.8 (26 Apr 2012)

  26 Apr 2012; Mike Gilbert <floppym@gentoo.org> +python-2.6.8.ebuild,
  +python-2.7.3-r1.ebuild, +python-3.1.5.ebuild, +python-3.2.3.ebuild:
  Version bumps for security bug 396329. Ebuilds and patchsets based on work by
  Arfrever in Progress overlay.
Comment 32 Arfrever Frehtes Taifersar Arahesis 2012-09-23 08:30:40 UTC
(In reply to comment #31)
> This bug was fixed in this commit

In 3.2 slot. Other slots were fixed earlier.
Comment 33 Markos Chandras (RETIRED) gentoo-dev 2013-06-29 09:35:44 UTC
Created attachment 352222 [details]
build.log

python-3.3.2 fails again

python-3.2.5 works ok


Portage 2.1.12.9 (default/linux/mips/13.0/n32, gcc-4.6.3, glibc-2.17, 3.4.27-rt37-Cavium-Octeon mips64)
=================================================================
KiB Mem:     2044516 total,    180900 free
KiB Swap:    1048572 total,   1038136 free
Timestamp of tree: Fri, 28 Jun 2013 07:45:01 +0000
ld GNU ld (GNU Binutils) 2.23.1
app-shells/bash:          4.2_p45
dev-lang/python:          2.7.5, 3.2.5
dev-util/pkgconfig:       0.28
sys-apps/baselayout:      2.2
sys-apps/openrc:          0.11.8
sys-apps/sandbox:         2.6-r1
sys-devel/autoconf:       2.69
sys-devel/automake:       1.12.4, 1.13.2
sys-devel/binutils:       2.23.1
sys-devel/gcc:            4.6.3, 4.7.3
sys-devel/gcc-config:     1.8
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r4
sys-kernel/linux-headers: 3.9 (virtual/os-headers)
sys-libs/glibc:           2.17
Repositories: gentoo
ACCEPT_KEYWORDS="mips ~mips"
ACCEPT_LICENSE="* -@EULA"
CBUILD="mips64-unknown-linux-gnu"
CFLAGS="-O2 -march=mips64 -mplt -pipe"
CHOST="mips64-unknown-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=mips64 -mplt -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS=""
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="ftp://ftp.ucsb.edu/pub/mirrors/linux/gentoo/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j7"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://mirror.bytemark.co.uk/gentoo-portage"
USE="acl berkdb bzip2 cli cracklib crypt cxx gdbm gpm iconv ipv6 mips modules mudflap ncurses nls nptl pam pcre readline session ssl tcpd unicode zlib" ALSA_CARDS="au1x00" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev impact newport dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Comment 34 Marien Zwart (RETIRED) gentoo-dev 2013-06-29 10:10:35 UTC
I had a quick look at this, but I don't see what changed here either in the ebuild or in Python. I wish I remembered if I did anything particularly clever when debugging this before. I think I'd have to add a bunch of logging to detect_ctypes in setup.py and/or look at an strace of setup.py to see where it goes off the rails.
Comment 35 Markos Chandras (RETIRED) gentoo-dev 2013-07-01 18:44:21 UTC
(In reply to Marien Zwart from comment #34)
> I had a quick look at this, but I don't see what changed here either in the
> ebuild or in Python. I wish I remembered if I did anything particularly
> clever when debugging this before. I think I'd have to add a bunch of
> logging to detect_ctypes in setup.py and/or look at an strace of setup.py to
> see where it goes off the rails.

Thanks for looking at it. Well we used to have this problem in the patch, then you applied a magic patch that fixed it but somehow it seems more changes are needed in recent python versions. This kinda blocks up2date stages for MIPS64 multilib.
Comment 36 Markos Chandras (RETIRED) gentoo-dev 2013-07-10 19:58:26 UTC
@MIPS

Any objection to mask this python version for mips?
Comment 37 Markos Chandras (RETIRED) gentoo-dev 2013-07-21 18:18:14 UTC
*** Bug 471228 has been marked as a duplicate of this bug. ***
Comment 38 Mike Gilbert gentoo-dev 2013-07-21 18:46:24 UTC
Arfrever says this is a different problem than the one originally reported. Could someone please file a new bug and close this one.
Comment 39 Mike Gilbert gentoo-dev 2013-07-21 19:48:12 UTC
Based on a conversation between Arfrever and hwoarang in #gentoo-python, it seems that the ebuild in Progress overlay works find.

There is probably some difference between our patchsets.
Comment 40 Markos Chandras (RETIRED) gentoo-dev 2013-07-24 09:21:25 UTC
*** Bug 452758 has been marked as a duplicate of this bug. ***
Comment 41 Markos Chandras (RETIRED) gentoo-dev 2013-08-14 19:38:10 UTC
(In reply to Mike Gilbert from comment #39)
> Based on a conversation between Arfrever and hwoarang in #gentoo-python, it
> seems that the ebuild in Progress overlay works find.
> 
> There is probably some difference between our patchsets.

ping? anyone willing to copy/backport Arfrever's patch properly?
Comment 42 Mike Gilbert gentoo-dev 2013-08-15 00:48:37 UTC
I'll put it on my weekend to-do list if nobody gets it first.
Comment 43 Mike Gilbert gentoo-dev 2013-08-18 18:35:54 UTC
+*python-3.3.2-r2 (18 Aug 2013)
+
+  18 Aug 2013; Mike Gilbert <floppym@gentoo.org>
+  +files/CVE-2013-4073_py33.patch, +python-3.3.2-r2.ebuild:
+  Use Arfrever's patchset, bug 354877. Apply fix for CVS-2013-4238, bug 480856.

Please confirm that this resolves the issue.
Comment 44 Mike Gilbert gentoo-dev 2013-08-22 20:17:00 UTC
Markos reports that this is working now.
Comment 45 Weedy 2014-03-25 00:26:15 UTC
2.7.6, 3.2.5-r3, and 3.3.4 fail to build on my system.

Was this patch dropped?
Default 64bit multilib here.