Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 351206 - dev-python/cython fails tests with >=dev-python/numpy-1.5
Summary: dev-python/cython fails tests with >=dev-python/numpy-1.5
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All All
: High normal (vote)
Assignee: Python Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-01-09 13:59 UTC by Theo Chatzimichos (RETIRED)
Modified: 2011-12-27 07:42 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log 0.14-r1 (build.log,810.36 KB, text/plain)
2011-01-09 14:05 UTC, Theo Chatzimichos (RETIRED)
Details
update cython ebuild (cython-0.14.1-r1.ebuild,1.33 KB, text/plain)
2011-03-20 22:56 UTC, Neil Leathers
Details
updated ebuild (cython-0.14.1-r1.ebuild,1.33 KB, text/plain)
2011-03-20 23:02 UTC, Neil Leathers
Details
the build log (cython-0.15.1-tail-build.log,8.90 KB, text/plain)
2011-12-07 17:58 UTC, Ian Delaney (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Theo Chatzimichos (RETIRED) archtester gentoo-dev Security 2011-01-09 13:59:35 UTC
Portage 2.2.0_alpha13 (default/linux/amd64/10.0/desktop/kde, gcc-4.5.2, glibc-2.12.1-r3, 2.6.36-gentoo-r3 x86_64)
=================================================================
                        System Settings
=================================================================
System uname: Linux-2.6.36-gentoo-r3-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q6600_@_2.40GHz-with-gentoo-2.0.1
Timestamp of tree: Unknown
distcc 3.1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:     4.1_p9
dev-java/java-config: 2.1.11-r3
dev-lang/python:     2.6.6-r1, 2.7.1, 3.1.3
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.1-r1
sys-apps/openrc:     0.6.8
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.13, 2.68
sys-devel/automake:  1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:  2.21
sys-devel/gcc:       4.4.5, 4.5.2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.4-r1
sys-devel/make:      3.82
virtual/os-headers:  2.6.36.1 (sys-kernel/linux-headers)
Repositories: gentoo multimedia mysql kde qting-edge jedi-mind-builds
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=core2 -O2 -pipe -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/splash /etc/terminfo"
CXXFLAGS="-march=core2 -O2 -pipe -ggdb"
DISTDIR="/home/portage/distfiles"
FEATURES="assume-digests binpkg-logs buildpkg candy distlocks fixlafiles fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict test unknown-features-warn unmerge-logs unmerge-orphans userfetch"
GENTOO_MIRRORS="http://ftp.cc.uoc.gr/mirrors/linux/gentoo/ http://ftp.ntua.gr/pub/linux/gentoo/ http://distfiles.gentoo.org/"
LANG="en_US.utf8"
LDFLAGS="-Wl,--as-needed"
LINGUAS="en el"
MAKEOPTS="-j5"
PKGDIR="/home/portage/binaries"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/home/tampakrap/Source_Code/gentoo/gentoo-x86"
PORTDIR_OVERLAY="/var/lib/layman/multimedia /var/lib/layman/mysql /home/tampakrap/Source_Code/gentoo/kde /home/tampakrap/Source_Code/gentoo/qting-edge /home/tampakrap/Source_Code/gentoo/tampakrap"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="64bit 7zip X X509 a52 aac aalib acl acpi addressbook akonadi alsa amd64 amr apache2 asf bash-completion berkdb bidi bluetooth branding bzip2 cairo cdda cddb cdio cdparanoia cdr cgi chm cli consolekit cpudetection cracklib crypt cups curl cvs cxx dbus directfb dmx dri dts dv dvb dvd dvdr emboss encode esd exif expat fam fame fbcon ffmpeg firefox flac foomaticdb fortran gdbm gdu gif gimp git glitz gnutls gphoto2 gpm graphviz hal handbook hash hddtemp hpn iconv idn ieee1394 imagemagick ipod ithreads java joystick jpeg jpeg2k kde kdrive kipi kqemu kvm lame lcms ldap libcaca libnotify lirc lm_sensors lzma mad matroska mikmod mjpeg mmx mmxext mng modules mono mp3 mp4 mpeg mplayer mudflap multilib musepack mysql mysqli nas ncurses network networkmanager nfs nls nodrm npp nptl ogg openexr opengl openmp pam pango pcre pdf perl phonon php plasma png policykit ppds pppd python qt qt3support qt4 quicktime radio rar raster rdesktop readline ruby samba sasl scanner sdl semantic-desktop sensord session sftp simplexml smp sndfile snmp speex spell sql sqlite srt sse sse2 sse3 ssl ssse3 startup-notification subtitles subversion svg sysfs syslog tcpd theora threads tiff truetype udev unicode usb v4l v4l2 vcd vdpau vhosts vim-syntax vorbis webkit winbind wma x264 xcb xcomposite xine xinerama xml xmlreader xmlrpc xmlwriter xorg xpm xscreensaver xsl xulrunner xv xvid xvmc yv12 zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse joystick" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en el" LIRC_DEVICES="devinput hauppauge" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nv nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
                        Package Settings
=================================================================

dev-python/cython-0.13_beta0 was built with the following:
USE="(multilib) -doc -examples"
Comment 1 Theo Chatzimichos (RETIRED) archtester gentoo-dev Security 2011-01-09 14:05:04 UTC
Created attachment 259369 [details]
build.log 0.14-r1
Comment 2 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-01-09 15:13:38 UTC
It's a known problem occuring only when >=dev-python/numpy-1.5 is installed:
http://trac.cython.org/cython_trac/ticket/630
Comment 3 Neil Leathers 2011-03-20 22:56:42 UTC
Created attachment 266661 [details]
update cython ebuild

Adds a dependency blocker for the versions of numpy that upstream states are unsupported.
Comment 4 Neil Leathers 2011-03-20 23:02:02 UTC
Created attachment 266665 [details]
updated ebuild

Upload the correct file
Comment 5 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-03-22 11:27:43 UTC
Comment on attachment 266665 [details]
updated ebuild

It doesn't make sense to block new NumPy.
Comment 6 Neil Leathers 2011-03-26 07:58:13 UTC
> It doesn't make sense to block new NumPy.

In the opinion of the upstream developers this version of Cython (and earlier versions of Cython) does not work with NumPy version 1.5 or newer and Cython code needs to be fixed before Cython is compatible. Thus NumPy version 1.5 or later cannot be installed if Cython is install. This is why the package block concept exists. You must choose one or the other otherwise packages fail. Fix the problem with a block until there is a code fix.
Comment 7 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-03-26 15:39:56 UTC
NumPy 1.5 has introduced support for Python 3 and many other fixes (including fix for building on ppc and ppc64), so blocking it is unacceptable. Older versions of NumPy will be removed after stabilization of NumPy 1.5.*.
Comment 8 Kevin F. Quinn 2011-05-02 00:09:01 UTC
If anyone else is looking for Cython's take on this - they believe it's a bug in NumPy (took me a little while to find it as their trac ticket doesn't hyperlink to the commit...):

https://github.com/cython/cython/commit/fcd59a76dd679feda9c24c75cf21c78ea4a0cbb7

however that commit doesn't apply to cython 1.14.1 so we'll just have to wait.
Comment 9 Johan Bergström 2011-09-06 06:24:11 UTC
FWIW: I just ran the python 0.15 test suite against bumpy 1.5.1 on AMD64. These tests pass on python 2.4,2.5,2.6 and 2.7
Comment 10 Ian Delaney (RETIRED) gentoo-dev 2011-12-07 17:22:00 UTC
an update on this.
after repeating the test phase many times, it seems that currently;
cython test phases passes for python2* for numpy 1.5.1
cython test phase fails only against python3.2 for numpy 1.5.1

cython test phase fails on every CPython for numpy 1.6.0 up.

Now the  failures for the former are;
File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 181, in _compile
seemingly repeated 4 times, but looks like just the 1. Just the 1 little error.

Now, testing against numpy 1.6.0 up yields a curious outcome.

File "/mnt/gen2/tmpdir/portage/dev-python/cython-0.15.1/work/Cython-0.15.1/tests-2.5/run/c/numpy_test.so", line 226, in numpy_test
Failed example:
    print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a', 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python2.5/doctest.py", line 1228, in __run
        compileflags, 1) in test.globs
      File "<doctest numpy_test[72]>", line 1, in <module>
        print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a', 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
      File "numpy_test.pyx", line 451, in numpy_test.test_partially_packed_align (numpy_test.c:7028)
    ValueError: Buffer dtype mismatch; next field is at offset 13 but 16 expected

and a second 1 for a second partially_packed struct, basically the same thing twice.

On bringing up the numpy_test.pyx, we find 
The following expose bugs in Numpy (versions prior to 2011-04-02):
        
        >>> print(test_partially_packed_align(np.zeros((1,), dtype=np.dtype([('a', 'b'), ('b', 'i'), ('sub', np.dtype('b,i')), ('c', 'i')], align=True))))
        array([(22, 23, (24, 25), 26)], 
              dtype=[('a', '|i1'), ('', '|V3'), ('b', '!i4'), ('sub', [('f0', '|i1'), ('f1', '!i4')]), ('', '|V3'), ('c', '!i4')])
    
        >>> print(test_partially_packed_align_2(np.zeros((1,), dtype=np.dtype([('a', 'b'), ('b', 'i'), ('c', 'b'), ('sub', np.dtype('b,i', align=True))]))))
        array([(22, 23, 24, (27, 28))], 
              dtype=[('a', '|i1'), ('b', '!i4'), ('c', '|i1'), ('sub', [('f0', '|i1'), ('', '|V3'), ('f1', '!i4')])])


well from the Changelog, 
*numpy-1.6.0_beta2 (07 Apr 2011)

  07 Apr 2011;
heralded the intro if the first pre-release of numpy-1.6
What is not clear to me is the meaning of 
"The following expose bugs ".
That means to me these tests will make it trip and pull up, when the opposite occurs.

What is clear is that only an installed numpy-1.5.1 will see a cython test phase pass and I have the logs to prove it.
Now, what do we find in  numpy_test.pyx;

def test_partially_packed_align(np.ndarray[PartiallyPackedStruct] arr):
    arr[0].a = 22
    arr[0].b = 23
    arr[0].sub.a = 24
    arr[0].sub.b = 25
    arr[0].c = 26
    return repr(arr).replace('<', '!').replace('>', '!')

def test_partially_packed_align_2(np.ndarray[PartiallyPackedStruct2] arr):
    arr[0].a = 22
    arr[0].b = 23
    arr[0].c = 24
    arr[0].sub.a = 27
    arr[0].sub.b = 28
    return repr(arr).replace('<', '!').replace('>', '!')

The exact test quoted above, and sure enough that is exactly where the cython tests trip for > numpy 1.5.1; more than a coincidence.

Perhaps the choice of words "The following expose bugs "is just completely mis-leading, but the outcome of testing the test phase of cython gives gentoo a headache for cython test phase failing against all installed numpy(s) but the oldest version in the tree.

Is it that these two PartiallyPackedStruct & PartiallyPackedStruct2 are out of place in the current test suite seeing they only work in version 1.5.1?
It does to me.
Ought they be retained and hold the entire test phase to ransom?
Will someone savant of numpy structs and arrays actually fix it?
Ought they be removed from the test suite and allow cython to pass?
Do we await upstream to fix it?
Comment 11 Ian Delaney (RETIRED) gentoo-dev 2011-12-07 17:58:43 UTC
Created attachment 295103 [details]
the build log
Comment 12 Johan Bergström 2011-12-27 07:42:22 UTC
Fixed with cython 0.15.1 and numpy 1.6.1-r1