| Summary: | Python Numeric 23.x has broken random sampling libraries on amd64 | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Chris Monson <shiblon> |
| Component: | [OLD] Development | Assignee: | Python Gentoo Team <python> |
| Status: | VERIFIED NEEDINFO | ||
| Severity: | normal | ||
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | AMD64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
Cut and paste error in the patch. It should be:
--- RandomArray.py Mon Sep 13 13:01:36 2004
***************
*** 1,4 ****
--- 1,5 ----
import ranlib
+ import random as pythonrandom
import Numeric
import LinearAlgebra
import sys
*************** def multivariate_normal(mean, cov, shape
*** 127,133 ****
# Create a matrix of independent standard normally distributed random
# numbers. The matrix has rows with the same length as mean and as
# many rows are necessary to form a matrix of shape final_shape.
! x = ranlib.standard_normal(Numeric.multiply.reduce(final_shape))
x.shape = (Numeric.multiply.reduce(final_shape[0:len(final_shape)-1]),
mean.shape[0])
# Transform matrix of standard normals into matrix where each row
--- 128,135 ----
# Create a matrix of independent standard normally distributed random
# numbers. The matrix has rows with the same length as mean and as
# many rows are necessary to form a matrix of shape final_shape.
! #x = ranlib.standard_normal(Numeric.multiply.reduce(final_shape))
! x = Numeric.array([pythonrandom.normalvariate(0,1) for z in xrange(Numeric.multiply.reduce(final_shape))])
x.shape = (Numeric.multiply.reduce(final_shape[0:len(final_shape)-1]),
mean.shape[0])
# Transform matrix of standard normals into matrix where each row
Patches as plain text attachment, please. I committed numeric-23.6. Is this still an issue with this version? If so, please add a comment and report upstream please, otherwise this bug could be closed. Works for me, now. Thanks. Sorry about the inline patches. Oops. I think that I changed the status wrong... Now closed. |
It appears that ranlib.c is broken under amd64 in the dev-python/numeric ebuild. I have narrowed the problem down somewhat and have a patch that works for me, but that does not fix the real problem. The patch is here: --- RandomArray.py Mon Sep 13 13:01:36 2004 *************** *** 1,4 **** --- 1,5 ---- import ranlib + import random as pythonrandom import Numeric import LinearAlgebra import sys *************** def multivariate_normal(mean, cov, shape *** 127,133 **** # Create a matrix of independent standard normally distributed random # numbers. The matrix has rows with the same length as mean and as # many rows are necessary to form a matrix of shape final_shape. ! x = ranlib.standard_normal(Numeric.multiply.reduce(final_shape)) x.shape = (Numeric.multiply.reduce(final_shape[0:len(final_shape)-1]), mean.shape[0]) # Transform matrix of standard normals into matrix where each row --- 128,135 ---- # Create a matrix of independent standard normally distributed random # numbers. The matrix has rows with the same length as mean and as # many rows are necessary to form a matrix of shape final_shape. ! #x = ranlib.standard_normal(Numeric.multiply.reduce(final_shape)) ! x = Numeric.array([pythonrandom.normalvariate(0,1) for z in xrange(Nume-->ric.multiply.reduce(final_shape))]) x.shape = (Numeric.multiply.reduce(final_shape[0:len(final_shape)-1]), mean.shape[0]) # Transform matrix of standard normals into matrix where each row The patch applies to RandomArray.py and basically circumvents the call to ranlib.standard_normal (but only for the multivariate_normal call), which calls the 'snorm' function in the ranlib.c file. Something in snorm is broken for amd64. Unfortunately, it looks like the file was generated by an automatic Fortran => C tool, so it may be difficult to track down the true problem. The problem, for example, may be in one of the deeper function calls resulting from the call to snorm. It's hard to say. Numeric is not currently in active development, but numarray (it's eventual replacement) does not duplicate this functionality, and it looks like SciPy uses the Numeric libraries to do its work as well, making this seem like the right forum for this bug. It should be noted that the snorm.f file from which this particular function was taken (it can be found in the octave distribution) looks exactly the same and either works or is not used as the basis for Octave's 'randn' (ranlib, after all, is located in octave's libcruft directory, which says a lot). One final note: this is not just broken for Numeric-23.3, which has the ~amd64 keyword, but also for Numeric-23.1, which has the 'amd64' keyword, indicating that it works fine. It does not, and probably should at least have the ~ reapplied for that ebuild. Reproducible: Always Steps to Reproduce: 1. Run the following python code #!/usr/bin/env python from RandomArray import standard_normal print standard_normal(2) Actual Results: The code outputs something like [ 1.59162593e-314 5.22396115e-315] The numbers it outputs are not particularly important in and of themselves, but for a standard normal distribution they are consistently far too close to zero (variance should be 1, after all). Expected Results: It should output more reasonable values, like [-1.7442323 -0.43547371] Recall that this is a random sample, so YMMV. The key here is that the way the numbers are sampled is always wrong on amd64. The numbers always end up being something e-31x, which is consistently far too close to zero to be correct. Portage 2.0.50-r10 (default-x86-2004.2, gcc-3.3.4, glibc-2.3.3.20040420-r1, 2.6.8-gentoo-r3) ================================================================= System uname: 2.6.8-gentoo-r3 i686 Intel(R) Xeon(TM) CPU 2.80GHz Gentoo Base System version 1.4.16 Autoconf: sys-devel/autoconf-2.59-r4 Automake: sys-devel/automake-1.8.5-r1 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CFLAGS="-O2 -march=i686 -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" COMPILER="" CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.2/share/config /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=i686 -fomit-frame-pointer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoaddcvs ccache sandbox" GENTOO_MIRRORS="ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/ http://gentoo.ccccom.com ftp://gentoo.ccccom.com http://ftp.ntua.gr/pub/linux/gentoo/" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X alsa apm arts avi berkdb crypt cups encode esd foomaticdb gdbm gif gnome gpm gtk gtk2 imlib jpeg kde ldap libg++ libwww mad mikmod motif mozilla mpeg ncurses nls oggvorbis opengl oss pam pdflib perl png python qt quicktime readline sdl slang spell ssl svga tcpd tetex truetype x86 xml2 xmms xv zlib"