Benjamin C. Wiley Sittler reports: hi, i discovered a bug yesterday in repr() for unicode strings. this causes an unpatched non-debug wide (UTF-32/UCS-4) build of python to abort: python2.4 -c 'assert(repr(u"\U00010000" * 39 + u"\uffff" * 4096)) == (repr(u"\U00010000" * 39 + u"\uffff" * 4096))' the problem is fixed by a change to unicodeobject.c. in the process of fixing it i also found and fixed another bug in repr() on UCS-4 python builds -- previously paired unicode surrogates were being repr()'ed as a single "character" even though they are not treated as such by a UCS-4 python build -- i.e. eval(repr(u'\ud800\udc00')) != u'\ud800\udc00' in an unpatched UCS-4 build. Package: python2.4 Version: 2.4.3-7ubuntu2 Severity: important when i run this command: python -c "repr(u'\u24ea\u059c\u200a\U0001d77e\uff07\u202f\u0747\u202f \U0001d56b\U0001d5b9\U0001d4e9\u20052\u14bf\U0001d7f8\u200a\U0001d795 \U0001d6e7Z\u2006\u2002\U0001d50a\uff27\u13c0\u2000\uff16\u0411\uff16 \U0001d7e7\uff4c\u2006\u2001\ufe39\u2008\u0313]\u2008\u3014\u3015')" python aborts with the following backtrace and memory dump: *** glibc detected *** python: realloc(): invalid next size: 0x081521e8 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6[0xb7e8acd4] /lib/tls/i686/cmov/libc.so.6(__libc_realloc+0xff)[0xb7e8cc5f] python(_PyString_Resize+0x80)[0x8082b4b] python[0x80991f7] python(PyObject_Repr+0x58)[0x807d1fd] python(PyEval_EvalFrame+0x4b37)[0x80b5270] python(PyEval_EvalCodeEx+0x836)[0x80b65d6] python(PyEval_EvalCode+0x57)[0x80b6640] python(PyRun_SimpleStringFlags+0xa8)[0x80d8b7c] python(Py_Main+0x685)[0x8055862] python(main+0x22)[0x80550e2] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xd8)[0xb7e378b8] python[0x8055041] ======= Memory map: ======== 08048000-0811a000 r-xp 00000000 08:03 622736 /usr/bin/python2.4 0811a000-0813b000 rw-p 000d1000 08:03 622736 /usr/bin/python2.4 0813b000-081b5000 rw-p 0813b000 00:00 0 [heap] b7c00000-b7c21000 rw-p b7c00000 00:00 0 b7c21000-b7d00000 ---p b7c21000 00:00 0 b7d40000-b7d4a000 r-xp 00000000 08:03 376899 /lib/libgcc_s.so.1 b7d4a000-b7d4b000 rw-p 00009000 08:03 376899 /lib/libgcc_s.so.1 b7d68000-b7d9b000 r--p 00000000 08:03 82634 /usr/lib/locale/en_US.utf8/LC_CTYPE b7d9b000-b7d9e000 r-xp 00000000 08:03 625529 /usr/lib/python2.4/lib-dynload/_locale.so b7d9e000-b7d9f000 rw-p 00003000 08:03 625529 /usr/lib/python2.4/lib-dynload/_locale.so b7d9f000-b7e22000 rw-p b7d9f000 00:00 0 b7e22000-b7f51000 r-xp 00000000 08:03 66543 /lib/tls/i686/cmov/libc-2.4.so b7f51000-b7f53000 r--p 0012e000 08:03 66543 /lib/tls/i686/cmov/libc-2.4.so b7f53000-b7f55000 rw-p 00130000 08:03 66543 /lib/tls/i686/cmov/libc-2.4.so b7f55000-b7f58000 rw-p b7f55000 00:00 0 b7f58000-b7f7c000 r-xp 00000000 08:03 66547 /lib/tls/i686/cmov/libm-2.4.so b7f7c000-b7f7e000 rw-p 00023000 08:03 66547 /lib/tls/i686/cmov/libm-2.4.so b7f7e000-b7f80000 r-xp 00000000 08:03 68161 /lib/tls/i686/cmov/libutil-2.4.so b7f80000-b7f82000 rw-p 00001000 08:03 68161 /lib/tls/i686/cmov/libutil-2.4.so b7f82000-b7f83000 rw-p b7f82000 00:00 0 b7f83000-b7f85000 r-xp 00000000 08:03 66546 /lib/tls/i686/cmov/libdl-2.4.so b7f85000-b7f87000 rw-p 00001000 08:03 66546 /lib/tls/i686/cmov/libdl-2.4.so b7f87000-b7f96000 r-xp 00000000 08:03 68156 /lib/tls/i686/cmov/libpthread-2.4.so b7f96000-b7f98000 rw-p 0000f000 08:03 68156 /lib/tls/i686/cmov/libpthread-2.4.so b7f98000-b7f9a000 rw-p b7f98000 00:00 0 b7fb0000-b7fb7000 r--s 00000000 08:03 2130015 /usr/lib/gconv/gconv-modules.cache b7fb7000-b7fb9000 rw-p b7fb7000 00:00 0 b7fb9000-b7fd2000 r-xp 00000000 08:03 2737127 /lib/ld-2.4.so b7fd2000-b7fd4000 rw-p 00018000 08:03 2737127 /lib/ld-2.4.so bf99b000-bf9b3000 rw-p bf99b000 00:00 0 [stack] ffffe000-fffff000 ---p 00000000 00:00 0 [vdso] Aborted
Python please advise and bump as necessary.
Just committed python-2.3.5-r3 and 2.4.3-r4 with this patched. 2.5 is unaffected. 2.2 and 2.1 are probably affected but I do not think it is worth it to patch them: we cannot keep supporting them forever. 2.3.5-r3 (which is 2.3.5-r2 with a patch for this single issue) and 2.4.3-r4 (which contains some other fixes from the ~arch 2.4.3-r3) should go stable. <2.3 should probably be package.masked, but I have not discussed that with all the python project members yet.
Thx Marien. Arches please test and mark stable. Target keywords are: python-2.3.5-r3.ebuild:KEYWORDS="alpha amd64 arm hppa ia64 m68k mips ppc ppc64 s390 sh sparc x86" python-2.4.3-r4.ebuild:KEYWORDS="alpha amd64 arm hppa ia64 m68k mips ppc ppc64 s390 sh sparc x86 ~x86-fbsd"
ppc64 stable
2.4.3-r4 1) emerges fine so far: Listing /usr/lib/python24.zip ... Can't list /usr/lib/python24.zip [...] Listing /usr/lib/python2.4/lib-tk ... Can't list /usr/lib/python2.4/lib-tk [...] 2) passes collision test 3) passes test suite, but 1 skip unexpected on linux2: test_locale 4) works (tested a system update with portage 2.1.1-r1, see bug #149062) 2.3.5-r3 1) emerges fine 2) fails collision test (slotted with python 2.4) * checking 2531 files for package collisions existing file /usr/bin/idle is not owned by this package existing file /usr/bin/pydoc is not owned by this package existing file /usr/bin/python-config is not owned by this package existing file /usr/sbin/python-updater is not owned by this package 3) passes test suite, but 1 skip unexpected on linux2: test_locale mv: cannot stat `/var/tmp/portage/python-2.3.5-r3/temp/test_subprocess.py': No such file or directory mv: cannot stat `/var/tmp/portage/python-2.3.5-r3/temp/test_tcl.py': No such file or directory 4) works Portage 2.1.1 (default-linux/x86/2006.1/desktop, gcc-4.1.1, glibc-2.4-r3, 2.6.17-gentoo-r8 i686) ================================================================= System uname: 2.6.17-gentoo-r8 i686 AMD Athlon(tm) XP 2500+ Gentoo Base System version 1.12.5 Last Sync: Tue, 26 Sep 2006 07:20:02 +0000 app-admin/eselect-compiler: [Not Present] dev-java/java-config: 1.2.11-r1 dev-lang/python: 2.4.3-r1 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: [Not Present] dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.13-r3 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r1 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/splash /etc/terminfo" CXXFLAGS="-O2" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache collision-protect distlocks metadata-transfer parallel-fetch sandbox sfperms strict test" GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo/" LANG="de_DE@euro" LC_ALL="de_DE@euro" LINGUAS="de" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.informatik.rwth-aachen.de/gentoo-portage" USE="x86 3dnow 3dnowext X Xaw3d a52 alsa artworkextra asf audiofile bash-completion beagle berkdb bidi bitmap-fonts bootsplash branding bzip2 cairo cdda cddb cdparanoia cdr cli cracklib crypt css cups curl custom-cflags dbus dga directfb divx4linux dlloader dri dts dvd dvdr dvdread dvi eds elibc_glibc emacs emboss encode esd evo exif expat fam fat fbcon ffmpeg firefox fortran ftp gb gcj gdbm gif gnome gpm gstreamer gtk gtk2 gtkhtml hal icq idn imagemagick imap input_devices_keyboard input_devices_mouse ipv6 isdnlog java javascript jikes jpeg jpeg2k kde kernel_linux ldap leim libg++ linguas_de lm_sensors mad maildir matroska mbox mhash mikmod mime mmx mmxext mng mono mp3 mpeg mpeg2 mule nautilus ncurses nforce2 nls nocardbus nptl nptlonly nsplugin nvidia objc ogg opengl pam pcre pdf perl plotutils pmu png ppds pppd preview-latex print python qt3 qt4 quicktime readline reflection reiserfs samba sdk session slang spell spl sse ssl svg svga t1lib tcltk tcpd tetex theora thunderbird tiff truetype truetype-fonts type1-fonts udev usb userland_GNU vcd video_cards_fbdev video_cards_radeon video_cards_vesa videos vorbis win32codecs wmf wxwindows xine xml xorg xosd xv xvid zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LDFLAGS, PORTAGE_RSYNC_EXTRA_OPTS
ppc stable
amd64 all stable
hppa stable.
python-2.3.5-r3: 1.) emerges fine on x86, following QA Info: QA Notice: USE Flag 'elibc_uclibc' not in IUSE for dev-lang/python-2.3.5-r3 2.) passes collision test 3.) passes test suite, but test_dbm test_dbm skipped -- No module named dbm 1 skip unexpected on linux2: test_dbm 4.) works python-2.4.3-r4: 1.) emerges fine on x86, with the following QA Info: QA Notice: USE Flag 'elibc_uclibc' not in IUSE for dev-lang/python-2.4.3-r4 2.) passes collision test 3.) passes test suite, but test_dbm fails as above 4.) works emerge --info Portage 2.1.1 (default-linux/x86/2006.1/desktop, gcc-4.1.1, glibc-2.4-r3, 2.6.17.13 i686) ================================================================= System uname: 2.6.17.13 i686 AMD Athlon(TM) XP1800+ Gentoo Base System version 1.12.5 Last Sync: Tue, 26 Sep 2006 14:50:01 +0000 ccache version 2.3 [enabled] app-admin/eselect-compiler: [Not Present] dev-java/java-config: 1.2.11-r1 dev-lang/python: 2.3.5-r3, 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 dev-util/ccache: 2.3 dev-util/confcache: [Not Present] sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.13-r3 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r1 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=i686 -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c" CXXFLAGS="-O2 -march=i686 -fomit-frame-pointer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache collision-protect distlocks fixpackages metadata-transfer parallel-fetch sandbox sfperms strict test userfetch userpriv usersandbox" GENTOO_MIRRORS="http://mirror.switch.ch/mirror/gentoo/ http://gentoo.inode.at/" LANG="en_GB.utf8" LINGUAS="en de en_GB" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude='/distfiles' --exclude='/local' --exclude='/packages'" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage/normal /usr/local/portage/testing" SYNC="rsync://192.168.2.1/gentoo-portage" USE="x86 3dnow 3dnowext X a52 aac acpi alsa apache2 bash-completion berkdb bitmap-fonts bzip2 cairo cdr cli crypt css cups dbus divx4linux dlloader dri dts dvd dvdr dvdread elibc_glibc emboss exif fam ffmpeg firefox font-server fortran gdbm gif gnome gphoto2 gpm gstreamer gtk gtk2 gtkhtml hal input_devices_keyboard input_devices_mouse ipv6 isdnlog java jpeg kde kernel_linux ldap libclamav libg++ linguas_de linguas_en linguas_en_GB logitech-mouse mad mikmod mmx mmxext mono mozcalendar mozdevelop mozsvg mp3 mpeg ncurses network nls nptl nptlonly nvidia oav ogg opengl oss pam pcre perl png ppds pppd python qt qt3 qt4 quicktime readline reflection samba sdl seamonkey session spell spl ssl tcltk tcpd test tetex tiff truetype truetype-fonts type1-fonts udev unicode usb userland_GNU vcd video_cards_none video_cards_nv vorbis win32codecs xine xinerama xml xorg xorg-x11 xprint xv xvg xvid zlib" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, MAKEOPTS, PORTAGE_RSYNC_EXTRA_OPTS
SPARC stable
security comes before problems for x86 :(
Stable on Alpha.
GLSA 200610-07
This change to the repr() behavior was not really related to this bug, and causes the Gentoo Python to behave differently than other Pythons at the same version number. I'm curious both why it was included in this patch (which otherwise just fixed a bug) and why it was not reported to upstream (or, if it was, I can't find it).
Sorry, it was reported upstream (from ubuntu, but whatever): http://sourceforge.net/tracker/index.php?func=detail&aid=1541585&group_id=5470&atid=305470 (apparently a search for 'unicode repr', among other things, won't find that?) So I withdraw my question.
