Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 246091 - dev-lang/python-2.5.2-r7 - import readline in python writes strange character to stdout
Summary: dev-lang/python-2.5.2-r7 - import readline in python writes strange character...
Status: RESOLVED OBSOLETE
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: High normal with 1 vote (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-11-08 20:01 UTC by David Flogeras
Modified: 2016-01-02 03:28 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
strace output of python executing 'import readline' (strace.txt.gz,5.65 KB, application/octet-stream)
2008-11-10 00:40 UTC, David Flogeras
Details
required definitions from ubuntu (from_ubuntu_hardy.tgz,289.93 KB, application/octet-stream)
2009-03-31 20:58 UTC, David Flogeras
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Flogeras 2008-11-08 20:01:25 UTC
Running a x86 stable:
sys-libs/ncurses-5.6-r2
sys-libs/readline-5.2_p12-r1
dev-lang/python-2.5.2-r7

If I execute 'import readline' inside of a python script, the sequence "\33[?1034h" is written to stdout (I verified using strace).

I believe it is coming from the ncurses library.  I grepped through its source and found \E[?1034h on line 3148 of misc/terminfo.src.  I have no idea what that does, but its the same byte-sequence.

I have tried this on 3 different gentoo boxes (one amd64). same behaviour.  I tried on an Ubuntu box, it does not display this behaviour.

Reproducible: Always

Steps to Reproduce:
1. put 'import readline' into a file test.py
2. strace -o trace.txt python test.py, look for the sequency 1034h
3. alternatively python test.py > output.txt, open in vim, see an unexpected char

Actual Results:  
Strange unicode character written to stdout

Expected Results:  
Nothing written to stdout
Comment 1 Wormo (RETIRED) gentoo-dev 2008-11-09 08:13:51 UTC
I am unable to reproduce this on an amd64 box with matching versions of the listed packages. Please post 'emerge --info' output and also output from 'env' on one of your affected boxes.
Comment 2 David Flogeras 2008-11-09 11:54:52 UTC
Portage 2.1.4.5 (default/linux/x86/2008.0/desktop, gcc-4.1.2, glibc-2.6.1-r0, 2.6.25-gentoo-r9 i686)
=================================================================
System uname: 2.6.25-gentoo-r9 i686 Genuine Intel(R) CPU T2500 @ 2.00GHz
Timestamp of tree: Sat, 08 Nov 2008 13:01:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
app-shells/bash:     3.2_p33
dev-java/java-config: 1.3.7, 2.1.6
dev-lang/python:     2.5.2-r7
dev-python/pycrypto: 2.0.1-r6
dev-util/cmake:      2.4.6-r1
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.61-r2
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1
sys-devel/binutils:  2.18-r3
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.26
virtual/os-headers:  2.6.23-r3
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d"
CXXFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo"
LANG="en_US"
LDFLAGS="-Wl,-O1"
LINGUAS="en"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 aac acl acpi alsa apache2 arts asf audiofile berkdb bluetooth branding bzip2 cairo cdparanoia cdr cli cracklib crypt cups dbus dell doc dri dvd dvdr dvdread eds emboss encode esd evo fam ffmpeg firefox flac font-server fortran gdbm gif gpm gstreamer gtk hal i8x0 iconv ipv6 isdnlog jpeg kde ldap libnotify mad midi mikmod mmx mozbranding mp3 mpeg mudflap ncurses nls nptl nptlonly nsplugin nvidia ogg opengl openmp pam pcre pdf perl png ppds pppd python qt qt3 qt3support qt4 quicktime readline reflection samba sdl session spell spl sse sse2 ssl startup-notification subversion svg sysfs tcpd theora threads tiff truetype unicode usb vcd vorbis wifi win32codecs x86 xcomposite xine xml xorg xv zeroconf zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" USERLAND="GNU" VIDEO_CARDS="vesa fbdev nvidia"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Comment 3 David Flogeras 2008-11-09 11:56:08 UTC
MANPATH=/home/dave//.gentoo/java-config-2/current-user-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/i686-pc-linux-gnu/2.18/man:/usr/share/gcc-data/i686-pc-linux-gnu/4.1.2/man:/opt/sun-jdk-1.4.2.18/man:/etc/java-config/system-vm/man/:/usr/kde/3.5/share/man:/usr/qt/3/doc/man:/opt/vmware/player/man
KDE_MULTIHEAD=false
SSH_AGENT_PID=8535
DM_CONTROL=/var/run/xdmctl
TERM=xterm
SHELL=/bin/bash
XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic
GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/dave//.gtkrc-2.0:/home/dave/.kde3.5/share/config/gtkrc-2.0
GTK_RC_FILES=/etc/gtk/gtkrc:/home/dave//.gtkrc:/home/dave/.kde3.5/share/config/gtkrc
GS_LIB=/home/dave//.fonts
WINDOWID=52428805
QTDIR=/usr/qt/3
ANT_HOME=/usr/share/ant
KDE_FULL_SESSION=true
SGML_CATALOG_FILES=/etc/sgml/sgml-docbook.cat:/etc/sgml/openjade-1.3.2.cat:/etc/sgml/xml-docbook-4.4.cat:/etc/sgml/sgml-ent.cat:/etc/sgml/xml-simple-docbook-1.0.cat:/etc/sgml/xml-simple-docbook-4.1.2.4.cat:/etc/sgml/xml-docbook-4.1.2.cat:/etc/sgml/sgml-docbook-3.0.cat:/etc/sgml/sgml-docbook-3.1.cat:/etc/sgml/sgml-docbook-4.0.cat:/etc/sgml/sgml-docbook-4.1.cat:/etc/sgml/sgml-docbook-4.2.cat:/etc/sgml/sgml-docbook-4.4.cat:/etc/sgml/sgml-lite.cat:/etc/sgml/dsssl-docbook-stylesheets.cat
USER=dave
LD_LIBRARY_PATH=/opt/tools/i686/lib/debug:/tmp/trunk/lib/debug/i686/:/opt/tools/i686/lib
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:
GDK_USE_XFT=1
SSH_AUTH_SOCK=/tmp/ssh-spPRfU8534/agent.8534
KDEDIR=/usr/kde/3.5
SESSION_MANAGER=local/adamwest.akoostix.com:/tmp/.ICE-unix/8579
VMHANDLE=sun-jdk-1.4
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/env.d/java/ /etc/udev/rules.d /etc/fonts/fonts.conf /etc/gconf /etc/terminfo /etc/ca-certificates.conf /etc/texmf/web2c /etc/revdep-rebuild
FLTK_DOCDIR=/usr/share/doc/fltk-1.1.7-r2/html
XDG_CONFIG_DIRS=/usr/kde/3.5/etc/xdg
KONSOLE_DCOP=DCOPRef(konsole-25072,konsole)
DESKTOP_SESSION=kde-3.5
PATH=/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.1.2:/opt/sun-jdk-1.4.2.18/bin:/opt/sun-jdk-1.4.2.18/jre/bin:/opt/sun-jdk-1.4.2.18/jre/javaws:/usr/kde/3.5/bin:/usr/qt/3/bin:/usr/games/bin:/opt/vmware/player/bin:/opt/vmware/server/console/bin
KONSOLE_DCOP_SESSION=DCOPRef(konsole-25072,session-1)
DISTCC_LOG=
PWD=/home/dave
JAVA_HOME=/home/dave//.gentoo/java-config-2/current-user-vm
GENERATION=2
EDITOR=/usr/bin/vim
JAVAC=/home/dave//.gentoo/java-config-2/current-user-vm/bin/javac
LANG=en_US
KDE_SESSION_UID=1000
QMAKESPEC=linux-g++
KDEDIRS=/usr:/usr/local:/usr/kde/3.5
DISTCC_VERBOSE=0
DCCC_PATH=/usr/lib/distcc/bin
JDK_HOME=/home/dave//.gentoo/java-config-2/current-user-vm
SHLVL=2
HOME=/home/dave/
CG_COMPILER_EXE=/usr/bin/cgc
XCURSOR_THEME=default
PYTHONPATH=/usr/lib/portage/pym
LESS=-R -M --shift 5
LOGNAME=dave
GCC_SPECS=
CVS_RSH=ssh
XDG_DATA_DIRS=/usr/share:/usr/kde/3.5/share:/usr/local/share
CLASSPATH=.
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-CqRQjhqvc8,guid=7a5800475279a09cd69842144915c34b
PKG_CONFIG_PATH=/usr/qt/3/lib/pkgconfig
LESSOPEN=|lesspipe.sh %s
INFOPATH=/usr/share/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.18/info:/usr/share/gcc-data/i686-pc-linux-gnu/4.1.2/info
DISPLAY=:0.0
QT_PLUGIN_PATH=/opt/tools/i686/lib
OPENGL_PROFILE=nvidia
LDPATH=/usr/kde/3.5/lib:
CONFIG_PROTECT=/usr/kde/3.5/share/config /usr/kde/3.5/env /usr/kde/3.5/shutdown /usr/share/config
COLORTERM=
_=/usr/bin/env
Comment 4 David Flogeras 2008-11-09 12:04:02 UTC
Also please note, I do have unicode support on (well, my profile does, I think its default).

I did however try recompiling ncurses with "-unicode", then recompiled python to update its readline.so plugin, but the problem persisted.  I have since recompiled both back with unicode.
Comment 5 Wormo (RETIRED) gentoo-dev 2008-11-09 23:52:51 UTC
One more thing, could you attach one of those strace logs you were looking at? gzip it before attaching if it's pretty large.
Comment 6 David Flogeras 2008-11-10 00:40:16 UTC
Created attachment 171252 [details]
strace output of python executing 'import readline'

see line 729
Comment 7 David Flogeras 2009-02-24 00:10:23 UTC
Tried again with readline-6.0, re-emerged python to link the readline plugin against the new readline.  has the same results
Comment 8 David Flogeras 2009-03-03 22:45:20 UTC
I seem to have found a workaround accidentally.

I was building an embedded gentoo machine, and noticed this behaviour was not present.  I thought it might have something to do with the fact that I was running python in a VT, versus a konsole or some such.  So I tried on one of my affected boxes, and sure enough if I run the following in a VT (and pipe to a file), I don't see the weird unicode before the hello string:

#!/usr/bin/python
import readline
print 'hello'

But if I go back to a konsole, I do get the weirdness.

I looked at the difference in environments and noticed in a VT, TERM=linux, and in konsole TERM=xterm.  So from a konsole, I re-ran as 'TERM=linux ./test.py' and I no longer got the garbage.

Hopefully a dev can make something of this.  I don't like the solution of just randomly changing my TERM variable, because I don't know what else this affects.
Comment 9 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2009-03-23 22:48:26 UTC
This is definetly beyond python/perl.

Minimal testcases:
$ TERM=xterm python -c 'import readline' |xxd
0000000: 1b5b 3f31 3033 3468                      .[?1034h
$ emerge dev-perl/Term-ReadLine-Gnu
$ TERM=xterm perl -e 'use Term::ReadLine; $term=Term::ReadLine->new("");' |xxd
0000000: 1b5b 3f31 3033 3468                      .[?1034h
Comment 10 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2009-03-23 22:57:25 UTC
tracing the terminfo.src,
it's coming from the definition of xterm-basic, specifically either the rmm or smm settings:
rmm=\E[?1034l
smm=\E[?1034h

rmm and smm are meta_off and meta_on respectively, this means that characters are a full 8-bits wide, instead of 7-bits.

Can somebody grab the xterm-basic and xterm-new definitions from Ubuntu?
Comment 11 David Flogeras 2009-03-31 20:58:52 UTC
Created attachment 186892 [details]
required definitions from ubuntu

I have attached (hopefully) the requested info from Ubuntu.  I have little to no working knowledge of readline/ncurses, but I want to help progress this bug.

What I have done, is grabbed the readline and Ubuntu patches (Hardy, I know this bug is not present on that release).  I unpacked the original readline, and applied the patchset.  Next I did a find/grep for any files including the xterm-basic/xterm-new and created a tarball with what I think are the definitions.  Please let me know if these are not enough.
Comment 12 David Flogeras 2009-03-31 21:00:41 UTC
pardon me, the files I attached came from the Ubuntu patched ncurses, not readline.  shouldn't post when tired.
Comment 13 Thomas Dickey 2009-09-22 10:50:02 UTC
ultimately this is a bug in libreadline5, which
turns the feature on unconditionally.  Analogously,
if it decided to enable the xterm mouse feature
just because it was there it would also be a nuisance
(the latter is a defect of gpm).
Comment 14 David Flogeras 2016-01-01 20:36:28 UTC
Can no longer reproduce this, unless there is value in keeping it open, shall I close it?  It's ancient
Comment 15 SpanKY gentoo-dev 2016-01-02 03:28:14 UTC
no worries.  thanks for following up.