Running a x86 stable: sys-libs/ncurses-5.6-r2 sys-libs/readline-5.2_p12-r1 dev-lang/python-2.5.2-r7 If I execute 'import readline' inside of a python script, the sequence "\33[?1034h" is written to stdout (I verified using strace). I believe it is coming from the ncurses library. I grepped through its source and found \E[?1034h on line 3148 of misc/terminfo.src. I have no idea what that does, but its the same byte-sequence. I have tried this on 3 different gentoo boxes (one amd64). same behaviour. I tried on an Ubuntu box, it does not display this behaviour. Reproducible: Always Steps to Reproduce: 1. put 'import readline' into a file test.py 2. strace -o trace.txt python test.py, look for the sequency 1034h 3. alternatively python test.py > output.txt, open in vim, see an unexpected char Actual Results: Strange unicode character written to stdout Expected Results: Nothing written to stdout
I am unable to reproduce this on an amd64 box with matching versions of the listed packages. Please post 'emerge --info' output and also output from 'env' on one of your affected boxes.
Portage 2.1.4.5 (default/linux/x86/2008.0/desktop, gcc-4.1.2, glibc-2.6.1-r0, 2.6.25-gentoo-r9 i686) ================================================================= System uname: 2.6.25-gentoo-r9 i686 Genuine Intel(R) CPU T2500 @ 2.00GHz Timestamp of tree: Sat, 08 Nov 2008 13:01:01 +0000 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled] app-shells/bash: 3.2_p33 dev-java/java-config: 1.3.7, 2.1.6 dev-lang/python: 2.5.2-r7 dev-python/pycrypto: 2.0.1-r6 dev-util/cmake: 2.4.6-r1 sys-apps/baselayout: 1.12.11.1 sys-apps/sandbox: 1.2.18.1-r2 sys-devel/autoconf: 2.13, 2.61-r2 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10.1-r1 sys-devel/binutils: 2.18-r3 sys-devel/gcc-config: 1.4.0-r4 sys-devel/libtool: 1.5.26 virtual/os-headers: 2.6.23-r3 ACCEPT_KEYWORDS="x86" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/config" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c /etc/udev/rules.d" CXXFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FEATURES="distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LANG="en_US" LDFLAGS="-Wl,-O1" LINGUAS="en" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa apache2 arts asf audiofile berkdb bluetooth branding bzip2 cairo cdparanoia cdr cli cracklib crypt cups dbus dell doc dri dvd dvdr dvdread eds emboss encode esd evo fam ffmpeg firefox flac font-server fortran gdbm gif gpm gstreamer gtk hal i8x0 iconv ipv6 isdnlog jpeg kde ldap libnotify mad midi mikmod mmx mozbranding mp3 mpeg mudflap ncurses nls nptl nptlonly nsplugin nvidia ogg opengl openmp pam pcre pdf perl png ppds pppd python qt qt3 qt3support qt4 quicktime readline reflection samba sdl session spell spl sse sse2 ssl startup-notification subversion svg sysfs tcpd theora threads tiff truetype unicode usb vcd vorbis wifi win32codecs x86 xcomposite xine xml xorg xv zeroconf zlib" ALSA_CARDS="hda-intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" USERLAND="GNU" VIDEO_CARDS="vesa fbdev nvidia" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
MANPATH=/home/dave//.gentoo/java-config-2/current-user-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/i686-pc-linux-gnu/2.18/man:/usr/share/gcc-data/i686-pc-linux-gnu/4.1.2/man:/opt/sun-jdk-1.4.2.18/man:/etc/java-config/system-vm/man/:/usr/kde/3.5/share/man:/usr/qt/3/doc/man:/opt/vmware/player/man KDE_MULTIHEAD=false SSH_AGENT_PID=8535 DM_CONTROL=/var/run/xdmctl TERM=xterm SHELL=/bin/bash XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/dave//.gtkrc-2.0:/home/dave/.kde3.5/share/config/gtkrc-2.0 GTK_RC_FILES=/etc/gtk/gtkrc:/home/dave//.gtkrc:/home/dave/.kde3.5/share/config/gtkrc GS_LIB=/home/dave//.fonts WINDOWID=52428805 QTDIR=/usr/qt/3 ANT_HOME=/usr/share/ant KDE_FULL_SESSION=true SGML_CATALOG_FILES=/etc/sgml/sgml-docbook.cat:/etc/sgml/openjade-1.3.2.cat:/etc/sgml/xml-docbook-4.4.cat:/etc/sgml/sgml-ent.cat:/etc/sgml/xml-simple-docbook-1.0.cat:/etc/sgml/xml-simple-docbook-4.1.2.4.cat:/etc/sgml/xml-docbook-4.1.2.cat:/etc/sgml/sgml-docbook-3.0.cat:/etc/sgml/sgml-docbook-3.1.cat:/etc/sgml/sgml-docbook-4.0.cat:/etc/sgml/sgml-docbook-4.1.cat:/etc/sgml/sgml-docbook-4.2.cat:/etc/sgml/sgml-docbook-4.4.cat:/etc/sgml/sgml-lite.cat:/etc/sgml/dsssl-docbook-stylesheets.cat USER=dave LD_LIBRARY_PATH=/opt/tools/i686/lib/debug:/tmp/trunk/lib/debug/i686/:/opt/tools/i686/lib LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36: GDK_USE_XFT=1 SSH_AUTH_SOCK=/tmp/ssh-spPRfU8534/agent.8534 KDEDIR=/usr/kde/3.5 SESSION_MANAGER=local/adamwest.akoostix.com:/tmp/.ICE-unix/8579 VMHANDLE=sun-jdk-1.4 PAGER=/usr/bin/less CONFIG_PROTECT_MASK=/etc/env.d/java/ /etc/udev/rules.d /etc/fonts/fonts.conf /etc/gconf /etc/terminfo /etc/ca-certificates.conf /etc/texmf/web2c /etc/revdep-rebuild FLTK_DOCDIR=/usr/share/doc/fltk-1.1.7-r2/html XDG_CONFIG_DIRS=/usr/kde/3.5/etc/xdg KONSOLE_DCOP=DCOPRef(konsole-25072,konsole) DESKTOP_SESSION=kde-3.5 PATH=/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.1.2:/opt/sun-jdk-1.4.2.18/bin:/opt/sun-jdk-1.4.2.18/jre/bin:/opt/sun-jdk-1.4.2.18/jre/javaws:/usr/kde/3.5/bin:/usr/qt/3/bin:/usr/games/bin:/opt/vmware/player/bin:/opt/vmware/server/console/bin KONSOLE_DCOP_SESSION=DCOPRef(konsole-25072,session-1) DISTCC_LOG= PWD=/home/dave JAVA_HOME=/home/dave//.gentoo/java-config-2/current-user-vm GENERATION=2 EDITOR=/usr/bin/vim JAVAC=/home/dave//.gentoo/java-config-2/current-user-vm/bin/javac LANG=en_US KDE_SESSION_UID=1000 QMAKESPEC=linux-g++ KDEDIRS=/usr:/usr/local:/usr/kde/3.5 DISTCC_VERBOSE=0 DCCC_PATH=/usr/lib/distcc/bin JDK_HOME=/home/dave//.gentoo/java-config-2/current-user-vm SHLVL=2 HOME=/home/dave/ CG_COMPILER_EXE=/usr/bin/cgc XCURSOR_THEME=default PYTHONPATH=/usr/lib/portage/pym LESS=-R -M --shift 5 LOGNAME=dave GCC_SPECS= CVS_RSH=ssh XDG_DATA_DIRS=/usr/share:/usr/kde/3.5/share:/usr/local/share CLASSPATH=. DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-CqRQjhqvc8,guid=7a5800475279a09cd69842144915c34b PKG_CONFIG_PATH=/usr/qt/3/lib/pkgconfig LESSOPEN=|lesspipe.sh %s INFOPATH=/usr/share/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.18/info:/usr/share/gcc-data/i686-pc-linux-gnu/4.1.2/info DISPLAY=:0.0 QT_PLUGIN_PATH=/opt/tools/i686/lib OPENGL_PROFILE=nvidia LDPATH=/usr/kde/3.5/lib: CONFIG_PROTECT=/usr/kde/3.5/share/config /usr/kde/3.5/env /usr/kde/3.5/shutdown /usr/share/config COLORTERM= _=/usr/bin/env
Also please note, I do have unicode support on (well, my profile does, I think its default). I did however try recompiling ncurses with "-unicode", then recompiled python to update its readline.so plugin, but the problem persisted. I have since recompiled both back with unicode.
One more thing, could you attach one of those strace logs you were looking at? gzip it before attaching if it's pretty large.
Created attachment 171252 [details] strace output of python executing 'import readline' see line 729
Tried again with readline-6.0, re-emerged python to link the readline plugin against the new readline. has the same results
I seem to have found a workaround accidentally. I was building an embedded gentoo machine, and noticed this behaviour was not present. I thought it might have something to do with the fact that I was running python in a VT, versus a konsole or some such. So I tried on one of my affected boxes, and sure enough if I run the following in a VT (and pipe to a file), I don't see the weird unicode before the hello string: #!/usr/bin/python import readline print 'hello' But if I go back to a konsole, I do get the weirdness. I looked at the difference in environments and noticed in a VT, TERM=linux, and in konsole TERM=xterm. So from a konsole, I re-ran as 'TERM=linux ./test.py' and I no longer got the garbage. Hopefully a dev can make something of this. I don't like the solution of just randomly changing my TERM variable, because I don't know what else this affects.
This is definetly beyond python/perl. Minimal testcases: $ TERM=xterm python -c 'import readline' |xxd 0000000: 1b5b 3f31 3033 3468 .[?1034h $ emerge dev-perl/Term-ReadLine-Gnu $ TERM=xterm perl -e 'use Term::ReadLine; $term=Term::ReadLine->new("");' |xxd 0000000: 1b5b 3f31 3033 3468 .[?1034h
tracing the terminfo.src, it's coming from the definition of xterm-basic, specifically either the rmm or smm settings: rmm=\E[?1034l smm=\E[?1034h rmm and smm are meta_off and meta_on respectively, this means that characters are a full 8-bits wide, instead of 7-bits. Can somebody grab the xterm-basic and xterm-new definitions from Ubuntu?
Created attachment 186892 [details] required definitions from ubuntu I have attached (hopefully) the requested info from Ubuntu. I have little to no working knowledge of readline/ncurses, but I want to help progress this bug. What I have done, is grabbed the readline and Ubuntu patches (Hardy, I know this bug is not present on that release). I unpacked the original readline, and applied the patchset. Next I did a find/grep for any files including the xterm-basic/xterm-new and created a tarball with what I think are the definitions. Please let me know if these are not enough.
pardon me, the files I attached came from the Ubuntu patched ncurses, not readline. shouldn't post when tired.
ultimately this is a bug in libreadline5, which turns the feature on unconditionally. Analogously, if it decided to enable the xterm mouse feature just because it was there it would also be a nuisance (the latter is a defect of gpm).
Can no longer reproduce this, unless there is value in keeping it open, shall I close it? It's ancient
no worries. thanks for following up.
