Current stable (1.1.1.20060917) and ~x86 (1.1.1.20070114) ebuild fail to install due to encoding error. Seems to be utf8 related. Reproducible: Always Steps to Reproduce: 1. emerge linuxtv-dvb-apps 2. 3. Actual Results: merging fails Expected Results: merging should succeed kohni ~ # emerge --info Portage 2.1.2.2 (default-linux/x86/2006.1, gcc-4.1.1, glibc-2.5-r0, 2.6.19-gentoo-r5 i686) ================================================================= System uname: 2.6.19-gentoo-r5 i686 AMD Athlon(TM) XP 3000+ Gentoo Base System release 1.12.9 Timestamp of tree: Fri, 30 Mar 2007 20:00:09 +0000 dev-java/java-config: 1.3.7, 2.0.31 dev-lang/python: 2.4.3-r4 dev-python/pycrypto: 2.0.1-r5 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.61 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.14 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -mtune=athlon-xp -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.4/env /usr/kde/3.4/share/config /usr/kde/3.4/shutdown /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/java-config/vms/ /etc/php/apache1-php5/ext-active/ /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c" CXXFLAGS="-O2 -mtune=athlon-xp -fomit-frame-pointer -pipe" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig distlocks metadata-transfer sandbox sfperms strict" GENTOO_MIRRORS="http://ftp.easynet.nl/mirror/gentoo" LANG="de_DE.UTF8" LINGUAS="de zh_CN zh_TW ko" MAKEOPTS="-j1" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/home/portage" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="3dfx 3dnow 3dnowext X X509 a52 aac acpi aiglx alsa arts asf audiofile bash-completion berkdb bitmap-fonts bonobo bzip2 cddb cdr cjk cli cracklib crypt cups curl dba dbus divx doc dri dvb dvd dvdr dvdread eds emerald encode exif expat ffmpeg flac foomaticdb fortran gd gdbm gif glitz glut gmedia gmp gstreamer gtk gtkhtml hal iconv idn imagemagick imlib ipv6 isdnlog jack jack-esd java jpeg kde kdexdeltas ladspa lcms libg++ libsamplerate libwww lm_sensors mad mhash midi mjpeg mmx mmxext mng motif mozilla mp3 mpeg mplayer musicbrainz nas ncurses nls nptl nptlonly nsplugin offensive ogg old-crypt openexr opengl pam pcre pdf perl pmount png ppds pppd python qt qt3 qt4 quicktime readline real realmedia recode reflection samba scanner sdl session slang spell spl sse sse2 ssl subtitles subversion svg svga swat tcltk tcpd tetex tiff truetype truetype-fonts type1-fonts unicode usb v4l v4l2 vcd vorbis win32codecs winbind wmf wmp x86 xcomposite xine xml xml2 xorg xprint xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" DVB_CARDS="usb-wt220u" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de zh_CN zh_TW ko" USERLAND="GNU" VIDEO_CARDS="radeon fglrx vesa v4l r128 via" Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS kohni ~ #
Created attachment 115029 [details] Stack trace of installing trial
I cannot reproduce this with x86, nor with ~x86 portage+linuxtv-dvb-apps.
Any idea how to debug this? It might be a bug in portage, so I will try installing ~x86 portage and emergeing once again.
OK, tested. Updating portage does not change anything.
OK, further testing resulted in a utf8-problem in my installation. I need to check that. But the package seems not to be broken. Closing. Sorry for the noise.
Sorry for again making noise: I finally got the package installed. I simply added in the ebuild in src_unpack: # make the french region files use utf8 encoding convmv -f iso8859-15 -t utf8 --notest util/scan/dvb-t/fr-* Thats because two french cities-names are encoded iso8859-15 (the files are located in util/scan/dvb-t/) but my system uses utf8 for every file (defaulted by kernel-config). That cannot be a solution, but as I'm not a developer, I don't know where to fix that. But anyway: as it seems to be a bug in that package to me again, I'll reopen. Feel free to close, if it should be fixed somewhere else or if I'm still missing something and got it completely wrong. Best regards Jan
Can you please attach your kernel configuration. This is the .config file in the kernel directory.
I have LANG=en_US.UTF-8 and linuxtv-dvb-apps-1.1.1.20070114 installs with no problems. (In reply to comment #0) > PORTAGE_TMPDIR="/home/portage" Is there anything about the filesystem mounted on /home/portage that makes it different from /usr where the file was being merged when the error occurred?
(In reply to comment #7) > Can you please attach your kernel configuration. This is the .config file in > the kernel directory. Only if I want it to be there. :) I'll check and post as soon as I have acces to the machine again.
Problem is still existent in 1.1.1.20070114 for me. I'll attach the output of 'zcat /proc/config.gz > config'. Regards Jan
Created attachment 117828 [details] Kernelconfiguration
I forgot to mention: My proposed fix works also for this version.
(In reply to comment #8) > (In reply to comment #0) > > PORTAGE_TMPDIR="/home/portage" > > Is there anything about the filesystem mounted on /home/portage that makes it > different from /usr where the file was being merged when the error occurred? /home is xfs, /usr is jfs mounted with iocharset=utf8. Could this causing that error?
(In reply to comment #13) > /home is xfs, /usr is jfs mounted with iocharset=utf8. Could this causing that > error? Yes, apparently. According to `man mount` the default is to do no conversion (when iocharset is not specified). It might just work if you remove that mount option.
(In reply to comment #6) > I finally got the package installed. I simply added in the ebuild in > src_unpack: > # make the french region files use utf8 encoding > convmv -f iso8859-15 -t utf8 --notest util/scan/dvb-t/fr-* > > Thats because two french cities-names are encoded iso8859-15 (the files are > located in util/scan/dvb-t/) but my system uses utf8 for every file (defaulted > by kernel-config). > > That cannot be a solution, but as I'm not a developer, I don't know where to > fix that. The encoding may vary from one system to another, so hardcoded utf8 conversion won't necessarily work for everyone. The file names should be encoded according to whatever the current locale settings are.
Same here. On my amd64 (no multilib) system version 1.1.1.20070114 fails to build, too. It does not matter wether both LC_ALL or LANG are unset or set to de_DE.utf8 When PORTAGE_TMPDIR is on a reiser3 System, it builds, but fails during installation: >>> /usr/share/dvb/scan/dvb-t/se-Goteborg_Brudaremossen >>> /usr/share/dvb/scan/dvb-t/fi-Utsjoki >>> /usr/share/dvb/scan/dvb-t/fr-Bergerac >>> /usr/share/dvb/scan/dvb-t/se-Bollebygd >>> /usr/share/dvb/scan/dvb-t/uk-DoverB Traceback (most recent call last): File "/usr/bin/emerge", line 5676, in ? retval = emerge_main() File "/usr/bin/emerge", line 5669, in emerge_main myopts, myaction, myfiles, spinner) File "/usr/bin/emerge", line 5136, in action_build retval = mergetask.merge(pkglist, favorites, mtimedb) File "/usr/bin/emerge", line 3273, in merge prev_mtimes=ldpath_mtimes) File "/usr/lib64/portage/pym/portage.py", line 3872, in doebuild vartree=vartree, prev_mtimes=prev_mtimes) File "/usr/lib64/portage/pym/portage.py", line 4086, in merge mydbapi=mydbapi, prev_mtimes=prev_mtimes) File "/usr/lib64/portage/pym/portage.py", line 8047, in merge cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes) File "/usr/lib64/portage/pym/portage.py", line 7641, in treewalk if self.mergeme(srcroot,destroot,outfile,secondhand,"",cfgfiledict,mymtime): File "/usr/lib64/portage/pym/portage.py", line 7919, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage.py", line 7919, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage.py", line 7919, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage.py", line 7919, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage.py", line 7919, in mergeme join(offset, x), cfgfiledict, thismtime): File "/usr/lib64/portage/pym/portage.py", line 7802, in mergeme mydmode = os.lstat(mydest).st_mode OSError: [Errno 22] Invalid argument: '/usr/share/dvb/scan/dvb-t/fr-Besan\xe7on' Putting PORTAGE_TMPDIR on a jfs system with iocharset=utf8 it does not even unpack: >>> Unpacking source... >>> Unpacking linuxtv-dvb-apps-20070114.tar.bz2 to /home/tmp/portage/media-tv/linuxtv-dvb-apps-1.1.1.20070114/work tar: linuxtv-dvb-apps-20070114/util/scan/dvb-t/fr-Besan\347on: Kann open nicht ausführen: Das Argument ist ungültig tar: linuxtv-dvb-apps-20070114/util/scan/dvb-t/fr-Alen\347on: Kann open nicht ausführen: Das Argument ist ungültig tar: Fehler beim Beenden, verursacht durch vorhergehende Fehler. !!! ERROR: media-tv/linuxtv-dvb-apps-1.1.1.20070114 failed. Call stack: ebuild.sh, line 1638: Called dyn_unpack ebuild.sh, line 763: Called qa_call 'src_unpack' ebuild.sh, line 44: Called src_unpack linuxtv-dvb-apps-1.1.1.20070114.ebuild, line 25: Called unpack 'linuxtv-dvb-apps-20070114.tar.bz2' ebuild.sh, line 439: Called die !!! failure unpacking linuxtv-dvb-apps-20070114.tar.bz2 !!! If you need support, post the topmost build error, and the call stack if relevant. !!! A complete build log is located at '/var/log/portage/media-tv:linuxtv-dvb-apps-1.1.1.20070114:20070913-205302.log'.
As long as the ebuild installs files names in strange encodings, there are 2 possible workarounds: 1) Use INSTALL_MASK to exclude the ones with the strange filenames. 2) Disable the iocharset mount option. I suspect that by disabling the iocharset mount option you won't really loose any functionality. For example, ext3 has no such mount option, yet utf8 works just fine because it's just treated as raw bytes with no attempt to do conversion.
1) The reiserfs partition has no iocharset option, thats why I posted the difference, so workaround 2 is void. It was just to add some more information to the problem, just in case someone cares. 2) Being off topic, but FYI: the iocharset option is absolutely crucial, as you can see exactly with this very problem. The iocharset option is the only way to fight that i18n mess that linux has. The question is not, wether ext3 is able to save utf8 files, but that it will also accept non utf8 files if the app above creates its own character mess, which hell of a lot of applications do. As long as there is no consistent way of character treatment among the whole system, it is an absolute must to have at least the filesystem enforce some standard on which one can rely.
Still valid with 1.1.1.20100223?
Since bug 382199, portage forces UTF-8 encoding regardless of locale. When files with other encodings are detected, it triggers a QA Notice, and the files are automatically renamed to be valid UTF-8 (without any character set conversion).
