Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 587606 - app-portage/gentoolkit- "equery uses" with python2.7 cannot handle non-ascii characters in USE descriptions
Summary: app-portage/gentoolkit- "equery uses" with python2.7 cannot handle non...
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage Tools Team
Keywords: InVCS
: 589362 589388 591256 (view as bug list)
Depends on:
Blocks: 346443
  Show dependency tree
Reported: 2016-06-30 18:15 UTC by Ben Kohler
Modified: 2017-10-12 18:27 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Ben Kohler gentoo-dev 2016-06-30 18:15:21 UTC
There is a non-ascii character in the new profiles/desc/l10n.desc file:

nb - Norwegian Bokmål

This appears to be making equery u explode when python2.7 is used.  To reproduce:

EPYTHON="python2.7" equery uses texlive

[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for app-text/texlive-2015:
 U I
 + + X                 : Add support for X11
 - - cjk               : Add support for Multi-byte character languages (Chinese, Japanese,
 - - context           : Add support for the ConTeXt format (dev-texlive/texlive-context)
 - - detex             : Add support for dev-tex/detex, a filter program that removes the
                         LaTeX (or TeX) control sequences
 - - dvi2tty           : Add support for dev-tex/dvi2tty to preview dvi-files on text-only
 - - epspdf            : Add support for app-text/epspdf, a tool for converting [e]ps files
                         to pdf.
 - - extra             : Add support for extra TeXLive packages
 - - games             : Add typesetting support for games (chess, etc.)
 - - graphics          : Add support for several graphics packages (pgf, tikz,...)
 - - humanities        : Add LaTeX support for the humanities
 - - jadetex           : Add support for app-text/jadetex (for processing tex files produced
                         by the TeX backend of Jade)
 - - l10n_af           : Afrikaans
 - - l10n_ar           : Arabic
 - - l10n_as           : Assamese
 - - l10n_bg           : Bulgarian
 - - l10n_bn           : Bengali
 - - l10n_br           : Breton
 - - l10n_ca           : Catalan
 - - l10n_cs           : Czech
 - - l10n_cy           : Welsh
 - - l10n_da           : Danish
 - - l10n_de           : German
 - - l10n_el           : Modern Greek
 + + l10n_en           : English
 - - l10n_en-GB        : English (United Kingdom)
 - - l10n_eo           : Esperanto
 - - l10n_es           : Spanish
 - - l10n_et           : Estonian
 - - l10n_eu           : Basque
 - - l10n_fa           : Persian
 - - l10n_fi           : Finnish
 - - l10n_fr           : French
 - - l10n_ga           : Irish
 - - l10n_gl           : Galician
 - - l10n_gu           : Gujarati
 - - l10n_he           : Hebrew
 - - l10n_hi           : Hindi
 - - l10n_hr           : Croatian
 - - l10n_hsb          : Upper Sorbian
 - - l10n_hu           : Hungarian
 - - l10n_hy           : Armenian
 - - l10n_ia           : Interlingua
 - - l10n_id           : Indonesian
 - - l10n_is           : Icelandic
 - - l10n_it           : Italian
 - - l10n_ja           : Japanese
 - - l10n_kn           : Kannada
 - - l10n_ko           : Korean
 - - l10n_la           : Latin
 - - l10n_lo           : Lao
 - - l10n_lt           : Lithuanian
 - - l10n_lv           : Latvian
 - - l10n_ml           : Malayalam
 - - l10n_mn           : Mongolian
 - - l10n_mr           : Marathi
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/equery", line 38, in <module>
  File "/usr/lib64/python2.7/site-packages/gentoolkit/equery/", line 357, in main
  File "/usr/lib64/python2.7/site-packages/gentoolkit/equery/", line 310, in main
  File "/usr/lib64/python2.7/site-packages/gentoolkit/equery/", line 118, in display_useflags
    desc = twrap.fill(desc)
  File "/usr/lib64/python2.7/", line 338, in fill
    return "\n".join(self.wrap(text))
  File "/usr/lib64/python2.7/", line 329, in wrap
    return self._wrap_chunks(chunks)
  File "/usr/lib64/python2.7/site-packages/gentoolkit/", line 95, in _wrap_chunks
    lines.append(indent + ''.join(cur_line))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)
ben@ris-ben ~ $
Comment 1 Ben Kohler gentoo-dev 2016-06-30 18:16:39 UTC
FYI I have seen bug 498 and bug 566504 but I don't think this is quite the same thing, this bug triggers even on a fully utf8-enabled system, and no mixed locale components.

emerge --info

Portage 2.3.0 (python 3.4.4-final-0, default/linux/amd64/13.0/desktop/plasma, gcc-5.4.0, glibc-2.23-r2, 4.6.2-gentoo x86_64)
System uname: Linux-4.6.2-gentoo-x86_64-Intel-R-_Core-TM-_i5-3570_CPU_@_3.40GHz-with-gentoo-2.2
KiB Mem:    16295504 total,   5207880 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Thu, 30 Jun 2016 15:32:58 +0000
sh bash 4.3_p46
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          4.3_p46::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.24.0-r1::gentoo
dev-lang/python:          2.7.11-r2::gentoo, 3.4.4::gentoo
dev-util/cmake:           3.5.2-r1::gentoo
sys-apps/baselayout:      2.2-r1::gentoo
sys-apps/openrc:          0.21.1::gentoo
sys-apps/sandbox:         2.10-r2::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r2::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.12.6-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            5.4.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.6::gentoo (virtual/os-headers)
sys-libs/glibc:           2.23-r2::gentoo

    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://
    priority: -1000
    sync-rsync-extra-opts: --exclude=/.git

    location: /mnt/btr/iamben-overlay
    sync-type: git
    masters: gentoo

Installed sets: @system
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA FraunhoferFDK google-chrome Intel-SDP googleearth Oracle-BCLA-JavaSE dlj-1.1 PUEL AdobeFlash-11.x"
CFLAGS="-O2 -pipe -march=native"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/xdg/autostart/"
CXXFLAGS="-O2 -pipe -march=native"
EMERGE_DEFAULT_OPTS="--autounmask=n -j2 --with-bdeps=y"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
USE="X a52 aac acl acpi alsa amd64 berkdb branding bzip2 cairo caps cdda cdr cli consolekit cracklib crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam ffmpeg filecaps firefox flac fortran gdbm gif glamor gpm gtk gtk3 iconv icu ipv6 jpeg kde kipi lcms libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt3support qt4 qt5 readline samba sdl seccomp session spell sse sse2 ssl startup-notification svg tcpd theora threads tiff truetype udev udisks unicode upower usb vaapi vorbis vpx widgets x264 xattr xcb xcomposite xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_4" QEMU_SOFTMMU_TARGETS="i386 x86_64" RUBY_TARGETS="ruby20 ruby21 ruby22" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Comment 2 Ben Kohler gentoo-dev 2016-06-30 18:17:07 UTC
Oops I meant bug 498748 on that previous comment
Comment 3 Paul Varner (RETIRED) gentoo-dev 2016-06-30 20:46:05 UTC
I have pushed the fix for this specific issue to git

It can be tested with gentoolkit-9999
Comment 4 Ben Kohler gentoo-dev 2016-06-30 21:05:03 UTC
Confirmed working in 9999, thanks!
Comment 5 Paul Varner (RETIRED) gentoo-dev 2016-06-30 23:39:52 UTC
I just pushed more changes that fixes this in a different way and hopefully gets rid of all of the UnicodeDecodeErrors still lurking in the code.  My testing was successful, but when you get a chance can you retest with gentoolkit-9999
Comment 6 Ben Kohler gentoo-dev 2016-07-01 12:53:38 UTC
Retested the original case, and all other cases I can think of (python2 utf8, python2 no-utf8, python3 utf8, python3 no-utf8), all looks good
Comment 7 Ortwin Glueck 2016-07-06 20:11:45 UTC
yes, this fixes this quite prominent one:
equery u kde-apps/kde4-l10n
Comment 8 Paul Varner (RETIRED) gentoo-dev 2016-07-22 14:26:47 UTC
*** Bug 589362 has been marked as a duplicate of this bug. ***
Comment 9 Alex Xu (Hello71) 2016-07-22 19:40:22 UTC
*** Bug 589388 has been marked as a duplicate of this bug. ***
Comment 10 Paul Varner (RETIRED) gentoo-dev 2016-07-22 20:49:57 UTC
This was released in gentoolkit-0.3.2
Comment 11 Anthony Staines 2016-08-10 14:40:14 UTC
works well for me
thank you!
Comment 12 Alex Xu (Hello71) 2016-08-15 20:09:43 UTC
*** Bug 591256 has been marked as a duplicate of this bug. ***
Comment 13 Till Schäfer 2016-09-16 13:01:10 UTC
Works here too.
-> This bug should be closed then, or am I wrong?
Comment 14 Raymond Jennings 2016-09-17 07:05:57 UTC
(In reply to Till Schäfer from comment #13)
> Works here too.
> -> This bug should be closed then, or am I wrong?

That depends, is 0.3.2 in the tree yet?

It appears so.
Comment 15 Paul Varner (RETIRED) gentoo-dev 2016-09-19 20:02:07 UTC
(In reply to Till Schäfer from comment #13)
> Works here too.
> -> This bug should be closed then, or am I wrong?

This still affects stable users, so not yet.