Bug 657766 – app-emulation/qemu-2.12.0-r1(python3.5) - "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: ordinal not in range(128)" if en_US.UTF-8 locale is not available

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description PrSo 2018-06-10 22:28:44 UTC]

Created attachment 535604 [details]
build log

Hi all.
After recent update there is a problem during compilation of qemu-2.12.0-r1 with python 3.5. It seems that there is something wrong with 'ascii' codec on python side. I have no idea if it is related but I am using Polish-utf8 language set in my environment.
Build log and output of "emerge info" attached.

[Comment 1 PrSo 2018-06-10 22:30:37 UTC]

Created attachment 535606 [details]
output of emerge --info

[Comment 2 Aidan Marks 2018-06-10 22:33:50 UTC]

Same here.  Nothing to do with locale / Polish language.

[Comment 3 Wes 2018-06-11 23:52:07 UTC]

Same problem with python 3.4 and 3.6 by the way, not jot 3.5

[Comment 4 Wes 2018-06-11 23:52:53 UTC]

*not just

[Comment 5 Matthias Maier 2018-06-12 03:02:34 UTC]

Above error message occurs if you try to convert a string to utf-8 while being on a non-utf8 locale. According to [1] the qemu build system has a workaround for exactly that problem. It sets (in Makefile)

  PYTHON_UTF8 = LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 $(PYTHON)

But that only works if the en_US.UTF-8 locale is available...

Can you please try again but generate the en_US.UTF-8 locale first:
 - add "en_US.UTF-8 UTF-8" to /etc/locale.gen
 - run # locale-gen

[1] https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg01123.html

[Comment 6 Wes 2018-06-12 03:05:14 UTC]

can confirm the workaround succeeded for me with all python3 targets

[Comment 7 PrSo 2018-06-12 05:48:38 UTC]

(In reply to Matthias Maier from comment #5)
> Above error message occurs if you try to convert a string to utf-8 while
> being on a non-utf8 locale. According to [1] the qemu build system has a
> workaround for exactly that problem. It sets (in Makefile)
> 
>   PYTHON_UTF8 = LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 $(PYTHON)
> 
> But that only works if the en_US.UTF-8 locale is available...
> 
> Can you please try again but generate the en_US.UTF-8 locale first:
>  - add "en_US.UTF-8 UTF-8" to /etc/locale.gen
>  - run # locale-gen
> 
> [1] https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg01123.html

Setting en_US.UTF-8 in /etc/locale.gen indeed works, but that's little odd.

If I understood qemu-devel mailing list post correctly the patch sets LANG to "C" overriding en_US.UTF-8, and since mine was set to pl_PL.UTF-8 it seems that in qemu case setting en_US.UTF-8 is a MUST (beside others UTF-8 locales).

Maybe we should add pre-emerge checks to the qemu ebuild, similar to those related to kernel .config file, or try to modify makefile to override other locales too.

Thanks,
Przemek.

[Comment 8 Arfrever Frehtes Taifersar Arahesis 2018-06-12 05:56:05 UTC]

Created attachment 535682 [details, diff]
Patch

QEMU upstream do not know how to use Python properly...
It is easy to use explicit decoding/encoding, which is then independent of locale.

I attach untested patch.

[Comment 9 Arfrever Frehtes Taifersar Arahesis 2018-06-12 05:58:13 UTC]

Please test this patch with LC_ALL etc.

[Comment 10 Arfrever Frehtes Taifersar Arahesis 2018-06-12 05:58:38 UTC]

(In reply to Arfrever Frehtes Taifersar Arahesis from comment #9)
> Please test this patch with LC_ALL etc.

I meant LC_ALL=C

[Comment 11 PrSo 2018-06-12 06:23:23 UTC]

(In reply to Arfrever Frehtes Taifersar Arahesis from comment #8)
> Created attachment 535682 [details, diff] [details, diff]
> Patch
> 
> QEMU upstream do not know how to use Python properly...
> It is easy to use explicit decoding/encoding, which is then independent of
> locale.
> 
> I attach untested patch.

It seems that the patch works without any drawbacks with python 3.5.

I have set back the locale to pl_PL.UTF-8 only in locale.gen, and quemu compiles fine.

Thanks,
Przemek.

[Comment 12 Neil Bothwick 2018-06-12 08:07:15 UTC]

The patch works for me too, thanks!

% grep -v '^#' /etc/locale.gen

en_GB ISO-8859-1
en_GB.UTF-8 UTF-8

[Comment 13 Larry the Git Cow 2018-06-12 15:41:08 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d5141d7cae6dfa31b8b7023b6f984ea4725289ae

commit d5141d7cae6dfa31b8b7023b6f984ea4725289ae
Author:     Matthias Maier <[e-mail]>
AuthorDate: 2018-06-12 15:33:08 +0000
Commit:     Matthias Maier <[e-mail]>
CommitDate: 2018-06-12 15:40:57 +0000

    app-emulation/qemu: fix compilation if en_US.UTF-8 is unavailable
    
    Thanks to Arfrever for the proper patch.
    
    Closes: https://bugs.gentoo.org/657766
    Package-Manager: Portage-2.3.40, Repoman-2.3.9

 app-emulation/qemu/Manifest              | 1 +
 app-emulation/qemu/qemu-2.12.0-r1.ebuild | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)