| Summary: | www-client/firefox-58.0 - .../work/firefox-58.0/python/mozbuild/mozbuild/controller/building.py", line 548, in emit self.fh.write(msg) UnicodeEncodeError: 'ascii' codec can't encode characters in position 156-157: ordinal not in range(128) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Erich Seifert <eseifert> |
| Component: | Current packages | Assignee: | Mozilla Gentoo Team <mozilla> |
| Status: | RESOLVED INVALID | ||
| Severity: | normal | CC: | aballier, adamcarter3, chris, gentoo, kaikaikai, petr.pisar, progmachine, tsmksubc |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=727110 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | www-client:firefox-58.0:20180129-130029.log | ||
|
Description
Erich Seifert
2018-01-29 18:06:29 UTC
Exception in thread ProcessReader:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 986, in _read
callback(line.rstrip())
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 904, in __call__
e(*args, **kwargs)
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/python/mach/mach/mixin/process.py", line 86, in handleLine
line_handler(line)
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/python/mozbuild/mozbuild/controller/building.py", line 720, in on_line
self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/python/mach/mach/mixin/logging.py", line 54, in log
extra={'action': action, 'params': params})
File "/usr/lib64/python2.7/logging/__init__.py", line 1231, in log
self._log(level, msg, args, **kwargs)
File "/usr/lib64/python2.7/logging/__init__.py", line 1286, in _log
self.handle(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 1296, in handle
self.callHandlers(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 1336, in callHandlers
hdlr.handle(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 759, in handle
self.emit(record)
File "/var/tmp/portage.notmpfs/portage/www-client/firefox-58.0/work/firefox-58.0/python/mozbuild/mozbuild/controller/building.py", line 548, in emit
self.fh.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 156-157: ordinal not in range(128)
Could this be connected to FEATURES="distcc"? (In reply to Torsten Kurbad from comment #2) > Could this be connected to FEATURES="distcc"? Yes, thanks for the hint. It seems to be the case. In the meantime an update to www-client/firefox-58.0.1 was released, but it showed the same problems. After emerging without distcc, fiefox was built and installed without issues. (In reply to Erich Seifert from comment #3) > (In reply to Torsten Kurbad from comment #2) > > Could this be connected to FEATURES="distcc"? > > Yes, thanks for the hint. It seems to be the case. In the meantime an update > to www-client/firefox-58.0.1 was released, but it showed the same problems. > After emerging without distcc, fiefox was built and installed without issues. FEATURES="-distcc" worked for me too when building www-client/firefox-58.0.1. If there was a PMS-legal way to disable or restrict distcc in mozilla packages I would do it. I haven't had a reliable distcc-based build in many versions, and although they do often compile "fine" with local retries there are occasions that I've seen random runtime failures that don't occur with a local-only build. I recommend not using FEATURES="distcc" for firefox, thunderbird, and seamonkey. You can disable features on a per-package basis using /etc/portage/package.env (google for instructions on setting per-package environment variables) (In reply to Ian Stakenvicius from comment #5) > If there was a PMS-legal way to disable or restrict distcc in mozilla > packages I would do it. I haven't had a reliable distcc-based build in many > versions, and although they do often compile "fine" with local retries there > are occasions that I've seen random runtime failures that don't occur with a > local-only build. > > I recommend not using FEATURES="distcc" for firefox, thunderbird, and > seamonkey. You can disable features on a per-package basis using > /etc/portage/package.env (google for instructions on setting per-package > environment variables) perhaps: if has "distcc" $FEATURES ; then # fatal die "..." # non-fatal ewarn "..." fi in pkg_pretend; but i am not sure whether the features variable actually contains that information within the scope of the ebuild. (In reply to Sven B. from comment #6) > perhaps: > > if has "distcc" $FEATURES ; then > # fatal > die "..." > # non-fatal > ewarn "..." > fi > > in pkg_pretend; but i am not sure whether the features variable actually > contains that information within the scope of the ebuild. Although last I checked it does when portage is the package manager, this is an undocumented feature that just happens to be true until now. It's not PMS compliant and so it can't be used in the gentoo repo. *** Bug 670662 has been marked as a duplicate of this bug. *** A post on f.g.o made me wonder... are the LANG_* variables exported to the helper hosts, such that when gcc returns error messages back to the machine requesting services, that their locale messages match... Or should all helper machines always return the 'C' locale? Okay, I found on *systemd* distcc helper machines: Forcing LANG to be unset seems to make distcc work better for firefox, or at least let it match OpenRC distcc helpers as OpenRC helper boxes do not set LANG by default. Workaround-hack-maybe should be permanent: In the supplied file: /etc/systemd/system/distccd.service.d, add: Environment="LANG=''" to the [Service] section, then restart distccd: # systemctl daemon-reload; systemctl restart distccd Then see if distcc works better. For me, this finally lets firefox build fine with the distcc feature enabled in firefox. Chalk another issue to systemd issues^H^H^H^H^H^H^H differences (though perhaps this is distcc or firefox build oversight that distcc behaves differently depending on locale.) I take that back, I found one of my OpenRC boxes with LANG set, so this helper machine also sets LANG. I forgot that I had tried distcc with this OpenRC box by itself and it likewise failed in the same manner. The assumed workaround to fix this is to add LANG="" at the end of /etc/conf.d/distccd , for at least this OpenRC box [untested as of yet, but verified the environment no longer sets it to en_US.utf8 where it had been before.] So we finally have confirmation that the famous
> UnicodeEncodeError: 'ascii' codec can't encode characters in position 156-157: ordinal not in range(128)
bug (see bugs like 666948) is really caused by different LANG= settings across distcc build cluster? Thanks!
Yeah I probably should have wrote this in the other bug report (I found this one first...) but likely they may be one and the same "bug". Need others to verify this workaround is usable. I just verified that clearing environment variable LANG for my other OpenRC distccd box allows firefox to build properly with FEATURES=distcc for a different machine. Note that this problem will also manifest with distcc running in loopback mode (localhost) which really baffled me. However perhaps the correct solution is for distcc to also pass LANG* env variables to the helpers? Anyway, thanks to fedeliallalinea for suggesting the idea to find this workaround (as well as finding that Firefox forces LANG to C in its build scripts). (In reply to Ben from comment #11) > The assumed workaround to fix this is to add > > LANG="" > > at the end of /etc/conf.d/distccd Sure that is sufficient ? I use OpenRC. I added LANG="" at the end of all my /etc/conf.d/distccd I restarted all my distccd But I always have the error : UnicodeEncodeError: 'ascii' codec can't encode characters in position 147-148: ordinal not in range(128) I don't have error with FEATURES="-distcc" Not sure if I have anything else set, but I just had another successful run. Perhaps it's www-client/firefox-60.3.0-r1, not sure if my observations apply to other versions. Without distcc: (1.6GHz x86 single core laptop, no helpers) firefox: Fri Nov 9 09:49:35 2018: 30686 seconds With distcc: (1.6GHz x86 single core laptop with -j8 -l1.5 (due to past bad experiences) and two x86_64 quad core multilib helpers) firefox: Mon Nov 19 21:09:07 2018: 9812 seconds (In reply to Ben from comment #15) > Not sure if I have anything else set, but I just had another successful run. > Perhaps it's www-client/firefox-60.3.0-r1, not sure if my observations apply > to other versions. > > Without distcc: (1.6GHz x86 single core laptop, no helpers) > firefox: Fri Nov 9 09:49:35 2018: 30686 seconds > > With distcc: (1.6GHz x86 single core laptop with -j8 -l1.5 (due to past bad > experiences) and two x86_64 quad core multilib helpers) > firefox: Mon Nov 19 21:09:07 2018: 9812 seconds I've also had success with 60.2+ and 62+ on distcc, in my case I have a common /etc/locale.gen and the same 'eselect locale' across all systems. I'm not sure how best to deal with this but it certainly does seem to be related to a mismatch in locales or perhaps a lack of support for the locale on the distcc host compared to the client. Which locale was used? I have a feeling only the "C" locale will allow Firefox to compile, mainly because apparently the Firefox build routines hardcode to C. I had another machine (LANG=en_US.utf8) successfully build Firefox with distcc with the helpers (LANG=C) workaround. On all I have : # eselect locale list Available targets for the LANG variable: [1] C [2] en_US [3] en_US.iso88591 [4] en_US.utf8 [5] fr_FR [6] fr_FR@euro [7] fr_FR.iso88591 [8] fr_FR.iso885915@euro * [9] fr_FR.utf8 [10] POSIX [ ] (free form) Could check what locale distcc returns errors from some of your hosts?
Create a file foo.c that is "dirty" to test with, perhaps like this:
---cut---
int goobers(int bar) {
char *foo;
foo=&bar;
return (foo);
}
---cut---
Now, try to compile it:
$ DISTCC_HOSTS=helpermachine distcc gcc -Wall -c foo.c -o foo.o
foo.c: Dans la fonction ‘goobers’:
foo.c:3:4: attention : assignment from incompatible pointer type [-Wincompatible-pointer-types]
foo.c:4:8: attention : return makes integer from pointer without a cast [-Wint-conversion]
See what locale it prints the messages in. I hacked my machine to force LANG=fr_FR.utf8 above. As far as I know, this will fail firefox builds.
Incidentally, I found what is different in en_US.utf8 and C: (Look carefully)
< foo.c: In function ‘goobers’:
> foo.c: In function 'goobers':
This is sufficient to trip up Python if was scraping the output and expecting LANG=C as distcc does not pass LANG.
(In reply to Ben from comment #19) > Now, try to compile it: > > $ DISTCC_HOSTS=helpermachine distcc gcc -Wall -c foo.c -o foo.o In my case : $ DISTCC_HOSTS=192.168.0.11 distcc gcc -Wall -c foo.c -o foo.o foo.c: Dans la fonction « goobers »: foo.c:3:4: warning: affectation depuis un type pointeur incompatible [-Wincompatible-pointer-types] foo.c:4:8: warning: le retour transforme un pointeur en entier sans transtypage [-Wint-conversion] $ DISTCC_HOSTS=serveur2 distcc gcc -Wall -c foo.c -o foo.o foo.c: Dans la fonction « goobers »: foo.c:3:4: warning: affectation depuis un type pointeur incompatible [-Wincompatible-pointer-types] foo.c:4:8: warning: le retour transforme un pointeur en entier sans transtypage [-Wint-conversion] But I don't understand what that means. This means that your machines are still returning something other than LANG=C which means the hack presented didn't work for you... (i.e., /etc/conf.d/distccd didn't get applied to the running instance of distccd.) ok, understood. Thanks Were you able to get distccd to return results in locale 'C' (you should see it returning English results and use the straight single quote marks that match the single quote mark generated by your keyboard) -- and then see if firefox will build with those distccd machines? (In reply to Ben from comment #23) > Were you able to get distccd to return results in locale 'C' (you should see > it returning English results and use the straight single quote marks that > match the single quote mark generated by your keyboard) -- and then see if > firefox will build with those distccd machines? No, sorry. /etc/conf.d/distccd is read. Allow, listen logfile, loglevel directives are well used. But LANG=C or LANG="C" or LANG="", doesn't do anything. I don't really know what to try. (In reply to Christophe PEREZ from comment #24) > (In reply to Ben from comment #23) > > Were you able to get distccd to return results in locale 'C' (you should see > > it returning English results and use the straight single quote marks that > > match the single quote mark generated by your keyboard) -- and then see if > > firefox will build with those distccd machines? > > No, sorry. > > /etc/conf.d/distccd is read. Allow, listen logfile, loglevel directives are > well used. > But LANG=C or LANG="C" or LANG="", doesn't do anything. > > I don't really know what to try. This is a bit more invasive, but try to edit /etc/init.d/distccd and add 'unset LANG' in start() above the start-stop-daemon call, then restart distccd? (In reply to Ian Stakenvicius from comment #25) > This is a bit more invasive, but try to edit /etc/init.d/distccd and add > 'unset LANG' in start() above the start-stop-daemon call, then restart > distccd? root@serveur2 ~ # grep -B1 "start-stop-daemon --start" /etc/init.d/distccd unset LANG start-stop-daemon --start --quiet --exec "${DISTCCD_EXEC}" --user distcc -- \ root@serveur2 ~ # rc-service distccd restart * Stopping distccd ... [ ok ] * Starting distccd ... [ ok ] chris@medion ~/tmp $ DISTCC_HOSTS=serveur2 distcc gcc -Wall -c foo.c -o foo.o foo.c: Dans la fonction « goobers »: foo.c:3:4: warning: affectation depuis un type pointeur incompatible [-Wincompatible-pointer-types] foo.c:4:8: warning: le retour transforme un pointeur en entier sans transtypage [-Wint-conversion] Nothing changed :( What if you explicitly set LANG=C instead of unsetting? I'm not sure how these localizations work, haven't really looked into it mainly because usually the C locale is the same as en_US without the annoying high bit characters ... Also try setting LC_ALL=C as well? (In reply to Ben from comment #27) > What if you explicitly set LANG=C instead of unsetting? Not better > Also try setting LC_ALL=C as well? I should try it before. I have english message, even with LC_ALL=C in /etc/conf.d/distccd : $ DISTCC_HOSTS=serveur2 distcc gcc -Wall -c foo.c -o foo.o foo.c: In function 'goobers': foo.c:3:4: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types] foo.c:4:8: warning: return makes integer from pointer without a cast [-Wint-conversion] But, same error compiling firefox with distcc. Please feel free to reopen and update any bug report that can be duplicated with current esr builds, 60.x. If you feel your feature needs to be re looked at in any of these bugs reopen and update, please attach patches when appropriate. Thank you Mozilla Team I always have the same problem, with all my PC (3), with all firefox version (actually www-client/firefox-60.7.0). I have to compile without distcc each time. |
