Hello, Every time I build firefox, now with 94, with PGO I get this type of error messages ``` console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/tmp/portage/www-client/firefox-94.0/temp/tmp4bt0ilyf/search.json.mozlz4", (void 0))) ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost jarlog: /var/tmp/portage/www-client/firefox-94.0/work/firefox_build/jarlog/en-US.log JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"MediaTrackGraph 7f81618cf000 shutdown","state":"(none)","filename":"/var/tmp/portage/www-client/firefox-94.0/work/firefox-94.0/dom/media/MediaTrackGraph.cpp","lineNumber":1809,"stack":"MediaTrackGraph shutdown"}] Barrier: xpcom-will-shutdown JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 5098: NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIWindowMediator.getEnumerator] ``` And it seems that the emerge is idle at these moments (htop reports no core at full/any significant load), and it does not seem to complete with firefox 94 (it's been more than an hour). Emerging with PGO used to take around 25mins on my machine and 8mins without (firefox 94 completes without PGO). I am not sure anymore that PGO is doing its mojo properly given the errors/timeouts that occur so I am opening this bug report to get feedback on the matter. Thanks! Adel
Yeah, I've reproduced this hang too. Can you attach the log up to the point of hang and emerge --info please?
Created attachment 748290 [details] Firefox 94 PGO build log
Created attachment 748293 [details] emerge info
I get the same hang while building with [pgo]. Building with [-pgo] gets me a hangy firefox with messages likes this in journalctl: Nov 03 15:19:04 saga.alfsgard.dk firefox.desktop[584836]: ###!!! [Parent][MessageChannel] Error: (msgtype=0x5B0013,name=PHttpChannel::Msg_DeleteSelf) Channel error: cannot send/recv Nov 03 15:19:04 saga.alfsgard.dk firefox.desktop[584836]: ###!!! [Parent][MessageChannel] Error: (msgtype=0x390078,name=PContent::Msg_DestroyBrowsingContextGroup) Channel error: cannot send/recv Nov 03 15:19:04 saga.alfsgard.dk firefox.desktop[584836]: ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost Nov 03 15:19:04 saga.alfsgard.dk systemd[1329]: app-gnome-firefox-584836.scope: Consumed 10min 24.218s CPU time. Downgrading the following packages fixes the build hang with [pgo] and the runtime hang with [-pgo]: (entries from package.mask) sys-devel/clang:13 sys-devel/llvm:13 sys-devel/clang-runtime:13.0.0 >=sys-devel/llvmgold-13 >=sys-devel/lld-13 =sys-devel/llvmgold-13* >=dev-lang/rust-1.56 >=virtual/rust-1.56 I'm unsure if it's the llvm or the rust downgrade that fixes this. Probably llvm, since that was my main suspect but rust-1.56[system-bootstrap] requires llvm-13 so that got downgraded alongside it.
Yes I confirm that the non-pgo firefox 94 is buggy on my side too, when built with rust 1.56.1 and llvm 13.0.0.
The hangs were present in firefox-93* as well.
Created attachment 748347 [details, diff] ebuild patch to build with older llvm This patch forces firefox to use the llvm and rust versions.
(In reply to Patrick McLean from comment #6) > The hangs were present in firefox-93* as well. I can confirm I had issues with 93 as well. Still building without PGO with 94.
I too can confirm the pgo problem using llvm 13 and rust 1.56.1 with Firefox 93/94. My (-pgo) 94.0.1 build works fine and I don't see the hangy/buggy problems reported in comments 4 and 5
Created attachment 748557 [details] Firefox 94 with PGO build fail log Same thing happens with my build. I also noticed that there's a line with: ACCESS DENIED: open_wr: /dev/nvidiactl Not sure if it has to do anything with the actual fail or is a separate issue.
I downloaded the source code tar of Firefox 94, created the following `mozconfig` file ``` ac_add_options --with-libclang-path=/usr/lib/llvm/13/lib64 ac_add_options MOZ_PGO=1 mk_add_options MOZ_PARALLEL_BUILD=32 export RUSTC_OPT_LEVEL=2 ``` then did this ``` export MACH_USE_SYSTEM_PYTHON=1 export RANLIB=llvm-ranlib export NM=llvm-nm export CXX=clang++ export CC=clang export AR=llvm-ar ./mach build ``` At the end of the first build, a Firefox window opens and runs many tests by opening a tab, running a test, closing the tab. In the middle of it the same error messages show ``` 15:29.10 gmake: Leaving directory '/media/personal/Downloads/firefox-94.0.source/firefox-94.0/obj-x86_64-pc-linux-gnu/instrumented' ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/tmphqqrgys3/search.json.mozlz4", (void 0))) ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost jarlog: /media/personal/Downloads/firefox-94.0.source/firefox-94.0/obj-x86_64-pc-linux-gnu/jarlog/en-US.log ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"MediaTrackGraph 7f4f54c9c400 shutdown","state":"(none)","filename":"/media/personal/Downloads/firefox-94.0.source/firefox-94.0/dom/media/MediaTrackGraph.cpp","lineNumber":1809,"stack":"MediaTrackGraph shutdown"}] Barrier: xpcom-will-shutdown JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 5098: NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIWindowMediator.getEnumerator] ExceptionHandler::GenerateDump cloned child 80708 ExceptionHandler::SendContinueSignalToChild sent continue signal to child ExceptionHandler::WaitForContinueSignal waiting for continue signal... Firefox exited with code 11 during profiling Error running mach: ['build'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You can invoke |./mach busted| to check if this issue is already on file. If it isn't, please use |./mach busted file build| to report it. If |./mach busted| is misbehaving, you can also inspect the dependencies of bug 1543241. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: subprocess.CalledProcessError: Command '['/media/personal/Downloads/firefox-94.0.source/firefox-94.0/obj-x86_64-pc-linux-gnu/instrumented/_virtualenvs/build/bin/python', '/media/personal/Downloads/firefox-94.0.source/firefox-94.0/build/pgo/profileserver.py']' returned non-zero exit status 11. File "/media/personal/Downloads/firefox-94.0.source/firefox-94.0/python/mozbuild/mozbuild/build_commands.py", line 214, in build subprocess.check_call(pgo_cmd, cwd=instr.topobjdir, env=pgo_env) File "/usr/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) ``` But the build does fail after timeout. So it seems to be something not related to the EBUILD itself but something upstream. And given the previous comments, LLVM and Rust seem to not be playing well. I tried PGO with GCC 11.2.0 and the same issue happened, though.
I could reproduce neither the pgo timeout nor hang / issues with the final application, with llvm 13 and rust 1.56.1 The pgo timeout during build seems to be an issue with sandbox (see also bug 821532 ), but I am still puzzled about the runtime hang in comment 4
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=17e0a35b87856e065b94c42bb96c8cfd4286c933 commit 17e0a35b87856e065b94c42bb96c8cfd4286c933 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2021-11-04 12:44:20 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2021-11-04 12:45:04 +0000 www-client/firefox: allow access to GPU for USE=pgo Bug: https://bugs.gentoo.org/821505 Closes: https://bugs.gentoo.org/821526 Package-Manager: Portage-3.0.28, Repoman-3.0.3 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/firefox-91.3.0.ebuild | 28 ++++++++++++++++++++++++++++ www-client/firefox/firefox-94.0.1.ebuild | 28 ++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
(In reply to Patrick McLean from comment #7) > Created attachment 748347 [details, diff] [details, diff] > ebuild patch to build with older llvm > > This patch forces firefox to use the llvm and rust versions. Thanks, tried your patch and tried building FF-94 with older rust 1.55.0 (binary version if that matters). Here's the result. 55:01.12 We know it took a while, but your build finally finished successfully! 55:01.12 If you are building Firefox often, SCCache can save you a lot of time. You can learn more here: https://firefox-source-docs.mozilla.org/setup/configuring_build_options.html#sccache To take your build for a test drive, run: |mach run| For more information on what to do now, see https://firefox-source-docs.mozilla.org/setup/contributing_code.html >>> Source compiled. * ----------------------- SANDBOX ACCESS VIOLATION SUMMARY ----------------------- * LOG FILE: "/tmp/portage/www-client/firefox-94.0/temp/sandbox.log" * VERSION 1.0 FORMAT: F - Function called FORMAT: S - Access Status FORMAT: P - Path as passed to function FORMAT: A - Absolute Path (not canonical) FORMAT: R - Canonical Path FORMAT: C - Command Line F: open_wr S: deny P: /dev/nvidiactl A: /dev/nvidiactl R: /dev/nvidiactl C: /tmp/portage/www-client/firefox-94.0/work/firefox_build/instrumented/dist/firefox/firefox data:text/html,<script>Quitter.quit()</script> -profile /tmp/portage/www-client/firefox-94.0/temp/tmpxi7sq84w F: open_wr S: deny P: /dev/nvidiactl A: /dev/nvidiactl R: /dev/nvidiactl C: /tmp/portage/www-client/firefox-94.0/work/firefox_build/instrumented/dist/firefox/firefox http://localhost:8888/index.html -profile /tmp/portage/www-client/firefox-94.0/temp/tmpxi7sq84w * -------------------------------------------------------------------------------- emerge --info: https://pastebin.com/ZLZ7Fj8H
Ivan, the sandbox violation you experienced was addressed 15min before you posted your answer. Please update repository and try again. Regarding the issue PGO problem itself: I am unable to reproduce. Building firefox with PGO/LTO using both CLANG or GCC worked. Tried on stable (12) and unstable (13). All testings were done in fresh images. Due to other bugs I suspect that LLVM/CLANG toolchain for some might be busted. I recommend rebuilding the entire toolchain to be safe.
I rebuilt LLVM/Clang/lld with GCC then rebuilt them with LLVM/Clang/lld, I encounter the same issue. Should I also rebuild rust ?
Created attachment 748629 [details] Firefox 94.0.1 PGO build with fresh LLVM bootstrapped toolchain
(In reply to Thomas Deutschmann from comment #15) > Ivan, the sandbox violation you experienced was addressed 15min before you > posted your answer. Please update repository and try again. > > Regarding the issue PGO problem itself: > > I am unable to reproduce. Building firefox with PGO/LTO using both CLANG or > GCC worked. Tried on stable (12) and unstable (13). All testings were done > in fresh images. Due to other bugs I suspect that LLVM/CLANG toolchain for > some might be busted. I recommend rebuilding the entire toolchain to be safe. Thanks. So far I have same issue as mentioned in #17 (https://bugs.gentoo.org/821505#c17). I will try to rebuild toolchain and see if this will help. Could be glibc-2.34 that I have installed
(In reply to Ivan from comment #18) > (In reply to Thomas Deutschmann from comment #15) > > Ivan, the sandbox violation you experienced was addressed 15min before you > > posted your answer. Please update repository and try again. > > > > Regarding the issue PGO problem itself: > > > > I am unable to reproduce. Building firefox with PGO/LTO using both CLANG or > > GCC worked. Tried on stable (12) and unstable (13). All testings were done > > in fresh images. Due to other bugs I suspect that LLVM/CLANG toolchain for > > some might be busted. I recommend rebuilding the entire toolchain to be safe. > > Thanks. > > So far I have same issue as mentioned in #17 > (https://bugs.gentoo.org/821505#c17). > > I will try to rebuild toolchain and see if this will help. Could be > glibc-2.34 that I have installed I am experiencing the hangs with glibc-2.33-r7, downgrading rust and llvm fixed the issue for me.
Rebuilt toolchain, rebuilt all clang-related packages. Still cannot build FF 94 with PGO, building just 'freezes' at some point and nothing happens. Usually FF builds in about an 1 hour with PGO, but now I see this: > current merge time: 12 hours, 32 minutes and 37 seconds. > ETA: any time now. So I had to manually interrupt compilation. Here's the last lines of log: https://pastebin.com/eAnqDUr8 Guess I should try building rust, because I used -bin version so far.
Well, unsurprisingly, building rust from the sources didn't help in this particular issue.
Is this related? Seems like clang/llvm-13 will be supported in firefox-95: https://bugzilla.mozilla.org/show_bug.cgi?id=1731582
I'm just here to say that I have this issue as well and rebuilding the LLVM toolchain doesn't help.
I had to - uninstall rust-1.56, llvm 13 - install llvm 12, rust-1.55 - switch to rust-1.55 - check by saying "rustc -v --version" that it's using LLVM 12 - may have to "ldconfig" if it can't find .so files - restrict the firefox ebuild to llvm 12 using the patched ebuild in my "local" portage repo - I also changed its dependency on virtual/rust to || ( ( >=dev-lang/rust-1.51 <dev-lang/rust-1.56 ) ( >=dev-lang/rust-bin-1.51 <dev-lang/rust-bin-1.56 ) ) since I couldn't figure out a better way to do it. virtual/rust doesn't seem to be slotted. are virtuals implicitly slotted? That was enough to get Firefox 94 to build and install, but now I'm struggling with portage wanting to only have one version of dev-lang/rust installed. Isn't it slotted? Why is installing rust-1.55 considered a downgrade ("UD") instead of a new slot ("NS")?
(In reply to Alexandra Parker from comment #24) > That was enough to get Firefox 94 to build and install, but now I'm > struggling with portage wanting to only have one version of dev-lang/rust > installed. Isn't it slotted? Why is installing rust-1.55 considered a > downgrade ("UD") instead of a new slot ("NS")? Rust isn't slotted (even though I thought it was until 2 days ago!).
I'm re-reading this until it makes sense: https://wiki.gentoo.org/wiki/Sub-slots_and_Slot-Operators Unless some compatibility fix is backported I think it makes sense to mask rust-1.56 to prevent the version flapping.
(In reply to Alexandra Parker from comment #26) > I'm re-reading this until it makes sense: > > https://wiki.gentoo.org/wiki/Sub-slots_and_Slot-Operators > > Unless some compatibility fix is backported I think it makes sense to mask > rust-1.56 to prevent the version flapping. (and by that I mean it makes sense for people trying to work around this problem)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ef9940a9394e7a0311bb9d5987e0d442d7188c52 commit ef9940a9394e7a0311bb9d5987e0d442d7188c52 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2021-11-09 20:51:51 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2021-11-09 20:53:32 +0000 www-client/firefox: fix runtime errors Closes: https://bugs.gentoo.org/822492 Closes: https://bugs.gentoo.org/822138 Bug: https://bugs.gentoo.org/821505 Package-Manager: Portage-3.0.28, Repoman-3.0.3 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/Manifest | 100 +- ...efox-94.0.1.ebuild => firefox-94.0.1-r1.ebuild} | 2 +- www-client/firefox/firefox-94.0.ebuild | 1209 -------------------- 3 files changed, 2 insertions(+), 1309 deletions(-)
-r1 doesn't seem to help: console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/tmp/portage/www-client/firefox-94.0.1-r1/temp/tmpef3jyhwi/search.json.mozlz4", (void 0))) ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost jarlog: /var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/jarlog/en-US.log JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. 1636553855045 addons.xpi ERROR System addon update list error Error: Failed downloading XML, status: 0, reason: error
Created attachment 750198 [details] build.log.xz
(In reply to John Helmert III from comment #29) > -r1 doesn't seem to help: > > console.warn: SearchSettings: "get: No settings file exists, new profile?" > (new NotFoundError("Could not open the file at > /var/tmp/portage/www-client/firefox-94.0.1-r1/temp/tmpef3jyhwi/search.json. > mozlz4", (void 0))) > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > jarlog: > /var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/jarlog/en- > US.log > JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't > find profile directory. > JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't > find profile directory. > 1636553855045 addons.xpi ERROR System addon update list error > Error: Failed downloading XML, status: 0, reason: error Worked for me last night. Unmasked clang/llvm/rust stuff, made sure that was all updated and then updated to 94.0.1-r1 and it compiled with PGO for me.
Is anyone who is experiencing PGO build failures _not_ using USE=pulseaudio? For me it looks like USE=pulseaudio is the trigger...
Looks like -r1 does the trick for me. Here's list of USE-flags I used: www-client/firefox-94.0.1-r1:0/94::gentoo USE="clang dbus gmp-autoupdate hwaccel lto openh264 pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland -debug -eme-free -geckodriver -hardened -jack -screencast (-selinux) -sndio -wifi" I used rust-bin-1.56.1 and latest llvm. As far as I can say it used LLVM 13 to build FF. Builds fine, no hangs detected so far. Huge thanks to Gentoo developers.
I tried again after Comment #33 but I still get the same issue Useflags: clang dbus gmp-autoupdate lto openh264 pgo pulseaudio screencast system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland wifi -debug -eme-free -geckodriver -hardened -hwaccel -jack (-selinux) -sndio
I'm getting what looks like a similar issue. Building firefox 91.3.0 times out: 19:10.24 gmake[1]: Leaving directory '/var/tmp/portage/www-client/firefox-91.3.0/work/firefox_build/instrumented/browser/installer' 19:10.24 gmake: Leaving directory '/var/tmp/portage/www-client/firefox-91.3.0/work/firefox_build/instrumented' console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/tmp/portage/www-client/firefox-91.3.0/temp/tmpisyuhggx/search.json.mozlz4", (void 0))) JavaScript error: resource:///modules/Interactions.jsm, line 209: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIUserIdleService.removeIdleObserver] ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost jarlog: /var/tmp/portage/www-client/firefox-91.3.0/work/firefox_build/jarlog/en-US.log JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource:///modules/Interactions.jsm, line 209: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIUserIdleService.removeIdleObserver] ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"MediaTrackGraph 7f22e12afc00 shutdown","state":"(none)","filename":"/var/tmp/portage/www-client/firefox-91.3.0/work/firefox-91.3.0/dom/media/MediaTrackGraph.cpp","lineNumber":1784,"stack":"MediaTrackGraph shutdown"}] Barrier: xpcom-will-shutdown JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 4995: NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIWindowMediator.getEnumerator] emerge --info: Portage 3.0.28 (python 3.9.7-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-11.2.0, glibc-2.33-r7, 5.15.1-gentoo-cirno x86_64) ================================================================= System uname: Linux-5.15.1-gentoo-cirno-x86_64-11th_Gen_Intel-R-_Core-TM-_i7-11850H_@_2.50GHz-with-glibc2.33 KiB Mem: 32582408 total, 5118432 free KiB Swap: 16383996 total, 15869456 free Timestamp of repository gentoo: Sat, 13 Nov 2021 00:21:52 +0000 Head commit of repository gentoo: 9f16e61545b3eb770da7cefac83b9a23da92412f Timestamp of repository gentoo-zh: Sat, 13 Nov 2021 00:21:37 +0000 Head commit of repository gentoo-zh: c511984399adc425cbb2364a910ffeb7fc1fad73 Timestamp of repository menelkir: Thu, 11 Nov 2021 19:13:35 +0000 Head commit of repository menelkir: 69acb3d75dac7ee9de9db847822b054d0858bb53 Timestamp of repository steam-overlay: Tue, 09 Nov 2021 10:53:07 +0000 Head commit of repository steam-overlay: b21d724e69a176a398cae2db70172cb9dfede48a sh bash 5.1_p8 ld GNU ld (Gentoo 2.37_p1 p0) 2.37 distcc 3.4 x86_64-pc-linux-gnu [disabled] app-shells/bash: 5.1_p8::gentoo dev-lang/perl: 5.34.0-r3::gentoo dev-lang/python: 2.7.18_p13::gentoo, 3.9.7_p1::gentoo dev-lang/rust: 1.56.1::gentoo dev-util/cmake: 3.20.5::gentoo sys-apps/baselayout: 2.7::gentoo sys-apps/sandbox: 2.25::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.71-r1::gentoo sys-devel/automake: 1.16.4::gentoo sys-devel/binutils: 2.37_p1::gentoo sys-devel/gcc: 11.2.0::gentoo sys-devel/gcc-config: 2.4::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.3::gentoo sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers) sys-libs/glibc: 2.33-r7::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git priority: -1000 ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=tigerlake -O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=tigerlake -O2 -pipe" DISTDIR="/var/cache/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-march=tigerlake -O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-march=tigerlake -O2 -pipe" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j15" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi activities alsa amd64 bash-completion bluetooth branding bzip2 cairo cdda cdr cli colord crypt cups dbus declarative dri dts dvd dvdr emboss encode exif flac fortran gdbm gif gles2 gpm gstreamer gtk gui iconv icu ipv6 jpeg kde kipi kwallet lcms libglvnd libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses nls nptl offensive ogg opencl opengl openmp pam pango pcre pdf pipewire plasma png policykit ppds pulseaudio qml qt5 readline screencast sdl seccomp semantic-desktop spell split-usr ssl startup-notification svg systemd theora tiff truetype udev udisks unicode upower usb v4l vaapi vdpau vorbis vulkan wayland widgets wxwidgets x264 x265 xattr xcb xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2019" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 avx512f avx512dq avx512cd avx512bw avx512vl avx512vbmi f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="intel i965 iris nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS Flags I used: USE="clang dbus gmp-autoupdate lto openh264 pgo pulseaudio screencast system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland -debug -eme-free -geckodriver -hardened -hwaccel -jack (-selinux) -sndio -wifi" ABI_X86="(64)" L10N="-ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -ca-valencia -cak -cs -cy -da -de -dsb -el -en-CA -en-GB -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -ne -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -sco -si -sk -sl -son -sq -sr -sv -szl -ta -te -th -tl -tr -trs -uk -ur -uz -vi -xh -zh-CN -zh-TW"
Builds with pgo again using rust 1.56.1 llvm 13 and re-keyworded glibc 2.34-r1 www-client/firefox-94.0.1-r1:0/94::gentoo USE="clang dbus hardened lto openh264 pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland -debug -eme-free -geckodriver (-gmp-autoupdate) -hwaccel -jack -screencast (-selinux) -sndio -wifi" Many thanks to all involved.
Following Comment #36 I tried emerging firefox 94.0.1-r1 again and it successfully built. But the error messages still show, as I reported in my first description message: ``` 11:42.59 gmake: Leaving directory '/var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/instrumented' console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/tmp/portage/www-client/firefox-94.0.1-r1/temp/tmpbhtdum7x/search.json.mozlz4", (void 0))) ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost jarlog: /var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/jarlog/en-US.log JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory. ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ``` I am afraid PGO actually isn't (entirely) working, even when enabled.
Created attachment 751158 [details] firefox-94.0.1-r1 pgo+lto successfull merge but with error message
(In reply to Adel KARA SLIMANE from comment #37) > Following Comment #36 I tried emerging firefox 94.0.1-r1 again and it > successfully built. But the error messages still show, as I reported in my > first description message: > > ``` > 11:42.59 gmake: Leaving directory > '/var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/ > instrumented' > console.warn: SearchSettings: "get: No settings file exists, new profile?" > (new NotFoundError("Could not open the file at > /var/tmp/portage/www-client/firefox-94.0.1-r1/temp/tmpbhtdum7x/search.json. > mozlz4", (void 0))) > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > jarlog: > /var/tmp/portage/www-client/firefox-94.0.1-r1/work/firefox_build/jarlog/en- > US.log > JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't > find profile directory. > JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't > find profile directory. > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > > > ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, > messages will be lost > ``` > > I am afraid PGO actually isn't (entirely) working, even when enabled. Unless someone knows otherwise, I think these messages can be ignored because PGO is run in a temporary environment. You would have to run a PGO build outside of Portage to determine otherwise.
In my Comment #11 I attempted a build from source outside of Portage and encountered the same error messages. I will retry with 94.0.1 and report back.
Okay I tried building from source on Firefox 94.0.1 tar source code, this time with LTO enabled in `mozconfig` ``` ac_add_options --enable-lto=cross ac_add_options --with-libclang-path=/usr/lib/llvm/13/lib64 ac_add_options MOZ_PGO=1 mk_add_options MOZ_PARALLEL_BUILD=32 export RUSTC_OPT_LEVEL=2 ``` and the it looks like the message still shows, so maybe somthing upstream ? ``` 9:42.64 gmake: Leaving directory '/media/personal/Downloads/firefox-94.0.1.source/firefox-94.0.1/obj-x86_64-pc-linux-gnu/instrumented' ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. ATTENTION: default value of option mesa_glthread overridden by environment. console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/tmpamnknw5p/search.json.mozlz4", (void 0))) ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
Created attachment 751182 [details] Self built directly from source firefox lto+pgo
Same here. USE-flag "pgo" is enabled. Last strokes of emerge output: ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"MediaTrackGraph 7f1e0c044000 shutdown","state":"(none)","filename":"/var/tmp/portage/www-client/firefox-91.3.0/work/firefox-91.3.0/dom/media/MediaTrackGraph.cpp","lineNumber":1784,"stack":"MediaTrackGraph shutdown"}] Barrier: xpcom-will-shutdown JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 4995: NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIWindowMediator.getEnumerator]
A general comment: The bug summary itself ("Error: Channel closing: too late to send/recv, messages will be lost") is totally misleading and doesn't indicate a PGO problem/unsuccessful build. For example, see this Fedora build.log (https://kojipkgs.fedoraproject.org//packages/firefox/94.0/1.fc33/data/logs/x86_64/build.log) for their PGO build which was successful and still contain "Error: Channel closing: too late to send/recv, messages will be lost". The problem is that during instrumented build, firefox will hang or crash. To make progress, we need to catch these crashes. However, this is not easy and at the moment I don't know how to get to these information.
Shouldn't it be reported upstream then ?
For the record, I have observed the same problems - i.e. with USE=pgo emerge hangs, with USE=-pgo firefox can be built but crashes on a regular basis - with =www-client/firefox-91.3.0, our current ESR ebuild. Downgrading to llvm-12 and rust-1.53 appears to solve both. Interestingly enough this only happens on only some of my Gentoo boxes, even though they have got almost identical software configuration.
@Marek: Yes, ESR needs the patch for latest rust, see https://bugs.gentoo.org/822138#c13 Leaving this bug for the unknown problems.
*** Bug 826954 has been marked as a duplicate of this bug. ***
I tried to unmask firefox-95.0 and it has been compiled successfully. The other thing to achieve it was unmasking dev-libs/nss-3.72. $ tail -n2 /etc/portage/package.accept_keywords/i_want_to_get_new_firefox =www-client/firefox-95.0::gentoo ~amd64 =dev-libs/nss-3.72::gentoo ~amd64 USE-flags enabled for firefox: clang dbus gmp-autoupdate lto openh264 pgo pulseaudio system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp KERNEL="linux" L10N="ru
I seem to be able to reproduce on Firefox 95.
I went ahead and opened a bug report at Mozilla's bugzilla [1], hopefully it will help them and help us figure out what's at play and how to deal with this annoying issue. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1745641
I just attempted to build firefox 95 with PGO + LTO, it hung for quite a while on those "send/recv" error messages then, to my surprise, went on and successfully got built.
Created attachment 758697 [details] emerge --info for successful firefox 95 PGO+LTO build
Created attachment 758698 [details] Build log for successful firefox 95 build (still with error messages and a finite hang)
(In reply to Adel KARA SLIMANE from comment #52) > I just attempted to build firefox 95 with PGO + LTO, it hung for quite a > while on those "send/recv" error messages then, to my surprise, went on and > successfully got built. I had seen the same behaviour of building process. It had been stopped compiling in "send/recv error" stage and I even decided to press 'Ctrl+C' but somebody had draw me away. When I got back firefox-95 was ready to use. Unfortunately I forgot to save build.log so I have nothing to show in here.
Haven't seen these in a while, I presume this is fixed by upstream (and/or related to our pgo patches)
