Created attachment 563424 [details] emerge --info After build Firefox 65 with flags pgo and lto, FF crash with error "invalid instruction".
Created attachment 563426 [details] Run in GDB
Created attachment 563428 [details] Strace
Created attachment 563430 [details] Firefox build settings
Forgot to add - Firefox build with GCC 8.2.0-r6
I have the same on one of my boxes but at the moment it looks like it is caused by a somehow broken toolchain (rust, llvm...). I am not conviced yet that this caused by firefox itself.
With USE="lto -pgo" same error Program received signal SIGILL, Illegal instruction. 0x00007fffe68e967a in ?? () from /usr/lib64/firefox/libxul.so
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=da3675965824a307cd16ffc48193929ba8125d98 commit da3675965824a307cd16ffc48193929ba8125d98 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2019-02-10 00:50:26 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2019-02-10 21:49:56 +0000 www-client/firefox: USE=pgo when using GCC will require AVX2 support Bug: https://bugs.gentoo.org/677052 Package-Manager: Portage-2.3.59, Repoman-2.3.12 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/firefox-65.0-r1.ebuild | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
I think its this one? https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88561
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=28ae5ce2652cc4e1d5488034a6dd1c3d1cf26781 commit 28ae5ce2652cc4e1d5488034a6dd1c3d1cf26781 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2019-02-11 15:21:32 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2019-02-11 15:21:53 +0000 www-client/firefox: use -fdisable-ipa-cdtor workaround when AVX2 is not available This commit changes commit da3675965824a307cd16ffc48193929ba8125d98. Bug: https://bugs.gentoo.org/677052 Package-Manager: Portage-2.3.59, Repoman-2.3.12 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/firefox-65.0-r1.ebuild | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
(In reply to tt_1 from comment #8) > I think its this one? > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88561 No. We are carrying a patch for this one (6012_fix-pgo-gcc-bug-in-spellchecker.patch). The problem from this bug is, that GCC will produce AVX2 instructions even when AVX2 is not available. E.g. when you copy the binaries which are crashing with "invalid instructions" to a system with AVX2 support, they will work...
Just so you know, firefox[pgo] is possible even on non-avx2-capable system. www-client/firefox-65.0-r1::local USE="custom-cflags custom-optimization dbus eme-free hardened hwaccel jack lto pgo pulseaudio screenshot system-harfbuzz system-jpeg system-sqlite wayland -bindist -clang -debug -geckodriver (-gmp-autoupdate) (-neon) (-selinux) -startup-notification -system-icu -system-libevent -system-libvpx -system-webp -test -wifi" sys-devel/gcc-8.2.0-r6::gentoo USE="cxx fortran go graphite hardened mpx (multilib) nls nptl objc objc++ objc-gc openmp pgo (pie) (ssp) vtv (-altivec) -debug -doc (-fixed-point) (-jit) (-libssp) (-pch) -regression-test (-sanitize) -systemtap -vanilla" ABI_X86="(64)" dev-lang/rust-1.31.1::gentoo USE="clippy jemalloc rls rustfmt system-llvm -debug -doc -libressl -wasm" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="sse2" LLVM_TARGETS="(X86) -AArch64 -AMDGPU -ARM -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -Sparc -SystemZ -XCore" > grep -q avx2 /proc/cpuinfo || echo nosuchthing > nosuchthing firefox builds and runs fine
Forgot to mention that's locally edited ebuild without the '!clang? ( cpu_flags_x86_avx2 )' restriction
(In reply to ak from comment #12) > Forgot to mention that's locally edited ebuild without the '!clang? ( > cpu_flags_x86_avx2 )' restriction Comments like this doesn't help. Maybe you haven't set CPU USE flags but maybe your processor supports AVX2. In the end the GCC bug is well known and upstream fixed the problem. So yes, it is confirmed that current released GCC versions will produce AVX2 instructions when building firefox with USE=pgo... But it doesn't matter anymore. Restriction was removed and we are now using a workaround for systems without AVX2.
(In reply to Thomas Deutschmann from comment #13) > (In reply to ak from comment #12) > > Forgot to mention that's locally edited ebuild without the '!clang? ( > > cpu_flags_x86_avx2 )' restriction > > Comments like this doesn't help. Maybe you haven't set CPU USE flags but > maybe your processor supports AVX2. In the end the GCC bug is well known and > upstream fixed the problem. So yes, it is confirmed that current released > GCC versions will produce AVX2 instructions when building firefox with > USE=pgo... Thank you for very helpful comment. Yes I do realise that not setting CPU flag make my CPU support avx2. > But it doesn't matter anymore. Restriction was removed and we are now using > a workaround for systems without AVX2. Good news. Less burden to my local overlay. Cheers.
(In reply to Thomas Deutschmann from comment #13) > (In reply to ak from comment #12) > > Forgot to mention that's locally edited ebuild without the '!clang? ( > > cpu_flags_x86_avx2 )' restriction > > Comments like this doesn't help. Maybe you haven't set CPU USE flags but > maybe your processor supports AVX2. In the end the GCC bug is well known and > upstream fixed the problem. So yes, it is confirmed that current released > GCC versions will produce AVX2 instructions when building firefox with > USE=pgo... > > But it doesn't matter anymore. Restriction was removed and we are now using > a workaround for systems without AVX2. Hello. As I understand it, Firefox did not start because GCC forcibly builds Firefox with AVX2 instructions, and ebuild Firefox-65.0.1 had to fix this with the -fdisable-ipa-cdtor option. If this is the case, Firefox still does not start, the error is the same "Illegal instruction".
May I ask, did you use dev-lang/rust-1.32.0 to compile, with system-llvm?
(In reply to tt_1 from comment #16) > May I ask, did you use dev-lang/rust-1.32.0 to compile, with system-llvm? No, this flag disabled.
I ran into this issue on my old MSI GX700 Laptop with CoreDuo T7500: illegal instruction in ... libxul.so It seems indeed to be a compiler issue. After upgrading to unstable gcc-8.3.0 and rebuilding Firefox, it runs fine. ervin
OK, now I’ll compile gcc 8.3.0 and then Firefox.
With GCC 8.3.0 0x00007fffe8980f13 in ?? () from /usr/lib64/firefox/libxul.so Segmentation fault I will try as Ervin Peters to build Firefox on an old laptop with a CoreDuo T2250 processor.
Just saw bug 672946, could you please add the two patches from bug 672946 comment #42, rebuild gcc and retry?
(In reply to Thomas Deutschmann from comment #21) > Just saw bug 672946, could you please add the two patches from bug 672946 > comment #42, rebuild gcc and retry? All the same error as without patches Thread 1 "firefox" received signal SIGSEGV, Segmentation fault. 0x00007fffe8980f13 in ?? () from /usr/lib64/firefox/libxul.so
Can you please compile without USE="lto pgo" so we can be sure that FF itself works for you and this is lto/pgo related?
This is a bug in GCC that was fixed in GCC commit abf54c8d65cb18b7c67b11fa8a02cbcf5103870a That commit was added to 8.3.0 but not 7.4.0. GCC 8.3.0 ipa.c history: https://gcc.gnu.org/git/?p=gcc.git;a=history;f=gcc/ipa.c;h=2e6621ccc8fcdece6ce992688c1a69e00e663217;hb=4ac50a4913ed81cc83a8baf865e49a2c62a5fe5d GCC 7.4.0 ipa.c history: https://gcc.gnu.org/git/?p=gcc.git;a=history;f=gcc/ipa.c;h=a74f28c2aa1236e4e8877a647e924f146a90799e;hb=67641d6f9b47bf82d1c2de808b391f1332f1ef84 Details about this bug may be found here: https://gcc.gnu.org/ml/gcc-patches/2018-12/msg01142.html And here: http://hubicka.blogspot.com/2018/12/firefox-64-built-with-gcc-and-clang.html Alec
(In reply to Thomas Deutschmann from comment #23) > Can you please compile without USE="lto pgo" so we can be sure that FF > itself works for you and this is lto/pgo related? Yes, without these flags, Firefox runs.
@ Alec: 1) We have a workaround for cdtor problem (https://gitweb.gentoo.org/repo/gentoo.git/log/?qt=grep&q=cdtor) 2) Alexander said that he still gets the same error with gcc-8.3.0. @ Alexander: 1) You are building on your own local machine for the same system, right? 2) No distcc, ccache or other things like that involved, right? Please test FF66 again. If you still end up with "Illegal Instructions" please share the created image with us (quickpkg). I have to analyze the binary...
It works for Ervin Peters and myself. BDVER2 here, CPU supports AVX but no AVX2.
On a Core2 system (Xeon X7460; Penryn derivative): CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3 sse4_1" Kernel-reported flags (from /proc/cpuinfo): flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority dtherm FF65.0.2, GCC7.3.0, USE="lto pgo" => SIGILL FF65.0.2, GCC7.3.0, USE="lto pgo", -fdisable-ipa-cdtor => SIGILL FF65.0.2, GCC7.3.0, USE="-lto -pgo" => works FF65.0.2, GCC8.2.0, USE="lto pgo" => SIGILL FF65.0.2, GCC8.2.0, USE="lto pgo", -fdisable-ipa-cdtor => SIGILL FF65.0.2, GCC8.2.0, USE="lto -pgo", -fdisable-ipa-cdtor => SIGILL FF65.0.2, GCC8.2.0, USE="-lto -pgo", -fdisable-ipa-cdtor => works The missing combinations are currently unknowns. Hopefully I didn't make any mistakes while creating the above table. No distcc or ccache in use here. Apparently LTO is the problem, at least on my system...? ######## @Thomas: I have a couple .xpaks that you might be interested in... These were not built by the X7460 system above, but the system that built them has a near-identical portage config & depgraph. They have CFLAGS="-march=nocona -mtune=core2" and CPU_FLAGS_X86="mmx mmxext sse sse2". I think they were both built with GCC8.2, but I'm not seeing the compiler version in the xpak metadata so I'm not completely certain of that. firefox-65.0.2-2.xpak #USE="lto pgo" => SIGILL on X7460 firefox-65.0.2-3.xpak #USE="lto -pgo" => SIGILL on X7460
> 1) You are building on your own local machine for the same system, right? Yes. > 2) No distcc, ccache or other things like that involved, right? Yes. > Please test FF66 again. If you still end up with "Illegal Instructions" > please share the created image with us (quickpkg). I have to analyze the > binary... OK.
(In reply to Will Simoneau from comment #28) > Apparently LTO is the problem, at least on my system...? No, I have the same problem. By the way, my processor also does not support AVX2 and AVX instructions (AMD Phenom II x6).
Build FF 66 fails at: 69:53.50 JARLOG_FILE=jarlog/en-US.log /var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/ff/_virtualenvs/init/bin/python /var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/build/pgo/profileserver.py 69:53.50 Running Firefox as root in a regular user's session is not supported. ($HOME is /var/calculate/tmp/portage/www-client/firefox-66.0-r1/homedir which is owned by portage.) 69:53.51 Firefox exited with code 1 during profile initialization 69:53.51 gmake[1]: *** [Makefile:199: profiledbuild] Error 1 69:53.51 gmake[1]: Leaving directory '/var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/ff' 69:53.51 gmake: *** [client.mk:125: build] Error 2 69:53.53 110 compiler warnings present.
(In reply to Will Simoneau from comment #28) > @Thomas: I have a couple .xpaks that you might be interested in... Not exactly sure what you mean with xpaks but I need the install image. Using quickpkg would create an complete package file when you already have a non-working firefox installed. (In reply to Alexander from comment #31) > Build FF 66 fails at: > > 69:53.50 JARLOG_FILE=jarlog/en-US.log > /var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/ff/ > _virtualenvs/init/bin/python > /var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/ > build/pgo/profileserver.py > 69:53.50 Running Firefox as root in a regular user's session is not > supported. ($HOME is > /var/calculate/tmp/portage/www-client/firefox-66.0-r1/homedir which is owned > by portage.) > 69:53.51 Firefox exited with code 1 during profile initialization > 69:53.51 gmake[1]: *** [Makefile:199: profiledbuild] Error 1 > 69:53.51 gmake[1]: Leaving directory > '/var/calculate/tmp/portage/www-client/firefox-66.0-r1/work/firefox-66.0/ff' > 69:53.51 gmake: *** [client.mk:125: build] Error 2 > 69:53.53 110 compiler warnings present. If building FF66 is failing for you, this is a different issue (build issue). Your excerpt doesn't help because it doesn't contain the failure reason (error).
(In reply to Thomas Deutschmann from comment #32) > If building FF66 is failing for you, this is a different issue (build > issue). Your excerpt doesn't help because it doesn't contain the failure > reason (error). I should create a new FF-66 build error report, or just put the build log here, maybe the reason for the crash is pretty simple, I just can't find it in the log?
It was necessary to add FEATURES = "userpriv", then there will be no problems with the rights to the directory. Firefox builds with pgo and lto, so the bug is fixed.
