Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 680934 - www-client/firefox-66.0 - cargo: thread 'main' panicked at 'stack backtrace: ???
Summary: www-client/firefox-66.0 - cargo: thread 'main' panicked at 'stack backtrace: ???
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Mozilla Gentoo Team
: 680990 (view as bug list)
Depends on:
Reported: 2019-03-19 14:27 UTC by Luke Bratch
Modified: 2019-04-12 00:52 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---

build.log.gz (build.log.gz,111.85 KB, application/gzip)
2019-03-19 14:28 UTC, Luke Bratch
build-backtrace.log.gz (build-backtrace.log.gz,109.60 KB, application/gzip)
2019-03-19 22:03 UTC, Luke Bratch
build-nobacktrace.log.gz (build-nobacktrace.log.gz,112.78 KB, application/gzip)
2019-03-19 22:06 UTC, Luke Bratch
build-backtrace-success.log.gz (build-backtrace-success.log.gz,557.88 KB, application/gzip)
2019-03-20 10:39 UTC, Luke Bratch
backtrace.log (firefox66-style-rust-cargo-error.log,77.43 KB, text/plain)
2019-03-20 15:35 UTC, Thomas Deutschmann
build-rust-bin.log.gz (build-rust-bin.log.gz,112.87 KB, application/gzip)
2019-03-20 16:33 UTC, Luke Bratch
Build log (www-client:firefox-66.0-r1:20190321-132743.log.lzma,70.50 KB, application/x-lzma)
2019-03-21 13:43 UTC, Saul Peebsen
firefox-66.0.2-nopanic.log.gz (firefox-66.0.2-nopanic.log.gz,111.21 KB, application/gzip)
2019-03-29 12:32 UTC, Luke Bratch
firefox-66.0.2-nopanic-finished.log.gz (firefox-66.0.2-nopanic-finished.log.gz,567.55 KB, application/gzip)
2019-03-29 12:33 UTC, Luke Bratch
Successful build using MAKEOPTS=-j1 (build.log.xz,658.31 KB, application/x-xz)
2019-04-02 19:40 UTC, Thomas Deutschmann

Note You need to log in before you can comment on or make changes to this bug.
Description Luke Bratch 2019-03-19 14:27:11 UTC
www-client/firefox-66.0 fails to compile, build log attached.

I see the warnings at the start of the log about clang/llvm slots 7 and 8, but I get the same warnings on another box that compiles firefox-66.0 without problem so I don't think that's the problem.

emerge --info:
Portage 2.3.62 (python 2.7.15-final-0, default/linux/amd64/17.0, gcc-8.2.0, glibc-2.27-r6, 4.20.16-gentoo x86_64)                                                                                                                                                               
System uname: Linux-4.20.16-gentoo-x86_64-Intel-R-_Core-TM-_i5-2520M_CPU_@_2.50GHz-with-gentoo-2.6                                                                                                                                                                              
KiB Mem:     8040200 total,   1776748 free                                                                                                                                                                                                                                      
KiB Swap:    4194300 total,   4194300 free                                                                                                                                                                                                                                      
Timestamp of repository gentoo: Tue, 19 Mar 2019 12:30:01 +0000                                                                                                                                                                                                                 
Head commit of repository gentoo: e3ea551322689ebe50de119d860dc5a9d9fc0d5e                                                                                                                                                                                                      
sh bash 4.4_p23-r1                                                                                                                                                                                                                                                              
ld GNU ld (Gentoo 2.30 p5) 2.30.0                                                                                                                                                                                                                                               
Head commit of repository gentoo: e3ea551322689ebe50de119d860dc5a9d9fc0d5e
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.26.2::gentoo
dev-lang/python:          2.7.15::gentoo, 3.5.5::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.38.3-r1::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.30-r4::gentoo
sys-devel/gcc:            8.2.0-r6::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.14-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo

    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

    location: /usr/local/portage
    masters: gentoo
    priority: 0

    location: /var/lib/layman/anyc
    masters: gentoo
    priority: 50

    location: /var/lib/layman/steam-overlay
    masters: gentoo
    priority: 50

    location: /var/lib/layman/tlp
    masters: gentoo
    priority: 50

CFLAGS="-march=sandybridge -O2 -pipe"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=sandybridge -O2 -pipe"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
USE="X a52 aac aacplus aalib acl aiglx alsa amd64 avx berkdb bluetooth bzip2 cleartype cli crypt cups cxx dbus directfb divx dri dv dvd dvdr dvdread encode exif ffmpeg firefox flac fortran ftp g3dvl gdbm gif gopher gpm gtk http2 iconv id3tag idn imagemagick ipv6 java jpeg lame ldap libnotify libtirpc libv4l lua midi mmx mmxext mp3 mpeg mtp multilib mysql ncurses njb nls nptl nptlonly ogg openal opengl openmp pam pcre png projectm quicktime readline rtmp samba scanner sdl sdl2 seccomp smp speex sse sse2 sse2_4way sse3 sse4 sse4_1 ssh ssl ssse3 tcltk tcpd theora tiff timidity truetype unicode usb v4l v4l2 vaapi vdpau vorbis webp wifi win32codecs wmf x264 xattr xinerama xv xvid zlib" ABI_X86="64 32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" L10N="en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

Comment 1 Luke Bratch 2019-03-19 14:28:20 UTC
Created attachment 569768 [details]

Had to attach build log gzipped due to file size.
Comment 2 Thomas Deutschmann gentoo-dev Security 2019-03-19 18:57:10 UTC
> I see the warnings at the start of the log about clang/llvm slots 7 and 8, but I get the same warnings on another box that compiles firefox-66.0 without problem so I don't think that's the problem.
Right, as long as this block ends with

> * Will use LLVM slot X!
you are fine -- no error.

I cannot reproduce with

- gcc-8.3.0 (you used 8.2.0)
- rust-1.32 (same)
- cbindgen-0.7.1 (same)
- clang:6 (same)
- llvm:6 (same)

Please re-install rust with symbols enabled.

If you are still able to reproduce, show us the error with the following env variables set:

Comment 3 Luke Bratch 2019-03-19 22:03:54 UTC
Created attachment 569912 [details]

Thanks for your reply.

I have rebuilt Rust with FEATURES="nostrip" and attempted to build Firefox with RUST_BACKTRACE=1 RUST_LOG=debug and it fails with different errors, please see the attached.

Just to confirm, my other machine that works fine (also amd64) works with the same packages including gcc-8.2:

- gcc-8.2.0-r6
- rust-1.32
- cbindgen-0.7.1
- clang:6
- llvm:6

Comment 4 Luke Bratch 2019-03-19 22:06:11 UTC
Created attachment 569914 [details]

In case it is useful, here is Rust rebuilt with FEATURES="nostrip" and Firefox re-attempted without RUST_BACKTRACE or RUST_LOG set.  Same error as in the initial report.
Comment 5 Thomas Deutschmann gentoo-dev Security 2019-03-19 22:29:17 UTC
Yes, you cannot build entire firefox with RUST_LOG=debug set. But you can set RUST_LOG=debug when the error occurs and then re-run to get details:

> cd $(portageq get_repo_path / gentoo)/www-client/firefox
> ebuild firefox-66.0.ebuild clean compile
> building will fail...
> RUST_BACKTRACE=1 RUST_LOG=debug ebuild firefox-66.0.ebuild compile

This should give us the details we are looking for.
If not, please show us build.log only with RUST_BACKTRACE=1 set...
Comment 6 Luke Bratch 2019-03-20 10:39:25 UTC
Created attachment 569964 [details]

In an unexpected twist, continuing the failed build with ebuild compile actually let the compile finish.  Now installed and qmerged successfully.

I let it run with RUST_BACKTRACE=1 RUST_LOG=debug anyway and have attached the results.

I have another (very similar) machine that is failing as well, so I will test letting it fail and resume with ebuild compile on that machine with symbol-less rust and without RUST_BACKTRACE or RUST_LOG to see what happens.
Comment 7 Thomas Deutschmann gentoo-dev Security 2019-03-20 15:28:47 UTC
*** Bug 680990 has been marked as a duplicate of this bug. ***
Comment 8 Thomas Deutschmann gentoo-dev Security 2019-03-20 15:35:56 UTC
Created attachment 569972 [details]

Extracted backtrace.
Comment 9 Thomas Deutschmann gentoo-dev Security 2019-03-20 15:37:29 UTC
Are you using rust-bin? If not, can you please install rust-bin in parallel, switch via `eselect rust` and try again?
Comment 10 Luke Bratch 2019-03-20 16:33:32 UTC
Created attachment 569974 [details]

Done.  Was using dev-lang/rust-1.32.0 before, now using dev-lang/rust-bin-1.32.0:

$ eselect rust list
Available Rust versions:
  [1]   rust-bin-1.32.0 *
  [2]   rust-1.32.0

Same panic as before though.

FYI one difference compared to yesterday is that since firefox-66.0-r1 is now in Portage I had to unmask and install media-libs/libaom-1.0.0-r1 due to a failure during ./configure.  This log is for firefox-66.0-r1.

I haven't had a chance to test the build/fail/resume trick on a second machine yet as it is switched off at home.
Comment 11 Saul Peebsen 2019-03-21 13:43:56 UTC
Created attachment 570110 [details]
Build log

Have tried to upgrade in three computers, all fail.

 ~ $ emerge -pv firefox
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild     U  ] www-client/firefox-66.0-r1::gentoo [65.0.2::gentoo] USE="gmp-autoupdate hwaccel lto screenshot system-av1%* system-harfbuzz system-icu system-jpeg system-libevent system-sqlite system-webp -bindist -clang -custom-cflags -custom-optimization -dbus -debug -eme-free -geckodriver -hardened -jack (-neon) -pgo -pulseaudio (-selinux) -startup-notification -system-libvpx -test -wayland -wifi" CPU_FLAGS_X86="avx2" L10N="-ach -af -an -ar -as -ast -az -bg -bn-BD -bn-IN -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-GB -en-ZA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mai -mk -ml -mr -ms -nb -nl -nn -or -pa -pl -pt-BR -pt-PT -rm -ro -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -uz -vi -xh -zh-CN -zh-T
Comment 12 Luke Bratch 2019-03-21 18:07:35 UTC
(In reply to Luke Bratch from comment #6)
> I have another (very similar) machine that is failing as well, so I will
> test letting it fail and resume with ebuild compile on that machine with
> symbol-less rust and without RUST_BACKTRACE or RUST_LOG to see what happens.

I have now done this - it worked.  A machine with nothing special done other than resuming the compile once it fails using "ebuild /usr/portage/www-client/firefox/firefox-66.0-r1.ebuild compile" works fine.
Comment 13 Robert Schedel 2019-03-25 19:42:13 UTC
Also suffered from similar cargo error on firefox 66 builds (but could not find any panic print).
Interesting observation:
- With "-j8", firefox 66.0.1 built fine for me on my i7-6700K.
- With "-j1" afterwards, the same build again fails reproducably (just like the original poster had used "-j1" in MAKEOPTS).

Might be an option to try out. Maybe some race condition or dependency issue.
Comment 14 Luke Bratch 2019-03-29 12:32:13 UTC
Created attachment 571178 [details]

firefox-66.0.2 also fails on the same machine, and again can be resumed by issuing an ebuild compile.

Unlike in the initial report, I no longer see a panic, much like what Robert mentions in the previous post.

Attached is the log from when it failed (before resuming).
Comment 15 Luke Bratch 2019-03-29 12:33:15 UTC
Created attachment 571180 [details]

And here's the finished log from after issuing the ebuild resume and it ultimately succeeding.
Comment 16 Saul Peebsen 2019-03-29 13:22:45 UTC
Using -j2 instead of -j1 and it builds. First time ever I see parallel build is required for successful build.
Comment 17 Hank Leininger 2019-04-01 01:03:49 UTC
(In reply to Saul Peebsen from comment #16)
> Using -j2 instead of -j1 and it builds. First time ever I see parallel build
> is required for successful build.

Wild.  Same here.

Some time ago I had added package.env settings to set MAKEOPTS="-j1" for firefox - I think ~2018-09-10, because 62.x could not compile with my default "-j10".

But no version of www-client/firefox-66.x would build with -j1.  Upped it to -j4 and it worked!

In case it matters:

Comment 18 Thomas Deutschmann gentoo-dev Security 2019-04-02 19:40:04 UTC
Created attachment 571684 [details]
Successful build using MAKEOPTS=-j1

The claim that you cannot build current firefox with MAKEOPTS=-j1 is wrong:

www-client/firefox-66.0.2 with USE="abi_x86_64 amd64 cpu_flags_x86_avx2 elibc_glibc gmp-autoupdate kernel_linux l10n_de lto pgo screenshot system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp userland_GNU"

real    203m33,001s
user    239m41,843s
sys     11m40,439s

GCC 8.3.0 and rust-1.33 was used:

> Portage 2.3.62 (python 3.7.3-final-0, default/linux/amd64/17.0, gcc-8.3.0, glibc-2.29-r1, 4.19.30-gentoo x86_64)
> =================================================================
> System uname: Linux-4.19.30-gentoo-x86_64-AMD_Ryzen_Threadripper_2950X_16-Core_Processor-with-gentoo-2.6
> KiB Mem:   131953288 total, 123205776 free
> KiB Swap:   16777212 total,  16777212 free
> Timestamp of repository gentoo: Tue, 02 Apr 2019 12:44:16 +0000
> Head commit of repository gentoo: 0553bb476d25269069361ef972888fe53ecac2e0
> sh bash 5.0_p3-r1
> ld GNU ld (Gentoo 2.32 p1) 2.32.0
> app-shells/bash:          5.0_p3-r1::gentoo
> dev-lang/perl:            5.28.0::gentoo
> dev-lang/python:          2.7.16::gentoo, 3.6.8::gentoo, 3.7.3::gentoo
> dev-util/cmake:           3.14.1::gentoo
> sys-apps/baselayout:      2.6-r1::gentoo
> sys-apps/openrc:          0.41.2::gentoo
> sys-apps/sandbox:         2.17::gentoo
> sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
> sys-devel/automake:       1.16.1-r1::gentoo
> sys-devel/binutils:       2.32::gentoo
> sys-devel/gcc:            8.3.0::gentoo
> sys-devel/gcc-config:     2.0::gentoo
> sys-devel/libtool:        2.4.6-r5::gentoo
> sys-devel/make:           4.2.1-r4::gentoo
> sys-kernel/linux-headers: 5.0-r1::gentoo (virtual/os-headers)
> sys-libs/glibc:           2.29-r1::gentoo
Comment 19 Saul Peebsen 2019-04-02 19:46:46 UTC
Wrong is a strong word. Anyhow, something has been changed and FF builds with -j1 now, I can confirm.
Comment 20 John Covici 2019-04-12 00:52:52 UTC
I had similar problems, once a panic and then none, and I did an ebuild … compile and then install and then merge.  This was with J1, gcc 7.3 and (I think) rust 1.33.