Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 925101 - www-client/firefox-123.0[pgo] fails to build during/after linking stage
Summary: www-client/firefox-123.0[pgo] fails to build during/after linking stage
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 2 votes (vote)
Assignee: Mozilla Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-02-20 19:23 UTC by Johannes Penßel
Modified: 2024-04-24 19:58 UTC (History)
24 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
full build log (zstd compressed) (firefox-123.0.log.zst,873.40 KB, application/zstd)
2024-02-20 19:23 UTC, Johannes Penßel
Details
another emerge --info www-client/firefox (emerge.info,8.34 KB, text/plain)
2024-02-21 08:42 UTC, Anders Larsson
Details
Build logs (xz compressed) (build.log.xz,818.44 KB, application/x-xz)
2024-02-21 09:04 UTC, Anders Larsson
Details
www-client:firefox-123.0:20240221-234927.log (build.log.bz2,897.95 KB, application/x-bzip2)
2024-02-22 08:55 UTC, Andreas Thalhammer
Details
emerge --info (emerge-info.txt.gz,7.15 KB, application/gzip)
2024-02-22 09:01 UTC, Andreas Thalhammer
Details
www-client:firefox-123.0:20240222-103851.log (build.log.bz2,891.42 KB, application/x-bzip2)
2024-02-22 11:50 UTC, Andreas Thalhammer
Details
www-client:firefox-123.0:20240222-115537.log (build.log.bz2,916.01 KB, application/x-bzip2)
2024-02-22 13:25 UTC, Andreas Thalhammer
Details
emerge info on the system where the build works (emerge-working.info,8.36 KB, text/plain)
2024-03-12 10:55 UTC, rodolfo
Details
emerge info on the system where the build breaks (emerge-broken.info,8.00 KB, text/plain)
2024-03-12 10:57 UTC, rodolfo
Details
emerge--info--firefox (emerge--info--firefox,20.68 KB, text/plain)
2024-04-24 19:55 UTC, demo
Details
compressed build log (firefox-125.0.2:20240424-143613.log.xz,513.37 KB, text/plain)
2024-04-24 19:56 UTC, demo
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Penßel 2024-02-20 19:23:42 UTC
Created attachment 885535 [details]
full build log (zstd compressed)

final log snippet:

73:44.54 Failed to hack libxul.so: basic_ios::clear: iostream error
73:44.54 x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)
73:44.54 gmake[4]: *** [/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox-123.0/config/rules.mk:541: libxul.so] Error 1
73:44.54 gmake[4]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox_build/toolkit/library/build'
73:44.54 gmake[3]: *** [/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox-123.0/config/recurse.mk:72: toolkit/library/build/target] Error 2
73:44.54 gmake[3]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox_build'
73:44.54 gmake[2]: *** [/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox-123.0/config/recurse.mk:34: compile] Error 2
73:44.54 gmake[2]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox_build'
73:44.55 gmake[1]: *** [/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox-123.0/config/rules.mk:361: default] Error 2
73:44.55 gmake[1]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-123.0/work/firefox_build'
73:44.55 gmake: *** [client.mk:60: build] Error 2
73:44.55 W 156 compiler warnings present.

$ emerge --info firefox
Portage 3.0.61 (python 3.11.8-final-0, default/linux/amd64/17.1/desktop/plasma/systemd/merged-usr, gcc-13, glibc-2.39, 6.7.5-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-6.7.5-gentoo-x86_64-12th_Gen_Intel-R-_Core-TM-_i5-1240P-with-glibc2.39
KiB Mem:    32710916 total,   9311232 free
KiB Swap:   25165816 total,  25154296 free
Timestamp of repository gentoo: Tue, 20 Feb 2024 17:34:07 +0000
Head commit of repository gentoo: c2aac933e187159fdfbed08ea987700872fd748f

Timestamp of repository guru: Tue, 20 Feb 2024 09:03:11 +0000
Head commit of repository guru: 2b4b499a6f6a14e6fcbbdbb132c53134a220063a

Timestamp of repository kde: Sat, 17 Feb 2024 20:48:15 +0000
Head commit of repository kde: 815b3c1b090ff03d5b1a4cf48fd1599d426de029

sh bash 5.2_p26
ld GNU ld (Gentoo 2.42 p3) 2.42.0
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p26::gentoo
dev-build/autoconf:        2.13-r8::gentoo, 2.72-r1::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.28.3::gentoo
dev-build/libtool:         2.4.7-r2::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.3.2::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r1::gentoo
dev-lang/python:           3.11.8_p1::gentoo
dev-lang/rust:             1.75.0-r1::gentoo
sys-apps/baselayout:       2.14-r2::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-apps/systemd:          255.3::gentoo
sys-devel/binutils:        2.42-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           17.0.6::gentoo
sys-devel/gcc:             13.2.1_p20240210::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/lld:             17.0.6::gentoo
sys-devel/llvm:            17.0.6::gentoo
sys-kernel/linux-headers:  6.7::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000
    volatile: False

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo
    volatile: False

kde
    location: /var/db/repos/kde
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/kde.git
    masters: gentoo
    volatile: False

local
    location: /var/db/repos/local
    masters: gentoo
    priority: 10
    volatile: False

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -Wall -g1 -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -flto=auto -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -Wall -g1 -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -flto=auto -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
F77FLAGS="-O2 -pipe -Wall -g1 -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -flto=auto -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing"
FCFLAGS="-O2 -pipe -Wall -g1 -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -flto=auto -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news nodoc noinfo parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -Wall -g1 -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -flto=auto -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="/usr/share/gtk-doc/html /usr/share/help /usr/share/locale/a* /usr/share/locale/b* /usr/share/locale/c* /usr/share/locale/da* /usr/share/locale/dv* /usr/share/locale/dz* /usr/share/locale/ee* /usr/share/locale/el* /usr/share/locale/en_C* /usr/share/locale/en_A* /usr/share/locale/eo* /usr/share/locale/es* /usr/share/locale/et* /usr/share/locale/eu* /usr/share/locale/f* /usr/share/locale/g* /usr/share/locale/h* /usr/share/locale/i* /usr/share/locale/j* /usr/share/locale/k* /usr/share/locale/la /usr/share/locale/lg /usr/share/locale/li /usr/share/locale/ln /usr/share/locale/lo /usr/share/locale/lt* /usr/share/locale/lv* /usr/share/locale/m* /usr/share/locale/n* /usr/share/locale/o* /usr/share/locale/p* /usr/share/locale/q* /usr/share/locale/r* /usr/share/locale/s* /usr/share/locale/t* /usr/share/locale/u* /usr/share/locale/v* /usr/share/locale/w* /usr/share/locale/x* /usr/share/locale/y* /usr/share/locale/z* "
LANG="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--build-id=sha1"
LEX="flex"
LINGUAS="en en_GB en_US de de_DE"
MAKEOPTS="-j16"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RUSTFLAGS="-Ctarget-cpu=x86-64-v3 -Clink-arg=-Wl,-z,pack-relative-relocs -Clink-arg=-Wl,-z,relro -Clink-arg=-Wl,-z,now -Copt-level=3 -Cdebuginfo=1 -Cforce-frame-pointers=yes -Ccodegen-units=1"
SHELL="/bin/zsh"
USE="X a52 aac acl acpi activities aio alsa amd64 apparmor archive audit bash-completion bluetooth boost bpf branding brotli btrfs bzip2 cairo caps cdr cet cjk clang cli cloudproviders colord context crypt cups curl cxx dbus declarative device-mapper djvu drm dts duktape dvdr elf encode exif ffmpeg firewalld flac flatpak fontconfig fontforge fonts fortran freetype fuse gbm gdbm gdk-pixbuf geoclue geolocation gif gles2 gmp gsettings gssapi gtk gui harfbuzz highlight iconv icu idn io-uring ios ipv6 jit jpeg jpegxl json kde kerberos keyutils kf6compat ktls kwallet ladspa latex lcms ldap libevent libidn2 libnotify libproxy libssh2 libtirpc libxml2 lmdb lto lv2 lz4 lzma lzo mad mng modemmanager mp3 mp4 mpeg mtp native-extensions natspec ncurses netlink networkmanager nfs nftables nls ogg opencl openconnect opengl openh264 openmp opus pam pango pcap pcre pcre2 pdf perl pgo pic pie pim pipewire pkcs11 plasma plymouth png policykit postproc ppds psl pulseaudio qml qsv qt6 raw readline rubberband samba sasl scanner screencast sdl seccomp secureboot semantic-desktop sound spell sqlite ssl sssd startup-notification stemmer sudo svg svt-av1 system-sqlite systemd systemtap taglib tdb telemetry test-rust theora threads tiff tpm truetype twolame udev udisks uki ukify unicode unwind upower usb uuid v4l vaapi verify-sig vorbis vulkan wayland webengine webp widgets wifi wxwidgets x264 xattr xcb xetex xft xml xv xvid xxhash zeroconf zlib zstd" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 ssse3 vpclmulqdq" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GRUB_PLATFORMS="efi-64 pc coreboot" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-US en-GB de de-DE" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby33" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, FC, GCOV, GPROF, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

www-client/firefox-122.0.1::gentoo was built with the following:
USE="X clang dbus gmp-autoupdate jumbo-build libproxy lto openh264 pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp telemetry wayland wifi -debug -eme-free -geckodriver -hardened -hwaccel -jack (-selinux) -sndio -system-png (-system-python-libs) (-valgrind)" L10N="de en-GB -ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -ca-valencia -cak -cs -cy -da -dsb -el -en-CA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fur -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -ne -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -sc -sco -si -sk -sl -son -sq -sr -sv -szl -ta -te -th -tl -tr -trs -uk -ur -uz -vi -xh -zh-CN -zh-TW"
CFLAGS="-pipe -Wall -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -mno-omit-leaf-frame-pointer"
CXXFLAGS="-pipe -Wall -fno-omit-frame-pointer -march=x86-64-v3 -fcf-protection -mno-omit-leaf-frame-pointer"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--build-id=sha1 -Wl,-z,nopack-relative-relocs -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"
Comment 1 Johannes Penßel 2024-02-20 20:40:29 UTC
Builds fine with USE=-pgo.
Comment 2 Johannes Penßel 2024-02-20 20:48:23 UTC
Builds fine with USE=-pgo.
Comment 3 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2024-02-20 20:49:21 UTC
[1003258.958377] llvm-worker-6[154725]: segfault at 60 ip 00007f2c2f99409b sp 00007f2bda9fa080 error 4 in libLLVM-17.so[7f2c2deb8000+3f31000] likely on CPU 15 (core 7, socket 0)
[1003258.958403] Code: c6 48 89 85 f8 f6 ff ff 48 8b 87 28 09 00 00 48 8d bd 58 f7 ff ff 48 89 85 50 f7 ff ff c5 f9 7f 85 40 f7 ff ff e8 65 16 b5 fe <49> 8b 47 60 48 89 85 00 f7 ff ff 48 85 c0 74 35 48 8d bd 40 f7 ff

encountered this here as well

building with 
X clang dbus gmp-autoupdate hwaccel jack jumbo-build libproxy lto pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-png system-webp wayland
Comment 4 Joonas Niilola gentoo-dev 2024-02-21 06:25:32 UTC
(In reply to Johannes Penßel from comment #0)
> LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs
> -Wl,--build-id=sha1 -Wl,-z,nopack-relative-relocs
> -Wl,--compress-debug-sections=zlib
> -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"

Try with sane LDFLAGS, I'm pretty sure "-z,nopack-relative-relocs" doesn't work.
Comment 5 Johannes Penßel 2024-02-21 06:42:34 UTC
(In reply to Joonas Niilola from comment #4)
> (In reply to Johannes Penßel from comment #0)
> > LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs
> > -Wl,--build-id=sha1 -Wl,-z,nopack-relative-relocs
> > -Wl,--compress-debug-sections=zlib
> > -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"
> 
> Try with sane LDFLAGS, I'm pretty sure "-z,nopack-relative-relocs" doesn't
> work.

While it looks kinda weird, -z,nopack-relative-relocs is a valid linker flag. (see ld man page) I've been using it for over a year with fireox only because global -z,pack-relative-relocs causes its build to fail.
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-02-21 06:47:35 UTC
It is indeed a real flag, although nowadays at least, FF builds fine for me with DT_RELR.
Comment 7 Joonas Niilola gentoo-dev 2024-02-21 07:05:32 UTC
(In reply to Johannes Penßel from comment #5)
> 
> While it looks kinda weird, -z,nopack-relative-relocs is a valid linker
> flag. (see ld man page) I've been using it for over a year with fireox only
> because global -z,pack-relative-relocs causes its build to fail.

I'm aware, but it's broke building firefox before. As I managed to build 123.0 fine with +pgo (both +clang -clang) I suspect that could be the difference.
Comment 8 Anders Larsson 2024-02-21 08:42:37 UTC
Created attachment 885587 [details]
another emerge --info www-client/firefox

Hello, I'm also getting the same linking errors with defaults(?):
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"
Comment 9 Joonas Niilola gentoo-dev 2024-02-21 08:51:29 UTC
(In reply to Anders Larsson from comment #8)
> Created attachment 885587 [details]
> another emerge --info www-client/firefox
> 
> Hello, I'm also getting the same linking errors with defaults(?):
> LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--compress-debug-sections=zlib
> -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"

Full build.log please.

Also what are your mesa use flags? Post

 emerge -pv mesa --nodeps
Comment 10 Anders Larsson 2024-02-21 09:04:31 UTC
Created attachment 885595 [details]
Build logs (xz compressed)

[ebuild   R    ] media-libs/mesa-24.0.1::gentoo  USE="X gles2 (opengl) proprietary-codecs vdpau wayland zstd -d3d9 -debug -gles1 -llvm -lm-sensors -opencl -osmesa (-selinux) -test -unwind -vaapi -valgrind -vulkan -vulkan-overlay -xa (-zink)" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="sse2" LLVM_SLOT="17 -15 -16" VIDEO_CARDS="-d3d12 (-freedreno) -intel -lavapipe (-lima) -nouveau (-panfrost) -r300 -r600 -radeon -radeonsi (-v3d) (-vc4) -virgl (-vivante) -vmware" 19,484 KiB
Comment 11 Johannes Penßel 2024-02-21 09:09:24 UTC
(In reply to Joonas Niilola from comment #7)
> (In reply to Johannes Penßel from comment #5)
> > 
> > While it looks kinda weird, -z,nopack-relative-relocs is a valid linker
> > flag. (see ld man page) I've been using it for over a year with fireox only
> > because global -z,pack-relative-relocs causes its build to fail.
> 
> I'm aware, but it's broke building firefox before. As I managed to build
> 123.0 fine with +pgo (both +clang -clang) I suspect that could be the
> difference.


With -z,nopack-relative-relocs removed, the PGO build still fails, unfortunately. 
(With USE=-pgo, it works just fine now. Thanks for the info, Sam!)

Looking through the complete list of closed tickets for FF123, this one looks like a potential culprit to me:
https://bugzilla.mozilla.org/show_bug.cgi?id=1839832

Apparently, Firefox now uses "temporal instrumentation" (-pgo-temporal-instrumentation flag) for PGO if supported by the compiler. Maybe that is causing issues somehow.
Comment 12 Joonas Niilola gentoo-dev 2024-02-21 09:28:28 UTC
(In reply to Anders Larsson from comment #10)
> Created attachment 885595 [details]
> Build logs (xz compressed)
> 

For you, always try without ccache/sccache if a build fails. Although since you're getting the same error, I doubt it's going to fix this. But just in general.


(In reply to Johannes Penßel from comment #11)
> 
> Looking through the complete list of closed tickets for FF123, this one
> looks like a potential culprit to me:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1839832
> 
> Apparently, Firefox now uses "temporal instrumentation"
> (-pgo-temporal-instrumentation flag) for PGO if supported by the compiler.
> Maybe that is causing issues somehow.

Hmm, there are some regressions listed which have patches, but since they were able to build 123.0 upstream and I was able to build it, there has to be something different "locally" that breaks it.
Comment 13 Andreas Thalhammer 2024-02-22 08:55:19 UTC
Created attachment 885674 [details]
www-client:firefox-123.0:20240221-234927.log

I got the same error with those LDFLAGS:

LDFLAGS="-Wl,-O2 -Wl,--as-needed -Wl,--sort-common -Wl,--hash-style=both -Wl,-z,relro -Wl,-z,now -fstack-protector-strong -fno-plt -fexceptions -fcf-protection"

Full build log attached.

I didn't have any problems with previous versions, with clang. I recompiled 123.0 with USE="-clang", which worked.
Comment 14 Andreas Thalhammer 2024-02-22 09:01:49 UTC
Created attachment 885678 [details]
emerge --info


The useflags are (not all in make.conf, some are by package):

[ebuild   R   ~] www-client/firefox-123.0:rapid::gentoo  USE="X clang* dbus gmp-autoupdate hardened hwaccel jumbo-build libproxy lto openh264 pgo pulseaudio sndio system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-png system-webp wayland wifi -debug -eme-free -geckodriver -jack (-selinux) (-system-av1) (-system-python-libs) -telemetry (-valgrind)" L10N="de -ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -ca-valencia -cak -cs -cy -da -dsb -el -en-CA -en-GB -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fur -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -ne -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -sc -sco -si -sk -sl -son -sq -sr -sv -szl -ta -te -th -tl -tr -trs -uk -ur -uz -vi -xh -zh-CN -zh-TW" LLVM_SLOT="17 -16" 0 KiB

(NOTE: clang is marked* as "new" useflag, because it installed with USE="-clang" for now...)
Comment 15 Joonas Niilola gentoo-dev 2024-02-22 09:24:52 UTC
I wonder if it worked with llvm-16 instead when using +clang?
Comment 16 Andreas Thalhammer 2024-02-22 09:33:59 UTC
(In reply to Joonas Niilola from comment #15)
> I wonder if it worked with llvm-16 instead when using +clang?

... which would lead to also downgrading rust. I checked, and decided to go with -clang.

# LLVM_SLOT="16" emerge -pv www-client/firefox

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 6.24 s (backtrack: 0/20).

[ebuild     UD ] dev-lang/rust-1.71.1:stable/1.71::gentoo [1.74.1:stable/1.74::gentoo] USE="lto verify-sig (-big-endian) -clippy -debug -dist -doc (-llvm-libunwind) (-miri) (-nightly) (-parallel-compiler) -profiler -rust-analyzer -rust-src -rustfmt (-system-bootstrap) (-system-llvm) -test -wasm" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" LLVM_TARGETS="AMDGPU BPF (X86) -AArch64 -ARM -AVR -Hexagon -Lanai -LoongArch -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -VE -WebAssembly -XCore (-ARC%) (-CSKY%) (-DirectX%) (-M68k%) (-SPIRV%) (-Xtensa%)" 308.049 KiB
[ebuild     UD ] virtual/rust-1.71.1-r1:0/llvm-16::gentoo [1.74.1:0/llvm-17::gentoo] USE="-rustfmt" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild   R   ~] www-client/firefox-123.0:rapid::gentoo  USE="X clang* dbus gmp-autoupdate hardened hwaccel jumbo-build libproxy lto openh264 pgo pulseaudio sndio system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-png system-webp wayland wifi -debug -eme-free -geckodriver -jack (-selinux) (-system-av1) (-system-python-libs) -telemetry (-valgrind)" L10N="de -ach -af -an -ar -ast -az -be -bg -bn -br -bs -ca -ca-valencia -cak -cs -cy -da -dsb -el -en-CA -en-GB -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fur -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -ia -id -is -it -ja -ka -kab -kk -km -kn -ko -lij -lt -lv -mk -mr -ms -my -nb -ne -nl -nn -oc -pa -pl -pt-BR -pt-PT -rm -ro -ru -sc -sco -si -sk -sl -son -sq -sr -sv -szl -ta -te -th -tl -tr -trs -uk -ur -uz -vi -xh -zh-CN -zh-TW" LLVM_SLOT="16* -17*" 0 KiB

Total: 3 packages (2 downgrades, 1 reinstall), Size of downloads: 308.049 KiB


Do you want me to try it?
Comment 17 Johannes Penßel 2024-02-22 09:34:37 UTC
Looks like Fedora is also skipping PGO for this release because of build failures:
https://src.fedoraproject.org/rpms/firefox/c/6411e9e37788448993f6be62400bb6a27f9c94cc?branch=rawhide

(In reply to Joonas Niilola from comment #7)
> I'm aware, but it's broke building firefox before. As I managed to build
> 123.0 fine with +pgo (both +clang -clang) I suspect that could be the
> difference.

Would you mind posting your $ emerge --info www-client/firefox, please? I wonder if I can replicate your successful build if I try to match your settings.
Comment 18 Joonas Niilola gentoo-dev 2024-02-22 10:11:52 UTC
(In reply to Johannes Penßel from comment #17)
> Looks like Fedora is also skipping PGO for this release because of build
> failures:
> https://src.fedoraproject.org/rpms/firefox/c/
> 6411e9e37788448993f6be62400bb6a27f9c94cc?branch=rawhide

I'm pretty sure Fedora build failed during the profiling, not during linking. Canonical also had an issue with PGO and they forced software rendering to be used. They tracked this issue to an older mesa that they're currently shipping, so it could be related to Fedora's issues as well. We don't have that mesa even present in our repo anymore.


> 
> Would you mind posting your $ emerge --info www-client/firefox, please? I
> wonder if I can replicate your successful build if I try to match your
> settings.

It's done with the most basic settings and rust-bin. Latest ~unstable packages, so llvm-17 and clang-17 were used.
https://github.com/juippis/incus-gentoo-github-pullrequest-tester/tree/master/container/etc/portage

So it could be mesa, it could be upstream pgo-updates (and the new flag), it could be unique system-related stuff, it could be the ffvpx-av1 changes since the linking dies immediately after libmozavcodec... and I wonder if it could be related our wrong addpredict too.

I'm just throwing stuff because it worked for me and therefore I can't test a "fix". However would be great to eliminate one thing:
https://gitweb.gentoo.org/repo/gentoo.git/tree/www-client/firefox/firefox-123.0.ebuild#n552

add

  addpredict /dev

somewhere inside the "if use pgo ; then" block to test whether colon-limited addpredict is breaking stuff.
Comment 19 Joonas Niilola gentoo-dev 2024-02-22 10:14:47 UTC
(In reply to Joonas Niilola from comment #18)
> 
> I'm pretty sure Fedora build failed during the profiling, not during
> linking. Canonical also had an issue with PGO and they forced software
> rendering to be used. They tracked this issue to an older mesa that they're
> currently shipping, so it could be related to Fedora's issues as well. We
> don't have that mesa even present in our repo anymore.
> 
> 

Right and here's the link
https://raw.githubusercontent.com/canonical/firefox-snap/stable/patches/pgo-with-software-webrender.patch
Comment 20 Joonas Niilola gentoo-dev 2024-02-22 11:41:44 UTC
Oh hmm one more idea: could it be related to the new llvm-r1.eclass? If someone wants to diff 122.0.1 <-> 123.0 and revert llvm changes, using the old llvm.eclass instead, it could prove helpful too.
Comment 21 Andreas Thalhammer 2024-02-22 11:50:49 UTC
Created attachment 885708 [details]
www-client:firefox-123.0:20240222-103851.log

(In reply to Joonas Niilola from comment #18)
> add
> 
>   addpredict /dev
> 
> somewhere inside the "if use pgo ; then" block to test whether colon-limited
> addpredict is breaking stuff.

Did that (and ONLY that), didn't work. Build log attached, looks the same to me.
Comment 22 Andreas Thalhammer 2024-02-22 13:25:03 UTC
Created attachment 885713 [details]
www-client:firefox-123.0:20240222-115537.log

I added pgo-with-software-webrender.patch as a user patch (according to https://wiki.gentoo.org/wiki//etc/portage/patches) and recompiled. Funny enough, this worked:

 * Applying user patches from /etc/portage/patches ...
 * Applying pgo-with-software-webrender.patch ...
 [ ok ]
 * User patches applied.

Build log of successful build attached. What does this mean? Is it due to media-libs/mesa? I'm using currently stable version 23.3.5...

# emerge -pv media-libs/mesa

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 2.54 s (backtrack: 0/20).

[ebuild   R    ] media-libs/mesa-23.3.5::gentoo  USE="X d3d9 gles2 llvm lm-sensors osmesa proprietary-codecs vaapi vdpau vulkan wayland xa zstd -debug -gles1 -opencl (-selinux) -test -unwind -valgrind -vulkan-overlay (-zink)" ABI_X86="(64) -32 (-x32)" CPU_FLAGS_X86="sse2" VIDEO_CARDS="d3d12 nouveau radeon radeonsi (-freedreno) -intel -lavapipe (-lima) (-panfrost) -r300 -r600 (-v3d) (-vc4) -virgl (-vivante) -vmware" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

Why does it work without the patch for some, and what exactly does the patch fix? My build -- with the patch -- is no longer using hardware rendering, yes? So wouldn't it be preferred to use -clang with hardware rendering instead?

(As in important side note: the build log says that I've nouveau installed -- but it is not in use since my laptop has hybrid graphics and amdgpu is in use. Nouveau is installed, yes, but it is also disabled: the dedicated Nvidia graphics card is dormant...)
Comment 23 Johannes Penßel 2024-02-22 13:32:16 UTC
Adding addpredict /dev didn't work for me either, I'm afraid. Same thing with reverting to llvm.eclass. Currently building with the software webrender patch.
Comment 24 Johannes Penßel 2024-02-22 16:10:36 UTC
Can confirm, the patch does the trick! My mesa:

media-libs/mesa-24.0.1  USE="X gles2 llvm lm-sensors opencl (opengl) proprietary-codecs unwind vulkan wayland zstd -d3d9 -debug -gles1 -osmesa (-selinux) -test -vaapi -valgrind -vdpau -vulkan-overlay -xa (-zink)" CPU_FLAGS_X86="sse2" LLVM_SLOT="17 -15 -16" VIDEO_CARDS="intel -d3d12 (-freedreno) -lavapipe (-lima) -nouveau (-panfrost) -r300 -r600 -radeon -radeonsi (-v3d) (-vc4) -virgl (-vivante) -vmware"
Comment 25 Bernard Cafarelli gentoo-dev 2024-02-25 00:54:26 UTC
Using that software webrender patch also allows compilation with mesa[zink] (from bug #923054)
Comment 26 bob 2024-03-12 05:03:40 UTC
I compiled with the patch and firefox now segfaults.
Comment 27 bob 2024-03-12 05:04:37 UTC
To clarify, it segfaults on start, even on firefox --help.
Comment 28 bob 2024-03-12 08:25:16 UTC
ok that was unrelated to this patch
Comment 29 rodolfo 2024-03-12 10:54:47 UTC
I have 2 machines,
- one on ~amd64 where the build breaks
- one on stable with selected packages on ~ where the build works

I could use them to do some testing.
Comment 30 rodolfo 2024-03-12 10:55:28 UTC
Created attachment 887474 [details]
emerge info on the system where the build works
Comment 31 rodolfo 2024-03-12 10:57:47 UTC
Created attachment 887475 [details]
emerge info on the system where the build breaks

I got this file building with USE=-pgo. If I leave the pgo use flag on the build breaks with the llvm error as others reported.
Comment 32 12101111 2024-04-17 16:46:05 UTC
upstream issue: https://github.com/llvm/llvm-project/issues/84062
Comment 33 Joonas Niilola gentoo-dev 2024-04-18 07:05:28 UTC
Again, 125.0.1 with +pgo worked fine for me with -clang and +clang. I guess people still hitting this can compile with gcc (-clang use flag) or use the patch:
https://raw.githubusercontent.com/canonical/firefox-snap/stable/patches/pgo-with-software-webrender.patch

until upstream llvm is fixed? Note that llvm-18 can't be enabled on Firefox before a matching rust is bumped. rust-1.78 should be built with llvm-18.
Comment 34 Maxim P. Dementiev 2024-04-20 08:56:35 UTC
FYI: there is a discussion in https://github.com/llvm/llvm-project/issues/87894

My citation from there:

> In fact, seeing multilib in this output, I've got two hosts with Gentoo, and on one host it compiles, but on an other - it fails.
> The difference is in gentoo profiles:
>
> * default/linux/amd64/23.0/no-multilib - it compiles and works
> * default/linux/amd64/17.1/desktop - it fails.
>
> So, maybe the key of the problem is this multilib/no- problem.
Comment 35 demo 2024-04-24 19:55:07 UTC
Created attachment 891666 [details]
emerge--info--firefox
Comment 36 demo 2024-04-24 19:56:08 UTC
Created attachment 891667 [details]
compressed build log
Comment 37 demo 2024-04-24 19:58:03 UTC
Hello ...

The ubuntu patch didn't work for the 12.0.2 build. I've added some extra config options that I think shouldn't add to the problem, at least not reproduce the same exact result, namely "--enable-jemalloc --enable-replace-malloc --enable-tests --enable-rust-tests".