Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 837122 - www-client/firefox-99.0: Needs ~ 64 GB RAM to build with specific use flags
Summary: www-client/firefox-99.0: Needs ~ 64 GB RAM to build with specific use flags
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal with 1 vote (vote)
Assignee: Mozilla Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-07 16:41 UTC by Simeon Simeonov
Modified: 2022-04-13 18:07 UTC (History)
13 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (build.log.xz,844.03 KB, application/x-xz)
2022-04-07 16:44 UTC, Simeon Simeonov
Details
emerge --info (emergeinfo,7.55 KB, text/plain)
2022-04-09 20:02 UTC, Joe Kappus
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Simeon Simeonov 2022-04-07 16:41:35 UTC
Firefox 99.0 fails to build with clang13 and the following USE flags.

USE="clang dbus hwaccel lto openh264 pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp wayland -debug -eme-free -geckodriver -gmp-autoupdate -hardened -jack -libproxy -screencast (-selinux) -sndio -system-png -wifi"

complete build.log attached

102:52.52 clang-13: error: unable to execute command: Killed
102:52.57 clang-13: error: linker command failed due to signal (use -v to see invocation)
102:52.57 gmake[4]: *** [/var/tmp/portage/www-client/firefox-99.0/work/firefox-99.0/config/rules.mk:531: libxul.so] Error 254
102:52.57 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-99.0/work/firefox_build/toolkit/library/build'
102:52.57 gmake[3]: *** [/var/tmp/portage/www-client/firefox-99.0/work/firefox-99.0/config/recurse.mk:72: toolkit/library/build/target] Error 2
102:52.60 gmake[3]: Leaving directory '/var/tmp/portage/www-client/firefox-99.0/work/firefox_build'
102:52.61 gmake[2]: *** [/var/tmp/portage/www-client/firefox-99.0/work/firefox-99.0/config/recurse.mk:34: compile] Error 2
102:52.64 gmake[2]: Leaving directory '/var/tmp/portage/www-client/firefox-99.0/work/firefox_build'
102:52.65 gmake[1]: *** [/var/tmp/portage/www-client/firefox-99.0/work/firefox-99.0/config/rules.mk:352: default] Error 2
102:52.67 gmake[1]: Leaving directory '/var/tmp/portage/www-client/firefox-99.0/work/firefox_build'
102:52.67 gmake: *** [client.mk:63: build] Error 2
102:52.71 247 compiler warnings present.
 * ERROR: www-client/firefox-99.0::gentoo failed (compile phase):
 *   Failed to run './mach build --verbose'
 * 


Reproducible: Always




Portage 3.0.30 (python 3.10.2-final-0, default/linux/amd64/17.1/no-multilib, gcc-11.2.1, glibc-2.34-r10, 5.17.1 x86_64)
=================================================================
System uname: Linux-5.17.1-x86_64-Intel-R-_Core-TM-_i7-5820K_CPU_@_3.30GHz-with-glibc2.34
KiB Mem:    16294892 total,  11272212 free
KiB Swap:   16777212 total,  16505940 free
Timestamp of repository gentoo: Thu, 07 Apr 2022 16:00:02 +0000
Head commit of repository gentoo: 6220200bfeaaaf53a956ee3a27710fc25b23e60e
Timestamp of repository sgs: Thu, 07 Apr 2022 10:05:46 +0000
Head commit of repository sgs: 48a37257aaf15e4ef9fa6c8c21ba82a85ae18843

sh bash 5.1_p16
ld GNU ld (Gentoo 2.37_p1 p2) 2.37
app-misc/pax-utils:        1.3.3::gentoo
app-shells/bash:           5.1_p16::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.34.0-r6::gentoo
dev-lang/python:           3.9.9-r1::gentoo, 3.10.2_p1::gentoo
dev-lang/rust:             1.59.0::gentoo
dev-util/cmake:            3.22.2::gentoo
dev-util/meson:            0.60.3::gentoo
sys-apps/baselayout:       2.7-r3::gentoo
sys-apps/openrc:           0.44.10::gentoo
sys-apps/sandbox:          2.29::gentoo
sys-devel/autoconf:        2.13-r1::gentoo, 2.71-r1::gentoo
sys-devel/automake:        1.16.4::gentoo
sys-devel/binutils:        2.37_p1-r2::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/clang:           13.0.1::gentoo, 14.0.0-r1::gentoo
sys-devel/gcc:             11.2.1_p20220115::gentoo
sys-devel/gcc-config:      2.5-r1::gentoo
sys-devel/libtool:         2.4.6-r6::gentoo
sys-devel/lld:             13.0.1::gentoo
sys-devel/llvm:            13.0.1::gentoo, 14.0.0::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  5.15-r3::gentoo (virtual/os-headers)
sys-libs/glibc:            2.34-r10::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 

sgs
    location: /var/db/repos/sgs
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/sgs.git
    masters: gentoo

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE all-rights-reserved free-noncomm unRAR NPSL NVIDIA-r2 Skype-TOS"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=haswell -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=haswell -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--autounmask=n"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="https://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/usr/portage/packages"
PORTAGE_BZIP2_COMMAND="/bin/bzip2"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac acpi alsa amd64 bash-completion bindist bluetooth bluray branding bzip2 cairo cdda cdr cli crypt cups custom-cflags dav1d dbus dri dts dvd dvdr egl elogind emboss encode exif fam ffmpeg flac fortran fuse gdbm gif glamor gles2 gpm gtk gtk3 gtk4 http2 iconv icu ipv6 jpeg jpeg2k lcms libglvnd libnotify libtirpc lto mad mng mp3 mp4 mpeg ncurses nls nouveau nptl ogg opengl openmp pam pango pcre pdf pgo png ppds python qt qt5 readline schroedinger sdl seccomp secure-delete smp spell split-usr ssl startup-notification svg theora threads threadsafe tiff truetype type1 udev unicode upower usb v4l v4l2 vdpau verify-sig vorbis vpx wavpack wayland wxwidgets x264 x265 xattr xcb xml xv xvid xvmc zlib {USE}" ABI_X86="64" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev keyboard libinput mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="luajit" LUA_TARGETS="luajit" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby26" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LEX, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, MAKEOPTS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 1 Simeon Simeonov 2022-04-07 16:44:37 UTC
Created attachment 769286 [details]
build.log
Comment 2 Ingo Brunberg 2022-04-07 18:20:13 UTC
I have had this, too. Despite having plenty of RAM, lld consumed more and more until the OOM killer kicked in.

If nobody has a better idea, I will perhaps try again without lto. Note that I do not have pgo enabled.
Comment 3 Simeon Simeonov 2022-04-07 19:18:14 UTC
USE="-clang" seems to fix the build for me
Comment 4 Christian Birchinger 2022-04-07 21:18:39 UTC
Getting hit by this too, having 16GB RAM with nothing else running on a system
that was previously able to build Firefox 98 even with /var/tmp mounted as zram
including pgo and lto support.

I re-tried with PORTAGE_TMPDIR set to a HDD to give it the maximum amount of ram.
The linker seems to loop until it fully runs out of memory and the OOM kicks in, regardless of the available memory.
Comment 5 cyrillic 2022-04-08 00:02:54 UTC
With 64GB of RAM, I get this result :
 9:49.78 Your build was successful!

I was watching "top", and memory used got up to 15-16GB during certain parts of the build. You may be able to squeak-by using some memory saving tricks like :

MAKEOPTS="-j1" and/or booting the machine with no GUI ...
Comment 6 Jan Psota 2022-04-08 06:44:08 UTC
Same here. Using clang, too. Can it be related with:
--with-libclang-path=/usr/lib/llvm/13/lib64
[...]
checking the target C compiler version... 14.0.0 ?

And the error is:
/usr/lib/llvm/14/bin/../../../../lib/clang/14.0.0/include/emmintrin.h:2378:19: error: use of undeclared identifier '__builtin_elementwise_max', err: true

No lto, 8GB RAM, no oom.
Comment 7 Ingo Brunberg 2022-04-08 08:01:02 UTC
Yes, I would also guess the problem has something to do with mixing llvm 13 and 14. And then there is the issue of having built rust with system-llvm, so being tied to lld-13.

So now I will go on with USE=-clang.
Comment 8 Marius Stoica 2022-04-08 09:46:05 UTC
lld seems to hang when trying to link libmozavcodec, so if any use flag disables avcodec that would be a workaround for this issue. I don't think any use flag does this (altough openh264, system-av1, system-libvpx and system-webp seem somewhat related...)

Maybe I'll try to switch to building without /var/tmp mounted as tmpfs or to USE="-clang" later this evening.
Comment 9 Maciej S. Szmigiero 2022-04-08 14:30:54 UTC
I an unable to build www-client/firefox-99.0[clang,lto] too.

Specifically, /usr/bin/ld.lld OOMs when linking libxul.so.

Build log snippets:
>  --with-libclang-path=/usr/lib/llvm/13/lib64  Gentoo default
> checking for the target C++ compiler... /usr/lib/llvm/13/bin/x86_64-pc-linux-gnu-clang++

Versions:
$ /usr/bin/ld.lld --version 
LLD 13.0.1 (compatible with GNU linkers)

[IP-] [  ] sys-devel/lld-13.0.1:0
[IP-] [  ] sys-devel/clang-11.1.0:11/11.1
[IP-] [  ] sys-devel/clang-13.0.1:13
[IP-] [  ] sys-devel/clang-14.0.0-r1:14
[IP-] [  ] sys-devel/llvm-11.1.0:11
[IP-] [  ] sys-devel/llvm-13.0.1:13
[IP-] [  ] sys-devel/llvm-14.0.0:14
Comment 10 Maciej S. Szmigiero 2022-04-08 20:17:09 UTC
> Specifically, /usr/bin/ld.lld OOMs when linking libxul.so.

Update: with enough swap it does eventually build successfully in this USE="clang lto" config.
The peak memory usage of the ld.lld process while linking libxul.so is about 40 GiB, however.
Comment 11 Joe Kappus 2022-04-09 01:28:54 UTC
I really think this is because of mixing lld-13 on a clang-14 system and the title should be updated to match that. 

Digging around mozilla's bugzilla and it doesn't appear like we should be blocking 14 in both cases (see: https://bugzilla.mozilla.org/show_bug.cgi?id=1758780 ). Doing a test compile right now and will open a new bug to lift the max slot if it succeeds.
Comment 12 Joe Kappus 2022-04-09 03:14:08 UTC
Nope, still uses obscene amounts of ram with lld-14, can't build it on a 32GB ram system.
Comment 13 rodolfo 2022-04-09 05:50:33 UTC
I have this problem on a system with only llvm and clang 13 so it doesn't seem a problem of conflicting llvm versions.
Comment 14 Ivan 2022-04-09 09:28:10 UTC
Managed to successfully build FF99 with the following USE flags with lld 14: 

clang dbus gmp-autoupdate hwaccel lto openh264 pgo pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-png system-webp wayland -debug -eme-free -geckodriver -hardened -jack -libproxy -screencast -selinux -sndio -wifi

emerge --info here: https://pastebin.com/JWPK0hzs

Basically what I did: I changed the ebuild manually to accept lld 14 with simple digit swaps.

Also guess that rust-1.60 binary played some role too.

I have 32G RAM and set tmp ZRAM to '36G'. Peak mem consumption was about 29-30GB (KDE Desktop + Google Chrome with few tabs to watch YT) and peak amount of data in tmp was about 5G compressed (which is 12G+ uncompressed). Guess that saved me from running out of RAM.

Building process took about 49-50 mins, when usually it was 33-34 mins, and lld phase (or what it is) was ~15 minutes, while lld ate 66% of available RAM. Did it twice.
Comment 15 Михаил 2022-04-09 10:19:25 UTC
sys-devel/clang-13.0.1
dev-lang/rust-1.59.0

www-client/firefox-99.0::gentoo was built with the following:
USE="clang dbus geckodriver gmp-autoupdate hardened hwaccel lto openh264 pulseaudio screencast system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-png system-webp wayland wifi -debug -eme-free -jack -libproxy -pgo (-selinux) -sndio" ABI_X86="(64)"


16g ram + 18g swap is not enough to link libxul.so

Adding 10g swap file solves link issue.

Peak ld.ldd memory comsumption near to 30G

PID  %MEM    VIRT   SWAP    RES   CODE    DATA    SHR nMaj nDRT  %CPU COMMAND                                                                                                                                                           
***  78,6   28,8g   15,2g  12,1g   0,0m   27,9g 136,8m 3,3m    0 197,4 /usr/bin/ld.lld --eh-frame-hdr -m elf_x86_64 -shared -o libxul.so ...
Comment 16 Sebastian Doering 2022-04-09 11:05:03 UTC
Just another voice in the chorus: Ran into the same system.

Was able to get it compiled by adding >20 GB swapfile.
Comment 17 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2022-04-09 14:38:32 UTC
Maybe it's time to add

  append-ldflags "-Wl,--no-keep-memory"

somewhere in the ebuild. This memory requirement is insane...
Comment 18 Maciej S. Szmigiero 2022-04-09 15:08:07 UTC
(In reply to Lars Wendler (Polynomial-C) from comment #17)
> Maybe it's time to add
> 
>   append-ldflags "-Wl,--no-keep-memory"
> 
> somewhere in the ebuild.

Does this flag even exist for lld?
Comment 19 Larry the Git Cow gentoo-dev 2022-04-09 15:25:54 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd5635e0c7269edb0261a35ff9e779cfc26288e9

commit fd5635e0c7269edb0261a35ff9e779cfc26288e9
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2022-04-09 15:25:02 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2022-04-09 15:25:02 +0000

    www-client/firefox: enable llvm:14 for 99.0
    
    Bug: https://bugs.gentoo.org/836587
    Bug: https://bugs.gentoo.org/837122
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 www-client/firefox/firefox-99.0.ebuild | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Comment 20 Joonas Niilola gentoo-dev 2022-04-09 15:29:32 UTC
So I haven't been able to reproduce this. When I pushed firefox-99, I had rust-1.59 built with llvm:13 stack and nothing from llvm:14 installed. Today, I've done multiple clang+lto+pgo runs while testing llvm:14, and the container capped at 22 GiB memory usage with MAKEOPTS="-j16". The lowest was 19 GiB.

And while that seems higher than ever, and higher compared to ESR, it's not out of the ordinary for linking to require 1 GiB memory per thread. And for me it seems to be pretty close to 1 GiB, maybe few hundred MB higher.

So I hope this has been caused by some mismatched versions between rust, llvm, clang and lld - which wouldn't be the first time it's caused troubles...
Comment 21 Joe Kappus 2022-04-09 20:02:27 UTC
Created attachment 769703 [details]
emerge --info

Since you can't reproduce please include your emerge --info. 

I managed to get it compiled disabling tmpfs and having things be clang/lld-14 pure and not mixed. Still, it cuts it close on a 32GB system without swap.
Comment 22 Perfect Gentleman 2022-04-10 03:13:17 UTC
(In reply to Joonas Niilola from comment #20)
> So I haven't been able to reproduce this. When I pushed firefox-99, I had
> rust-1.59 built with llvm:13 stack and nothing from llvm:14 installed.

I have Rust-1.60, LLVM-14. Cannot build Firefox-99 with LTO. Have no problems with <=FF-98.0.2.
IMO, that Mozilla's issue.
Comment 23 Joonas Niilola gentoo-dev 2022-04-10 09:36:15 UTC
(In reply to Perfect Gentleman from comment #22)
> (In reply to Joonas Niilola from comment #20)
> > So I haven't been able to reproduce this. When I pushed firefox-99, I had
> > rust-1.59 built with llvm:13 stack and nothing from llvm:14 installed.
> 
> I have Rust-1.60, LLVM-14. Cannot build Firefox-99 with LTO. Have no
> problems with <=FF-98.0.2.
> IMO, that Mozilla's issue.

Yeah, Firefox-ESR capped at <7 GB with clang+lto+pgo. Could you just for reference tell the max amount of RAM used with 98.0.2?

99.0 got stuck linking libxul.so for 20 minutes which also raised memory usage to 20 GB, but it passed and the browser runs fine. So there's definitely something going on in there. 

FYI I've disabled domstreams, but it didn't seem to affect the huge RAM usage when building Firefox.
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8bc3be8f915ac1fe436546c0b007c7e293600b9f

I'm gonna get a new build.log and flag a Mozilla ticket about this. Weird that no other distro seems to be suffering from this, we can't be the only ones using clang+lld to build Firefox. So I fear it's something to do with our rust+lld connections.
Comment 24 Florian K. 2022-04-10 09:44:45 UTC
I think the issue might be introduced by one of the patches from the new patchset of firefox 99.

When there was no official ebuild available I copied firefox-98.0.2.ebuild to a local repository and renamed it to firefox-99.0.ebuild.
Then I removed 0033-resolve-fs-symlinks-bmo1753182.patch from the firefox-98-patches-04j-org.tar.xz patchset as it did not apply to firefox 99.
Compiling firefox 99 using the reduced patchset from firefox 98 with clang and lto works fine without using a huge amount of memory.

When I try to emerge firefox 99 using the official ebuild and official patchset one lld process takes forever and fails with huge memory consumption:

/usr/bin/ld.lld --eh-frame-hdr -m elf_x86_64 -shared -o libxul.so /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.1/../../../../lib64/crti.o /usr/lib/llvm/14/bin/../../../../lib/clang/14.0.0/lib/linux/clang_rt.crtbegin-x86_64.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.1 -L/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.1/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/11.2.1/../../../../x86_64-pc-linux-gnu/lib -L/lib -L/usr/lib -plugin-opt=mcpu=znver3 -plugin-opt=O2 -plugin-opt=-function-sections -plugin-opt=-data-sections -z defs --gc-sections -h libxul.so /mnt/portage/compilespace/portage/www-client/firefox-99.0/work/firefox_build/toolkit/library/build/libxul_so.list -plugin-opt=-import-instr-limit=10 -plugin-opt=-import-hot-multiplier=30 -lpthread -O2 --as-needed -z relro -z now -z relro -z now --compress-debug-sections=zlib -rpath=/usr/lib64/firefox --enable-new-dtags -z noexecstack -z text -z relro -z nocopyreloc -Bsymbolic-functions -rpath-link /mnt/portage/compilespace/portage/www-client/firefox-99.0/work/firefox_build/dist/bin -rpath-link /usr/lib ../../../js/src/build/libjs_static.a /mnt/portage/compilespace/portage/www-client/firefox-99.0/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a ../../../security/sandbox/linux/libmozsandbox.so ../../../config/external/lgpllibs/liblgpllibs.so ../../../config/external/sqlite/libmozsqlite3.so ../../../widget/gtk/mozgtk/libmozgtk.so --version-script symverscript -licui18n -licuuc -licudata -laom -ldav1d -lasound -lrt -lm -ldl -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrandr -lXrender -lXtst -lpthread -lc -lffi -lplds4 -lplc4 -lnspr4 -lz -lssl3 -lsmime3 -lnss3 -lnssutil3 -lfreetype -lfontconfig -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgraphite2 -lwebpdemux -lwebp -levent -lvpx -lpixman-1 -ldbus-glib-1 -ldbus-1 -lxcb-shm -lX11-xcb -lxcb -lXcursor -lXi -lstdc++ -lm /usr/lib/llvm/14/bin/../../../../lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a -l:libunwind.so -lpthread -lc /usr/lib/llvm/14/bin/../../../../lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a -l:libunwind.so /usr/lib/llvm/14/bin/../../../../lib/clang/14.0.0/lib/linux/clang_rt.crtend-x86_64.o /usr/lib/gcc/x86_64-pc-linux-gnu/11.2.1/../../../../lib64/crtn.o

Firefox is build with "clang dbus gmp-autoupdate hardened hwaccel lto openh264 pulseaudio system-av1 system-harfbuzz system-icu system-jpeg system-libevent system-libvpx system-webp -debug -eme-free -geckodriver -jack -libproxy -pgo -screencast (-selinux) -sndio -system-png -wayland -wifi"

app-misc/pax-utils:        1.3.3::gentoo
app-shells/bash:           5.1_p16::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.34.1::gentoo
dev-lang/python:           3.9.12::gentoo, 3.10.4::gentoo
dev-lang/rust:             1.60.0::gentoo
dev-util/cmake:            3.23.0::gentoo
dev-util/meson:            0.61.4-r2::gentoo
sys-apps/baselayout:       2.8::gentoo
sys-apps/sandbox:          2.29::gentoo
sys-apps/systemd:          250.4-r1::gentoo
sys-devel/autoconf:        2.13-r1::gentoo, 2.71-r1::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.38-r1::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/clang:           14.0.0-r1::gentoo
sys-devel/gcc:             11.2.1_p20220115::gentoo
sys-devel/gcc-config:      2.5-r1::gentoo
sys-devel/libtool:         2.4.7::gentoo
sys-devel/lld:             14.0.0::gentoo
sys-devel/llvm:            14.0.0::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  5.17::gentoo (virtual/os-headers)
sys-libs/glibc:            2.35-r2::gentoo
Comment 25 Joonas Niilola gentoo-dev 2022-04-10 11:11:34 UTC
(In reply to Florian K. from comment #24)
> I think the issue might be introduced by one of the patches from the new
> patchset of firefox 99.
> 

What a great hint. We have 3 patches for gcc+pgo and I had to rebase them for 99.0 because gcc+pgo failed, so I may have missed something there. On it!
Comment 26 Daniel Harding 2022-04-10 11:18:59 UTC
So just looking at the patch listings, there is one new patch for Firefox 99, 0031-pgo-use-toolchain-disable-watchdog-fix-on-gcc.patch, which seems to change -flto=thin to -flto in build/moz.configure/lto-pgo.configure.  I'm trying a rebuild right now with just those lines from that patch reverted and will I report when finished (previously, I could not successfully link libxul for Firefox 99 even with 32GB RAM and 32GB swap).
Comment 27 Joonas Niilola gentoo-dev 2022-04-10 11:27:51 UTC
Yep, can already say that's extra. Gonna do runs with clang/gcc +lto +pgo and see what happens.
Comment 28 Daniel Harding 2022-04-10 11:52:46 UTC
So dropping -flto lines from the 0031-pgo-use-toolchain-disable-watchdog-fix-on-gcc.patch was sufficient for firefox-99 to build for me.
Comment 29 Perfect Gentleman 2022-04-10 12:47:05 UTC
(In reply to Joonas Niilola from comment #23)
> (In reply to Perfect Gentleman from comment #22)
> > (In reply to Joonas Niilola from comment #20)
> > > So I haven't been able to reproduce this. When I pushed firefox-99, I had
> > > rust-1.59 built with llvm:13 stack and nothing from llvm:14 installed.
> > 
> > I have Rust-1.60, LLVM-14. Cannot build Firefox-99 with LTO. Have no
> > problems with <=FF-98.0.2.
> > IMO, that Mozilla's issue.
> 
> Yeah, Firefox-ESR capped at <7 GB with clang+lto+pgo. Could you just for
> reference tell the max amount of RAM used with 98.0.2?
> 
> 99.0 got stuck linking libxul.so for 20 minutes which also raised memory
> usage to 20 GB, but it passed and the browser runs fine. So there's
> definitely something going on in there. 
> 
> FYI I've disabled domstreams, but it didn't seem to affect the huge RAM
> usage when building Firefox.
> https://gitweb.gentoo.org/repo/gentoo.git/commit/
> ?id=8bc3be8f915ac1fe436546c0b007c7e293600b9f
> 
> I'm gonna get a new build.log and flag a Mozilla ticket about this. Weird
> that no other distro seems to be suffering from this, we can't be the only
> ones using clang+lld to build Firefox. So I fear it's something to do with
> our rust+lld connections.

Don't know. I've 16GB of RAM, portage_tmp_dir in tmpfs. Built FF with LTO+PGO. Before 99 it was okay.
Comment 30 Larry the Git Cow gentoo-dev 2022-04-10 12:54:42 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=7ed1fb8a42366e60ea020b1dc40983dd7b414243

commit 7ed1fb8a42366e60ea020b1dc40983dd7b414243
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2022-04-10 12:53:21 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2022-04-10 12:54:40 +0000

    www-client/firefox: update patch set for 99.0
    
     - accidentally enabled full-lto instead of thinlto with clang, which
       requires tons more of RAM. With bfd (gcc) we already use
       '--no-keep-memory'.
    
    Closes: https://bugs.gentoo.org/837122
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 www-client/firefox/Manifest            | 2 +-
 www-client/firefox/firefox-99.0.ebuild | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 31 Joonas Niilola gentoo-dev 2022-04-10 12:56:23 UTC
Thanks! Here's an image to show the difference we had I guess.

https://bug1644409.bmoattachments.org/attachment.cgi?id=9156178
Comment 32 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2022-04-10 13:01:04 UTC
(In reply to Joonas Niilola from comment #31)
> Thanks! Here's an image to show the difference we had I guess.
> 
> https://bug1644409.bmoattachments.org/attachment.cgi?id=9156178

Thanks juippis!
Comment 33 Maciej S. Szmigiero 2022-04-10 13:09:20 UTC
(In reply to Larry the Git Cow from comment #30)
>      - accidentally enabled full-lto instead of thinlto with clang, which

So I guess at least these people who managed to build the package in its previous config (despite huge memory requirements) got themselves a really well optimized browser.
Comment 34 Михаил 2022-04-13 18:07:56 UTC
ld.lld: warning: Linking two modules of different target triples: '/var/tmp/portage/www-client/firefox-99.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a(encoding_c-49418a4a95d4a761.encoding_c.509369a0-cgu.0.rcgu.o at 97910880)' is 'x86_64-unknown-linux-gnu' whereas '/var/tmp/portage/www-client/firefox-99.0.1/work/firefox_build/toolkit/library/build/../../../dom/media/eme/Unified_cpp_dom_media_eme0.o' is 'x86_64-pc-linux-gnu'

This warnings during link occurs only if -flto=thin.

May be it is ld.ldd bug.