Created attachment 633866 [details] emerge --info I can build =www-client/firefox-75.0 with CLANG ang PGO, i.e. USE="clang pgo lto" using CCACHE, i.e. FEATURES=ccache, without any problems. However, on second run, i.e. when cache is already populated, build will fail because lld will segfault: > * ERROR: www-client/firefox-75.0::gentoo failed (compile phase): > * Failed to run './mach build --verbose' > * while error was > 9:35.79 gmake[4]: Entering directory '/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app' > 9:35.79 browser/app/firefox > 9:35.79 /usr/lib64/ccache/bin/x86_64-pc-linux-gnu-clang++ -std=gnu++17 -o ../../dist/bin/firefox -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Wunused-function -Wunused-variable -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-error=tautological-type-limit-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -fno-sized-deallocation -fno-aligned-new -pipe -march=znver1 -mtune=znver1 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -O3 -fomit-frame-pointer -funwind-tables -fprofile-use=/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/instrumented/merged.profdata -Wno-error=backend-plugin /var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app/firefox.list -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -Wl,--compress-debug-sections=zlib -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -fstack-protector-strong -rdynamic -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/dist/bin -Wl,-rpath-link,/usr/lib -fcolor-diagnostics -pie -ldl > 9:35.79 LLVM ERROR: Function Import: link error > 9:35.79 clang-10: error: linker command failed with exit code 1 (use -v to see invocation) > ================================================================= > Package Settings > ================================================================= > > dev-util/ccache-3.7.9::gentoo was built with the following: > USE="-test" ABI_X86="(64)" > FEATURES="multilib-strict binpkg-docompress strict sandbox merge-sync usersync binpkg-dostrip unmerge-orphans unmerge-logs userpriv distlocks protect-owned sfperms news qa-unresolved-soname-deps userfetch assume-digests config-protect-if-modified downgrade-backup binpkg-logs ccache ebuild-locks usersandbox preserve-libs cgroup fixlafiles unknown-features-warn pid-sandbox xattr parallel-fetch network-sandbox ipc-sandbox"
Created attachment 633868 [details] gdb.log When trying to re-run x86_64-pc-linux-gnu-clang++ with "-v" like told: > dev1 ~ # ulimit -c > unlimited > dev1 ~ # su portage --shell /bin/bash > portage@dev1 /root $ cd ~/www-client/firefox-75.0/work/firefox-75.0 > portage@dev1 ~/www-client/firefox-75.0/work/firefox-75.0 $ echo $PATH > /bin:/usr/bin > portage@dev1 ~/www-client/firefox-75.0/work/firefox-75.0 $ export PATH="/usr/lib64/ccache/bin:/usr/lib/llvm/10/bin:/var/tmp/portage/www-client/firefox-75.0/temp/python3.8/bin:/usr/lib/portage/python3.8/ebuild-helpers/xattr:/usr/lib/portage/python3.8/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/10/bin:/usr/lib/llvm/9/bin" > portage@dev1 ~/www-client/firefox-75.0/work/firefox-75.0 $ cd ff/browser/app > portage@dev1 ~/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app $ /usr/lib64/ccache/bin/x86_64-pc-linux-gnu-clang++ -v -std=gnu++17 -o ../../dist/bin/firefox -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Wunused-function -Wunused-variable -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-error=tautological-type-limit-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -fno-sized-deallocation -fno-aligned-new -pipe -march=znver1 -mtune=znver1 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -O3 -fomit-frame-pointer -funwind-tables -fprofile-use=/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/instrumented/merged.profdata -Wno-error=backend-plugin /var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app/firefox.list -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -Wl,--compress-debug-sections=zlib -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -fstack-protector-strong -rdynamic -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/dist/bin -Wl,-rpath-link,/usr/lib -fcolor-diagnostics -pie -ldl > clang version 10.0.0 > Target: x86_64-pc-linux-gnu > Thread model: posix > InstalledDir: /usr/lib/llvm/10/bin > Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0 > Candidate multilib: .;@m64 > Candidate multilib: 32;@m32 > Selected multilib: .;@m64 > "/usr/bin/ld.lld" -pie -export-dynamic --eh-frame-hdr -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o ../../dist/bin/firefox /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64/Scrt1.o /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/crtbeginS.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0 -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64 -L/usr/lib64/llvm/10/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../.. -L/lib -L/usr/lib -plugin /usr/lib64/llvm/10/bin/../lib64/LLVMgold.so -plugin-opt=mcpu=znver1 -plugin-opt=O3 -plugin-opt=thinlto -plugin-opt=-function-sections -plugin-opt=-data-sections -plugin-opt=cs-profile-path=/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/instrumented/merged.profdata /var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app/firefox.list -plugin-opt=-import-instr-limit=10 -lpthread -O1 --as-needed -rpath=/usr/lib64/firefox --enable-new-dtags --compress-debug-sections=zlib -z noexecstack -z text -z relro -z nocopyreloc -Bsymbolic-functions -rpath-link /var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/dist/bin -rpath-link /usr/lib -ldl -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64/crtn.o > LLVM ERROR: Function Import: link error > clang-10: error: unable to execute command: Segmentation fault (core dumped) > clang-10: error: linker command failed due to signal (use -v to see invocation) >
Tested with sys-devel/llvm, sys-devel/lld, sys-devel/clang at 9.0.1 and 10.0.0.
I'll try to reproduce locally as well. I wonder if it still happens for you in the chroot without -march= specific flags. New code in codegen tends to generate invalid code in very unexpected places.
On top of that can you attach your successful and failed build logs?
Created attachment 633958 [details] build.log from first successful build It's already built without custom CFLAGS. Note that USE=custom-cflags nor USE=custom-optimization is enabled. You only see "-march=znver1 -mtune=znver1" used for psutils but that doesn't matter.
Created attachment 633962 [details] build.log failed
Reproduced locally as well \o/ """ 42:54.79 /usr/lib/ccache/bin/x86_64-pc-linux-gnu-clang++ -std=gnu++17 -o ../../dist/bin/firefox -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Wunused-function -Wunused-variable -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-error=tautological-type-limit-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -fno-sized-deallocation -fno-aligned-new -pipe -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -O3 -fomit-frame-pointer -funwind-tables -fprofile-use=/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/instrumented/merged.profdata -Wno-error=backend-plugin /var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app/firefox.list -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -lpthread -Wl,-O1 -Wl,--as-needed -pipe -Wl,--hash-style=gnu -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -Wl,--compress-debug-sections=zlib -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -fstack-protector-strong -rdynamic -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/dist/bin -Wl,-rpath-link,/usr/lib -fcolor-diagnostics -pie -ldl 42:54.79 LLVM ERROR: Function Import: link error 42:54.79 clang-10: error: linker command failed with exit code 1 (use -v to see invocation) 42:54.79 gmake[4]: *** [/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/config/rules.mk:464: ../../dist/bin/firefox] Error 1 42:54.79 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/ff/browser/app' 42:54.79 gmake[3]: *** [/var/tmp/portage/www-client/firefox-75.0/work/firefox-75.0/config/recurse.mk:74: browser/app/target] Error 2 42:54.79 gmake[3]: *** Waiting for unfinished jobs.... """
Built llvm[debug] to get some insight. The error is in profiling data conflicts: Program aborted due to an unhandled Error: linking module flags 'ProfileSummary': IDs have conflicting values in '../../memory/build/Unified_cpp_memory_build0.o' and 'nsBrowserApp.o' ccache probably caches too much and/or llvm's "'ProfileSummary': IDs", whatever it is, are not deterministic across runs and state on disk.
Looks like semantics of -fprofile-use= flag is very different across gcc and clang: gcc assumes -fprofile-use= to be a directory with *.gcda files. clang assumes -fprofile-use= to be a single file in it's own format. ccache knows how to traverse .gcda files to build cache key. My hypothesis is that it does not take into account clang's file itself at all and fetches from cache objects against different profile. Will try to craft a small example.
(In reply to Sergei Trofimovich from comment #9) > Looks like semantics of -fprofile-use= flag is very different across gcc and > clang: > > gcc assumes -fprofile-use= to be a directory with *.gcda files. > clang assumes -fprofile-use= to be a single file in it's own format. > > ccache knows how to traverse .gcda files to build cache key. My hypothesis > is that it does not take into account clang's file itself at all and fetches > from cache objects against different profile. > > Will try to craft a small example. Filed upstream bug for ccache: https://github.com/ccache/ccache/issues/582 You might want to disable ccache for PGO meanwhile. I'll try to poke al LLVM's SIGSEGV if it's feasible to report a profile mismatch error instead of crashing like that.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5cce531e716803b47711303fa77d052a2e472e56 commit 5cce531e716803b47711303fa77d052a2e472e56 Author: Thomas Deutschmann <whissi@gentoo.org> AuthorDate: 2020-04-21 19:21:08 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2020-04-21 19:21:36 +0000 www-client/firefox: disable FEATURES=ccache with USE=clang and USE=pgo Bug: https://bugs.gentoo.org/718632 Package-Manager: Portage-2.3.99, Repoman-2.3.22 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> www-client/firefox/firefox-68.7.0.ebuild | 4 ++++ www-client/firefox/firefox-75.0.ebuild | 4 ++++ 2 files changed, 8 insertions(+)
> I'll try to poke al LLVM's SIGSEGV if it's feasible to report a profile > mismatch error instead of crashing like that. Filed llvm/lld bug as https://bugs.llvm.org/show_bug.cgi?id=45636
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0722911b79f21d7c5d574758ae147934aa5a992c commit 0722911b79f21d7c5d574758ae147934aa5a992c Author: Sergei Trofimovich <slyfox@gentoo.org> AuthorDate: 2020-06-22 21:16:08 +0000 Commit: Sergei Trofimovich <slyfox@gentoo.org> CommitDate: 2020-06-22 21:16:21 +0000 dev-util/ccache: bump up to 3.7.10 Among other things new ccache improves handling of profile handling on clang. Should help for cases like bug #718632 Reported-by: Thomas Deutschmann Bug: https://bugs.gentoo.org/718632 Package-Manager: Portage-2.3.101, Repoman-2.3.22 Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> dev-util/ccache/Manifest | 1 + dev-util/ccache/ccache-3.7.10.ebuild | 70 ++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+)
