Created attachment 705234 [details] emerge-info Portage is configured to "splitdebug -nostrip installsources". Debug information for firefox and its libraries appeared in /usr/lib64/debug, but: 1. Sources didn't installed 2. `.gnu_debuglink` in firefox binaries has only entries for Scrt|crti|crtn ~$ readelf -p .gnu_debuglink /usr/lib64/firefox/firefox String dump of section '.gnu_debuglink': [ 0] Scrt1.o.debug [ 12] ]fcrti.o.debug [ 27] ^crtn.o.debug [ 3a] NF ~$ readelf -p .gnu_debuglink /usr/lib64/firefox/libxul.so String dump of section '.gnu_debuglink': [ 0] crti.o.debug [ 13] ^crtn.o.debug [ 26] NF 3. Manual adding debug information in gdb through `add-symbol-file` has no effect, e.g. gdb didn't resolve function names in backtraces and didn't stop at breakpoints. Will attach full build log later.
Created attachment 705237 [details] emerge log
Created attachment 705240 [details] produced firefox executable
https://tinystash.undef.im/dl/4dEiqZwFrprZrpLwNgSKtKNrg7gWFfMeLYap8pQWyBE2MdgcfiAhtNCdi9CRAUY7AXBu1dicbEFvTnqCutz6TP6E.debug produced debug info for firefox executable (too large to attach on bugzilla)
Hrm, seems like adding correct debuglink with `objcopy --add-gnu-debuglink` to binary makes debugging work, please ignore third point.
Build log here: https://tinystash.undef.im/il/47YbyfYZ88QLTuredzDQbHn4SJuTbn7DdbToc15WoCvdk1MVwaMyMN19yaHWKuKyuBCoV1d3nqfN4s8uBN2K6Qpv.log Stripping phase looks interesting: strip: x86_64-pc-linux-gnu-strip --strip-unneeded -N __gentoo_check_ldflags__ -R .comment -R .GCC.command.line -R .note.gnu.gold-version /usr/lib64/firefox/liblgpllibs.so /usr/lib64/firefox/libmozavutil.so /usr/lib64/firefox/libmozavcodec.so /usr/lib64/firefox/libmozsandbox.so /usr/lib64/firefox/pingsender /usr/lib64/firefox/libmozsqlite3.so /usr/lib64/firefox/firefox /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/liblgpllibs.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libmozsandbox.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libmozavcodec.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/pingsender: DWARF version 0 unhandled/usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libmozavutil.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libmozsqlite3.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/firefox: DWARF version 0 unhandled x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stDBE7s1: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stHUleOi: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stk1qcSv: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/sttEqL5F: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/st1FcVkO: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stPUsmZX: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stbLUpad: debuglink section already exists /usr/lib64/firefox/libmozgtk.so /usr/lib64/firefox/libxul.so /usr/lib64/firefox/plugin-container /usr/lib64/firefox/gmp-clearkey/0.1/libclearkey.so /usr/lib64/firefox/gtk2/libmozgtk.so /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libmozgtk.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/plugin-container: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/gtk2/libmozgtk.so: DWARF version 0 unhandled /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/gmp-clearkey/0.1/libclearkey.so: DWARF version 0 unhandled x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stuDDKsa: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/gtk2/stBxmDuH: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/gmp-clearkey/0.1/stVO4HYV: debuglink section already exists x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stAazH1P: debuglink section already exists /usr/bin/debugedit: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/libxul.so: DWARF version 0 unhandled x86_64-pc-linux-gnu-objcopy: /var/tmp/portage/www-client/firefox-78.10.0/image/usr/lib64/firefox/stNahJyH: debuglink section already exists
Please compress and attach logs here for posterity.
(In reply to Sam James from comment #6) > Please compress and attach logs here for posterity. Firefox build log is about 15MB, bugzilla attachments is limited to 1MB for me.
Created attachment 705261 [details] build log
Well, xz magically shrinks it to 340K, don't expected such high ratio :)
this issue is actually more general. it doesn't really have any relation to firefox. somehow lld chokes on the glibc crt1.o object's debuglink and insists on inserting it into the generated binaries for some unknown reason. the fix, or at least a workaround for this is disabling splitdebug for sys-libs/glibc specifically. I ran into this while doing a (almost) full clang/libc++ system rebuild
To add to my previous comment, this seems to happen everytime lld needs to add some object file for runtime functionality into an executable, e.g. the C runtime crt?.o files, compiler-rt's libclang_rt.builtins, gcc's crtfastmath. I've had to turn off splitdebug for glibc, compiler-rt, and gcc so far. If I find more, I'll list them in another followup comment. I will try to track down the problem within LLVM that's causing this issue (doesn't seem to be inside LLD itself, but in the LLVM core)
I think this is the issue I hit last year with systemd-coredump and very recently with Ardour in gdb producing almost entirely garbage stack traces where even open source's best wizards got stumped by how broken they were. May I suggest this bug is renamed to something more obvious and prominent, since it probably affects all clang built systems with splitdebug enabled.
You can also fix this issue by adding "--remove-section=.gnu_debuglink" before the "--add-gnu-debuglink=..." in the objcopy command where you're adding the debuglink.
(In reply to Patrick Stewart from comment #13) > You can also fix this issue by adding "--remove-section=.gnu_debuglink" > before the "--add-gnu-debuglink=..." in the objcopy command where you're > adding the debuglink. Ah, in Portage then. Interesting..
I wonder if this was fixed by https://sourceware.org/git/?p=debugedit.git;a=commit;h=3e7aeeab4f744ad15108775685db68d3a35b0735 which we backported in bug 911306?
Created attachment 892115 [details, diff] estrip: Don't use splitdebug for .o files I sent this to gentoo-portage-dev a year ago but it silently ignored my emails.
(In reply to Alex Xu (Hello71) from comment #16) > Created attachment 892115 [details, diff] [details, diff] > estrip: Don't use splitdebug for .o files > > I sent this to gentoo-portage-dev a year ago but it silently ignored my > emails. Thanks, I'll take a look. I think it might be subscriber only.
(In reply to Sam James from comment #17) Could you add Signed-off-by please? In a comment is fine.
This portage patch fixes binaries to not have double debuglink entries for me, gdb stops complaining about the *.o stuff mismatching with the actual library (presumably it tries to use them instead of the real splitdebug file for the debug symbols) and iwdevtools abigail ABI report integration starts working again instead of reporting ABI(nodebug).
Created attachment 892237 [details, diff] estrip: Don't use splitdebug for .o files (with Signed-off-by)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=51f6ad158fc3de14df3f87d1242ff950b2189f05 commit 51f6ad158fc3de14df3f87d1242ff950b2189f05 Author: Alex Xu (Hello71) <alex_y_xu@yahoo.ca> AuthorDate: 2023-03-11 17:27:23 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-05-04 01:32:47 +0000 estrip: Don't use splitdebug for .o files As the process_ar comment explained, object files cannot be splitdebug. Bug: https://bugs.gentoo.org/787623 ("www-client/firefox[clang]: .gnu_debuglink is busted when using LLD") Fixes: 51579fb34c19 ("prepstrip: add support for elfutils strip") Signed-off-by: Alex Xu (Hello71) <alex_y_xu@yahoo.ca> Signed-off-by: Sam James <sam@gentoo.org> bin/estrip | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
Thank you!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9a9d8628ddadbbbeb595c8f0007c9e5ba89c8a71 commit 9a9d8628ddadbbbeb595c8f0007c9e5ba89c8a71 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-05-10 18:25:01 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-05-10 18:25:27 +0000 sys-apps/portage: backport fix for splitdebug splitting Affects Clang in particular. Bug: https://bugs.gentoo.org/787623 Signed-off-by: Sam James <sam@gentoo.org> .../files/portage-3.0.64-clang-splitdebug.patch | 76 +++++++ sys-apps/portage/portage-3.0.64-r1.ebuild | 246 +++++++++++++++++++++ 2 files changed, 322 insertions(+)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe commit 1dd08f05355b14c7594d0e0ccf46cdb2b1486cbe Author: Sam James <sam@gentoo.org> AuthorDate: 2024-06-04 01:31:07 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-06-04 01:31:07 +0000 sys-apps/portage: add 3.0.65 Closes: https://bugs.gentoo.org/930802 Closes: https://bugs.gentoo.org/787623 Closes: https://bugs.gentoo.org/922629 Closes: https://bugs.gentoo.org/927241 Closes: https://bugs.gentoo.org/930992 Closes: https://bugs.gentoo.org/932739 Closes: https://bugs.gentoo.org/932804 Closes: https://bugs.gentoo.org/933297 Closes: https://bugs.gentoo.org/933385 Closes: https://bugs.gentoo.org/933442 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.65.ebuild | 242 +++++++++++++++++++++++++++++++++ 2 files changed, 243 insertions(+)