Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 917287 - FEATURES=splitdebug should follow usrmerge symlinks in /usr/lib/debug/
Summary: FEATURES=splitdebug should follow usrmerge symlinks in /usr/lib/debug/
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - External Interaction (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 937749 940345 (view as bug list)
Depends on:
Blocks: usrmerge, usrmerge-fixes 933986
  Show dependency tree
 
Reported: 2023-11-13 09:45 UTC by Maxim Britov
Modified: 2025-04-15 00:20 UTC (History)
14 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
gdb: Don't use realpath on objfile names (gdb_Dont_use_realpath_on_objfile_names.patch,5.83 KB, patch)
2024-03-17 18:10 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Maxim Britov 2023-11-13 09:45:41 UTC
For now FEATURE=splitdebug creating old style split bin, sbin, usr/bin, usr/sbin in /usr/lib/debug/

tree -d -n /usr/lib/debug/
/usr/lib/debug/
├── lib64
├── sbin
└── usr
    ├── bin
    ├── lib64
    └── sbin

This confuses tools like gdb/valgrind, etc.

For example gdb /usr/[s]bin/squid can't find symbols.
I have to copy /usr/lib/debug/usr/sbin/squid.debug into /usr/lib/debug/usr/bin/
Then gdb /usr/[s]bin/squid load debug symbols.

Reproducible: Always
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-11-13 09:49:53 UTC
This came up on the gdb side at https://sourceware.org/bugzilla/show_bug.cgi?id=24143 too...
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-11-13 10:22:03 UTC
Could you consider filing a bug for Valgrind upstream if there isn't one already, and possibly elfutils if it also suffers from the same problem?

Thank you.
Comment 3 Matt Whitlock 2023-11-13 19:13:27 UTC
What if you create the same symlinks beneath /usr/lib/debug as exist within the top level? Then the package manager will install the split debug files in /usr/lib/debug/usr/bin regardless of whether they were split from binaries intending to be installed in /bin, /sbin, /usr/bin, or /usr/sbin, and debug tools should then find them regardless of whether they're implementing the original or patched lookup behavior.
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-11-13 20:12:19 UTC
That sounds like a good idea. Could I tempt one of you to have a go at implementing it? I'm swamped at the moment.
Comment 5 Matt Whitlock 2023-11-13 20:39:47 UTC
Seems to work...


$ /sbin/fdisk --version
fdisk from util-linux 2.39.2

$ readlink -e /sbin/fdisk
/usr/bin/fdisk

$ readlink -e /usr/lib/debug/sbin/fdisk.debug
/usr/lib/debug/sbin/fdisk.debug

$ gdb /sbin/fdisk </dev/null | tail -n3 | head -n2
Reading symbols from /sbin/fdisk...
(No debugging symbols found in /sbin/fdisk)

$ gdb /usr/bin/fdisk </dev/null | tail -n3 | head -n2
Reading symbols from /usr/bin/fdisk...
(No debugging symbols found in /usr/bin/fdisk)


# cd /usr/lib/debug
# mv --no-clobber {{,s}bin,usr/sbin}/* usr/bin/
# rmdir {,s}bin usr/sbin
# for each in {,s}bin usr/sbin ; do ln -rsfnv usr/bin "${each}" ; done
'bin' -> 'usr/bin'
'sbin' -> 'usr/bin'
'usr/sbin' -> 'bin'


$ readlink -e /usr/lib/debug/sbin/fdisk.debug
/usr/lib/debug/usr/bin/fdisk.debug

$ gdb /sbin/fdisk </dev/null | tail -n3 | head -n2
Reading symbols from /sbin/fdisk...
Reading symbols from /usr/lib/debug//usr/bin/fdisk.debug...

$ gdb /usr/bin/fdisk </dev/null | tail -n3 | head -n2
Reading symbols from /usr/bin/fdisk...
Reading symbols from /usr/lib/debug//usr/bin/fdisk.debug...
Comment 6 Zac Medico gentoo-dev 2024-03-17 18:10:43 UTC
Created attachment 887847 [details, diff]
gdb: Don't use realpath on objfile names

This gdb patch still applies cleanly, mentioned in https://sourceware.org/bugzilla/show_bug.cgi?id=24143#c5 and downloaded from  https://sourceware.org/pipermail/gdb-patches/2022-April/187545.html.
Comment 7 Zac Medico gentoo-dev 2024-03-18 03:05:39 UTC
(In reply to Zac Medico from comment #6)
> Created attachment 887847 [details, diff] [details, diff]
> gdb: Don't use realpath on objfile names
> 
> This gdb patch still applies cleanly, mentioned in
> https://sourceware.org/bugzilla/show_bug.cgi?id=24143#c5 and downloaded from
> https://sourceware.org/pipermail/gdb-patches/2022-April/187545.html.

I applied this patch to dev-debug/gdb-14.1-r1 and it appears to correctly resolve /usr/lib/debug/sbin/fdisk.debug regardless of how I refer to fdisk:

> $ gdb /sbin/fdisk </dev/null | tail -n4 | head -n2
> Reading symbols from /sbin/fdisk...
> Reading symbols from /usr/lib/debug/sbin/fdisk.debug...
> $ gdb /bin/fdisk </dev/null | tail -n4 | head -n2
> Reading symbols from /bin/fdisk...
> Reading symbols from /usr/lib/debug/sbin/fdisk.debug...
> $ gdb /usr/bin/fdisk </dev/null | tail -n4 | head -n2
> Reading symbols from /usr/bin/fdisk...
> Reading symbols from /usr/lib/debug/sbin/fdisk.debug...
> $ gdb /usr/sbin/fdisk </dev/null | tail -n4 | head -n2
> Reading symbols from /usr/sbin/fdisk...
> Reading symbols from /usr/lib/debug/sbin/fdisk.debug...
Comment 8 Zac Medico gentoo-dev 2024-03-18 03:21:43 UTC
Comment on attachment 887847 [details, diff]
gdb: Don't use realpath on objfile names

Actually dev-debug/gdb-14.1-r1 without this patch also resolves /usr/lib/debug/sbin/fdisk.debug for me correctly for all the same cases shown in comment #7.
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-03-18 08:35:12 UTC
I think I last tried w/ gdb-14.1 (14.2 has very few changes on top) but even that patch didn't seem to help me.. interesting!
Comment 10 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-08-11 08:54:28 UTC
*** Bug 937749 has been marked as a duplicate of this bug. ***
Comment 11 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-08-11 08:54:50 UTC
The dupe bug 937749 implies it's still an issue.
Comment 12 Emanuele Torre 2024-08-11 12:13:16 UTC
(In reply to Sam James from comment #11)
> The dupe bug 937749 implies it's still an issue.

Thank you for adding me to this bug, Sam.

My current workaround to have debug informations for bash is:

    $ cat /etc/portage/env/nostrip
    FEATURES="${FEATURES} -splitdebug nostrip"
    $ grep bash /etc/portage/package.env
    app-shells/bash nostrip

But I have just noticed that, indeed, this problems affects more
packages than just bash.

Even more annoyingly, I don't even have debug informations for glibc
because the debug informations for the core parts are installed in
/usr/lib/debug/lib{,64} instead of /usr/lib/debug/usr/lib{,64}.

Simple example:

    $ gdb -p "$BASHPID" --batch-silent --ex 'call free(0)'
    'free' has unknown return type; cast the call to its declared return type
    $ gdb -p "$BASHPID" --batch-silent --ex 'call (void)free(0)'
    $ 

The nosplit workaround also works for glibc

    $ grep glibc /etc/portage/package.env
    sys-libs/glibc nostrip
    $ gdb -p "$BASHPID" --batch-silent --ex 'call free(0)'
    $ 

I think for now I will just use this nosplit workaround on every package
that tries to install in /usr/lib/debug outside of /usr/lib/debug/usr
instead of messing with /usr/lib/debug moving files manually and
creating symlinks. Hopefully this problem will be resolved soon though.

I think not having debug informations for glibc is a pretty big deal for
debugging so I am considering adding a note that says that splitdebug
with merge-usr is broken for some core packages, and mentions this
bug and the nostrip workaround on the wiki page for Debugging later.

o/
 emanuele6
Comment 13 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-08-30 23:52:03 UTC
negril suggested in -portage running `merge-usr --prefix /usr/lib/debug/` and it did the charm for me.
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-08-30 23:53:39 UTC
... did the trick, or worked like a ...
Comment 15 Matt Whitlock 2024-08-31 00:07:32 UTC
(In reply to Sam James from comment #13)
> negril suggested in -portage running `merge-usr --prefix /usr/lib/debug/`

Doesn't that effectively do the same as…

(In reply to Matt Whitlock from comment #3)
> What if you create the same symlinks beneath /usr/lib/debug as exist within
> the top level?

In other words, I think I manually performed the usr merge beneath /usr/lib/debug. I have had no issues with tools not finding debugging symbols since I did that. I think maybe Portage still warns about the links when unmerging packages, though.
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-08-31 00:29:14 UTC
Yes, it does. The revelation (to me) is that we have a nice tool to do it already (I didn't realise it had a --prefix option).
Comment 17 Mike Gilbert gentoo-dev 2024-09-29 15:49:35 UTC
*** Bug 940345 has been marked as a duplicate of this bug. ***