Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 551936

Summary: sys-libs/glibc: splitdebug leads to broken core dumps (from gdb generate-core-file) for mulithreaded apps if /usr/lib/debug/*/libpthread-*.so.debug is not available
Product: Gentoo Linux Reporter: Zac Medico <zmedico>
Component: Current packagesAssignee: Gentoo Toolchain Maintainers <toolchain>
Status: RESOLVED OBSOLETE    
Severity: normal CC: rini17
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
URL: https://sourceware.org/gdb/wiki/FAQ#GDB_does_not_see_any_threads_besides_the_one_in_which_crash_occurred.3B_or_SIGTRAP_kills_my_program_when_I_set_a_breakpoint.
Whiteboard:
Package list:
Runtime testing required: ---

Description Zac Medico gentoo-dev 2015-06-13 01:59:59 UTC
At my workplace, we build system images with splitdebug enabled, but we keep /usr/lib/debug in a separate tarball that we only unpack when needed. This results in broken core dumps for multi-threaded apps.

It would be nice if our use case was supported, so that we could have working core dumps even when /usr/lib/debug/*/libpthread-*.so.debug is not available. Perhaps the logic inside src_strip of src_install.eblit could be adjusted to support this use case?
Comment 1 SpanKY gentoo-dev 2015-07-17 07:08:36 UTC
i'm not sure this has been triaged correctly.  core dumps are created by the kernel and have no interaction with userspace, so by definition, the core dumps cannot be broken.

further, we're already doing what that FAQ says to do:
(1) we do not strip libthread_db at all
(2) we strip libpthread, but only with --strip-debug

but as long as the session where you run gdb (not where the program crashed) has a coredump and runtime and debug files that all match, i'm not seeing a problem.  since it's also limited to where gdb itself runs and not where the program crashes, i don't see what use case we need to support here.
Comment 2 Zac Medico gentoo-dev 2015-07-17 13:53:18 UTC
(In reply to SpanKY from comment #1)
> i'm not sure this has been triaged correctly.  core dumps are created by the
> kernel and have no interaction with userspace, so by definition, the core
> dumps cannot be broken.

The case that I have tested involves using gdb to generate a core dump of a running process like this:

 gdb -batch \
    -ex "set auto-load safe-path /" \
    -ex "attach ${PID}" \
    -ex "generate-core-file ${FILE}"

Maybe it's the involvement of gdb that seems to break the core dump.

> further, we're already doing what that FAQ says to do:
> (1) we do not strip libthread_db at all
> (2) we strip libpthread, but only with --strip-debug

The relevant code from src_install.eblit looks like this:

local pthread=$(has splitdebug ${FEATURES} && echo "libthread_db" || echo "lib{pthread,thread_db}")

env \
    -uRESTRICT \
    CHOST=${CTARGET} \
    STRIP_MASK="/*/{,tls/}${pthread}*" \
    prepallstrip

So, when splitdebug is enabled, it doesn't add libpthread to STRIP_MASK.

> but as long as the session where you run gdb (not where the program crashed)
> has a coredump and runtime and debug files that all match, i'm not seeing a
> problem.  since it's also limited to where gdb itself runs and not where the
> program crashes, i don't see what use case we need to support here.

I'll have to try trigger a multi-threaded core dump without using gdb generate-core-file and see what happens.
Comment 3 Zac Medico gentoo-dev 2015-07-17 15:24:02 UTC
(In reply to Zac Medico from comment #2)
> (In reply to SpanKY from comment #1)
> > i'm not sure this has been triaged correctly.  core dumps are created by the
> > kernel and have no interaction with userspace, so by definition, the core
> > dumps cannot be broken.
> 
> The case that I have tested involves using gdb to generate a core dump of a
> running process like this:
> 
>  gdb -batch \
>     -ex "set auto-load safe-path /" \
>     -ex "attach ${PID}" \
>     -ex "generate-core-file ${FILE}"
> 
> Maybe it's the involvement of gdb that seems to break the core dump.

I've tested this, and the problem seems to occur only when gdb generate-core-file is used on a running process. With such a core file, gdb commands like "info threads" and "thread apply all bt" to not work correctly.

Since this problem doesn't occur for core dumps generated by the kernel when a program crashes, I suppose we can simply tell people that they need to have /usr/lib/debug present when they want to use gdb generate-core-file.
Comment 4 Juraj Variny 2016-01-27 12:26:10 UTC
> further, we're already doing what that FAQ says to do:
> (1) we do not strip libthread_db at all

This is not true, With splitdebug FEATURE I have found /lib/libthread_db-1.0.so indeed stripped and it caused gdb to disable thread debugging, with message:

> Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

My gdb-7.10.1 *is* able to find splitdebug files normally but not in libthread_db case. I have recompiled glibc-2.21 with following configuration which fixed the problem:

# cat /etc/portage/package.env   
sys-libs/glibc nosplitdebug.conf 
# cat /etc/portage/env/nosplitdebug.conf 
FEATURES="${FEATURES} -splitdebug nostrip"
Comment 5 Sergei Trofimovich (RETIRED) gentoo-dev 2019-10-23 18:53:03 UTC
The stripping was simplified in bug #697910 and should not strip libpthread at all.