Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 697910 - sys-libs/glibc: gdb unable to find glibc libthread_db due to stripped libpthread.so
Summary: sys-libs/glibc: gdb unable to find glibc libthread_db due to stripped libpthr...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: High major (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-10-17 06:51 UTC by Minux
Modified: 2019-10-17 23:22 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Minux 2019-10-17 06:51:49 UTC
After bug #587296 reverted the fix to bug #46186, bug #46186 resurfaces.

When using gdb to debug a multithreaded program, it will complain:
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Reproducible: Always

Steps to Reproduce:
1. rebuild glibc without nostrip or splitdebug feature (If you have enabled splitdebug before, you need to make sure to clean up /usr/lib/debug/lib64/libpthread-2.29.so.debug so that gdb won't find debugging symbols there.)

Alternatively, if you are using splitdebug, just rename /usr/lib/debug/lib64/libpthread-2.29.so.debug.
If you are using nostrip (unlikely), backup /lib64/libpthread-2.29.so and then strip /lib64/libpthread-2.29.so.
(The key here is to simulate the environment of a stock gentoo installation without any FEATUREs set.)

2. gdb -q -ex run -ex q --args xzdec -V # any program that links libpthread will work here

Actual Results:  
Reading symbols from xzdec...
(No debugging symbols found in xzdec)
Starting program: /usr/bin/xzdec -V
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
xzdec (XZ Utils) 5.2.4
liblzma 5.2.4
[Inferior 1 (process 1583) exited normally]

Expected Results:  
Reading symbols from xzdec...
(No debugging symbols found in xzdec)
Starting program: /usr/bin/xzdec -V
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
xzdec (XZ Utils) 5.2.4
liblzma 5.2.4
[Inferior 1 (process 1583) exited normally]

The real problem is not that gdb relies on debug info in libpthread.so, on the contrary, gdb doesn't care.
It's *glibc* libthread_db that requires a local symbol (nptl_version) in libpthread.so to verify that
their versions match.
(See https://sourceware.org/gdb/wiki/FAQ#GDB_does_not_see_any_threads_besides_the_one_in_which_crash_occurred.3B_or_SIGTRAP_kills_my_program_when_I_set_a_breakpoint
for the official explanation.)

Thanks.
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2019-10-17 07:21:18 UTC
Please provide the 'emerge --info' for the affected system.
Comment 2 Minux 2019-10-17 07:25:03 UTC
emerge --info (it's a up-to-date stock gentoo stage3 installation without any tweaks to /etc/portage/make.conf):

Portage 2.3.76 (python 2.7.16-final-0, default/linux/amd64/17.1, gcc-8.3.0, glibc-2.29-r2, 5.2.0+ x86_64)
=================================================================
System uname: Linux-5.2.0+-x86_64-Common_KVM_processor-with-gentoo-2.6
KiB Mem:     2030548 total,   1877348 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Wed, 16 Oct 2019 00:45:02 +0000
Head commit of repository gentoo: bd4f1637960181a2b27465027991e0c82382675a
sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.28.2-r1::gentoo
dev-lang/python:          2.7.16::gentoo, 3.6.9::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.41.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo
sys-devel/gcc:            8.3.0-r1::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.19::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 berkdb bzip2 cli crypt cxx dri fortran gdbm iconv ipv6 libtirpc multilib ncurses nls nptl openmp pam pcre readline seccomp split-usr ssl tcpd unicode xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 3 Sergei Trofimovich (RETIRED) gentoo-dev 2019-10-17 08:01:15 UTC
(In reply to Minux from comment #0)
> After bug #587296 reverted the fix to bug #46186, bug #46186 resurfaces.
> 
> When using gdb to debug a multithreaded program, it will complain:
> warning: Unable to find libthread_db matching inferior's thread library,
> thread debugging will not be available.
> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1. rebuild glibc without nostrip or splitdebug feature (If you have enabled
> splitdebug before, you need to make sure to clean up
> /usr/lib/debug/lib64/libpthread-2.29.so.debug so that gdb won't find
> debugging symbols there.)
> 
> Alternatively, if you are using splitdebug, just rename
> /usr/lib/debug/lib64/libpthread-2.29.so.debug.
> If you are using nostrip (unlikely), backup /lib64/libpthread-2.29.so and
> then strip /lib64/libpthread-2.29.so.
> (The key here is to simulate the environment of a stock gentoo installation
> without any FEATUREs set.)
> 
> 2. gdb -q -ex run -ex q --args xzdec -V # any program that links libpthread
> will work here
> 
> Actual Results:  
> Reading symbols from xzdec...
> (No debugging symbols found in xzdec)
> Starting program: /usr/bin/xzdec -V
> warning: Unable to find libthread_db matching inferior's thread library,
> thread debugging will not be available.
> xzdec (XZ Utils) 5.2.4
> liblzma 5.2.4
> [Inferior 1 (process 1583) exited normally]
> 
> Expected Results:  
> Reading symbols from xzdec...
> (No debugging symbols found in xzdec)
> Starting program: /usr/bin/xzdec -V
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> xzdec (XZ Utils) 5.2.4
> liblzma 5.2.4
> [Inferior 1 (process 1583) exited normally]
> 
> The real problem is not that gdb relies on debug info in libpthread.so, on
> the contrary, gdb doesn't care.
> It's *glibc* libthread_db that requires a local symbol (nptl_version) in
> libpthread.so to verify that
> their versions match.

I think it's a bit more complicate than that. glibc relies on gdb (or any other host process) to provide 'ps_pglobal_lookup' symbol. Which in turn inspects '.symtab' is a libpthread.so. debug info stripping does affect '.symtab'. I would say gdb very much relies on debugging info.

I agree the original change broke the mechanism. I'll restore selective stripping at least in newer ebuilds and probably disable stripping altogether in older ones.

> (See
> https://sourceware.org/gdb/wiki/
> FAQ#GDB_does_not_see_any_threads_besides_the_one_in_which_crash_occurred.
> 3B_or_SIGTRAP_kills_my_program_when_I_set_a_breakpoint
> for the official explanation.)

Unfortunately it has no mechanical explanation at all.
Comment 4 Minux 2019-10-17 08:27:49 UTC
(In reply to Sergei Trofimovich from comment #3)
> I think it's a bit more complicate than that. glibc relies on gdb (or any
> other host process) to provide 'ps_pglobal_lookup' symbol. Which in turn
> inspects '.symtab' is a libpthread.so. debug info stripping does affect
> '.symtab'. I would say gdb very much relies on debugging info.
> 
> I agree the original change broke the mechanism. I'll restore selective
> stripping at least in newer ebuilds and probably disable stripping
> altogether in older ones.

Yes, the underlying mechanism is as you described.

But the issue is that it's glibc libthread_db that requests gdb to find a private symbol
(nptl_version) in libpthread.so, see:
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl_db/td_ta_new.c;h=63ffe4ca2fe097fb719db4ea8a61de336ac9a30d;hb=HEAD#l42

glibc tries to hide the implementation details with local symbols, see
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl_db/structs.def;hb=HEAD
for the list of private symbols that libthread_db.so need to access.

The best solution for gentoo is still to only strip --strip-debug libpthread.so
if ! has nostrip $FEATURES && ! has splitdebug $FEATURES.
Comment 5 Sergei Trofimovich (RETIRED) gentoo-dev 2019-10-17 19:36:31 UTC
(In reply to Minux from comment #4)
> The best solution for gentoo is still to only strip --strip-debug
> libpthread.so
> if ! has nostrip $FEATURES && ! has splitdebug $FEATURES.

Gentoo does not provide fine-grained control over strip process. We can either strip or not strip with help of portage in ebuild level. in EAPI=7 we can do it on individual file level.

glibc ebuild could strip on rest of files itself but I'd rather not do it and rely on users' CFLAGS to target amount of debugging info available.
Comment 6 Larry the Git Cow gentoo-dev 2019-10-17 23:22:28 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f3872a506edc7da0d987bcf0a90d4709945328a7

commit f3872a506edc7da0d987bcf0a90d4709945328a7
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2019-10-17 23:19:20 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2019-10-17 23:22:21 +0000

    sys-libs/glibc: restore strip quirk for 'libpthread.so.0'
    
    This change effectively reverts commit e14229b10b513a
    ("sys-libs/glibc: drop prepallstrip, bug #587296")
    
    gdb still relies on libpthread being unstripped.
    See https://bugs.gentoo.org/697910 for details and reproducer.
    
    Also see ebuild note that explains indirect mechanics
    of how glibc relies on gdb's ability to traverse local symbols.
    
    In EAPI=7 ebuilds (glibc-9999) we do selective stripping.
    
    Reported-by: Minux
    Closes: https://bugs.gentoo.org/697910
    Bug: https://bugs.gentoo.org/587296
    Bug: https://bugs.gentoo.org/46186
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
    Package-Manager: Portage-2.3.76, Repoman-2.3.17
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-libs/glibc/glibc-2.19-r2.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.24-r4.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.25-r11.ebuild | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.26-r7.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.27-r6.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.28-r6.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.29-r2.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.29-r4.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.29-r5.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.30-r1.ebuild  | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-2.30.ebuild     | 25 ++++++++++++++++++++++---
 sys-libs/glibc/glibc-9999.ebuild     | 31 ++++++++++++++++++++++++++++---
 12 files changed, 270 insertions(+), 36 deletions(-)