Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 655478

Summary: app-misc/pax-utils: lddtree.py unable to detect default shared lib search path to mimic ld.so
Product: Gentoo Linux Reporter: Eli Schwartz <eschwartz93>
Component: Current packagesAssignee: Gentoo Toolchain Maintainers <toolchain>
Status: CONFIRMED ---    
Severity: normal CC: dominik, sam
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Eli Schwartz 2018-05-11 13:47:20 UTC
lddtree is a very neat tool, which I use in Arch Linux a lot. As a bugwrangler it is convenient to tell people reporting an issue with some package not being able to load a shared library, "use lddtree to figure out which lib is actually problematic".

For example, every time gcc is updated, we get complains that numpy doesn't work, because people use their own self-compiled blas/lapack packages which need to be rebuilt...

But it doesn't work:

$ lddtree /usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
/usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so (interpreter => None)
    libcblas.so.3 => None
    libm.so.6 => None
    libpython3.6m.so.1.0 => None
    libpthread.so.0 => None
    libc.so.6 => None
    ld-linux-x86-64.so.2 => None


Using -x, the following additional info is reported:

$ lddtree -x /usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
root = /
ParseLdSoConf(//etc/ld.so.conf)
  glob: /etc/ld.so.conf.d/*.conf
  ParseLdSoConf(/etc/ld.so.conf.d/fakechroot.conf)
  ParseLdSoConf(/etc/ld.so.conf.d/fakeroot.conf)
ldpaths[conf] = ['/usr/lib/libfakeroot/fakechroot', '/usr/lib/libfakeroot']
ldpaths[env]  = []
argv[x]       = /usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  +auto-root  = /usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  globbed     = /usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
ParseELF(/usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so)
  ldpaths[rpath]   = []
  ldpaths[runpath] = []
  FindLib(libcblas.so.3)
    checking: /usr/lib/libfakeroot/fakechroot/libcblas.so.3
    checking: /usr/lib/libfakeroot/libcblas.so.3
  FindLib(libm.so.6)
    checking: /usr/lib/libfakeroot/fakechroot/libm.so.6
    checking: /usr/lib/libfakeroot/libm.so.6
  FindLib(libpython3.6m.so.1.0)
    checking: /usr/lib/libfakeroot/fakechroot/libpython3.6m.so.1.0
    checking: /usr/lib/libfakeroot/libpython3.6m.so.1.0
  FindLib(libpthread.so.0)
    checking: /usr/lib/libfakeroot/fakechroot/libpthread.so.0
    checking: /usr/lib/libfakeroot/libpthread.so.0
  FindLib(libc.so.6)
    checking: /usr/lib/libfakeroot/fakechroot/libc.so.6
    checking: /usr/lib/libfakeroot/libc.so.6
  FindLib(ld-linux-x86-64.so.2)
    checking: /usr/lib/libfakeroot/fakechroot/ld-linux-x86-64.so.2
    checking: /usr/lib/libfakeroot/ld-linux-x86-64.so.2

...

The issue seems to be it cannot detect the python interpreter, and we don't add the default /usr/lib(64) to our ld.so.conf, so lddtree.py never looks for libs in /usr/lib, even though the python interpreter *does* pull in /lib64/ld-linux-x86-64.so.2 as its own interpreter.

I'm not sure if the correct solution is to just fall back on ld.so as the interpreter, or somehow detect/special-case python, or simply seed the ldpaths with common defaults.

Reproducible: Always
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2018-05-13 10:16:07 UTC
JUst to clarify: do I understand correctly this is the output of an arch system and not gentoo?
Comment 2 Sergei Trofimovich (RETIRED) gentoo-dev 2018-05-13 10:44:59 UTC
I think presence of python interpreter is unrelated. shared libraries don't have intepreter set in ELF and it's fine. They don't have to be selfcontained binaries and in theory any dynamic loader (with it's own defaults) can load them.

What seems to differ on your system is lack of path to /lib* in /etc/ld.so.conf* files.
Gentoo has:
  $ cat /etc/ld.so.conf
  /lib64
  /lib32
  /lib
  <and many others>

You are correct it does not perfectly match search order of default dynamic loader. glibc's ld has a few more defaults set:

$ LD_DEBUG=all /lib64/ld-linux-x86-64.so.2 --inhibit-cache --list /bin/ls 2>&1 | fgrep 'search path'
      1270:      search path=/lib64/tls/x86_64/x86_64:/lib64/tls/x86_64:/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64/x86_64:/lib64/x86_64:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64/x86_64:/usr/lib64/tls/x86_64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64/x86_64:/usr/lib64/x86_64:/usr/lib64/x86_64:/usr/lib64           (system search path)

Default configuration can vary from system to system. For example on prefixed system there is no /lib* or /usr/lib* paths at all:

$ LD_DEBUG=all ./gentoo/g821/lib64/ld-linux-x86-64.so.2 --inhibit-cache --list /bin/ls 2>&1 | fgrep 'search path'
      1445:      search path=/home/slyfox/gentoo/g821/lib64/tls/x86_64/x86_64:/home/slyfox/gentoo/g821/lib64/tls/x86_64:/home/slyfox/gentoo/g821/lib64/tls/x86_64:/home/slyfox/gentoo/g821/lib64/tls:/home/slyfox/gentoo/g821/lib64/x86_64/x86_64:/home/slyfox/gentoo/g821/lib64/x86_64:/home/slyfox/gentoo/g821/lib64/x86_64:/home/slyfox/gentoo/g821/lib64:/home/slyfox/gentoo/g821/usr/lib64/tls/x86_64/x86_64:/home/slyfox/gentoo/g821/usr/lib64/tls/x86_64:/home/slyfox/gentoo/g821/usr/lib64/tls/x86_64:/home/slyfox/gentoo/g821/usr/lib64/tls:/home/slyfox/gentoo/g821/usr/lib64/x86_64/x86_64:/home/slyfox/gentoo/g821/usr/lib64/x86_64:/home/slyfox/gentoo/g821/usr/lib64/x86_64:/home/slyfox/gentoo/g821/usr/lib64            (system search path)

I think simplest workarounds for you would be to:
- populate /etc/ld.so.conf (or equivalent)
- or inject ld's defaults into ParseLdSoConf function
Comment 3 Eli Schwartz 2018-05-13 12:49:45 UTC
> shared libraries don't have intepreter set in ELF and it's fine

Except when conceptually you're using it even though technically you don't use it directly at all. Though I suppose it is true you don't usually need to check the search path of a library, since it is usually used directly by some other binary, whereas python (and other scripting languages) needs you to import it.

> - or inject ld's defaults into ParseLdSoConf function

Yes, it worked fine when changing

> paths = []

to

> paths = ['/usr/lib']

But I'd prefer something that didn't require patching. Would it be possible to build this during configure? Maybe ParseELF() should default to the system ld.so for interp, if none is detected in the ELF object itself?
Comment 4 Sergei Trofimovich (RETIRED) gentoo-dev 2018-05-13 14:30:30 UTC
(In reply to Eli Schwartz from comment #3)
> > - or inject ld's defaults into ParseLdSoConf function
> 
> Yes, it worked fine when changing
> 
> > paths = []
> 
> to
> 
> > paths = ['/usr/lib']
> 
> But I'd prefer something that didn't require patching. Would it be possible
> to build this during configure? Maybe ParseELF() should default to the
> system ld.so for interp, if none is detected in the ELF object itself?

Having a configure option to inject defaults sounds fine.
Comment 5 SpanKY gentoo-dev 2018-06-08 19:01:07 UTC
defaulting to any set of paths at configure time is incorrect.  lddtree.py is, by design, agnostic of the active runtime environment.  if you hardcode e.g. /usr/lib64 because CHOST is x86_64-linux-gnu, you break lddtree.py usage for cross targets.  this is precisely why lddtree.py uses the interp binary to try and figure out a default.

however, if ld.so.conf doesn't contain the system paths, and you're probing an ELF that doesn't have an interp set (like a shared lib), then yes lddtree won't be able to discover the right default paths.

we could fallback to probing the interp from $ROOT/bin/sh to recover.  that should hopefully handle the majority of real world systems.
Comment 6 SpanKY gentoo-dev 2021-07-06 01:55:54 UTC
*** Bug 800221 has been marked as a duplicate of this bug. ***