Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 290974 - linker scripts and dlopening direct library names
Summary: linker scripts and dlopening direct library names
Status: RESOLVED WONTFIX
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: x86 Linux
: High normal with 2 votes (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
: 294093 346095 (view as bug list)
Depends on:
Blocks: 311361
  Show dependency tree
 
Reported: 2009-10-29 05:21 UTC by Ivan
Modified: 2015-12-08 21:06 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Sample C file that breaks dlopen() (breakDlopen.c,425 bytes, text/plain)
2009-10-29 05:28 UTC, Ivan
Details
zlib-1.2.4.ebuild.patch (zlib-1.2.4.ebuild.patch,274 bytes, patch)
2010-03-25 23:35 UTC, Maciej Piechotka
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan 2009-10-29 05:21:05 UTC
Please do not mark this as a duplicate of bug 4411 and close it; or if you do so, please re-open bug 4411 as the problem is still occurring.

Whilst in most cases linker scripts in /usr/lib suffice for libraries in /lib, they do not in cases of interpreted languages (e.g. using ghci from dev-lang/ghc, or even Template Haskell when compiling with GHC proper, as TH acts as an interpreter).  This is even a problem with C programs using dlopen(), as I'll show in the next entry with an attachment.

Linker scripts _do_ break dlopen:  if you try and dlopen("libpcre.so", _) then it will fail.  The problem is that whilst /lib is searched before /usr/lib, there is no libpcre.so in /lib, just libpcre.so.0{,.0.1} (which people don't know to look for when writing their code).  So the first "valid" file the dynamic linker finds when you dlopen() is the linker script in /usr/lib, and since that isn't a valid ELF file dlopen() fails.

If we're going to keep with these linker scripts in /usr/lib, can we at least
consider having symlinks in /lib for libpcre, etc. like there are for most
other libraries?  Otherwise we'll keep having problems with interpreted
languages that need to open the libs at runtime rather than using the static
linker.  As an example of where this is a problem, dev-haskell/pcre-light (this and the following packages are in the Haskell overlay) dynamically links to libpcre, and then dev-haskell/highlighting-kate uses it; but when we try to actually use highliting-kate in app-text/pandoc, it fails to build due to not being able to find libpcre.  However, if we manually create symlink /lib/libpcre.so -> /lib/libpcre.so.0.0.1, then it will successfully build.
Comment 1 Ivan 2009-10-29 05:28:30 UTC
Created attachment 208605 [details]
Sample C file that breaks dlopen()

This is an example based upon the one from dlopen's manpage using libpcre instead of libm.  Compile this using "$ gcc -rdynamic -o breakDlopen breakDlopen.c -ldl" and then try to run it; it will fail.  However, if you edit the source to explicitly ask for "libpcre.so.0.1" or "libpcre.so.0" then it will work.

Thus, usage of linking scripts is still a problem.  Please either include foo.so symlinks in /lib or replace linker scripts with symlinks.
Comment 2 Sebastian Luther (few) 2009-10-29 07:40:32 UTC
That means it's a bug in dev-libs/libpcre, not to create the /lib/libpcre.so symlink, right?
Comment 3 Ivan 2009-10-29 07:43:02 UTC
(In reply to comment #2)
> That means it's a bug in dev-libs/libpcre, not to create the /lib/libpcre.so
> symlink, right?
> 

Yes, and for all other libs that have linking scripts.
Comment 4 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-07 01:54:33 UTC
Uhm. No.

dlopen() should be used with a soname just like the linker does, since you're *assuming* a given ABI (and thus a given soname).

To accept *any* soname means that you're loading stuff without knowing the ABI it has.

See http://blog.flameeyes.eu/2009/10/27/a-shared-library-by-any-other-name about the details of symlinks.
Comment 5 Ivan 2009-11-09 09:24:51 UTC
(In reply to comment #4)
> dlopen() should be used with a soname just like the linker does, since you're
> *assuming* a given ABI (and thus a given soname).

OK, that's a fair point for "installed" packages and we should refer to the sonames rather than just the library name.  But how do we get from a "source" package (which just specifies that it needs pcre; see http://hackage.haskell.org/packages/archive/pcre-light/0.3.1/pcre-light.cabal for example) to resolve the exact soname that we can use for package registration (using ghc-pkg in our case)?  We can't use ld since we're not generating .so files (more akin to special bytecode files like a java JAR).

For a more "manual" example, consider doing something like this:
  ghci Foo.hs -lpcre
(where ghci is the GHC interpreter).  We have here the source package of Foo.hs, and we want to pick the current pcre version (same as if we did "gcc foo.c -lpcre").  If this is successfully compiled to bytecode and loaded, then we know the ABI is right for the "current" version of the library; but there's no obvious way of getting the exact soname corresponding to the ABI that i'ts compiled against that we can use.
Comment 6 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-23 12:33:18 UTC
*** Bug 294093 has been marked as a duplicate of this bug. ***
Comment 7 Sebastian Luther (few) 2009-11-23 13:37:58 UTC
Assigning to QA now. Should something happen here? If not please close.
Comment 8 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-11-23 13:54:33 UTC
At best it's stuff for toolchain; dlopen("libfoo.so") is IMHO always wrong anyway.
Comment 9 SpanKY gentoo-dev 2009-11-23 16:35:33 UTC
not only is dlopen("libfoo.so") broken for the reasons Diego cites, it also simply wont work on a lot of distributions out there that dont include the debug packages by default.  they only include the SONAMEs because that is the only thing packages link against.  so in one way, the current behavior is a good thing as it prevents people from doing stupid stuff.

do you have any other examples for us to consider ?
Comment 10 Howard B. Golden 2009-11-23 20:43:33 UTC
I'm trying to reconcile the need (i.e., link against the current package) in Ivan's comment #5 with the current situation. If linking against libpcre.so is not good practice, then what is? How do we find out what the current soname is? Is there some way to use ld to get us what we want?

(Note: for ghc I have a patch that reads the linker script and uses a regular expression to find the GROUP ( ... ) command. Is this generalizable? Is it the right way to go?)

Comment 11 SpanKY gentoo-dev 2009-11-23 20:50:34 UTC
you use the SONAME for whatever ABI you're blindly encoding into your code.  or avoid the pointless dlopen-ing and simply link against it in the first place.
Comment 12 Howard B. Golden 2009-11-23 21:30:39 UTC
(In reply to comment #11)
> you use the SONAME for whatever ABI you're blindly encoding into your code.  or
> avoid the pointless dlopen-ing and simply link against it in the first place.

I think we may be thinking of different use cases. ghc's interactive version is an interpreter, so it needs to be able to link dynamically. (I wonder how Python does it. I'll investigate.)

When you talk about blindly encoding an ABI, isn't that what you do in a C program? I don't call a function with an soname in my C code. I expect the linker to find it for me. Please correct me if I'm wrong.

Comment 13 SpanKY gentoo-dev 2009-11-23 21:39:25 UTC
no, that isnt what you do when compiling & linking.  you include the header, compile it, and then link against the same library version (thus encoding the SONAME).  if you did something stupid like packaged up the .o objects for someone else to link against, that would be blindly encoding the ABI without the explicit SONAME linkage.

whether you are banging on the library from compiled code or interpreted code, the result is the same -- you're making assumptions about the ABI which really are only valid for 1 specific SONAME.  and again, any interpreted script that bangs directly on "libfoo.so" is going to break on every distribution out there (if only for different reasons).
Comment 14 Sergei Trofimovich gentoo-dev 2009-12-17 21:36:19 UTC
(In reply to comment #13)
> whether you are banging on the library from compiled code or interpreted code,
> the result is the same -- you're making assumptions about the ABI which really
> are only valid for 1 specific SONAME.  and again, any interpreted script that
(almost irrelevant to haskell problems)
So all static .a(==bunch of .o files) libraries gentoo builds - break that assumptions about ABI, because noone can be sure what headers they were built against (those .a libs can be dependant on another libs/headers), right?

How do I know what -lpcre version I need for, say, /usr/lib64/libglib-2.0.a ?
                 U pcre_compile2
                 U pcre_config
                 U pcre_dfa_exec
                 U pcre_exec
                 U pcre_fullinfo
                 U pcre_get_stringnumber
                 U pcre_get_stringtable_entries
                 U pcre_study
Comment 15 Maciej Piechotka 2010-03-25 23:35:59 UTC
Created attachment 225265 [details, diff]
zlib-1.2.4.ebuild.patch

Is there any reason why the symlinks from /usr/lib to /lib would not work? Attached patch does not meet QA (it uses hand move instead of do* as I couldn't find move replacement)

% cat test.c     
#include <zlib.h>

int main() {
	z_stream str;
	deflateInit(&str, 6);
}

% ls -l /usr/lib/libz.*
-rw-r--r-- 1 root root 354530 Mar 25 23:29 /usr/lib/libz.a
lrwxrwxrwx 1 root root     16 Mar 25 23:29 /usr/lib/libz.so -> /lib64/libz.so.1
% gcc -lz test.c

% ldd a.out 
	linux-vdso.so.1 =>  (0x00007fff61a30000)
	libz.so.1 => /lib/libz.so.1 (0x00007f183cf9b000)
	libc.so.6 => /lib/libc.so.6 (0x00007f183cc42000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f183d1b3000)

    Package information:
        app-admin/eselect-compiler: (none)
        app-shells/bash:       4.1_p2-r1
        dev-java/java-config:  2.1.10
        dev-lang/python:       2.6.5 3.1.2
        dev-python/pycrypto:   (none)
        dev-util/ccache:       (none)
        dev-util/cmake:        2.8.0-r2
        dev-util/confcache:    (none)
        sys-apps/baselayout:   2.0.1
        sys-apps/openrc:       0.6.1-r1
        sys-apps/sandbox:      2.2
        sys-devel/autoconf:    2.13 2.65
        sys-devel/automake:    1.10.3 1.11.1 1.9.6-r3
        sys-devel/binutils:    2.20.1
        sys-devel/gcc:         4.4.3
        sys-devel/gcc-config:  1.4.1
        sys-devel/libtool:     2.2.6b
        virtual/os-headers:    2.6.33 (for sys-kernel/linux-headers::installed)


% ghci -package zlib
GHCi, version 6.12.1: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package bytestring-0.9.1.5 ... linking ... done.
Loading package zlib-0.5.2.0 ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
Prelude>
Comment 16 SpanKY gentoo-dev 2010-03-26 02:39:58 UTC
i missed comment #14 ... the assumption is that the headers and the static library are from the same version.  if they arent, that's a broken package manager, or someone screwed up their system.

but yes, if you were to take a static library and deploy it with different headers, you could possibly be breaking things and this isnt a "bug".

i still havent seen anything worth changing, so ...
Comment 17 SpanKY gentoo-dev 2010-11-19 22:41:00 UTC
*** Bug 346095 has been marked as a duplicate of this bug. ***
Comment 18 Howard B. Golden 2010-12-25 23:13:18 UTC
At least for GHC-7.0, the linker script patch (GHC ticket #2615) I mentioned in comment #10 is finally available. This resolves Ivan's original need.

GHC 6.12.3 is now stable. The current version of GHC 6.12.3 doesn't have the linker script patch, but if you want it, please contact the Gentoo Haskell project at haskell@gentoo.org and request it.