Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 213277 - which library style to prefer on AIX for both the 'soname' feature and runtime linking enabled
Summary: which library style to prefer on AIX for both the 'soname' feature and runtim...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: PPC AIX
: High normal (vote)
Assignee: Michael Haubenwallner (RETIRED)
URL: http://lists.gnu.org/archive/html/lib...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-03-13 14:59 UTC by Michael Haubenwallner (RETIRED)
Modified: 2011-12-16 07:40 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Variant A: Traditional variant to create shared libraries on AIX, but with runtime linking (aixrtl-traditional,1.14 KB, text/plain)
2011-01-21 14:08 UTC, Michael Haubenwallner (RETIRED)
Details
Variants B and C: Using Import Files to create shared libraries on AIX with runtime linking (aixrtl-importfiles,7.52 KB, text/plain)
2011-01-21 14:09 UTC, Michael Haubenwallner (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Haubenwallner (RETIRED) gentoo-dev 2008-03-13 14:59:20 UTC
We'd better avoid libX.so style shared libs on AIX in general for various reasons:
http://archives.gentoo.org/gentoo-alt/msg_40445b0575511ef2709cbc3727011805.xml

Currently known-to-be-broken are these packages in @system:

app-arch/bzip2
dev-libs/apr
dev-libs/apr-util
dev-libs/openssl
dev-util/subversion
sys-devel/libperl
sys-libs/ncurses
sys-libs/readline
sys-libs/zlib
Comment 1 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2008-03-13 16:31:17 UTC
haubi - 
I only partially grasp the idea but my EPREFIX is not of production importance so it can be broken when "the switch" happens.
Comment 2 Fabian Groffen gentoo-dev 2008-03-13 16:56:19 UTC
Then I vote for hell with current AIX.  I haven't heard from Marshall for ages.  Benefit from the bleeding edgeness of the AIX port now you still can, and do it the right way, NOW.

(Just my opinion ;) )
Comment 3 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2008-09-23 03:38:55 UTC
haubi: This must be done by now. I have bootstrapped a fresh aix prefix some time ago and didn't notice any issue.
Comment 4 Michael Haubenwallner (RETIRED) gentoo-dev 2009-03-26 14:10:06 UTC
elibtoolize has aixrtl patches for up to libtool-2.2.6 now, to ignore '-brtl' in LDFLAGS, but still use '-G' linker flag, to create the runtime-linking enabled shared object packed into the archive library:
http://overlays.gentoo.org/proj/alt/changeset/40368
Comment 5 Michael Haubenwallner (RETIRED) gentoo-dev 2009-05-04 15:08:39 UTC
When a package does not really utilize the 'soname' feature for multiple shared library versions (because it does not build with libtool, like net-misc/mico fex), and it provides the full-versioned link-name via its config-helper scripts (like 'mico-config --libs'), there is no real need to force it putting the shared object (lib*.so) into the archive (lib*.a).

While the ld-wrapper (binutils-config) already passes '-brtl' to the linker to search for 'lib*.so' before 'lib*.a' (besides enabling runtime linking), gcc's collect2 also has to use this same search order, to not collect con/destructors for the global symbols in static libs when the shared libs actually get linked.
Comment 6 Michael Haubenwallner (RETIRED) gentoo-dev 2009-05-05 11:20:39 UTC
(In reply to comment #5)
> gcc's collect2 also has to use this same search order

For gcc-3.4.6 and gcc-4.2.2 done in r44662.

Still missing: gcc-4.3, eventually 4.1.
Comment 7 Fabian Groffen gentoo-dev 2009-10-25 09:41:26 UTC
let's keep it in the herd, eventually it'll be you (haubi) to fix it anyway, so don't worry about that :)
Comment 8 Michael Haubenwallner (RETIRED) gentoo-dev 2009-11-23 15:10:45 UTC
Hmm...
As long as a shared library does not have undefined symbols, it does work to use "Import Files" to implement some "soname" feature.

For packages not using libtool, I tend to not put the shared object into an archive any more, but leave it as 'libNAME.so.X', and create an 'Import File' named 'libNAME.so', pointing the loader to 'libNAME.so.X' (the "soname").
This is "Variant B1" in the description this bug's URL points to.

Many non-libtool packages already create libNAME.so.X for AIX, except that they also create libNAME.so as symlink to libNAME.so.X.

In general, it is easier (=smaller patches) to replace the symlink with an Import File instead of introducing the old AIX shared lib model.

It is questionable if portage should replace a symlinked named 'lib*.so' by the Import File, especially as this is required already when linking subsequent libraries/executables inside one package.

Using Import Files does have another advantage: The linker does not have to read the shared object any more, which can drastically reduce linktime.

Because of this, and the fact that GNU ld does have '--as-needed', which also pokes package maintainers to fix their libraries to not have undefined symbols, I started to prefer Import Files over traditional shared archive members, even if we still have to support the latter.
Comment 9 Michael Haubenwallner (RETIRED) gentoo-dev 2009-11-24 10:46:16 UTC
dev-libs/openssl-0.9.8l-r1 creates import files on aix now.
Comment 10 Michael Haubenwallner (RETIRED) gentoo-dev 2009-12-03 08:42:50 UTC
Playing around with "Variant B1" in libtool testsuite shows that collect2 of gcc-3.4 *sometimes* doesn't like standalone import files. Haven't figured out what is different in some libtool tests compared to already working situations.
Although these specific libtool tests seem to work with >=gcc-4.4.

So I've added a "Variant B2", packing the import file libNAME.imp into libNAME.so, which becomes an archive then. This works with gcc-3.4 too.
Comment 11 Michael Haubenwallner (RETIRED) gentoo-dev 2009-12-03 08:51:20 UTC
Never ending story:
Now there is a hard reason to use import files with standalone shared objects (=not packed into an archive): We do have some java generator tool, which runs 'cpp' to prepare its idl input files.
What happens is:

*) 'cc1' (used by cpp) is linked against GNU "libiconv.a[libiconv.so.2]". The embedded runpath would point to the correct libiconv.a.
*) IBM-Java does set LIBPATH="/usr/lib:/multiple/path/to/java/libs:${LIBPATH}" for whatever reason.
*) 'cpp' now does find /usr/lib/libiconv.a, which obviously does not contain libiconv.so.2, and thus cannot start. Error message is:

exec(): 0509-036 Cannot load program /gentoo/prefix/usr/libexec/gcc/powerpc-ibm-aix5.3.0.0/3.4.6/cc1 because of the following errors:
  0509-150   Dependent module /usr/lib/libiconv.a(libiconv.so.2) could not be loaded.
  0509-152   Member libiconv.so.2 is not found in archive 

This would not happen if cc1 were linked against 'libiconv.so.2' (via import file libiconv.so) instead of 'libiconv.a(libiconv.so.2)' (via shared archive libiconv.a).
Comment 12 Michael Haubenwallner (RETIRED) gentoo-dev 2010-04-16 14:02:39 UTC
(In reply to comment #10)
> So I've added a "Variant B2", packing the import file libNAME.imp into
> libNAME.so, which becomes an archive then. This works with gcc-3.4 too.

As the shared object itself is packed into the archive too (second), it should have the F_LOADONLY (strip -e) set, as it may export more symbols (from static objects in /lib/libc.a) not listed in the import file. Have seen subsequent binaries needing both libNAME.so.X and libNAME.so(libNAME.so.X) at runtime because of this.
Comment 13 Michael Haubenwallner (RETIRED) gentoo-dev 2010-05-20 08:24:22 UTC
As I have good experience with my current ELT-patches for AIX, I've committed them to gx86 now.

This requires switching libtool to force -no-undefined behaviour on AIX.
Additionally, get_libname() now does return .so and versions like Linux.
Comment 14 Michael Haubenwallner (RETIRED) gentoo-dev 2011-01-14 15:25:19 UTC
(In reply to comment #11)
> Never ending story:
> Now there is a hard reason to use import files with standalone shared objects
> (=not packed into an archive):

It isn't necessary for the shared object to be standalone, but to be distinguished along the filename, even if that is an archive containing the shared object.

Although it doesn't apply for Prefix: The reason to prefer archives containing shared objects over standalone shared object is ... multilib: AIX traditionally does both 32bit and 64bit shared objects in one archive library. They even can have the same member name within one archive.

> This would not happen if cc1 were linked against 'libiconv.so.2' (via import
> file libiconv.so) instead of 'libiconv.a(libiconv.so.2)' (via shared archive
> libiconv.a).

In this sample, it also should have helped when linked against 'libiconv.so.2(shr.o)'.

This basically is Variant C, however the shared object's name always should be (something like) "shr.o" eventually.

Now time to convince libtool upstream...
Comment 15 Michael Haubenwallner (RETIRED) gentoo-dev 2011-01-21 14:08:16 UTC
Created attachment 260433 [details]
Variant A: Traditional variant to create shared libraries on AIX, but with runtime linking
Comment 16 Michael Haubenwallner (RETIRED) gentoo-dev 2011-01-21 14:09:56 UTC
Created attachment 260434 [details]
Variants B and C: Using Import Files to create shared libraries on AIX with runtime linking
Comment 17 Michael Haubenwallner (RETIRED) gentoo-dev 2011-02-09 09:09:22 UTC
Unfortunately, GNU mailing list manager does not preserve threads across archive split times. which is done montly at the libtool list.

So here are the current thread entry points per month:
Initial: http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
2011-02: http://lists.gnu.org/archive/html/libtool/2011-02/msg00000.html
Comment 18 Michael Haubenwallner (RETIRED) gentoo-dev 2011-02-09 16:36:32 UTC
Now sys-devel/native-cctools wraps AIX linker to support '-soname' and similar flags, to provide a little help for packages neither using libtool nor doing aix specifics at all.
Additionally, moved sys-devel/native-cctools to gx86.

This alone does not break existing Prefixes on AIX, but subsequent commits using the new shared library model either via -soname flag or via new still-to-be-committed ELT-patches will.
Comment 19 Michael Haubenwallner (RETIRED) gentoo-dev 2011-02-10 11:08:49 UTC
(In reply to comment #17)
> So here are the current thread entry points per month:
> Initial: http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
> 2011-02: http://lists.gnu.org/archive/html/libtool/2011-02/msg00000.html

Have extended the same poll to binutils list now:
http://sourceware.org/ml/binutils/2011-02/msg00099.html
Comment 20 Michael Haubenwallner (RETIRED) gentoo-dev 2011-02-10 15:35:01 UTC
Have done a bunch of commits now, switching to that shared library model.
Unfortunately, there is no upgrade path, as @preserved-rebuild cannot help here.

Keeping this bug open, as the final decision still isn't done yet.
Comment 21 Michael Haubenwallner (RETIRED) gentoo-dev 2011-02-11 09:42:46 UTC
(In reply to comment #19)
> > So here are the current thread entry points per month:
> > Initial: http://lists.gnu.org/archive/html/libtool/2011-01/msg00023.html
> > 2011-02: http://lists.gnu.org/archive/html/libtool/2011-02/msg00000.html
> 
> Have extended the same poll to binutils list now:
> http://sourceware.org/ml/binutils/2011-02/msg00099.html

Forgot to update the currently preferred variant here, which is also posted in the IBM AIX Developers forum:
http://www.ibm.com/developerworks/forums/thread.jspa?threadID=358893

*) Create the shared object "shr.o" (using '-G' linker flag).
*) Set the LOADONLY flag for "shr.o" (using 'strip -e').
*) Create the Import File "shr.imp", containing
   - this header line, being the "soname" emulation:
       #! libNAME.so.1(shr.o)
   - another header line telling the bits used:
       # 32
     or
       # 64
   - the list of symbols exported.
*) Create the archive library "libNAME.so.1.2.3" from both
   "shr.imp" and "shr.o".
*) Create the symlinks as usual:
   libNAME.so.1 -> libNAME.so.1.2.3
   libNAME.so -> libNAME.so.1.2.3
*) Create the static library archive "libNAME.a" from static objects.

While the big agreement on that variant is still not there - maybe because of
less interest (there's not much response anywhere), I've implemented that in
Prefix now for both prove-of-concept as well as to identify possible problems.

While there are a few packages that do not use this way still, there's no more package so far that creates standalone shared objects used as libraries (lib*.so*). I'm not sure if standalone shared objects used as modules (seen in python) actually should be archives (without Import File).

What the remaining packages do is the traditional variant, having libNAME.so.1 as archive member of libNAME.a. But here @preserved-rebuild works to update from the traditional variant to this new "aix-soname" variant - as long as any library is an archive.
Comment 22 Michael Haubenwallner (RETIRED) gentoo-dev 2011-06-17 12:21:20 UTC
(In reply to comment #21)
> What the remaining packages do is the traditional variant, having libNAME.so.1
> as archive member of libNAME.a. But here @preserved-rebuild works to update
> from the traditional variant to this new "aix-soname" variant - as long as any
> library is an archive.

The preserve-libs feature had a bug to not list the executables in the NEEDED.XCOFF.1 file, but the libraries only - leading to miss some packages in @preserved-rebuild.
I've just added a patch for portage-2.2.01.18826 for this, although without revbump, as this requires an 'emerge -e @system @world' anyway.

Additionally, gcc-4.2.4-r01.3 (without revbump) now also does full 'aix-soname' for libgcc_s and libstdc++, that is: no libgcc_s.a and libstdc++.a any more as shared libraries.

Therefore, gcc-config-1.4.1-r00.2 (again without revbump) now can do the ELF-thing even on AIX.

So either you start over from scratch (once again) - or try this upgrade path:
1) cp usr/powerpc-ibm-aix*/lib/gcc/libstdc++.a lib/ # to keep aixdll-query working
2) emerge --sync
3) emerge -1 portage            # for the preserve-libs portage bugs
4) emerge -1 gcc-config gcc     # for libgcc_s.so and libstdc++.so
5) emerge -e @system @world     # to list executables into NEEDED.XCOFF.1
6) rm lib/libgcc_s.a lib/libstdc++.a  # clean up orphaned files
Comment 23 Fabian Groffen gentoo-dev 2011-12-15 18:04:15 UTC
haubi, correct me if I'm wrong, I think you made your choice here eventually.
Comment 24 Michael Haubenwallner (RETIRED) gentoo-dev 2011-12-16 07:40:48 UTC
Indeed, the choice is done, and I shouldn't abuse this bug to track the upstream implementation.