Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 603594 - portage's prestrip phase breaks LTO static libraries
Summary: portage's prestrip phase breaks LTO static libraries
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Ebuild Support (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL: https://sourceware.org/bugzilla/show_...
Whiteboard:
Keywords: PATCH
: 591542 (view as bug list)
Depends on:
Blocks: lto 671498
  Show dependency tree
 
Reported: 2016-12-23 16:56 UTC by Sergei Trofimovich (RETIRED)
Modified: 2022-08-26 04:31 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
estrip: Fix static library index (estrip-Fix-static-library-index-bug-603594.patch,1.24 KB, patch)
2018-10-19 16:44 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sergei Trofimovich (RETIRED) gentoo-dev 2016-12-23 16:56:38 UTC
Tl;DR:
    'prestrip' ebuild helper needs a 'gcc-ranlib' call after 'strip' to stop breaking -flto users.

Longer version:

The problem occasionally gets reported
    https://forums.gentoo.org/viewtopic-t-1014912-start-0.html
but it's not a widely known fact. Let's do it here.

How to reproduce:
  make.conf:
    CHOST="x86_64-pc-linux-gnu"

    CFLAGS="-O2 -pipe -flto"
    CXXFLAGS="${CFLAGS}"
    LDFLAGS="${LDFLAGS} ${CFLAGS}"

    AR=${CHOST}-gcc-ar
    NM=${CHOST}-gcc-nm
    RANLIB=${CHOST}-gcc-ranlib

    ACCEPT_KEYWORDS="${ACCEPT_KEYWORDS} ~$ARCH"

  Building an example static package (USE=static btrfsprogs):

    # FEATURES="nostrip" USE="static static-libs" emerge -1 e2fsprogs e2fsprogs-libs lzo zlib util-linux btrfs-progs -j5
    <works>

    # USE="static static-libs" emerge -1 e2fsprogs e2fsprogs-libs lzo zlib util-linux btrfs-progs -j5
    <fails as:>

/dev/shm/portage/sys-fs/btrfs-progs-4.8.5/temp/cc6590N6.ltrans5.ltrans.o: In function `cmd_filesystem_show.lto_priv.526':
<artificial>:(.text.cmd_filesystem_show.lto_priv.526+0x15b): undefined reference to `uuid_unparse'
<artificial>:(.text.cmd_filesystem_show.lto_priv.526+0x396): undefined reference to `uuid_unparse'
<artificial>:(.text.cmd_filesystem_show.lto_priv.526+0x753): undefined reference to `uuid_unparse'
/dev/shm/portage/sys-fs/btrfs-progs-4.8.5/temp/cc6590N6.ltrans5.ltrans.o:<artificial>:(.text.cmd_filesystem_show.lto_priv.526+0xaaf): more undefined references to `uuid_unparse' follow

The workarounds:
    1. use FEATURES=nostrip
    2. use CFLAGS=-ffat-lto-objects
    3. run manually 'gcc-ranlib' on already installed libs to unbreak them. Breaks checksums :(

The problem here lies in how strip treats .a files created by gcc-ar / -flto. One of last lines: of USE=static-libs emerge -1 zlib:

strip: x86_64-pc-linux-gnu-strip --strip-unneeded -R .comment -R .GCC.command.line -R .note.gnu.gold-version
   lib64/libz.so.1.2.8
   usr/lib64/libz.a
x86_64-pc-linux-gnu-strip: /dev/shm/portage/sys-libs/zlib-1.2.8-r1/image/usr/lib64/stzjlms5/adler32.o: plugin needed to handle lto object
x86_64-pc-linux-gnu-strip: /dev/shm/portage/sys-libs/zlib-1.2.8-r1/image/usr/lib64/stzjlms5/crc32.o: plugin needed to handle lto object
x86_64-pc-linux-gnu-strip: /dev/shm/portage/sys-libs/zlib-1.2.8-r1/image/usr/lib64/stzjlms5/deflate.o: plugin needed to handle lto object
...

as a result strip does not rebuild index and .a library is not useful
for linking. It can be fixed later by running gcc-ranlib:

# gcc-ranlib /usr/lib64/libz.a /usr/lib64/libuuid.a /usr/lib64/libblkid.a /usr/lib64/liblzo2.a /usr/lib64/libext2fs.a /usr/lib64/libcom_err.a
# USE=static emerge -1 btrfs-progs
<works again>

Thanks!
Comment 1 Steffen Hau 2017-05-16 14:58:19 UTC
Some more packages causing build errors of other packages:

dev-libs/crossguid-0_pre20150817 -> media-tv/kodi-9999
dev-qt/designer-5.7.1 -> kde-plasma/kwin-5.9.5
dev-qt/qtgui-5.7.1 -> dev-qt/qtwayland-5.7.2

I'm also using FEATURE="nostrip" as workaround.
Comment 2 Steffen Hau 2017-05-17 08:36:19 UTC
The bug has been reported upstream: https://sourceware.org/bugzilla/show_bug.cgi?id=21479
Comment 3 Alexander Miller 2018-03-14 15:57:18 UTC
If you want to avoid FEATURE="nostrip" you can use STRIP_MASK="*/lib*.a" instead as a more fine-grained workaround.
Comment 4 Shane Peelar 2018-05-23 15:12:48 UTC
What should we do for EAPI 7? It appears that STRIP_MASK is ignored for this EAPI version. We really depend on this for lto-overlay, and we would appreciate any help we could get.
Comment 5 Zac Medico gentoo-dev 2018-05-23 15:19:11 UTC
(In reply to Shane Peelar from comment #4)
> What should we do for EAPI 7? It appears that STRIP_MASK is ignored for this
> EAPI version. We really depend on this for lto-overlay, and we would
> appreciate any help we could get.

A call to dostrip in src_install like this should do the trick:

> set -o noglob
> has ${EAPI} {0..6} || dostrip -x ${STRIP_MASK}
> set +o noglob
Comment 6 Zac Medico gentoo-dev 2018-05-23 15:31:19 UTC
Actually, dostrip -x does not support globs, so if you need globs then you'll have to do something like this:

> set -o noglob
> if ! has ${EAPI} {0..6}; then
>    pushd "${ED}" || die
>    local x y
>    for x in ${STRIP_MASK}; do
>        set +o noglob
>        for y in "${ED%/}"/${x#/}; do
>           dostrip -x "${y#${ED%/}}"
>        done
>        set -o noglob
>    done
>    popd
> set +o noglob
Comment 7 Steven Newbury 2018-10-19 07:59:29 UTC
Why are static libraries stripped anyway? If it's just for space saving, does it really make any sense since the common case of splitdebug means it is a best awash.

It's not going to make any difference to application load times or memory usage since any binary using static libraries is going to be stripped itself anyway.

It just seems to me to be a hold over from the days when *many* packages were installed with static libraries by default, and prior to splitdebug, when space savings were paramount.

What is the current rationale?
Comment 8 Zac Medico gentoo-dev 2018-10-19 16:44:39 UTC
Created attachment 551860 [details, diff]
estrip: Fix static library index

(In reply to Steven Newbury from comment #7)
> What is the current rationale?
According to comments, unstripped are archives tend to be very large, there was talk of putting the files in a -debuginfo package:

https://bugzilla.redhat.com/show_bug.cgi?id=1395280

Maybe it makes sense to split static libraries into a -static-libs package. There could even be two variants, one stripped and one unstripped.

I've already got plans to add support for separate debuginfo and source packages, see bug 639376.

> If it's just for space saving, does it really make any sense since the common
> case of splitdebug means it is a best awash.
In fact portage does not strip static libraries when splitdebug is enabled, the code has this comment about it:

> # If we have split debug enabled, then do not strip this.
> # There is no concept of splitdebug for objects not yet
> # linked in (only for finally linked ELFs), so we have to
> # retain the debug info in the archive itself.
Comment 9 Larry the Git Cow gentoo-dev 2018-10-31 16:37:55 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=2404ddca9d5db7992bf6853cbde8ca944224560c

commit 2404ddca9d5db7992bf6853cbde8ca944224560c
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2018-10-29 16:41:45 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2018-10-31 16:37:35 +0000

    estrip: Run RANLIB after stripping static archives to fix LTO
    
    Run RANLIB after stripping static archives in order to fix potentially
    mangled LTO symbol indexes.  According to the user's report, strip lacks
    support for LTO symbols (and for binutils plugins that could add
    the missing support) and breaks them.  Upstream suggests fixing
    the archive by running ranlib -- and since there should be no harm
    in doing that, we can do that unconditionally.
    
    Suggested-by: Shane Peelar
    Bug: https://github.com/mgorny/portage-mgorny/issues/21
    Bug: https://bugs.gentoo.org/603594
    Reviewed-by: Zac Medico <zmedico@gentoo.org>
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 bin/estrip | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Comment 10 Zac Medico gentoo-dev 2019-04-10 04:36:38 UTC
Fixed in portage-2.3.62.
Comment 11 Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-29 14:06:42 UTC
*** Bug 591542 has been marked as a duplicate of this bug. ***