Bug 667854 – /usr/bin/armv7a-unknown-linux-gnueabihf-strip fails to find various link sections while striping statically linked files during src_install (affected packages: libcxx-7.0.1, libcxxabi-7.0.1, compiler-rt-7.0.1, compiler-rt-plugins-7.0.1)

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description tt_1 2018-10-05 21:44:59 UTC]

Created attachment 549484 [details]
compressed build logs

followup from #667174

this might be related to stripping of statically linked files only

I'm going to attach all four log files, and a list of llvm/clang updates from 6.0.1 to 7.0.0 which I ran today. At first I've been suspicious that something went wrong during the upgrade, but as I uninstalled the whole llvm/clang toolchain and recompiled it from scratch, the emerge produced the very same warnings as before.

[Comment 1 tt_1 2018-10-05 21:57:27 UTC]

Created attachment 549486 [details]
output of emerge --info

my emerge --info 

(from qemu-static chroot)

since /usr/bin/armv7a-unknown-linux-gnueabihf-strip is part of binutils - I would like to raise attention to the fact that I upgraded binutils (and binutils-libs) just before doing the upgrade of llvm/clang from 6.0.1 to 7.0.0

I'm fairly certain that this behaviour didn't happen with the stable binutils and llvm/clang-6.0.1

I can offer to reproduce the combination of binutils-2.30-r4 and clang/llvm-6.0.1 with a fresh stage upon request (it still takes 3 hrs within a qemu-static chroot)

[Comment 2 tt_1 2018-10-06 19:10:38 UTC]

Note to myself: downgrading binutils to 2.30-r3 from -r4 did not make this vanish. Didn't yet do a rebuild of the whole llvm/clang toolchain due to a lack of spare cycles.

[Comment 3 tt_1 2018-10-07 08:14:34 UTC]

Grabbed a fresh armv7-stage3 and reproduced this with the stable binutils-2.30-r2

[Comment 4 tt_1 2018-10-07 17:45:11 UTC]

Also ruled out: this has nothing to do with USE="static-analyzer"; flipping the flag has no meaning for the linking. 

[ebuild   R   *] sys-devel/clang-7.0.0:7::gentoo  USE="static-analyzer -debug -default-compiler-rt -default-libcxx -doc {-test} -xml (-z3)" LLVM_TARGETS="(ARM) -AArch64 -AMDGPU -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -Sparc -SystemZ -X86 -XCore" PYTHON_TARGETS="python2_7" 0 KiB

[Comment 5 tt_1 2018-10-09 12:08:20 UTC]

So this seems to be related to the statically linked files (don't know if these are binaries or libraries) in all of the four packages. It doesn't matter if gcc is used or clang itself for compiling. 

Todo: test llvm/clang-6 from a clean stage3

[Comment 6 tt_1 2018-10-09 16:15:23 UTC]

llvm/clang-6.0.1 is clean of the linking failures. 

@jer - Since I flipped stable and unstable binutils with llvm/clang-7.0.0, without any different behavior, would you mind to assing this to the llvm team?

[Comment 7 Michał Górny 2018-10-15 15:22:41 UTC]

Could you test if using lld to link makes any difference?

[Comment 8 tt_1 2018-10-15 15:35:42 UTC]

Sure, it's scheduled for tomorrow. 

I'm going to set via package.env: 

LDFLAGS="${LDFLAGS},-fuse-ld=lld" 

And build with gcc, unless there's a clang useflag, for all llvm/clang related packages.

[Comment 9 Michał Górny 2018-10-15 16:20:48 UTC]

I don't think gcc supports that.

[Comment 10 tt_1 2018-10-15 17:08:43 UTC]

Created attachment 551336 [details]
compressed build logs

Ok, so I did only libcxx-7.0.0, libcxxabi-7.0.0, compiler-rt-7.0.0 and compiler-rt-plugins-7.0.0 with: 

CC="clang"
CXX="clang++"
LDFLAGS="${LDFLAGS},-fuse-ld=lld" 

is this what you intended?

[Comment 11 Michał Górny 2018-10-15 17:43:44 UTC]

Nope.  '-fuse-ld=lld' must go to clang, not the linker, so without '-Wl,'.

[Comment 12 tt_1 2018-10-15 18:23:19 UTC]

Sorry, but I don't understand how to enable that, package.env wise. 

LDFLAGS="${LDFLAGS},-fuse-ld=lld"  is wrong, got that. You want me to use '-fuse-ld=lld' like a C/CPP flag?

[Comment 13 Michał Górny 2018-10-15 18:28:01 UTC]

Just put space instead of ','.

[Comment 14 tt_1 2018-10-15 18:39:50 UTC]

Created attachment 551404 [details]
compressed build log from compiler-rt-7.0.0

Ok, got it. Now there is four times -fuse-ld=lld in the flags?

And there are still the same warnings about link sections not being found. 

Injected values from package.env are: 

LDFLAGS="${LDFLAGS} -fuse-ld=lld"

[Comment 15 Michał Górny 2018-10-15 18:47:45 UTC]

Ok, I'm sorry, I was really stupid and missed the fact that compiler-rt is just static libs, so it doesn't use link editor at all.

@arm, can you reproduce something like it?  Can it be related to the errors from qemu?

[Comment 16 tt_1 2018-10-15 20:16:14 UTC]

I'm almost certain that I hit this the first time when compiling llvm/clang nativly, but to be sure I did set up a chroot: 

Calculating dependencies... done!
[ebuild  N    ~] sys-devel/llvm-common-7.0.0::gentoo  0 KiB
[ebuild  N    *] sys-libs/libomp-7.0.0::gentoo  USE="(-cuda) -hwloc -offload -ompt -test" 890 KiB
[ebuild  N    ~] sys-devel/llvm-7.0.0:7::gentoo  USE="gold libffi ncurses -debug -doc (-exegesis) -libedit -test -xar -xml" LLVM_TARGETS="(ARM) -AArch64 -AMDGPU -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -Sparc -SystemZ -X86 -XCore" 0 KiB
[ebuild  N    ~] sys-devel/llvmgold-7::gentoo  0 KiB
[ebuild  N    *] sys-libs/llvm-libunwind-7.0.0::gentoo  USE="static-libs -debug -test" 78 KiB
[ebuild  N    *] sys-devel/lld-7.0.0::gentoo  USE="-test" 895 KiB
[ebuild  N    *] sys-libs/libcxxabi-7.0.0::gentoo  USE="libunwind static-libs -test" 0 KiB
[ebuild  N    *] sys-libs/libcxx-7.0.0::gentoo  USE="libcxxabi libunwind static-libs -libcxxrt -test" 0 KiB
[ebuild  N    *] sys-devel/clang-7.0.0:7::gentoo  USE="-debug -default-compiler-rt -default-libcxx -doc -static-analyzer -test -xml (-z3)" LLVM_TARGETS="(ARM) -AArch64 -AMDGPU -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -Sparc -SystemZ -X86 -XCore" PYTHON_TARGETS="python2_7" 882 KiB
[ebuild  N    *] sys-devel/clang-common-7.0.0::gentoo  0 KiB
[ebuild  N    *] sys-libs/compiler-rt-7.0.0:7.0.0::gentoo  USE="clang -test" 0 KiB
[ebuild  N    *] sys-libs/compiler-rt-sanitizers-7.0.0:7.0.0::gentoo  USE="clang -test" 0 KiB
[ebuild  N    *] sys-devel/clang-runtime-7.0.0:7.0.0::gentoo  USE="compiler-rt libcxx openmp sanitize -crt" 0 KiB

Total: 13 packages (13 new), Size of downloads: 2.743 KiB

it's going to take a bit more than 24hrs for an answer.

[Comment 17 tt_1 2018-10-17 06:23:21 UTC]

I don't think this is related to the qemu chroot. The native compile finished, and apart from the obvious qemu: Unsupported syscall: 383 and 391 there is no difference in the log. 100% the same warnings, same files, same segments. 

However, I added the patchset from libcxxabi git to make it compile when gcc is used, and in this case the stripping of the static files didn't trigger the warnings. But it's not the patchset solving this, since I applied it anyway using CC/X=clang/++ in the logs of my initial posts which produced the warnings. 

So, do we agree this happens when stripping static-libs build by clang-7, on arm?

[Comment 18 tt_1 2018-10-17 07:47:30 UTC]

Created attachment 551634 [details]
build log from app-arch/bzip2

I made a short test to compile app-arch/bzip USE="static-libs" and clang instead of gcc, and results are as expected: 

strip: armv7a-unknown-linux-gnueabihf-strip --strip-unneeded -R .comment -R .GCC.command.line -R .note.gnu.gold-version
   lib/libbz2.so.1.0.6
   usr/bin/bzip2recover
   bin/bzip2
   usr/lib/libbz2.a
armv7a-unknown-linux-gnueabihf-strip: /var/tmp/portage/app-arch/bzip2-1.0.6-r10/image/usr/lib/stImUpsE/bzlib.o: Failed to find link section for section 11
armv7a-unknown-linux-gnueabihf-strip: /var/tmp/portage/app-arch/bzip2-1.0.6-r10/image/usr/lib/stImUpsE/bzlib.o: Failed to find link section for section 11

[Comment 19 tt_1 2018-10-17 20:46:52 UTC]

I think I found it. 

With clang-7 there is a new flag (cflag?) called -faddrsig, and it seems to be broken on arm. Disabling it with -fno-addrsig resolves the warning during striping of static-libs. 

How did I found out? I used readelf to fuzz with the misslinked static-libs and found out that they are linked to .llvm_addrsig. Clang-6.0.1 didn't produce static-libs with that value, and upon greping through the clang source code I found llvm_addrsig being linked to -faddrsig/-fno-addrsig. 

Disabled it via package.env:

CFLAGS="${CFLAGS} -fno-addrsig"

problem solved. 


@mgorny: Is there a way to mitigate the issue for the portage tree? I mean, it's possible to mask package, use flags, but cflags?

[Comment 20 tt_1 2018-10-17 20:51:57 UTC]

Created attachment 551754 [details]
static lib, readelf output and build log of compiler-rt

Adding a sample (static-lib, build log and output from readelf) from compiler-rt

It seems not all instances of addrsig are affected.

[Comment 21 Michał Górny 2018-10-19 23:01:23 UTC]

Thank you for the detailed analysis.  Could you try to create a small test case for it?  I mean, using some small code instead of those big libraries.

@toolchain, could you look whether it's specifically an issue with LLVM or with binutils?

[Comment 22 Sergei Trofimovich (RETIRED) 2018-10-20 11:06:19 UTC]

Does not seem to be qemu-specific. Cross-compiling on amd64 yields the same index  mismatch:

$ LANG=C USE=static-libs CC="/usr/lib64/llvm/7/bin/clang -target armv7a-unknown-linux-gnueabihf --sysroot=/usr/armv7a-unknown-linux-gnueabihf" CFLAGS="-march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard" armv7a-unknown-linux-gnueabihf-emerge -v1 app-arch/bzip2 --quiet-build=n
...
armv7a-unknown-linux-gnueabihf-strip: /tmp/portage-tmpdir/portage/app-arch/bzip2-1.0.6-r10/image/usr/lib/stSjRW4N/blocksort.o: failed to find link section for section 10
armv7a-unknown-linux-gnueabihf-strip: /tmp/portage-tmpdir/portage/app-arch/bzip2-1.0.6-r10/image/usr/lib/stSjRW4N/blocksort.o: failed to find link section for section 10
...

Minimal reproducer:

    $ echo 'void f(void){}' > a.c
    $ /usr/lib64/llvm/7/bin/clang -target armv7a-unknown-linux-gnueabihf -c a.c -o a.o
    $ LANG=C armv7a-unknown-linux-gnueabihf-strip --strip-unneeded -R .comment -R .GCC.command.line -R .note.gnu.gold-version a.o
armv7a-unknown-linux-gnueabihf-strip: st5EpxDK: failed to find link section for section 6
armv7a-unknown-linux-gnueabihf-strip: st5EpxDK: failed to find link section for section 6

Neither readelf not elflint understands the section type of .llvm_addrsig:

    $ LANG=C readelf -a a.o
    Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .strtab           STRTAB          00000000 000105 000066 00      0   0  1
  [ 2] .text             PROGBITS        00000000 000034 000004 00  AX  0   0  4
  [ 3] .ARM.exidx        ARM_EXIDX       00000000 000038 000008 00  AL  2   0  4
  [ 4] .rel.ARM.exidx    REL             00000000 0000fc 000008 08      9   3  4
  [ 5] .comment          PROGBITS        00000000 000040 00002e 01  MS  0   0  1
  [ 6] .note.GNU-stack   PROGBITS        00000000 00006e 000000 00      0   0  1
  [ 7] .ARM.attributes   ARM_ATTRIBUTES  00000000 00006e 00003c 00      0   0  1
  [ 8] .llvm_addrsig     LOOS+0xfff4c03  00000000 000104 000001 00   E  9   0  1
  [ 9] .symtab           SYMTAB          00000000 0000ac 000050 10      1   4  4

    $ eu-elflint a.o
  section [ 8] '.llvm_addrsig' has unsupported type 1879002115
  section [ 8] '.llvm_addrsig' contains invalid processor-specific flag(s) 0x80000000

It does not seem to be arm-specific either:

  $ cat a.c
  void g(void){}
  $ /usr/lib64/llvm/7/bin/clang -c a.c -o b.o
  $ LANG=C strip --strip-unneeded -R .comment -R .GCC.command.line -R .note.gnu.gold-version b.o
  strip: stNViIFP: failed to find link section for section 5
  strip: stNViIFP: failed to find link section for section 5
  $ LANG=C file b.o
  b.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

https://reviews.llvm.org/D47744 added new LLVM-specific section header type. I don't see support of it in binutils.

Thus warnings of not being able to process this section are expected.

At a glance I don't understand what exactly those sections contain. It seems that data is symbol indices and not relocations. strip probably breaks that section.

[Comment 23 tt_1 2018-10-24 20:08:50 UTC]

@Sergej: thank you for your effort, if I understand your posting correctly it means that this is a bug in binutils then?

[Comment 24 Sergei Trofimovich (RETIRED) 2018-10-24 20:49:15 UTC]

It's a good question. I'm not sure. I see a few avenues to explore:

1. On binutils side perhaps binutils should not strip binaries with section types it does not understand and leave them as-is (and emit a warning). I'll file a binutils bug and we'll see how it goes.

2. On user's side I suggest trying STRIP=llvm-strip in cross-make.conf to avoid corrupting binaries. I'm not sure about right syntax. Or FEATURES=nostrip as a last-resort hammer.

3. On portage's side portage's strip could also detect presence of such sections and avoid stripping.

4. On clang's side enabling -faddrsig unconditionally might be too optimistic at least in Gentoo. But it also conveys important information about symbols.

[Comment 25 Sergei Trofimovich (RETIRED) 2018-10-24 22:05:23 UTC]

(In reply to Sergei Trofimovich from comment #24)
> 1. On binutils side perhaps binutils should not strip binaries with section
> types it does not understand and leave them as-is (and emit a warning). I'll
> file a binutils bug and we'll see how it goes.

Filed https://sourceware.org/PR23817 for additional input.

[Comment 26 Sergei Trofimovich (RETIRED) 2018-10-25 07:05:43 UTC]

> > 1. On binutils side perhaps binutils should not strip binaries with section
> > types it does not understand and leave them as-is (and emit a warning). I'll
> > file a binutils bug and we'll see how it goes.
> 
> Filed https://sourceware.org/PR23817 for additional input.

Alan explained the situation in https://sourceware.org/PR23817#c1.
I suggest coming back to llvm upstream and deciding there what should be done about incompatibility.

[Comment 27 Michał Górny 2018-10-25 10:39:21 UTC]

I'm going to see if we cam get it disabled upstream by default.

[Comment 28 Michał Górny 2018-11-02 20:26:07 UTC]

Upstream pointed out to me that it happens only when stripping object files (and therefore static archives) but not executables.  Maybe we should consider stopping to strip those instead, as suggested in bug 603594.  It seems that premature stripping of objects is not really necessary, and is apt to cause issues.

[Comment 29 Michał Górny 2018-11-02 20:37:11 UTC]

Created attachment 553912 [details, diff]
gentoo-fnoaddrsig.patch (for clang)

Alternatively, please test the attached clang patch.  It switches the default for Gentoo systems to -fno-addrsig; though I honestly don't like diverging from upstream here.

[Comment 30 tt_1 2018-11-03 16:13:42 UTC]

The patch from #28 is going to make it into portage-2.3.52, right? And the clang patch is against clang-9999 it seems?

[Comment 31 Michał Górny 2018-11-03 16:16:40 UTC]

(In reply to tt_1 from comment #30)
> The patch from #28 is going to make it into portage-2.3.52, right? And the
> clang patch is against clang-9999 it seems?

Yes and yes.  However, it should be rather easy to rebase.

[Comment 32 tt_1 2018-11-10 23:09:17 UTC]

Could you push out a portage-2.3.51-r1 with 2404ddca9d5db7992bf6853cbde8ca944224560c, in case 2.3.52 isn't ready when llvm/clang-7.0.1 hits the tree?

[Comment 33 tt_1 2018-11-17 21:47:38 UTC]

Created attachment 555474 [details, diff]
patch backported to clang-7.0.1

So, I took a fresh stage and used qemu-static for faster results: 

portage-9999 doesn't prevent the beforementioned errors/warnings with static linking. 

clang-9999 + patch from #29 does work, which means the warnings/errors are gone from the logs. I'll add some examples and logs by tomorrow. 

@mgorny: I tried to backport the patch to 7.0.1_rc2, is it correct?

[Comment 34 Michał Górny 2018-12-21 21:47:46 UTC]

I still don't have a strong opinion on this but with more targets actually disabling -faddrsig, I suppose Gentoo could do the same.  However, I need to also fix tests for it.

[Comment 35 Michał Górny 2018-12-30 09:59:49 UTC]

This is now in trunk.  I'll probably backport it into -r1 before it goes stable.

[Comment 36 tt_1 2019-03-20 16:13:24 UTC]

fixed in clang-8.0.0, thanks for all the effort! :-)