Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 600978

Summary: sys-devel/binutils: BFD assertion failure in bfd/elfxx-mips.c:9020 during mips64 cross-toolchain build w/ gcc-6.2.0
Product: Gentoo Linux Reporter: Joshua Kinard <kumba>
Component: Current packagesAssignee: Gentoo Crossdev team <crossdev>
Status: RESOLVED FIXED    
Severity: normal CC: mips, slyfox
Priority: Normal    
Version: unspecified   
Hardware: MIPS   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 627914    

Description Joshua Kinard gentoo-dev 2016-11-27 11:36:59 UTC
I've about given up on this issue.  It looks like bad code generation via gcc-6.2.0 when that compiler is used to build binutils, but which only manifests itself during crossdev runs to build a full stage4 cross-toolchain (with gdb).  I've tried with binutils-2.25, 2.26, and 2.27, as well as glibc-2.23 and 2.24.  gcc-5.x works fine to build a cross-toolchain, but not 6.2.0.

In a real userland on a working MIPS machine, this problem does not appear at all.  I've fully compiled gcc-6.2.0 and binutils-2.27 in both glibc and uclibc-ng chroots, rebuilt libtool, then rebuilt as much of @system as I could, with the only failures being eudev due to Bug #580140 and dev-util/pkgconfig due to Bug #600378.

Here's the relevant error:
mips64-unknown-linux-gnu-gcc -mabi=64 -Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed  -shared -static-libgcc -Wl,-O1  -Wl,-z,defs -Wl,-dynamic-linker=/lib64/ld.so.1 -B/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl/ -B/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/csu/ -B/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl/ -Wl,--version-script=/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/libpthread.map -Wl,-soname=libpthread.so.0 -Wl,-z,relro -Wl,--enable-new-dtags,-z,nodelete,-z,initfirst -e __nptl_main -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/math -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/elf -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/dlfcn -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nss -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nis -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/rt -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/resolv -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/crypt -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/mathvec -L/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl -Wl,-rpath-link=/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/math:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/elf:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/dlfcn:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nss:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nis:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/rt:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/resolv:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/crypt:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/mathvec:/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl -o /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl/libpthread.so -T /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/shlib.lds /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/csu/abi-note.o -Wl,--whole-archive /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl/libpthread_pic.a -Wl,--no-whole-archive  -Wl,--start-group /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/libc.so /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/libc_nonshared.a -Wl,--as-needed /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/elf/ld.so -Wl,--no-as-needed -Wl,--end-group
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
/usr/libexec/gcc/mips64-unknown-linux-gnu/ld: BFD (Gentoo 2.27 p1.0) 2.27 assertion fail /ramfs/portage/cross-mips64-unknown-linux-gnu/binutils-2.27/work/binutils-2.27/bfd/elfxx-mips.c:9020
collect2: error: ld returned 1 exit status
make[2]: *** [../Makerules:541: /ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl/nptl/libpthread.so] Error 1
make[2]: Leaving directory '/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/glibc-2.24/nptl'
make[1]: *** [Makefile:215: nptl/others] Error 2
make[1]: Leaving directory '/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/glibc-2.24'
make: *** [Makefile:9: all] Error 2
make: Leaving directory '/ramfs/portage/cross-mips64-unknown-linux-gnu/glibc-2.24/work/build-n64-mips64-unknown-linux-gnu-nptl'
 * ERROR: cross-mips64-unknown-linux-gnu/glibc-2.24::local failed (compile phase):
 *   emake failed


When I go to look at the source of binutils, Line 9020 in bfd/elfxx-mips.c, there's no block of code that appears to match that assertion failure.  So I assume the line number is only after some preprocessing is done, which makes it all-but-impossible to figure out where the broken code is even at.

I haven't opened an upstream bug yet, as I am not 100% sure if this is something coming from our local patches or a legit gcc-6.2 bug.
Comment 1 Felix Janda 2016-11-27 13:25:43 UTC
Around line 9020, we have assertion:

BFD_ASSERT (dynobj != NULL
            && (h->needs_plt
                || h->u.weakdef != NULL
                || (h->def_dynamic
                    && h->ref_regular
                    && !h->def_regular)));

Can you reproduce the linking failure by manually executing the line
from your log? Can the same files be linked successfully with binutils
compiled from an earlier version of gcc?
Comment 2 Joshua Kinard gentoo-dev 2016-11-27 13:33:30 UTC
(In reply to Felix Janda from comment #1)
> Around line 9020, we have assertion:
> 
> BFD_ASSERT (dynobj != NULL
>             && (h->needs_plt
>                 || h->u.weakdef != NULL
>                 || (h->def_dynamic
>                     && h->ref_regular
>                     && !h->def_regular)));
> 
> Can you reproduce the linking failure by manually executing the line
> from your log? Can the same files be linked successfully with binutils
> compiled from an earlier version of gcc?

I haven't tried manually executing that line yet.  Have to put that on the list for later, though I doubt it'll change much.

For the other question, yes, gcc-5.4.0 has no problems on binutils-2.25 at least.  I think I tried 2.26.1 as well, but there was another, unrelated, issue w/ glibc (vfork aliases) that forced me to back down to 2.25 until glibc-2.24 came out.  gcc-6.2.0, however, causes this issue reliably on binutils 2.25 to 2.27, so the fault is likely in gcc itself generating bad code in one of the objects that's triggering the assertion.  There's a lot of objects that'd need checking, though, and I know little about debugging binutils.
Comment 3 Andreas K. Hüttel archtester gentoo-dev 2017-10-11 20:06:44 UTC
How about newer gcc (6.4)?
Comment 4 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-12 09:17:51 UTC
Last time I looked USE=-pie on gcc helped me to restore cross-building toolchain on mips.

Given it's a binutils assertion failure I would suspect a binutils bug to fail to handle PIC/PIE. Gentoo enabled PIE-by-default on gcc-6.
Comment 5 Joshua Kinard gentoo-dev 2017-11-08 14:34:00 UTC
(In reply to Andreas K. Hüttel from comment #3)
> How about newer gcc (6.4)?

I've confirmed it's still affected in gcc-7.2.x.


(In reply to Sergei Trofimovich from comment #4)
> Last time I looked USE=-pie on gcc helped me to restore cross-building
> toolchain on mips.
> 
> Given it's a binutils assertion failure I would suspect a binutils bug to
> fail to handle PIC/PIE. Gentoo enabled PIE-by-default on gcc-6.

Well, the odd thing is, this failure only happens on crossdev builds on my x86_64 box.  Native compiles work fine up to gcc-6.4.0 and binutils-2.29*.  So if it is PIE-related, it's specific to something in the logic that only happens in a cross-compile scenario.  And so far, at least with glibc.  Crossdev can't handle uclibc-ng just yet, so I can't test other libc's at the moment.
Comment 6 Sergei Trofimovich (RETIRED) gentoo-dev 2017-11-08 20:26:37 UTC
(In reply to Joshua Kinard from comment #5)
> (In reply to Andreas K. Hüttel from comment #3)
> > How about newer gcc (6.4)?
> 
> I've confirmed it's still affected in gcc-7.2.x.
> 
> 
> (In reply to Sergei Trofimovich from comment #4)
> > Last time I looked USE=-pie on gcc helped me to restore cross-building
> > toolchain on mips.
> > 
> > Given it's a binutils assertion failure I would suspect a binutils bug to
> > fail to handle PIC/PIE. Gentoo enabled PIE-by-default on gcc-6.
> 
> Well, the odd thing is, this failure only happens on crossdev builds on my
> x86_64 box.  Native compiles work fine up to gcc-6.4.0 and binutils-2.29*. 
> So if it is PIE-related, it's specific to something in the logic that only
> happens in a cross-compile scenario.  And so far, at least with glibc. 
> Crossdev can't handle uclibc-ng just yet, so I can't test other libc's at
> the moment.

There is a few things to keep in mind with crossdev:

1. crossdev generates a package named cross-*/gcc and things like 'package.mask sys-libs/gcc pie' do not apply to it
2. when you emerge 'cross-*/gcc' in system you install it on host's ARCH which means you are using amd64 profile, not mips.

Thus yes, it's expected pie is masked on native system but not masked in cross-environment. It's easy to verify by building cross-toolchain with USE=-pie.

I personally have the following package.use:

  /etc/portage/package.use/haskell:cross-mips64-unknown-linux-gnu/ghc -gmp
  /etc/portage/package.use/cross:cross-*/gcc::crossdev -fortran
  /etc/portage/package.use/cross:cross-*/gcc::crossdev -sanitize
  /etc/portage/package.use/cross:cross-*/gcc::crossdev -ssp
  /etc/portage/package.use/cross:cross-*/gcc::crossdev -vtv

because quite a few targets are broken. And it's an upstream issue thus
I was not too eager to report gentoo bugs.
Comment 7 Sergei Trofimovich (RETIRED) gentoo-dev 2017-12-28 19:58:01 UTC
How are things on today's toolchain? Looks like the following toolchain Just Works:

- cross-mips64-unknown-linux-gnu/gcc-7.2.0
- cross-mips64-unknown-linux-gnu/glibc-2.26-r4
- cross-mips64-unknown-linux-gnu/binutils-2.29.1-r1
Comment 8 Joshua Kinard gentoo-dev 2017-12-30 23:31:56 UTC
(In reply to Sergei Trofimovich from comment #7)
> How are things on today's toolchain? Looks like the following toolchain Just
> Works:
> 
> - cross-mips64-unknown-linux-gnu/gcc-7.2.0
> - cross-mips64-unknown-linux-gnu/glibc-2.26-r4
> - cross-mips64-unknown-linux-gnu/binutils-2.29.1-r1

I'll retest and try disabling PIE to see what happens.

Also, when did the -v flag to crossdev get removed?  I like seeing the build output.  That should be added back, IMHO.
Comment 9 Joshua Kinard gentoo-dev 2017-12-30 23:35:57 UTC
(In reply to Joshua Kinard from comment #8)
> 
> Also, when did the -v flag to crossdev get removed?  I like seeing the build
> output.  That should be added back, IMHO.

n/m, I see what the "new way" is.
Comment 10 Joshua Kinard gentoo-dev 2017-12-31 01:18:17 UTC
(In reply to Joshua Kinard from comment #8)
> (In reply to Sergei Trofimovich from comment #7)
> > How are things on today's toolchain? Looks like the following toolchain Just
> > Works:
> > 
> > - cross-mips64-unknown-linux-gnu/gcc-7.2.0
> > - cross-mips64-unknown-linux-gnu/glibc-2.26-r4
> > - cross-mips64-unknown-linux-gnu/binutils-2.29.1-r1
> 
> I'll retest and try disabling PIE to see what happens.

Looks like it works now!  I saw in the git log that crossdev itself has some changes related to handling PIE support now, so that is likely what fixed things.

Thanks!