Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 544756 - gcc-4.9.2 with uclibc-0.9.2.33-r14 on mips32r2 with -O1 and above fails to build any version of python
Summary: gcc-4.9.2 with uclibc-0.9.2.33-r14 on mips32r2 with -O1 and above fails to bu...
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Embedded Gentoo Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: uclibc-porting gcc-4.9
  Show dependency tree
 
Reported: 2015-03-28 12:02 UTC by Anthony Basile
Modified: 2019-12-30 17:45 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Build failure with python-3.4.3 (build.log,96.69 KB, text/plain)
2015-03-28 12:08 UTC, Anthony Basile
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Anthony Basile gentoo-dev 2015-03-28 12:02:00 UTC
When building python 2.7.9-r2 3.3.5-r1 and 3.4.3 with gcc-4.9.2 and uclibc-0.9.2.33-r14 on mips32r2, the build fails with "Illegal instruction" when CFLAGS has -O1, -O2 or -O3.  It succeeds with -O0.  It also succeeds at all levels of optimization before gcc-4.9 so this is a regression.  All other packages in @system succeed with 4.9 at -O2, so this bug is only triggered when building python.

I'll attach the full build.log below but here is the final build error:

LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.4.3/work/mips-gentoo-linux-uclibc ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
/bin/sh: line 5:  6899 Illegal instruction     LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.4.3/work/mips-gentoo-linux-uclibc ./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
Makefile:569: recipe for target 'pybuilddir.txt' failed

The problem is actually in libpython3.4.so.1.0 since I can get the freshly built ./python to run when linking against the host library by just removing the LD_LIBRARY_PATH.  (The host library being built with gcc-4.8.4.)

Sounds to me like any level of optimization in gcc-4.9 is causing a jump to a wrong address with garbage.  I'm narrowing it down with gdb and I'll try to produce reduced code for upstream.

I'm masking >=gcc-4.9 on profiles/linux/uclibc/mips while I sort this out.
Comment 1 Anthony Basile gentoo-dev 2015-03-28 12:08:03 UTC
Created attachment 399950 [details]
Build failure with python-3.4.3
Comment 2 Anthony Basile gentoo-dev 2015-03-29 12:33:54 UTC
Forgot to add the mips folks.  I should also say I hit hit this with isa=mips32r2 as stated above and mips I big endian.  I have yet to test little endian.  Also abi=o32 obviously since its the only 32-bit abi we support for mips.
Comment 3 Joshua Kinard gentoo-dev 2015-12-07 03:37:19 UTC
(In reply to Anthony Basile from comment #2)
> Forgot to add the mips folks.  I should also say I hit hit this with
> isa=mips32r2 as stated above and mips I big endian.  I have yet to test
> little endian.  Also abi=o32 obviously since its the only 32-bit abi we
> support for mips.

This might be fixed with gcc-5.x.  I found one of your old ~2012 mips3 uclibc stages and have successfully compiled gcc-5.3.0 and at least Python-2.7.10-r3 in it, and did not get any illegal instruction errors yet.  Just rebuilt uclibc itself (after chasing down why NPTL was disabled...), got sandbox updated, and now python-3.5.0-r1 is building.  Building with mips4/R1x000 compile flags, so I can't speak for mips32r2.  Hoping to use this chroot as a seed stage for new uclibc-based stages, which means new netboots (hopefully).

I am probably jinxing myself...
Comment 4 Anthony Basile gentoo-dev 2015-12-07 08:14:15 UTC
(In reply to Joshua Kinard from comment #3)
> (In reply to Anthony Basile from comment #2)
> > Forgot to add the mips folks.  I should also say I hit hit this with
> > isa=mips32r2 as stated above and mips I big endian.  I have yet to test
> > little endian.  Also abi=o32 obviously since its the only 32-bit abi we
> > support for mips.
> 
> This might be fixed with gcc-5.x.  I found one of your old ~2012 mips3
> uclibc stages and have successfully compiled gcc-5.3.0 and at least
> Python-2.7.10-r3 in it, and did not get any illegal instruction errors yet. 
> Just rebuilt uclibc itself (after chasing down why NPTL was disabled...),
> got sandbox updated, and now python-3.5.0-r1 is building.  Building with
> mips4/R1x000 compile flags, so I can't speak for mips32r2.  Hoping to use
> this chroot as a seed stage for new uclibc-based stages, which means new
> netboots (hopefully).
> 
> I am probably jinxing myself...

1) I'd like a copy of that updated mips3.  I lost it and never did get around to reconstructing it.

2) It might be ISA dependent.   You're running a mips3 cpu while I'm running a mips32r2.
Comment 5 Joshua Kinard gentoo-dev 2015-12-07 08:57:43 UTC
(In reply to Anthony Basile from comment #4)
> 
> 1) I'd like a copy of that updated mips3.  I lost it and never did get
> around to reconstructing it.

Google says it's on your university's server:
http://opensource.dyc.edu/pub/ZOLD/

Slow download speed, though.  Caps at ~128kbps.

If I can get updated stage runs out of catalyst, I'll make sure to get you a copy of those as well.


> 2) It might be ISA dependent.   You're running a mips3 cpu while I'm running
> a mips32r2.

Entirely possible.  I'd try w/ 5.3, though, as apparently a lot of fixes went into that release.  I still need to verify that PR66038 is fixed, which in 5.0 and 5.1, was preventing gcc from building under glibc N32.  Never tested 5.2, so I'm hoping 5.3 did fix whatever was causing it.

If the illegal instruction issue is still present, it might be a GCC bug and maybe filing upstream might yield results.
Comment 6 Anthony Basile gentoo-dev 2015-12-07 13:36:37 UTC
(In reply to Joshua Kinard from comment #5)
> (In reply to Anthony Basile from comment #4)
> > 
> > 1) I'd like a copy of that updated mips3.  I lost it and never did get
> > around to reconstructing it.
> 
> Google says it's on your university's server:
> http://opensource.dyc.edu/pub/ZOLD/
> 
> Slow download speed, though.  Caps at ~128kbps.
> 
> If I can get updated stage runs out of catalyst, I'll make sure to get you a
> copy of those as well.
> 
> 

OMG, i'm getting old and senile.  i put it there and forgot.
Comment 7 Joshua Kinard gentoo-dev 2015-12-28 03:42:06 UTC
I'm pretty confident now that whatever regression was causing this bug is fixed in gcc-5.3.0.  I've rebuilt/updated most of the packages in the 2012 chroot using gcc-5.3.0, including all three major python versions and haven't had any errors like this one.

I think gcc-4.9* should be masked on uclibc and at least 5.3 unmasked.  I haven't tested 5.2, but that one is probably safe as well.  Working around the maskings via catalyst is proving to be a bit....challenging.
Comment 8 Joshua Kinard gentoo-dev 2016-11-27 09:24:28 UTC
Successfully built gcc-6.2.0 in a chroot under binutils-2.27 and uclibc-ng-1.0.19.  Had to remove the toolchain.eclass hack, though (see Bug #600966).  I am pretty confident we can drop this mask and close this bug now to open up the newer gcc versions for wider testing.  gcc-6.2.x rebuilt all-but-one @system package in my chroot, the failing one being dev-util/pkgconfig, due to what looks like over-zealous -W flags (warning treated as an error).  I can also state that Python 3.4.x definitely builds, having randomly glanced at the SSH terminal when 3.4.5 was getting unpacked and built.
Comment 9 Joshua Kinard gentoo-dev 2017-07-12 13:12:09 UTC
I have newer stages built with gcc-6.3.0, uclibc-ng-1.0.24 (no objstack), and Python-3.4, 3.5, and 2.7 all compiled fine.  Not sure about old uclibc, but I'm starting to think we need to deprecate it sooner rather than later and move onto uclibc-ng by releasing the masks and that toolchain.eclass block.  I'm going to start testing an upgrade to gcc-7.1.x and uclibc-ng-1.0.25 next.
Comment 10 Anthony Basile gentoo-dev 2017-07-12 18:20:07 UTC
(In reply to Joshua Kinard from comment #9)
> I have newer stages built with gcc-6.3.0, uclibc-ng-1.0.24 (no objstack),
> and Python-3.4, 3.5, and 2.7 all compiled fine.  Not sure about old uclibc,
> but I'm starting to think we need to deprecate it sooner rather than later
> and move onto uclibc-ng by releasing the masks and that toolchain.eclass
> block.  I'm going to start testing an upgrade to gcc-7.1.x and
> uclibc-ng-1.0.25 next.

i'm worried about my mips32r2 hardware which is very slow.  its what i use to make the stage3's.  i'm in the middle of a batch right now, afterwards, i can try bumping to gcc-6, but i'm afraid of how long it might take.  do you have any idea what the build time is for gcc-6 compared to gcc-4.8.5?
Comment 11 Joshua Kinard gentoo-dev 2017-07-13 07:02:28 UTC
(In reply to Anthony Basile from comment #10)
> (In reply to Joshua Kinard from comment #9)
> > I have newer stages built with gcc-6.3.0, uclibc-ng-1.0.24 (no objstack),
> > and Python-3.4, 3.5, and 2.7 all compiled fine.  Not sure about old uclibc,
> > but I'm starting to think we need to deprecate it sooner rather than later
> > and move onto uclibc-ng by releasing the masks and that toolchain.eclass
> > block.  I'm going to start testing an upgrade to gcc-7.1.x and
> > uclibc-ng-1.0.25 next.
> 
> i'm worried about my mips32r2 hardware which is very slow.  its what i use
> to make the stage3's.  i'm in the middle of a batch right now, afterwards, i
> can try bumping to gcc-6, but i'm afraid of how long it might take.  do you
> have any idea what the build time is for gcc-6 compared to gcc-4.8.5?

If I recall correctly, the speed issues w/ your mips32r2 hardware was lack of L2 cache, right?  Wasn't your system at least a ~600MHz processor?  And what was the endianness of that platform?  mipsel?

I use my SGI Octane for builds, which has dual R14K CPUs at 600MHz and 2MB L2 cache.  But gcc is still a monster to build.  Per genlop, this is my build history for gcc since my reinstall to switch to the n32 ABI userland (using MAKEOPTS of -j3):

 * sys-devel/gcc

     Sun Aug 16 10:11:58 2015 >>> sys-devel/gcc-4.9.3
       merge time: 5 hours, 39 minutes and 41 seconds.

     Wed Feb 24 07:33:35 2016 >>> sys-devel/gcc-5.3.0
       merge time: 1 minute and 43 seconds.

     Thu Jun 16 16:05:48 2016 >>> sys-devel/gcc-5.4.0
       merge time: 7 hours, 24 minutes and 14 seconds.

     Wed Oct 12 12:29:08 2016 >>> sys-devel/gcc-6.2.0-r1
       merge time: 8 hours, 17 minutes and 34 seconds.

     Wed Nov 16 17:13:12 2016 >>> sys-devel/gcc-6.2.0-r1
       merge time: 8 hours, 17 minutes and 16 seconds.

     Fri Jan  6 12:31:12 2017 >>> sys-devel/gcc-6.3.0
       merge time: 8 hours, 22 minutes and 21 seconds.

The Feb 2016 entry is a fluke where I re-installed from binpkg.  gcc-7 doesn't seem to change the build times too much.  uclibc-ng builds are *much* faster, probably on the order of ~2-3 hours less than glibc-based userlands.  The smaller library footprint of uclibc-ng really makes itself known during large builds like w/ gcc.

I'm updating my new uclibc_mips2 stage3 chroot right now with gcc-7.1.x.  You need uclibc-ng-1.0.25 to do so, though, as it contains a build fix for gcc-7.1.x's final build of libstdc++.  If that works, I'll update the remaining chroots before trying catalyst runs all over again.
Comment 12 Sergei Trofimovich gentoo-dev 2019-12-30 17:45:52 UTC
blueness@ asked to add toolchain@ back in hopes to track bug down to gcc problem.

To make it more actionable on toolchain side ideally I would need a few things:
1. emerge --info output (to get the idea which CPU has the problem)
2. gdb backtrace and a disassembly of python crash to get the idea whan instruction and and where it crashed
3. extracted self-contained source from python and a gcc command that generates bad instruction. This will ease gcc bisection down to a fix if it happened upstream.