Bug 622056 – sys-devel/gcc-5.4.0 random Internal Compiler Errors in various packages at various points

• Product: Gentoo Linux
• Status: RESOLVED OBSOLETE

[Description toraboracaveman 2017-06-17 23:21:29 UTC]

Recently, I made the following changes:
 1. Upgraded the GCC C++ ABI.
 2. Switched my profile from AMD64 into AMD64/Systemd.

Then, as I compiled certain packages, I noticed some weird errors, which folks in the Freenode IRC channel, #gentoo, pointed me out to read the associated ABI breakage news item.

Then, I executed the following command:

 revdep-rebuild --library 'libstdc++.so.6' -- --exclude gcc

After which the following happened:
 1. As GCC was compiling packages, in some cases it used to freeze at seemingly random points with random applications. I recall this happening with ncurses, llvm, and qtwebengine. I call this 'freezing' because of the fact that emerge was stuck while seemingly executing some GCC command, but with a CPU utilization of 0% as I found by the command 'htop'.
 2. Similarly, GCC generates internal compiler errors when compiling large packages, such as qtwebengine. Similar to the previous case, such internal compiler errors appear at random points.
 3. By repeating again, I managed to get llvm and ncurses compiled. But I never managed to get qtwebengine. My theory is that qtwebengine is so large that the probability of having GCC crash randomly is about 1. But this is only a feeling, as I never managed to emerge qtwebengine.

Worth noting that:
 1. memtest86+ for 2.5 passes found no memory errors.
 2. prime95/mprime (from package GIMPS) found no CPU errors for running about 10 or 15 minutes, with a CPU temperature of 100+ Celsius degrees.

I'm going to attach 4 build logs for qtwebengine, that demonstrate the internal compiler errors, along with other information.



Reproducible: Always

Steps to Reproduce:
If I wish to reproducce this, I can do it simply by:
1. emerge -1 qtwebengine

That suffices for me. If you wish to know what I did to reach this state, the description of what I did is above.
Actual Results:  
GCC internal compiler error when installing qtwebengine, and other packages, including random freezing.

Expected Results:  
Have emerge -1 qtwebengine installed successfully.

Portage 2.3.5 (python 3.4.5-final-0, default/linux/amd64/13.0/systemd, gcc-5.4.0, glibc-2.23-r3, 4.9.16-gentoo x86_64)
=================================================================
System uname: Linux-4.9.16-gentoo-x86_64-Intel-R-_Core-TM-_i7-5600U_CPU_@_2.60GHz-with-gentoo-2.3
KiB Mem:     3838052 total,    230720 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sat, 17 Jun 2017 04:00:01 +0000
sh bash 4.3_p48-r1
ld GNU ld (Gentoo 2.26.1 p1.0) 2.26.1
app-shells/bash:          4.3_p48-r1::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.24.1-r2::gentoo
dev-lang/python:          2.7.12::gentoo, 3.4.5::gentoo
dev-util/cmake:           3.7.2::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/openrc:          0.26.3::gentoo
sys-apps/sandbox:         2.10-r3::gentoo
sys-devel/autoconf:       2.69::gentoo
sys-devel/automake:       1.11.6-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.26.1::gentoo, 2.28-r2::gentoo
sys-devel/gcc:            5.4.0-r3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.23-r3::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://91.121.188.201/gentoo-distfiles/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X aac acl acpi alsa amd64 berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 jpeg lm_sensors modules mp3 mp4 multilib ncurses nls nptl ogg opengl openmp pam pcre pdf png postscript pulseaudio readline seccomp session ssl svg systemd tcpd threads tiff udev unicode wav xattr xft xmp zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" L10N="en en-US ar ar-AE" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US ar ar_AE" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby21 ruby22" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

[Comment 1 toraboracaveman 2017-06-17 23:25:12 UTC]

Created attachment 476770 [details]
Build log 1

This is one of the oldest build logs representing one of my oldest attempts to emerge qtwebengine.

[Comment 2 toraboracaveman 2017-06-17 23:27:44 UTC]

Created attachment 476772 [details]
Build log 2

Same as build log 1, except for being yet another attempt.

[Comment 3 toraboracaveman 2017-06-17 23:28:56 UTC]

Created attachment 476774 [details]
Build log 3

Similar to the previous ones, except for me lowering the CPU frequency, but I *think* concurrent jobs (MAKEOPTS="-jX") was not 1. Not sure though.

[Comment 4 toraboracaveman 2017-06-17 23:32:48 UTC]

Created attachment 476776 [details]
Build log 4

Same, except:
 1. MAKEOPTS="-j1" -- I'm sure.
 2. I also configured the intel_pstate driver (by cpupower) such that the maximum CPU power is around 1GHz. I also confirmed this via lscpu, and cpupower commands. Additionally.
 3. I confirm that CPU temp. was about 58 Celsius degrees.
 4. I confirm that dmesg and journalctl did not show any CPU throttling due to raised temperature.

In my view, this build log represents my sanest shot to get this done, but I failed nonetheless.

[Comment 5 toraboracaveman 2017-06-17 23:40:44 UTC]

Created attachment 476778 [details]
CPU Information

This file is essentially my /proc/cpuinfo

[Comment 6 toraboracaveman 2017-06-17 23:41:20 UTC]

Created attachment 476780 [details]
Build environment

This file contains the build environment when I emerge qtwebengine.

[Comment 7 Matthias Maier 2017-06-18 00:26:57 UTC]

> Similar to the previous case, such internal compiler errors appear at random points.
> 3. By repeating again, I managed to get llvm and ncurses compiled. But I never managed to get qtwebengine. My theory is that qtwebengine is so large that the probability of having GCC crash randomly is about 1. But this is only a feeling, as I never managed to emerge qtwebengine.


gcc randomly segfaulting pretty much sounds like a memory issue on your side.

[Comment 8 toraboracaveman 2017-06-18 01:02:46 UTC]

Do you still think it is a memory issue? Could you please be more specific about what kind of an issue? Exceeding memory capacity? Or faulty memory?

As for exceeding, I see no oom-killer in dmesg or journalctl. As for faulty memory card, memtest86+ reports no memory errors for 2.5 passes.

[Comment 9 Thomas Beutin 2017-10-05 13:34:50 UTC]

(In reply to Matthias Maier from comment #7)
> > Similar to the previous case, such internal compiler errors appear at random points.
> > 3. By repeating again, I managed to get llvm and ncurses compiled. But I never managed to get qtwebengine. My theory is that qtwebengine is so large that the probability of having GCC crash randomly is about 1. But this is only a feeling, as I never managed to emerge qtwebengine.
> 
> 
> gcc randomly segfaulting pretty much sounds like a memory issue on your side.

yes, it seems to this is a "dupe" of bug 20600 (just my two cents)

[Comment 10 Sergei Trofimovich (RETIRED) 2019-01-27 22:35:46 UTC]

Does it still happen on modern gcc?