Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 828400 - =sys-devel/gcc-11.2.1_p20211127[lto] fails on any ARCH != amd64: ICE (lto1: internal compiler error: in read_cgraph_and_symbols, at lto/lto-common.c:2739)
Summary: =sys-devel/gcc-11.2.1_p20211127[lto] fails on any ARCH != amd64: ICE (lto1: i...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: gcc-11 822036
  Show dependency tree
 
Reported: 2021-12-05 04:22 UTC by matoro
Modified: 2022-01-18 13:19 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build logs (gcc-build-logs.tar.bz2,102.69 KB, application/x-bzip2)
2021-12-05 04:23 UTC, matoro
Details
build.log.gz (build.log.gz,174.27 KB, application/gzip)
2021-12-05 04:37 UTC, matoro
Details
build logs from arm64 (gcc-build-logs.tar.bz2,104.53 KB, application/x-bzip2)
2021-12-05 04:39 UTC, matoro
Details
build.log.gz from arm64 (build.log.gz,168.30 KB, application/gzip)
2021-12-05 04:39 UTC, matoro
Details
0001-11.3.0-fix-CET-patch.patch (file_828400.txt,4.48 KB, patch)
2021-12-28 03:48 UTC, Sam James
Details | Diff
26_all_enable-cet.patch (fixed) (file_828400.txt,1.90 KB, patch)
2021-12-28 03:48 UTC, Sam James
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description matoro archtester 2021-12-05 04:22:41 UTC
When building =sys-devel/gcc-11.2.1_p20211127 with USE="lto" on my non-amd64 hosts, the stage1 build succeeds, but it generates a stage2 xgcc that always crashes with the ICE:

lto1: internal compiler error: in read_cgraph_and_symbols, at lto/lto-common.c:2739

The generated xgcc cannot compile anything - there is no reproducer input because all source inputs trigger the ICE.

This occurs on my ppc64 host and my arm64 host.  It does *not* occur on two amd64 hosts I've tested (one Xeon, one Ryzen).

It does NOT occur if USE="-lto" is set.

It does NOT occur on ~sys-devel/gcc-11.2.0 on any arch.  Attached is a sample set of build logs from the ppc64 host, using MAKEOPTS="-j1".  Unfortunately I don't have other architectures I can test.

The only possible reference I could find is https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99448, but that occurs while compiling an application, not during the gcc bootstrap.  In the additional info section, I'll include emerge --info from the ppc64 host, but can include the arm64 one as well if needed.

Reproducible: Always




Portage 3.0.28 (python 3.9.9-final-0, default/linux/ppc64le/17.0, gcc-11.2.0, glibc-2.34-r3, 5.15.5-gentoo-dist ppc64le)
=================================================================
System uname: Linux-5.15.5-gentoo-dist-ppc64le-POWER9,_altivec_supported-with-glibc2.34
KiB Mem:    32978880 total,  12206272 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sat, 04 Dec 2021 23:29:52 +0000
Head commit of repository gentoo: 53883750a317f0990e1419ae77e64a2475b7ca51

Timestamp of repository chaoslab: Fri, 01 May 2020 10:40:59 +0000
sh bash 5.1_p12
ld GNU ld (Gentoo 2.37_p1 p1) 2.37
app-shells/bash:          5.1_p12::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.34.0-r5::gentoo
dev-lang/python:          3.9.9::gentoo, 3.10.0_p1::gentoo
dev-util/cmake:           3.22.0::gentoo
sys-apps/baselayout:      2.8::gentoo
sys-apps/openrc:          0.44.9::gentoo
sys-apps/sandbox:         2.29::gentoo
sys-devel/autoconf:       2.71-r1::gentoo
sys-devel/automake:       1.16.5::gentoo
sys-devel/binutils:       2.37_p1-r1::gentoo
sys-devel/gcc:            11.2.0::gentoo
sys-devel/gcc-config:     2.5-r1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.15-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.34-r3::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo
    sync-user: portage:portage
    priority: -1000
    sync-git-verify-commit-signature: yes

matoro
    location: /var/db/repos/overlay
    masters: gentoo

chaoslab
    location: /var/lib/layman/chaoslab
    sync-type: laymansync
    sync-uri: https://github.com/LarsImNetz/chaoslab.git
    masters: gentoo
    priority: 50

lto-overlay
    location: /var/lib/layman/lto-overlay
    sync-type: laymansync
    sync-uri: https://github.com/InBetweenNames/gentooLTO.git
    masters: gentoo mv
    priority: 50

mv
    location: /var/lib/layman/mv
    sync-type: laymansync
    sync-uri: https://anongit.gentoo.org/git/user/mv.git
    masters: gentoo
    priority: 50

nest
    location: /var/lib/layman/nest
    sync-type: laymansync
    sync-uri: https://github.com/SpiderX/portage-overlay.git
    masters: gentoo
    priority: 50

nymphos
    location: /var/lib/layman/nymphos
    sync-type: laymansync
    sync-uri: https://github.com/neeshy/nymphos
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="ppc64 ~ppc64"
ACCEPT_LICENSE="@FREE"
CBUILD="powerpc64le-unknown-linux-gnu"
CFLAGS="-O3 -mcpu=native -mtune=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=72 -fuse-linker-plugin -pipe"
CHOST="powerpc64le-unknown-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.8/conf /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.0/ext-active/ /etc/php/cgi-php8.0/ext-active/ /etc/php/cli-php8.0/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O3 -mcpu=native -mtune=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=72 -fuse-linker-plugin -pipe"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--usepkg --autounmask=n --complete-graph --keep-going --with-bdeps=y"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="https://mirror.leaseweb.com/gentoo/ https://mirror.rackspace.com/gentoo/ https://mirrors.rit.edu/gentoo/ https://gentoo.osuosl.org/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j72"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl bash-completion bzip2 caps cli crypt dist-kernel dri elogind fortran gdbm gentoo-vm graphite headless-awt iconv ipv6 libbsd libglvnd lm-sensors lto ncurses nls nptl openmp pam pcre pgo ppc64 readline seccomp split-usr ssl threads udev unicode verify-sig vhosts vim-syntax xattr zlib" ABI_PPC="64" ADA_TARGET="gnat_2020" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_PPC="altivec vsx vsx2 vsx3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS
Comment 1 matoro archtester 2021-12-05 04:23:30 UTC
Created attachment 757445 [details]
build logs
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-05 04:23:57 UTC
(Note that it does build fine with PGO + LTO on arm* for me).
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-05 04:32:51 UTC
I'd also be interested in a few other things:
- emerge --info gcc
- the build.log of when it fails (I can't _see_ it in the tarball, but it's late, and I can see the eclass tries to include it, so if it's in there, forgive me!)
- whether it happens with more vanilla CFLAGS (although most problematic things should be stripped out already)
- ideally a log from another machine which crashes (e.g. maybe an arm64 one to compare with the ppc64 one).

I've been running with this version of GCC PGO + LTO'd without any issues on arm and arm64 for quite some time, but trying it on sparc now.
Comment 4 matoro archtester 2021-12-05 04:37:49 UTC
Created attachment 757446 [details]
build.log.gz
Comment 5 matoro archtester 2021-12-05 04:39:13 UTC
Created attachment 757447 [details]
build logs from arm64
Comment 6 matoro archtester 2021-12-05 04:39:28 UTC
Created attachment 757448 [details]
build.log.gz from arm64
Comment 7 matoro archtester 2021-12-05 04:42:18 UTC
Sorry about that - here's the build.log and also the same from the arm64 box.  Yes those are from https://github.com/InBetweenNames/gentooLTO.git but they should be stripped out because I am not using USE=custom-cflags.  Just to test I tried disabling them, using this minimal set:

CFLAGS="-O3 -mcpu=native -mtune=native -pipe"

but got the same result.

I have also tried enabling/disabling PGO to no effect - it works fine if LTO is disabled, does not work if LTO is enabled.
Comment 8 matoro archtester 2021-12-05 04:43:12 UTC
=================================================================
                        Package Settings
=================================================================

sys-devel/gcc-11.2.0::gentoo was built with the following:
USE="(cxx) graphite lto nls nptl openmp pch pgo (pie) sanitize ssp (-ada) -custom-cflags -d -debug -doc (-fixed-point) -fortran -go (-hardened) -jit (-libssp) (-multilib) -objc -objc++ -objc-gc -systemtap -test -valgrind -vanilla (-vtv) -zstd"
CFLAGS="-mcpu=native -mtune=native -pipe -Wl,-O1 -Wl,--as-needed -O2"
CXXFLAGS="-mcpu=native -mtune=native -pipe -Wl,-O1 -Wl,--as-needed -O2"
FEATURES="pid-sandbox sandbox assume-digests parallel-fetch splitdebug preserve-libs binpkg-docompress unmerge-orphans distlocks network-sandbox multilib-strict buildpkg usersandbox ipc-sandbox parallel-install binpkg-logs config-protect-if-modified userpriv strict xattr compressdebug ebuild-locks news userfetch merge-sync unmerge-logs protect-owned unknown-features-warn qa-unresolved-soname-deps usersync compress-build-logs fixlafiles binpkg-dostrip sfperms"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -mcpu=native -mtune=native -pipe"
Comment 9 matoro archtester 2021-12-22 18:07:06 UTC
Note, I have just received my new sparc host and have confirmed that this bug also affects sparc (running under 64-bit userland), so that's at least 3 arches that cannot build this gcc patchset with LTO.
Comment 10 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-23 00:06:16 UTC
(In reply to matoro from comment #9)
> Note, I have just received my new sparc host and have confirmed that this
> bug also affects sparc (running under 64-bit userland), so that's at least 3
> arches that cannot build this gcc patchset with LTO.

It definitely can for me, but of course, I don't deny it's happening for you.

Will note that your last *FLAGS don't have -O2.

Can you report this upstream? I'm struggling to get any debugging information out of this, sadly.
Comment 11 matoro archtester 2021-12-23 01:36:39 UTC
(In reply to Sam James from comment #10)
> (In reply to matoro from comment #9)
> > Note, I have just received my new sparc host and have confirmed that this
> > bug also affects sparc (running under 64-bit userland), so that's at least 3
> > arches that cannot build this gcc patchset with LTO.
> 
> It definitely can for me, but of course, I don't deny it's happening for you.
> 
> Will note that your last *FLAGS don't have -O2.
> 
> Can you report this upstream? I'm struggling to get any debugging
> information out of this, sadly.

So, out of respect for upstream I figured I should submit details with USE=vanilla, but when I tested just now it surprisingly worked with this flag set (tested on ppc64, with LTO on).  If I'm understanding what this means, doesn't that basically confirm that a Gentoo-specific patch is the cause?  Given that it also does not appear on ~sys-devel/gcc-11.2.0, what are the new patches added that I can attempt to narrow it down to?
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-23 01:47:26 UTC
(In reply to matoro from comment #11)
> (In reply to Sam James from comment #10)
> > (In reply to matoro from comment #9)
> > > Note, I have just received my new sparc host and have confirmed that this
> > > bug also affects sparc (running under 64-bit userland), so that's at least 3
> > > arches that cannot build this gcc patchset with LTO.
> > 
> > It definitely can for me, but of course, I don't deny it's happening for you.
> > 
> > Will note that your last *FLAGS don't have -O2.
> > 
> > Can you report this upstream? I'm struggling to get any debugging
> > information out of this, sadly.
> 
> So, out of respect for upstream I figured I should submit details with
> USE=vanilla, but when I tested just now it surprisingly worked with this
> flag set (tested on ppc64, with LTO on).  If I'm understanding what this
> means, doesn't that basically confirm that a Gentoo-specific patch is the
> cause?  Given that it also does not appear on ~sys-devel/gcc-11.2.0, what
> are the new patches added that I can attempt to narrow it down to?

Here's the list of patches: https://gitweb.gentoo.org/proj/gcc-patches.git/tree/11.3.0/gentoo.

We generally don't patch GCC (or glibc) in any interesting way to avoid problems like this. That's why I didn't even think to suggest it!

The only new patches were CET related which shouldn't affect non-amd64.

We can compare the dir with https://gitweb.gentoo.org/proj/gcc-patches.git/tree/11.2.0/gentoo. But this sounds really odd!
Comment 13 matoro archtester 2021-12-23 05:32:11 UTC
(In reply to Sam James from comment #12)
> (In reply to matoro from comment #11)
> > (In reply to Sam James from comment #10)
> > > (In reply to matoro from comment #9)
> > > > Note, I have just received my new sparc host and have confirmed that this
> > > > bug also affects sparc (running under 64-bit userland), so that's at least 3
> > > > arches that cannot build this gcc patchset with LTO.
> > > 
> > > It definitely can for me, but of course, I don't deny it's happening for you.
> > > 
> > > Will note that your last *FLAGS don't have -O2.
> > > 
> > > Can you report this upstream? I'm struggling to get any debugging
> > > information out of this, sadly.
> > 
> > So, out of respect for upstream I figured I should submit details with
> > USE=vanilla, but when I tested just now it surprisingly worked with this
> > flag set (tested on ppc64, with LTO on).  If I'm understanding what this
> > means, doesn't that basically confirm that a Gentoo-specific patch is the
> > cause?  Given that it also does not appear on ~sys-devel/gcc-11.2.0, what
> > are the new patches added that I can attempt to narrow it down to?
> 
> Here's the list of patches:
> https://gitweb.gentoo.org/proj/gcc-patches.git/tree/11.3.0/gentoo.
> 
> We generally don't patch GCC (or glibc) in any interesting way to avoid
> problems like this. That's why I didn't even think to suggest it!
> 
> The only new patches were CET related which shouldn't affect non-amd64.
> 
> We can compare the dir with
> https://gitweb.gentoo.org/proj/gcc-patches.git/tree/11.2.0/gentoo. But this
> sounds really odd!

So what I've done is created a reverse patch using interdiff for each of the patches in question and dumped them in my user-patches folder.  Using that, I got a successful compile.  Now, I will remove each reverse-patch one at a time after each build until I can identify which one is causing the problem.  Will update with findings...
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-23 05:42:36 UTC
(In reply to matoro from comment #13)
> So what I've done is created a reverse patch using interdiff for each of the
> patches in question and dumped them in my user-patches folder.  Using that,
> I got a successful compile.  Now, I will remove each reverse-patch one at a
> time after each build until I can identify which one is causing the problem.
> Will update with findings...

Thank you for your work here so far & sorry I've not been able to be that helpful so far.

If you have any questions which need to be answered a bit quicker which don't fit in the bug, feel free to come to #gentoo-toolchain on libera.chat (IRC).
Comment 15 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-23 05:59:21 UTC
(In reply to Sam James from comment #14)
> (In reply to matoro from comment #13)
> > So what I've done is created a reverse patch using interdiff for each of the
> > patches in question and dumped them in my user-patches folder.  Using that,
> > I got a successful compile.  Now, I will remove each reverse-patch one at a
> > time after each build until I can identify which one is causing the problem.
> > Will update with findings...
> 
> Thank you for your work here so far & sorry I've not been able to be that
> helpful so far.
> 
> If you have any questions which need to be answered a bit quicker which
> don't fit in the bug, feel free to come to #gentoo-toolchain on libera.chat
> (IRC).

Something else I'm interested in (because none of the patches are that interesting other than say, the CET one, which I'd really hope isn't related here (and shouldn't be)): 

this is happening for you on every one of your machines, and you've just setup a new one. Can you walk me through _anything_ interesting about this?

The fact it's on *every one* of your !amd64 boxes is suspicious given I've not managed to hit it. My gut is that there's some additional environmental factor, possibly something you do naturally so you've forgotten.

(I'd note again that not all of these flags are safe globally and are generally likely to trigger compiler bugs at least on random arches).

Was binutils and the rest of it also built with vanilla *FLAGS when testing?

Does it happen if you fire up a chroot on one of these machines from a vanilla stage3? How much can you vary a chroot before it starts to fail like the host does?
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-23 06:00:49 UTC
(In reply to Sam James from comment #15)
> 
> Was binutils and the rest of it also built with vanilla *FLAGS when testing?
> 
(also, please verify at least @system, including stuff like gmp, mpfr, ... is built with normal flags too when testing)
Comment 17 Georgy Yakovlev archtester gentoo-dev 2021-12-23 08:44:58 UTC
I've reproduced on ppc64le system


^[[01m^[[Klto1:^[[m^[[K ^[[01;31m^[[Kinternal compiler error: ^[[m^[[Kin read_cgraph_and_symbols, at lto/lto-common.c:2739
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.gentoo.org/> for instructions.
lto-wrapper: fatal error: /var/tmp/portage/sys-devel/gcc-11.2.1_p20211127/work/build/./prev-gcc/xgcc returned 1 exit status
compilation terminated.
/usr/powerpc64le-unknown-linux-gnu/bin/ld: error: lto-wrapper failed
Comment 18 Georgy Yakovlev archtester gentoo-dev 2021-12-23 08:51:58 UTC
ok it's not LTO to blame, it's graphite/isl

without USE=graphite xgcc works for me.

so probably another weird graphite ricerbug =)
Comment 19 Georgy Yakovlev archtester gentoo-dev 2021-12-23 08:57:13 UTC
actually NVM, build still failed later.
not graphite.
Comment 20 Georgy Yakovlev archtester gentoo-dev 2021-12-23 09:12:54 UTC
ok I pinpointed it down to

-flto=jobserver

trying jobserver is what triggers an ice in read_cgraph_and_symbols

on this line

      gcc_assert (num_objects == nfiles);
Comment 21 Georgy Yakovlev archtester gentoo-dev 2021-12-23 11:08:53 UTC
will dump it here for history


<trofi> lto/lto-common.c:2739 is an assert in mismatch on number of objects seen in .res file and amount of objects collected by linker in it's input arguments. you probably want to compare actual file list in .res file and compare it to '**fnames' of read_cgraph_and_symbols() to find the discrepancy.
<+gyakovlev> trofi: hey! thanks
<+gyakovlev> yeah I slowly crawling thru this.
<+gyakovlev> but my gdb crashes with SIGABRT itself, does not help at all.
<+gyakovlev> I collected all temps output here, just in case:
<+gyakovlev> https://dpaste.com/BLMSMAM56
<trofi> my guess is that number of collected files is collected and written by plugin at https://github.com/gcc-mirror/gcc/blob/master/lto-plugin/lto-plugin.c#L557 (which read then fails to reconcile with plugin's state)
<+gyakovlev> interesting thing is that -flto=<int> does not trigger it, just -flto=jobserver
<+gyakovlev> so I'm guessing it's related to parallel/auto_parallel handling
<trofi> empty '' parameter looks slightly suspicious
<+gyakovlev> seeing that 
<+gyakovlev>       else if (parallel > 1)
<+gyakovlev>         {
<+gyakovlev>           char buf[256];
<+gyakovlev>           sprintf (buf, "-fwpa=%i", parallel);
<+gyakovlev> set -fwpa
<+gyakovlev> and it's 32 for me ( number of cores)
<+gyakovlev> it takes that codepath, I guess.
<+gyakovlev> hmm. it started failing with -flto=1 too
<+gyakovlev> I need to get some sleep and jump at it with working gdb and fresh brain.
<+gyakovlev> ok I compared args to working gcc and borked one.
<+gyakovlev> difference is on borked gcc
<+gyakovlev> instead of
<+gyakovlev> -fcf-protection=none
<+gyakovlev> I have ""
<+gyakovlev> here's normal flags file
<+gyakovlev> ==> tempswpa.args.0 <==
<+gyakovlev> tempst.o
<+gyakovlev> and here's borked
<+gyakovlev> ==> tempswpa.args.0 <==
<+gyakovlev> ""
<+gyakovlev> tempst.o
<+gyakovlev> so I guess gcc interprets that "" that came out of nowhere as a file and number of files does not mach actual objects
<+gyakovlev> manually adding '-fcf-protection=none' to args solves the ICE...
<+gyakovlev> now need to figure out why it's empty on borked gcc =)
<+gyakovlev> sam_: we have  winner!
<+gyakovlev> 26_all_enable-cet.patch messes up with fcf-protection
<+gyakovlev> so the args gets emptied somehow
<+gyakovlev> this leads to this lto weirdness
<+gyakovlev> note it breaks even if USE=cet is not in use
Comment 22 matoro archtester 2021-12-23 14:50:33 UTC
My testing has also confirmed that 26_all_enable-cet.patch is the cause.  I suppose that also explains why it appears only on non-amd64.  For now I have deployed the reverse patch to my hosts.  Will this patch just be reverted entirely?
Comment 23 Georgy Yakovlev archtester gentoo-dev 2021-12-23 20:29:24 UTC
yeah it will be fixed or reverted ofc, package will be bumped when it happens.
working on finding best way to do it.
Comment 24 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-28 03:48:26 UTC
Created attachment 760602 [details, diff]
0001-11.3.0-fix-CET-patch.patch

Attached a patch-to-the-patch.
Comment 25 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-28 03:48:51 UTC
Created attachment 760603 [details, diff]
26_all_enable-cet.patch (fixed)

matoro, could you test this for me please? Revert the old patch first / drop it.
Comment 26 matoro archtester 2021-12-28 05:54:33 UTC
(In reply to Sam James from comment #25)
> Created attachment 760603 [details, diff] [details, diff]
> 26_all_enable-cet.patch (fixed)
> 
> matoro, could you test this for me please? Revert the old patch first / drop
> it.

Confirmed working on ppc64, thanks for the effort!
Comment 27 Larry the Git Cow gentoo-dev 2021-12-28 05:57:47 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=2b36f3ad2ba0114eae1d32bae5e395e098b3714b

commit 2b36f3ad2ba0114eae1d32bae5e395e098b3714b
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-12-28 03:44:47 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-12-28 03:55:44 +0000

    11.3.0: fix CET patch
    
    Our patch was causing unhandled state to leak into the LTO metadata writer,
    it shouldn't have got that far though. Instead of messing about with GCC's
    option handling, use the macro they provide for purposes like this, which
    makes things far simpler (and less fragile).
    
    Bug: https://bugs.gentoo.org/828400
    Bug: https://bugs.gentoo.org/822036
    Thanks-to: Sergei Trofimovich <slyich@gmail.com> (debugging help in #gentoo-toolchain)
    Thanks-to: Georgy Yakovlev <gyakovlev@gentoo.org> (debugging)
    Reported-by: matoro <matoro@airmail.cc>
    Signed-off-by: Sam James <sam@gentoo.org>

 11.3.0/gentoo/26_all_enable-cet.patch | 65 +++++------------------------------
 1 file changed, 9 insertions(+), 56 deletions(-)
Comment 28 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-12-28 05:58:38 UTC
(In reply to matoro from comment #26)
> (In reply to Sam James from comment #25)
> > Created attachment 760603 [details, diff] [details, diff] [details, diff]
> > 26_all_enable-cet.patch (fixed)
> > 
> > matoro, could you test this for me please? Revert the old patch first / drop
> > it.
> 
> Confirmed working on ppc64, thanks for the effort!

Thank you!

I'm going to see if soap wants to take a newer snapshot (there's not that many changes) or not before deciding if we should just revbump the current version.
Comment 29 Larry the Git Cow gentoo-dev 2022-01-18 13:19:54 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b96fd11e3e5626181c32c38381f814aba21fb9f0

commit b96fd11e3e5626181c32c38381f814aba21fb9f0
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2022-01-18 13:16:42 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2022-01-18 13:19:30 +0000

    sys-devel/gcc: add 11.2.1_p20220115
    
    Fairly minor changes upstream since the last snapshot of the 11
    stable branch. Includes more CET fixes and the upstream cross-compile
    patch.
    
    Also, the PCH ICE fix, although we've since masked PCH globally
    due to its instability.
    
    Bug: https://bugs.gentoo.org/822036
    Closes: https://bugs.gentoo.org/803371
    Closes: https://bugs.gentoo.org/828400
    Closes: https://bugs.gentoo.org/822690
    Signed-off-by: Sam James <sam@gentoo.org>

 profiles/base/package.use.mask            |  2 +-
 sys-devel/gcc/Manifest                    |  3 +++
 sys-devel/gcc/gcc-11.2.1_p20220115.ebuild | 26 ++++++++++++++++++++++++++
 3 files changed, 30 insertions(+), 1 deletion(-)