Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 730406 - sys-kernel/genkernel-4.0.9-r1: Failed to create binpkg of boost-1.73.0: *** stack smashing detected ***: terminated
Summary: sys-kernel/genkernel-4.0.9-r1: Failed to create binpkg of boost-1.73.0: *** s...
Status: RESOLVED DUPLICATE of bug 724314
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Gentoo Genkernel Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-01 19:45 UTC by Stig Nielsen
Modified: 2020-12-29 10:14 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
genkernel.log (genkernel.tar.gz,184.51 KB, application/gzip)
2020-07-01 19:47 UTC, Stig Nielsen
Details
output from "emerge --info" (emerge-info.log,17.63 KB, text/plain)
2020-07-21 15:27 UTC, Jack
Details
genkernel log (genkernel.log,534.50 KB, text/x-log)
2020-07-21 17:06 UTC, Jack
Details
output of gcc (crash.1.log,4.02 KB, text/plain)
2020-07-22 20:37 UTC, Jack
Details
crash with backtrace (crashbt.1.log,17.51 KB, text/plain)
2020-07-22 21:57 UTC, Jack
Details
gdb output (crash2.log,9.54 KB, text/plain)
2020-07-23 20:18 UTC, Jack
Details
gdb output (gdb.out,278.60 KB, text/plain)
2020-07-24 22:12 UTC, Jack
Details
output of gcc and of gdb (gdb2.log,14.02 KB, text/plain)
2020-07-26 18:21 UTC, Jack
Details
gcc-10.2.0-backport-PR95820-ICE-fundecl.patch (gcc-10.2.0-backport-PR95820-ICE-fundecl.patch,657 bytes, patch)
2020-07-26 20:20 UTC, Sergei Trofimovich (RETIRED)
Details | Diff
Minimize self-contained source using creduce (gcc_p_statistics.txt,12.04 KB, text/plain)
2020-07-30 08:07 UTC, Stig Nielsen
Details
Reproducer (instantiate_cpp_grammar.ii,84.61 KB, text/plain)
2020-07-30 08:08 UTC, Stig Nielsen
Details
Shellscript (test.sh,378 bytes, application/x-shellscript)
2020-07-30 14:17 UTC, Stig Nielsen
Details
debug - stack smashing detected (debug.txt,6.48 KB, text/plain)
2020-08-03 21:13 UTC, Stig Nielsen
Details
gcc -v and statistics for creduce (gcc_p_statistics.txt,8.05 KB, text/plain)
2020-08-03 21:15 UTC, Stig Nielsen
Details
instantiate_cpp_grammar.ii (instantiate_cpp_grammar.ii,2.64 KB, text/plain)
2020-08-03 21:17 UTC, Stig Nielsen
Details
new reduced test case and gcc output (xxx,41.74 KB, text/plain)
2020-08-07 16:07 UTC, Jack
Details
gcc output (gcc_out.fullasan.txt,42.87 KB, text/plain)
2020-08-09 20:11 UTC, Jack
Details
reduced boost 1.74 xml_grammar file (xml_grammar.ii,2.51 KB, text/plain)
2020-08-26 13:00 UTC, cJ
Details
testcase (testcase,2.08 KB, text/plain)
2020-08-27 01:31 UTC, cJ
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stig Nielsen 2020-07-01 19:45:59 UTC
I get the following error when trying to run genkernel --lvm --luks --install initramfs

ERROR: Command 'b2 --user-config=/var/tmp/genkernel/gk_ldJZhGwB/boost/boost_1_73_0/user-config.jam --without-python gentoorelease -j17 -q -d+2 pch=off --disable-icu boost.locale.icu=off --without-mpi --without-locale --without-context --without-coroutine --without-fiber --without-stacktrace --boost-build=/var/tmp/genkernel/gk_ldJZhGwB/boost/buildroot/usr/share/boost-build --prefix=/usr --layout=system --no-cmake-config threading=multi link=shared,static -sNO_BZIP2=1 -sNO_LZMA=1 -sNO_ZLIB=1 -sNO_ZSTD=1' failed!
* ERROR: create_initramfs(): append_data(): append_lvm(): populate_binpkg(): populate_binpkg(): gkbuild(): Failed to create binpkg of boost-1.73.0!

It seems to be a problem with boost-1.73.0. I didnt have the problem before upgrading to version 1.73.0.

Reproducible: Always

Steps to Reproduce:
I have boost 1.73.0 installed together with Genkernel 4.0.9. I am trying to upgrade the kernel (gentoo-sources) to version 5.7.6.
Actual Results:  
ERROR: Command 'b2 --user-config=/var/tmp/genkernel/gk_ldJZhGwB/boost/boost_1_73_0/user-config.jam --without-python gentoorelease -j17 -q -d+2 pch=off --disable-icu boost.locale.icu=off --without-mpi --without-locale --without-context --without-coroutine --without-fiber --without-stacktrace --boost-build=/var/tmp/genkernel/gk_ldJZhGwB/boost/buildroot/usr/share/boost-build --prefix=/usr --layout=system --no-cmake-config threading=multi link=shared,static -sNO_BZIP2=1 -sNO_LZMA=1 -sNO_ZLIB=1 -sNO_ZSTD=1' failed!
* ERROR: create_initramfs(): append_data(): append_lvm(): populate_binpkg(): populate_binpkg(): gkbuild(): Failed to create binpkg of boost-1.73.0

Expected Results:  
Working version of genkernel installed in /boot
Comment 1 Stig Nielsen 2020-07-01 19:47:44 UTC
Created attachment 647330 [details]
genkernel.log
Comment 2 Stig Nielsen 2020-07-01 19:48:22 UTC
Portage 2.3.103 (python 3.7.8-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-10.1.0, glibc-2.31-r5, 5.6.13-gentoo x86_64)
=================================================================
System uname: Linux-5.6.13-gentoo-x86_64-AMD_Ryzen_7_1700X_Eight-Core_Processor-with-gentoo-2.7
KiB Mem:    65814752 total,  63491736 free
KiB Swap:   75497468 total,  75497468 free
Timestamp of repository gentoo: Wed, 01 Jul 2020 18:00:01 +0000
Head commit of repository gentoo: 56911e5b7b0e6a0e71a495f7768123184ac7ac3b
sh bash 5.0_p17
ld GNU ld (Gentoo 2.34 p4) 2.34.0
app-shells/bash:          5.0_p17::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3-r1::gentoo
dev-lang/python:          2.7.18::gentoo, 3.6.10-r2::gentoo, 3.7.8::gentoo, 3.8.3::gentoo, 3.9.0_beta3::gentoo
dev-util/cmake:           3.17.3::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.20::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.16.2::gentoo
sys-devel/binutils:       2.34-r1::gentoo
sys-devel/gcc:            8.4.0-r1::gentoo, 10.1.0-r1::gentoo
sys-devel/gcc-config:     2.3.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.7::gentoo (virtual/os-headers)
sys-libs/glibc:           2.31-r5::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-jobs: 1

guru
    location: /var/lib/layman/guru
    masters: gentoo
    priority: 50

steam-overlay
    location: /var/lib/layman/steam-overlay
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=znver1 -O2 -pipe -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/libvirt/libvirtd.conf /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/sddm/scripts/Xsetup"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--with-bdeps=y --keep-going=y --jobs 3"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ ftp://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ rsync://ftp-stud.hs-esslingen.de/gentoo/"
LANG="da_DK.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j17"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/tmp"
USE="X a52 aac acl acpi activities alsa amd64 bash-completion berkdb bluetooth branding bzip2 cairo cdda cdr cli crypt cups cxx dbus declarative dri dts dvd dvdr elogind emboss encode exif fam flac fortran gdbm gif glamor gtk hardened iconv icu ipv6 java jpeg kde kerberos kipi kwallet lcms ldap libnotify libtirpc mad mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt5 readline sasl sdl seccomp semantic-desktop spell split-usr ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis widgets wxwidgets x264 xattr xcb xcomposite xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" L10N="da" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python3_6 python3_7" RUBY_TARGETS="ruby25 ruby27" SANE_BACKENDS="hp" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 3 David Seifert gentoo-dev 2020-07-03 12:20:28 UTC
Paging in slyfox, as smells a lot like bug 724314:

*** stack smashing detected ***: terminated
Comment 4 Thomas Deutschmann gentoo-dev Security 2020-07-03 12:56:44 UTC
Is it reproducible for you? Or does it work after multiple attempts?

Do you have same problem with =dev-libs/boost-1.73.0?

I was told that this is likely an obscure bug with gcc-10 which only triggers under very specific conditions...  you can be even lucky that 1.72.0 did work for you.
Comment 5 Thomas Deutschmann gentoo-dev Security 2020-07-03 12:58:24 UTC
You could also test if reducing MAKEOPTS helps for some reason.
Comment 6 Stig Nielsen 2020-07-03 16:16:03 UTC
(In reply to Thomas Deutschmann from comment #4)
> Is it reproducible for you? Or does it work after multiple attempts?
> 
> Do you have same problem with =dev-libs/boost-1.73.0?
> 
> I was told that this is likely an obscure bug with gcc-10 which only
> triggers under very specific conditions...  you can be even lucky that
> 1.72.0 did work for you.

I have tried 5 times, over two days, with the same outcome (waited a day and emerge --sync'd, trying my luck). I also tried reemerging dev-libs/boost. 

I just tried to run genkernel without makeopts=j17, same results. I will now try with =dev-libs/boost-1.73.0 and the retry genkernel.  

Btw thank you all for helping.
Comment 7 Stig Nielsen 2020-07-03 16:43:43 UTC
(In reply to Thomas Deutschmann from comment #5)
> You could also test if reducing MAKEOPTS helps for some reason.

I h(In reply to Stig Nielsen from comment #6)
> (In reply to Thomas Deutschmann from comment #4)
> > Is it reproducible for you? Or does it work after multiple attempts?
> > 
> > Do you have same problem with =dev-libs/boost-1.73.0?
> > 
> > I was told that this is likely an obscure bug with gcc-10 which only
> > triggers under very specific conditions...  you can be even lucky that
> > 1.72.0 did work for you.
> 
> I have tried 5 times, over two days, with the same outcome (waited a day and
> emerge --sync'd, trying my luck). I also tried reemerging dev-libs/boost. 
> 
> I just tried to run genkernel without makeopts=j17, same results. I will now
> try with =dev-libs/boost-1.73.0 and the retry genkernel.  
> 
> Btw thank you all for helping.

I have tried it now and it gives the same results. I will try on my Intel machine and see if the CPU/AM4 is one of the reasons.
Comment 8 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-03 18:48:10 UTC
As in bug #724314, -march=znver1 on znver1 managed to break your gcc. Are you building whole system with flags as in your make.conf?
    CFLAGS="-march=znver1 -O2 -pipe -ggdb"
    CXXFLAGS="-O2 -pipe"
Or you have some overrides in /etc/portage/package.env or the equivalent?

The flags are fine, I'm just trying to find any pattern why some systems manage to exhibit this behaviour while most of znvers don't. Rebuilding gcc with CFLAGS="--O2 -pipe -ggdb" will probably workaround it. Worth a try.

If you are san spend some time debugging actual failure can you try to minimize the sample source file that gcc fails on? https://wiki.gentoo.org/wiki/Gcc-ICE-reporting-guide#.5Bbonus.5D_minimize_self-contained_source_using_creduce has step by step instruction.
Comment 9 Stig Nielsen 2020-07-03 19:23:51 UTC
(In reply to Sergei Trofimovich from comment #8)
> As in bug #724314, -march=znver1 on znver1 managed to break your gcc. Are
> you building whole system with flags as in your make.conf?
>     CFLAGS="-march=znver1 -O2 -pipe -ggdb"
>     CXXFLAGS="-O2 -pipe"
> Or you have some overrides in /etc/portage/package.env or the equivalent?
> 
> The flags are fine, I'm just trying to find any pattern why some systems
> manage to exhibit this behaviour while most of znvers don't. Rebuilding gcc
> with CFLAGS="--O2 -pipe -ggdb" will probably workaround it. Worth a try.
> 
> If you are san spend some time debugging actual failure can you try to
> minimize the sample source file that gcc fails on?
> https://wiki.gentoo.org/wiki/Gcc-ICE-reporting-guide#.5Bbonus.
> 5D_minimize_self-contained_source_using_creduce has step by step instruction.

Thank you for the comment. I dont have any other overrides but I do have cross-x86_64-mingw32 set up, could that be source of the error?

I will try to recompile with the CFLAGS that you suggest. 

Thank you for the link. I will read it and follow it, so we dont get swamped in information.

Btw the Intel machine works without problems.
Comment 10 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-03 19:29:44 UTC
(In reply to Stig Nielsen from comment #9)
> (In reply to Sergei Trofimovich from comment #8)
> > As in bug #724314, -march=znver1 on znver1 managed to break your gcc. Are
> > you building whole system with flags as in your make.conf?
> >     CFLAGS="-march=znver1 -O2 -pipe -ggdb"
> >     CXXFLAGS="-O2 -pipe"
> > Or you have some overrides in /etc/portage/package.env or the equivalent?
> > 
> > The flags are fine, I'm just trying to find any pattern why some systems
> > manage to exhibit this behaviour while most of znvers don't. Rebuilding gcc
> > with CFLAGS="--O2 -pipe -ggdb" will probably workaround it. Worth a try.
> > 
> > If you are san spend some time debugging actual failure can you try to
> > minimize the sample source file that gcc fails on?
> > https://wiki.gentoo.org/wiki/Gcc-ICE-reporting-guide#.5Bbonus.
> > 5D_minimize_self-contained_source_using_creduce has step by step instruction.
> 
> Thank you for the comment. I dont have any other overrides but I do have
> cross-x86_64-mingw32 set up, could that be source of the error?

I would not expect it to interfere. But it's an interesting hypothesis. I'll try to install it on znver2 machine to see if it has any effect for me.
Comment 11 Stig Nielsen 2020-07-05 15:02:07 UTC
I recompiled gcc without znver1 and reemerged libtool and then reemerged boost , as you hinted and it works.

I will have some time these next days so I will try to work with the error on my other ZNVER1 (same set up hardware wise, but smaller gentoo build (I use it as a server)).
Comment 12 Jack 2020-07-21 15:24:46 UTC
I get the same error with sys-kernel/genkernel-9999, dev-libs/boost-1.73.0, and sys-kernel/gentoo-sources-5-7-9.  I'll post emerge --info in a minute.
Comment 13 Jack 2020-07-21 15:27:26 UTC
Created attachment 650084 [details]
output from "emerge --info"

output from "emerge --info"
Comment 14 Thomas Deutschmann gentoo-dev Security 2020-07-21 16:51:42 UTC
FYI: `emerge --info` do not really help here. Genkernel isn't using emerge or anything from portage. Please always add genkernel.log from failing run.

And please notice slyfox's comment that your toolchain could be broken. Try rebuilding/switching compiler for example.
Comment 15 Jack 2020-07-21 17:06:34 UTC
Created attachment 650112 [details]
genkernel log

I suppose broken toochain is possible, but after my last gcc upgrade, I did an "emerge -e @system" and the few failures were likely timing issues, as they all rebuilt with no errors when done one at a time.  However, I'll go ahead and rebuild gcc (I only have one version installed.)  Let me know if there is anything else useful I cay try, or information to provide.
Comment 16 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-21 18:33:39 UTC
> System uname: Linux-5.7.7-gentoo-x86_64-01-x86_64-AMD_Ryzen_5_2600_Six-Core_Processor-with-gentoo-2.6
> CFLAGS="-march=native -O2 -pipe"

You also hit a jackpot. -march=znver1 somehow breaks gcc-10.1.0. Rebuild would likely not help (unless you remove -march=native).

https://bugs.gentoo.org/730406#c8 suggests to extract minimal example that can reproduce crash on your gcc. That might ease stepping through gcc with gdb to see where stack is corrupted.
Comment 17 Jack 2020-07-21 23:10:05 UTC
Taking the first error, I have the original cpp file (52 lines, 1939 bytes) the .ii file created using --save-temps (147096 lines, 4129327 bytes) and the command which fails.  I can try stepping through gcc with gdb, but I'm not sure I even know what to look for.  Also, if I'm looking for info on where gcc is when failing, do I need to re-emerge gcc with any flags for debug?  I'm willing to put some time into this, but I need some guidance.
Comment 18 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-21 23:13:11 UTC
I added a https://wiki.gentoo.org/wiki/Project:Toolchain/724314-gcc-10-and-znver1#More_info to get some hints how to extract more info to debug it further.

Specifically we need to track down where stack gets corrupted: https://wiki.gentoo.org/wiki/Stack-smashing-debugging-guide
Comment 19 Jack 2020-07-22 00:12:18 UTC
(Minor typo in that first link - the Symptom line has mach instead of march.)

I'll work through that, but I'm curious why I didn't have any trouble emerging boost 1.73.0, but I do trip the bug when genkernel tries to compile the same version for the initramfs.  Would tracking down the differences between those two builds be of any help?
Comment 20 Thomas Deutschmann gentoo-dev Security 2020-07-22 01:51:46 UTC
We should be carrying same patches (except python stuff we don't need) https://gitweb.gentoo.org/proj/genkernel.git/tree/patches/boost-build/1.73.0?h=v4.0.10. The only difference I can spot is in gkbuild, we are removing stuff we don't anymore in the ebuild (https://gitweb.gentoo.org/proj/genkernel.git/tree/gkbuilds/boost-build.gkbuild?h=v4.0.10#n15) but this should be harmless.
Comment 21 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-22 06:38:06 UTC
(In reply to Jack from comment #17)
> Taking the first error, I have the original cpp file (52 lines, 1939 bytes)
> the .ii file created using --save-temps (147096 lines, 4129327 bytes) and
> the command which fails.  I can try stepping through gcc with gdb, but I'm
> not sure I even know what to look for.  Also, if I'm looking for info on
> where gcc is when failing, do I need to re-emerge gcc with any flags for
> debug?  I'm willing to put some time into this, but I need some guidance.

Try shrinking the example first with cvise or creduce: https://wiki.gentoo.org/wiki/Gcc-ICE-reporting-guide#.5Bbonus.5D_minimize_self-contained_source_using_creduce

(In reply to Jack from comment #19)
> (Minor typo in that first link - the Symptom line has mach instead of march.)
> 
> I'll work through that, but I'm curious why I didn't have any trouble
> emerging boost 1.73.0, but I do trip the bug when genkernel tries to compile
> the same version for the initramfs.  Would tracking down the differences
> between those two builds be of any help?

It probably comes down to gcc flag difference. At the very least I see '-O2 -march=native' (make.conf) vs -Os -fomit-frame-pointer (genkernel's flags) difference. It should not matter. If just makes gcc behave slightly different on different options. We need to debug whatever crashes.
Comment 22 Jack 2020-07-22 20:37:31 UTC
Created attachment 650230 [details]
output of gcc

Unfortunately, the crash is now sporadic, and I've gotten genkernel to complete a build of initramfs with no crash.  I also rebuild gcc with debugging on, and got the attached output.  I'm just starting with creduce, but perhaps the attached can provide some hints.

The log includes the command line, and it was run in a directory with only the .ii and .s files produced by -save-temps.
Comment 23 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-22 20:52:34 UTC
(In reply to Jack from comment #22)
> Created attachment 650230 [details]
> output of gcc
> 
> Unfortunately, the crash is now sporadic, and I've gotten genkernel to
> complete a build of initramfs with no crash.  I also rebuild gcc with
> debugging on, and got the attached output.  I'm just starting with creduce,
> but perhaps the attached can provide some hints.
> 
> The log includes the command line, and it was run in a directory with only
> the .ii and .s files produced by -save-temps.

The backtrace looks similar to backtrace reported by others. Unfortunately it still does not tell where stack corruption happens.
Comment 24 Jack 2020-07-22 21:57:50 UTC
Created attachment 650246 [details]
crash with backtrace

I'm terrible at reading backtraces, so I don't know if this is useful or not.  I have not exited gdb yet, in case I can get any more useful information from it.  It has some similarities to https://bugs.gentoo.org/724314#c10 but it's somewhat shorter, and DOES have source names and line numbers.
Comment 25 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-22 23:17:04 UTC
(In reply to Jack from comment #24)
> Created attachment 650246 [details]
> crash with backtrace
> 
> I'm terrible at reading backtraces, so I don't know if this is useful or
> not.  I have not exited gdb yet, in case I can get any more useful
> information from it.  It has some similarities to
> https://bugs.gentoo.org/724314#c10 but it's somewhat shorter, and DOES have
> source names and line numbers.

Unfortunately this crash happened after stack smashing happened before. You will need to catch stack smash as it happens when canary is overwritten to find the offending line of code. https://wiki.gentoo.org/wiki/Stack-smashing-debugging-guide has a step-by-step how to do it.
Comment 26 Jack 2020-07-23 20:18:35 UTC
Created attachment 650402 [details]
gdb output

Next attempt.  I don't know if it matters, but I couldn't get gcc to build without pie.  The only thing I'm really not sure of is if I got the watch location correct.  From the example from "mov    %rax,-0x8(%rbp)" was to watch "*(long*)($rbp-8)".  I used "mov    %rax,0x78(%rsp)" to watch "*(long*)($rsp+78)".  At least I'm getting used to gdb, so let me know what to change if I need to try again.
Comment 27 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-23 21:06:40 UTC
(In reply to Jack from comment #26)
> Created attachment 650402 [details]
> gdb output
> 
> Next attempt.  I don't know if it matters, but I couldn't get gcc to build
> without pie.  The only thing I'm really not sure of is if I got the watch
> location correct.  From the example from "mov    %rax,-0x8(%rbp)" was to
> watch "*(long*)($rbp-8)".  I used "mov    %rax,0x78(%rsp)" to watch
> "*(long*)($rsp+78)".  At least I'm getting used to gdb, so let me know what
> to change if I need to try again.

No need to build gcc without pie. It's actually is never a pie binary (one of the exceptions).

Looks like you are very close! You did it almost correctly. The small annoyance is that 0x78 is different from 78:

>   0x00000000006595de <+46>:    mov    %rax,0x78(%rsp)
> ...
> (gdb) watch *(long*)($rsp+78)

It has to be 'watch *(long*)($rsp+0x78)'. Otherwise you are off by 42 bytes.

"""
(gdb) print 78
$1 = 78
(gdb) print 0x78
$2 = 120
"""
Comment 28 Jack 2020-07-24 03:33:21 UTC
posted to #gentoo-toolchain: hits the watch at the same location, but with differernt old and new.  Then, hitting "c" repeatedly hits the watch in different functions (including gimplify_bind_expr, all with different values.  Is there some value change to look for, or just keep going and capture the output?
-------
multiple (c)ontinue's don't seem to end up anywhere useful.  would a screen-share be of any use here, or can you suggest anything I may have missed?
Comment 29 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-24 06:30:43 UTC
(In reply to Jack from comment #28)
> posted to #gentoo-toolchain: hits the watch at the same location, but with
> differernt old and new.  Then, hitting "c" repeatedly hits the watch in
> different functions (including gimplify_bind_expr, all with different
> values.  Is there some value change to look for, or just keep going and
> capture the output?
> -------
> multiple (c)ontinue's don't seem to end up anywhere useful.  would a
> screen-share be of any use here, or can you suggest anything I may have
> missed?

If gimplify_bind_expr() was called from that very cp_gimplify_expr() maybe that was it. Hard to say without seeing the output.

Ideally you need to add a break and watch on every 'cp_gimplify_expr()' occurence as we can get many nested calls.

Do you still run gdb against full preprocessed file? Or did you reduce it with creduce/cvise?

Full preprocessed file will likely be very hard to follow as it has a lot (millions?) of *_gimplify_* calls that repeat. Hopefully reduced example will be a lot smaller.
Comment 30 Jack 2020-07-24 14:04:24 UTC
Run on full preprocessed file.  I'll try C-reduce again, but don't really trust it will work correctly if test.sh is not guaranteed to fail consistently.  More correctly, if the full run doesn't always fail, then test.sh may give wrong answer, and mislead creduce.
Comment 31 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-24 17:47:04 UTC
(In reply to Jack from comment #30)
> Run on full preprocessed file.  I'll try C-reduce again, but don't really
> trust it will work correctly if test.sh is not guaranteed to fail
> consistently.  More correctly, if the full run doesn't always fail, then
> test.sh may give wrong answer, and mislead creduce.

It's OK. It will not reduce too much. It will reduce too little. Once it stabilizes you can make test.sh to run gcc 10 times (or similar) to reduce it further.
Comment 32 Jack 2020-07-24 18:37:25 UTC
I altered test.sh to try up to ten times, and only return 1 if it works all ten times, otherwise return 0.  creduce has now been running for almost two hours.  The original cpp file was 52 lines, 1939 bytes.  The current cpp is 5 lines, under 300 bytes.

Before I got that started, I made a modified version of test.sh that would put out the error or else OK, and then sort the results and do uniq -d.  On multiple runs, it averaged about
20% OK
40% ./boost/spirit/home/classic/core/composite/composite.hpp:67:31: internal compiler error: Aborted
40% ./boost/wave/grammars/cpp_predef_macros_grammar.hpp:94:31: internal compiler error: Aborted

Above is running on the .ii file.  Running on the .cpp file (in a fresh directory, out of the boost build dir) those results start in /usr/includ/boost instead of ./boost.

Is there anything worth my posting now?  Othewise, I'll just wait until I can try working with the reduced file?
Comment 33 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-24 18:44:08 UTC
(In reply to Jack from comment #32)
> I altered test.sh to try up to ten times, and only return 1 if it works all
> ten times, otherwise return 0.  creduce has now been running for almost two
> hours.  The original cpp file was 52 lines, 1939 bytes.  The current cpp is
> 5 lines, under 300 bytes.

Woohoo! That one should be easier to step through in gdb!

> Before I got that started, I made a modified version of test.sh that would
> put out the error or else OK, and then sort the results and do uniq -d.  On
> multiple runs, it averaged about
> 20% OK
> 40% ./boost/spirit/home/classic/core/composite/composite.hpp:67:31: internal
> compiler error: Aborted
> 40% ./boost/wave/grammars/cpp_predef_macros_grammar.hpp:94:31: internal
> compiler error: Aborted
> 
> Above is running on the .ii file.  Running on the .cpp file (in a fresh
> directory, out of the boost build dir) those results start in
> /usr/includ/boost instead of ./boost.
> 
> Is there anything worth my posting now?  Othewise, I'll just wait until I
> can try working with the reduced file?

That sounds like great progress! You can attach minimized example for posterity (I will also try to break gcc locally as well).

I did not understand if the sample you got is selfcontained or it still uses external includes. If it's selfcontained you can try again to trace gcc through gdb and see if you can catch stack corruption.
Comment 34 Jack 2020-07-24 22:10:59 UTC
I attempted for a self contained example, but when I removed the -I pointing into the build area, I think it found some includes in the installed system, so the example will work for others, if they have boost 1.73.0 installed.  If necessary, I suppose I can go back and try to find the necessary includes.

creduce took almost 3 hours, and ended up with 

#include <boost/wave/cpplexer/cpp_lex_iterator.hpp>
#include <boost/wave/grammars/cpp_predef_macros_grammar.hpp>
typedef boost::wave::cpplexer::lex_token<> a;
typedef boost::wave::cpplexer::lex_iterator<a> b;
template struct boost::wave::grammars::predefined_macros_grammar_gen<b>;

I'll attach my gdb output, but I feel like I'm just going in circles.  I feel like I've missed something subtle, or maybe I'm just not really sure what I'm supposed to be looking for in the gdb output.
Comment 35 Jack 2020-07-24 22:12:39 UTC
Created attachment 650644 [details]
gdb output

using the creduced source shown in previous comment.
Comment 36 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-24 23:19:00 UTC
(In reply to Jack from comment #34)
> I attempted for a self contained example, but when I removed the -I pointing
> into the build area, I think it found some includes in the installed system,
> so the example will work for others, if they have boost 1.73.0 installed. 
> If necessary, I suppose I can go back and try to find the necessary includes.
> 
> creduce took almost 3 hours, and ended up with 
> 
> #include <boost/wave/cpplexer/cpp_lex_iterator.hpp>
> #include <boost/wave/grammars/cpp_predef_macros_grammar.hpp>
> typedef boost::wave::cpplexer::lex_token<> a;
> typedef boost::wave::cpplexer::lex_iterator<a> b;
> template struct boost::wave::grammars::predefined_macros_grammar_gen<b>;

Ah, that's not a self-contained example. I'm surprised .ii file contained them. It should not. You can expand includes with 'gcc -E <other-options> file.cpp -o file.pp.cpp' and continue reducing preprocessed file.

> I'll attach my gdb output, but I feel like I'm just going in circles.  I
> feel like I've missed something subtle, or maybe I'm just not really sure
> what I'm supposed to be looking for in the gdb output.

We can try to do it backwards once/if you'll be able to shrink the example further.
Comment 37 Jack 2020-07-24 23:38:55 UTC
I am getting confused.  I ran creduce on the .cpp, not on the .ii.   Are those five lines still too big, or is the problem the includes?  Or, should I run your suggestion and then run creduce on that?  The discussion(s) on tracking this down need to be more explicit about all this.  It's not at all obvious to folks who aren't very experienced in gdb and gcc details.
Comment 38 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-25 07:14:07 UTC
(In reply to Jack from comment #37)
> I am getting confused.  I ran creduce on the .cpp, not on the .ii.   Are
> those five lines still too big, or is the problem the includes?

Correct. Currently we see the crash in conversion of c++ expression (sort of) into lower-level representation in gcc. Boost includes import many complicated expressions into the final result. reduction of .ii (if possible) where all includes are inlined into a single file will make an example shorter.

To make it easily inspectable source file should consists of tens of expressions, not tens of thousands.

> Or, should I run your suggestion and then run creduce on that?

There should be no fundamental differece between .ii reduction and reduction of 'gcc -E' preprocessed file. Reducing 'gcc -E' preprocessed file can be faster as you already made it so small.

> The discussion(s) on tracking this down need to be more explicit about all this.  It's not at all obvious to folks who aren't very experienced in gdb and gcc details.

It's not specific to gcc or gdb. We need smallest possible input for a faulty program because we are tracing very low-level per-instruction execution of the program to see where it got miscompiled.

External includes make input relatively large.
Comment 39 Jack 2020-07-25 16:33:05 UTC
the creduced version of the -E preprocessed file only fails <5% of the time.  I have to wonder if my test.sh wasn't really right.  I'm going to try from scratch again.
Comment 40 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-25 19:30:56 UTC
(In reply to Jack from comment #39)
> the creduced version of the -E preprocessed file only fails <5% of the time.
> I have to wonder if my test.sh wasn't really right.  I'm going to try from
> scratch again.

Having slightly lower failure probability does not sound bad. As long as gcc still crashes it means we have something wrong happening.
Comment 41 Jack 2020-07-26 18:21:32 UTC
Created attachment 650886 [details]
output of gcc and of gdb

Started again.  Preprocessed with gcc -E -P.  Creduce ran about 13 hours and reduced to a single line

template <long()()->decltype(0)

which fails 100%, although the failures seem different from the pre-reduced file.

Attachment has output of gcc, showing "instantiate_predef_macros.pp.cpp:1:21: internal compiler error: in splice_late_return_type, at cp/pt.c:29095"

In gdb, with breakpoint on that line, the break is:
Thread 2.1 "cc1plus" hit Breakpoint 2, 0x000000000072ad8c in splice_late_return_type(tree_node*, tree_node*) ()
    at /var/tmp/portage/sys-devel/gcc-10.1.0-r2/work/gcc-10.1.0/gcc/wide-int-bitmask.h:86
I don't understand how a breakpoint in one .c file seems to trigger in a different .h file.

I'm open to suggestions on what else I can do.
Comment 42 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-26 18:40:10 UTC
(In reply to Jack from comment #41)
> Created attachment 650886 [details]
> output of gcc and of gdb
> 
> Started again.  Preprocessed with gcc -E -P.  Creduce ran about 13 hours and
> reduced to a single line
> 
> template <long()()->decltype(0)
> 
> which fails 100%, although the failures seem different from the pre-reduced
> file.
> 
> Attachment has output of gcc, showing
> "instantiate_predef_macros.pp.cpp:1:21: internal compiler error: in
> splice_late_return_type, at cp/pt.c:29095"
> 
> In gdb, with breakpoint on that line, the break is:
> Thread 2.1 "cc1plus" hit Breakpoint 2, 0x000000000072ad8c in
> splice_late_return_type(tree_node*, tree_node*) ()
>     at
> /var/tmp/portage/sys-devel/gcc-10.1.0-r2/work/gcc-10.1.0/gcc/wide-int-
> bitmask.h:86
> I don't understand how a breakpoint in one .c file seems to trigger in a
> different .h file.
> 
> I'm open to suggestions on what else I can do.

Well done! I can reproduce the failure on gcc-10.2.0 locally as well:

$ cat a.cc
template <long()()->decltype(0)
$ LANG=C gcc-10.2.0 a.cc
a.cc:1:21: internal compiler error: in splice_late_return_type, at cp/pt.c:29152
    1 | template <long()()->decltype(0)
      |                     ^~~~~~~~~~~

It might happen to be yet another problem. But let me fix a fix for this one first and then we can retry with fixed gcc.
Comment 43 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-26 18:43:00 UTC
(In reply to Sergei Trofimovich from comment #42)
> (In reply to Jack from comment #41)
> > Created attachment 650886 [details]
> > output of gcc and of gdb
> > 
> > Started again.  Preprocessed with gcc -E -P.  Creduce ran about 13 hours and
> > reduced to a single line
> > 
> > template <long()()->decltype(0)
> > 
> > which fails 100%, although the failures seem different from the pre-reduced
> > file.
> > 
> > Attachment has output of gcc, showing
> > "instantiate_predef_macros.pp.cpp:1:21: internal compiler error: in
> > splice_late_return_type, at cp/pt.c:29095"
> > 
> > In gdb, with breakpoint on that line, the break is:
> > Thread 2.1 "cc1plus" hit Breakpoint 2, 0x000000000072ad8c in
> > splice_late_return_type(tree_node*, tree_node*) ()
> >     at
> > /var/tmp/portage/sys-devel/gcc-10.1.0-r2/work/gcc-10.1.0/gcc/wide-int-
> > bitmask.h:86
> > I don't understand how a breakpoint in one .c file seems to trigger in a
> > different .h file.
> > 
> > I'm open to suggestions on what else I can do.
> 
> Well done! I can reproduce the failure on gcc-10.2.0 locally as well:
> 
> $ cat a.cc
> template <long()()->decltype(0)
> $ LANG=C gcc-10.2.0 a.cc
> a.cc:1:21: internal compiler error: in splice_late_return_type, at
> cp/pt.c:29152
>     1 | template <long()()->decltype(0)
>       |                     ^~~~~~~~~~~
> 
> It might happen to be yet another problem. But let me fix a fix for this one
> first and then we can retry with fixed gcc.

Probably a https://gcc.gnu.org/PR95820.
Comment 44 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-26 20:20:03 UTC
Created attachment 650896 [details, diff]
gcc-10.2.0-backport-PR95820-ICE-fundecl.patch
Comment 45 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-26 20:23:36 UTC
gcc-10.2.0-backport-PR95820-ICE-fundecl.patch should fix the "internal compiler error: in splice_late_return_type, at cp/pt.c:29095" failure.

To enable it drop the file to /etc/portage/patches/sys-devel/gcc:10/ and rebuild gcc. Once done check if "splice_late_return_type, at cp/pt.c:29095" ICE is gone  try to re-reduce preprocessed boost file again.
Comment 46 Jack 2020-07-27 00:05:33 UTC
gcc rebuilt with the patch.  Running on the one line file complains "error: 'parameter' function with trailing return type not declared with 'auto' type specifier" and "error: expected '>' at end of input".  Has required syntax changed?

Running on the previously preprocessed file failed 28 out of 100.
Running on the original file (with the #include's) failed 21 out of 100.
I re-emerge boost-1.73.0 with no error, but I had no error before.
I ran genkernel with no error, and even though I told it not to cleanup, the boost build dir was cleaned, so I can't try on the new version of the original file which failed.

I wonder why some of the old files still fail, even sometimes, but perhaps it's best to call it fixed for now, and see if the patch also works for the few others who also found the problem.
Comment 47 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-27 06:46:26 UTC
(In reply to Jack from comment #46)
> gcc rebuilt with the patch.  Running on the one line file complains "error:
> 'parameter' function with trailing return type not declared with 'auto' type
> specifier" and "error: expected '>' at end of input".  Has required syntax
> changed?

The one-liner reduced file had an incorrect c++ syntax. It even lacked ';' after the statement. So the proper compiler error instead of compiler ICE is fine. That's why we needed to start from any syntactically valid state for re-reduction.

> Running on the previously preprocessed file failed 28 out of 100.
> Running on the original file (with the #include's) failed 21 out of 100.

The original bug is still present then. Just harder to hit.
Comment 48 Larry the Git Cow gentoo-dev 2020-07-27 06:50:35 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=fafbb4148cb5c2cf7e1ae02679240cba43e95992

commit fafbb4148cb5c2cf7e1ae02679240cba43e95992
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-07-27 06:48:54 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-07-27 06:48:54 +0000

    10.2.0: backport ICE on invalid function declarations
    
    Reported-by: Jack Ostroff
    Bug: https://bugs.gentoo.org/730406
    Bug: https://gcc.gnu.org/PR95820
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 10.2.0/gentoo/34_all_fundecl-ICE-PR95820.patch | 25 +++++++++++++++++++++++++
 10.2.0/gentoo/README.history                   |  1 +
 2 files changed, 26 insertions(+)
Comment 49 Stig Nielsen 2020-07-27 19:16:36 UTC
First of all, thank all of you for the hard work and a special thank you to Jack and Sergei. 

I have tried the patch and am able to rebuild GCC etc. and then reemerge boost. After that genkernel is working without problem (I did not reinstall genkernel). So the patch is working for a 1700x ryzen.

I haven't yet have time to try the path Jack has taken to find the bug and honestly, I am not sure what to do but will try on my alternative setup.

BTW I am not sure if it means anything but when I have CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" set in make.conf, boost fails to build. 
@Sergei, please advice if I should open a new bug for this error or if it is related and should stay here. In any case commenting it out, allows boost to build.
Comment 50 Stig Nielsen 2020-07-27 19:26:57 UTC
ARRHHHH I wrote to soon. 

I reemerge genkernel and tried:genkernel --lvm --luks --install initramfs and got the same error again with: *** stack smashing detected ***: terminated
In file included from libs/serialization/src/xml_grammar.cpp:64:......
 
Sorry guys I got excited about the patch. So to be clear, the patch is not resolving the error on an 1700x ryzen.
Comment 51 Jack 2020-07-27 21:19:58 UTC
Note I've never had this error emerging boost, and have now run genkernel several times without seeing the error.  I think there is something non-deterministic involved here.  I have a looping test set up, and with gcc rebuilt with the patch, I am getting 20-4o% errors and the rest complete OK (with some warnings, but no errors.)  Some of the intermediate versions while creduce is running fail with errors other than the stack smashing, but as those or marked uninteresting to creduce, it doesn't matter.  

It might have to to with the order in which things get done, with multiple threads, although I'm not even certain if that compiler is multithreaded or not.  Further, with a single invocation of x86_64-pc-linux-gnu-g++, I'm not sure how to control that, where "-j 1" and related incantations would work with emerge or maybe even genkernel.
Comment 52 Stig Nielsen 2020-07-30 08:07:08 UTC
Created attachment 651584 [details]
Minimize self-contained source using creduce

Hi all, 

I have tried to follow the Gcc-Ice guide and have gotten the following output. But I need help with where to go from here. I am not sure how to apply the https://wiki.gentoo.org/wiki/Stack-smashing-debugging-guide on this code.

Please advice.

Regards Stig
Comment 53 Stig Nielsen 2020-07-30 08:08:06 UTC
Created attachment 651586 [details]
Reproducer
Comment 54 Sergei Trofimovich (RETIRED) gentoo-dev 2020-07-30 13:16:23 UTC
(In reply to Stig Nielsen from comment #52)
> Created attachment 651584 [details]
> Minimize self-contained source using creduce

Oh, nice! Can you also post the exact compiler command you run against this source and shell script you used for creaduce?

> Hi all, 
> 
> I have tried to follow the Gcc-Ice guide and have gotten the following
> output. But I need help with where to go from here. I am not sure how to
> apply the https://wiki.gentoo.org/wiki/Stack-smashing-debugging-guide on
> this code.
> 
> Please advice.
> 
> Regards Stig


Normally you can run gcc under gdb. For that you need to extract raw command. I usually use '-v' command:

$ gcc-10.2.0 -c a.cc -Os -v
...
 /usr/libexec/gcc/x86_64-pc-linux-gnu/10.2.0/cc1plus -quiet -v -D_GNU_SOURCE a.cc -quiet -dumpbase a.cc -mtune=generic -march=x86-64 -auxbase-strip a.o -Os -version -fdiagnostics-color=always -o /tmp/ccCW1wtS.s
...

Now we can run it under gdb:

$ gdb --args /usr/libexec/gcc/x86_64-pc-linux-gnu/10.2.0/cc1plus -quiet -v -D_GNU_SOURCE a.cc -quiet -dumpbase a.cc -mtune=generic -march=x86-64 -auxbase-strip a.o -Os -version -fdiagnostics-color=always -o /tmp/ccCW1wtS.s
> start
Temporary breakpoint 1, main (argc=17, argv=0x7fffffffd748) at /usr/src/debug/sys-devel/gcc-10.2.0/gcc-10.2.0/gcc/main.c:35

It's ready to debug.
Comment 55 Stig Nielsen 2020-07-30 14:17:26 UTC
Created attachment 651680 [details]
Shellscript

I used this command: creduce test.sh instantiate_cpp_grammar.ii

@Sergei, I hope this is what you were asking for. 

The debug I will try when I get back home.

Thank you for helping me.
Comment 56 Jack 2020-08-02 23:48:21 UTC
I was away for the weekend, and the latest creduce (which took over five days) came up with 

struct a {
  template <typename> static char ao;
  template <class> struct ao<>:

which looks like another invalid syntax that triggers an ICE.  Unless someone does it first, I should get to debugging within a few days.
Comment 57 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-03 06:50:32 UTC
(In reply to Stig Nielsen from comment #55)
> Created attachment 651680 [details]
> Shellscript
> 
> I used this command: creduce test.sh instantiate_cpp_grammar.ii
> 
> @Sergei, I hope this is what you were asking for. 
> 
> The debug I will try when I get back home.
> 
> Thank you for helping me.

Oh, you will need to tweak the script for this specific failure.

> LANG=C++ x86_64-pc-linux-gnu-g++ -fvisibility-inlines-hidden -march=znver1 -O2 -pipe -ggdb -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1 -DNDEBUG  -I"." -c instantiate_cpp_grammar.ii -o instantiate_cpp_grammar.o \
> >gcc_out.txt 2>&1
> grep "internal compiler error" gcc_out.txt >/dev/null 2>&1

LANG=C is a human language locale, not programming language flavour. It's used to make gcc produce english errors. Otherwise 'internal compiler error' may be localised.

'grep "internal compiler error" gcc_out.txt >/dev/null 2>&1' is a typical string to search for any problem in compiler. We are specifically interested in smashed stack, thus command should be:

"""
LANG=C x86_64-pc-linux-gnu-g++ -fvisibility-inlines-hidden -march=znver1 -O2 -pipe -ggdb -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1 -DNDEBUG  -I"." -c instantiate_cpp_grammar.ii -o instantiate_cpp_grammar.o \
>gcc_out.txt 2>&1
grep "stack smashing detected" gcc_out.txt >/dev/null 2>&1
"""

This script should maintain the property of a compiler broken by stack corruption instead of any other invalid C++ construct.
Comment 58 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-03 07:04:37 UTC
(In reply to Jack from comment #56)
> I was away for the weekend, and the latest creduce (which took over five
> days) came up with 
> 
> struct a {
>   template <typename> static char ao;
>   template <class> struct ao<>:
> 
> which looks like another invalid syntax that triggers an ICE.  Unless
> someone does it first, I should get to debugging within a few days.

I filed https://gcc.gnu.org/PR96425 to fix ICE on invalid code.

My guess that similar to https://bugs.gentoo.org/730406#c54 you used 'internal compiler error' as an interestingness test. But to track down stack corruption we will need a 'stack smashing detected' presence. Otherwise we will uncover more unrelated (or maybe related!) crashes.
Comment 59 Stig Nielsen 2020-08-03 21:13:45 UTC
Created attachment 652808 [details]
debug - stack smashing detected

Hi all, 

I have tried to debug and get a stack smashing detected. The error is: 

*** stack smashing detected ***: terminated
I fil inkluderet fra libs/serialization/src/xml_grammar.cpp:64:
libs/serialization/src/basic_xml_grammar.ipp: In constructor »boost::archive::basic_xml_grammar<CharType>::basic_xml_grammar() [with CharType = char]«:
libs/serialization/src/basic_xml_grammar.ipp:328:9: intern oversætterfejl: Afbrudt (SIGABRT)
  325 |         str_p(BOOST_ARCHIVE_XML_CLASS_ID()) >> NameTail
      |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  326 |         >> Eq
      |         ~~~~~
  327 |         >> L'"'
      |         ~~~~~~~
  328 |         >> int_p [xml::assign_object(rv.class_id)]
      |         ^~~~~~~~~~
0xc7662f crash_signal

Does it look right. It is the first time I am trying it and there was no debugging other then running gcc-10.1.0 -c libs/serialization/src/xml_grammar.cpp -Os -v in the Boost workdirectory (/tmp/portage/dev-libs/boost-1.73.0/work/boost_1_73_0-abi_x86_64.amd64)
Comment 60 Stig Nielsen 2020-08-03 21:15:45 UTC
Created attachment 652810 [details]
gcc -v and statistics for creduce

I have tried to run creduce as instructed in earlier comments. Here are the gcc -v out put and statistics
Comment 61 Stig Nielsen 2020-08-03 21:17:15 UTC
Created attachment 652812 [details]
instantiate_cpp_grammar.ii

The script for creduce (test.sh) is:
 
LANG=C x86_64-pc-linux-gnu-g++ -fvisibility-inlines-hidden -march=znver1 -O2 -pipe -ggdb -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden  -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1 -DNDEBUG  -I"." -c instantiate_cpp_grammar.ii -o instantiate_cpp_grammar.o \
>gcc_out.txt 2>&1
grep "stack smashing detected" gcc_out.txt >/dev/null 2>&1
Comment 62 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-03 21:58:20 UTC
(In reply to Stig Nielsen from comment #59)
> Does it look right. It is the first time I am trying it and there was no
> debugging other then running gcc-10.1.0 -c
> libs/serialization/src/xml_grammar.cpp -Os -v in the Boost workdirectory
> (/tmp/portage/dev-libs/boost-1.73.0/work/boost_1_73_0-abi_x86_64.amd64)

Yeah, this looks correct and looks similar enough to other failures.
Comment 63 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-03 22:03:53 UTC
(In reply to Stig Nielsen from comment #61)
> Created attachment 652812 [details]
> instantiate_cpp_grammar.ii
> 
> The script for creduce (test.sh) is:
>  
> LANG=C x86_64-pc-linux-gnu-g++ -fvisibility-inlines-hidden -march=znver1 -O2
> -pipe -ggdb -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline
> -Wall -fvisibility=hidden  -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1
> -DNDEBUG  -I"." -c instantiate_cpp_grammar.ii -o instantiate_cpp_grammar.o \
> >gcc_out.txt 2>&1
> grep "stack smashing detected" gcc_out.txt >/dev/null 2>&1

Interesting. The reduction looks legitimate. The resulting file is invalid c++ (for me g++-10.2.0 rejects it). That probably means corruption happens very early in c++ frontend in template matches/instantiations. Which might be a good thing as before it crashed in GIMPLE internals (way later in gcc stages).

Now you can try to get exact place where stack gets corrupted. https://bugs.gentoo.org/730406#c54
Comment 64 Jack 2020-08-06 20:47:14 UTC
My run of creduce was stopped by an Isaias induced power loss, and the last version of the cpp file it left did not trigger the error in over 100 tries.  Based on a message from the creduce-dev list, it had to have triggered the error at least once, or it would not have been left there as a candidate file.  I've restarted creduce from an intermediate file I saved earlier.  However, part of the list response included "The real answer here is to avoid nondeterminism, for example by disabling ASLR on your machine. I'm not sure that you're going to get a useful result out of C-Reduce here, if you don't do that."  
So - is it reasonable (or maybe even necessary) to turn off ASLR while debugging this stack smashing debugging, including running creduce?
Comment 65 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-06 21:06:26 UTC
(In reply to Jack from comment #64)
> My run of creduce was stopped by an Isaias induced power loss, and the last
> version of the cpp file it left did not trigger the error in over 100 tries.

That means the breakage is probably somehow dependent on some volatile state, like hardware memory layout.

> Based on a message from the creduce-dev list, it had to have triggered the
> error at least once, or it would not have been left there as a candidate
> file.  I've restarted creduce from an intermediate file I saved earlier. 
> However, part of the list response included "The real answer here is to
> avoid nondeterminism, for example by disabling ASLR on your machine. I'm not
> sure that you're going to get a useful result out of C-Reduce here, if you
> don't do that."  
> So - is it reasonable (or maybe even necessary) to turn off ASLR while
> debugging this stack smashing debugging, including running creduce?

I agree that determinism would help. Unfortunately we have to work with what we have.

Disabling ASLR might help a bit. By either making the error more frequent or not happen at all. The knob is https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/sysctl/kernel.rst#randomize-va-space
Comment 66 Jack 2020-08-07 16:07:54 UTC
Created attachment 653616 [details]
new reduced test case and gcc output

Disabling aslr did not seem to change anything - about 20% fails for this file.  New attachment is the latest reduced cpp file and output of gcc.  I wonder if this isn't another syntax error, and not the real bug?
Comment 67 Jack 2020-08-07 17:07:30 UTC
Someone on the creduce-dev list also suggested building gcc with  --with-build-config=bootstrap-asan.  Does that make any sense?  I don't see how to do so, so I'd need some pointers or instruction.
Comment 68 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-07 18:35:05 UTC
(In reply to Jack from comment #66)
> Created attachment 653616 [details]
> new reduced test case and gcc output
> 
> Disabling aslr did not seem to change anything - about 20% fails for this
> file.  New attachment is the latest reduced cpp file and output of gcc.  I
> wonder if this isn't another syntax error, and not the real bug?

This attachment contains both c file and gcc errors. I struggle to find where on finishes and another starts. It also does not contain actual gcc command. Can you post them separately?
Comment 69 Jack 2020-08-07 18:38:23 UTC
The gcc command was the same one I've been using all along.  The first line of the gcc output is the first line that contains "error:" and the error reflects the first line of the file.  I'll repost separately when I'm back at my desktop.
Comment 70 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-07 18:45:11 UTC
(In reply to Jack from comment #67)
> Someone on the creduce-dev list also suggested building gcc with 
> --with-build-config=bootstrap-asan.  Does that make any sense?  I don't see
> how to do so, so I'd need some pointers or instruction.

Might work for you. I tried to build such a gcc and it did not flag anything for me on znver2. Maybe you will be luckier. To build asan'd gcc you need to build it as:

  # EXTRA_ECONF=--with-build-config=bootstrap-asan GCC_MAKE_TARGET=all FEATURES="-sandbox -usersandbox" emerge -v1 gcc

FEATURES="-sandbox -usersandbox" will probably be needed for all emerge invocations that will try to use this gcc as asan is not compatible with sandbox.
Comment 71 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-07 18:47:03 UTC
(In reply to Jack from comment #69)
> The gcc command was the same one I've been using all along.  The first line
> of the gcc output is the first line that contains "error:" and the error
> reflects the first line of the file.  I'll repost separately when I'm back
> at my desktop.

I assume it's a command
"""
/usr/bin/x86_64-pc-linux-gnu-g++ -fvisibility-inlines-hidden -Os -pipe -fomit-frame-pointer -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1 -DNDEBUG -I. -c -o instantiate_predef_macros.o instantiate_predef_macros.cpp
"""
Comment 72 Jack 2020-08-08 17:44:01 UTC
(In reply to Sergei Trofimovich from comment #70)
[snip]
> To build asan'd gcc you need to build it as:
> 
>   # EXTRA_ECONF=--with-build-config=bootstrap-asan GCC_MAKE_TARGET=all
> FEATURES="-sandbox -usersandbox" emerge -v1 gcc
The build failed during install phase with five cases of 

==14750==ERROR: LeakSanitizer: detected memory leaks

but I think it's a delayed report, as the previous lines are

make[2]: Leaving directory '/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/build/x86_64-pc-linux-gnu/libatomic'
make[1]: Leaving directory '/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/build'

I assume this is not the place to discuss that issue, but where is?
Comment 73 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-08 21:13:55 UTC
(In reply to Jack from comment #72)
> (In reply to Sergei Trofimovich from comment #70)
> [snip]
> > To build asan'd gcc you need to build it as:
> > 
> >   # EXTRA_ECONF=--with-build-config=bootstrap-asan GCC_MAKE_TARGET=all
> > FEATURES="-sandbox -usersandbox" emerge -v1 gcc
> The build failed during install phase with five cases of 
> 
> ==14750==ERROR: LeakSanitizer: detected memory leaks
> 
> but I think it's a delayed report, as the previous lines are
> 
> make[2]: Leaving directory
> '/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/build/x86_64-pc-linux-gnu/
> libatomic'
> make[1]: Leaving directory
> '/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/build'

It means something is broken. ./config/bootstrap-asan.mk already disables leat detection: export ASAN_OPTIONS=detect_leaks=0:use_odr_indicator=1.

You can try to also set the variable explicitly in case it leaks out somewhere else. But it should not. It works for me as is.

> I assume this is not the place to discuss that issue, but where is?

A separate ticket bug on bugs.gentoo.org might be ok.
Comment 74 Jack 2020-08-09 00:02:30 UTC
Thanks.  That worked.  Some quick tests show lots of leaks detected.  I'm going to try to figure out if I can work with the creduced version, or if the leaks on compiling the original preprocessed version need reporting separately.
Comment 75 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-09 08:03:17 UTC
(In reply to Jack from comment #74)
> Thanks.  That worked.  Some quick tests show lots of leaks detected.  I'm
> going to try to figure out if I can work with the creduced version, or if
> the leaks on compiling the original preprocessed version need reporting
> separately.

Memory leaks should not be important and possibly a distraction in this case. We are looking for memory corruption here.

Note that asan-ed version of gcc might not smash stack anymore. The error could transform into asan report of memory write into out-of-bounds location.

To avoid memory leaks getting in the way you will probably need something like:

    export ASAN_OPTIONS=detect_leaks=0:use_odr_indicator=1

added to you test script for creduce. Otherwise memory leaks will prevent you from discovering possible memory corruption.
Comment 76 Jack 2020-08-09 19:25:30 UTC
So far, using asan enabled gcc, I have not seen one stack smashing.  Could that imply that some out-of-bound write (trapped by asan) was causing the stack smash?  I'm going to keep trying to debug using the latest creduced version, but that only gives me a stack smash 15-30% of the time.
Comment 77 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-09 19:39:40 UTC
(In reply to Jack from comment #76)
> So far, using asan enabled gcc, I have not seen one stack smashing.  Could
> that imply that some out-of-bound write (trapped by asan) was causing the
> stack smash?

It's not clear if asan detects out-of-bound writes for you. Or everything just works? If it does can you post exact gcc output with a write failure?
Comment 78 Jack 2020-08-09 20:11:15 UTC
Created attachment 653939 [details]
gcc output

You're right - I only see memory leaks, no out-of-bound writes.  Attached is the gcc output from 

#!/bin/bash
export ASAN_OPTIONS=detect_leaks=0:use_odr_indicator=1
x86_64-pc-linux-gnu-g++  -fvisibility-inlines-hidden -Os -pipe -fomit-frame-pointer -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden -DBOOST_ALL_DYN_LINK=1 -DBOOST_ALL_NO_LIB=1 
-DNDEBUG -c -o instantiate_predef_macros.o instantiate_predef_macros.pp.cpp > gcc_out.txt 2>&1
grep "stack smashing detected" gcc_out.txt || echo OK

where instantiate_predef_macros.pp.cpp is the first part of my previous attachment.
Without setting the ASAN_OPTIONS, I get exactly the same, but without the memory leak detections at the end.
Comment 79 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-09 20:30:27 UTC
(In reply to Jack from comment #78)
> Created attachment 653939 [details]
> gcc output
> 
> You're right - I only see memory leaks, no out-of-bound writes.  Attached is
> the gcc output from 

Yeah, that means gcc-asan generated code deifferent enough not to exhibit the failure. We would have to continue with standard gcc build.
Comment 80 Jack 2020-08-20 17:35:09 UTC
I haven't given up, but I'm pretty close.  I've generated several additional initramfs files with genkernel, and not seen the error again.  I can no longer generate the stack smash on the original file.  The last output from creduce only fails 2 to 5 times out of 100, and I have not yet gotten a stack smash while in gdb.  Since it was never 100% reproducible, I'll guess that some other package update (other than gcc) has changed the environment during compilation to decrease the probability of the failure.  There are plenty of syntax errors in the gcc output, but I have not checked if they are consistent or not, and in any case they would be errors in boost, not gcc.
Comment 81 Larry the Git Cow gentoo-dev 2020-08-23 09:14:37 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=9bba1f72a9210743fddf664b716b5cf288132922

commit 9bba1f72a9210743fddf664b716b5cf288132922
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-08-23 09:11:31 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-08-23 09:11:31 +0000

    10.2.0: cut 2 patchset
    
    Four new patches:
    + 33_all_lto-O0-mix-ICE-ipa-PR96291.patch: fix -O0 crash for ipa/lto
    + 34_all_fundecl-ICE-PR95820.patch: fix ICE on invalid templates
    + 35_all_ipa-fix-bit-CP.patch: fix bad code generation in ipa bit constprop
    + 36_all_ipa-fix-bit-CP-p2.patch: part 2 of previous patch
    
    Bug: https://bugs.gentoo.org/733482
    Bug: https://gcc.gnu.org/PR96291
    Bug: https://bugs.gentoo.org/730406
    Bug: https://gcc.gnu.org/PR95820
    Bug: https://bugs.gentoo.org/736685
    Bug: https://gcc.gnu.org/PR96482
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 10.2.0/gentoo/README.history | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 82 Larry the Git Cow gentoo-dev 2020-08-23 09:23:32 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=653d6bf6ea15bdf2db033e3099913bac47f5b0e0

commit 653d6bf6ea15bdf2db033e3099913bac47f5b0e0
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-08-23 09:17:12 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-08-23 09:17:12 +0000

    sys-devel/gcc:  cut 2 patchset
    
    Four new patches:
    + 33_all_lto-O0-mix-ICE-ipa-PR96291.patch: fix -O0 crash for ipa/lto
    + 34_all_fundecl-ICE-PR95820.patch: fix ICE on invalid templates
    + 35_all_ipa-fix-bit-CP.patch: fix bad code generation in ipa bit constprop
    + 36_all_ipa-fix-bit-CP-p2.patch: part 2 of previous patch
    
    Closes: https://bugs.gentoo.org/733482
    Bug: https://gcc.gnu.org/PR96291
    Closes: https://bugs.gentoo.org/730406
    Bug: https://gcc.gnu.org/PR95820
    Closes: https://bugs.gentoo.org/736685
    Bug: https://gcc.gnu.org/PR96482
    Package-Manager: Portage-3.0.4, Repoman-3.0.1
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-devel/gcc/Manifest             |  1 +
 sys-devel/gcc/gcc-10.2.0-r1.ebuild | 13 +++++++++++++
 2 files changed, 14 insertions(+)
Comment 83 cJ 2020-08-25 17:57:06 UTC
I'm still encountering a gcc stack smashing on boost with this new gcc version, but only on AMD systems, and it doesn't happen all the time.

Currently working on reducing the source that reproduces the problem.
Comment 84 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-25 22:15:11 UTC
(In reply to cJ from comment #83)
> I'm still encountering a gcc stack smashing on boost with this new gcc
> version, but only on AMD systems, and it doesn't happen all the time.

Oh, I closed the bug by accident. We only fixed ICE on invalid code here. You are seeing stack smash with gcc-10.2.0-r1, right?

> Currently working on reducing the source that reproduces the problem.

Thank you!
Comment 85 cJ 2020-08-26 13:00:24 UTC
Created attachment 656910 [details]
reduced boost 1.74 xml_grammar file

Attaching current reduction code, don't know if it's exploitable yet. It's still running... bear in mind that there is a lot of CPU days behind this.
Comment 86 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-26 21:57:35 UTC
(In reply to cJ from comment #85)
> Created attachment 656910 [details]
> reduced boost 1.74 xml_grammar file
> 
> Attaching current reduction code, don't know if it's exploitable yet.

To try to to reproduce crash locally I'll need more details:

1. exact reduction script including compiler parameters used
2. 'emerge --info' output
Comment 87 cJ 2020-08-27 01:31:19 UTC
Created attachment 656980 [details]
testcase

Attaching testcase that I used; it runs compilation with a simple
 ${CXX} -o -c ${x}.o ${x}.ii

But with several attempts pipelined and distributed on remote hosts.

Sample output:

20200826T212900 __main__ INFO Attempt 0
20200826T212901 __main__ INFO Attempt 1
20200826T212901 __main__ INFO Attempt 2
20200826T212901 __main__ INFO Attempt 3
20200826T212902 __main__ INFO Attempt 4
20200826T212902 __main__ INFO Attempt 5
20200826T212902 __main__ INFO Attempt 6
20200826T212902 __main__ INFO Attempt 7
20200826T212902 __main__ INFO Attempt 8
20200826T212902 __main__ INFO Attempt 9
20200826T212902 __main__ INFO Interesting: 2020-08-26T21:29:00.678114 on MacGyver
Comment 88 cJ 2020-08-27 01:47:39 UTC
The emerge --info is not very useful with the configuration that I have (per-package options), so I give you what (I think) may be relevant:

- Epyc (3000 or 7000) with ECC RAM, no EDAC issues found
- Linux-5.8.{0-2}
- all systems were @system up-to-date on ~amd64 as of 2 days ago (glibc 2.32 dev-libs/gmp-6.2.0-r1 dev-libs/mpfr-4.1.0 dev-libs/mpc-1.2.0) except that I was using binutils 2.33.1
- CFLAGS = -pipe -march=znver1 -mno-3dnow -mno-lwp -mno-fma4 -mno-xop -mno-tbm -mno-hle -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mtune=znver1 -O2
- gcc USE = abi_x86_64 amd64 cxx elibc_glibc fortran kernel_linux multilib nls nptl openmp pch pie sanitize ssp userland_GNU vtv

I also forgot to put LC_LANG=C in the tests (but adding it still reproduces the problem) and I didn't disable ASLR.
Comment 89 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-27 22:15:02 UTC
If the stack smash is reasonably reproducible try to extract the backtrace where stack smash happens: https://wiki.gentoo.org/wiki/Stack-smashing-debugging-guide
Comment 90 Thomas Deutschmann gentoo-dev Security 2020-09-05 22:23:34 UTC
Count me in, I just got

> *** stack smashing detected ***: terminated

while building boost via genkernel on a

> processor       : 31
> vendor_id       : AuthenticAMD
> cpu family      : 23
> model           : 8
> model name      : AMD Ryzen Threadripper 2950X 16-Core Processor
> stepping        : 2
> microcode       : 0x800820d

for the first time myself :(
Comment 91 Thomas Deutschmann gentoo-dev Security 2020-09-05 22:26:17 UTC
> gcc.compile.c++ bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/link-static/pch-off/threading-multi/visibility-hidden/xml_wgrammar.o
> 
>     "x86_64-pc-linux-gnu-g++"   -fvisibility-inlines-hidden -Os -pipe -fomit-frame-pointer -I/var/tmp/genkernel/gk_F0bcQHTT/boost/buildroot/usr/include -std=c++14 -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden -ftemplate-depth-255 -fvisibility=hidden -fvisibility-inlines-hidden -DBOOST_ALL_NO_LIB=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/link-static/pch-off/threading-multi/visibility-hidden/xml_wgrammar.o" "libs/serialization/src/xml_wgrammar.cpp"
> 
> *** stack smashing detected ***: terminated
> In file included from libs/serialization/src/xml_wgrammar.cpp:146:
> libs/serialization/src/basic_xml_grammar.ipp: In constructor 'boost::archive::basic_xml_grammar<CharType>::basic_xml_grammar() [with CharType = wchar_t]':
> libs/serialization/src/basic_xml_grammar.ipp:364:9: internal compiler error: Aborted
>   360 |         str_p(BOOST_ARCHIVE_XML_CLASS_NAME())
>       |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   361 |         >> Eq
>       |         ~~~~~
>   362 |         >> L'"'
>       |         ~~~~~~~
>   363 |         >> ClassName
>       |         ~~~~~~~~~~~~
>   364 |         >> L'"'
>       |         ^~~~~~~
> 0xc77b4f crash_signal
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/toplev.c:328
> 0x7f1ae543837f ???
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
> 0x7f1ae5438301 __GI_raise
> 	../sysdeps/unix/sysv/linux/raise.c:50
> 0x7f1ae5421535 __GI_abort
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/stdlib/abort.c:79
> 0x7f1ae547b1d6 __libc_message
> 	../sysdeps/posix/libc_fatal.c:155
> 0x7f1ae550cbf1 __GI___fortify_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/fortify_fail.c:26
> 0x7f1ae550cbcf __stack_chk_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/stack_chk_fail.c:24
> 0x65a941 cp_gimplify_expr(tree_node**, gimple**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/cp/cp-gimplify.c:955
> 0x9f5d6b gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13465
> 0x9ff060 gimplify_target_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6744
> 0x9f70bf gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13952
> 0x9fc3c0 gimplify_addr_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6171
> 0x9f6cde gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13603
> 0x9fc7e4 gimplify_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:14603
> 0x9fd165 gimplify_call_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:3497
> 0x9f70e2 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13528
> 0x9fbc20 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> 0x9fbc20 gimplify_cleanup_point_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6567
> 0x9f7073 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13948
> 0x9f6f05 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> Please submit a full bug report,
> with preprocessed source if appropriate.
> Please include the complete backtrace with any bug report.
> See <https://bugs.gentoo.org/> for instructions.
> gcc.compile.c++ bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden/xml_grammar.o
> 
>     "x86_64-pc-linux-gnu-g++"   -fvisibility-inlines-hidden -Os -pipe -fomit-frame-pointer -I/var/tmp/genkernel/gk_F0bcQHTT/boost/buildroot/usr/include -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden -ftemplate-depth-255 -fvisibility=hidden -fvisibility-inlines-hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_SERIALIZATION_DYN_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden/xml_grammar.o" "libs/serialization/src/xml_grammar.cpp"
> 
> *** stack smashing detected ***: terminated
> In file included from libs/serialization/src/xml_grammar.cpp:64:
> libs/serialization/src/basic_xml_grammar.ipp: In constructor 'boost::archive::basic_xml_grammar<CharType>::basic_xml_grammar() [with CharType = char]':
> libs/serialization/src/basic_xml_grammar.ipp:328:9: internal compiler error: Aborted
>   325 |         str_p(BOOST_ARCHIVE_XML_CLASS_ID()) >> NameTail
>       |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   326 |         >> Eq
>       |         ~~~~~
>   327 |         >> L'"'
>       |         ~~~~~~~
>   328 |         >> int_p [xml::assign_object(rv.class_id)]
>       |         ^~~~~~~~~~
> 0xc77b4f crash_signal
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/toplev.c:328
> 0x7fa532c2b37f ???
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
> 0x7fa532c2b301 __GI_raise
> 	../sysdeps/unix/sysv/linux/raise.c:50
> 0x7fa532c14535 __GI_abort
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/stdlib/abort.c:79
> 0x7fa532c6e1d6 __libc_message
> 	../sysdeps/posix/libc_fatal.c:155
> 0x7fa532cffbf1 __GI___fortify_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/fortify_fail.c:26
> 0x7fa532cffbcf __stack_chk_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/stack_chk_fail.c:24
> 0x65a941 cp_gimplify_expr(tree_node**, gimple**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/cp/cp-gimplify.c:955
> 0x9f5d6b gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13465
> 0x9ff060 gimplify_target_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6744
> 0x9f70bf gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13952
> 0x9fc3c0 gimplify_addr_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6171
> 0x9f6cde gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13603
> 0x9fc7e4 gimplify_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:14603
> 0x65a720 cp_gimplify_expr(tree_node**, gimple**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/cp/cp-gimplify.c:891
> 0x9f5d6b gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13465
> 0xa02403 gimplify_modify_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:5766
> 0x9f6bb3 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13556
> 0x9ff060 gimplify_target_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6744
> 0x9f70bf gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13952
> Please submit a full bug report,
> with preprocessed source if appropriate.
> Please include the complete backtrace with any bug report.
> See <https://bugs.gentoo.org/> for instructions.
> ...skipped <pbin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden>libboost_serialization.so.1.73.0 for lack of <pbin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden>xml_grammar.o...
> ...skipped <p/var/tmp/genkernel/gk_F0bcQHTT/boost/boost_1_73_0/stage/lib>libboost_serialization.so.1.73.0 for lack of <pbin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden>libboost_serialization.so.1.73.0...
> ...skipped <p/var/tmp/genkernel/gk_F0bcQHTT/boost/boost_1_73_0/stage/lib>libboost_serialization.so.1 for lack of <p/var/tmp/genkernel/gk_F0bcQHTT/boost/boost_1_73_0/stage/lib>libboost_serialization.so.1.73.0...
> gcc.compile.c++ bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden/xml_wgrammar.o
> 
>     "x86_64-pc-linux-gnu-g++"   -fvisibility-inlines-hidden -Os -pipe -fomit-frame-pointer -I/var/tmp/genkernel/gk_F0bcQHTT/boost/buildroot/usr/include -std=c++14 -fPIC -m64 -pthread -finline-functions -Wno-inline -Wall -fvisibility=hidden -ftemplate-depth-255 -fvisibility=hidden -fvisibility-inlines-hidden -DBOOST_ALL_NO_LIB=1 -DBOOST_SERIALIZATION_DYN_LINK=1 -DNDEBUG  -I"." -c -o "bin.v2/libs/serialization/build/gcc-10.2/gentoorelease/pch-off/threading-multi/visibility-hidden/xml_wgrammar.o" "libs/serialization/src/xml_wgrammar.cpp"
> 
> *** stack smashing detected ***: terminated
> In file included from ./boost/spirit/home/classic/core/non_terminal/rule.hpp:33,
>                  from ./boost/spirit/include/classic_rule.hpp:11,
>                  from ./boost/archive/impl/basic_xml_grammar.hpp:53,
>                  from libs/serialization/src/xml_wgrammar.cpp:19:
> ./boost/spirit/home/classic/core/non_terminal/impl/rule.ipp: In constructor 'boost::spirit::classic::impl::concrete_parser<ParserT, ScannerT, AttrT>::concrete_parser(const ParserT&) [with ParserT = boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::sequence<boost::spirit::classic::strlit<const char*>, boost::spirit::classic::rule<boost::spirit::classic::scanner<__gnu_cxx::__normal_iterator<wchar_t*, std::__cxx11::basic_string<wchar_t> >, boost::spirit::classic::scanner_policies<> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t> >, boost::spirit::classic::chlit<wchar_t> >, boost::spirit::classic::action<boost::spirit::classic::uint_parser<unsigned int>, boost::archive::xml::assign_impl<unsigned int> > >, boost::spirit::classic::chlit<wchar_t> >; ScannerT = boost::spirit::classic::scanner<__gnu_cxx::__normal_iterator<wchar_t*, std::__cxx11::basic_string<wchar_t> >, boost::spirit::classic::scanner_policies<> >; AttrT = boost::spirit::classic::nil_t]':
> ./boost/spirit/home/classic/core/non_terminal/impl/rule.ipp:235:54: internal compiler error: Aborted
>   235 |             concrete_parser(ParserT const& p_) : p(p_) {}
>       |                                                      ^
> 0xc77b4f crash_signal
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/toplev.c:328
> 0x7f318499137f ???
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
> 0x7f3184991301 __GI_raise
> 	../sysdeps/unix/sysv/linux/raise.c:50
> 0x7f318497a535 __GI_abort
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/stdlib/abort.c:79
> 0x7f31849d41d6 __libc_message
> 	../sysdeps/posix/libc_fatal.c:155
> 0x7f3184a65bf1 __GI___fortify_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/fortify_fail.c:26
> 0x7f3184a65bcf __stack_chk_fail
> 	/var/tmp/portage/sys-libs/glibc-2.32-r1/work/glibc-2.32/debug/stack_chk_fail.c:24
> 0x65a941 cp_gimplify_expr(tree_node**, gimple**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/cp/cp-gimplify.c:955
> 0x9f5d6b gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13465
> 0x9fbc20 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> 0x9fbc20 gimplify_cleanup_point_expr
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6567
> 0x9f7073 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13948
> 0x9f6f05 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> 0x9f6f05 gimplify_statement_list
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:1869
> 0x9f6f05 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:14000
> 0x9f6ab1 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> 0x9f6ab1 gimplify_and_add(tree_node*, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:486
> 0x9f6ab1 gimplify_expr(tree_node**, gimple**, gimple**, bool (*)(tree_node*), int)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:13907
> 0x9f6f05 gimplify_stmt(tree_node**, gimple**)
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:6825
> 0x9f6f05 gimplify_statement_list
> 	/var/tmp/portage/sys-devel/gcc-10.2.0-r1/work/gcc-10.2.0/gcc/gimplify.c:1869
> Please submit a full bug report,
> with preprocessed source if appropriate.
> Please include the complete backtrace with any bug report.
> See <https://bugs.gentoo.org/> for instructions.
>
Comment 92 Jack 2020-09-05 23:22:58 UTC
Now see if you can extract the exact command and source file that triggered the stack smash.  See comment 36 for creating a fully standalone example, and then comment 8 (and others) for using creduce to create a minimal example.  I'd also be curious how reproducible the error is.  I finally gave up when I could only get the stack smash one or two times out of a hundred tries.
Comment 93 Thomas Deutschmann gentoo-dev Security 2020-09-05 23:53:02 UTC
It already passed on second attempt.
Comment 94 Jack 2020-09-11 16:23:59 UTC
I now get the stack smash emerging boost - either 1.73.0 or 1.74.0.  gcc 10.2.0-r1.  In a few days, I should have enough time to make a new attempt to see how consistent it is (I've had three or four failures so far) and try a new creduce attempt.
Comment 95 cJ 2020-09-11 16:42:57 UTC
Sorry guys I'm catching up on other stuff right now.
Comment 96 Jack 2020-09-16 22:14:15 UTC
The emerge of boost failed repeatedly for me last week.  After some days away, I extracted the failing command, and created a preprocessed file with gcc -E -P.  That version had the stack smash 50-60% of tries.  Emerging creduce also pulled in compiler-rt and compiler-rt-sanitizers 9.01.  (10.0.0 were already installed).  After that, I have not had a single stack smash.  Given the compile command uses x86_64-pc-linux-gnu-g++ I have no idea why that would make any difference.  When the creduce emerge is completely done, I'll quickpkg the newly re-emerge slotted packages and see if the stack smashing returns.  

Does this make any sense to anyone?
Comment 97 Sergei Trofimovich (RETIRED) gentoo-dev 2020-09-16 22:37:34 UTC
(In reply to Jack from comment #96)
> The emerge of boost failed repeatedly for me last week.  After some days
> away, I extracted the failing command, and created a preprocessed file with
> gcc -E -P.  That version had the stack smash 50-60% of tries.  Emerging
> creduce also pulled in compiler-rt and compiler-rt-sanitizers 9.01.  (10.0.0
> were already installed).  After that, I have not had a single stack smash. 
> Given the compile command uses x86_64-pc-linux-gnu-g++ I have no idea why
> that would make any difference.  When the creduce emerge is completely done,
> I'll quickpkg the newly re-emerge slotted packages and see if the stack
> smashing returns.  
> 
> Does this make any sense to anyone?

llvm install should not affect gcc install.
Comment 98 Jack 2020-09-21 18:32:24 UTC
It wasn't llvm, it was just the irreproducibility showing up again.  I did another creduce, and ended up with a 24 line file.  On runs of 100, I generally get 60-80 stack smashes.  However, I've had times when I got 0 (that's ZERO) smashes in 100 runs, three or four times in a row.  Is it conceivably possible it depends on which core it runs on?  I wouldn't think so, but trying to go further is no fun with the failure that inconsistent.
Comment 99 Sergei Trofimovich (RETIRED) gentoo-dev 2020-09-21 18:51:29 UTC
(In reply to Jack from comment #98)
> It wasn't llvm, it was just the irreproducibility showing up again.  I did
> another creduce, and ended up with a 24 line file.  On runs of 100, I
> generally get 60-80 stack smashes.  However, I've had times when I got 0
> (that's ZERO) smashes in 100 runs, three or four times in a row.  Is it
> conceivably possible it depends on which core it runs on?  I wouldn't think
> so, but trying to go further is no fun with the failure that inconsistent.

Core might be it. Or a physical memory layout. Cores could be tested by pinning a binary to specific cpu, say with taskset:
    $ taskset -c 0 ./the-command
to run on cpu0.
Comment 100 Jack 2020-11-28 21:31:36 UTC
I'm wondering if this should just be closed as "worksforme" since I've since done several clean emerges of boost 1.73, and now one of 1.74, and also multiple successful runs of genkernel.  Using the latest result from creduce, run in a loop of 100 tries, I generally get 0 stack smashes, and I don't think I've had more than about 30% in over a month.  I've also not identified any pattern of when I get no failures and when I do get 20-30%.
Comment 101 Sergei Trofimovich (RETIRED) gentoo-dev 2020-11-28 21:56:05 UTC
In bug #724314 i got access to a system where it is somewhat reliably reproducible. Let's close it as a dupe of that bug.

*** This bug has been marked as a duplicate of bug 724314 ***
Comment 102 Larry the Git Cow gentoo-dev 2020-12-29 09:55:18 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=7adab39a82fd07085f600603bdc5c440aa1c142a

commit 7adab39a82fd07085f600603bdc5c440aa1c142a
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-12-29 09:51:52 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-12-29 09:51:52 +0000

    10.2.0: revert PR95820 backporting
    
    The backport breaks parsing as seen in https://gcc.gnu.org/PR98441
    
    Bug: https://gcc.gnu.org/PR95820
    Bug: https://bugs.gentoo.org/730406
    Reported-by: Daniel Santos
    Bug: https://gcc.gnu.org/PR98441
    Bug: https://bugs.gentoo.org/762382
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 10.2.0/gentoo/34_all_fundecl-ICE-PR95820.patch | 25 -------------------------
 10.2.0/gentoo/README.history                   |  3 +++
 2 files changed, 3 insertions(+), 25 deletions(-)
Comment 103 Larry the Git Cow gentoo-dev 2020-12-29 10:07:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=3779ee05e8c035d435c62198f3b761516e63fdf0

commit 3779ee05e8c035d435c62198f3b761516e63fdf0
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-12-29 10:05:50 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-12-29 10:05:50 +0000

    10.2.0: cut 6 patchset
    
    Single dropped patch:
    - 34_all_fundecl-ICE-PR95820.patch: revert PR95820 backporting
    
    Bug: https://gcc.gnu.org/PR95820
    Bug: https://bugs.gentoo.org/730406
    Reported-by: Daniel Santos
    Bug: https://gcc.gnu.org/PR98441
    Bug: https://bugs.gentoo.org/762382
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 10.2.0/gentoo/README.history | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 104 Larry the Git Cow gentoo-dev 2020-12-29 10:14:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8667621e1a1fb7684b9b8d112255f33f8d5ec978

commit 8667621e1a1fb7684b9b8d112255f33f8d5ec978
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-12-29 10:11:31 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-12-29 10:14:44 +0000

    sys-devel/gcc: 10.2.0: cut 6 patchset
    
    Single dropped patch:
    - 34_all_fundecl-ICE-PR95820.patch: revert PR95820 backporting
    
    Bug: https://gcc.gnu.org/PR95820
    Bug: https://bugs.gentoo.org/730406
    Reported-by: Daniel Santos
    Bug: https://gcc.gnu.org/PR98441
    Closes: https://bugs.gentoo.org/762382
    Package-Manager: Portage-3.0.12, Repoman-3.0.2
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-devel/gcc/Manifest             |  1 +
 sys-devel/gcc/gcc-10.2.0-r5.ebuild | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)