Created attachment 863842 [details] Gcc build logs In trying to test the CET feature I picked a clean ~amd64 nomultilib stage3, updated it and unmasked the USE and rebuilt (with C{,XX}FLAGS as "-O2 -pipe -march=native"): USE="cet" emerge -uDNav world These are the packages that would be merged, in order: Calculating dependencies... done! Dependency resolution took 1.77 s. [ebuild R ] sys-devel/binutils-2.40-r5:2.40::gentoo USE="cet* nls plugins -doc -gold -gprofng -multitarget -pgo -static-libs -test -vanilla -zstd" 0 KiB [ebuild R ] sys-libs/glibc-2.37-r3:2.2::gentoo USE="cet* multiarch ssp (static-libs) -audit -caps -compile-locales (-crypt) (-custom-cflags) -doc -gd -hash-sysv-compat -headers-only (-multilib) -multilib-bootstrap -nscd -perl -profile (-selinux) (-stack-realign) -suid -systemd -systemtap -test (-vanilla)" 0 KiB [ebuild R ] sys-devel/gcc-13.1.1_p20230527:13::gentoo USE="cet* (cxx) fortran nls nptl openmp (pie) sanitize ssp -ada (-custom-cflags) -d -debug -default-stack-clash-protection -default-znow -doc (-fixed-point) -go -graphite -hardened (-ieee-long-double) -jit (-libssp) -lto -modula2 (-multilib) -objc -objc++ -objc-gc (-pch) -pgo -systemtap -test -valgrind -vanilla -vtv -zstd" 60419 KiB sys-devel/gcc (gcc-12.3.1_p20230609 and gcc-13.1.1_p20230527 tested with same error) fails to compile with (full build log for the later version will be attached): Comparing stages 2 and 3 Bootstrap comparison failure! gcc/crtprec32.o differs gcc/crtprec64.o differs gcc/crtprec80.o differs gcc/crtfastmath.o differs x86_64-pc-linux-gnu/libgcc/_absvdi2.o differs x86_64-pc-linux-gnu/libgcc/_clz.o differs x86_64-pc-linux-gnu/libgcc/_mulvsi3.o differs x86_64-pc-linux-gnu/libgcc/_ctzdi2.o differs x86_64-pc-linux-gnu/libgcc/_ctzsi2.o differs x86_64-pc-linux-gnu/libgcc/_ffssi2.o differs x86_64-pc-linux-gnu/libgcc/_ffsdi2.o differs (...) So I discovered which flags are set by -march=native (on same compiler version) with `gcc -march=native -E -v - </dev/null 2>&1 | grep cc1`. If I unmask custom-cflags and replace native with "-march=alderlake -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -mno-sse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mgfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mno-clzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mmovdir64b -mmovdiri -mno-mwaitx -mno-pconfig -mpku -mno-prefetchwt1 -mprfchw -mptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mserialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mwaitpkg -mno-wbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mhreset -mno-kl -mno-widekl -mavxvnni -mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex --param l1-cache-size=48 --param l1-cache-line-size=64 --param l2-cache-size=36864 -mtune=alderlake" in C{,XX}FLAGS then gcc compiles fine with CET. This might be something about changing native binary output somehow when CET is enabled?
Created attachment 863843 [details] emerge --info
Created attachment 863844 [details] /proc/cpuinfo
(Please don't CC arch teams, they're used for keywording/stabilisation, not general bugs.)
Could you upload a tarball with both the bad files and the new files clearly in separate directories? Please also share the output of: `arch=native; for t in param target; do cmd="gcc -Q -O2 -march=$arch --help=$t"; diff -U0 <(LANG=C $cmd -march=x86-64) <(LANG=C $cmd -march=$arch); done`.
Created attachment 863890 [details] arch=native; for t in param target; do cmd="gcc -Q -O2 -march=$arch --help=$t"; diff -U0 <(LANG=C $cmd -march=x86-64) <(LANG=C $cmd -march=$arch); done
Stage2 and stage3 files can be downloaded in https://mega.nz/file/zVFiADRA#h0k1plkJnOgNENj6SbMSMZE2wWvT3BfFErzUgh-2nKk since it is about 30MB and can not be attached. Thanks for your time, hope it is enough
Thanks. This is going to take me a few days to get the chance to dig into. I'll let you know if I need anything more - thanks also for the prompt responses. I've just grabbed those files so I have them safe now too.
(In reply to Sam James from comment #7) > Thanks. This is going to take me a few days to get the chance to dig into. > I'll let you know if I need anything more - thanks also for the prompt > responses. > > I've just grabbed those files so I have them safe now too. Just as a hint to help you, this compilation was made on a no-multilib profile in amd64 environment, so no 32bit at all :)
Same issue on a fresh install with last week's openrc stage3 and current stable gcc (13.2.1_p20230826) for amd64 architecture Using native I have comparison failures Using alderlake (my specific arch) it goes through I first saw this issue for a much older gcc here: https://forums.gentoo.org/viewtopic-p-7348012.html
There's no way that forum post is the same root problem though ;) Anyway, see bug 915389.
I'm going to add a janky workaround later today where we add some fixed cache size for these CPUs.
Could someone with a hybrid CPU show me: * cat /proc/cpuinfo * lscpu -p Thanks
Created attachment 888274 [details] cpuinfo $ lscpu -p # The following is the parsable format, which can be fed to other # programs. Each different item in every column has an unique ID # starting usually from zero. # CPU,Core,Socket,Node,,L1d,L1i,L2,L3 0,0,0,0,,0,0,0,0 1,0,0,0,,0,0,0,0 2,1,0,0,,4,4,1,0 3,1,0,0,,4,4,1,0 4,2,0,0,,8,8,2,0 5,2,0,0,,8,8,2,0 6,3,0,0,,12,12,3,0 7,3,0,0,,12,12,3,0 8,4,0,0,,16,16,4,0 9,4,0,0,,16,16,4,0 10,5,0,0,,20,20,5,0 11,5,0,0,,20,20,5,0 12,6,0,0,,24,24,6,0 13,6,0,0,,24,24,6,0 14,7,0,0,,28,28,7,0 15,7,0,0,,28,28,7,0 16,8,0,0,,32,32,8,0 17,9,0,0,,33,33,8,0 18,10,0,0,,34,34,8,0 19,11,0,0,,35,35,8,0 20,12,0,0,,36,36,9,0 21,13,0,0,,37,37,9,0 22,14,0,0,,38,38,9,0 23,15,0,0,,39,39,9,0
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4fba38fc35fe2966574dd6bfd68ee82cd354976c commit 4fba38fc35fe2966574dd6bfd68ee82cd354976c Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-23 14:16:38 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-23 14:48:47 +0000 toolchain.eclass: add workaround for hybrid CPUs Hybrid/big.little/PE CPUs may report an inconsistent cache size across cores which can cause GCC's bootstrapping to fail its self-comparison. When CBUILD is amd64 or x86 and -march=native is in CFLAGS, iterate over all cores and record l1-cache-size. If any differ, use the first one we found. Bug: https://gcc.gnu.org/PR111768 Closes: https://bugs.gentoo.org/904426 Closes: https://bugs.gentoo.org/908523 Closes: https://bugs.gentoo.org/915389 Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
Thank you everyone for helping!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f64da62e50ad607de3a95b6a1d0d2ed8ba8bd268 commit f64da62e50ad607de3a95b6a1d0d2ed8ba8bd268 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-24 14:05:00 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-24 14:05:31 +0000 toolchain.eclass: improve hybrid check Thanks to stikonas for debugging on IRC. Bug: https://bugs.gentoo.org/904426 Bug: https://bugs.gentoo.org/908523 Bug: https://bugs.gentoo.org/915389 Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e897f91e8e81b8168e7045db2f6d8dd9ebdb9ddf commit e897f91e8e81b8168e7045db2f6d8dd9ebdb9ddf Author: Sam James <sam@gentoo.org> AuthorDate: 2024-03-24 17:19:46 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-03-24 17:47:08 +0000 toolchain.eclass: abort if hybrid CPU detected w/ -march=native Unfortunately, the previous approach can't work. --param doesn't fully wipe out the previous value added by -march=native, so we still get a failed comparison. Users hitting this should install app-misc/resolve-march-native, run resolve-march-native, and use that in their *FLAGS instead of -march=native - at least for sys-devel/gcc via package.env, if not in make.conf. Therefore, our only real option is to just abort when we detect a problematic situation and tell users what to do. The only other idea I had was to try taskset in src_compile which feels super brittle and not sure it'd even work at all. Thanks to Andrei for testing and debugging with us on IRC & the bug. Bug: https://bugs.gentoo.org/904426 Bug: https://bugs.gentoo.org/908523 Bug: https://bugs.gentoo.org/915389 Bug: https://bugs.gentoo.org/927688 Thanks-to: Andrei Liavonchykau <andreil499@gmail.com> Signed-off-by: Sam James <sam@gentoo.org> eclass/toolchain.eclass | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
(In reply to Larry the Git Cow from comment #17) > The bug has been referenced in the following commit(s): > > https://gitweb.gentoo.org/repo/gentoo.git/commit/ > ?id=e897f91e8e81b8168e7045db2f6d8dd9ebdb9ddf > > commit e897f91e8e81b8168e7045db2f6d8dd9ebdb9ddf > Author: Sam James <sam@gentoo.org> > AuthorDate: 2024-03-24 17:19:46 +0000 > Commit: Sam James <sam@gentoo.org> > CommitDate: 2024-03-24 17:47:08 +0000 > > toolchain.eclass: abort if hybrid CPU detected w/ -march=native > > Unfortunately, the previous approach can't work. --param doesn't fully > wipe out the previous value added by -march=native, so we still get a > failed > comparison. > > Users hitting this should install app-misc/resolve-march-native, run > resolve-march-native, and use that in their *FLAGS instead of > -march=native - > at least for sys-devel/gcc via package.env, if not in make.conf. > > Therefore, our only real option is to just abort when we detect a > problematic > situation and tell users what to do. > > The only other idea I had was to try taskset in src_compile which feels > super > brittle and not sure it'd even work at all. > > Thanks to Andrei for testing and debugging with us on IRC & the bug. > > Bug: https://bugs.gentoo.org/904426 > Bug: https://bugs.gentoo.org/908523 > Bug: https://bugs.gentoo.org/915389 > Bug: https://bugs.gentoo.org/927688 > Thanks-to: Andrei Liavonchykau <andreil499@gmail.com> > Signed-off-by: Sam James <sam@gentoo.org> > > eclass/toolchain.eclass | 11 +++++++++-- > 1 file changed, 9 insertions(+), 2 deletions(-) Sam, I remember having to do the workaround you suggested in the past for gcc to be compiled instead of using native directly and I need to ask: Is then the "native-cflags" needed to apply this workaround? I am not sure but I remember that some CPU flags got filtered without the use flag and it is masked. Can you please check?
See https://bugs.gentoo.org/915389#c42, I think
