Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 954406 - sys-devel/gcc-15.1.0: upgrade from 14.2.1_p20250419 (Bootstrap comparison failure in gcc/i386.o)
Summary: sys-devel/gcc-15.1.0: upgrade from 14.2.1_p20250419 (Bootstrap comparison fai...
Status: IN_PROGRESS
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: gcc-15
  Show dependency tree
 
Reported: 2025-04-25 22:04 UTC by Karl Ernst Brunk
Modified: 2025-05-02 14:00 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge-info (emerge-info.txt,16.58 KB, text/plain)
2025-04-25 22:05 UTC, Karl Ernst Brunk
Details
actual emerge use-flags (emerge-use-flags.txt,422 bytes, text/plain)
2025-04-25 22:06 UTC, Karl Ernst Brunk
Details
build-log (buld-log.xz,121.31 KB, application/x-xz)
2025-04-25 22:07 UTC, Karl Ernst Brunk
Details
gcc-build-logs (gcc-build-logs.tar.xz,187.05 KB, application/x-xz)
2025-04-25 22:08 UTC, Karl Ernst Brunk
Details
stage2-split-1 (stage2-i386.oaa,731.06 KB, application/x-object)
2025-04-26 12:11 UTC, Karl Ernst Brunk
Details
stage2-split2 (stage2-i386.oab,730.48 KB, application/octet-stream)
2025-04-26 12:12 UTC, Karl Ernst Brunk
Details
stage2-split3 (stage2-i386.oac,730.49 KB, application/octet-stream)
2025-04-26 12:12 UTC, Karl Ernst Brunk
Details
stage2-split-part1 (stage2-i386.oaa,731.06 KB, application/x-object)
2025-04-26 12:13 UTC, Karl Ernst Brunk
Details
stage3-split1 (stage3-i386.oaa,731.04 KB, application/x-object)
2025-04-26 12:14 UTC, Karl Ernst Brunk
Details
stage3-split2 (stage3-i386.oab,730.48 KB, application/octet-stream)
2025-04-26 12:15 UTC, Karl Ernst Brunk
Details
stage3-split3 (stage3-i386.oac,730.49 KB, application/octet-stream)
2025-04-26 12:15 UTC, Karl Ernst Brunk
Details
gcc-bug.sh (file_954406.txt,1.33 KB, text/plain)
2025-04-27 18:09 UTC, Sam James
Details
gcc-bug.sh (file_954406.txt,1.40 KB, text/plain)
2025-04-27 21:47 UTC, Sam James
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Karl Ernst Brunk 2025-04-25 22:04:36 UTC
Error while comparing stage 2 with 3

Reproducible: Always

Steps to Reproduce:
1.emerge -u sys-devl/gcc

Actual Results:  
Error while comparing stage 2 with 3

Expected Results:  
should pass comparision
Comment 1 Karl Ernst Brunk 2025-04-25 22:05:52 UTC
Created attachment 926187 [details]
emerge-info
Comment 2 Karl Ernst Brunk 2025-04-25 22:06:48 UTC
Created attachment 926188 [details]
actual emerge use-flags
Comment 3 Karl Ernst Brunk 2025-04-25 22:07:27 UTC
Created attachment 926189 [details]
build-log
Comment 4 Karl Ernst Brunk 2025-04-25 22:08:05 UTC
Created attachment 926190 [details]
gcc-build-logs
Comment 5 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-25 22:11:12 UTC
Thanks. Could you upload the stage2 and stage3 copies of gcc/i386.o in /var/tmp/portage/sys-devel/gcc-15.1.0/work/build?
Comment 6 Karl Ernst Brunk 2025-04-26 12:11:31 UTC
Created attachment 926212 [details]
stage2-split-1
Comment 7 Karl Ernst Brunk 2025-04-26 12:12:03 UTC
Created attachment 926213 [details]
stage2-split2
Comment 8 Karl Ernst Brunk 2025-04-26 12:12:33 UTC
Created attachment 926214 [details]
stage2-split3
Comment 9 Karl Ernst Brunk 2025-04-26 12:13:52 UTC
Created attachment 926215 [details]
stage2-split-part1
Comment 10 Karl Ernst Brunk 2025-04-26 12:14:41 UTC
Created attachment 926216 [details]
stage3-split1
Comment 11 Karl Ernst Brunk 2025-04-26 12:15:12 UTC
Created attachment 926217 [details]
stage3-split2
Comment 12 Karl Ernst Brunk 2025-04-26 12:15:59 UTC
Created attachment 926218 [details]
stage3-split3
Comment 13 Karl Ernst Brunk 2025-04-26 12:17:57 UTC
Sorry for the delay and the kind of files. Compressing did not do the job to get smaller than 1000K. So concating results in the proper .o's.
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 12:47:23 UTC
Thanks.

This part is normal (-fchecking=1 vs -fno-checking) is supposed to give identical code generation:
```
│ -'-fno-openmp' '-fno-openacc' '-m64' '-march=znver4' '-mtune=znver4' '-O2' '-Wno-error=narrowing' '-fno-checking' '-flto=jobserver' '-frandom-seed=1' '-fcf-protection=full' '-fno-exceptions' '-fasynchronous-unwind-tables' '-fPIE'
│ +'-fno-openmp' '-fno-openacc' '-m64' '-march=znver4' '-mtune=znver4' '-O2' '-Wno-error=narrowing' '-fchecking=1' '-flto=jobserver' '-frandom-seed=1' '-fcf-protection=full' '-fno-exceptions' '-fasynchronous-unwind-tables' '-fPIE'
```

I need to figure out how to use lto-dump for the rest.
Comment 15 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 12:50:46 UTC
OK, running `git diff --no-index --word-diff <(/usr/x86_64-pc-linux-gnu/gcc-bin/15/lto-dump -demangle -list stage2.o -objects) <(/usr/x86_64-pc-linux-gnu/gcc-bin/15/lto-dump -demangle -list stage3.o -objects)`,
```
diff --git a/dev/fd/63 b/dev/fd/62
--- a/dev/fd/63
+++ b/dev/fd/62
@@ -1,5 +1,5 @@

    LTO Object Name: [-stage2.o-]{+stage3.o+}

No.    Offset    Size       Section Name

@@ -8,770 +8,770 @@ No.    Offset    Size       Section Name
 3   331015    17274   .gnu.lto_.icf.1
 4   348289     8930   .gnu.lto_.ipa_sra.1
 5   357219    47260   .gnu.lto_.inline.1
 6   404479    [-34137-]{+34114+}   .gnu.lto_.jmpfuncs.1
 7   [-438616-]{+438593+}     1917   .gnu.lto_.pureconst.1
 8   [-440533-]{+440510+}     9744   .gnu.lto_.ipa_modref.1
 9   [-450277-]{+450254+}        8   .gnu.lto_.lto.1
[...]
```

.gnu.lto_.jmpfuncs.1 seems to be the only section other than the opts one (which is fine) that changed size
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 12:58:20 UTC
While I ask for advice from others, could you try the following:
* let me know if it's reproducible (so re-run emerge -v1 sys-devel/gcc:15, does it fail, and is it the same comparison issue on x86.o (hopefully it is));
* see if you can narrow down which of *FLAGS causes it (e.g. does dropping -fuse-ld=mold help, and so on)
Comment 17 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 13:14:22 UTC
(In reply to Sam James from comment #16)

s/x86.o/i386.o/
Comment 18 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 14:53:22 UTC
amonakov gave the next step and spotted the problem (thank you).

The sections are compressed (by zstd here):
```
$ objcopy -O binary --set-section-flags *=ALLOC -j .gnu.lto_.jmpfuncs.1 stage2.o jmpfuncs.stage2.zstd && zstd -d jmpfuncs.stage2.zstd
$ objcopy -O binary --set-section-flags *=ALLOC -j .gnu.lto_.jmpfuncs.1 stage3.o jmpfuncs.stage3.zstd && zstd -d jmpfuncs.stage3.zstd
```

```
$ diffoscope jmpfuncs jmpfuncs.stage3
--- jmpfuncs
+++ jmpfuncs.stage3
@@ -431,15 +431,15 @@
 00001ae0: 0201 9602 0001 0202 e203 0120 0119 2001  ........... .. .
 00001af0: 1920 0119 2001 0004 0206 c778 012f 0001  . .. ......x./..
 00001b00: 0202 dc03 0120 012f 2001 2f20 012f 2001  ..... ./ ./ ./ .
 00001b10: 0002 0628 0108 0001 0202 ce03 0120 0108  ...(......... ..
 00001b20: 2001 0820 0108 2001 0081 0104 0101 0101   .. .. .........
 00001b30: ff01 7f02 197f 0219 7f02 cb03 7f02 d603  ................
 00001b40: 0202 0196 0200 0102 02e2 0301 2001 1920  ............ ..
-00001b50: 0119 2001 1920 0100 0400 0001 0202 ce03  .. .. ..........
+00001b50: 0119 2001 1920 0100 0400 0001 0202 e603  .. .. ..........
 00001b60: 0120 0100 2001 ffff 0320 0100 2001 ffff  . .. .... .. ...
 00001b70: 0300 0000 0800 0001 0202 dc03 0120 0111  ............. ..
 00001b80: 2001 1220 0100 2001 1304 8101 0101 0000   .. .. .........
 00001b90: 0000 0002 012b 0001 0202 d603 0101 0100  .....+..........
 00001ba0: 0101 0001 0100 0101 0082 0101 0103 7f02  ................
 00001bb0: ce03 0200 0000 8301 0201 010b 0402 1902  ................
 00001bc0: 0219 0603 2801 8608 bb77 0001 0202 a304  ....(....w......
[...]
```

ce03 changes to e603 in a bunch of places. The size difference just comes from the stage2 version compressing better.
Comment 19 Karl Ernst Brunk 2025-04-26 15:39:14 UTC
Hi again,

it's the lto Flag !
So USE="-lto" emerge sys-devel/gcc did do the trick.
I left all the others including mold like in attachment 2 [details].
So maybe some of the other tracked issues have a link to lto also.
Thanks for your help and advice
Karl
Comment 20 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 15:41:50 UTC
Thanks. If you try again w/ USE=lto, does it fail? Just i386.o again if so? Could you upload the files again (use different filenames to distinguish from before) if so too?

I'd like to understand if it's either some corruption (maybe failing memory) or if the changed bits are actually meaningful.
Comment 21 Karl Ernst Brunk 2025-04-26 15:42:48 UTC
It is strange - the automatic produced links of the second attachment in the last comment links to some completely different bug.
I referenced the actual emerge use-flags.
Comment 22 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 15:46:44 UTC
(In reply to Sam James from comment #20)
> Thanks. If you try again w/ USE=lto, does it fail? Just i386.o again if so?
> Could you upload the files again (use different filenames to distinguish
> from before) if so too?
> 
> I'd like to understand if it's either some corruption (maybe failing memory)
> or if the changed bits are actually meaningful.

... amonakov made an interesting suggestion as well given mold is threaded: could you try with LTO and without mold too?
Comment 23 Karl Ernst Brunk 2025-04-26 16:08:32 UTC
Hi Sam,

So compiling gcc-15.1.0 and including again lto with the new gcc-15.1.0(-lto) results in the same.
Now I am not exactly sure what files you want to have.
Just tell me before i upload useless files.

i386.o's from failed compilation with the 15.1.0(-lto)-version ?
Comment 24 Karl Ernst Brunk 2025-04-26 16:10:05 UTC
(In reply to Sam James from comment #22)
> (In reply to Sam James from comment #20)
> > Thanks. If you try again w/ USE=lto, does it fail? Just i386.o again if so?
> > Could you upload the files again (use different filenames to distinguish
> > from before) if so too?
> > 
> > I'd like to understand if it's either some corruption (maybe failing memory)
> > or if the changed bits are actually meaningful.
> 
> ... amonakov made an interesting suggestion as well given mold is threaded:
> could you try with LTO and without mold too?

That was the first thing i did before trying with reduced use-flags.
No without mold was the same error.
Comment 25 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 18:33:02 UTC
I can reproduce..
Comment 26 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-26 18:34:19 UTC
(In reply to Sam James from comment #25)
> I can reproduce..

Used:

(cd /var/db/repos/gentoo/sys-devel/gcc && MAKEOPTS="-j$(nproc) -l$(nproc)" ADAFLAGS="-pipe -march=znver4 -mtune=znver4 -O2" CFLAGS="-pipe -march=znver4 -mtune=znver4 -O2" CXXFLAGS="-pipe -march=znver4 -mtune=znver4 -O2" FFLAGS="-pipe -march=znver4 -mtune=znver4 -O2" FCFLAGS="-pipe -march=znver4 -mtune=znver4 -O2" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--sort-common" USE="-* abi_x86_64 amd64 cet cxx default-stack-clash-protection default-znow elibc_glibc fortran graphite kernel_linux lto nls openmp pie sanitize ssp zstd" ebuild gcc-15.1.0.ebuild clean install)
Comment 27 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-27 12:40:26 UTC
If I use -ggdb3 as well, I get (only) a comparison failure in gimple-match-1.o.

```
@@ -8,136 +8,136 @@ No.    Offset    Size       Section Name
 3   994122     6108   .gnu.lto_.icf.1
 4  1000230     4631   .gnu.lto_.ipa_sra.1
 5  1004861    11060   .gnu.lto_.inline.1
 6  1015921    [-15612-]{+15652+}   .gnu.lto_.jmpfuncs.1
 7  [-1031533-]{+1031573+}      304   .gnu.lto_.pureconst.1
 8  [-1031837     2129-]{+1031877     2128+}   .gnu.lto_.ipa_modref.1
 9  [-1033966-]{+1034005+}        8   .gnu.lto_.lto.1
[...]
132  [-2386021   152067-]{+2386060   152062+}   .gnu.lto_.decls.1
133  [-2538088-]{+2538122+}    41649   .gnu.lto_.symtab.1
134  [-2579737-]{+2579771+}      973   .gnu.lto_.ext_symtab.1
135  [-2580710      242-]{+2580744      241+}   .gnu.lto_.opts
```

The difference between extracted & decompressed .gnu.lto_.jmpfuncs.1 there is much bigger, but maybe some of that is just exaggerated by the large size of gimple-match-1.cc and it's just whatever is happening in i386.o but many more times.

Uploaded gimple-match-1.o from stage2/stage3 at https://dev.gentoo.org/~sam/bugs/gentoo/954406/gimple-match.tar.xz.
Comment 28 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-27 18:09:00 UTC
Created attachment 926359 [details]
gcc-bug.sh

OK, repro'd manually at least which fails on gimple-match-6.o. Progress is slow so far, sorry.
Comment 29 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-27 21:47:49 UTC
Created attachment 926812 [details]
gcc-bug.sh

Alexander mentioned it could be a diff w/ -fchecking=1, so I tried:

[20:24] <sam_> @@ -552,8 +552,8 @@ STAGE2_CONFIGURE_FLAGS += --with-libphobos-druntime-only
[20:24] <sam_>  # stage2 and stage3 we are merely avoid doing redundant work, plus we apply
[20:24] <sam_>  # checking when building all target libraries for release builds.
[20:24] <sam_>  STAGE1_TFLAGS += -fno-checking
[20:24] <sam_> -STAGE2_CFLAGS += -fno-checking
[20:24] <sam_> -STAGE2_TFLAGS += -fno-checking
[20:24] <sam_> +STAGE2_CFLAGS += -fchecking=1
[20:24] <sam_> +STAGE2_TFLAGS += -fchecking=1
[20:24] <sam_>  STAGE3_CFLAGS += -fchecking=1
[20:24] <sam_>  STAGE3_TFLAGS += -fchecking=1

and indeed it compared fine. I'm bisecting now for gimple-match* (as gimple-match-6 consistently fails on this machine & with the script as-attached).

I did look for some obvious gcc_checking_asserts w/ side-effects but didn't spot any yet.
Comment 30 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-28 06:07:42 UTC
Going to bed now but I think there might be another issue, as I can't reproduce one of them on gcc-15.1.0, but can on master.
Comment 31 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2025-04-28 14:30:51 UTC
(In reply to Sam James from comment #30)
> Going to bed now but I think there might be another issue, as I can't
> reproduce one of them on gcc-15.1.0, but can on master.

I've filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119977 for the gimple-match-6.o issue which I can only hit on master. It may well be the same as this but that one at least I could reproduce manually w/o any default changes.