Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 889948 - dev-libs/gmp: miscompiled with sys-devel/gcc LTO on arm64
Summary: dev-libs/gmp: miscompiled with sys-devel/gcc LTO on arm64
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: ARM64 Linux
: Normal normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: lto 915000
  Show dependency tree
 
Reported: 2023-01-06 14:45 UTC by gluker
Modified: 2024-02-28 22:30 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info (emerge.info,7.05 KB, text/plain)
2023-01-06 14:45 UTC, gluker
Details
portage_all_installed (portage_all_installed,63.48 KB, text/plain)
2023-01-07 16:16 UTC, gluker
Details
liblinear.emerge.log (liblinear.emerge.log,1.42 KB, text/plain)
2023-01-07 16:17 UTC, gluker
Details
cmake.emerge.log (cmake.emerge.log,312.13 KB, text/plain)
2023-01-07 16:18 UTC, gluker
Details

Note You need to log in before you can comment on or make changes to this bug.
Description gluker 2023-01-06 14:45:14 UTC
Created attachment 847690 [details]
emerge --info

I've found the problem related to GCC LTO optimization for GMP for ARM aarch64
under QEMU with ARM Neoverse-N1 virt-4.2 (Oracle Cloud) and ARM virt-7.2 (Apple M1)
both running the latest Gentoo with gmp-6.2.1-r5 compiled by both
gcc version 11.3.1 20221209 and gcc version 12.2.1 20221231.

To reproduce:
1. Compile GMP with LTO optimization -- compiles OK
2. Try to compile iproute2-6.1.0 or cmake-3.25.1 -- compilation hangs during the process

Disabling LTO optimization for GMP solves the problem.
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-01-07 07:12:23 UTC
- Please share the full iproute2 log up until the point it hangs, ditto CMake.
- If you take a fresh stage3 then compile GMP with LTO, is that sufficient to then make building iproute2 or CMake hang?
Comment 2 gluker 2023-01-07 16:16:44 UTC
Created attachment 847840 [details]
portage_all_installed
Comment 3 gluker 2023-01-07 16:17:22 UTC
That's my bug report to gmp-bugs at gmplib.org:

--- report starts ---

Let me try to reproduce the whole process with original GMP source:

1. I've downloaded https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz

2. Exported my CFLAGS with LTO enabled:
export COMMON_FLAGS="-O3 -march=native -mcpu=native -pipe -funroll-loops \
    -fomit-frame-pointer -fforce-addr -ftracer -fno-stack-protector -flto"
export CFLAGS="${COMMON_FLAGS}"
export CXXFLAGS="${COMMON_FLAGS}"

3. Configure, make & install GMP with gcc version 12.2.1 20221231:
./configure --prefix=/usr/ --libdir=/usr/lib64/ --enable-cxx
make -j4
make install

4. Cloned iproute2:
git clone git://git.kernel.org/pub/scm/network/iproute2/iproute2.git

5. Try to compile iproute2 with hang as follows:
    CC       tc_cbq.o
    CC       tc_estimator.o
    ... hangs here ...

6. Commented out the following piece of iproute2 code in tc/tc_estimator.c:
           // if (A/(-log(w)) > time_const)
           //      break;

7. Continued to compile iproute2 and compilation went OK

Same error was reproduced with current https://gmplib.org/repo/gmp-6.2/

Switching off LTO let iproute2 compile OK

--- report ends ---

AFAIK that's not only causing problems with iproute2/cmake but with any other packages (e.g.: dev-libs/liblinear) using some portion of math of GMP lib. 

My current Gentoo is based on stage3-arm64-openrc-20230101T231658Z, all others packages are up to date, please check the attached `portage_all_installed` for all packages with their current versions.

As I retried to recompile iproute2 for collecting all the logs, I've found out
that it started to compile OK. However other packages (liblinear & cmake) hang 
during the compilation.

Please find `liblinear.emerge.log` & `cmake.emerge.log` attached.
Comment 4 gluker 2023-01-07 16:17:42 UTC
Created attachment 847842 [details]
liblinear.emerge.log
Comment 5 gluker 2023-01-07 16:18:04 UTC
Created attachment 847844 [details]
cmake.emerge.log
Comment 6 gluker 2023-01-07 16:32:50 UTC
As I've just researched iproute2 started to compile since I had upgraded
dev-libs/mpfr from 4.1.1_p1 to 4.2.0, however dev-libs/liblinear-243 and 
dev-util/cmake-3.25.1 still hang
Comment 7 gluker 2023-01-09 13:30:02 UTC
As Torbjörn Granlund from gmplib.org answered:

---
Unfortunately, I don't have time to isolate this problem.

I long shot would be to disable assembly in the GMP build (by means of
.../configure --disable-assembly).  The arm64 asm in GMP 6.2.1 followed
the ABI docs of the time, which turned out to cause problems at least in
some environments.
---

However enabling --disable-assembly didn't solve the problem.
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-01-09 13:32:22 UTC
(In reply to Sam James from comment #1)
> - If you take a fresh stage3 then compile GMP with LTO, is that sufficient
> to then make building iproute2 or CMake hang?

What about this? (You would do it in a chroot in /tmp or something.)

Also, please try minimise precisely which CFLAGS cause it. Is it just "-O3 -flto", for example?
Comment 9 gluker 2023-01-09 14:01:26 UTC
Sorry, I was mistaken so further research showed that 
`configure --disable-assembly` solves the problem.
Comment 10 gluker 2023-01-09 14:10:38 UTC
(In reply to Sam James from comment #8)
> (In reply to Sam James from comment #1)
> > - If you take a fresh stage3 then compile GMP with LTO, is that sufficient
> > to then make building iproute2 or CMake hang?
> 
> What about this? (You would do it in a chroot in /tmp or something.)
> 
> Also, please try minimise precisely which CFLAGS cause it. Is it just "-O3
> -flto", for example?

Pure CFLAGS `-O2 -flto` and `-Os -flto` for GMP leads to hang with liblinear's 
compilation, except if GMP is configured with `--disable-assembly`
Comment 12 Larry the Git Cow gentoo-dev 2023-05-07 23:54:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=570a3e436ff2a642f9470809fdc8ef63ce560494

commit 570a3e436ff2a642f9470809fdc8ef63ce560494
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-05-07 23:53:29 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-05-07 23:53:49 +0000

    dev-libs/gmp: strip-flags & filter-lto for miscompilation
    
    Miscompiled on arm64 with LTO at least. strip-flags as well because gmp is so
    brittle (see e.g. the past configure mess).
    
    Bug: https://bugs.gentoo.org/889948
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-libs/gmp/gmp-6.2.1-r2.ebuild | 5 +++++
 dev-libs/gmp/gmp-6.2.1-r5.ebuild | 5 +++++
 2 files changed, 10 insertions(+)
Comment 13 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-05-07 23:55:45 UTC
I think I can hit this with -O2 -flto and -O3 -flto with GCC 12/13/14.

I get an ICE with 'echo hi | aarch64-unknown-linux-gnu-gcc -E -dM -':
```
$ echo hi  | aarch64-unknown-linux-gnu-gcc -E -dM -
<built-in>: internal compiler error: in real_to_decimal_for_mode, at real.cc:1899

0xe036af real_to_decimal_for_mode(char*, real_value const*, unsigned long, unsigned long, int, machine_mode)
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/real.cc:1899
0x80cdeb builtin_define_with_hex_fp_value
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/c-family/c-cppbuiltin.cc:1877
0x80e8bf builtin_define_float_constants
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/c-family/c-cppbuiltin.cc:277
0x810103 c_cpp_builtins(cpp_reader*)
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/c-family/c-cppbuiltin.cc:1245
0x82f853 c_finish_options
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/c-family/c-opts.cc:1489
0x831fb3 c_common_init()
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/c-family/c-opts.cc:1226
0x708243 lang_dependent_init
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/toplev.cc:1820
0x708243 do_compile
        /usr/src/debug/sys-devel/gcc-14.0.0.9999/gcc-14.0.0.9999/gcc/toplev.cc:2115
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.gentoo.org/> for instructions.
```

and 'echo hi | aarch64-unknown-linux-gnu-gfortran -Werror -xf77 -v -c -' hangs for me.
Comment 14 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-05-08 02:23:16 UTC
(In reply to gluker from comment #7)
> As Torbjörn Granlund from gmplib.org answered:
> 

For the record, this was at https://gmplib.org/list-archives/gmp-bugs/2023-January/005223.html.
Comment 15 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-10-01 06:18:00 UTC
(In reply to Sam James from comment #14)
> (In reply to gluker from comment #7)
> > As Torbjörn Granlund from gmplib.org answered:
> > 
> 
> For the record, this was at
> https://gmplib.org/list-archives/gmp-bugs/2023-January/005223.html.

... and my own report was: https://marc.info/?l=gmp-bugs&m=168367093126416&w=2
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-10-01 07:26:33 UTC
The release notes for 6.3.0 mention something about arm64+LTO, but I kept the filter for now in:

commit a879c9754ae3013f31fae0cac7b5d93c078543a7
Author: Sam James <sam@gentoo.org>
Date:   Sun Jul 30 14:49:28 2023 +0100

    dev-libs/gmp: add 6.3.0

    Signed-off-by: Sam James <sam@gentoo.org>

The possible fixes are https://gmplib.org/repo/gmp/rev/9c324044f4b5 or https://gmplib.org/repo/gmp/rev/73d9ef70d14f. But they don't seem to (at a glance anyway) resemble the bisect result Jannik hit in https://bugs.gentoo.org/889948#c11 so I've no idea if this is really fixed or if it just suppresses LTO for the assembly and calls it a day.

In any case, it's fair to say given this bug and some others on the ML that LTO isn't a particularly well supported configuration upstream.
Comment 17 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-02-28 22:30:48 UTC
Let's call this fixed given the above. I'm not going to drop the filter any time soon though.