| Summary: | dev-lang/python-3.8.0 with -flto - Unknown float word ordering | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Craig Andrews <candrews> |
| Component: | Current packages | Assignee: | Python Gentoo Team <python> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | alecm, candrews, eugene.shalygin, johannes.hirte, matoro_gentoo, mgorny, pascal, sam, steffen |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=724002 https://github.com/autoconf-archive/autoconf-archive/pull/279 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 618550 | ||
| Attachments: |
config.log
build.log python-3.8.0 fat log |
||
|
Description
Craig Andrews
2019-11-13 14:39:28 UTC
Created attachment 595964 [details]
config.log
Created attachment 595966 [details]
build.log
My first guess is that your CFLAGS are triggering this. Please start with a simpler set of flags, (eg. -O2 -pipe) and work up from there. On my system, I had to remove -lto. O3 and graphite seems to be fine. Confirmed, disabling lto works around this issue. The relevant autoconf macro is running grep on an object file, which seems like a recipe for disaster. https://github.com/python/cpython/blob/v3.8.0/m4/ax_c_float_words_bigendian.m4#L52 I believe appending -ffat-lto-objects remedies this bug by adding the fat that -flto optimizes out. Created attachment 596238 [details]
python-3.8.0 fat log
media-libs/x264 uses a similar approach and failes with LTO. Adding -ffat-lto-objects solves the issue as stated in comment #7 The thing is that .o files generated with LTO do not contain the normal program code and therefor grep and strings do not work as intended, using fat-lto-objects leads to normal program code in .o files. Python has a configure option --with-lto. This sets the following flags for compiling with lto: LTOFLAGS="-flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none" (In reply to Johannes Hirte from comment #10) Thanks for pointing that out. I'm not sure how best to respect user CFLAGS/LDFLAGS in the case of -flto. Here are a few options: 1. Unconditionally remove -flto from CFLAGS. Add a 'lto' USE flag to dev-lang/python to toggle the --with-lto configure option. 2. Inspect the user CFLAGS, and add -ffat-lto-objects if -flto is present. 2. Inspect the user CFLAGS, and warn if -flto is present but -ffat-lto-objects is missing. 4. Inspect CFLAGS, and pass --with-lto if -lto is present. (In reply to Michał Górny from comment #12) > 4. Inspect CFLAGS, and pass --with-lto if -lto is present. I generally prefer not to mess with configure flags based on CFLAGS, but that seems reasonable in this case. Just ran into this on ARM. Are there any changes pending for python 3.8 with LTO? Option 4 doesn't work; the configure script does not append -ffat-lto-objects to CFLAGS until after the float endian check. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=337c1cbdae45b58405e96f06077fdd29203e0ad0 commit 337c1cbdae45b58405e96f06077fdd29203e0ad0 Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2020-01-05 23:18:45 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2020-01-05 23:19:48 +0000 dev-lang/python: fix build with -flto in CFLAGS Bug: https://bugs.gentoo.org/700012 Package-Manager: Portage-2.3.84_p2, Repoman-2.3.20_p24 Signed-off-by: Mike Gilbert <floppym@gentoo.org> dev-lang/python/python-3.8.1.ebuild | 7 ++++++- dev-lang/python/python-3.9.0_alpha2.ebuild | 5 +++++ 2 files changed, 11 insertions(+), 1 deletion(-) (In reply to Larry the Git Cow from comment #16) > The bug has been referenced in the following commit(s): > > https://gitweb.gentoo.org/repo/gentoo.git/commit/ > ?id=337c1cbdae45b58405e96f06077fdd29203e0ad0 > > commit 337c1cbdae45b58405e96f06077fdd29203e0ad0 > Author: Mike Gilbert <floppym@gentoo.org> > AuthorDate: 2020-01-05 23:18:45 +0000 > Commit: Mike Gilbert <floppym@gentoo.org> > CommitDate: 2020-01-05 23:19:48 +0000 > > dev-lang/python: fix build with -flto in CFLAGS > > Bug: https://bugs.gentoo.org/700012 > Package-Manager: Portage-2.3.84_p2, Repoman-2.3.20_p24 > Signed-off-by: Mike Gilbert <floppym@gentoo.org> > > dev-lang/python/python-3.8.1.ebuild | 7 ++++++- > dev-lang/python/python-3.9.0_alpha2.ebuild | 5 +++++ > 2 files changed, 11 insertions(+), 1 deletion(-) This only catches '-flto'. If someone is specifying the number of parallel processes like '-flto=8' it doesn't work. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bc22f76af021e82362cb4be63e8f9ba452ac4906 commit bc22f76af021e82362cb4be63e8f9ba452ac4906 Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2020-01-13 17:28:29 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2020-01-13 17:28:29 +0000 dev-lang/python: also match '-flto=*' Bug: https://bugs.gentoo.org/700012 Package-Manager: Portage-2.3.84_p2, Repoman-2.3.20_p24 Signed-off-by: Mike Gilbert <floppym@gentoo.org> dev-lang/python/python-3.8.1.ebuild | 2 +- dev-lang/python/python-3.9.0_alpha2.ebuild | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) I just hit this when using Clang with -flto=thin. Any chance you can catch that as well? (In reply to Calum Knight from comment #19) > I just hit this when using Clang with -flto=thin. Any chance you can catch > that as well? Nevermind, It turns out that, unsurprisingly, you can't have "fat" thin LTO objects so this can't be fixed the same way. I just submitted PGO+LTO patches against bug 61541 I suggest closing this bug in favor of that one. I agree that -flto=thin should be caught. But I really wish Python had a better test. LLD with thin LTO got a speedup of roughly 36 when linking sys-libs/llvm, which is quite significant and I really like keeping thin LTO around. Alright so I've just updated clang to 11, which may be relevant, but -ffat-lto-objects is not supported. % clang -march=native -O3 -pipe -flto -ffat-lto-objects blank.c -c clang-11: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument] Personally, I was confused why the ebuild wasn't appending the argument even though I had a plain old '-flto' argument. So I guess I'm back to hacking up ebuilds. reporting that this is still a thing in version 3.9.2. To WA this, I added the performance and lto options as described here: https://docs.python.org/3/using/configure.html#performance-options The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=074a78fe03e77fe3337dae773c3d46d0af47eb95 commit 074a78fe03e77fe3337dae773c3d46d0af47eb95 Author: Sam James <sam@gentoo.org> AuthorDate: 2021-11-17 10:49:17 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-11-17 10:49:17 +0000 dev-lang/python: backport PGO, LTO to 3.9.x and 3.10.x Bug: https://bugs.gentoo.org/615412 Bug: https://bugs.gentoo.org/700012 Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.0_p1.ebuild | 18 +++++++++++++++++- dev-lang/python/python-3.9.9.ebuild | 18 +++++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b85750dd2326759e7f102d81e5943bd4e5557a8a commit b85750dd2326759e7f102d81e5943bd4e5557a8a Author: Sam James <sam@gentoo.org> AuthorDate: 2021-11-17 10:06:12 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-11-17 10:46:28 +0000 dev-lang/python: add LTO Bug: https://bugs.gentoo.org/615412 Bug: https://bugs.gentoo.org/700012 Signed-off-by: Sam James <sam@gentoo.org> Closes: https://github.com/gentoo/gentoo/pull/22853 Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/metadata.xml | 1 + dev-lang/python/python-3.11.0_alpha2.ebuild | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) I'm going to say this is fixed given that we have USE=lto for enabling LTO 'correctly' now. The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ff65f5c822417e8347598fcb75a1e575ba35bf58 commit ff65f5c822417e8347598fcb75a1e575ba35bf58 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-28 22:40:54 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-28 22:40:54 +0000 dev-lang/python: avoid confused LTO configure test w/ cross CBUILD Python No real need to take care here as it gets thrown away, so just bang in -fno-lto. Bug: https://bugs.gentoo.org/700012 Signed-off-by: Sam James <sam@gentoo.org> dev-lang/python/python-3.10.13.ebuild | 8 ++++++-- dev-lang/python/python-3.11.5.ebuild | 8 ++++++-- dev-lang/python/python-3.12.0_beta4_p2.ebuild | 8 ++++++-- dev-lang/python/python-3.12.0_rc1_p5.ebuild | 8 ++++++-- dev-lang/python/python-3.12.0_rc1_p6.ebuild | 8 ++++++-- dev-lang/python/python-3.8.18.ebuild | 8 ++++++-- dev-lang/python/python-3.9.18.ebuild | 8 ++++++-- 7 files changed, 42 insertions(+), 14 deletions(-) > I'm going to say this is fixed given that we have USE=lto for enabling LTO 'correctly' now. lto USE flag was dropped in the last commit. https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.12.1_p1.ebuild#n32 (In reply to Boris Staletic from comment #28) > > I'm going to say this is fixed given that we have USE=lto for enabling LTO 'correctly' now. > > lto USE flag was dropped in the last commit. > > https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.12. > 1_p1.ebuild#n32 Yes, that's fine. See the commit message which did it and https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.12.1_p1.ebuild#n288. Python since 3.11 supports choosing between full and thin lto: https://github.com/python/cpython/blob/3.11/configure#L1796 Maybe this could be added to the ebuilds? |
