| Summary: | dev-libs/gmp-6.2.1-r2: econf ruins gmp-specific platform autodetection | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Nikita Zlobin <cook60020tmp> |
| Component: | Current packages | Assignee: | Gentoo Toolchain Maintainers <toolchain> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | sam |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=739298 https://bugs.gentoo.org/show_bug.cgi?id=235697 https://bugs.gentoo.org/show_bug.cgi?id=545442 https://bugs.gentoo.org/show_bug.cgi?id=894134 https://bugs.gentoo.org/show_bug.cgi?id=892321 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | zhu3d-4.2.6-r2.ebuild | ||
|
Description
Nikita Zlobin
2022-11-26 17:52:33 UTC
So, I confirm, that it's due to econf, more exactly - at this moment at configure stage:
* econf: updating gmp-6.2.1/config.sub with /usr/share/gnuconfig/config.sub
* econf: updating gmp-6.2.1/config.guess with /usr/share/gnuconfig/config.guess
If trying to grep in source root dir: grep -nFl -e sandy ./*
It will show lines from these files:
./acinclude.m4
./_config.guess
./config.guess
./config.in
./_config.sub
./config.sub
./configure.ac
./configure.wrapped
where ./_config.{sub,guess} is backup, I did before to try 'configure' step.
After restoring backuped files, I can just run: ./config.guess, and it will print value:
$ ./config.guess
sandybridge-pc-linux-gnu
while variants, substituted with econf, print: x86_64-pc-linux-gnu
The reason it forces a "generic" build is because otherwise binpkgs would be unusable. Investigation will be required as to whether the produced build is genuinely generic or not, as this is a bizarre abuse of CHOST (which is typically used for compiler invocations and such). (In reply to Sam James from comment #2) > The reason it forces a "generic" build is because otherwise binpkgs would be > unusable. Investigation will be required as to whether the produced build is > genuinely generic or not, as this is a bizarre abuse of CHOST (which is > typically used for compiler invocations and such). (Wrt genuinely generic: it could well be that either -march makes things work fine, or it does runtime detection instead.) gmp docs recommend configure --host option only for cross-compilation, for own build like in gentoo --build must be enough. But still - config stuff better be kept intact. Of course, I really don't know, is it correct to propose e.g. sandybridge or like that for 32bit - but that could be gmp specifics as well. There could be make.conf variable like GMP_BUILD, similar to OPENBLAS_TARGET from openblas ebuild. > (Wrt genuinely generic: it could well be that either -march makes things work fine, or it does runtime detection instead.)
gmp optimization is not limited by flags, there are specially coded asms, aimed to push best cpu features to their limits.
(In reply to Sam James from comment #2) > The reason it forces a "generic" build is because otherwise binpkgs would be > unusable. Investigation will be required as to whether the produced build is > genuinely generic or not, as this is a bizarre abuse of CHOST (which is > typically used for compiler invocations and such). There is what I learned and consider going fine, as must: First for above: I checked "ebuild --info" and found both CBUILD and CHOST set to "x86_64-pc-linux-gnu", so this part is fine. This explains, why exactly this value is picked by gmp ./configure. In case of gmp it's stated, that while --build specifies machine, used to build, machine to run at is set by --host (no wonder, lone CBUILD will not work). Thus, everything is fine, this part is checked. About MPN_PATH, printed in begining of ./configure output: I determined, that it depends on picked host type, and for ebuild it matches to supplied value "x86_64-pc-linux-gnu". Also nothing to worry about. The only remaining problem is config* files substitution, as it makes originally valid configurations, specific not only to ABI, but also to CPU targets (nehalem, sandybridge, skylike, etc) not be recognized by configure system and be treated with error, causing the very configuration to fail. Just one interesting part - when configuring for 32bit, it picks "i686-pc-linux-gnu". I guess - this is due to config* stuff substitution, as ./configure itself has special option for proper ABI= (either 32 or 64), which doesn't affect host type. E.g. if I run: "./configure ABI=32 --build=x86_64-pc-linux-gnu", I get this summary: Version: GNU MP 6.2.1 Host type: x86_64-pc-linux-gnu ABI: 32 Install prefix: /usr/local Compiler: gcc Static libraries: yes Shared libraries: yes Of, forgot - it would be great to see difference between using i686 and x86_64 used for 32bit ABI. Also
Watch for MPN_PATH and resulting CFLAGS.
Unrelated noise is intentionally omited.
./configure ABI=32 --build=i686-pc-linux-gnu
using ABI="32"
CC="gcc"
CFLAGS="-m32 -O2 -pedantic -fomit-frame-pointer -mtune=pentiumpro -march=pentiumpro"
CPPFLAGS=""
MPN_PATH=" x86/p6 x86 generic"
./configure ABI=64 --build=i686-pc-linux-gnu
configure.wrapped: error: ABI=64 is not among the following valid choices: 32
./configure ABI=32 --build=x86_64-pc-linux-gnu
using ABI="32"
CC="gcc"
CFLAGS="-m32 -O2 -pedantic -fomit-frame-pointer -mtune=k8 -march=k8"
CPPFLAGS=""
MPN_PATH=" x86/k8 x86/k7/mmx x86/k7 x86/mmx x86 generic"
./configure ABI=64 --build=x86_64-pc-linux-gnu:
using ABI="64"
CC="gcc"
CFLAGS="-O2 -pedantic -fomit-frame-pointer -m64 -mtune=k8 -march=k8"
CPPFLAGS=""
MPN_PATH=" x86_64/k8 x86_64 generic"
./configure ABI=32 --build=sandybridge-pc-linux-gnu
using ABI="32"
CC="gcc"
CFLAGS="-m32 -O2 -pedantic -fomit-frame-pointer -mtune=sandybridge -march=sandybridge"
CPPFLAGS=""
MPN_PATH=" x86/coreisbr x86/p6/sse2 x86/p6/p3mmx x86/p6/mmx x86/p6 x86/mmx x86 generic"
./configure ABI=64 --build=sandybridge-pc-linux-gnu
using ABI="64"
CC="gcc"
CFLAGS="-O2 -pedantic -fomit-frame-pointer -m64 -mtune=sandybridge -march=sandybridge"
CPPFLAGS=""
MPN_PATH=" x86_64/coreisbr x86_64/coreinhm x86_64/core2 x86_64 generic"
Created attachment 838073 [details]
zhu3d-4.2.6-r2.ebuild
Comment on attachment 838073 [details]
zhu3d-4.2.6-r2.ebuild
Wrong thread. Is it possible to remove it from there?
Comment on attachment 838073 [details]
zhu3d-4.2.6-r2.ebuild
This was for different package.
(In reply to Nikita Zlobin from comment #6) > > (Wrt genuinely generic: it could well be that either -march makes things work fine, or it does runtime detection instead.) > > gmp optimization is not limited by flags, there are specially coded asms, > aimed to push best cpu features to their limits. Various projects correctly deduce this based on preprocessor macros, so again that's not ruled out. What I'm saying is: I agree there _may_ be a problem and it should be investigated, but obsessing over the thing upstream deviate from vanilla on may be a red herring. We'll see. Rest assured I get your point. There's no really obsession. I just misunderstood, what exactly happened in my config, so decided to clarify in last two posts :). (In reply to Nikita Zlobin from comment #14) > There's no really obsession. I just misunderstood, what exactly happened in > my config, so decided to clarify in last two posts :). No worries, didn't mean it derogatorily, I just mean it might be that it's not that. Or it might be and then I'll tear my hair out :( But in any case, thanks for raising it. It looks like --enable-fat will give us a truly generic build with lots of extra bits installed for various CPUs. (In reply to Sam James from comment #16) > It looks like --enable-fat will give us a truly generic build with lots of > extra bits installed for various CPUs. uh, forgive me: generic in the sense that it's going to work with binpkgs (instruction set agnostic) but install bonus loadables which can be used at runtime. (In reply to Sam James from comment #17) > (In reply to Sam James from comment #16) > > It looks like --enable-fat will give us a truly generic build with lots of > > extra bits installed for various CPUs. > > uh, forgive me: generic in the sense that it's going to work with binpkgs > (instruction set agnostic) but install bonus loadables which can be used at > runtime. Ugh, fat doesn't work for !x86: > Fat binary, --enable-fat > > Using --enable-fat selects a “fat binary” build on x86, where optimized low >level subroutines are chosen at runtime according to the CPU detected. This >means more code, but gives good performance on all x86 chips. (This option >might become available for more architectures in the future.) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=78b4d4ebbcee12f5ef4c1dfa89f80c98a582db9d commit 78b4d4ebbcee12f5ef4c1dfa89f80c98a582db9d Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-30 07:37:50 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-30 07:48:13 +0000 dev-libs/gmp: pass --enable-fat for x86 (includes more asm) Vanilla (_not_ from an ebuild) gmp will probe CHOST and then deduce its *own* tuple based on the currently used CPU, abusing its own custom config.guess (which our econf overwrites). This value is used to decide which accelerated asm is compiled in (MPN_PATH). In ebuilds, we override and refresh config.guess and friends because it's a common portability problem to have stale versions which don't even recognise e.g. riscv as an arch. But gmp's non-standard use of these files poses a problem for us. We can't simply resort to just not refreshing/changing the files because we need to be able to build binary packages on a system without them being super specific to the machine they were built on with no way of controlling it (we may want to look at a USE flag for this going forward though). For now, we can use --enable-fat which produces a generic libgmp which can load various asm at runtime based on the CPU in use - success! Unfortunately, --enable-fat only does anything on x86 for the time being, so !x86 is still in the same situation as before this commit, but this is progress at least. Thanks goes to Nikita Zlobin <nick87720z@gmail.com> for noticing the problem originally (and being persistent with investigating it). Temporarily unkeyworded until I can test this a bit more. Bug: https://bugs.gentoo.org/883201 Signed-off-by: Sam James <sam@gentoo.org> dev-libs/gmp/gmp-6.2.1-r3.ebuild | 130 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bf286a7973d8b95dbef90f17cc8e2d66c19a47c8 commit bf286a7973d8b95dbef90f17cc8e2d66c19a47c8 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-11-30 08:30:15 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-11-30 08:31:33 +0000 dev-libs/gmp: tie --enable-fat to USE=cpudetection Note that USE=-cpudetection still gives a very generic build rather than one detected at build time to be precisely the machine compiled on (for now?) Bug: https://bugs.gentoo.org/883201 Signed-off-by: Sam James <sam@gentoo.org> dev-libs/gmp/gmp-6.2.1-r3.ebuild | 7 ++++--- dev-libs/gmp/metadata.xml | 23 ++++++++++++----------- profiles/arch/amd64/package.use.mask | 4 ++++ profiles/arch/base/package.use.mask | 4 ++++ profiles/arch/x86/package.use.mask | 4 ++++ 5 files changed, 28 insertions(+), 14 deletions(-) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=121152c38caefd8e94fb8c90cb51102693b6d6a8 commit 121152c38caefd8e94fb8c90cb51102693b6d6a8 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-12-01 03:26:54 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-12-01 03:41:54 +0000 dev-libs/gmp: allow automatic detection of host for USE=-cpudetection Bug: https://bugs.gentoo.org/883201 Closes: https://bugs.gentoo.org/235697 Closes: https://bugs.gentoo.org/545442 Signed-off-by: Sam James <sam@gentoo.org> .../{gmp-6.2.1-r3.ebuild => gmp-6.2.1-r4.ebuild} | 32 +++++++++++++++++++--- dev-libs/gmp/metadata.xml | 13 ++++++++- profiles/arch/amd64/package.use.mask | 4 --- profiles/arch/base/package.use.mask | 4 --- profiles/arch/x86/package.use.mask | 4 --- 5 files changed, 40 insertions(+), 17 deletions(-) Okay, all done, I think: There's now two improvements: 1. on amd64/x86, we benefit from "fat binaries" which lets you still make binpkgs easily, but if you do USE=-cpudetection (cpudetection is default, "fat mode"), you can have it tuned for your CPU. 2. for non-amd64/x86, the default behaviour for them still sucks a bit because there's no fat support, but they get a warning telling them to do USE=-cpudetection if they don't mind no binpkgs. But the point is now we now allow people to get that native detection if they want it. In summary: - amd64/x86 && USE=cpudetection (--enable-fat): *slightly* (100KB?) larger library which contains routines for various subarches, optimised, can use binpkgs as a result - amd64/x86 && USE=-cpudetection: optimised specifically for CPU used to do the building, can't use binpkgs as a result - !amd64 && !x86 && USE=cpudetection: the same build as before all of these changes, generic C implementation, can use binpkgs as a result. Upstream does not support fat builds for these arches for now. - !amd64 && !x86 && USE=-cpudetection: optimised specifically for CPU used to do the building, can't use binpkgs as a result Nice! Did not know about cure from Denis (still had to be used from ebuild). Now we leverage autoconf intrinsics, hope it's stable :) The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=2e58af672b41a624f84eb7fada145ab2f6c2b1b6 commit 2e58af672b41a624f84eb7fada145ab2f6c2b1b6 Author: Sam James <sam@gentoo.org> AuthorDate: 2022-12-06 10:48:07 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2022-12-06 10:52:50 +0000 dev-libs/gmp: further econf fixes * Use gnuconfig.eclass to update configfsf.{guess,sub} before the wrapped versions call them. * Export CBUILD too just in case for the deduced host from upstream's wrapped config.guess (not the same as gnuconfig's at all, see bugs for details). * Save/restore upstream's config.{guess,sub} to make sure that for the second multilib build we still have it available to run. In practice, this may not have mattered too much, as the exported ac_cv_host should've persisted, but it's still worth being (clearly) correct as is the case now. * Move cpudetection warning into pkg_pretend so it's easier to spot and we also don't then repeat it for each ABI. Bug: https://bugs.gentoo.org/235697 Bug: https://bugs.gentoo.org/545442 Bug: https://bugs.gentoo.org/883201 See: 78b4d4ebbcee12f5ef4c1dfa89f80c98a582db9d See: 3fe1b5370c0cc1537d1249e8ec9d2fbcd8619499 See: bf286a7973d8b95dbef90f17cc8e2d66c19a47c8 See: 121152c38caefd8e94fb8c90cb51102693b6d6a8 Signed-off-by: Sam James <sam@gentoo.org> .../{gmp-6.2.1-r4.ebuild => gmp-6.2.1-r5.ebuild} | 32 ++++++++++++++++++---- 1 file changed, 26 insertions(+), 6 deletions(-) I can't see, how this can work. There' some mess. econf runs own update unconditionally, affecting resulting MPN_PATH, but than summary still prints host. Variables like ac_cv_build / ac_cv_host are only in configure.wrapped, but there's no references for this file. Regular configure is not using them. Also: Just discovered, that MPN_PATH may be configured, but that's clearly expert-level. err, I was blinded by endless mpn/generic in the config end, MPN_PATH is as expected. (Note that I did check the final binaries are different both in size and in symbols too.) |
