| Summary: | dev-python/libvirt-python-9.3.0 fails to build with lto (parallel build issue) | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | CaptainBlood <luc_pierard_de_maujouy> |
| Component: | Current packages | Assignee: | Matthias Maier <tamiko> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | eschwartz, floppym, kocelfc, python, virtualization |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: |
https://bugs.gentoo.org/show_bug.cgi?id=660754 https://bugs.gentoo.org/show_bug.cgi?id=856835 https://github.com/pypa/setuptools/issues/3942 |
||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 351559 | ||
| Attachments: |
faulty with lto activated
emerge --info |
||
Provide emerge --info and change the status to UNCONFIRMED. Created attachment 863109 [details]
emerge --info
From the log:
> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> ...
> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> lto1: error: build/temp.linux-x86_64-cpython-310/libvirt-utils.o: file too short
It looks like libvirt-utils.o is being built twice, possibly in parallel.
Does compiling with MAKEOPTS="-j1" help?
I believe it does help, yeah. I mean, I am able to reproduce (even though I get a slightly different error, still LTO related though and given that an .o file is overwritten whilst another process is using it we may see a wide variety of errors), and MAKEOPTS="-j1" solves it. I'm not sure how to solve it though. Apparently, setuptools is the default when it comes to building python modules (so one might argue that the bug is de facto in setuptools and not libvirt-python), but then we have much more advanced build systems than setuptools (e.g. libvirt uses meson, and even autotools solved this problem many years ago). I think it's worth trying to explore our options fixing the bug in setuptools prior to switching libvirt-python away from it. Other packages/python modules are affected too. (In reply to Michal Prívozník from comment #4) > I believe it does help, yeah. I mean, I am able to reproduce (even though I > get a slightly different error, still LTO related though and given that an > .o file is overwritten whilst another process is using it we may see a wide > variety of errors), and MAKEOPTS="-j1" solves it. > > I'm not sure how to solve it though. Apparently, setuptools is the default > when it comes to building python modules (so one might argue that the bug is > de facto in setuptools and not libvirt-python), but then we have much more > advanced build systems than setuptools (e.g. libvirt uses meson, and even > autotools solved this problem many years ago). > > I think it's worth trying to explore our options fixing the bug in > setuptools prior to switching libvirt-python away from it. Other > packages/python modules are affected too. Right, distutils nor setuptools can't handle parallel builds of C extensions. This is a longstanding problem, unfortunately. We've tried looking into this before for numpy and scipy. There's a bug somewhere that mgorny filed upstream, including some workarounds, but it was kind of hopeless. (I see https://github.com/pypa/setuptools/issues/3119 but it's not the one I'm thinking of.) > Apparently, setuptools is the default when it comes to building python modules (so one might argue that the bug is de facto in setuptools and not libvirt-python)
Those are really two separate points.
- setuptools is asserted the default when it comes to building python modules
- the bug is in setuptools and not libvirt-python
The latter is independently true based on the sheer fact that nothing in libvirt-python's source code is incompatible with LTO such that gcc would write the same file twice. The problem really and truly is just down to the fact that setuptools isn't well suited to parallelizing builds.
As for setuptools being the default... I'm not entirely sure what the argument is here. It's the default therefore everyone should use it? It's the default therefore it's better to fix what has gone unfixed for several decades instead of migrating away from the default?
But it really isn't the default. Setuptools is a series of monkey patches on top of distutils, and distutils is the default... in versions of python less than 3.12. The Python developers don't love this, and the python packaging community doesn't either, so they built standards to get setuptools out of the game of being a gatekeeper in the hope of encouraging competition. By and large, that competition isn't much good, but that's mostly because everyone reinvents how to distribute .py files.
With my upstream meson hat on, I very much do believe that meson is a great option here, because it's a cross platform, cross language, robust build system well suited for C/C++ among other things, that has proper parallelism and LTO support and a lot more features. And it can also build python extensions and package up wheels for you.
It's a well trodden path by now -- SciPy and NumPy have moved over despite maintaining one of the few serious attempts to extend distutils (numpy.distutils) and having been long troubled by the parallelism issue. Parallelism sort of exists, just not very well due to virtually unsolvable limitations in setuptools. Namely, you could parallelize over files in a single .so, but not over multiple .so files, and even doing that was an unpredictable hack that only usually worked.
Despite their best wishes, continuing to monkey patch distutils via setuptools in a kind of elaborate three-project game of monkey where nothing really has a stable internal API, was just unsustainable.
As a data science project, they wanted to get out of the game of being a build system project :P and stick to being active participants in an externally maintained build system that has the features they need. That's meson.
I strongly advise you to do the same. The main advantage of setuptools was always that it was distributed by default (distutils in the stdlib, and setuptools installed by default with ensurepip) and because pip natively understood how to `pip install` an sdist using setup.py. But none of these advantages are true anymore -- distutils is gone in 3.12, so only actual setuptools is supported, and it's not installed by default, only in build venvs if actually requested. And pip natively understands any build backend.
This is also an opportunity to get some excellent integration between components. Libvirt-python's README already talks about how to build against a meson uninstalled build of libvirt -- migrating the python bindings over to meson too, means that you can just use a meson subproject of libvirt, inside libvirt-python, and then simply build libvirt-python and have it automatically grab a copy of libvirt.
By the way, the problem you have encountered here is that you build a few different .so extensions, with the following pattern:
- ""
- "_qemu"
- "_lxc"
Each of them builds an extension named libvirtmod{variant}, using two unique source file "libvirt-{variant/_/-/}override.c" and "build/libvirt{variant/_/-/}.c", and two universal shared files, "typewrappers.c" and "libvirt-utils.c".
For the shared files, setuptools has no concept of their sharedness, so each extension module tries to rebuild those files -- their output files are based on input filename, and don't have a key based on the output extension -- but maybe doesn't if it sees that it already exists (regardless of the current contents). Fortunately, you build each one with the same options, so fully single threaded building is "safe".
Unlike meson (or autotools), setuptools has no concept of shared utility libraries. With meson, you'd build each file exactly once into a static_library(..., build_by_default: false, install: false) and then link each extension to the static library.
Reported here: https://github.com/pypa/setuptools/issues/3942 (In reply to Eli Schwartz from comment #7) > By the way, the problem you have encountered here is that you build a few > different .so extensions, with the following pattern: Yep, this matches my findings exactly (but was too tired to report it yesterday anywhere). So if there's already a precedence of other python modules switching away from setuptools to meson, then maybe I can convince libvirt's community too. (In reply to Michal Prívozník from comment #4) > ... and MAKEOPTS="-j1" solves it. Indeed setting so in /etc/portage/packages.env works fine here. Despite I have no real understanding of possible cause, I just want to remind multiple threaded build works fine with gcc:12 here. (In reply to CaptainBlood from comment #10) > Despite I have no real understanding of possible cause, I just want to > remind multiple threaded build works fine with gcc:12 here. This has nothing to do with the GCC version. I agree, I just had a collision while apologizing about that, sorry. The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=86035d3667a85f92aa5e6596fdd8cf5815e4b47b commit 86035d3667a85f92aa5e6596fdd8cf5815e4b47b Author: Sam James <sam@gentoo.org> AuthorDate: 2023-07-28 08:20:02 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-28 08:20:03 +0000 dev-python/libvirt-python: set -j1 to avoid setuptools breakage May ICE when building w/ GCC LTO or even be miscompiled (even w/o LTO). Bug: https://bugs.gentoo.org/856835 Closes: https://bugs.gentoo.org/907718 Signed-off-by: Sam James <sam@gentoo.org> dev-python/libvirt-python/libvirt-python-9.3.0.ebuild | 5 +++++ dev-python/libvirt-python/libvirt-python-9.4.0.ebuild | 5 +++++ dev-python/libvirt-python/libvirt-python-9.5.0.ebuild | 5 +++++ dev-python/libvirt-python/libvirt-python-9999.ebuild | 5 +++++ 4 files changed, 20 insertions(+) |
Created attachment 863070 [details] faulty with lto activated Whereas 8.2.0, 8.7.0, 8.9.0 could built with lto. 9.3.0 builds fine without lto.