Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 907718 - dev-python/libvirt-python-9.3.0 fails to build with lto (parallel build issue)
Summary: dev-python/libvirt-python-9.3.0 fails to build with lto (parallel build issue)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Matthias Maier
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: parallel-make
  Show dependency tree
 
Reported: 2023-06-02 17:19 UTC by CaptainBlood
Modified: 2023-07-28 08:20 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
faulty with lto activated (dev-python:libvirt-python-9.3.0:20230602-054247.log.gz,3.23 KB, application/gzip)
2023-06-02 17:19 UTC, CaptainBlood
Details
emerge --info (emerge-info-libvirt-python.xz,4.98 KB, application/x-xz)
2023-06-03 12:59 UTC, CaptainBlood
Details

Note You need to log in before you can comment on or make changes to this bug.
Description CaptainBlood 2023-06-02 17:19:43 UTC
Created attachment 863070 [details]
faulty with lto activated

Whereas 8.2.0, 8.7.0, 8.9.0 could built with lto.

9.3.0 builds fine without lto.
Comment 1 Mike Gilbert gentoo-dev 2023-06-03 01:51:52 UTC
Provide emerge --info and change the status to UNCONFIRMED.
Comment 2 CaptainBlood 2023-06-03 12:59:44 UTC
Created attachment 863109 [details]
emerge --info
Comment 3 Mike Gilbert gentoo-dev 2023-06-04 01:50:57 UTC
From the log:

> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> ...
> x86_64-pc-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -march=native -O2 -flto -fuse-linker-plugin -fno-fat-lto-objects -pipe -Wa,-mbranches-within-32B-boundaries -DNDEBUG -fPIC -I. -I/usr/include/python3.10 -c libvirt-utils.c -o build/temp.linux-x86_64-cpython-310/libvirt-utils.o
> lto1: error: build/temp.linux-x86_64-cpython-310/libvirt-utils.o: file too short

It looks like libvirt-utils.o is being built twice, possibly in parallel.

Does compiling with MAKEOPTS="-j1" help?
Comment 4 Michal Prívozník 2023-06-04 19:56:39 UTC
I believe it does help, yeah. I mean, I am able to reproduce (even though I get a slightly different error, still LTO related though and given that an .o file is overwritten whilst another process is using it we may see a wide variety of errors), and MAKEOPTS="-j1" solves it.

I'm not sure how to solve it though. Apparently, setuptools is the default when it comes to building python modules (so one might argue that the bug is de facto in setuptools and not libvirt-python), but then we have much more advanced build systems than setuptools (e.g. libvirt uses meson, and even autotools solved this problem many years ago).

I think it's worth trying to explore our options fixing the bug in setuptools prior to switching libvirt-python away from it. Other packages/python modules are affected too.
Comment 5 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-06-04 19:58:52 UTC
(In reply to Michal Prívozník from comment #4)
> I believe it does help, yeah. I mean, I am able to reproduce (even though I
> get a slightly different error, still LTO related though and given that an
> .o file is overwritten whilst another process is using it we may see a wide
> variety of errors), and MAKEOPTS="-j1" solves it.
> 
> I'm not sure how to solve it though. Apparently, setuptools is the default
> when it comes to building python modules (so one might argue that the bug is
> de facto in setuptools and not libvirt-python), but then we have much more
> advanced build systems than setuptools (e.g. libvirt uses meson, and even
> autotools solved this problem many years ago).
> 
> I think it's worth trying to explore our options fixing the bug in
> setuptools prior to switching libvirt-python away from it. Other
> packages/python modules are affected too.

Right, distutils nor setuptools can't handle parallel builds of C extensions. This is a longstanding problem, unfortunately.

We've tried looking into this before for numpy and scipy. There's a bug somewhere that mgorny filed upstream, including some workarounds, but it was kind of hopeless.

(I see https://github.com/pypa/setuptools/issues/3119 but it's not the one I'm thinking of.)
Comment 6 Eli Schwartz 2023-06-04 20:34:15 UTC
> Apparently, setuptools is the default when it comes to building python modules (so one might argue that the bug is de facto in setuptools and not libvirt-python)

Those are really two separate points.

- setuptools is asserted the default when it comes to building python modules

- the bug is in setuptools and not libvirt-python

The latter is independently true based on the sheer fact that nothing in libvirt-python's source code is incompatible with LTO such that gcc would write the same file twice. The problem really and truly is just down to the fact that setuptools isn't well suited to parallelizing builds.

As for setuptools being the default... I'm not entirely sure what the argument is here. It's the default therefore everyone should use it? It's the default therefore it's better to fix what has gone unfixed for several decades instead of migrating away from the default?

But it really isn't the default. Setuptools is a series of monkey patches on top of distutils, and distutils is the default... in versions of python less than 3.12. The Python developers don't love this, and the python packaging community doesn't either, so they built standards to get setuptools out of the game of being a gatekeeper in the hope of encouraging competition. By and large, that competition isn't much good, but that's mostly because everyone reinvents how to distribute .py files. 

With my upstream meson hat on, I very much do believe that meson is a great option here, because it's a cross platform, cross language, robust build system well suited for C/C++ among other things, that has proper parallelism and LTO support and a lot more features. And it can also build python extensions and package up wheels for you.

It's a well trodden path by now -- SciPy and NumPy have moved over despite maintaining one of the few serious attempts to extend distutils (numpy.distutils) and having been long troubled by the parallelism issue. Parallelism sort of exists, just not very well due to virtually unsolvable limitations in setuptools. Namely, you could parallelize over files in a single .so, but not over multiple .so files, and even doing that was an unpredictable hack that only usually worked.

Despite their best wishes, continuing to monkey patch distutils via setuptools in a kind of elaborate three-project game of monkey where nothing really has a stable internal API, was just unsustainable.

As a data science project, they wanted to get out of the game of being a build system project :P and stick to being active participants in an externally maintained build system that has the features they need. That's meson.

I strongly advise you to do the same. The main advantage of setuptools was always that it was distributed by default (distutils in the stdlib, and setuptools installed by default with ensurepip) and because pip natively understood how to `pip install` an sdist using setup.py. But none of these advantages are true anymore -- distutils is gone in 3.12, so only actual setuptools is supported, and it's not installed by default, only in build venvs if actually requested. And pip natively understands any build backend.

This is also an opportunity to get some excellent integration between components. Libvirt-python's README already talks about how to build against a meson uninstalled build of libvirt -- migrating the python bindings over to meson too, means that you can just use a meson subproject of libvirt, inside libvirt-python, and then simply build libvirt-python and have it automatically grab a copy of libvirt.
Comment 7 Eli Schwartz 2023-06-04 20:53:07 UTC
By the way, the problem you have encountered here is that you build a few different .so extensions, with the following pattern:

- ""
- "_qemu"
- "_lxc"

Each of them builds an extension named libvirtmod{variant}, using two unique source file "libvirt-{variant/_/-/}override.c" and "build/libvirt{variant/_/-/}.c", and two universal shared files, "typewrappers.c" and "libvirt-utils.c".

For the shared files, setuptools has no concept of their sharedness, so each extension module tries to rebuild those files -- their output files are based on input filename, and don't have a key based on the output extension -- but maybe doesn't if it sees that it already exists (regardless of the current contents). Fortunately, you build each one with the same options, so fully single threaded building is "safe".

Unlike meson (or autotools), setuptools has no concept of shared utility libraries. With meson, you'd build each file exactly once into a static_library(..., build_by_default: false, install: false) and then link each extension to the static library.
Comment 8 Michal Prívozník 2023-06-05 03:55:04 UTC
Reported here:

https://github.com/pypa/setuptools/issues/3942
Comment 9 Michal Prívozník 2023-06-05 03:58:50 UTC
(In reply to Eli Schwartz from comment #7)
> By the way, the problem you have encountered here is that you build a few
> different .so extensions, with the following pattern:

Yep, this matches my findings exactly (but was too tired to report it yesterday anywhere).
So if there's already a precedence of other python modules switching away from setuptools to meson, then maybe I can convince libvirt's community too.
Comment 10 CaptainBlood 2023-06-07 16:35:07 UTC
(In reply to Michal Prívozník from comment #4)
> ... and MAKEOPTS="-j1" solves it.

Indeed setting so in /etc/portage/packages.env works fine here.

Despite I have no real understanding of possible cause, I just want to remind multiple threaded build works fine with gcc:12 here.
Comment 11 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-06-07 16:43:19 UTC
(In reply to CaptainBlood from comment #10)
> Despite I have no real understanding of possible cause, I just want to
> remind multiple threaded build works fine with gcc:12 here.

This has nothing to do with the GCC version.
Comment 12 CaptainBlood 2023-06-07 17:15:45 UTC
I agree, I just had a collision while apologizing about that, sorry.
Comment 13 Larry the Git Cow gentoo-dev 2023-07-28 08:20:32 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=86035d3667a85f92aa5e6596fdd8cf5815e4b47b

commit 86035d3667a85f92aa5e6596fdd8cf5815e4b47b
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-07-28 08:20:02 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-07-28 08:20:03 +0000

    dev-python/libvirt-python: set -j1 to avoid setuptools breakage
    
    May ICE when building w/ GCC LTO or even be miscompiled (even w/o LTO).
    
    Bug: https://bugs.gentoo.org/856835
    Closes: https://bugs.gentoo.org/907718
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-python/libvirt-python/libvirt-python-9.3.0.ebuild | 5 +++++
 dev-python/libvirt-python/libvirt-python-9.4.0.ebuild | 5 +++++
 dev-python/libvirt-python/libvirt-python-9.5.0.ebuild | 5 +++++
 dev-python/libvirt-python/libvirt-python-9999.ebuild  | 5 +++++
 4 files changed, 20 insertions(+)