All the ebuilds for dev-lang/python have this logic in the build_cbuild_python function: # pass system CFLAGS & LDFLAGS as _NODIST, otherwise they'll get # propagated to sysconfig for built extensions # # -fno-lto to avoid bug #700012 (not like it matters for mini-CBUILD Python anyway) local -x CFLAGS_NODIST="${BUILD_CFLAGS} -fno-lto" local -x LDFLAGS_NODIST=${BUILD_LDFLAGS} local -x CFLAGS= LDFLAGS= local -x BUILD_CFLAGS="${CFLAGS_NODIST}" local -x BUILD_LDFLAGS=${LDFLAGS_NODIST} This causes distutils/setuptools to use an extremely restricted set of compilation options when compiling C extensions. $ python3 -c 'import sysconfig; print(sysconfig.get_config_var("CFLAGS"))' -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG In particular note that there are no -g or -O options whatsoever in this list, and that means, if you use `pip install` to install packages that include compiled-code extensions from PyPI into a virtualenv, they are compiled with no optimization and no debug information. This is undesirable. I recognize that not all of the options included in CFLAGS_NODIST are safe when compiling random C extensions, but I think that what I have in CFLAGS in /etc/portage/make.conf (namely `-ggdb3 -O2 -march=native -pipe`) really ought to be getting used when I compile C extensions using pip. (Why aren't I packaging these C extensions for Portage? Because I'm _developing_ those extensions. `source .venv/bin/activate && pip install -e .` in a local git checkout is easy, fast, and doesn't require root.) Reproducible: Always Steps to Reproduce: The easiest way to see the problem is to run `pip install -e .` on a package containing a C extension whose source code contains an intentional compile-time error; this will cause pip to print out the failed compilation command, which will include only the restricted CFLAGS listed above. Here is a complete recipe in the form of a shell script: ``` #! /bin/sh mkdir no-optimization-demo cd no-optimization-demo mkdir demo echo > demo/demo.c '#error "force compilation error"' echo > pyproject.toml \ '[project] name = "no-optimization-demo" version = "0.0.1" [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [tool.setuptools] packages = ["demo"]' echo > setup.py \ 'from setuptools import Extension, setup setup(ext_modules = [ Extension("demo.demo", sources = ["demo/demo.c"]) ])' python3 -m venv venv . venv/bin/activate pip install -e . 2>&1 | grep demo.c | grep -v CalledProcessError ``` Note that this script creates a directory named `no-optimization-demo` and does not clean it up afterward. Actual Results: Output of the script above will be something like x86_64-pc-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fPIC -I/tmp/no-optimization-demo/venv/include -I/usr/include/python3.12 -c demo/demo.c -o /tmp/tmp8y_eensp.build-temp/demo/demo.o demo/demo.c:1:2: error: #error "force compilation error" Expected Results: On my machine, with my /etc/portage/make.conf, the output should have been something like x86_64-pc-linux-gnu-gcc -ggdb3 -O2 -march=native -pipe -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fPIC -I/tmp/no-optimization-demo/venv/include -I/usr/include/python3.12 -c demo/demo.c -o /tmp/tmp8y_eensp.build-temp/demo/demo.o demo/demo.c:1:2: error: #error "force compilation error" Will attach emerge --info output
Created attachment 914272 [details] emerge --info output
This change goes back to bug 831901. > I think that what I have in CFLAGS in /etc/portage/make.conf (namely `-ggdb3 -O2 -march=native -pipe`) really ought to be getting used when I compile C extensions using pip. -march=native is problematic because I can see us getting a bug report down the line where somebody accidentally produced non-portable wheels and didn't expect that leakage to be happening. On the other hand, I agree that we need at least something in there, and if we're going to have something, it makes sense for it to be configurable to some degree: do we just take CFLAGS and strip out -march=* and pray? or just pick -O2 and hope nobody complains? (I'm hoping here for "sensible default" and that ideally there'd be some way to override this as an environment variable or in a configuration file for pip-based installs; I'm not sure myself if such a mechanism exists.)
> -march=native is problematic because I can see us getting a bug report > down the line where somebody accidentally produced non-portable wheels > and didn't expect that leakage to be happening. ugh, good point. counter point, though: people doing HPC really want their local builds of non-packaged math acceleration modules to be optimized to death. but if we can get _some_ level of -O (anything but the default -O0 is night and day) and also -g in the defaults, that would be Good Enough most of the time.
> ideally there'd be some way to override this as an environment variable > or in a configuration file for pip-based installs Just checked and you *can* override what's in sysconfig by setting CFLAGS as an environment variable. However, https://setuptools.pypa.io/en/latest/userguide/ext_modules.html does not say anything about per-user or per-venv config files or anything like that.
The ultimate config file is, naturally, ~/.bashrc. It's also possible to port math acceleration modules which you have some control over, to a sane python build system such as Meson. Meson defaults to buildtype=debug but you can update the default per-project to debugoptimized, which gives you -g -O2. You also get various other improvements such as parallel compilation, which C extensions often benefit heavily from. I'd also argue that in general it's surprising for software optimization flags to be configured by "what was the language runtime configured with" rather than "what is CFLAGS" and that if you're building software you should `. /etc/portage/make.conf && export CFLAGS CXXFLAGS LDFLAGS FFLAGS FCFLAGS`. I'm not sure *anything* other than setuptools inspects the recorded sysconfig.get_config_var copy of "all Makefile variables from building python"
> The ultimate config file is, naturally, ~/.bashrc. That is in fact my current workaround. > port math acceleration modules which you have some control over, > to a sane python build system This is not an option for me, for reasons which are thoroughly off-topic for this bug report. In general, I don't think "your workflow is bogus" is ever a constructive response.
(In reply to Zack Weinberg from comment #6) > In general, I don't think "your workflow is bogus" is ever a > constructive response. I don't understand what you mean by that.
In any case, it doesn't matter, as the workflow exists and we have a way of making it less-broken (by choosing something simple & unobjectionable) even if we don't like it.
It is not currently broken, it just works like a ./configure project in that if you want optimization or debugging info you have to configure that yourself. It was previously broken (generated SIGILL'ing binaries, leaking dangerous experimental settings, violating the principle of least astonishment by forcing undesirable production settings into debug modules, or undesirable debug settings into release modules) and fixing the brokenness is now seen as a negative, per the issuance of this ticket. Note that this is about setuptools specifically, not python modules using any other build backend. Why is a specific C/C++ build system expected to check what flags another software was configured with, and then use it internally (with no override whatsoever!!!) -- and why not instead configure GCC itself to default to -O2 -g? This does remind me -- I recently learned Red Hat configures their GCC to inject -march=x86-64-v2 into ALL binaries built on Red Hat operating systems, so it is now impossible to use Red Hat as a build platform for compiling standalone executables that are distributed in tarballs for users to download and install directly. Should Gentoo do the same? And if not, what's the difference between that and setuptools?
(In reply to Eli Schwartz from comment #9) The difference is that Python defaults to storing these, and setuptools defaults to retrieving them, and blanking it isn't something that anyone else does. The problem is worse on Gentoo because of this, rather than us just picking something bland for it to hardcode. It's not users' fault that the thing they're `pip install`ing uses setuptools. As for prior art, there's: * https://fedoraproject.org/wiki/Changes/Python_Extension_Flags * https://fedoraproject.org/wiki/Changes/Python_Extension_Flags_Reduction
Another option would be to patch setuptools to have a sane default (like "-O2") rather than doing it in Python (which is fine with me), but your objection would apply to that too, AFAICT.
(In reply to Eli Schwartz from comment #9) > It is not currently broken, it just works like a ./configure project in that > if you want optimization or debugging info you have to configure that > yourself. (This is not a good example -- autoconf defaults to -O2 -g for GCC at least.)
(In reply to Sam James from comment #10) > (In reply to Eli Schwartz from comment #9) > > The difference is that Python defaults to storing these, and setuptools > defaults to retrieving them, and blanking it isn't something that anyone > else does. The problem is worse on Gentoo because of this, rather than us > just picking something bland for it to hardcode. > > It's not users' fault that the thing they're `pip install`ing uses > setuptools. > > As for prior art, there's: > * https://fedoraproject.org/wiki/Changes/Python_Extension_Flags > * https://fedoraproject.org/wiki/Changes/Python_Extension_Flags_Reduction For prior art there's also bug 831901 :) Fedora's flags are the opposite of bland: -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -O3 But indeed, your second link *specifically* claims that they are going out of their way to remove all flags from sysconfig that don't specifically relate to binary compatibility, which means they are going to remove -O2 -g. (In reply to Sam James from comment #11) > Another option would be to patch setuptools to have a sane default (like > "-O2") rather than doing it in Python (which is fine with me), but your > objection would apply to that too, AFAICT. Absolutely not what I said at all! But perhaps best demonstrated by talking about this instead: (In reply to Sam James from comment #12) > (In reply to Eli Schwartz from comment #9) > > It is not currently broken, it just works like a ./configure project in that > > if you want optimization or debugging info you have to configure that > > yourself. > > (This is not a good example -- autoconf defaults to -O2 -g for GCC at least.) ... no, my example was chosen with exactitude and specificity. I meant exactly what I said and I continue to stand by it. In fact, above I also mentioned meson as another example, which defaults to -O0 -g but can be told to default to -O2 -g by setting the per-project default buildtype to debugoptimized instead of debug. The reason I meant what I said is because setuptools does NOT default to whatever python says, and I am NOT arguing against restoring a sane default. I'm saying it's inappropriate to specify any flags at all. At the moment you need to have forked python ebuilds in a private overlay if you want to build setuptools extensions on Gentoo that don't forcibly disable debug asserts and reduce the set of permitted codegen optimizations around pointer overflow due to CPython having potentially much worse code than extension authors do. setuptools does not default to sysconfig CFLAGS! setuptools *unconditionally uses* sysconfig CFLAGS, and then if you export $CFLAGS via an environment variable it uses those *also*. That is the entire point of bug 831901 (which happened in an ebuild that used setuptools with make.conf exporting CFLAGS, where dev-lang/python had been built with package.env / similar in order to enable some experimental flags on that one package). It's also the point of the Fedora change proposal as well. They want -fexceptions specifically because that "has to be used" for potential ABI reasons, although funnily enough that then only applies to setuptools packages, not anything built via meson-python or scikit-build-core. But they do not want any other forms of contamination, such as -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 which *cannot* be overridden by *any* method whatsoever other than downloading pyenv and compiling your own cpython interpreter in $HOME and using that to pip install your setuptools packages. tl;dr setuptools is treating sysconfig CFLAGS as the same kind of mandatory as `pkg-config --cflags --libs python-3.12`. We could wish that setuptools defaulted to CFLAGS="-O2 -g" the way ./configure does, but currently no such setuptools concept exists. Therefore the only sane thing to do is to assume there are no defaults, and set what you need via ~/.bashrc.
Ah, it wasn't clear to me at all that was your position. I think we agree then. I'm going to send a patch for https://github.com/pypa/distutils/issues/299 and hope they accept it.
A very simple jerry-rigged solution for existing setuptools projects is to implement the extremely sane and reasonable thing that ./configure and meson already do. Add this to the top of your setup.py: import os if 'CFLAGS' not in os.environ: os.environ['CFLAGS'] = '-O2 -g' It plays nicely with *any* cpython build, including the gentoo one, it is trivially overridable by users exporting $CFLAGS themselves (including demanding to build with zero flags by exporting CFLAGS=""), and it's not *that* much boilerplate to add to every existing setup.py in the world. :)
(In reply to Sam James from comment #14) > Ah, it wasn't clear to me at all that was your position. I think we agree > then. I'm going to send a patch for > https://github.com/pypa/distutils/issues/299 and hope they accept it. And then separately propose that we at least (and maybe upstream, let's see how the first PR goes) patch setuptools to default to -O2 -g or something.
By the way: https://fedoraproject.org/wiki/Changes/Python_built_with_gcc_O3 > Instead of Fedora's default -O2 compiler flag, we will use -O3 > to build CPython. This only impacts the interpreter and Python > standard library, not any 3rd party extension modules built as > RPM or on developer machines. This aligns with the way Python is > built upstream. According to our performance measurements, it > makes Python significantly faster (pyperformance geometric > mean: 1.04x faster). And: > This change is limited to CPython interpreter and extension > modules from the Python standard library only thanks to > Changes/Python_Extension_Flags_Reduction (since Fedora 39). Other > Python extension modules will remain bulidng as before, e.g. in > RPM packages, they will still be built with -O2, unless Fedora > changes that globally. The extension modules built with -O2 still > work with Python built with -O3. They have good intentions, and Fedora *specifically* arranged to require you as a pip install user to specify $CFLAGS if you want optimizations. It's just that Fedora never tested their change to ensure it does what the change proposal says it would do -- -O3 is indeed leaking into their sysconfigdata. It didn't, before that change of theirs, e.g. Fedora 40 doesn't have any -O values when using pip install on setuptools projects.
> setuptools does not default to sysconfig CFLAGS! setuptools > *unconditionally uses* sysconfig CFLAGS, and then if you export > $CFLAGS via an environment variable it uses those *also*. I have spent quite a bit of time writing CPython extensions, and I don't feel good about the idea of *dropping* the sysconfig CFLAGS whenever CFLAGS are set in the environment... > At the moment you need to have forked python ebuilds in a private > overlay if you want to build setuptools extensions on Gentoo that > don't forcibly disable debug asserts and reduce the set of permitted > codegen optimizations around pointer overflow due to CPython having > potentially much worse code than extension authors do. ... because aspects of that "potentially much worse code" you're talking about are baked into the CPython API. In particular, I don't think it is safe to build CPython extensions without -fno-strict-overflow or equivalent, no matter how careful the extension's authors have been in their own code. I see that the setuptools folks merged Sam's patches, but I suspect they don't work down in the guts of the CPython interpreter and don't realize that at least some of the sysconfig CFLAGS are necessary for correctness of extension code.
(In reply to Zack Weinberg from comment #18) > > setuptools does not default to sysconfig CFLAGS! setuptools > > *unconditionally uses* sysconfig CFLAGS, and then if you export > > $CFLAGS via an environment variable it uses those *also*. > > I have spent quite a bit of time writing CPython extensions, and I > don't feel good about the idea of *dropping* the sysconfig CFLAGS > whenever CFLAGS are set in the environment... > It's consistent with behaviour of every other build system for Python extensions, though, as far as I'm aware. > > At the moment you need to have forked python ebuilds in a private > > overlay if you want to build setuptools extensions on Gentoo that > > don't forcibly disable debug asserts and reduce the set of permitted > > codegen optimizations around pointer overflow due to CPython having > > potentially much worse code than extension authors do. > > ... because aspects of that "potentially much worse code" you're > talking about are baked into the CPython API. In particular, > I don't think it is safe to build CPython extensions without > -fno-strict-overflow or equivalent, no matter how careful the > extension's authors have been in their own code. > That flag isn't present in CPython's pkgconfig files either, which means that extensions built using Meson at the very least (but I believe no other backend was doing this) wouldn't have been using it before now either. With regard to -fno-strict-overflow, I'm not yet convinced by this argument, because CPython only added it if the compiler used to build CPython supported it *and* --with-strict-overflow wasn't passed to configure. I don't think it's something anyone ought to have been relying on.
On -f{no-,}strict-overflow specifically, see also https://github.com/python/cpython/issues/96821.
(In reply to Zack Weinberg from comment #18) > ... because aspects of that "potentially much worse code" you're > talking about are baked into the CPython API. In particular, > I don't think it is safe to build CPython extensions without > -fno-strict-overflow or equivalent, no matter how careful the > extension's authors have been in their own code. > > I see that the setuptools folks merged Sam's patches, but I suspect > they don't work down in the guts of the CPython interpreter and don't > realize that at least some of the sysconfig CFLAGS are necessary for > correctness of extension code. This is for example true on python 2, and was explicitly fixed in https://peps.python.org/pep-3123/ In general it's a significantly more devastating bug in software if its use of Undefined Behavior leaks into the public API such that people writing their own fully-correct code cannot compile it without Undefined Behavior forced on them by the API and ABI of the project they are linking to. As far as I'm aware, CPython doesn't have any "viral" Undefined Behavior, regardless of the safety (or lack thereof) of its own codebase. There's a single *.c file in the entire codebase compiled with -fno-strict-aliasing, and a global -fwrapv that was added back in 2007 after someone reported a one-line function call in test_str.py segfaulted. No headers or public API are incriminated, and loads of third-party code has been compiled over the decades without crashing due to strict-overflow issues as long as the cpython interpreter itself is built with -fwrapv. I'm going to need a citation on the dangers of this, sorry.
