The Gentoo python 3.4 ebuilds apply this patch: https://gitweb.gentoo.org/proj/python-gentoo-patches.git/diff/patches/21_all_distutils_c++.patch?h=3.3.4 The relevant section: +- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \ +- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', +- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS') ++ (cc, cxx, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \ ++ get_config_vars('CC', 'CXX', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED', ++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS') ++ ++ cflags = '' ++ cxxflags = '' As a result, all cflags from distutils are thrown away, presumably so that cflags supplied by the Gentoo build system in the CFLAGS environment variable may be used instead. This is great for emerging Python packages that use distutils. However, it wrecks manual building of Python packages that use distutils: ~/Desktop/pyFFTW_git]> python-config-3.4 --cflags -I/usr/include/python3.4 -I/usr/include/python3.4 -Wno-unused-result -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -DNDEBUG ~/Desktop/pyFFTW_git]> sudo python setup.py install running install running bdist_egg running egg_info creating pyFFTW.egg-info writing pbr to pyFFTW.egg-info/pbr.json writing pyFFTW.egg-info/PKG-INFO writing top-level names to pyFFTW.egg-info/top_level.txt writing dependency_links to pyFFTW.egg-info/dependency_links.txt writing manifest file 'pyFFTW.egg-info/SOURCES.txt' reading manifest file 'pyFFTW.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pyFFTW.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-3.4 creating build/lib.linux-x86_64-3.4/pyfftw copying pyfftw/_version.py -> build/lib.linux-x86_64-3.4/pyfftw copying pyfftw/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw creating build/lib.linux-x86_64-3.4/pyfftw/builders copying pyfftw/builders/builders.py -> build/lib.linux-x86_64-3.4/pyfftw/builders copying pyfftw/builders/_utils.py -> build/lib.linux-x86_64-3.4/pyfftw/builders copying pyfftw/builders/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw/builders creating build/lib.linux-x86_64-3.4/pyfftw/interfaces copying pyfftw/interfaces/_utils.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces copying pyfftw/interfaces/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces copying pyfftw/interfaces/cache.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces copying pyfftw/interfaces/numpy_fft.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces copying pyfftw/interfaces/scipy_fftpack.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces running build_ext skipping '/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c' Cython extension (up-to-date) building 'pyfftw.pyfftw' extension creating build/temp.linux-x86_64-3.4 creating build/temp.linux-x86_64-3.4/home creating build/temp.linux-x86_64-3.4/home/ehvatum creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw x86_64-pc-linux-gnu-gcc -pthread -fPIC -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c /home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c -o build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.o In file included from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0, from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:17, from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4, from /home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c:239: /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^ x86_64-pc-linux-gnu-gcc -pthread -shared build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.o -L/usr/lib64 -lfftw3 -lfftw3f -lfftw3l -lfftw3_threads -lfftw3f_threads -lfftw3l_threads -lpython3.4 -o build/lib.linux-x86_64-3.4/pyfftw/pyfftw.cpython-34.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/pyfftw creating build/bdist.linux-x86_64/egg/pyfftw/builders copying build/lib.linux-x86_64-3.4/pyfftw/builders/builders.py -> build/bdist.linux-x86_64/egg/pyfftw/builders copying build/lib.linux-x86_64-3.4/pyfftw/builders/_utils.py -> build/bdist.linux-x86_64/egg/pyfftw/builders copying build/lib.linux-x86_64-3.4/pyfftw/builders/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw/builders copying build/lib.linux-x86_64-3.4/pyfftw/_version.py -> build/bdist.linux-x86_64/egg/pyfftw copying build/lib.linux-x86_64-3.4/pyfftw/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw copying build/lib.linux-x86_64-3.4/pyfftw/pyfftw.cpython-34.so -> build/bdist.linux-x86_64/egg/pyfftw creating build/bdist.linux-x86_64/egg/pyfftw/interfaces copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/_utils.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/cache.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/numpy_fft.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/scipy_fftpack.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/builders.py to builders.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/_utils.py to _utils.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/__init__.py to __init__.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/_version.py to _version.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/__init__.py to __init__.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/_utils.py to _utils.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/__init__.py to __init__.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/cache.py to cache.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/numpy_fft.py to numpy_fft.cpython-34.pyc byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/scipy_fftpack.py to scipy_fftpack.cpython-34.pyc creating stub loader for pyfftw/pyfftw.cpython-34.so byte-compiling build/bdist.linux-x86_64/egg/pyfftw/pyfftw.py to pyfftw.cpython-34.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying pyFFTW.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying pyFFTW.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying pyFFTW.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying pyFFTW.egg-info/pbr.json -> build/bdist.linux-x86_64/egg/EGG-INFO copying pyFFTW.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... pyfftw.__pycache__.pyfftw.cpython-34: module references __file__ creating dist creating 'dist/pyFFTW-0.10.0-py3.4-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing pyFFTW-0.10.0-py3.4-linux-x86_64.egg removing '/usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg' (and everything under it) creating /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg Extracting pyFFTW-0.10.0-py3.4-linux-x86_64.egg to /usr/lib64/python3.4/site-packages pyFFTW 0.10.0 is already the active version in easy-install.pth Installed /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg Processing dependencies for pyFFTW==0.10.0 Finished processing dependencies for pyFFTW==0.10.0 ~/Desktop/pyFFTW_git]> cd .. ~/Desktop/pyFFTW_git]> python3 -c 'import pyfftw' zsh: segmentation fault python3 -c 'import pyfftw' Note that the following cflags are missing from the compilation command: -Wno-unused-result -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -DNDEBUG As a result, our in-house Python C extensions were built with INCORRECT options, causing segfaults and 3x run time increase for important production code. This was surprising as python-config --cflags suggests that the correct cflags will be used, when, in fact, Gentoo has specifically modified distutils so that these flags are not used. That's just evil. Changing the section of the patch pasted above to the following fixes the issue: +- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \ +- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', +- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS') ++ (cc, cxx, cflags, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \ ++ get_config_vars('CC', 'CXX', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED', ++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS') ++ ++ cxxflags = cflags After making an overlay with the patch thusly modified and re-emerging Python3, this surprising and incorrect behavior is thankfully gone: ~/Desktop/pyFFTW_git]> sudo rm -r build dist pyFFTW.egg-info ~/Desktop/pyFFTW_git]> sudo python setup.py install [snip] x86_64-pc-linux-gnu-gcc -pthread -Wno-unused-result -DNDEBUG -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -fPIC -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c [snip] Installed /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg Processing dependencies for pyFFTW==0.10.0 Finished processing dependencies for pyFFTW==0.10.0 ~/Desktop]> python3 -c 'import pyfftw' ~/Desktop]> No more segfault!
Which missing flag is causing the segfault?
(In reply to Mike Gilbert from comment #1) > Which missing flag is causing the segfault? Not the one I expected! -O2 seems to be the essential flag. With no optimization flag or -O0, the segfault occurs. It happens with gcc 4.6.3, 4.8.4, and 4.9.2, suggesting to me that it's not caused by a gcc optimizer bug that manifests when building pyFFTW. I can not get the segfault to occur with a debug Python 3.4.3 interpreter that I built from source (with its sysconfig.py file modified to have the same issue). I'm digging some more... it's possible that the segfault issue is merely exposed by the missing -O2 option, in which case the missing distutils cflags can only be said to trigger the segfault issue, not cause it. Nonetheless, I think many people would be surprised to find that distutils builds on Gentoo ignore the contents of the distutils CFLAGS config variable. It may make sense for emerge to ignore distutils CFLAGS so that Python does not need to be rebuilt in order for CFLAGS changes to make.conf to impact emerging of Python extensions, but the way this is currently implemented has the subtle and (to me) totally unexpected side effect of wiping out CFLAGS outside of the emerge context.
Right; I'm pretty sure we wipe out the CFLAGS intentionally. We have been doing this for a *long* time; the patch predates my time as a Gentoo developer. Therefore, it is hard for me to say personally *why* we do this.
Does anyone else on the python team have any insight here?
(In reply to Mike Gilbert from comment #4) > Does anyone else on the python team have any insight here? Looks like the patch originates from here: http://bugs.python.org/issue1222585 Diffing the latest version of the python3.4 patch on that page and the Gentoo patch, the only differences appear related to support for emxccompiler and ensuring that mingw and not cygwin gcc is used on win32. The loss of CFLAGS might be a simple oversight made as the patch was updated through the years. I'm leaving a comment there about the CFLAGS lossage. There is talk of applying the patch into the official Python 3.5 tree, so this seems like a good time to bring up my issue :)
Note that we've stopped applying this patch recently because of: 1. bug 831901 2. the confusion over the need for the patch (like this bug)