Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 548776

Summary: dev-lang/python: distutils CFLAGS are thrown away, subtly breaking all 3rd party distutils packages
Product: Gentoo Linux Reporter: Erik Hvatum <ice.rikh>
Component: Current packagesAssignee: Python Gentoo Team <python>
Status: UNCONFIRMED ---    
Severity: normal CC: arfrever.fta, jlec, mjo
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=236332
Whiteboard:
Package list:
Runtime testing required: ---

Description Erik Hvatum 2015-05-06 15:00:55 UTC
The Gentoo python 3.4 ebuilds apply this patch: https://gitweb.gentoo.org/proj/python-gentoo-patches.git/diff/patches/21_all_distutils_c++.patch?h=3.3.4

The relevant section:

+-        (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+-            get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+-                            'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++        (cc, cxx, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++            get_config_vars('CC', 'CXX', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++                            'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++        cflags = ''
++        cxxflags = ''

As a result, all cflags from distutils are thrown away, presumably so that cflags supplied by the Gentoo build system in the CFLAGS environment variable may be used instead.

This is great for emerging Python packages that use distutils.  However, it wrecks manual building of Python packages that use distutils:

~/Desktop/pyFFTW_git]> python-config-3.4 --cflags
-I/usr/include/python3.4 -I/usr/include/python3.4  -Wno-unused-result -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -DNDEBUG

~/Desktop/pyFFTW_git]> sudo python setup.py install         
running install
running bdist_egg
running egg_info
creating pyFFTW.egg-info
writing pbr to pyFFTW.egg-info/pbr.json
writing pyFFTW.egg-info/PKG-INFO
writing top-level names to pyFFTW.egg-info/top_level.txt
writing dependency_links to pyFFTW.egg-info/dependency_links.txt
writing manifest file 'pyFFTW.egg-info/SOURCES.txt'
reading manifest file 'pyFFTW.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyFFTW.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.4
creating build/lib.linux-x86_64-3.4/pyfftw
copying pyfftw/_version.py -> build/lib.linux-x86_64-3.4/pyfftw
copying pyfftw/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw
creating build/lib.linux-x86_64-3.4/pyfftw/builders
copying pyfftw/builders/builders.py -> build/lib.linux-x86_64-3.4/pyfftw/builders
copying pyfftw/builders/_utils.py -> build/lib.linux-x86_64-3.4/pyfftw/builders
copying pyfftw/builders/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw/builders
creating build/lib.linux-x86_64-3.4/pyfftw/interfaces
copying pyfftw/interfaces/_utils.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces
copying pyfftw/interfaces/__init__.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces
copying pyfftw/interfaces/cache.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces
copying pyfftw/interfaces/numpy_fft.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces
copying pyfftw/interfaces/scipy_fftpack.py -> build/lib.linux-x86_64-3.4/pyfftw/interfaces
running build_ext
skipping '/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c' Cython extension (up-to-date)
building 'pyfftw.pyfftw' extension
creating build/temp.linux-x86_64-3.4
creating build/temp.linux-x86_64-3.4/home
creating build/temp.linux-x86_64-3.4/home/ehvatum
creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop
creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git
creating build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw
x86_64-pc-linux-gnu-gcc -pthread -fPIC -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c /home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c -o build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.o
In file included from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0,
                 from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.c:239:
/usr/lib64/python3.4/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
x86_64-pc-linux-gnu-gcc -pthread -shared build/temp.linux-x86_64-3.4/home/ehvatum/Desktop/pyFFTW_git/pyfftw/pyfftw.o -L/usr/lib64 -lfftw3 -lfftw3f -lfftw3l -lfftw3_threads -lfftw3f_threads -lfftw3l_threads -lpython3.4 -o build/lib.linux-x86_64-3.4/pyfftw/pyfftw.cpython-34.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/pyfftw
creating build/bdist.linux-x86_64/egg/pyfftw/builders
copying build/lib.linux-x86_64-3.4/pyfftw/builders/builders.py -> build/bdist.linux-x86_64/egg/pyfftw/builders
copying build/lib.linux-x86_64-3.4/pyfftw/builders/_utils.py -> build/bdist.linux-x86_64/egg/pyfftw/builders
copying build/lib.linux-x86_64-3.4/pyfftw/builders/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw/builders
copying build/lib.linux-x86_64-3.4/pyfftw/_version.py -> build/bdist.linux-x86_64/egg/pyfftw
copying build/lib.linux-x86_64-3.4/pyfftw/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw
copying build/lib.linux-x86_64-3.4/pyfftw/pyfftw.cpython-34.so -> build/bdist.linux-x86_64/egg/pyfftw
creating build/bdist.linux-x86_64/egg/pyfftw/interfaces
copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/_utils.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces
copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/__init__.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces
copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/cache.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces
copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/numpy_fft.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces
copying build/lib.linux-x86_64-3.4/pyfftw/interfaces/scipy_fftpack.py -> build/bdist.linux-x86_64/egg/pyfftw/interfaces
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/builders.py to builders.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/_utils.py to _utils.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/builders/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/_version.py to _version.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/_utils.py to _utils.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/cache.py to cache.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/numpy_fft.py to numpy_fft.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/interfaces/scipy_fftpack.py to scipy_fftpack.cpython-34.pyc
creating stub loader for pyfftw/pyfftw.cpython-34.so
byte-compiling build/bdist.linux-x86_64/egg/pyfftw/pyfftw.py to pyfftw.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying pyFFTW.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pyFFTW.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pyFFTW.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pyFFTW.egg-info/pbr.json -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pyFFTW.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
pyfftw.__pycache__.pyfftw.cpython-34: module references __file__
creating dist
creating 'dist/pyFFTW-0.10.0-py3.4-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing pyFFTW-0.10.0-py3.4-linux-x86_64.egg
removing '/usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg' (and everything under it)
creating /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg
Extracting pyFFTW-0.10.0-py3.4-linux-x86_64.egg to /usr/lib64/python3.4/site-packages
pyFFTW 0.10.0 is already the active version in easy-install.pth

Installed /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg
Processing dependencies for pyFFTW==0.10.0
Finished processing dependencies for pyFFTW==0.10.0

~/Desktop/pyFFTW_git]> cd ..

~/Desktop/pyFFTW_git]> python3 -c 'import pyfftw'
zsh: segmentation fault  python3 -c 'import pyfftw'


Note that the following cflags are missing from the compilation command:
-Wno-unused-result -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -DNDEBUG

As a result, our in-house Python C extensions were built with INCORRECT options, causing segfaults and 3x run time increase for important production code.  This was surprising as python-config --cflags suggests that the correct cflags will be used, when, in fact, Gentoo has specifically modified distutils so that these flags are not used.  That's just evil.

Changing the section of the patch pasted above to the following fixes the issue:

+-        (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+-            get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+-                            'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++        (cc, cxx, cflags, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++            get_config_vars('CC', 'CXX', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++                            'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++        cxxflags = cflags

After making an overlay with the patch thusly modified and re-emerging Python3, this surprising and incorrect behavior is thankfully gone:

~/Desktop/pyFFTW_git]> sudo rm -r build dist pyFFTW.egg-info                   
~/Desktop/pyFFTW_git]> sudo python setup.py install                            
[snip]
x86_64-pc-linux-gnu-gcc -pthread -Wno-unused-result -DNDEBUG -march=native -O2 -pipe -fno-omit-frame-pointer -fwrapv -fPIC -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -I/home/ehvatum/Desktop/pyFFTW_git/include -I/home/ehvatum/Desktop/pyFFTW_git/pyfftw -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c 
[snip]
Installed /usr/lib64/python3.4/site-packages/pyFFTW-0.10.0-py3.4-linux-x86_64.egg
Processing dependencies for pyFFTW==0.10.0
Finished processing dependencies for pyFFTW==0.10.0

~/Desktop]> python3 -c 'import pyfftw'
~/Desktop]>

No more segfault!
Comment 1 Mike Gilbert gentoo-dev 2015-05-06 15:13:18 UTC
Which missing flag is causing the segfault?
Comment 2 Erik Hvatum 2015-05-06 18:38:52 UTC
(In reply to Mike Gilbert from comment #1)
> Which missing flag is causing the segfault?

Not the one I expected!  -O2 seems to be the essential flag.  With no optimization flag or -O0, the segfault occurs.  It happens with gcc 4.6.3, 4.8.4, and 4.9.2, suggesting to me that it's not caused by a gcc optimizer bug that manifests when building pyFFTW.  I can not get the segfault to occur with a debug Python 3.4.3 interpreter that I built from source (with its sysconfig.py file modified to have the same issue).

I'm digging some more... it's possible that the segfault issue is merely exposed by the missing -O2 option, in which case the missing distutils cflags can only be said to trigger the segfault issue, not cause it.

Nonetheless, I think many people would be surprised to find that distutils builds on Gentoo ignore the contents of the distutils CFLAGS config variable.  It may make sense for emerge to ignore distutils CFLAGS so that Python does not need to be rebuilt in order for CFLAGS changes to make.conf to impact emerging of Python extensions, but the way this is currently implemented has the subtle and (to me) totally unexpected side effect of wiping out CFLAGS outside of the emerge context.
Comment 3 Mike Gilbert gentoo-dev 2015-05-06 19:07:27 UTC
Right; I'm pretty sure we wipe out the CFLAGS intentionally.

We have been doing this for a *long* time; the patch predates my time as a Gentoo developer. Therefore, it is hard for me to say personally *why* we do this.
Comment 4 Mike Gilbert gentoo-dev 2015-05-06 19:20:19 UTC
Does anyone else on the python team have any insight here?
Comment 5 Erik Hvatum 2015-05-06 19:51:52 UTC
(In reply to Mike Gilbert from comment #4)
> Does anyone else on the python team have any insight here?

Looks like the patch originates from here: http://bugs.python.org/issue1222585

Diffing the latest version of the python3.4 patch on that page and the Gentoo patch, the only differences appear related to support for emxccompiler and ensuring that mingw and not cygwin gcc is used on win32.  The loss of CFLAGS might be a simple oversight made as the patch was updated through the years.

I'm leaving a comment there about the CFLAGS lossage.  There is talk of applying the patch into the official Python 3.5 tree, so this seems like a good time to bring up my issue :)