| Summary: | dev-python/pyzmq-14.4.1 - error: each element of 'ext_modules' option must be an Extension instance or 2-tuple | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Greg Turner <gmturner007> |
| Component: | [OLD] Development | Assignee: | Python Gentoo Team <python> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | alex3255, dustin |
| Priority: | Normal | Keywords: | PATCH |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | http://bugs.python.org/issue23102 | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=533926 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
pyzmq-14.4.1-use_setuptools_not_distutils.patch
result of: emerge --info > emergeinfo.out build log less_confusing_log.txt pyzmq-14.4.1-import_distutils_after_setuptools_possibly_monkey_patches_it_due_to_nose_usage.patch distutils_accomodate_extension_ducktypes.patch distutils_accomodate_distribution_ducktypes.patch |
||
|
Description
Greg Turner
2014-12-16 16:01:43 UTC
(In reply to Greg Turner from comment #0) > (Pdb) extensions > [<distutils.extension.Extension object at 0x7f7fd7e7dd68>, > <distutils.extension.Extension object at 0x7f7fd7d2ca90>, > <distutils.extension.Extension object at 0x7f7fd2acd0f0>, > <distutils.extension.Extension object at 0x7f7fd2ae92e8>, > <distutils.extension.Extension object at 0x7f7fd2ae9358>, > <distutils.extension.Extension object at 0x7f7fd2ae93c8>, > <distutils.extension.Extension object at 0x7f7fd2af2a90>, > <distutils.extension.Extension object at 0x7f7fd2af2ac8>, > <distutils.extension.Extension object at 0x7f7fd287d198>, > <distutils.extension.Extension object at 0x7f7fd2884438>]. WTF, setuptools?) maybe setuptools tries to monkey-patch distutils for this, but some load-ordering or module-namespace duplication problem is preventing the correct binding from going to the code object for check_extensions_list... that would be a bug I could report upstream (to setuptools, if I really nailed it), if I could just wrap my head around it. (In reply to Greg Turner from comment #0) > Not sure what is behind this. On my system I fail to build pyzmq 1) Please attach the entire build log to this bug report. 2) Please post your `emerge --info' output in a comment. Created attachment 391896 [details] result of: emerge --info > emergeinfo.out (In reply to Jeroen Roovers from comment #2) > (In reply to Greg Turner from comment #0) > 2) Please post your `emerge --info' output in a comment. Heh, you might not like it :) Well it's less mortifying than it used to be I guess. Created attachment 391898 [details]
build log
Portage output. Cut/pasted from the console so I can reverse my patch after the prepare phase (to get back the vanilla, non-working behavior).
Still no idea what you're trying to do. You have an original build failure you're not disclosing and now a new build failure that depends on a patch that supposedly fixes the issue you're not disclosing to begin with. (In reply to Jeroen Roovers from comment #5) > Still no idea what you're trying to do. You have an original build failure > you're not disclosing and now a new build failure that depends on a patch > that supposedly fixes the issue you're not disclosing to begin with. I reverted the patch I installed in /etc/portage/patches, which fixed the problem for me. Had I let the build proceed with that patch, it would have succeeded. The build failure I'm experiencing when using the gx86 ebuild without modification is as seen in the enclosed log. Sorry for the confusion, I should have just hid the evidence of that since it's not relevant. Created attachment 391902 [details]
less_confusing_log.txt
OK, sorry. I get that there's only so much time in a day to triage stuff like this.
Here is a revised log which should clarify both what I was doing when I reversed the patch last time, and that something is not right, at least, on my particular system, when using the vanilla, unpatched, gx86 ebuild.
One other note: In function `main': timer_create02m9pb.c:(.text+0x15): undefined reference to `timer_create' ain't it. It's like an autotools check -- an expected error occurring as part of the build process. Same issue with picard: [ebuild U ] media-sound/picard-1.3 [1.2-r1] USE="acoustid cdda nls" 0 KiB Total: 1 package (1 upgrade), Size of downloads: 0 KiB Would you like to merge these packages? [Yes/No] >>> Verifying ebuild manifests >>> Emerging (1 of 1) media-sound/picard-1.3::gentoo * picard-1.3.tar.gz SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ] >>> Unpacking source... >>> Unpacking picard-1.3.tar.gz to /var/tmp/portage/media-sound/picard-1.3/work >>> Source unpacked in /var/tmp/portage/media-sound/picard-1.3/work >>> Preparing source in /var/tmp/portage/media-sound/picard-1.3/work/picard-1.3 ... >>> Source prepared. >>> Configuring source in /var/tmp/portage/media-sound/picard-1.3/work/picard-1.3 ... >>> Source configured. >>> Compiling source in /var/tmp/portage/media-sound/picard-1.3/work/picard-1.3 ... * Building of media-sound/picard-1.3 with CPython 2.7... python2.7 setup.py build -b build-2.7 running build generating scripts/picard from scripts/picard.in running build_py creating build-2.7 creating build-2.7/lib.linux-x86_64-2.7 creating build-2.7/lib.linux-x86_64-2.7/picard copying picard/webservice.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/track.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/tagger.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/similarity.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/script.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/resources.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/releasegroup.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/plugin.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/metadata.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/mbxml.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/log.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/i18n.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/file.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/disc.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/dataobj.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/config_upgrade.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/config.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/collection.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/cluster.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/album.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/acoustidmanager.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/acoustid.py -> build-2.7/lib.linux-x86_64-2.7/picard copying picard/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard creating build-2.7/lib.linux-x86_64-2.7/picard/browser copying picard/browser/filelookup.py -> build-2.7/lib.linux-x86_64-2.7/picard/browser copying picard/browser/browser.py -> build-2.7/lib.linux-x86_64-2.7/picard/browser copying picard/browser/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/browser creating build-2.7/lib.linux-x86_64-2.7/picard/const copying picard/const/locales.py -> build-2.7/lib.linux-x86_64-2.7/picard/const copying picard/const/languages.py -> build-2.7/lib.linux-x86_64-2.7/picard/const copying picard/const/countries.py -> build-2.7/lib.linux-x86_64-2.7/picard/const copying picard/const/attributes.py -> build-2.7/lib.linux-x86_64-2.7/picard/const copying picard/const/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/const creating build-2.7/lib.linux-x86_64-2.7/picard/coverart copying picard/coverart/utils.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart copying picard/coverart/image.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart copying picard/coverart/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart creating build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers copying picard/coverart/providers/whitelist.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers copying picard/coverart/providers/caa_release_group.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers copying picard/coverart/providers/caa.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers copying picard/coverart/providers/amazon.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers copying picard/coverart/providers/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/coverart/providers creating build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/wav.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/vorbis.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/mp4.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/id3.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/asf.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/apev2.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats copying picard/formats/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats creating build-2.7/lib.linux-x86_64-2.7/picard/formats/mutagenext copying picard/formats/mutagenext/tak.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats/mutagenext copying picard/formats/mutagenext/compatid3.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats/mutagenext copying picard/formats/mutagenext/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/formats/mutagenext creating build-2.7/lib.linux-x86_64-2.7/picard/plugins copying picard/plugins/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/plugins creating build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/util.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_tagsfromfilenames.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_passworddialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_tags.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_script.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_renaming.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_releases.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_ratings.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_plugins.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_network.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_metadata.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_matching.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_interface.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_general.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_folksonomy.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_fingerprinting.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_cover.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_cdlookup_select.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_cdlookup.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_advanced.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options_about.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_options.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_infostatus.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_infodialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_edittagdialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ui_cdlookup.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/tagsfromfilenames.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/ratingwidget.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/passworddialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/metadatabox.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/mainwindow.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/logview.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/itemviews.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/item.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/infostatus.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/infodialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/filebrowser.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/edittagdialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/coverartbox.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/collectionmenu.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/cdlookup.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui copying picard/ui/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui creating build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/tags.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/scripting.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/renaming.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/releases.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/ratings.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/plugins.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/network.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/metadata.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/matching.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/interface.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/general.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/folksonomy.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/fingerprinting.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/dialog.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/cover.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/cdlookup.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/advanced.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/about.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options copying picard/ui/options/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/ui/options creating build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/webbrowser2.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/versions.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/thread.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/textencoding.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/tags.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/imageinfo.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/icontheme.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/filenaming.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/cdrom.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/bytes2human.py -> build-2.7/lib.linux-x86_64-2.7/picard/util copying picard/util/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/util creating build-2.7/lib.linux-x86_64-2.7/picard/util/devutil copying picard/util/devutil/__init__.py -> build-2.7/lib.linux-x86_64-2.7/picard/util/devutil warning: build_py: byte-compiling is disabled, skipping. running build_ext error: each element of 'ext_modules' option must be an Extension instance or 2-tuple * ERROR: media-sound/picard-1.3::gentoo failed (compile phase): * Building failed with CPython 2.7 in distutils_building() function * * Call stack: * ebuild.sh, line 93: Called src_compile * environment, line 4897: Called distutils_src_compile * environment, line 1208: Called python_execute_function 'distutils_building' * environment, line 3291: Called die * The specific snippet of code: * die "${failure_message}"; * * If you need support, post the output of `emerge --info '=media-sound/picard-1.3::gentoo'`, * the complete build log and the output of `emerge -pqv '=media-sound/picard-1.3::gentoo'`. * The complete build log is located at '/var/tmp/portage/media-sound/picard-1.3/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/media-sound/picard-1.3/temp/environment'. * Working directory: '/var/tmp/portage/media-sound/picard-1.3/work/picard-1.3' * S: '/var/tmp/portage/media-sound/picard-1.3/work/picard-1.3' I suspect the error about ext_modules is a symptom that is triggered by this error earlier in the log: cc -c /var/tmp/portage/dev-python/pyzmq-14.4.1/temp/timer_create02m9pb.c -o /var/tmp/portage/dev-python/pyzmq-14.4.1/work/pyzmq-14.4.1-python3_3/temp.linux-x86_64-3.3/scratch/var/tmp/portage/dev-python/pyzmq-14.4.1/temp/timer_create02m9pb.o cc /var/tmp/portage/dev-python/pyzmq-14.4.1/work/pyzmq-14.4.1-python3_3/temp.linux-x86_64-3.3/scratch/var/tmp/portage/dev-python/pyzmq-14.4.1/temp/timer_create02m9pb.o -o /var/tmp/portage/dev-python/pyzmq-14.4.1/work/pyzmq-14.4.1-python3_3/temp.linux-x86_64-3.3/scratch/a.out /var/tmp/portage/dev-python/pyzmq-14.4.1/work/pyzmq-14.4.1-python3_3/temp.linux-x86_64-3.3/scratch/var/tmp/portage/dev-python/pyzmq-14.4.1/temp/timer_create02m9pb.o: In function `main': timer_create02m9pb.c:(.text+0x15): undefined reference to `timer_create' (In reply to Mike Gilbert from comment #10) Nevermind; that's just a test to see if we need to link with librt. Seems to be a bug in setuptools. I bisected it to this changeset. https://bitbucket.org/pypa/setuptools/commits/dcd552da643c (In reply to Greg Turner from comment #1) > maybe setuptools tries to monkey-patch distutils for this, but some > load-ordering or module-namespace duplication problem is preventing the > correct binding from going to the code object for check_extensions_list... > that would be a bug I could report upstream (to setuptools, if I really > nailed it), if I could just wrap my head around it. I think you are on the right track here. Just playing with some imports in setup.py: This works: import setuptools from distutils.extension import Extension This fails: from distutils.extension import Extension import setuptools + 20 Dec 2014; Mike Gilbert <floppym@gentoo.org> pyzmq-14.3.1.ebuild, + pyzmq-14.4.0.ebuild, pyzmq-14.4.1.ebuild: + Force early import of setuptools to work around bug 532708. I'll leave this open in case someone figures out the root cause here. I created an issue for setuptools. https://bitbucket.org/pypa/setuptools/issue/309 What I find insane about it is that pyzmq very explicitly imports everything from distutils. Then, Shazam! out of nowhere, setuptools objects appear like a rabbit from an empty hat. Perhaps the way to fix it, then, is to figure out who is to blame for bringing in setuptools in the first place, say, by running python -v or replacing the first couple lines in all the setuptools modules with a pdb invocation (I recommend against using ipdb here -- it does not play nicely with these modules in my experience). If the distutils modules are frozen, then you'll have your work cut out for you. There is a way to "unfreeze" them. Get in there with your debugger during the import statement, somehow (one way is to monkey patch __import__ in __builtins__ at the pdb prompt) then, right as it's being imported, blow away the whatever-its-called-now that replaced the module-level __loader__ thingy in recent pythons. And just wedge an instance of the regular source loader (not exactly loader anymore, but, like I said, whatever-its-called-now that points to the loader) instead, pointing to the real source code you want to see. Or, maybe s/loader/meta-path-finder/ ? Something like that works... But it's nontrivial I'm afraid.... Then, re-import the module, and kill off the debugger. Next time it's imported, the thing you want to happen will. python -v and a fearless attitude help :) If nobody beats me to it, I might have time to take a closer look at it tonight, now that we're sure I'm not just a victim of my own rice. (In reply to Greg Turner from comment #16) > What I find insane about it is that pyzmq very explicitly imports everything > from distutils. Then, Shazam! out of nowhere, setuptools objects appear > like a rabbit from an empty hat. On my system, it gets imported by dev-python/nose, which gets imported in pyzmq's setup.py. % python3.4 setup.py build Traceback (most recent call last): File "setup.py", line 55, in <module> import nose File "/usr/lib64/python3.4/site-packages/nose/__init__.py", line 1, in <module> from nose.core import collector, main, run, run_exit, runmodule File "/usr/lib64/python3.4/site-packages/nose/core.py", line 11, in <module> from nose.config import Config, all_config_files File "/usr/lib64/python3.4/site-packages/nose/config.py", line 9, in <module> from nose.plugins.manager import NoPlugins File "/usr/lib64/python3.4/site-packages/nose/plugins/__init__.py", line 185, in <module> from nose.plugins.manager import * File "/usr/lib64/python3.4/site-packages/nose/plugins/manager.py", line 418, in <module> import pkg_resources File "/home/floppym/src/setuptools/pkg_resources.py", line 78, in <module> import setuptools._vendor.packaging.version File "/home/floppym/src/setuptools/setuptools/__init__.py", line 3, in <module> raise Exception("foo") Exception: foo OK. So, here is what I've been able to figure out. First, setuptools aggressively monkey patches the core distutils module layout. That's "just how it works."
Setuptools is effectively saying (so I surmise): "hey, we are like a distutils and then some. So, if you want to import this stuff, congratulations, you have now upgraded your distutils to really be setuptools. That way, your (presumably setuptools-based) setup.py can use all the fancy Setuptoolsian features but most legacy distutils-based code out there ought to work fine anyhow since we quack like a distutils."
I was previously thinking that, however setuptools is getting involved, must be a bug. Maybe it is; maybe not. I'd like to hear your thoughts, Mike et al.
Here's the deal. Nose has a plugin architecture of some kind, as we can see from the file paths in Mike's stack trace (be advised, I know approximately fuck-all about nose; I'm mostly at peace with my ignorance for now)
Quoting from that manager.py:
:class:`DefaultPluginMananger`
This is the manager class that will be used by default. If
setuptools is installed, it is a subclass of
:class:`EntryPointPluginManager` and :class:`BuiltinPluginManager`;
otherwise, an alias to :class:`BuiltinPluginManager`.
The corresponding code looks vaguely like:
class BuiltinPluginManager: pass
try:
import pkg_resources
class DefaultPluginManager(
EntryPointPluginManager, BuiltinPluginManager): pass
except ImportError:
class DefaultPluginManager(BuiltinPluginManager): pass
As can be seen from Mike's stack-trace, pkg_resources (a setuptools module) is what unleashes the monkeys.
So, AFAICS, this comes down to fundamentally differing assumptions about when it's appropriate to monkey-patch distutils:
pyzmq: Never. We are using distutils, full stop.
nose: If setuptools is installed, always, otherwise, never.
setuptools: Always, on import. We control the horizontal and the vertical.
Allright, fine. So.... whose bug is this?
I propose that the bug is mostly in nose, although one could perhaps make compelling arguments that setuptools should do something other than monkey-patch distutils, or that pyzmq should know about nose's behavior and take it into account.
The reason I say so: nose clearly wants to support usage scenarios with and without setuptools.
But nose is sort-of acting as though everyone is using pip+virtualenv -- that is, "Is setuptools around" is a piss-poor way to guess the answer to the question "is setup.py using setuptools?"
Seeking advice/thoughts here... shall I put this to the nose people, wherever they are to be found?
Meanwhile, I'll post an improved patch here, which works around the problem in pyzmq's setup.py simply by re-ordering the imports.
Created attachment 392216 [details, diff]
pyzmq-14.4.1-import_distutils_after_setuptools_possibly_monkey_patches_it_due_to_nose_usage.patch
A revised setup.py-level work-around which simply re-orders the imports to avoid the problem.
(In reply to Greg Turner from comment #19) > Created attachment 392216 [details, diff] [details, diff] > pyzmq-14.4.1- > import_distutils_after_setuptools_possibly_monkey_patches_it_due_to_nose_usag > e.patch > > A revised setup.py-level work-around which simply re-orders the imports to > avoid the problem. One other note. Clearly the behavior of setuptools did change. "import pkg_resources" presumably used not to cause monkey patching distutils; after the commit Mike mentioned above, apparently, it does. That might be a second bug, depending on whether or not setuptools deliberately decided this was the correct behavior, or not. Either way, I'd say that, given that some versions of setuptools will monkey patch, and some will not, when pkg_resources is imported, nose should provide some code-path that will avoid importing pkg_resources even if we import nose -- otherwise, nose effectively becomes subtly incompatible with at least the setup.py from pyzmq, and possibly more. BTW, picard looks to have a somewhat similar etiology although it does not use nose -- like pyzmq, picard does "from distutils.core import Extension". However, annoyingly, I can't recreate the failure reported by Jason -- the ebuild works, for me, OOTB (nb: my portage tree is a few days out of date by now). Also somewhat wierdly, Picard is clearly aware of the issue -- it's setup.py says: try: from py2app.build_app import py2app do_py2app = True except ImportError: do_py2app = False # this must be imported *after* py2app, because py2app imports setuptools # which "patches" (read: screws up) the Extension class from distutils import log from distutils.command.build import build from distutils.command.install import install as install from distutils.core import setup, Command, Extension from distutils.dep_util import newer from distutils.dist import Distribution from distutils.spawn import find_executable Maybe if I had py2app installed I would see the failure although this still seems pretty inexplicable to me :/ (In reply to Greg Turner from comment #18) > Allright, fine. So.... whose bug is this? You know, another idea just occurred to me. The real bug here, when you get down to it, is distutils checking issubclass() in the first place. It's not pythonic and is leading to a real-world problem -- hence, it's pretty much a bug, de-facto if not de-jure. Get rid of that check, and this whole mess just vanishes. Don't know why that didn't occur to me sooner; regardless of who's ultimately "to blame" for this mess, it certainly seems to me that of all the various ways one could attack this problem, loosening up that check would be the least semantically impactful and have the most concise fix. Unless someone convinces me otherwise, I'll file a cpython bug for this, some time tomorrow. Whatever function that issubclass check serves can almost surely be achieved with a hasattr() check that allows for duck-typing. Created attachment 392244 [details, diff]
distutils_accomodate_extension_ducktypes.patch
Here is an alternate fix, which I will probably file as a cpython bug.
If applied to "all" of your gentoo python installations (I applied it, for example, to dev-lang/python:{2.7,3.3,3.4} by placing it in /etc/portage/patches/dev-lang/python/distutils_accomodate_extension_ducktypes.patch and running:
emerge -1a $(qlist -IeSC dev-lang/python)
), I believe the problem is fixed.
I'd appreciate it, Jason, if you could verify that your picard hasn't magically fixed itself, somehow, and then try this patch in your python installations, and see if the picard issue doesn't disappear as well.
*** Bug 533442 has been marked as a duplicate of this bug. *** Just marked a duplicate, but now I'm not so sure... Tried the patch in comment #22 but still getting the same error installing markdown: >>> Install markdown-2.5.1 into /var/tmp/portage/dev-python/markdown-2.5.1/image/ category dev-python * python3_3: running distutils-r1_run_phase distutils-r1_python_install /usr/bin/python3.3 setup.py install --root=/var/tmp/portage/dev-python/markdown-2.5.1/image//_python3.3 running install running build running build_py running build_scripts running build_docs running install_lib Traceback (most recent call last): File "setup.py", line 249, in <module> 'Topic :: Text Processing :: Markup :: HTML', File "/usr/lib64/python3.3/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib64/python3.3/distutils/dist.py", line 930, in run_commands self.run_command(cmd) File "/usr/lib64/python3.3/distutils/dist.py", line 949, in run_command cmd_obj.run() File "/usr/lib64/python3.3/distutils/command/install.py", line 581, in run self.run_command(cmd_name) File "/usr/lib64/python3.3/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib64/python3.3/distutils/dist.py", line 947, in run_command cmd_obj = self.get_command_obj(command) File "/usr/lib64/python3.3/distutils/dist.py", line 822, in get_command_obj cmd_obj = self.command_obj[command] = klass(self) File "/usr/lib64/python3.3/site-packages/setuptools/__init__.py", line 125, in __init__ _Command.__init__(self,dist) File "/usr/lib64/python3.3/distutils/cmd.py", line 57, in __init__ raise TypeError("dist must be a Distribution instance") TypeError: dist must be a Distribution instance Yes, it is a different bug, with a highly similar etiology. My patch doesn't address it. To sum up, perhaps it's worth considering (In reply to Greg Turner from comment #26) > Yes, it is a different bug, with a highly similar etiology. My patch > doesn't address it. Actually, however, it's so similar, now that I think about it, that perhaps it's sensible to keep #533442 as a duplicate, and to consider this a new sequela of the same bug, for which no work-around currently has been developed. I'll take a look later and see if I can't think of some way to augment my patch to cover this as well. The same bug was mentioned by folks on the python (... or was it setuputils?) bugs. There is a lot of room here for everybody to decide that this is somebody else's bug. In my experience, when that is the case, it literally almost always occurs, and tends to go on for years. If somebody wants to prove me wrong, great, color me pleasantly surprised and kudos to whatever project took some proactive steps to make things go smoothly for end users. But, IMO, Gentoo should not wait for upstream fixes to these problems. Instead it should develop a hopefully-good-enough-for-now solution, deploy it to end-users, cross it's fingers and pray. Of course, if it didn't work, or caused regressions, we'd just have to deal with that when it happens, that is the downside of shooting first and asking questions later. Even so, in this case, I feel it's the right way to go. Carrying patches that upstream wants no business with is a really shitty position to be in as a distro; it means forward-porting the patch on any future release, even in small bug-fix-only releases, and forward-porting is often not trivial (particularly not in the case of the whole distutils/setuptools shebang). Meanwhile, we have to do serious effort on each new release; and I'd posit that our packaging gets delayed superlinearly for the amount of effort involved in doing the work. So, while there may be cases where carrying a patch upstream doesn't want is the right approach, it comes with significant costs that we should also be mindful of. So far no other failures (but still 200+ packages to go) but markdown-2.5* no longer installs correctly (the workaround is to add back the src_prepare sed hack from 2.3.1 and then it installs correctly). Note that markdown 2.5* craps out with or without Greg's patch (and I'm still running a deep world update with the patch applied and no problems so far). (In reply to Dirkjan Ochtman from comment #29) > Carrying patches that upstream wants no business with is a really shitty > position to be in as a distro Well this is all hypothetical. If upstreams are responsive, great, problem solved. Anyhow, I don't think your description quite captures what I'm predicting. My prediction is not that upstream(s) will reject these contributions, but that everybody will say "yes, somebody really ought to look at this eventually and figure out whose bug this is..." and nothing will happen. The ebuilds are broken, right now. Obviosly we have nothing to lose by engineering fixes to the problems and contributing our fixes to the appropriate upstream(s).... sort of like bring a horse to water is never a bad idea. But, until the horse drinks, we also have very little to lose, given that the result right now in gx86 is an ebuild crash, by fixing the problem in Gentoo, so our users can go on with their work and stop squinting at this bug. As for the maintenance burden... my one-line patch applies to all python versions since at least 2.7. That's about ten years of python history. If, ten years from now, upstreams still haven't gotten around to fixing this, and something breaks, then, I suppose you can say "I told you so" -- but then, so could I :) Once upstreams take care of it, we just drop our patch. No big deal. They surely will, sooner or later, or else the sun will swallow the earth -- either way, this problem is not eternal -- but I predict it might not be short-lived either. (In reply to Greg Turner from comment #32) > As for the maintenance burden... my one-line patch applies to all python > versions since at least 2.7. That's about ten years of python history. This is a bit hyperbolic. It's probably a bit less than ten years (although I wouldn't be surprised if my patch still applied and worked fine in python 2.2), and, presumably my patch will be a bit bigger if I can address the bug #533442 issue as well. The issue should be resolved in the next setuptools release. https://bitbucket.org/pypa/setuptools/issue/311 The change in behavior was that pkg_resouces was accidentally changed to import setuptools, which caused setuptools to get imported unintentionally by many packages. (In reply to Mike Gilbert from comment #34) > The issue should be resolved in the next setuptools release. > > https://bitbucket.org/pypa/setuptools/issue/311 > > The change in behavior was that pkg_resouces was accidentally changed to > import setuptools, which caused setuptools to get imported unintentionally > by many packages. Great news. Sounds like this will return us to how things were before -- mostly working OOTB. For the convenience of any hypothetical backporters or casual work-around seekers AFAICS: https://bitbucket.org/pypa/setuptools/commits/5092bf199e11fb243ab17de33461fea1c6d54ce2/raw/, https://bitbucket.org/pypa/setuptools/commits/e2181ecd6ccbc67b366a968d968b785b2941b31f/raw/, and https://bitbucket.org/pypa/setuptools/commits/f0cc850450fdfb6529676b8aebcf9256bb295700/raw/ are the corresponding commits. Sorry I haven't gotten to it yet, but I do still intend to try to rev "distutils_accomodate_extension_ducktypes.patch" with a corresponding fix for the Command problem (bug #533442) -- assuming, of course, that I can find a reasonable, low-impact, and frankly low-effort way to do so. Created attachment 392636 [details, diff]
distutils_accomodate_distribution_ducktypes.patch
Here as promised is the same trick for the very similar bug that's been cropping up, due to distutils.cmd.Command.__init__ checking isinstance(dist, Distribution).
This time, I had to finesse the patch a bit to make it apply in all pythons, but, it does. Same recipe (put it in /etc/portage/patches/dev-lang/python alongside the other one, and emerge -1a $(qlist -ICSe dev-lang/python) should apply.
Note to folks testing this -- I haven't ported either patch to pypy, so pypy will still suffer from these issues until that changes. Therefore please test with USE=-python_targets_pypy if pypy is in your PYTHON_TARGETS.
I just added dev-python/setuptools-9.1 to the tree. Please confirm that this resolves the issue with pyzmq. Regarding your patches for the distutils package: while I do appreciate the effort, we are probably not going to carry them as distro-specific. Please spend that effort getting them applied upstream. (In reply to Mike Gilbert from comment #37) > I just added dev-python/setuptools-9.1 to the tree. Please confirm that this > resolves the issue with pyzmq. > > Regarding your patches for the distutils package: while I do appreciate the > effort, we are probably not going to carry them as distro-specific. Please > spend that effort getting them applied upstream. Can't confirm or disconfirm yet. Will endeavour to guinea-pig this for you within the next 24 hours. (In reply to Mike Gilbert from comment #37) > I just added dev-python/setuptools-9.1 to the tree. Please confirm that this > resolves the issue with pyzmq. > > Regarding your patches for the distutils package: while I do appreciate the > effort, we are probably not going to carry them as distro-specific. Please > spend that effort getting them applied upstream. I assure you I have zero or, really, negative personal attachment to that outcome, but thanks for the encouraging words. Assuming the immediate "crisis" is resolved in ~amd64, as far as I'm concerned, I've "done my bit" -- upstream has my patches; they presumably know best whether to pipe them to /dev/null or /usr/bin/hg. + 30 Dec 2014; Mike Gilbert <floppym@gentoo.org> pyzmq-14.3.1.ebuild, + pyzmq-14.4.0.ebuild, pyzmq-14.4.1.ebuild: + Remove workaround for bug 532708; issue is resolved in setuptools-9.1. (In reply to Mike Gilbert from comment #40) > + 30 Dec 2014; Mike Gilbert <floppym@gentoo.org> pyzmq-14.3.1.ebuild, > + pyzmq-14.4.0.ebuild, pyzmq-14.4.1.ebuild: > + Remove workaround for bug 532708; issue is resolved in setuptools-9.1. A bit late, sorry, but I can now confirm that this W4M. Thanks all! |
