Summary: | dev-python/pycparser-2.18 breaks Portage "ebuild" command (ImportError: cannot import name '_pycparser') | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Karl-Johan Karlsson <creideiki+gentoo-bugzilla> |
Component: | Current packages | Assignee: | Python Gentoo Team <python> |
Status: | RESOLVED OBSOLETE | ||
Severity: | normal | CC: | dev-portage, drobbins, gyakovlev, ikelos, mgorny, python, saintdev, toralf, zmedico |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: |
https://github.com/eliben/pycparser/pull/198 https://bugs.gentoo.org/show_bug.cgi?id=667764 |
||
Whiteboard: | |||
Package list: | Runtime testing required: | --- |
Description
Karl-Johan Karlsson
2017-08-20 16:49:48 UTC
it also breaks env-update sys-apps/portage-2.3.8::gentoo was built with the following: USE="(ipc) native-extensions xattr -build -doc -epydoc (-selinux)" ABI_X86="(64)" LINGUAS="ru" PYTHON_TARGETS="python2_7 python3_4 -pypy -python3_5 -python3_6" # env-update Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in <module> from . import _pycparser as pycparser ImportError: cannot import name '_pycparser' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python-exec/python3.4/env-update", line 35, in <module> portage.env_update(makelinks) File "/usr/lib64/python3.4/site-packages/portage/proxy/objectproxy.py", line 30, in __call__ result = object.__getattribute__(self, '_get_target')() File "/usr/lib64/python3.4/site-packages/portage/proxy/lazyimport.py", line 130, in _get_target __import__(name) File "/usr/lib64/python3.4/site-packages/portage/util/env_update.py", line 15, in <module> from portage.checksum import prelink_capable File "/usr/lib64/python3.4/site-packages/portage/checksum.py", line 195, in <module> from Crypto.Hash import BLAKE2b File "/usr/lib64/python3.4/site-packages/Crypto/Hash/BLAKE2b.py", line 64, in <module> from Crypto.Util._raw_api import (load_pycryptodome_raw_lib, File "/usr/lib64/python3.4/site-packages/Crypto/Util/_raw_api.py", line 39, in <module> ffi = FFI() File "/usr/lib64/python3.4/site-packages/cffi/api.py", line 63, in __init__ from . import cparser File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 7, in <module> import pycparser File "/usr/lib64/python3.4/site-packages/pycparser/__init__.py", line 14, in <module> from .c_parser import CParser File "/usr/lib64/python3.4/site-packages/pycparser/c_parser.py", line 20, in <module> class CParser(PLYParser): File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 95, in template _create_param_rules(cls, method) File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 112, in _create_param_rules param_rule.__doc__ = func.__doc__.replace('xxx', xxx).replace('yyy', yyy) AttributeError: 'NoneType' object has no attribute 'replace' Confirm, back to =dev-python/pycparser-2.17 and looks fine. *** Bug 628550 has been marked as a duplicate of this bug. *** On #628550, we've also determined gcc6+ might be related (which would also explain why I don't suffer from it). (In reply to Georgy Yakovlev from comment #1) > it also breaks env-update > > sys-apps/portage-2.3.8::gentoo was built with the following: > USE="(ipc) native-extensions xattr -build -doc -epydoc (-selinux)" > ABI_X86="(64)" LINGUAS="ru" PYTHON_TARGETS="python2_7 python3_4 -pypy > -python3_5 -python3_6" > > # env-update > Traceback (most recent call last): > File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in > <module> > from . import _pycparser as pycparser > ImportError: cannot import name '_pycparser' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/lib/python-exec/python3.4/env-update", line 35, in <module> > portage.env_update(makelinks) > File "/usr/lib64/python3.4/site-packages/portage/proxy/objectproxy.py", > line 30, in __call__ > result = object.__getattribute__(self, '_get_target')() > File "/usr/lib64/python3.4/site-packages/portage/proxy/lazyimport.py", > line 130, in _get_target > __import__(name) > File "/usr/lib64/python3.4/site-packages/portage/util/env_update.py", line > 15, in <module> > from portage.checksum import prelink_capable > File "/usr/lib64/python3.4/site-packages/portage/checksum.py", line 195, > in <module> > from Crypto.Hash import BLAKE2b > File "/usr/lib64/python3.4/site-packages/Crypto/Hash/BLAKE2b.py", line 64, > in <module> > from Crypto.Util._raw_api import (load_pycryptodome_raw_lib, > File "/usr/lib64/python3.4/site-packages/Crypto/Util/_raw_api.py", line > 39, in <module> > ffi = FFI() > File "/usr/lib64/python3.4/site-packages/cffi/api.py", line 63, in __init__ > from . import cparser > File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 7, in > <module> > import pycparser > File "/usr/lib64/python3.4/site-packages/pycparser/__init__.py", line 14, > in <module> > from .c_parser import CParser > File "/usr/lib64/python3.4/site-packages/pycparser/c_parser.py", line 20, > in <module> > class CParser(PLYParser): > File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 95, > in template > _create_param_rules(cls, method) > File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line > 112, in _create_param_rules > param_rule.__doc__ = func.__doc__.replace('xxx', xxx).replace('yyy', yyy) > AttributeError: 'NoneType' object has no attribute 'replace' This backtrace happens on the following function: <function CParser.p_direct_xxx_declarator_1 at 0x7f68d0f559d8> @parameterized(('id', 'ID'), ('typeid', 'TYPEID'), ('typeid_noparen', 'TYPEID')) def p_direct_xxx_declarator_1(self, p): """ direct_xxx_declarator : yyy """ My guess is docstring not present due to decorator addition in pycparser-2.17->2.18. Whatever that decorator does it should populate docstring. Or _create_param_rules should learn to ignore docstring. > This backtrace happens on the following function: > <function CParser.p_direct_xxx_declarator_1 at 0x7f68d0f559d8> > > @parameterized(('id', 'ID'), ('typeid', 'TYPEID'), ('typeid_noparen', > 'TYPEID')) > def p_direct_xxx_declarator_1(self, p): > """ direct_xxx_declarator : yyy > """ > > My guess is docstring not present due to decorator addition in > pycparser-2.17->2.18. Whatever that decorator does it should populate > docstring. > > Or _create_param_rules should learn to ignore docstring. Having looked at it a bit more @parameterized decorator does not create a new function. It only attaches new slots to existing function thus __doc__ should be present here. This has something to do with portage's top-level module loading: https://github.com/gentoo/portage/blob/master/pym/portage/checksum.py#L193 'portage.checksum' module breaks only under ObjectProxy/LazyImport. Zac, is it obvious for you why it breaks? To clarify the reproducer is: - pick vanilla ~amd64 image - $ emerge -v1 pycryptodome pycparser - $ portageq best_visible / x11-misc/kapow If you do it on real system make sure portage uses python3.4 as python interpreter and you don't have pygcrypto, pygcrypt installed to get to the pycryptodome line: https://github.com/gentoo/portage/blob/master/pym/portage/checksum.py#L195 Ok, I finally can repro with Python 2.7&3.4. Newer versions work fine. No clue what's the cause though. (In reply to Michał Górny from comment #9) > Ok, I finally can repro with Python 2.7&3.4. Newer versions work fine. No > clue what's the cause though. #c5 and #c6 have some hints on how addition of decorator in pycparser-2.18 broke __doc__ population for p_direct_xxx_declarator_1 when portage lazy-loads it. Ok, we've been able to establish it's related to -OO dropping docstrings. Easy way to repro: python -OO -c 'import Crypto.Hash.BLAKE2b' AFAICS upstream is aware of the issue and they're like *shrug, we don't support -OO, kthxbai*. I've read the upstream discussion more closely, and it seems that they've applied a partial workaround for this [1] which replaces this specific error with a warning. However, it seems to cause another failure which -- according to the discussion -- is because we don't have the pregenerated tables installed. I'm going to see to verify this and possibly fix pycparser ebuild. However, the root of the issue is in poor design of dev-python/ply whose upstream determined that docstrings are a great place to describe syntax... [1]:https://github.com/eliben/pycparser/commit/673accec311a027c22b0718d753f8da922915305 I've pushed two related fixes. However, they don't seem to be enough to fix the issue for me. Maybe someone else will be able to figure out what else needs to be done from here. commit d1e3eb8faa09741a67cc56a0ff6d5332a7bc6068 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: Sun Aug 27 20:07:13 2017 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: Sun Aug 27 20:07:49 2017 dev-python/pycparser: Backport upstream -OO patch, #628386 commit d2a9def05500f4e6f403ff56ad9cc54aede11ee7 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: Sun Aug 27 19:59:40 2017 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: Sun Aug 27 20:07:49 2017 dev-python/pycparser: Enforce regenerating tables Remove the pregenerated tables provided in the distfile to force regenerating them. Generate the tables for every Python implementation separately to avoid incompatibility/underoptimization problems (tables built by Python 3.5+ are incompatible with older Python versions that have 100 group-in-regex limit). env-update and ebuild broke on my system with dev-python/pycparser-2.18. Downgrading to 2.17 fixed it. The error I am getting is this: https://pastebin.com/sb9xUxpj (In reply to Techwolf from comment #15) > env-update and ebuild broke on my system with dev-python/pycparser-2.18. > Downgrading to 2.17 fixed it. The error I am getting is this: > > https://pastebin.com/sb9xUxpj Confirm Due to other issues -- pycparser-2.18 seems to kill net-im/gajim (see #613896)-- I'd recommend masking 2.18 until this and the aforementioned issue is resolved. Cleanup of bugs open for old versions. |