Halfway through building Chromium, the OOM killer killed my GCC. When trying to continue the build process using the "ebuild" command, I immediately got this traceback: # ebuild /usr/portage/www-client/chromium/chromium-61.0.3163.49.ebuild compile install package qmerge Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in <module> from . import _pycparser as pycparser ImportError: cannot import name '_pycparser' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python-exec/python3.4/ebuild", line 246, in <module> portage.portdb._pull_valid_cache(cpv, ebuild, ebuild_portdir)[0] is None File "/usr/lib64/python3.4/site-packages/portage/dbapi/porttree.py", line 474, in _pull_valid_cache if auxdb.validate_entry(metadata, ebuild_hash, eclass_db): File "/usr/lib64/python3.4/site-packages/portage/cache/template.py", line 246, in validate_entry if self._validate_entry(chf_type, entry, ebuild_hash, eclass_db): File "/usr/lib64/python3.4/site-packages/portage/cache/template.py", line 258, in _validate_entry if entry_hash != getattr(ebuild_hash, chf_type): File "/usr/lib64/python3.4/site-packages/portage/eclass_cache.py", line 51, in __getattr__ if hashname not in checksum.get_valid_checksum_keys(): File "/usr/lib64/python3.4/site-packages/portage/proxy/objectproxy.py", line 22, in __getattribute__ result = object.__getattribute__(self, '_get_target')() File "/usr/lib64/python3.4/site-packages/portage/proxy/lazyimport.py", line 107, in _get_target __import__(name) File "/usr/lib64/python3.4/site-packages/portage/checksum.py", line 195, in <module> from Crypto.Hash import BLAKE2b File "/usr/lib64/python3.4/site-packages/Crypto/Hash/BLAKE2b.py", line 64, in <module> from Crypto.Util._raw_api import (load_pycryptodome_raw_lib, File "/usr/lib64/python3.4/site-packages/Crypto/Util/_raw_api.py", line 39, in <module> ffi = FFI() File "/usr/lib64/python3.4/site-packages/cffi/api.py", line 63, in __init__ from . import cparser File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 7, in <module> import pycparser File "/usr/lib64/python3.4/site-packages/pycparser/__init__.py", line 14, in <module> from .c_parser import CParser File "/usr/lib64/python3.4/site-packages/pycparser/c_parser.py", line 20, in <module> class CParser(PLYParser): File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 95, in template _create_param_rules(cls, method) File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 112, in _create_param_rules param_rule.__doc__ = func.__doc__.replace('xxx', xxx).replace('yyy', yyy) AttributeError: 'NoneType' object has no attribute 'replace' Playing around a bit, it seems that "ebuild" dies with that same error every time, and really early as well, before it's actually done anything: # ebuild /usr/portage/www-client/firefox-bin/firefox-bin-55.0.ebuild help Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in <module> from . import _pycparser as pycparser ImportError: cannot import name '_pycparser' "emerge" works, though. I had recently had updates of dev-python/pycparser (2.17 -> 2.18) and sys-apps/portage (2.3.6 -> 2.3.8). Downgrading Portage to 2.3.6 made no difference, but downgrading pycparser to 2.17 made "ebuild" work again. Portage 2.3.6 (python 3.4.6-final-0, hardened/linux/amd64, gcc-6.4.0, glibc-2.25-r4, 4.12.6-gentoo x86_64) ================================================================= System Settings ================================================================= System uname: Linux-4.12.6-gentoo-x86_64-Intel-R-_Core-TM-_i7-2640M_CPU_@_2.80GHz-with-gentoo-2.4.1 KiB Mem: 8115700 total, 2887252 free KiB Swap: 4000180 total, 2603440 free Timestamp of repository gentoo: Sun, 20 Aug 2017 11:00:01 +0000 sh bash 4.4_p12 ld GNU ld (Gentoo 2.27 p1.1) 2.27 app-shells/bash: 4.4_p12::gentoo dev-java/java-config: 2.2.0-r3::gentoo dev-lang/perl: 5.24.2::gentoo dev-lang/python: 2.7.13::gentoo, 3.4.6::gentoo, 3.6.1-r1::gentoo dev-util/cmake: 3.9.1::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.4.1-r1::gentoo sys-apps/openrc: 0.29::gentoo sys-apps/sandbox: 2.10-r4::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.15.1-r1::gentoo sys-devel/binutils: 2.27-r1::gentoo, 2.28-r2::gentoo, 2.28.1::gentoo sys-devel/gcc: 6.4.0::gentoo sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6-r4::gentoo sys-devel/make: 4.2.1-r1::gentoo sys-kernel/linux-headers: 4.10::gentoo (virtual/os-headers) sys-libs/glibc: 2.25-r4::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-extra-opts: --timeout=10 kde-sunset location: /usr/local/overlay/kde-sunset sync-type: git sync-uri: git://anongit.gentoo.org/proj/kde-sunset.git masters: gentoo priority: 0 creideiki location: /usr/local/portage masters: gentoo priority: 1 betagarden location: /var/lib/layman/betagarden sync-type: laymansync sync-uri: git://anongit.gentoo.org/proj/betagarden.git masters: gentoo priority: 50 hasufell location: /var/lib/layman/hasufell sync-type: laymansync sync-uri: https://bitbucket.org/hasufell/hasufell-overlay.git masters: gentoo priority: 50 seden location: /var/lib/layman/seden sync-type: laymansync sync-uri: git://anongit.gentoo.org/user/seden.git masters: gentoo priority: 50 steam-overlay location: /var/lib/layman/steam-overlay sync-type: laymansync sync-uri: git://github.com/anyc/steam-overlay.git masters: gentoo priority: 50 sunrise location: /var/lib/layman/sunrise sync-type: laymansync sync-uri: git://anongit.gentoo.org/proj/sunrise-reviewed.git masters: gentoo priority: 50 torbrowser location: /var/lib/layman/torbrowser sync-type: laymansync sync-uri: git://github.com/MeisterP/torbrowser-overlay.git masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA PUEL dlj-1.1 googleearth AdobeFlash-10.1 AdobeFlash-10.3 AdobeFlash-11.x google-talkplugin skype-eula QUAKE4 Oracle-BCLA-JavaSE Intel-SDP Introversion skype-4.0.0.7-copyright google-chrome AdobeAIRSDK" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -fweb -ftracer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-march=native -O2 -pipe -fweb -ftracer" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--alphabetical --keep-going --quiet-build=n --verbose-conflicts" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--hash-style=gnu -Wl,--enable-new-dtags" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_EXTRA_OPTS="--timeout=10" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac aacs acl acpi alsa amd64 anthy avi bash-completion berkdb bluetooth bluray bzip2 c++0x cairo canna cdr cjk cli consolekit cracklib crypt css cups cvs cxx dbus dri dts dvd dvdnav dvdr dvdread emacs exif ffmpeg fftw flac fontconfig fortran fuse gdbm gif gimp git gles gles1 gles2 glitz gphoto2 gstreamer gtk gtk3 hackrf hal handbook hardened iconv idn ipv6 javascript jingle jpeg justify kde laptop lcms lensfun lm_sensors logrotate mad matroska mmap mmx mmxext mng modules motif mp3 multilib ncurses nls nptl nsplugin offensive ogg opencl opengl openmp pam pax_kernel pcre pdf phonon pic pie plasma png postscript projectm qt qt3 qt3support qt4 quicktime qwt raw readline real resid rtmp s3tc samba sasl sdl seccomp session sid smp sndfile sqlite sse sse2 sse3 sse4_1 ssl ssp ssse3 steamgames_source_engine steamgames_tf2 steamruntime subversion svg tcpd theora tiff truetype unicode urandom usb v4l vaapi vorbis wifi win32codecs windows_games wma wmf x264 xattr xcb xine xinerama xrandr xtpax xv xvid xvmc zlib" ABI_X86="32 64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi krita sheets stage words" CAMERAS="canon" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev wacom" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_4" QEMU_SOFTMMU_TARGETS="i386 x86_64" RUBY_TARGETS="ruby22 ruby23 ruby24" SANE_BACKENDS="pixma hp" USERLAND="GNU" VIDEO_CARDS="intel i915 i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS ================================================================= Package Settings ================================================================= dev-python/pycparser-2.17::gentoo was built with the following: USE="-test" ABI_X86="(64)" PYTHON_TARGETS="-pypy -pypy3 python2_7 python3_4 -python3_5 -python3_6" dev-python/pycparser-2.18::gentoo was built with the following: USE="-test" ABI_X86="(64)" PYTHON_TARGETS="-pypy -pypy3 python2_7 python3_4 -python3_5 -python3_6"
it also breaks env-update sys-apps/portage-2.3.8::gentoo was built with the following: USE="(ipc) native-extensions xattr -build -doc -epydoc (-selinux)" ABI_X86="(64)" LINGUAS="ru" PYTHON_TARGETS="python2_7 python3_4 -pypy -python3_5 -python3_6" # env-update Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in <module> from . import _pycparser as pycparser ImportError: cannot import name '_pycparser' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python-exec/python3.4/env-update", line 35, in <module> portage.env_update(makelinks) File "/usr/lib64/python3.4/site-packages/portage/proxy/objectproxy.py", line 30, in __call__ result = object.__getattribute__(self, '_get_target')() File "/usr/lib64/python3.4/site-packages/portage/proxy/lazyimport.py", line 130, in _get_target __import__(name) File "/usr/lib64/python3.4/site-packages/portage/util/env_update.py", line 15, in <module> from portage.checksum import prelink_capable File "/usr/lib64/python3.4/site-packages/portage/checksum.py", line 195, in <module> from Crypto.Hash import BLAKE2b File "/usr/lib64/python3.4/site-packages/Crypto/Hash/BLAKE2b.py", line 64, in <module> from Crypto.Util._raw_api import (load_pycryptodome_raw_lib, File "/usr/lib64/python3.4/site-packages/Crypto/Util/_raw_api.py", line 39, in <module> ffi = FFI() File "/usr/lib64/python3.4/site-packages/cffi/api.py", line 63, in __init__ from . import cparser File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 7, in <module> import pycparser File "/usr/lib64/python3.4/site-packages/pycparser/__init__.py", line 14, in <module> from .c_parser import CParser File "/usr/lib64/python3.4/site-packages/pycparser/c_parser.py", line 20, in <module> class CParser(PLYParser): File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 95, in template _create_param_rules(cls, method) File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 112, in _create_param_rules param_rule.__doc__ = func.__doc__.replace('xxx', xxx).replace('yyy', yyy) AttributeError: 'NoneType' object has no attribute 'replace'
Confirm, back to =dev-python/pycparser-2.17 and looks fine.
*** Bug 628550 has been marked as a duplicate of this bug. ***
On #628550, we've also determined gcc6+ might be related (which would also explain why I don't suffer from it).
(In reply to Georgy Yakovlev from comment #1) > it also breaks env-update > > sys-apps/portage-2.3.8::gentoo was built with the following: > USE="(ipc) native-extensions xattr -build -doc -epydoc (-selinux)" > ABI_X86="(64)" LINGUAS="ru" PYTHON_TARGETS="python2_7 python3_4 -pypy > -python3_5 -python3_6" > > # env-update > Traceback (most recent call last): > File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 5, in > <module> > from . import _pycparser as pycparser > ImportError: cannot import name '_pycparser' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/lib/python-exec/python3.4/env-update", line 35, in <module> > portage.env_update(makelinks) > File "/usr/lib64/python3.4/site-packages/portage/proxy/objectproxy.py", > line 30, in __call__ > result = object.__getattribute__(self, '_get_target')() > File "/usr/lib64/python3.4/site-packages/portage/proxy/lazyimport.py", > line 130, in _get_target > __import__(name) > File "/usr/lib64/python3.4/site-packages/portage/util/env_update.py", line > 15, in <module> > from portage.checksum import prelink_capable > File "/usr/lib64/python3.4/site-packages/portage/checksum.py", line 195, > in <module> > from Crypto.Hash import BLAKE2b > File "/usr/lib64/python3.4/site-packages/Crypto/Hash/BLAKE2b.py", line 64, > in <module> > from Crypto.Util._raw_api import (load_pycryptodome_raw_lib, > File "/usr/lib64/python3.4/site-packages/Crypto/Util/_raw_api.py", line > 39, in <module> > ffi = FFI() > File "/usr/lib64/python3.4/site-packages/cffi/api.py", line 63, in __init__ > from . import cparser > File "/usr/lib64/python3.4/site-packages/cffi/cparser.py", line 7, in > <module> > import pycparser > File "/usr/lib64/python3.4/site-packages/pycparser/__init__.py", line 14, > in <module> > from .c_parser import CParser > File "/usr/lib64/python3.4/site-packages/pycparser/c_parser.py", line 20, > in <module> > class CParser(PLYParser): > File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line 95, > in template > _create_param_rules(cls, method) > File "/usr/lib64/python3.4/site-packages/pycparser/plyparser.py", line > 112, in _create_param_rules > param_rule.__doc__ = func.__doc__.replace('xxx', xxx).replace('yyy', yyy) > AttributeError: 'NoneType' object has no attribute 'replace' This backtrace happens on the following function: <function CParser.p_direct_xxx_declarator_1 at 0x7f68d0f559d8> @parameterized(('id', 'ID'), ('typeid', 'TYPEID'), ('typeid_noparen', 'TYPEID')) def p_direct_xxx_declarator_1(self, p): """ direct_xxx_declarator : yyy """ My guess is docstring not present due to decorator addition in pycparser-2.17->2.18. Whatever that decorator does it should populate docstring. Or _create_param_rules should learn to ignore docstring.
> This backtrace happens on the following function: > <function CParser.p_direct_xxx_declarator_1 at 0x7f68d0f559d8> > > @parameterized(('id', 'ID'), ('typeid', 'TYPEID'), ('typeid_noparen', > 'TYPEID')) > def p_direct_xxx_declarator_1(self, p): > """ direct_xxx_declarator : yyy > """ > > My guess is docstring not present due to decorator addition in > pycparser-2.17->2.18. Whatever that decorator does it should populate > docstring. > > Or _create_param_rules should learn to ignore docstring. Having looked at it a bit more @parameterized decorator does not create a new function. It only attaches new slots to existing function thus __doc__ should be present here. This has something to do with portage's top-level module loading: https://github.com/gentoo/portage/blob/master/pym/portage/checksum.py#L193
'portage.checksum' module breaks only under ObjectProxy/LazyImport. Zac, is it obvious for you why it breaks?
To clarify the reproducer is: - pick vanilla ~amd64 image - $ emerge -v1 pycryptodome pycparser - $ portageq best_visible / x11-misc/kapow If you do it on real system make sure portage uses python3.4 as python interpreter and you don't have pygcrypto, pygcrypt installed to get to the pycryptodome line: https://github.com/gentoo/portage/blob/master/pym/portage/checksum.py#L195
Ok, I finally can repro with Python 2.7&3.4. Newer versions work fine. No clue what's the cause though.
(In reply to Michał Górny from comment #9) > Ok, I finally can repro with Python 2.7&3.4. Newer versions work fine. No > clue what's the cause though. #c5 and #c6 have some hints on how addition of decorator in pycparser-2.18 broke __doc__ population for p_direct_xxx_declarator_1 when portage lazy-loads it.
Ok, we've been able to establish it's related to -OO dropping docstrings. Easy way to repro: python -OO -c 'import Crypto.Hash.BLAKE2b'
AFAICS upstream is aware of the issue and they're like *shrug, we don't support -OO, kthxbai*.
I've read the upstream discussion more closely, and it seems that they've applied a partial workaround for this [1] which replaces this specific error with a warning. However, it seems to cause another failure which -- according to the discussion -- is because we don't have the pregenerated tables installed. I'm going to see to verify this and possibly fix pycparser ebuild. However, the root of the issue is in poor design of dev-python/ply whose upstream determined that docstrings are a great place to describe syntax... [1]:https://github.com/eliben/pycparser/commit/673accec311a027c22b0718d753f8da922915305
I've pushed two related fixes. However, they don't seem to be enough to fix the issue for me. Maybe someone else will be able to figure out what else needs to be done from here. commit d1e3eb8faa09741a67cc56a0ff6d5332a7bc6068 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: Sun Aug 27 20:07:13 2017 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: Sun Aug 27 20:07:49 2017 dev-python/pycparser: Backport upstream -OO patch, #628386 commit d2a9def05500f4e6f403ff56ad9cc54aede11ee7 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: Sun Aug 27 19:59:40 2017 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: Sun Aug 27 20:07:49 2017 dev-python/pycparser: Enforce regenerating tables Remove the pregenerated tables provided in the distfile to force regenerating them. Generate the tables for every Python implementation separately to avoid incompatibility/underoptimization problems (tables built by Python 3.5+ are incompatible with older Python versions that have 100 group-in-regex limit).
env-update and ebuild broke on my system with dev-python/pycparser-2.18. Downgrading to 2.17 fixed it. The error I am getting is this: https://pastebin.com/sb9xUxpj
(In reply to Techwolf from comment #15) > env-update and ebuild broke on my system with dev-python/pycparser-2.18. > Downgrading to 2.17 fixed it. The error I am getting is this: > > https://pastebin.com/sb9xUxpj Confirm
Due to other issues -- pycparser-2.18 seems to kill net-im/gajim (see #613896)-- I'd recommend masking 2.18 until this and the aforementioned issue is resolved.
Cleanup of bugs open for old versions.
