Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 615620

Summary: sys-apps/portage-2.3.16 with dev-python/pygcrypt-1.0.0 - constant runtime crashes: raise errors.GcryptException(ffi.string(lib.gcry_strerror(error)), error) pygcrypt.errors.GcryptException: (b'Cannot allocate memory', 16810070)
Product: Portage Development Reporter: Helmut Jarausch <jarausch>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: mgorny, toralf
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 637452    

Description Helmut Jarausch 2017-04-15 09:11:18 UTC
Installing dev-python/pygcrypt-1.0.0 did work.

BUT from then on, any emerge (except with option -C) crashes - see below.

This is with the GIT version of portage.

And here an example:

ionice -c3 nice -19 emerge -auv1 -j4 --keep-going --verbose-conflicts \
>      app-text/pdfgrep \
>      dev-python/numpy dev-python/python-stdnum \
>      sys-devel/make 

These are the packages that would be merged, in order:

Calculating dependencies... done!
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.6/emerge", line 50, in <module>
    retval = emerge_main()
  File "/usr/lib64/python3.6/site-packages/_emerge/main.py", line 1229, in emerge_main
    return run_action(emerge_config)
  File "/usr/lib64/python3.6/site-packages/_emerge/actions.py", line 3281, in run_action
    retval = action_build(emerge_config, spinner=spinner)
  File "/usr/lib64/python3.6/site-packages/_emerge/actions.py", line 339, in action_build
    settings, trees, myopts, myparams, myaction, myfiles, spinner)
  File "/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 9389, in backtrack_depgraph
    myaction, myfiles, spinner)
  File "/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 9426, in _backtrack_depgraph
    success, favorites = mydepgraph.select_files(myfiles)
  File "/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 3681, in select_files
    return self._select_files(args)
  File "/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 3690, in _select_files
    self._load_vdb()
  File "/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 618, in _load_vdb
    scheduler.wait()
  File "/usr/lib64/python3.6/site-packages/_emerge/AsynchronousTask.py", line 54, in wait
    self._wait()
  File "/usr/lib64/python3.6/site-packages/portage/util/_async/AsyncScheduler.py", line 85, in _wait
    self._event_loop.iteration()
  File "/usr/lib64/python3.6/site-packages/portage/util/_eventloop/EventLoop.py", line 319, in iteration
    if not x.callback(f, event, *x.args):
  File "/usr/lib64/python3.6/site-packages/_emerge/EbuildMetadataPhase.py", line 151, in _output_handler
    self._unregister_if_appropriate(event)
  File "/usr/lib64/python3.6/site-packages/_emerge/AbstractPollTask.py", line 135, in _unregister_if_appropriate
    self.wait()
  File "/usr/lib64/python3.6/site-packages/_emerge/AsynchronousTask.py", line 57, in wait
    self._wait_hook()
  File "/usr/lib64/python3.6/site-packages/_emerge/AsynchronousTask.py", line 175, in _wait_hook
    self._exit_listener_stack.pop()(self)
  File "/usr/lib64/python3.6/site-packages/portage/util/_async/AsyncScheduler.py", line 64, in _task_exit
    self._schedule()
  File "/usr/lib64/python3.6/site-packages/_emerge/PollScheduler.py", line 154, in _schedule
    self._schedule_tasks()
  File "/usr/lib64/python3.6/site-packages/portage/util/_async/AsyncScheduler.py", line 55, in _schedule_tasks
    task.start()
  File "/usr/lib64/python3.6/site-packages/_emerge/AsynchronousTask.py", line 30, in start
    self._start()
  File "/usr/lib64/python3.6/site-packages/_emerge/EbuildMetadataPhase.py", line 119, in _start
    fd_pipes=fd_pipes, returnpid=True)
  File "/usr/lib64/python3.6/site-packages/portage/package/ebuild/doebuild.py", line 708, in doebuild
    mf.checkFileHashes("EBUILD", os.path.basename(myebuild))
  File "/usr/lib64/python3.6/site-packages/portage/manifest.py", line 614, in checkFileHashes
    ok, reason = verify_all(self._getAbsname(ftype, fname), digests)
  File "/usr/lib64/python3.6/site-packages/portage/checksum.py", line 455, in verify_all
    myhash = perform_checksum(filename, x, calc_prelink=calc_prelink)[0]
  File "/usr/lib64/python3.6/site-packages/portage/checksum.py", line 513, in perform_checksum
    myhash, mysize = hashfunc_map[hashname].checksum_file(myfilename)
  File "/usr/lib64/python3.6/site-packages/portage/checksum.py", line 87, in checksum_file
    checksum = self._hashobject()
  File "/usr/lib64/python3.6/site-packages/portage/checksum.py", line 149, in __init__
    self._obj = pygcrypt.hashcontext.HashContext(algo=algo)
  File "/usr/lib64/python3.6/site-packages/pygcrypt/hashcontext.py", line 54, in __init__
    raise errors.GcryptException(ffi.string(lib.gcry_strerror(error)), error)
pygcrypt.errors.GcryptException: (b'Cannot allocate memory', 16810070)


11 # emerge --info dev-python/pygcrypt
Portage 2.3.5 (python 3.6.1-final-0, default/linux/amd64/13.0/desktop, gcc-6.3.0, glibc-2.25, 4.11.0-rc6 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.11.0-rc6-x86_64-AMD_Phenom-tm-_II_X4_940_Processor-with-gentoo-2.3
KiB Mem:     7903440 total,     80996 free
KiB Swap:    4194300 total,   4194300 free
Timestamp of repository gentoo: Sat, 15 Apr 2017 07:45:01 +0000
sh bash 4.4_p12
ld GNU ld (Gentoo 2.27 p1.0) 2.27
ccache version 3.3.4 [disabled]
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r3::local
dev-lang/perl:            5.24.1-r1::gentoo
dev-lang/python:          2.7.13::gentoo, 3.5.3::gentoo, 3.6.1::gentoo
dev-util/ccache:          3.3.4::gentoo
dev-util/cmake:           3.8.0::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/openrc:          0.24.2::gentoo
sys-apps/sandbox:         2.10-r4::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r2::gentoo
sys-devel/automake:       1.9.6-r4::gentoo, 1.10.3-r2::gentoo, 1.11.6-r2::gentoo, 1.12.6-r1::gentoo, 1.13.4-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.27::gentoo
sys-devel/gcc:            5.4.0-r3::gentoo, 6.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r4::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.10::gentoo (virtual/os-headers)
sys-libs/glibc:           2.25::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.informatik.rwth-aachen.de/gentoo-portage
    priority: -1000

java
    location: /usr/local/portage/layman/java
    sync-type: git
    sync-uri: git://anongit.gentoo.org/proj/java.git
    masters: gentoo
    priority: 50

lisp
    location: /usr/local/portage/layman/lisp
    sync-type: git
    sync-uri: git://anongit.gentoo.org/proj/lisp.git
    masters: gentoo
    priority: 50

nx
    location: /usr/local/portage/layman/nx
    sync-type: laymansync
    sync-uri: https://anongit.gentoo.org/git/proj/nx.git
    masters: gentoo
    priority: 50

octave
    location: /usr/local/portage/layman/octave
    sync-type: git
    sync-uri: git://github.com/rafaelmartins/octave-overlay.git
    masters: gentoo
    priority: 50

palemoon
    location: /usr/local/portage/layman/palemoon
    sync-type: git
    sync-uri: https://github.com/deuiore/palemoon-overlay.git
    masters: gentoo
    priority: 50

science
    location: /usr/local/portage/layman/science
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/proj/sci.git
    masters: gentoo
    priority: 50

seden
    location: /usr/local/portage/layman/seden
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/user/seden.git
    masters: gentoo
    priority: 50

local
    location: /usr/local/portage
    masters: gentoo
    priority: 99

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-mtune=native -O2 -msse3 -pipe -fPIC"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.1/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-mtune=native -O2 -msse3 -pipe -fPIC"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/gentoo "
LANG="en_US.iso88591"
LC_ALL=""
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="3dnow 3dnowext 3dnowprefetch X a52 aac acl acpi alsa amd64 avahi berkdb branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam ffmpeg fftw firefox flac fortran fuse gcj gdbm gfortran gif gimp glamor gpm gtk gtk3 iconv ipv6 jpeg lapack lcms libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl nptlonly ogg opengl openmp pam pango pcre pdf png policykit ppds python qt qt3support qt4 qt5 readline sdl seccomp session smp spell sqlite sqlite3 sse sse2 sse3 sse4a ssl startup-notification svg tcl tcpd threads tiff tk truetype udev udisks unicode upower usb vorbis wxwidgets x264 xattr xcb xml xulrunner xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow mmx 3dnowext mmxext popcnt sse sse2 sse3 sse4a" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc multiboot" INPUT_DEVICES="keyboard mouse evdev wacom" KERNEL="linux" L10N="en de" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en de" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="radeon radeonsi r600" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

After removing dev-python/pygcrypt with 'emerge -C' (which still worked)
any further emerge works just find.

Thanks for looking into this,
Helmut
Comment 1 Zac Medico gentoo-dev 2017-04-16 16:16:10 UTC
Do the tests pass if you build dev-python/pygcrypt-1.0.0 with FEATURES=test enabled?
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-04-16 19:27:08 UTC
Upstream is already working on this. It's a regression in pygcrypt 1.0.0.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-12-05 16:28:56 UTC
*** Bug 639948 has been marked as a duplicate of this bug. ***
Comment 4 Larry the Git Cow gentoo-dev 2017-12-05 17:37:03 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=6076f4217dc463340547984b4f88551deec0cc55

commit 6076f4217dc463340547984b4f88551deec0cc55
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2017-12-05 16:33:32 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2017-12-05 17:36:46 +0000

    [checksum] Disable pygcrypt backend due to breakage
    
    Closes: https://bugs.gentoo.org/615620
    Reviewed-by: Brian Dolbec <dolsen@gentoo.org>

 pym/portage/checksum.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Comment 5 Larry the Git Cow gentoo-dev 2023-03-21 02:30:29 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=992cf6e5ecbc8de428792fe27bb821d04e03097c

commit 992cf6e5ecbc8de428792fe27bb821d04e03097c
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-03-21 02:29:34 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-03-21 02:30:24 +0000

    NEWS: update for checksum cleanups
    
    Bug: https://bugs.gentoo.org/597736
    Bug: https://bugs.gentoo.org/615620
    Bug: https://bugs.gentoo.org/885909
    Bug: https://bugs.gentoo.org/888884
    Signed-off-by: Sam James <sam@gentoo.org>

 NEWS | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=d8f9c763a180b5b014b333791dd7235cbf923d31

commit d8f9c763a180b5b014b333791dd7235cbf923d31
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-03-20 03:24:39 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-03-21 02:30:23 +0000

    checksum: drop pygcrypt support
    
    No longer in Gentoo and the codepath for this has been disabled since 2017(!)
    because of https://bugs.gentoo.org/615620.
    
    The last release on pypi is also from April 2017.
    
    Bug: https://bugs.gentoo.org/615620
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/checksum.py | 46 ----------------------------------------------
 1 file changed, 46 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=377a31434ae51b6e6465ba48b5132eb200a8cf40

commit 377a31434ae51b6e6465ba48b5132eb200a8cf40
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-03-20 03:16:37 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-03-21 02:30:23 +0000

    checksum: drop STREEBOG{256,512} & pygost support
    
    For STREEBOG{256,512}, we had three options that we "supported":
    
    1. using pygost;
    
       pygost is unpackaged in Gentoo and appears to, as of March 2023, been yanked
       from pypi too.
    
       Hence not only can we test it on the distribution where we do our development
       (Gentoo), but we also can't acquire it for CI easily either.
    
    2. using pygcrypt;
    
       No longer in Gentoo and the codepath for this has been disabled since 2017(!)
       because of https://bugs.gentoo.org/615620.
    
    3. a slow fallback implementation.
    
       This is not something we can rely on anyway, but this is especially wrong
       given the issues we've had recently with WHIRLPOOL. See https://bugs.gentoo.org/885909.
    
    As for whether we should support STREEBOG at all:
    - Per above, we don't have a decent implementation for it available.
    - We're not testing it very well.
    - Jugging obscure hash algorithms has become a pain because some of them have
      ended up being marked "legacy" or removed from OpenSSL by default.
    - I don't think we should have hash algorithms supported just for the sake of it.
      We're not in the business of supporting every single hash algorithm that a
      library with Python bindings out there provides.
    
    Bug: https://bugs.gentoo.org/597736
    Bug: https://bugs.gentoo.org/615620
    Bug: https://bugs.gentoo.org/885909
    Bug: https://bugs.gentoo.org/888884
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/checksum.py                 | 25 +------------------------
 lib/portage/tests/util/test_checksum.py | 26 --------------------------
 tox.ini                                 |  1 -
 3 files changed, 1 insertion(+), 51 deletions(-)