Bug 390735 – gnutls-2.10.5 misuses libgcrypt (1.5.0), causing random failures (libvirt, probably others)

• Product: Gentoo Linux
• Status: RESOLVED OBSOLETE

[Description Marien Zwart (RETIRED) 2011-11-16 14:07:11 UTC]

I caught this failure via libvirt's testsuite (virnettlscontexttest fails), but it also affects the gnutls testsuite (x509sign-verify fails), and I'm expecting it to cause random failures in many applications using it.

The bug is in pk-libgcrypt.c, which repeatedly calls gcry_sexp_nth_mpi with a third argument of 0. That third argument is documented in libgcrypt's info documentation as "If you use this function to parse results of a public key function, you most likely want to use `GCRYMPI_FMT_USG'.". libgcrypt interprets 0 as GCRYMPI_FMT_STD, which seems to be the same as GCRYMPI_FMT_USG but signed. The version of libgcrypt I have here (1.5.0-r1) has a FIXME for actually handling negative GCRYMPI_FMT_STD numbers (in mpicoder.c:gcry_mpi_scan). The effect of all this is that about half the calls from gnutls to gcry_sexp_nth_mpi incorrectly fail.

Changing the third argument of all calls to this function from 0 to GCRYMPI_FMT_USG causes the gnutls and libvirt testsuites to pass. I have not yet checked if it fixes other previously failing tests on this system. Ideally this fix would be run by upstream, but bug 389947 mentions upstream not caring about 2.10 anymore, and this code is not the same in gnutls 2.12. However, gnutls 2.12 does still call gcry_sexp_nth_mpi, and always uses a third argument of GCRYMPI_FMT_USG when doing so. The NEWS file mentions this fixing bugs with 1.5.0.

Please consider either applying the attached patch, restricting the libgcrypt dependency to a version gnutls passes its tests with, or unmasking a more recent version of gnutls.

Reproducible: Always

Steps to Reproduce:
1. install libgcrypt-1.5.0-r1
2. emerge gnutls or libvirt with FEATURES=test
Actual Results:  
tests fail

Expected Results:  
tests pass

Portage 2.2.0_alpha74 (default/linux/amd64/10.0/desktop/gnome, gcc-4.5.3, glibc-2.13-r4, 3.1.1-gentoo-m4-new-idmapper x86_64)
=================================================================
System uname: Linux-3.1.1-gentoo-m4-new-idmapper-x86_64-AMD_Athlon-tm-_II_X4_640_Processor-with-gentoo-2.1
Timestamp of tree: Wed, 16 Nov 2011 08:45:01 +0000
ccache version 3.1.6 [enabled]
app-shells/bash:          4.2_p10
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.6.7-r2, 2.7.2-r3, 3.1.4-r3, 3.2.2
dev-util/ccache:          3.1.6
dev-util/cmake:           2.8.6-r3
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.1
sys-apps/openrc:          0.9.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.10.3, 1.11.1-r1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.3-r1
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4.2
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo cross gamerlay-stable gnome bitcoin marienz
Installed sets: 
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA AdobeFlash-10.1 google-talkplugin"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-ggdb -O2 -march=amdfam10 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-ggdb -O2 -march=amdfam10 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs cached-virtuals candy ccache collision-protect distlocks ebuild-locks fixlafiles news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict test unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox verify-rdepend"
FFLAGS=""
GENTOO_MIRRORS="http://ftp.snt.utwente.nl/pub/linux/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_GB en_US nl fy fy_NL"
MAKEOPTS="-j4"
PKGDIR="/var/tmp/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage/cross /usr/local/portage/layman/gamerlay /usr/local/portage/layman/gnome /usr/local/portage/layman/bitcoin /usr/local/portage/private"
SYNC="rsync://rsync.nl.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac acpi alsa amd64 apng avahi bluetooth branding bzip2 cairo caps cdda cdr cli clutter colord consolekit crypt cups cxx dbus device-mapper doc dri dts dvd dvdr dvi emacs emboss encode equalizer evo exif expat ffmpeg firefox flac fontconfig fortran fuse gif gmp gnome gnome-keyring gpg gstreamer gtk gtk3 gtkstyle iconv idn imap inotify ipv6 irc jpeg lame latex lcms libcaca libffi libkms libnotify llvm lua lvm lzma macvtap maildir mmx mmxext mng modules mp3 mp4 mpeg multilib nautilus ncurses nfsidmap nls nptl nptlonly ntfsprogs objc offensive ogg opengl openmp pam pam_krb5 pango pch pdf playlist png policykit ppds pppd preview-latex pulseaudio python qemu qt3support qt4 readline realtime session sip speex spell spice sqlite sse sse2 sse3 ssl startup-notification svg sysfs test theora tokyocabinet toolkit-scroll-bars tools truetype udev udis86 unicode urwid usb vala vhost-net vorbis vpx wxwidgets x264 xattr xcb xcomposite xft xinerama xml xorg xulrunner xv xvfb xvmc zlib zsh-completion" ALSA_CARDS="hda_intel" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DRACUT_MODULES="plymouth caps lvm" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_GB en_US nl fy fy_NL" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" SANE_BACKENDS="plustek plustek_pp" USERLAND="GNU" VIDEO_CARDS="radeon r600" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 1 Marien Zwart (RETIRED) 2011-11-16 14:09:07 UTC]

Created attachment 292759 [details, diff]
fix libgcrypt api usage

whoops, forgot to attach the patch :(

[Comment 2 Arfrever Frehtes Taifersar Arahesis 2011-11-16 17:31:58 UTC]

(In reply to comment #0)
> Please consider either applying the attached patch, restricting the libgcrypt
> dependency to a version gnutls passes its tests with, or unmasking a more
> recent version of gnutls.

I suggest unmasking of GnuTLS 2.12.

[Comment 3 Doug Goldstein (RETIRED) 2012-03-08 00:08:58 UTC]

Please add a blocker in gnutls-2.10.x against libgcrypto-1.5.0 or make it depend on versions of libgcrypto less than 1.5.0 so others don't have this same issue.

[Comment 4 Tim Harder 2012-05-10 05:03:47 UTC]

Closing as obsolete since gnutls-2.10* is no longer in the tree. Reopen if it's still an issue with later versions.