Bug 590236

Summary:	app-arch/pbzip2-1.1.12/-1.1.13 fails to decompress files at random points returns -4
Product:	Gentoo Linux	Reporter:	deference
Component:	Current packages	Assignee:	No maintainer - Look at https://wiki.gentoo.org/wiki/Project:Proxy_Maintainers if you want to take care of it <maintainer-needed>
Status:	CONFIRMED ---
Severity:	normal	CC:	hydrapolic
Priority:	Normal
Version:	unspecified
Hardware:	Other
OS:	Linux
See Also:	https://launchpad.net/bugs/1783730 https://bugs.gentoo.org/show_bug.cgi?id=888777
Whiteboard:
Package list:		Runtime testing required:	---

Description deference 2016-08-01 19:11:04 UTC

Hello,
I can't seem to create an account at lauchpad so I'm reporting this here.
I downloaded the openstreetmap planet bz2 file and am trying to decompress it so that I can get an exact size for the decompressed file.
pbzip2 -vcdp6 --ignore-trailing-garbage=1 > pv -rab > /dev/null
It aborts weather or not --ignore-trailing-garbage is set.
It aborts at random points complaining that one of the threads returns -4.
The archive md5sum is good.
I've not finish decompressing the file yet, but bzip2 decompresses twice as much data as pbzip2 without error.
If you need any extra data just ask.

Comment 1 Tomáš Mózes 2018-07-25 08:47:38 UTC

Got the same error on 1.1.12 and 1.1.13. Ordinary bzip2 can decompress the file without problems. I was unable to find any open issue on the net regarding this, nor some patches on debian/fedora/arch :-(

(a 40GB bzip2 file)
# pbzip2 -cd data.sql.bz2 > /dev/null
pbzip2: *ERROR during BZ2_bzDecompress - failure exit code: ret=-4; block=82551; seq=-1; isLastInSeq=1; avail_in=0
Terminator thread: premature exit requested - quitting...
# echo $?
1

# emerge --info
Portage 2.3.40 (python 2.7.14-final-0, default/linux/amd64/17.0, gcc-7.3.0, glibc-2.26-r7, 4.14.55-gentoo x86_64)
=================================================================
System uname: Linux-4.14.55-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E5-2620_v3_@_2.40GHz-with-gentoo-2.4.1
KiB Mem:    80533756 total,   1905028 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Wed, 25 Jul 2018 00:15:01 +0000
Head commit of repository gentoo: 3cf88f8d4ae9db896054b12a03ff17e495adbdfd
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.5::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.13.4-r2::gentoo, 1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.14::gentoo (virtual/os-headers)
sys-libs/glibc:           2.26-r7::gentoo

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-mtune=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.1/ext-active/ /etc/php/apache2-php7.2/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-mtune=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl amd64 berkdb bzip2 cli crypt cxx dri fortran gdbm iconv ipv6 libtirpc modules multilib ncurses nptl openmp pam pcre readline seccomp ssl tcpd unicode xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="access_compat actions alias auth_basic auth_digest authn_alias authn_anon authn_core authn_file authz_core authz_host authz_user autoindex deflate dir env expires filter headers info log_config log_forensic mime mime_magic negotiation proxy proxy_fcgi reqtimeout rewrite setenvif socache_shmcb status unique_id unixd version vhost_alias" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 avx avx2 fma3 popcnt sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-1 php7-2" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Comment 2 Tomáš Mózes 2018-07-25 08:49:15 UTC

Just a side note, this has never happened before, we are decompressing sql backups for import every week. It has been like that for years now, this Saturday was the first time I saw this error.

Comment 3 Tomáš Mózes 2018-07-26 09:09:56 UTC

Reproduced on 2 machines but a dump from another day works ok with pbzip2, so probably a very specific input. Unfortunately cannot reproduce on osm planet bz2 (70GB).

Reported upstream:
https://bugs.launchpad.net/pbzip2/+bug/1783730

Comment 4 deference 2018-07-26 14:18:19 UTC

The planet I had tested with is now a few months old, it is the planet-160718.osm.bz2 file.

Comment 5 Tomáš Mózes 2018-08-06 11:14:00 UTC

The file that pbzip2 failed to decompressed can be decompressed by lbzip2 (thanks Poly-C for the tip).