sci-geosciences/googleearth-5.1.3535.3218 fails to emerge: >>> Unpacking GoogleEarthLinux-5.1.3535.3218.bin to /var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/work 2908+1 records in 25323+1 records out 25930826 bytes (26 MB) copied, 2.29607 s, 11.3 MB/s pbzip2: *ERROR unconsumed in after BZ2_bzDecompress loop:ret=4; block=0; seq=89; avail_in=7326 Terminator thread: premature exit requested - quitting... tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now * ERROR: sci-geosciences/googleearth-5.1.3535.3218 failed: * failure unpacking (bzip2 compressed data, block size = 900k) makeself GoogleEarthLinux-5.1.3535.3218.bin ('2.1.5' +8914) * * Call stack: * ebuild.sh, line 48: Called src_unpack * environment, line 2754: Called unpack_makeself * environment, line 3222: Called assert 'failure unpacking (bzip2 compressed data, block size = 900k) makeself GoogleEarthLinux-5.1.3535.3218.bin ('2.1.5' +8914)' * isolated-functions.sh, line 14: Called die * The specific snippet of code: * [[ $x -eq 0 ]] || die "$@" * * If you need support, post the output of 'emerge --info =sci-geosciences/googleearth-5.1.3535.3218', * the complete build log and the output of 'emerge -pqv =sci-geosciences/googleearth-5.1.3535.3218'. * The complete build log is located at '/var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/temp/environment'. * S: '/var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/work' Reproducible: Always
There cannot be a problem with the distfile if it passed the checksums. Do you have e.g. enough space?
(In reply to comment #1) > There cannot be a problem with the distfile if it passed the checksums. Do you > have e.g. enough space? > The problem seems to be pbzip2. I unmerged it and now googleearth emerges just fine: >>> Unpacking GoogleEarthLinux-5.1.3535.3218.bin to /var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/work 2908+1 records in 25323+1 records out 25930826 bytes (26 MB) copied, 5.32811 s, 4.9 MB/s bzip2: (stdin): trailing garbage after EOF ignored >>> Source unpacked in /var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/work >>> Preparing source in /var/tmp/portage/sci-geosciences/googleearth-5.1.3535.3218/work ... >>> Source prepared.
spatz: any idea what's wrong with pbzip2 and unpack_makeself() ?
Seems like a bug in pbzip2 to me, I was able to reproduce it with 1.1.1 (you didn't specify what version you were using). I sent an email upstream, let's wait for his response.
Where can I get the file which is being decompressed (GoogleEarthLinux-5.1.3535.3218.bin or whatever) so that I can try to reproduce the issue? Error seems like invalid/corrupted archive. pbzip2 doesn't tolerate invalid bzip2 archives and generates error as early as encounters ones. "There cannot be a problem with the distfile if it passed the checksums." Yes, it can - if the distfile has been corrupted before computing the checksums. Anyway - I'll need the original file so that I can easier test if the problem is in pbzip2 or in the archive file itself.
(In reply to comment #5) > Where can I get the file which is being decompressed > (GoogleEarthLinux-5.1.3535.3218.bin or whatever) so that I can try to reproduce > the issue? > > Error seems like invalid/corrupted archive. pbzip2 doesn't tolerate invalid > bzip2 archives and generates error as early as encounters ones. > > "There cannot be a problem with the distfile if it passed the checksums." > Yes, it can - if the distfile has been corrupted before computing the > checksums. > > Anyway - I'll need the original file so that I can easier test if the problem > is in pbzip2 or in the archive file itself. > The file is: http://dl.google.com/earth/client/current/GoogleEarthLinux.bin file size 25932414 bytes. I was usin app-arch/pbzip2-1.1.1
Thanks. That Gooogle...bin seems to be self-extracting bash script... Do we extract a clean .bz2 archive from it (how) or we replace bzip2 with link to pbzip2?
(In reply to comment #7) > Thanks. That Gooogle...bin seems to be self-extracting bash script... Do we > extract a clean .bz2 archive from it (how) or we replace bzip2 with link to > pbzip2? > the contents are extracted by portage with unpack_makeself() in /usr/portage/eclass/eutils.eclass
(In reply to comment #8) > (In reply to comment #7) > > Thanks. That Gooogle...bin seems to be self-extracting bash script... Do we > > extract a clean .bz2 archive from it (how) or we replace bzip2 with link to > > pbzip2? > > > > the contents are extracted by portage with unpack_makeself() > in /usr/portage/eclass/eutils.eclass > I don't have a running Gentoo in front of me to check that script but anyway: I did a simple test: 1. Replaced original bzip2 with pbzip2 mv /usr/bin/bzip2 /usr/bin/bzip2.org ln -s /home/oracle/admin/work/pbzip2-1.1.1/pbzip2 /usr/bin/bzip2 2. I ran the self-extracting script: ./GoogleEarthLinux.bin --noexec --target testout1 It finished successfully (I verified that it really took pbzip2 for decompression) So I don't see evidence for pbzip2 issue here! bzip2 has some behavior to consider archives with trailing garbage as valid ones. I consider this behavior evil since it may silently hide errors in corrupted/truncated archives.
The eclass extracts the tarball from the self-extracting script by using dd with the following arguments: dd ibs=8914 skip=1 obs=1024 conv=sync if=/usr/portage/distfiles/GoogleEarthLinux-5.1.3535.3218.bin That writes to stdout, so just redirect it to a file and there you have it. Extracting the resulting file succeeds with bzip2 and fails with pbzip2.
OK - that confirms the archive produced in that way is invalid: $ dd ibs=8914 skip=1 obs=1024 conv=sync if=GoogleEarthLinux.bin >GoogleEarthLinux.tar.bz2 $ bzip2 -t GoogleEarthLinux.tar.bz2 GoogleEarthLinux.tar.bz2: trailing garbage after EOF ignored ok Just bzip2 tolerates such archives. Why is that bad: bzip2 archives can be concatenated, i.e. bunzip2(bzip2(A)+bzip2(B)) = A+B /where + is concatenation/. So if we get a corruption in the beginning of the second segment: this will be considered by bzip2 a correct archive with trailing garbage... - decompression (with ordinary bzip2) will produce A. Attempt to decompress with pbzip2 will return an error (so that we know the archive has been corrupted). (The way pbzip2 works in order to parallelize it's processing generates such multi-segment archives). So - what could we do: 1) Fix that eclass since obviously it appends some unnecessary garbage. 2) Implement new feature(s) so that one can explicitly specify with additional options such kind of corruptions to be ignored. I'd prefer to keep the current behavior as default one. (There are different ways to handle the "garbage" on error - fully ignore it; output 1:1; seek next valid bzip2 stream /segment/ within it). 3) You can still force the original decompression with "pbzip2 -p1" but that's deprecated and will probably be changed (to raise an error in such situations). 4) Use the shell script to decompress itself. I vote for (1) - seems the correct way to handle the problem and also easier to fix. Here is how it can be implemented (instead of dd): MS_dd() { blocks=`expr $3 / 1024` bytes=`expr $3 % 1024` dd if="$1" ibs=$2 skip=1 obs=1024 conv=sync 2> /dev/null | \ { test $blocks -gt 0 && dd ibs=1024 obs=1024 count=$blocks ; \ test $bytes -gt 0 && dd ibs=1 obs=1024 count=$bytes ; } 2> /dev/null } filesizes="25923500" MS_dd GoogleEarthLinux.bin 8914 ${filesizes} When extracted in that way: the archive is fully valid and pbzip2 doesn't fail. The above code is in fact is hard-coded segment from the self-extracting shell script. Where filesizes is a variable available in the self-extracting .bin script. (If you let me know where to take the eclass source I may try to fix it directly). If you manage to convince me that (2) would be more feasible - I may give a bit higher priority on the relevant pbzip2 features and implement them sooner. Regards, Yavor > The eclass extracts the tarball from the self-extracting script by using dd > with the following arguments: > dd ibs=8914 skip=1 obs=1024 conv=sync > if=/usr/portage/distfiles/GoogleEarthLinux-5.1.3535.3218.bin > > That writes to stdout, so just redirect it to a file and there you have it. > Extracting the resulting file succeeds with bzip2 and fails with pbzip2. >
I've changed googleearth to run the script itself, but I doubt it's the only thing using unpack_makeself so I'll keep watching for a general solution. Adding vapier as it seems he wrote unpack_makeself.
you really shouldnt be changing ebuilds to work around issues that arent in the ebuild, especially when the only things breaking are currently unsupported setups http://sources.gentoo.org/eclass/eutils.eclass?r1=1.343&r2=1.344
*** Bug 322893 has been marked as a duplicate of this bug. ***
I've encountered similar issue with a single portage .tbz2 file. However, most of them work just fine...
Hi, Michał convinced me that existing portage format processing code has deeper dependencies (than the googleearth case) on that "ignore trailing garbage" feature of bzip2. So I already started implementing such a feature in pbzip2 (--ignore-trailing-garbage).
*** Bug 325697 has been marked as a duplicate of this bug. ***
*** Bug 328835 has been marked as a duplicate of this bug. ***
From the maintainer of pbzip2 (Jeff Gilchrist): """ Here is the upstream pbzip2 bug for the problem: https://bugs.launchpad.net/pbzip2/+bug/594868 This should be a feature in the next release of pbzip2 so that you gentoo users can specify or compile to ignore trailing garbage. """
*** Bug 341955 has been marked as a duplicate of this bug. ***
New upstream release 1.1.2 of pbzip2 is now available (http://compression.ca/pbzip2/) which adds option to silently ignore "trailing garbage" when uncompressing bzip2 archives: --ignore-trailing-garbage=1 # Silently ignore the "garbage" --ignore-trailing-garbage=0 # Encountered garbage will cause failure The default is defined with IGNORE_TRAILING_GARBAGE macro (considering 0 if undefined). I.e. you may uncomment following in Makefile to have the garbage ignored by default: CFLAGS += -DIGNORE_TRAILING_GARBAGE=1
(In reply to comment #21) > New upstream release 1.1.2 of pbzip2 is now available I bumped app-arch/pbzip2 to the new version, 1.1.2. I left the new flag untouched as it should now be compatible with bzip2 by default. Please test and report here if the new version does indeed fix this problem.
(In reply to comment #22) > (In reply to comment #21) > > New upstream release 1.1.2 of pbzip2 is now available > > I bumped app-arch/pbzip2 to the new version, 1.1.2. I left the new flag > untouched as it should now be compatible with bzip2 by default. Please test and > report here if the new version does indeed fix this problem. > The default behavior is still to not tolerate the garbage and fail (unless you patch the Makefile. I.e. - the -DIGNORE_TRAILING_GARBAGE=1 directive is commented in the upstream Makefile).
(In reply to comment #23) > The default behavior is still to not tolerate the garbage and fail (unless you > patch the Makefile. I.e. - the -DIGNORE_TRAILING_GARBAGE=1 directive is > commented in the upstream Makefile). According to the Makefile: # If IGNORE_TRAILING_GARBAGE is not defined: behavior is automatically determined # by program name: bzip2, bunzip2, bzcat - ignore garbage; otherwise - not. So it's supposed to be compatible with bzip2. To verify this I asked for more testing instead of just closing the bug.
(In reply to comment #24) > (In reply to comment #23) > > The default behavior is still to not tolerate the garbage and fail (unless you > > patch the Makefile. I.e. - the -DIGNORE_TRAILING_GARBAGE=1 directive is > > commented in the upstream Makefile). > > According to the Makefile: > # If IGNORE_TRAILING_GARBAGE is not defined: behavior is automatically > determined > # by program name: bzip2, bunzip2, bzcat - ignore garbage; otherwise - not. > > So it's supposed to be compatible with bzip2. To verify this I asked for more > testing instead of just closing the bug. > Actually, the statement in Makefile should be correct. I can't believe I have forgotten so deeply that this was already implemented... Anyway - more testing would be good idea.
I have also run into this issue with binary packages built by portage. I built a package for gcc on one system but was unable to install it on others... If I install pbzip2-1.1.1 and do a 'tar -tvf /usr/portage/packages/sys-devel/gcc-4.4.5.tbz2' I get: pbzip2: *ERROR unconsumed in after BZ2_bzDecompress loop:ret=4; block=0; seq=37; avail_in=4513 Terminator thread: premature exit requested - quitting... tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now All that is expected. However, if I upgrade to pbzip2-1.1.2 which ignores trailing garbage and run the same tar command it gets to the next to the last file within the archive and then simply hangs. An strace on the running pbzip2 -d process shows it stuck on: futex(0x410eabd8, FUTEX_WAIT, 15911, NULL The file is simply the result of an 'emerge -b gcc' but just in case it is hard to duplicate I have posted the 12MB file here: http://bikergeeks.com/gcc-4.4.5.tbz2 I have not had any other issues with binary packages built by portage on either version so I am not sure what is different about the gcc package. It does work fine with regular bzip2.
(In reply to comment #26) > I have also run into this issue with binary packages built by portage. I built > a package for gcc on one system but was unable to install it on others... > > If I install pbzip2-1.1.1 and do a 'tar -tvf > /usr/portage/packages/sys-devel/gcc-4.4.5.tbz2' I get: > pbzip2: *ERROR unconsumed in after BZ2_bzDecompress loop:ret=4; block=0; > seq=37; avail_in=4513 > Terminator thread: premature exit requested - quitting... > tar: Unexpected EOF in archive > tar: Error is not recoverable: exiting now > > All that is expected. However, if I upgrade to pbzip2-1.1.2 which ignores > trailing garbage and run the same tar command it gets to the next to the last > file within the archive and then simply hangs. > > An strace on the running pbzip2 -d process shows it stuck on: > futex(0x410eabd8, FUTEX_WAIT, 15911, NULL > > The file is simply the result of an 'emerge -b gcc' but just in case it is hard > to duplicate I have posted the 12MB file here: > http://bikergeeks.com/gcc-4.4.5.tbz2 > > I have not had any other issues with binary packages built by portage on either > version so I am not sure what is different about the gcc package. It does work > fine with regular bzip2. Thanks for reporting that, Kevin! What #CPU-s is used (or -p# parameter)?
(In reply to comment #27) > (In reply to comment #26) > > I have also run into this issue with binary packages built by portage. I built > > a package for gcc on one system but was unable to install it on others... > > > > If I install pbzip2-1.1.1 and do a 'tar -tvf > > /usr/portage/packages/sys-devel/gcc-4.4.5.tbz2' I get: > > pbzip2: *ERROR unconsumed in after BZ2_bzDecompress loop:ret=4; block=0; > > seq=37; avail_in=4513 > > Terminator thread: premature exit requested - quitting... > > tar: Unexpected EOF in archive > > tar: Error is not recoverable: exiting now > > > > All that is expected. However, if I upgrade to pbzip2-1.1.2 which ignores > > trailing garbage and run the same tar command it gets to the next to the last > > file within the archive and then simply hangs. > > > > An strace on the running pbzip2 -d process shows it stuck on: > > futex(0x410eabd8, FUTEX_WAIT, 15911, NULL > > > > The file is simply the result of an 'emerge -b gcc' but just in case it is hard > > to duplicate I have posted the 12MB file here: > > http://bikergeeks.com/gcc-4.4.5.tbz2 > > > > I have not had any other issues with binary packages built by portage on either > > version so I am not sure what is different about the gcc package. It does work > > fine with regular bzip2. > > Thanks for reporting that, Kevin! > What #CPU-s is used (or -p# parameter)? I had only tried it with 2 CPUs and no -p. I didn't see a way to inject the -p1 into the tar command so I just called the bzip2 symlink to pbzip2 directly. With -p1 it worked with the warning about trailing garbage. However, with -p2 it also worked. Therefore this may be more about the use of 'bzip2 -d' from within tar.
What is the output of pbzip2 --ignore-trailing-garbage=1 -d -k -v gcc-4.4.5.tbz2 It's not about tar - that's pbzip2 bug. I got a similar hang with -p4 and -p3 though things worked fine with -p2. (Maybe you have dual-core cpu-s or hyper-threading enabled or just the issue is not very deterministic).
(In reply to comment #29) > What is the output of > pbzip2 --ignore-trailing-garbage=1 -d -k -v gcc-4.4.5.tbz2 > > It's not about tar - that's pbzip2 bug. I got a similar hang with -p4 and -p3 > though things worked fine with -p2. (Maybe you have dual-core cpu-s or > hyper-threading enabled or just the issue is not very deterministic). Actually, I forgotten that I had switched to my quad core desktop for more rapid testing and re-installing of pbzip2. # pbzip2 --ignore-trailing-garbage=1 -d -k -v gcc-4.4.5.tbz2 Parallel BZIP2 v1.1.2 - by: Jeff Gilchrist [http://compression.ca] [Feb. 19, 2011] (uses libbzip2 by Julian Seward) Major contributions: Yavor Nikolov <nikolov.javor+pbzip2@gmail.com> # CPUs: 4 Maximum Memory: 100 MB Ignore Trailng Garbage: on ------------------------------------------- File #: 1 of 1 Input Name: gcc-4.4.5.tbz2 Output Name: gcc-4.4.5.tbz2.out BWT Block Size: 900k Input Size: 11626360 bytes Decompressing data... Completed: 58% [hang] This was done on my desktop which actually does have 4 CPUs. But if I add -p2 to force 2 threads like my dual core server I get: # pbzip2 --ignore-trailing-garbage=1 -d -k -v -p2 gcc-4.4.5.tbz2 Parallel BZIP2 v1.1.2 - by: Jeff Gilchrist [http://compression.ca] [Feb. 19, 2011] (uses libbzip2 by Julian Seward) Major contributions: Yavor Nikolov <nikolov.javor+pbzip2@gmail.com> # CPUs: 2 Maximum Memory: 100 MB Ignore Trailng Garbage: on ------------------------------------------- File #: 1 of 1 Input Name: gcc-4.4.5.tbz2 Output Name: gcc-4.4.5.tbz2.out BWT Block Size: 900k Input Size: 11626360 bytes Decompressing data... pbzip2: *WARNING: Trailing garbage after EOF ignored! Output Size: 33013760 bytes ------------------------------------------- Wall Clock: 2.201255 seconds and success.
Bug opened for this problem https://bugs.launchpad.net/pbzip2/+bug/740502
(In reply to comment #31) > Bug opened for this problem https://bugs.launchpad.net/pbzip2/+bug/740502 A patch is available at http://launchpadlibrarian.net/67140650/pbzip2_fix_bug_740502.patch We'll try to release a new version incorporating the above fix soon. In the meanwhile it would be nice if you can test the patch to make sure it helps for you too.
(In reply to comment #32) > (In reply to comment #31) > > Bug opened for this problem https://bugs.launchpad.net/pbzip2/+bug/740502 > > A patch is available at > http://launchpadlibrarian.net/67140650/pbzip2_fix_bug_740502.patch > > We'll try to release a new version incorporating the above fix soon. In the > meanwhile it would be nice if you can test the patch to make sure it helps for > you too. I tried the patch and it solves my problem. I am able to unpack that file successfully. However, there are differences from bzip2 that you may or may not want to address (they don't bother me but I thought I should point them out)... 1. When run as bzip2 -d pbzip2 filters out the trailing garbage and does not output a warning. This means if run through tar there is no warning at all about trailing garbage. When run with -v it does still output the warning. 2. When used to decompress file.tbz2 pbzip2 creates file.tbz2.out while regular bzip2 knows to call it file.tar.
Thanks for your feedback Kevin! pbzip2 1.1.3 released including fixes for the relevant bugs: http://www.compression.ca/pbzip2
I just added pbzip-1.1.4 to the tree. Can you please test with that version and see if the problem is fixed? Thanks.
Closing, feel free to reopen if the problem is still there. Thanks to all involved!
