When app-arch/pigz[symlink] is installed, some packages cannot be emerged, because the unpack step aborts. Reason is that pigz says that some tarballs have “trailing junk” at the end and exits with exitcode; GNU gzip does not do that. This happens every time. Steps to reproduce: 1. Install app-arch/pigz-2.2.4 with the ‘symlink’ use flag enabled. 2. Try to emerge dev-tex/feynmf-1.08-r3. Result: emerge aborts. You can manually reproduce this by performing the following steps: 1. Identify a gzipped tarball which exhibits this behavior. In this example, I’ll use feynmf-1.08.tar.gz from dev-tex/feynmf-1.08-r3. 2. /bin/gzip -dc < "${DISTDIR}"/feynmf-1.08.tar.gz > /dev/null ; echo $? 3. /usr/bin/gzip -dc < "${DISTDIR}"/feynmf-1.08.tar.gz > /dev/null ; echo $? Note that in step 2, GNU gzip is used, while in step 3, pigz is used. This may be a bug in pigz or in GNU gzip, or it may be a feature of one of these two packages. However, it’s annoying and causes builds to fail. An easy workaround is to uninstall app-arch/pigz. Another workaround is to emerge pigz without the ‘symlink’ use flag. Or you can temporarily move the symlink out of the way, try to change PATH in some way (so that /bin comes before /usr/bin), or emerge (or tar) might be forced to use /bin/gzip when unpacking. All of these workarounds cause pack/unpack operations to take longer on multicore/multiprocessor machines, of course; people install pigz in the first place to avoid that. :-) The unpack operation might check whether pigz is used and ignore exitcode 2. Don’t know whether it is used to signal something important. Or you could parse the output of pigz which goes to stderr.
It we add any special handling for pigz, then it will have to be in included in PMS.
Isn't this the same issue as bug 410935?
SIGPIPE results in exit code 141, but pigz is returning 2 instead: $ gzip -d -c feynmf-1.08.tar.gz | tar xof - $ echo ${PIPESTATUS[@]} 0 0 $ pigz -d -c feynmf-1.08.tar.gz | tar xof - feynmf-1.08.tar.gz OK, has trailing junk which was ignored $ echo ${PIPESTATUS[@]} 2 0
I mean the underlying problem, which is that there are a load of dodgy tarballs out there, and that Portage lets them through, so most of the time people don't notice.
I don't yet know whether feynmf-1.08.tar.gz really contains trailing garbage, or whether pigz is at fault.
$ gunzip feynmf-1.08.tar.gz | /usr/libexec/paludis/utils/strip_tar_corruption >/dev/null /usr/libexec/paludis/utils/strip_tar_corruption: tar file does not include an EOF marker
So, should we add a rule in PMS which says that unpac() must abort if given a tarfile with a missing EOF marker?
That's a good start, yes, although there are other ways things can go wrong too. This is what we've found so far: http://git.exherbo.org/paludis/paludis.git/tree/paludis/repositories/e/ebuild/utils/strip_tar_corruption.cc
(In reply to comment #7) > So, should we add a rule in PMS which says that unpac() must abort if given > a tarfile with a missing EOF marker? Of course not.
(In reply to comment #6) > $ gunzip feynmf-1.08.tar.gz | > /usr/libexec/paludis/utils/strip_tar_corruption >/dev/null > /usr/libexec/paludis/utils/strip_tar_corruption: tar file does not include > an EOF marker Something must be wrong with your program. The last file in feynmf-1.08.tar starts at byte position 0x88600: 00088600 66 65 79 6e 6d 66 2f 66 65 79 6e 6d 66 2e 69 6e |feynmf/feynmf.in| 00088610 73 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |s...............| 00088620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00088660 00 00 00 00 30 31 30 30 36 36 34 00 30 30 30 30 |....0100664.0000| 00088670 37 37 31 00 30 30 30 30 37 36 34 00 30 30 30 30 |771.0000764.0000| 00088680 30 30 31 30 30 37 34 00 30 36 32 35 30 34 31 36 |0010074.06250416| 00088690 30 30 30 00 30 31 31 32 36 35 00 20 30 00 00 00 |000.011265. 0...| 000886a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00088700 00 75 73 74 61 72 20 20 00 00 00 00 00 00 00 00 |.ustar ........| 00088710 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * The length of the file can be found in the 12 bytes starting at offset 124 in the header: 10074 octal = 4156 decimal, i.e. 9 blocks of 512 bytes. So the file ends at position 0x899ff. This is followed by two 512-byte blocks of zeros (from 0x89a00 to 0x89dff) which looks like a perfect end-of-file marker to me.
Nothing wrong with the program, I just forgot how gunzip works...
So, "gzip -dc feynmf-1.08.tar.gz | strip_tar_corruption" doesn't report any errors, I suppose?
Yeah. Looks like we need validity checking on gzip files as well as on the tar: $ gunzip -tv feynmf-1.08.tar.gz feynmf-1.08.tar.gz: gzip: feynmf-1.08.tar.gz: decompression OK, trailing zero bytes ignored
(In reply to comment #13) > Yeah. Looks like we need validity checking on gzip files as well as on the > tar: > > $ gunzip -tv feynmf-1.08.tar.gz > feynmf-1.08.tar.gz: > gzip: feynmf-1.08.tar.gz: decompression OK, trailing zero bytes ignored So we're going to say 'hey, upstream, you used tar -z, we don't support that'?
Well, it depends upon whether you want a system that works, or a system that mostly works most of the time.
(In reply to comment #15) > Well, it depends upon whether you want a system that works, or a system that > mostly works most of the time. I'd say gnu/bsd tar+gzip is our standard. If some custom pigz doesn't work fine with normal tar output, it should be fixed.
I've looked at the issue more closely. Previous versions of GNU tar padded their output with zeros up to a block boundary (which makes sense for a tape archiver), even when using compression. Current GNU tar still does this under certain circumstances. From the tar info file: | If the output goes directly to a local disk, and not through stdout, | then the last write is not extended to a full record size. Otherwise, | reblocking occurs. gzip -d accounts for this; it ignores trailing zero padding and exits with a good status: $ echo foo | gzip | dd conv=sync 2>/dev/null| gzip -d foo $ echo $? 0 But if the trailing garbage contains nonzero bytes, gzip -d will emit a warning and exit with nonzero status: $ ( echo foo | gzip; echo bar ) | gzip -d foo gzip: stdin: decompression OK, trailing garbage ignored $ echo $? 2 I've also tested with bzip2 and xz, they show the same behaviour as gzip. However, pigz doesn't tolerate the zero padding and will exit with an error even in that case: $ echo foo | gzip | dd conv=sync 2>/dev/null | pigz -d foo <stdin> OK, has trailing junk which was ignored $ echo $? 2
(In reply to comment #16) > (In reply to comment #15) > > Well, it depends upon whether you want a system that works, or a system that > > mostly works most of the time. If the three most used decompressors all agree that trailing zero padding is harmless, then I'd say that we shouldn't be more strict. > I'd say gnu/bsd tar+gzip is our standard. If some custom pigz doesn't work > fine with normal tar output, it should be fixed. +1
(In reply to comment #16) +1
*** Bug 485468 has been marked as a duplicate of this bug. ***
Still getting this behaviour or bug when emerging =dev-tex/xmltex-1.9-r2 with "=app-arch/pigz-2.3 symlink" installed. Only package on my year old Gentoo 64bit install, that breaks emerge/compile like this, after performing several "emerge -e @system && emerge -e @world"! Ran into this bug when installing Latex (AKA texlive-2012).
Created attachment 368012 [details, diff] Patch to pigz
I have report this problem to Mark Adler, the author of pigz, through Email in 10/8/13, but unfortunately have not get any response. Here is the patch that hopefully fix this problem. You may want to put it in /etc/portage/patches/app-arch/pigz/.
(In reply to Yichao Zhou from comment #22) > Created attachment 368012 [details, diff] [details, diff] > Patch to pigz This isn't an upstreamable patch since it overrides all return values instead of just fixing the specific case. See my attached patch for a targeted fix.
Created attachment 368016 [details, diff] pigz-2.3.1-trailing-junk-exit-status.patch
Fixed in 2.3.1-r1.
