Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 573642 - error: tar failed to extract the image. tarball is corrupt? (/var/tmp/portage/webrsync-eBIOwt/portage-20160201.tar.xz)
Summary: error: tar failed to extract the image. tarball is corrupt? (/var/tmp/portage...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
: 573678 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-02-02 08:17 UTC by charles17
Modified: 2016-02-05 07:56 UTC (History)
10 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Log of syncing with sync-type webrsync (sync.log,3.44 KB, text/plain)
2016-02-03 08:05 UTC, charles17
Details
Make emerge-webrsync skip over .xz archives, which are currently broken. (emerge-webrsync_no_xz.patch,496 bytes, patch)
2016-02-03 23:48 UTC, Hank Leininger
Details | Diff
webrsync with app-arch/tarsync installed still works (sync.log,1.06 KB, text/plain)
2016-02-04 07:29 UTC, charles17
Details

Note You need to log in before you can comment on or make changes to this bug.
Description charles17 2016-02-02 08:17:25 UTC
* Running $myvarcmd
>>> Syncing repository 'gentoo' into '/usr/portage'...
Fetching most recent snapshot ...
Trying to retrieve 20160201 snapshot from http://distfiles.gentoo.org ...
Fetching file portage-20160201.tar.xz.md5sum ...
Fetching file portage-20160201.tar.xz.gpgsig ...
Fetching file portage-20160201.tar.xz ...
Checking digest ...
Checking signature ...
gpg: Signature made Tue Feb  2 01:49:45 2016 CET using RSA key ID C9189250
gpg: Good signature from "Gentoo Portage Snapshot Signing Key (Automated Signing Key)" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DCD0 5B71 EAB9 4199 527F  44AC DB6B 8C1F 96D8 BF6D
     Subkey fingerprint: E1D6 ABB6 3BFC FB4B A02F  DF1C EC59 0EEA C918 9250
Getting snapshot timestamp ...
Syncing local tree ...
emerge-webrsync: error: tar failed to extract the image. tarball is corrupt? (/var/tmp/portage/webrsync-eBIOwt/portage-20160201.tar.xz)
Trying to retrieve 20160131 snapshot from http://distfiles.gentoo.org ...
Fetching file portage-20160131.tar.xz.md5sum ...
Fetching file portage-20160131.tar.bz2.md5sum ...
Fetching file portage-20160131.tar.bz2.gpgsig ...
Fetching file portage-20160131.tar.bz2 ...
Checking digest ...
Checking signature ...
gpg: Signature made Mon Feb  1 01:48:54 2016 CET using RSA key ID C9189250
gpg: Good signature from "Gentoo Portage Snapshot Signing Key (Automated Signing Key)" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DCD0 5B71 EAB9 4199 527F  44AC DB6B 8C1F 96D8 BF6D
     Subkey fingerprint: E1D6 ABB6 3BFC FB4B A02F  DF1C EC59 0EEA C918 9250
Getting snapshot timestamp ...
Syncing local tree ...


BTW: Why does it switch to the older 
portage-20160131.tar.bz2 if the newer 
portage-20160201.tar.bz2 is available?
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-02-02 17:58:03 UTC
*** Bug 573678 has been marked as a duplicate of this bug. ***
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-02-02 18:00:12 UTC
Please paste the output of:

  tar --version
  xz --version
Comment 3 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2016-02-02 18:06:19 UTC
I had the same thing,

tar (GNU tar) 1.27.1
xz (XZ Utils) 5.0.8
liblzma 5.0.8
Comment 4 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2016-02-02 18:15:00 UTC
testing this

app-arch/tar-1.28-r1::gentoo
app-arch/xz-utils-5.2.2::gentoo

still get the error

Syncing local tree ...
emerge-webrsync: error: tar failed to extract the image. tarball is corrupt? (/var/tmp/portage/webrsync-6CvNwh/portage-20160201.tar.xz)
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-02-02 18:41:44 UTC
So this is a bug in emerge-webrsync script.

I've been able to reproduce the issue doing:

$ xz -cd portage-20160201.tar.xz | tar -t >/dev/null
$ echo ${PIPESTATUS[*]}
141 0

Long story short, 'xz' is killed by SIGPIPE when tar reaches the end of the archive and emerge-webrsync wrongly assumes it failed.

This most likely started happening because snapshots are now compressed using pixz.
Comment 6 Markus Kaindl 2016-02-03 01:27:48 UTC
I'm seeing this on one of my gentoo machines, 3 others synced just fine today, all 4 using webrsync-gpg, the same mirror and the same versions of sys-apps/portage (2.2.26), app-arch/tar (1.27.1-r2) and app-arch/xz-utils (5.0.8), all with the same USE-Flags on all 4 machines.
Any ideas, where else to look for the difference?
Comment 7 charles17 2016-02-03 08:05:03 UTC
Created attachment 424520 [details]
Log of syncing with sync-type webrsync

(In reply to Michał Górny from comment #5)
> [...]
> This most likely started happening because snapshots are now compressed
> using pixz.

Could this please be redone for the time being until stage3 tarballs have a version of emerge-webrsync which is compatible with pixz compressed snapshot tarballs?

The point is that Gentoo handbook / Installation / Installing base system 
in section "Installing a Portage snapshot" ( https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Base#Installing_a_Portage_snapshot) recommends emerge-webrsync for the first installation of a snapshot.

Presently this would not install the latest tar.xz snapshot but only a three days old tar.bz2 snapshot (see my attachment).
Comment 8 Thomas Lindroth 2016-02-03 10:55:26 UTC
What is the plan for fixing this problem?

As I see it the old snapshot format that works with current emerge-webrsync should be restored. Emerge-webrsync can then be upgraded to a version that works with the new format. Then you wait for a year before rolling out the new format so users got a chance to upgrade to the new emerge-webrsync.
Comment 9 Hank Leininger 2016-02-03 23:47:26 UTC
I'd be interested in a real fix or even a real workaround, but in the meantime here's a stupid workaround, attached.  It makes emerge-webrsync pass over .xz compressed snapshots and prefer .bz2 instead.
Comment 10 Hank Leininger 2016-02-03 23:48:32 UTC
Created attachment 424584 [details, diff]
Make emerge-webrsync skip over .xz archives, which are currently broken.
Comment 11 charles17 2016-02-04 07:29:11 UTC
Created attachment 424590 [details]
webrsync with app-arch/tarsync installed still works

(In reply to Hank Leininger from comment #10)
> Created attachment 424584 [details, diff] [details, diff]
> Make emerge-webrsync skip over .xz archives, which are currently broken.

On a freshly expanded and chrooted-into stage3 this seems to be the only possibility by now.

For an existing installation one could emerge app-arch/tarsync as described in <https://wiki.gentoo.org/wiki//etc/portage/repos.conf/webrsync.conf>.

Just tested and it works for me.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-02-04 21:17:05 UTC
Robin has just updated the script to use threaded 'xz' instead of 'pixz'. Hopefully this will fix the issue when the next snapshot comes out.
Comment 13 charles17 2016-02-05 07:19:58 UTC
Thanks to Robin!

Just tested without app-arch/tarsync and it works again using the tar.xz tarball:


 * Running $myvarcmd
>>> Syncing repository 'gentoo' into '/usr/portage'...
Fetching most recent snapshot ...
Trying to retrieve 20160204 snapshot from http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo ...
Fetching file portage-20160204.tar.xz.md5sum ...
Fetching file portage-20160204.tar.xz.gpgsig ...
Fetching file portage-20160204.tar.xz ...
Checking digest ...
Checking signature ...
gpg: Signature made Fri Feb  5 01:56:57 2016 CET using RSA key ID C9189250
gpg: Good signature from "Gentoo Portage Snapshot Signing Key (Automated Signing Key)" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DCD0 5B71 EAB9 4199 527F  44AC DB6B 8C1F 96D8 BF6D
     Subkey fingerprint: E1D6 ABB6 3BFC FB4B A02F  DF1C EC59 0EEA C918 9250
Getting snapshot timestamp ...
Syncing local tree ...

Number of files: 208,115 (reg: 180,852, dir: 27,263)
Number of created files: 83 (reg: 79, dir: 4)
Number of deleted files: 67 (reg: 65, dir: 2)
Number of regular files transferred: 39,557
Total file size: 409.37M bytes
Total transferred file size: 85.71M bytes
Literal data: 85.71M bytes
Matched data: 0 bytes
File list size: 3.10M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 51.27M
Total bytes received: 786.67K

sent 51.27M bytes  received 786.67K bytes  127.43K bytes/sec
total size is 409.37M  speedup is 7.86
Cleaning up ...

Performing Global Updates
(Could take a couple of minutes if you have a lot of binary packages.)


=== Sync completed for gentoo
Comment 14 Branko Grubic 2016-02-05 07:44:23 UTC
I can also confirm that the current snapshot ('portage-20160204.tar.xz') works fine here.


Thanks!