Summary: | dev-util/diffball: webrsync fails with tarsync and the new gentoo-YYYYMMDD snapshot format | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Thomas Lindroth <thomas.lindroth> |
Component: | Current packages | Assignee: | Zac Medico <zmedico> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | infra-bugs, thomas.lindroth |
Priority: | Normal | Keywords: | InVCS |
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Attachments: | Reset diffball's XZ decompressor in cseek |
Description
Thomas Lindroth
2020-02-08 16:49:11 UTC
Created attachment 613622 [details, diff] Reset diffball's XZ decompressor in cseek I got another sync failure today with the gentoo-20200212.tar.xz tarball. I started looking into the problem and believe I've found the cause. Diffball's support for XZ is broken. Tarsync use dev-util/diffball for reading tar files. It goes through these steps: 1. Decompress the tarball in memory and generate a list of all files in it. 2. Check which files exists on the local filesystem. 3. Unpack the missing files from the tarball. Since it's not possible to seek in compressed files, step 3 must first reset the compressor and unpack the tarball from the beginning again. The reset is done in libcfile/cfile.c:cseek(). https://github.com/rafaelmartins/diffball/blob/master/libcfile/cfile.c#L660 The BZIP2_COMPRESSOR resets a lot of parameters of the compressor (cfh->bzs) including cfh->bzs->avail_in but the XZ_COMPRESSOR doesn't reset anything in cfh->xzs. The difference between the broken and working tarballs is that the broken tarballs have cfh->xzs->avail_in =! 0 at the end of step 1. https://github.com/rafaelmartins/diffball/blob/master/libcfile/cfile.c#L998 Because of that this check then assume there is still data left in ram to decompress and tries to decompress a stale buffer. The call to lzma_code() fails as a result. I've attached a patch against diffball to reset the XZ decompressor in cseek just like the BZIP2 decompressor. With that patch the broken tarball now work. (In reply to Thomas Lindroth from comment #1) > Created attachment 613622 [details, diff] [details, diff] > Reset diffball's XZ decompressor in cseek Good catch, fixes interaction with crefill logic here: > if(0 == cfh->xzs->avail_in && (cfh->raw.offset + > (cfh->raw.end - cfh->xzs->avail_in) < cfh->raw_total_len)) { The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f78d49bc40c58783eaff9ec874c2f733e63bcc5c commit f78d49bc40c58783eaff9ec874c2f733e63bcc5c Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-02-16 01:32:45 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-02-16 01:50:06 +0000 dev-util/diffball: 1.0.1-r2 revbump for bug 708736 Fix tarsync seek failure with xz compressed files. Bug: https://bugs.gentoo.org/708736 Package-Manager: Portage-2.3.89, Repoman-2.3.20 Signed-off-by: Zac Medico <zmedico@gentoo.org> dev-util/diffball/Manifest | 1 + dev-util/diffball/diffball-1.0.1-r2.ebuild | 42 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) |