Summary: | app-arch/zstd-1.5.2 checksum error on decompressing very large files | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Andy Figueroa <andy> |
Component: | Current packages | Assignee: | Gentoo's Team for Core System packages <base-system> |
Status: | UNCONFIRMED --- | ||
Severity: | normal | CC: | sam |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | AMD64 | ||
OS: | Linux | ||
See Also: | https://github.com/facebook/zstd/issues/3260 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- |
Description
Andy Figueroa
2022-04-16 21:00:57 UTC
Followup on issue with apparent zstd errors in my largest archives. Last night's archives were made with zstd-1.5.0 at my normal compression level 9, I continued to see an error upon testing the archive, but only on the largest archive in the collection of archives (total size of set of archives 93 GB at compression level 9), that one file being right at 30 GB. Simple test was done on the archive with $ zstd -t vboxmnt.tar.zst vboxmnt.tar.zst : 54.4 GiB... vboxmnt.tar.zst : Decoding error (36) : Corrupted block detected and: $ tar tf vboxmnt.tar.zst mnt/vbox/VDI/ mnt/vbox/VDI/MXLinux.vdi mnt/vbox/VDI/MXLinuxFrugal.vdi mnt/vbox/VDI/Windows10Home.vdi /*stdin*\ : Decoding error (36) : Corrupted block detected tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now Previously with version zstd-1.5.2, the error manifested as a checksum error. I therefore reinstalled zstd-1.5.2 and recreated the collection of archives at the default compression level of 3. Retesting the collection of archives as above, turned up no errors. Loss of compression between level 3 and level 9 was only about 5%, with a nearly 20% improvement in speed. That's a good trade off since I've moved to larger drives. My conclusion is that the apparent errors is more the product of higher, but still modest, compression levels, more than the changes between versions, though the errors manifest more readily with version 1.5.2 than 1.5.0. Strongly suspect the problem is upstream, though I've not been able to find reports of such errors by searching the web. I would suggest that you open a bug report upstream. It is unlikely that this problem is Gentoo-specific. Did you report this issue upstream? Thank you for asking, Sam. I failed to report it upstream at the time so I appreciate the reminder. I found recently reported similar issue decompressing large files at high compression. https://github.com/facebook/zstd/issues/3260 I tagged onto that and assume that I will be chastised if I did that wrongly. Fantastic, thank you for the quick update! |