Upon restoring or even reading content of very large tar.zstd archives (11 GB or larger) containing very large files (6 GB or larger) the following error results: $ tar xf /mnt/backup3/data/vboxmnt.tar.zst ... "/*stdin*\ : Decoding error (36) : Restored data doesn't match checksum tar: Child returned status 1 tar: Error is not recoverable: exiting now" The files appear to be extracted without error in spite of the error message. Tested by masking app-arch/zstd-1.5.2 and re-emerging app-arch/zstd-1.5.0, re-creating the archive and then again extracting the archive as above, but without error message. Archive was created with the following command: tar cpf /mnt/backup3/data/vboxmnt.tar.zst -I "zstd -9 -T0" mnt/vbox/VDI/ which contains several VirtualBox virtual machines. Smaller files are not affected, and the error message appears to be passed to tar from zstd, and only shows up once after all files have been extracted.
Followup on issue with apparent zstd errors in my largest archives. Last night's archives were made with zstd-1.5.0 at my normal compression level 9, I continued to see an error upon testing the archive, but only on the largest archive in the collection of archives (total size of set of archives 93 GB at compression level 9), that one file being right at 30 GB. Simple test was done on the archive with $ zstd -t vboxmnt.tar.zst vboxmnt.tar.zst : 54.4 GiB... vboxmnt.tar.zst : Decoding error (36) : Corrupted block detected and: $ tar tf vboxmnt.tar.zst mnt/vbox/VDI/ mnt/vbox/VDI/MXLinux.vdi mnt/vbox/VDI/MXLinuxFrugal.vdi mnt/vbox/VDI/Windows10Home.vdi /*stdin*\ : Decoding error (36) : Corrupted block detected tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now Previously with version zstd-1.5.2, the error manifested as a checksum error. I therefore reinstalled zstd-1.5.2 and recreated the collection of archives at the default compression level of 3. Retesting the collection of archives as above, turned up no errors. Loss of compression between level 3 and level 9 was only about 5%, with a nearly 20% improvement in speed. That's a good trade off since I've moved to larger drives. My conclusion is that the apparent errors is more the product of higher, but still modest, compression levels, more than the changes between versions, though the errors manifest more readily with version 1.5.2 than 1.5.0. Strongly suspect the problem is upstream, though I've not been able to find reports of such errors by searching the web.
I would suggest that you open a bug report upstream. It is unlikely that this problem is Gentoo-specific.
Did you report this issue upstream?
Thank you for asking, Sam. I failed to report it upstream at the time so I appreciate the reminder. I found recently reported similar issue decompressing large files at high compression. https://github.com/facebook/zstd/issues/3260 I tagged onto that and assume that I will be chastised if I did that wrongly.
Fantastic, thank you for the quick update!
upstream has closeed the issue due to lack of information and reproducer do we still need to keep this open?
Following upstream.
Currently running zstd 1.5.7-r1 and can no longer reproduce. Archives compressed at compression level 9 no longer show any errors upon decompression. Issue appears to be resolved.
(In reply to Andy Figueroa from comment #8) > Issue appears to be resolved. Even better :)
