Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 838862 - app-arch/zstd-1.5.2 checksum error on decompressing very large files
Summary: app-arch/zstd-1.5.2 checksum error on decompressing very large files
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Patrick Lauer
Depends on:
Reported: 2022-04-16 21:00 UTC by Andy Figueroa
Modified: 2022-04-17 18:40 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Andy Figueroa 2022-04-16 21:00:57 UTC
Upon restoring or even reading content of very large tar.zstd archives (11 GB or larger) containing very large files (6 GB or larger) the following error results:

$ tar xf /mnt/backup3/data/vboxmnt.tar.zst
"/*stdin*\ : Decoding error (36) : Restored data doesn't match checksum 
tar: Child returned status 1
tar: Error is not recoverable: exiting now"

The files appear to be extracted without error in spite of the error message.

Tested by masking app-arch/zstd-1.5.2 and re-emerging app-arch/zstd-1.5.0, re-creating the archive and then again extracting the archive as above, but without error message.

Archive was created with the following command:
tar cpf /mnt/backup3/data/vboxmnt.tar.zst -I "zstd -9 -T0" mnt/vbox/VDI/

which contains several VirtualBox virtual machines.

Smaller files are not affected, and the error message appears to be passed to tar from zstd, and only shows up once after all files have been extracted.
Comment 1 Andy Figueroa 2022-04-17 18:25:51 UTC
Followup on issue with apparent zstd errors in my largest archives. Last night's archives were made with zstd-1.5.0 at my normal compression level 9, I continued to see an error upon testing the archive, but only on the largest archive in the collection of archives (total size of set of archives 93 GB at compression level 9), that one file being right at 30 GB. Simple test was done on the archive with $ zstd -t vboxmnt.tar.zst
vboxmnt.tar.zst      : 54.4 GiB...     vboxmnt.tar.zst : Decoding error (36) : Corrupted block detected

$ tar tf vboxmnt.tar.zst
/*stdin*\ : Decoding error (36) : Corrupted block detected 
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Previously with version zstd-1.5.2, the error manifested as a checksum error.

I therefore reinstalled zstd-1.5.2 and recreated the collection of archives at the default compression level of 3. Retesting the collection of archives as above, turned up no errors.

Loss of compression between level 3 and level 9 was only about 5%, with a nearly 20% improvement in speed. That's a good trade off since I've moved to larger drives.

My conclusion is that the apparent errors is more the product of higher, but still modest, compression levels, more than the changes between versions, though the errors manifest more readily with version 1.5.2 than 1.5.0.

Strongly suspect the problem is upstream, though I've not been able to find reports of such errors by searching the web.
Comment 2 Mike Gilbert gentoo-dev 2022-04-17 18:40:34 UTC
I would suggest that you open a bug report upstream. It is unlikely that this problem is Gentoo-specific.