Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 609462 - sys-apps/portage: add binary package compression_probe support for app-arch/zstd
Summary: sys-apps/portage: add binary package compression_probe support for app-arch/zstd
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 611328
  Show dependency tree
 
Reported: 2017-02-15 19:20 UTC by Zac Medico
Modified: 2017-05-20 18:16 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
Support-for-zstandard-decompression.patch (0001-Add-support-for-zstandard-decompression.patch,1.14 KB, patch)
2017-02-20 15:16 UTC, Manuel Rüger (RETIRED)
Details | Diff
Support-for-zstandard-decompression-v2.patch (0001-Add-support-for-zstandard-decompression.patch,1.17 KB, patch)
2017-02-20 15:34 UTC, Manuel Rüger (RETIRED)
Details | Diff
Support-for-zstandard-decompression-v3.patch (0001-Add-support-for-zstandard-decompression.patch,1.15 KB, patch)
2017-02-21 17:08 UTC, Manuel Rüger (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2017-02-15 19:20:16 UTC
It would be nice to add zstd to the supported decompressors.
Comment 1 Manuel Rüger (RETIRED) gentoo-dev 2017-02-20 15:16:40 UTC
Created attachment 464436 [details, diff]
Support-for-zstandard-decompression.patch

Magic number taken from: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2017-02-20 15:20:00 UTC
Comment on attachment 464436 [details, diff]
Support-for-zstandard-decompression.patch

>From 211caa3ab8fce4c5f284d81a6673930a13907635 Mon Sep 17 00:00:00 2001
>From: =?UTF-8?q?Manuel=20R=C3=BCger?= <mrueg@gentoo.org>
>Date: Mon, 20 Feb 2017 16:08:44 +0100
>Subject: [PATCH] Add support for zstandard decompression
>
>X-Gentoo-Bug: 609462
>X-Gentoo-Bug-Url: https://bugs.gentoo.org/show_bug.cgi?id=609462
>
>diff --git a/pym/portage/util/compression_probe.py b/pym/portage/util/compression_probe.py
>index 74f74b163..b8c1a02e0 100644
>--- a/pym/portage/util/compression_probe.py
>+++ b/pym/portage/util/compression_probe.py
>@@ -18,6 +18,7 @@ _decompressors = {
> 	"lzip": "lzip -d",
> 	"lzop": "lzop -d",
> 	"xz": "xz -d",
>+	"zst": "zstd -d",

Any reason you're naming it 'zst' instead of 'zstd'?
Comment 3 Manuel Rüger (RETIRED) gentoo-dev 2017-02-20 15:34:10 UTC
Created attachment 464442 [details, diff]
Support-for-zstandard-decompression-v2.patch

Renamed it to zstd and fixed the magic number. Added Zstandard/LZ4 skippable frames as in  https://github.com/file/file/blob/master/magic/Magdir/compress#L223
Comment 4 Brian Dolbec gentoo-dev 2017-02-20 16:13:50 UTC
missed one?

@@ -41,6 +43,7 @@ def compression_probe(f):
 		lzip
 		lzop
 		xz
+		zst
Comment 5 Zac Medico gentoo-dev 2017-02-21 07:16:01 UTC
Do we really need to support files that begin with a skippable frame? If so, are such files distinguishable from LZ4 files? The frames are documented here:

https://github.com/facebook/zstd/blob/v1.1.3/doc/zstd_compression_format.md#skippable-frames

Apparently the first 4 bytes of a skippable frame can range from 0x184D2A50 to 0x184D2A5F, so we'd have to adjust the regex to account for that.
Comment 6 Manuel Rüger (RETIRED) gentoo-dev 2017-02-21 17:08:20 UTC
Created attachment 464550 [details, diff]
Support-for-zstandard-decompression-v3.patch

Okay, let's only support non-skippable frames first. I think it's easier to add it later than to deprecate.

Fixed the comment as well.
Comment 7 Zac Medico gentoo-dev 2017-02-21 17:45:27 UTC
Thanks, this is in the master branch:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=de0a26200b05b67cbeb1b8449acde18381fbb64b
Comment 8 Zac Medico gentoo-dev 2017-05-20 18:16:13 UTC
Fixed in portage-2.3.5.