Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 924235 - Support parallel variants of compressors for binpkgs (e.g. by allowing command selection)
Summary: Support parallel variants of compressors for binpkgs (e.g. by allowing comman...
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-02-10 18:33 UTC by Esteve Varela Colominas
Modified: 2024-02-10 19:33 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Esteve Varela Colominas 2024-02-10 18:33:04 UTC
I need to preface this feature request by mentioning that, currently, the default BINPKG_COMPRESS is set to zstd, which trumps every other compression format by a landslide, when it comes to being used in the context of portage. I'm not sure if the other compression formats are left around as a legacy, or if they're intended to be supported to their fullest extent. This feature request concerns the latter case.

There currently exist a few alternative implementations to some of the supported binpkg compression formats:

- app-arch/bzip2: app-arch/lbzip2 app-arch/pbzip2
- app-arch/gzip: app-arch/pigz
- app-arch/lzip: app-arch/plzip

There's currently two ways to use any of these alternatives:
- Using app-alternatives (gzip and bzip2)
- Using PORTAGE_BZIP2_COMMAND

However, neither of these solutions seems to be meant to change the compressor used for creating and extracting binpkgs.
The app-alternatives solution isn't supported for all commands, and affects everything on the system, which may be undesireable.
The PORTAGE_BZIP2_COMMAND solution is exclusive to bzip2, and has other side-effects across the portage codebase:
- It's used in unpack() in bin/phase-helpers.sh to unpack any bzip2 archive
- It's used to compress and decompress environment.bz2, across the ebuild phases and the python code
This feature feels a bit legacy, and I'm not sure what its intended use was.

I'm not sure what to propose as a solution, as there's no obvious *good* way to solve this, but depending on preferences, one of these solutions may be chosen:
- Add PORTAGE_comp_COMMAND for each of the supported compressors, and similarly replace any use in unpack() and similar, to make it consistent with PORTAGE_BZIP2_COMMAND.
- Add a new BINPKG_COMPRESS_COMMAND, overriding the compression command for the current BINPKG_COMPRESS method. Wouldn't allow choosing a decompressor, however.
- Implement support for the parallel compressors by picking them over the default implementation if available. I've seen more and more programs do this lately, and it'd provide a nice default speed boost by simply having them installed. Not great for reproducible builds, however.
Comment 1 John Helmert III archtester Gentoo Infrastructure gentoo-dev Security 2024-02-10 18:51:58 UTC
> However, neither of these solutions seems to be meant to change the compressor used for creating and extracting binpkgs.

Does switching the default via app-alternatives not allow you to affect what Portage uses as `bzip2`, `gzip`, etc?
Comment 2 Esteve Varela Colominas 2024-02-10 19:33:30 UTC
They do, but they also change the system-wide defaults, which may be undesireable. In my case, I just don't want to deviate from the secure and well-tested reference implementations where I don't have a reason to fully trust the archive, or don't want to risk data loss.