bzip2 is pretty bad in comparison to more modern compression algorithms like xz and zstd. Since app-arch/xz-utils is part of @system, I think changing the default is simple enough.
Created attachment 626472 [details, diff] Change-BINPKG_COMPRESS-default-from-bzip2-to-xz.patch Stab in the dark.
Using www-client/firefox as a test, either xz or lzip both take 400% of the time for only a 20% size reduction. Adjusting BINPKG_COMPRESS_FLAGS to "-1", both xz and lz are still smaller than bzip2, but now are comparable in time spent.
zstd -9 is twice as fast as bzip2 for comparable file size
Let's make the default conditional on zstd USE flag as suggested in bug 719456. It will be similar to how the ebuild enables FEATURES=xattr for USE=xattr.
Add make.globals BINPKG_COMPRESS setting to be conditional on USE=zstd: https://archives.gentoo.org/gentoo-portage-dev/message/05ac26d918736a2c25e167b5e278d339 https://github.com/gentoo/portage/pull/545
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=73acbb1a7f235f3eeda6a289684e4f43ffdd7dd6 commit 73acbb1a7f235f3eeda6a289684e4f43ffdd7dd6 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-04-26 21:53:49 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-03 20:27:15 +0000 make.globals: add default BINPKG_COMPRESS setting (bug 715108) The ebuild will have a default enabled USE=zstd which changes the default to zstd here. Bug: https://bugs.gentoo.org/715108 Bug: https://bugs.gentoo.org/719456 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> cnf/make.globals | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6a381b421b8283c73c2bd6e5d4e4a46cbe9a43a2 commit 6a381b421b8283c73c2bd6e5d4e4a46cbe9a43a2 Author: Ian Leonard <antonlacon@gmail.com> AuthorDate: 2020-04-25 20:32:57 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-05 23:56:43 +0000 sys-apps/portage: add zstd USE flag The zstd USE flag sets the default binary package compression to use zstd and adds it as a dependency. Masked on riscv until app-arch/zstd is keyworded. Closes: https://github.com/gentoo/gentoo/pull/15520 Bug: https://bugs.gentoo.org/715108 Bug: https://bugs.gentoo.org/719456 Signed-off-by: Ian Leonard <antonlacon@gmail.com> Signed-off-by: Zac Medico <zmedico@gentoo.org> profiles/arch/riscv/package.use.mask | 4 ++++ sys-apps/portage/portage-9999.ebuild | 9 ++++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e435225e72637d8383e06b0e0244c697e4d1423f commit e435225e72637d8383e06b0e0244c697e4d1423f Author: Ian Leonard <antonlacon@gmail.com> AuthorDate: 2020-05-07 15:51:38 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-07 16:12:27 +0000 sys-apps/portage: convert USE zstd to USE flag binpkg-zstd Bug: https://bugs.gentoo.org/715108 Bug: https://bugs.gentoo.org/719456 Package-Manager: Portage-2.3.99, Repoman-2.3.22 Signed-off-by: Ian Leonard <antonlacon@gmail.com> Signed-off-by: Zac Medico <zmedico@gentoo.org> profiles/arch/riscv/package.use.mask | 4 ++-- sys-apps/portage/metadata.xml | 1 + sys-apps/portage/portage-9999.ebuild | 6 +++--- 3 files changed, 6 insertions(+), 5 deletions(-)
Since change the default BINPKG_COMPRESS can be a problem for consumers of binary packages if they don't have zstd installed yet, think it might be a good idea to make this new default apply only to new installations, like we did when changing default locations for bug 378603.
Patch posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/e6c159f77492000afbfe3360f6b5d9e1 https://github.com/gentoo/portage/pull/549
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=230595cf600cae6beb6ebf6f817d08ace433c3ea commit 230595cf600cae6beb6ebf6f817d08ace433c3ea Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-05-11 01:23:41 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-24 22:26:29 +0000 Default BINPKG_COMPRESSION to zstd (bug 715108) This includes a _compat_upgrade.binpkg_compression script that the ebuild can call in pkg_preinst in order to maintain a backward-compatible bzip2 default when appropriate, ensuring that binary package consumers are not caught off guard. Bug: https://bugs.gentoo.org/715108 Reviewed-by: Brian Dolbec <dolsen@gentoo.org> Signed-off-by: Zac Medico <zmedico@gentoo.org> .travis.yml | 4 +++ cnf/make.globals | 5 +-- lib/portage/_compat_upgrade/binpkg_compression.py | 40 +++++++++++++++++++++++ lib/portage/tests/resolver/ResolverPlayground.py | 1 + man/make.conf.5 | 4 +-- 5 files changed, 50 insertions(+), 4 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=57d801e990d0222d646040a227b3b987f0a7c37e commit 57d801e990d0222d646040a227b3b987f0a7c37e Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-05-24 23:29:43 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-24 23:31:03 +0000 sys-apps/portage: Add zstd dep to 9999 See: https://github.com/gentoo/portage/pull/549 Bug: https://bugs.gentoo.org/715108 Package-Manager: Portage-2.3.99, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/portage-9999.ebuild | 5 +++++ 1 file changed, 5 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77960c6cf984530dbcab9fe507e170e7a2fe7dcf commit 77960c6cf984530dbcab9fe507e170e7a2fe7dcf Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-05-25 00:20:07 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-05-25 00:20:58 +0000 sys-apps/portage: Bump to version 2.3.100 #715108 Change default BINPKG_COMPRESS to zstd #719456 Add dependency on app-arch/zstd #720866 Do not set PKG_CONFIG_PATH #721402 Hostname UnicodeEncodeError surrogates not allowed #721516 Suppress precompressed QA notice for docompress -x Bug: https://bugs.gentoo.org/721152 Bug: https://bugs.gentoo.org/715108 Bug: https://bugs.gentoo.org/719456 Bug: https://bugs.gentoo.org/720866 Bug: https://bugs.gentoo.org/721402 Bug: https://bugs.gentoo.org/721516 Package-Manager: Portage-2.3.100, Repoman-2.3.22 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-2.3.100.ebuild | 261 ++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9d6a5c43a4a051be88a0af76fdd648f6d0f232b1 commit 9d6a5c43a4a051be88a0af76fdd648f6d0f232b1 Author: Matt Turner <mattst88@gentoo.org> AuthorDate: 2021-01-18 18:23:34 +0000 Commit: Matt Turner <mattst88@gentoo.org> CommitDate: 2021-01-23 16:19:07 +0000 targets: Update BINPKG_COMPRESS to new zstd default In portage commit 230595cf600c (Default BINPKG_COMPRESSION to zstd (bug 715108)) the default BINPKG_COMPRESSION setting was changed to zstd. But to retain compatibility with binpkg consumers, the default is unchanged for existing installations, so we need to update our BINPKG_COMPRESSION. Bug: https://bugs.gentoo.org/715108 Signed-off-by: Matt Turner <mattst88@gentoo.org> targets/stage1/chroot.sh | 3 +++ 1 file changed, 3 insertions(+)
