Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 715108 - sys-apps/portage: Change default BINPKG_COMPRESS to zstd
Summary: sys-apps/portage: Change default BINPKG_COMPRESS to zstd
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: .tbz2 support (show other bugs)
Hardware: All Linux
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 721960
Blocks: 719456 721152
  Show dependency tree
 
Reported: 2020-03-27 23:24 UTC by Matt Turner
Modified: 2021-01-23 16:22 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Change-BINPKG_COMPRESS-default-from-bzip2-to-xz.patch (0001-Change-BINPKG_COMPRESS-default-from-bzip2-to-xz.patch,1.54 KB, patch)
2020-03-27 23:25 UTC, Matt Turner
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Turner gentoo-dev 2020-03-27 23:24:00 UTC
bzip2 is pretty bad in comparison to more modern compression algorithms like xz and zstd.

Since app-arch/xz-utils is part of @system, I think changing the default is simple enough.
Comment 1 Matt Turner gentoo-dev 2020-03-27 23:25:55 UTC
Created attachment 626472 [details, diff]
Change-BINPKG_COMPRESS-default-from-bzip2-to-xz.patch

Stab in the dark.
Comment 2 Gordon Pettey 2020-04-26 20:13:00 UTC
Using www-client/firefox as a test, either xz or lzip both take 400% of the time for only a 20% size reduction. Adjusting BINPKG_COMPRESS_FLAGS to "-1",  both xz and lz are still smaller than bzip2, but now are comparable in time spent.
Comment 3 Gordon Pettey 2020-04-26 20:16:18 UTC
zstd -9 is twice as fast as bzip2 for comparable file size
Comment 4 Zac Medico gentoo-dev 2020-04-26 21:15:06 UTC
Let's make the default conditional on zstd USE flag as suggested in bug 719456. It will be similar to how the ebuild enables FEATURES=xattr for USE=xattr.
Comment 5 Zac Medico gentoo-dev 2020-04-26 22:04:00 UTC
Add make.globals BINPKG_COMPRESS setting to be conditional on USE=zstd:

https://archives.gentoo.org/gentoo-portage-dev/message/05ac26d918736a2c25e167b5e278d339
https://github.com/gentoo/portage/pull/545
Comment 6 Larry the Git Cow gentoo-dev 2020-05-03 20:32:31 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=73acbb1a7f235f3eeda6a289684e4f43ffdd7dd6

commit 73acbb1a7f235f3eeda6a289684e4f43ffdd7dd6
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-04-26 21:53:49 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-03 20:27:15 +0000

    make.globals: add default BINPKG_COMPRESS setting (bug 715108)
    
    The ebuild will have a default enabled USE=zstd which changes the
    default to zstd here.
    
    Bug: https://bugs.gentoo.org/715108
    Bug: https://bugs.gentoo.org/719456
    Reviewed-by: Brian Dolbec <dolsen@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 cnf/make.globals | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Comment 7 Larry the Git Cow gentoo-dev 2020-05-05 23:56:54 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6a381b421b8283c73c2bd6e5d4e4a46cbe9a43a2

commit 6a381b421b8283c73c2bd6e5d4e4a46cbe9a43a2
Author:     Ian Leonard <antonlacon@gmail.com>
AuthorDate: 2020-04-25 20:32:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-05 23:56:43 +0000

    sys-apps/portage: add zstd USE flag
    
    The zstd USE flag sets the default binary package compression to use
    zstd and adds it as a dependency.
    
    Masked on riscv until app-arch/zstd is keyworded.
    
    Closes: https://github.com/gentoo/gentoo/pull/15520
    Bug: https://bugs.gentoo.org/715108
    Bug: https://bugs.gentoo.org/719456
    Signed-off-by: Ian Leonard <antonlacon@gmail.com>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 profiles/arch/riscv/package.use.mask | 4 ++++
 sys-apps/portage/portage-9999.ebuild | 9 ++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
Comment 8 Larry the Git Cow gentoo-dev 2020-05-07 16:15:03 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e435225e72637d8383e06b0e0244c697e4d1423f

commit e435225e72637d8383e06b0e0244c697e4d1423f
Author:     Ian Leonard <antonlacon@gmail.com>
AuthorDate: 2020-05-07 15:51:38 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-07 16:12:27 +0000

    sys-apps/portage: convert USE zstd to USE flag binpkg-zstd
    
    Bug: https://bugs.gentoo.org/715108
    Bug: https://bugs.gentoo.org/719456
    Package-Manager: Portage-2.3.99, Repoman-2.3.22
    Signed-off-by: Ian Leonard <antonlacon@gmail.com>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 profiles/arch/riscv/package.use.mask | 4 ++--
 sys-apps/portage/metadata.xml        | 1 +
 sys-apps/portage/portage-9999.ebuild | 6 +++---
 3 files changed, 6 insertions(+), 5 deletions(-)
Comment 9 Zac Medico gentoo-dev 2020-05-10 00:36:06 UTC
Since change the default BINPKG_COMPRESS can be a problem for consumers of binary packages if they don't have zstd installed yet, think it might be a good idea to make this new default apply only to new installations, like we did when changing default locations for bug 378603.
Comment 11 Larry the Git Cow gentoo-dev 2020-05-24 23:32:38 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=230595cf600cae6beb6ebf6f817d08ace433c3ea

commit 230595cf600cae6beb6ebf6f817d08ace433c3ea
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-05-11 01:23:41 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-24 22:26:29 +0000

    Default BINPKG_COMPRESSION to zstd (bug 715108)
    
    This includes a _compat_upgrade.binpkg_compression script that the
    ebuild can call in pkg_preinst in order to maintain a backward-compatible
    bzip2 default when appropriate, ensuring that binary package consumers
    are not caught off guard.
    
    Bug: https://bugs.gentoo.org/715108
    Reviewed-by: Brian Dolbec <dolsen@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 .travis.yml                                       |  4 +++
 cnf/make.globals                                  |  5 +--
 lib/portage/_compat_upgrade/binpkg_compression.py | 40 +++++++++++++++++++++++
 lib/portage/tests/resolver/ResolverPlayground.py  |  1 +
 man/make.conf.5                                   |  4 +--
 5 files changed, 50 insertions(+), 4 deletions(-)
Comment 12 Larry the Git Cow gentoo-dev 2020-05-24 23:32:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=57d801e990d0222d646040a227b3b987f0a7c37e

commit 57d801e990d0222d646040a227b3b987f0a7c37e
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-05-24 23:29:43 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-24 23:31:03 +0000

    sys-apps/portage: Add zstd dep to 9999
    
    See: https://github.com/gentoo/portage/pull/549
    Bug: https://bugs.gentoo.org/715108
    Package-Manager: Portage-2.3.99, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/portage-9999.ebuild | 5 +++++
 1 file changed, 5 insertions(+)
Comment 13 Larry the Git Cow gentoo-dev 2020-05-25 00:24:06 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=77960c6cf984530dbcab9fe507e170e7a2fe7dcf

commit 77960c6cf984530dbcab9fe507e170e7a2fe7dcf
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-05-25 00:20:07 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-05-25 00:20:58 +0000

    sys-apps/portage: Bump to version 2.3.100
    
     #715108 Change default BINPKG_COMPRESS to zstd
     #719456 Add dependency on app-arch/zstd
     #720866 Do not set PKG_CONFIG_PATH
     #721402 Hostname UnicodeEncodeError surrogates not allowed
     #721516 Suppress precompressed QA notice for docompress -x
    
    Bug: https://bugs.gentoo.org/721152
    Bug: https://bugs.gentoo.org/715108
    Bug: https://bugs.gentoo.org/719456
    Bug: https://bugs.gentoo.org/720866
    Bug: https://bugs.gentoo.org/721402
    Bug: https://bugs.gentoo.org/721516
    Package-Manager: Portage-2.3.100, Repoman-2.3.22
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest               |   1 +
 sys-apps/portage/portage-2.3.100.ebuild | 261 ++++++++++++++++++++++++++++++++
 2 files changed, 262 insertions(+)
Comment 14 Larry the Git Cow gentoo-dev 2021-01-23 16:22:54 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=9d6a5c43a4a051be88a0af76fdd648f6d0f232b1

commit 9d6a5c43a4a051be88a0af76fdd648f6d0f232b1
Author:     Matt Turner <mattst88@gentoo.org>
AuthorDate: 2021-01-18 18:23:34 +0000
Commit:     Matt Turner <mattst88@gentoo.org>
CommitDate: 2021-01-23 16:19:07 +0000

    targets: Update BINPKG_COMPRESS to new zstd default
    
    In portage commit 230595cf600c (Default BINPKG_COMPRESSION to zstd (bug
    715108)) the default BINPKG_COMPRESSION setting was changed to zstd. But
    to retain compatibility with binpkg consumers, the default is unchanged
    for existing installations, so we need to update our BINPKG_COMPRESSION.
    
    Bug: https://bugs.gentoo.org/715108
    Signed-off-by: Matt Turner <mattst88@gentoo.org>

 targets/stage1/chroot.sh | 3 +++
 1 file changed, 3 insertions(+)