I'm experimenting with using lbzip2 to speed up things like binpkg installation[*]. PORTAGE_BZIP2_COMMAND can be overridden in make.conf to specify lbzip2 (BUNZIP2 command as well, but if unset, it'll be $...BZIP2... -d). /var/db/pkg/*/*/environment.bz2 saves some related things PORTAGE_COMPRESSION_COMMAND, but not the most relevant to this, PORTAGE_BZIP2_COMMAND, and I think it comes down to: ./bin/phase-functions.sh: PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \ ... PORTAGE_BUILD_USER PORTAGE_BUNZIP2_COMMAND \ PORTAGE_BZIP2_COMMAND PORTAGE_COLORMAP PORTAGE_CONFIGROOT \ ... filtered_vars="___.* ${readonly_bash_vars} ${bash_misc_vars} ${PORTAGE_READONLY_VARS} ${misc_garbage_vars}" So, those vars are intentionally filtered out when building environment.bz2; as a result, you can't tell by reviewing the contents of /var/db/pkg/ if overriding PORTAGE_BZIP2_COMMAND in make.conf "took". PORTAGE_COMPRESSION_COMMAND on the other hand (which selects between bzip2, gzip, etc.) does carry over into environment.bz2. These elements were first committed to the PORTAGE_READONLY_VARS= list back in 2010 in commit 255af602e7c5f74460104fbfaa80c09a31b44a36 I don't know when support for setting PORTAGE_BZIP2_COMMAND / PORTAGE_BUNZIP2_COMMAND in make.conf was established, maybe it was after the filtering decisions were made. In _theory_, it shouldn't matter because any BZIP2_COMMAND should be expected to behave correctly, so recording it in environment.bz2. But you know what they say about theory and practice... [*] pbzip2 -d seems to only unpack faster when the .bz2 was created with pbzip2, but lbzip2 benefits from multiple CPUs regardless: # getconf _NPROCESSORS_ONLN 8 # ls -l linux-firmware-20241210-r1.tbz2 -rw-r--r-- 1 root root 616829204 Jan 11 03:07 linux-firmware-20241210-r1.tbz2 # time head -c 616771277 linux-firmware-20241210-r1.tbz2 | bzip2 -d -c >/dev/null real 1m34.614s user 1m29.306s sys 0m5.446s # time head -c 616771277 linux-firmware-20241210-r1.tbz2 | pbzip2 -d -c >/dev/null real 1m28.020s user 1m29.367s sys 0m2.610s # time head -c 616771277 linux-firmware-20241210-r1.tbz2 | lbzip2 -d -c >/dev/null real 0m8.632s user 1m1.365s sys 0m2.618s
The idea for filtering it out of environment.bz2 is that the user's current setting from make.conf should take precedence over a value from enviroment.bz2. The value from environment.bz2 can get stale and the program may not even be available if the value is stale.
Oh, huh, maybe I have environment.bz2 backwards. I thought it was a historical record of what the settings were at the time of emerge. Since these things were defined in make.conf, etc., I was expecting them to show up there. Sounds like you're saying it's what will be used during the emerge? If so, could it be both at once (say by allowing any otherwise-defined/enforced settings to take precedent)? https://wiki.gentoo.org/wiki//var/db/pkg says it "Stores information about installed packages" without elaboration (it's probably a "if you know enough to care then it shouldn't be necessary to spell it out"?).
The environment.bz2 is loaded for execution of pkg_config, pkg_prerm and pkg_postrm functions. If a package is created with quickpkg then it can also be used to execute pkg_setup, pkg_preinst, and pkg_postinst. I suppose we could preserve PORTAGE_BZIP2_COMMAND in environment.bz2 and then discard it when we load environment.bz2, so that make.conf takes precedence.
Oh, that makes sense, thanks for explaining! If there's some way to both capture "this is what it was at the time" and also "but current settings may override" that sounds ideal. There's lots of specific handling and categories of filter lists in ./bin/phase-functions.sh and I won't pretend I understand the rationales for all of them. Note that at least in the case of PORTAGE_BZIP2_COMMAND, portage already has fallback handling: # egrep ^PORTAGE_BZIP2_COMMAND /etc/portage/make.conf PORTAGE_BZIP2_COMMAND=foooo # quickpkg bash PORTAGE_BZIP2_COMMAND setting is invalid: 'foooo' * Building package for app-shells/bash-5.2_p37 ... [ ok ] [snip] * Packages now in '/usr/portage/packages': * app-shells/bash-5.2_p37: 1.1M # ln -s /bin/bzip2 /usr/local/bin/foooo # quickpkg bash * Building package for app-shells/bash-5.2_p37 ... [ ok ] [snip] * Packages now in '/usr/portage/packages': * app-shells/bash-5.2_p37: 1.1M But, this situation may apply to other variables and fallback-handling may not be a given for them, so I'm not saying that's a universal get-out-of-jail-free card.
We can filter it out in the __preprocess_ebuild_env function, so that make.conf values always override.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=64c7c4161ce9b2eac6611f491b00441832b784ad commit 64c7c4161ce9b2eac6611f491b00441832b784ad Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2025-01-14 04:31:52 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2025-01-15 21:48:55 +0000 bin/phase-functions.sh: Preserve PORTAGE_BZIP2_COMMAND in environment.bz2 Bug: https://bugs.gentoo.org/948067 Signed-off-by: Zac Medico <zmedico@gentoo.org> NEWS | 2 ++ bin/phase-functions.sh | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=03f41049a0fe0632eabd8cddaaca898e45943201 commit 03f41049a0fe0632eabd8cddaaca898e45943201 Author: Sam James <sam@gentoo.org> AuthorDate: 2025-01-22 00:29:50 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2025-01-22 00:30:02 +0000 sys-apps/portage: add 3.0.67 Closes: https://bugs.gentoo.org/703520 Closes: https://bugs.gentoo.org/707980 Closes: https://bugs.gentoo.org/904702 Closes: https://bugs.gentoo.org/906044 Closes: https://bugs.gentoo.org/923530 Closes: https://bugs.gentoo.org/938164 Closes: https://bugs.gentoo.org/939299 Closes: https://bugs.gentoo.org/940120 Closes: https://bugs.gentoo.org/942512 Closes: https://bugs.gentoo.org/942760 Closes: https://bugs.gentoo.org/945382 Closes: https://bugs.gentoo.org/945861 Closes: https://bugs.gentoo.org/946326 Closes: https://bugs.gentoo.org/947822 Closes: https://bugs.gentoo.org/948067 Closes: https://bugs.gentoo.org/939444 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.67.ebuild | 231 +++++++++++++++++++++++++++++++++ 2 files changed, 232 insertions(+)
