When emerging the same package multiple times, the binpkgs that get generated contain non-hermetic data in the XPAK header. In the spirit of https://reproducible-builds.org/ I would like to see portage doing what it can to generate hermetic binpkgs. I have been working on finding some of the sources of non-hermiticity: https://issuetracker.google.com/issues/259558106 https://issuetracker.google.com/issues/300690593 This has resulted in the following patches: * https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4018820 * https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4022490 * https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4876328 I plan on pushing 4876328 upstream, but I'm curious to hear others thoughts if I should push up the other two, or if I should take a different approach. I don't intend this bug to be about fixing non-hermetic behavior caused by the packages build system, but just about portage. Reproducible: Always Steps to Reproduce: 1. emerge --buildpkg <package> 2. emerge --buildpkg <package> 3. diff the two binpkgs Actual Results: Differences in XPAK Expected Results: No differences in XPAK
Do those changes really need to be behind a new FEATURES flag? At a glance, they don't seem like the kinds of changes that might be reverse-incompatible or otherwise necessitating feature-flagging.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=c16ff6913b3d0a4d4ea0f6ee811ece541865bd8f commit c16ff6913b3d0a4d4ea0f6ee811ece541865bd8f Author: Raul E Rangel <rrangel@chromium.org> AuthorDate: 2023-10-11 18:42:34 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-13 10:19:01 +0000 config: Sort USE_EXPAND variables Without the sort, we end up extending the USE_EXPAND variables with a non deterministic ordering. This makes binary package creation non hermetic. i.e., ``` -declare -x PYTHON_TARGETS="python3_8 python3_7 python3_6 python3_10 python3_9" +declare -x PYTHON_TARGETS="python3_8 python3_9 python3_10 python3_7 python3_6" ``` Assuming `PYTHON_TARGETS=python3_8` in make.conf, after this change we get: ``` declare -x PYTHON_TARGETS="python3_8 python3_10 python3_6 python3_7 python3_9" ``` Ideally we would completely sort the USE_EXPAND variables, but the LINGUAS variable appears to need a very specific ordering. Bug: https://bugs.gentoo.org/914441 Signed-off-by: Raul E Rangel <rrangel@chromium.org> Closes: https://github.com/gentoo/portage/pull/1098 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/package/ebuild/config.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=463af696914237bc6e210306fbc2f09f3a1005ce commit 463af696914237bc6e210306fbc2f09f3a1005ce Author: Raul E Rangel <rrangel@chromium.org> AuthorDate: 2022-11-10 04:26:18 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-13 10:19:00 +0000 bin/phase-functions: Move du stats into subshell These variables are only used inside this subshell. This avoids polluting the environment. Apparently this calculation isn't hermetic. I'm not sure why: @@ -268,10 +268,10 @@ declare -x cros_setup_hooks_run="booya" declare -a exclude_hermetic=([0]="--exclude-non-hermetic") declare -- f -declare -a isz=([0]="264" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/image/") -declare -a nsz=([0]="2803" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/work") +declare -a isz=([0]="16" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/image/") +declare -a nsz=([0]="2599" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/work") declare -- phase_func __eapi6_src_install () Bug: https://bugs.gentoo.org/914441 Signed-off-by: Raul E Rangel <rrangel@chromium.org> Signed-off-by: Sam James <sam@gentoo.org> bin/phase-functions.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) https://gitweb.gentoo.org/proj/portage.git/commit/?id=8c2d81eb205bf05445eca616818681e047639eb5 commit 8c2d81eb205bf05445eca616818681e047639eb5 Author: Raul E Rangel <rrangel@chromium.org> AuthorDate: 2023-09-19 20:43:21 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-13 10:19:00 +0000 config: Don't directly modify FEATURES Iterating on self.features returns items in a non-deterministic order, it also leaves self.features out of sync. If we instead use the `remove` method, the values get sorted and synchronized correctly. Bug: https://bugs.gentoo.org/914441 Signed-off-by: Raul E Rangel <rrangel@chromium.org> Signed-off-by: Sam James <sam@gentoo.org> lib/portage/package/ebuild/config.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
This is definitely something worth working on, thanks for looking at it.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=b150419d28bd7afb98404a829c639584d34efc03 commit b150419d28bd7afb98404a829c639584d34efc03 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-13 10:32:26 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-13 10:33:13 +0000 Revert "config: Don't directly modify FEATURES" This reverts commit 8c2d81eb205bf05445eca616818681e047639eb5. I get a bunch of lost USE=test with this: ``` $ emerge -p -uvDU @world [...] [ebuild R ] dev-libs/tree-sitter-bash-0.20.4-r1::gentoo USE="-test*" 0 KiB [ebuild R ] dev-libs/libffi-3.4.4-r1:0/8::gentoo USE="-debug (-exec-static-trampoline) -pax-kernel -static-libs -test*" ABI_X86="32 (64) (-x32)" 0 KiB [ebuild R ] dev-util/pkgconf-2.0.3:0/4::gentoo USE="-test*" ABI_X86="32 (64) (-x32)" 0 KiB [ebuild R ] dev-libs/libpipeline-1.5.7::gentoo USE="-test*" 0 KiB [ebuild R ] dev-util/fq-0.8.0::gentoo USE="-test*" 0 KiB [...] ``` Bug: https://bugs.gentoo.org/914441 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/package/ebuild/config.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3500483f75789c36e379c1b6546a09df7c11e2b1 commit 3500483f75789c36e379c1b6546a09df7c11e2b1 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-10-20 00:49:33 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-10-20 00:51:00 +0000 sys-apps/portage: add 3.0.53 Closes: https://bugs.gentoo.org/915120 Closes: https://bugs.gentoo.org/821529 Closes: https://bugs.gentoo.org/914441 Closes: https://bugs.gentoo.org/914722 Closes: https://bugs.gentoo.org/914873 Closes: https://bugs.gentoo.org/915099 Closes: https://bugs.gentoo.org/915123 Closes: https://bugs.gentoo.org/915128 Closes: https://bugs.gentoo.org/915136 Closes: https://bugs.gentoo.org/915330 Closes: https://bugs.gentoo.org/915494 Closes: https://bugs.gentoo.org/915834 Closes: https://bugs.gentoo.org/915903 Closes: https://bugs.gentoo.org/900224 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.53.ebuild | 238 +++++++++++++++++++++++++++++++++ 2 files changed, 239 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=72f41d07b5396195a98691fafeb3e5b207a76564 commit 72f41d07b5396195a98691fafeb3e5b207a76564 Author: Raul E Rangel <rrangel@chromium.org> AuthorDate: 2024-03-21 20:29:59 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-03-24 22:18:34 +0000 Reapply "config: Don't directly modify FEATURES" This reverts commit b150419d28bd7afb98404a829c639584d34efc03. It turns out we need to keep the open coded version to avoid creating a persistent setting. See https://github.com/gentoo/portage/pull/1098#issuecomment-1761638611 This change just adds a sorted() so we get deterministic ordering. Bug: https://bugs.gentoo.org/914441 Signed-off-by: Raul E Rangel <rrangel@chromium.org> Closes: https://github.com/gentoo/portage/pull/1312 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/portage/package/ebuild/config.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
