Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 914441 - Make binpkg creation hermetic
Summary: Make binpkg creation hermetic
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 915120
Blocks: 913920
  Show dependency tree
 
Reported: 2023-09-19 18:53 UTC by Raul Rangel
Modified: 2024-06-24 06:02 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Raul Rangel 2023-09-19 18:53:59 UTC
When emerging the same package multiple times, the binpkgs that get generated contain non-hermetic data in the XPAK header. In the spirit of https://reproducible-builds.org/ I would like to see portage doing what it can to generate hermetic binpkgs.

I have been working on finding some of the sources of non-hermiticity:
https://issuetracker.google.com/issues/259558106
https://issuetracker.google.com/issues/300690593

This has resulted in the following patches:
* https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4018820
* https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4022490
* https://chromium-review.googlesource.com/c/chromiumos/third_party/portage_tool/+/4876328

I plan on pushing 4876328 upstream, but I'm curious to hear others thoughts if I should push up the other two, or if I should take a different approach.

I don't intend this bug to be about fixing non-hermetic behavior caused by the packages build system, but just about portage.

Reproducible: Always

Steps to Reproduce:
1. emerge --buildpkg <package>
2. emerge --buildpkg <package>
3. diff the two binpkgs
Actual Results:  
Differences in XPAK

Expected Results:  
No differences in XPAK
Comment 1 John Helmert III archtester Gentoo Infrastructure gentoo-dev Security 2023-09-20 03:29:59 UTC
Do those changes really need to be behind a new FEATURES flag? At a glance, they don't seem like the kinds of changes that might be reverse-incompatible or otherwise necessitating feature-flagging.
Comment 2 Larry the Git Cow gentoo-dev 2023-10-13 10:19:05 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c16ff6913b3d0a4d4ea0f6ee811ece541865bd8f

commit c16ff6913b3d0a4d4ea0f6ee811ece541865bd8f
Author:     Raul E Rangel <rrangel@chromium.org>
AuthorDate: 2023-10-11 18:42:34 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-13 10:19:01 +0000

    config: Sort USE_EXPAND variables
    
    Without the sort, we end up extending the USE_EXPAND variables with
    a non deterministic ordering. This makes binary package creation non
    hermetic.
    
    i.e.,
    ```
    -declare -x PYTHON_TARGETS="python3_8 python3_7 python3_6 python3_10 python3_9"
    +declare -x PYTHON_TARGETS="python3_8 python3_9 python3_10 python3_7 python3_6"
    ```
    
    Assuming `PYTHON_TARGETS=python3_8` in make.conf, after this change we
    get:
    ```
    declare -x PYTHON_TARGETS="python3_8 python3_10 python3_6 python3_7 python3_9"
    ```
    
    Ideally we would completely sort the USE_EXPAND variables, but the
    LINGUAS variable appears to need a very specific ordering.
    
    Bug: https://bugs.gentoo.org/914441
    Signed-off-by: Raul E Rangel <rrangel@chromium.org>
    Closes: https://github.com/gentoo/portage/pull/1098
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/package/ebuild/config.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=463af696914237bc6e210306fbc2f09f3a1005ce

commit 463af696914237bc6e210306fbc2f09f3a1005ce
Author:     Raul E Rangel <rrangel@chromium.org>
AuthorDate: 2022-11-10 04:26:18 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-13 10:19:00 +0000

    bin/phase-functions: Move du stats into subshell
    
    These variables are only used inside this subshell. This avoids
    polluting the environment.
    
    Apparently this calculation isn't hermetic. I'm not sure why:
    
    @@ -268,10 +268,10 @@
     declare -x cros_setup_hooks_run="booya"
     declare -a exclude_hermetic=([0]="--exclude-non-hermetic")
     declare -- f
    -declare -a isz=([0]="264" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/image/")
    -declare -a nsz=([0]="2803" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/work")
    +declare -a isz=([0]="16" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/image/")
    +declare -a nsz=([0]="2599" [1]="/var/tmp/portage/dev-libs/libffi-3.1-r8/work")
     declare -- phase_func
     __eapi6_src_install ()
    
    Bug: https://bugs.gentoo.org/914441
    Signed-off-by: Raul E Rangel <rrangel@chromium.org>
    Signed-off-by: Sam James <sam@gentoo.org>

 bin/phase-functions.sh | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8c2d81eb205bf05445eca616818681e047639eb5

commit 8c2d81eb205bf05445eca616818681e047639eb5
Author:     Raul E Rangel <rrangel@chromium.org>
AuthorDate: 2023-09-19 20:43:21 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-13 10:19:00 +0000

    config: Don't directly modify FEATURES
    
    Iterating on self.features returns items in a non-deterministic order,
    it also leaves self.features out of sync. If we instead use the `remove`
    method, the values get sorted and synchronized correctly.
    
    Bug: https://bugs.gentoo.org/914441
    Signed-off-by: Raul E Rangel <rrangel@chromium.org>
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/package/ebuild/config.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-10-13 10:27:55 UTC
This is definitely something worth working on, thanks for looking at it.
Comment 4 Larry the Git Cow gentoo-dev 2023-10-13 10:33:48 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b150419d28bd7afb98404a829c639584d34efc03

commit b150419d28bd7afb98404a829c639584d34efc03
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-13 10:32:26 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-13 10:33:13 +0000

    Revert "config: Don't directly modify FEATURES"
    
    This reverts commit 8c2d81eb205bf05445eca616818681e047639eb5.
    
    I get a bunch of lost USE=test with this:
    ```
    $ emerge -p -uvDU @world
    [...]
    [ebuild   R    ] dev-libs/tree-sitter-bash-0.20.4-r1::gentoo  USE="-test*" 0 KiB
    [ebuild   R    ] dev-libs/libffi-3.4.4-r1:0/8::gentoo  USE="-debug (-exec-static-trampoline) -pax-kernel -static-libs -test*" ABI_X86="32 (64) (-x32)" 0 KiB
    [ebuild   R    ] dev-util/pkgconf-2.0.3:0/4::gentoo  USE="-test*" ABI_X86="32 (64) (-x32)" 0 KiB
    [ebuild   R    ] dev-libs/libpipeline-1.5.7::gentoo  USE="-test*" 0 KiB
    [ebuild   R    ] dev-util/fq-0.8.0::gentoo  USE="-test*" 0 KiB
    [...]
    ```
    
    Bug: https://bugs.gentoo.org/914441
    Signed-off-by: Sam James <sam@gentoo.org>

 lib/portage/package/ebuild/config.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 5 Larry the Git Cow gentoo-dev 2023-10-20 00:51:22 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3500483f75789c36e379c1b6546a09df7c11e2b1

commit 3500483f75789c36e379c1b6546a09df7c11e2b1
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-10-20 00:49:33 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-10-20 00:51:00 +0000

    sys-apps/portage: add 3.0.53
    
    Closes: https://bugs.gentoo.org/915120
    Closes: https://bugs.gentoo.org/821529
    Closes: https://bugs.gentoo.org/914441
    Closes: https://bugs.gentoo.org/914722
    Closes: https://bugs.gentoo.org/914873
    Closes: https://bugs.gentoo.org/915099
    Closes: https://bugs.gentoo.org/915123
    Closes: https://bugs.gentoo.org/915128
    Closes: https://bugs.gentoo.org/915136
    Closes: https://bugs.gentoo.org/915330
    Closes: https://bugs.gentoo.org/915494
    Closes: https://bugs.gentoo.org/915834
    Closes: https://bugs.gentoo.org/915903
    Closes: https://bugs.gentoo.org/900224
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.53.ebuild | 238 +++++++++++++++++++++++++++++++++
 2 files changed, 239 insertions(+)
Comment 6 Larry the Git Cow gentoo-dev 2024-03-24 22:20:11 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=72f41d07b5396195a98691fafeb3e5b207a76564

commit 72f41d07b5396195a98691fafeb3e5b207a76564
Author:     Raul E Rangel <rrangel@chromium.org>
AuthorDate: 2024-03-21 20:29:59 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-03-24 22:18:34 +0000

    Reapply "config: Don't directly modify FEATURES"
    
    This reverts commit b150419d28bd7afb98404a829c639584d34efc03.
    
    It turns out we need to keep the open coded version to avoid creating
    a persistent setting.
    See https://github.com/gentoo/portage/pull/1098#issuecomment-1761638611
    
    This change just adds a sorted() so we get deterministic ordering.
    
    Bug: https://bugs.gentoo.org/914441
    Signed-off-by: Raul E Rangel <rrangel@chromium.org>
    Closes: https://github.com/gentoo/portage/pull/1312
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/package/ebuild/config.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)