Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 931837 - --buildpkg builds binary, even if it already exists and you emerge that very package
Summary: --buildpkg builds binary, even if it already exists and you emerge that very ...
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-05-12 22:13 UTC by soundbastlerlive
Modified: 2024-05-13 03:23 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description soundbastlerlive 2024-05-12 22:13:22 UTC
--buildpkg builds binary, even if it already exists, so if you have --buildpkg and --usepkg (or --getbinpkg) in your default settings and emerge a binary, it will after installing create another binary (sometimes overwriting, sometimes adding a new one, growing the repository - i don't see the difference why sometimes).

i couldn't find any flag that will tell portage to build a pkg only if it doesn't exist yet (and especially if it just merged that exact pkg).
Comment 1 soundbastlerlive 2024-05-12 22:37:42 UTC
I don't think it's the same.
The usecases for me are:
*) always build bins, if they don't exist, so other machines can access them
*) delete all, reemerge everything on 1 host, reemerge all binaries on other hosts (e.g. after gcc upgrade, etc.)

emerging a binary should IMHO never build that same exact binary again, just merge it
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-13 01:28:39 UTC
it's a 'See Also', not a dupe.
Comment 3 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-13 01:29:11 UTC
(In reply to soundbastlerlive from comment #1)
> I don't think it's the same.
> The usecases for me are:
> *) always build bins, if they don't exist, so other machines can access them
> *) delete all, reemerge everything on 1 host, reemerge all binaries on other
> hosts (e.g. after gcc upgrade, etc.)
> 

Use eclean-pkg regularly for this?

> emerging a binary should IMHO never build that same exact binary again, just
> merge it

How does it know if you changed compiler, or added a new userpatch?
Comment 4 soundbastlerlive 2024-05-13 01:37:36 UTC
if I want a full rebuild, I manually delete all packages, emerge on one host and then emerge on others (which all have both --buildpkg and --usepkg), but right now emerge will create/overwrite the existing binary package when emerging one with these switches (like after a source emerge), that is the bug.
Comment 5 soundbastlerlive 2024-05-13 01:38:29 UTC
emerging a binary package should IMHO never before installing compress those same same exact binaries into a package again (like it was a source emerge), just merge it
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-13 01:44:17 UTC
I think you're mixing up issues here -- emerge --buildpkg X twice is supposed to make 2 binpkgs, but emerge --buildpkg --usepkg X, where a binpkg already exists, indeed shouldn't unpack-the-binpkg-install-it-and-then-make-a-new-one.
Comment 7 Eli Schwartz 2024-05-13 02:41:29 UTC
I confirm I got some very alarming results with --usepkg --buildpkg when installing two packages available as binaries.

It is not redownloading sources and recompiling them. But the one I had built locally, superficially appears to have been recompressed into a new *.gpkg.tar with a build number incrementing past the build number count from the binhost.

The newly appearing .gpkg has a timestamp from a couple days ago and is smaller than the one I installed but I haven't dug deeper yet.
Comment 8 soundbastlerlive 2024-05-13 02:48:55 UTC
(In reply to Sam James from comment #6)
> I think you're mixing up issues here -- emerge --buildpkg X twice is
> supposed to make 2 binpkgs, but emerge --buildpkg --usepkg X, where a binpkg
> already exists, indeed shouldn't
> unpack-the-binpkg-install-it-and-then-make-a-new-one.

I'm not mixing up anything, but you just confirmed my bug (--buildpkg --usepkg makes another binary pkg), that's what I meant.

Sometimes the exsting xpak gets overwritten, sometimes a new multi version -2, -3 etc. is created (I don't know why, because all hosts use the same make.conf with multi version enabled, but all of that wouldn't matter if the bug was fixed anyway)
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-05-13 02:50:33 UTC
My point was 'i couldn't find any flag that will tell portage to build a pkg only if it doesn't exist yet (and especially if it just merged that exact pkg).' was mixing things, as that would be separate behaviour and not the same.
Comment 10 Eli Schwartz 2024-05-13 02:57:41 UTC
The new one is signed by the release keyring. The old one is signed by my local signing key. It looks like it always prefers the remote package if possible and downloads it again?

The new package is glib-2.78.6-4.gpkg.tar but contains glib-2.78.6-1/ inside. Seems like it's being renamed in place in order to bump the build id.


(In reply to soundbastlerlive from comment #5)
> emerging a binary package should IMHO never before installing compress those
> same same exact binaries into a package again (like it was a source emerge),
> just merge it

Can you double check if it is just downloading a different gpkg.tar from a binhost and installing *that*?
Comment 11 soundbastlerlive 2024-05-13 03:09:00 UTC
> Can you double check if it is just downloading a different gpkg.tar from a
> binhost and installing *that*?

i delete all packages (the entire folder), then reemerge world on 1 host -> gurantees that only 1 package file exists.
but then on the second host (with buildpkg and usepkg) it will download that 1 pkg, unpack it, create a new pkg (bug) and merge/install it.
Comment 12 Eli Schwartz 2024-05-13 03:11:54 UTC
(In reply to soundbastlerlive from comment #11)
> > Can you double check if it is just downloading a different gpkg.tar from a
> > binhost and installing *that*?
> 
> i delete all packages (the entire folder), then reemerge world on 1 host ->
> gurantees that only 1 package file exists.
> but then on the second host (with buildpkg and usepkg) it will download that
> 1 pkg, unpack it, create a new pkg (bug) and merge/install it.

The filenames may be different because portage bumps the build id.

Check the sha256sum of the gpkg.tar file on the binhost and the one that is downloaded. I suspect that "create a new pkg" means using "cp dev-libs/foo-1-1.gpkg.tar dev-libs/foo-1-2.gpkg.tar" before extraction.
Comment 13 soundbastlerlive 2024-05-13 03:20:57 UTC
> Check the sha256sum of the gpkg.tar file on the binhost and the one that is
> downloaded. I suspect that "create a new pkg" means using "cp
> dev-libs/foo-1-1.gpkg.tar dev-libs/foo-1-2.gpkg.tar" before extraction.

i don't understand what difference it would make? a new pkg shouldn't be created with --buildpkg --usepkg, because clearly it already exists as it is being emerged right now - that's a bug, don't you agree?

but here are the files, you can see the repackaged ones from the secondary hosts are slightly smaller and checksums are different because of "new" date in metadata i guess:

# l /var/cache/distfiles/_binpkg/znver4/sys-power/acpid/
total 245K
-rw-r--r-- 1 root root 84K May 12 20:26 acpid-2.0.34-r1-1.xpak
-rw-r----- 1 root root 81K May 13 00:37 acpid-2.0.34-r1-2.xpak
-rw-r----- 1 root root 81K May 13 00:37 acpid-2.0.34-r1-3.xpak

# md5sum /var/cache/distfiles/_binpkg/znver4/sys-power/acpid/*
713a42e8447fb4a40e5c64468399700a  /var/cache/distfiles/_binpkg/znver4/sys-power/acpid/acpid-2.0.34-r1-1.xpak
ab29fa2fa5c9dc1951e69afc7c57130c  /var/cache/distfiles/_binpkg/znver4/sys-power/acpid/acpid-2.0.34-r1-2.xpak
d2a06651fac5070871b32cb95695c627  /var/cache/distfiles/_binpkg/znver4/sys-power/acpid/acpid-2.0.34-r1-3.xpak
Comment 14 soundbastlerlive 2024-05-13 03:23:20 UTC
here are all 3 packages' metadata from Packages files, exactly the same, except for size, mtime and sha1

BDEPEND: virtual/pkgconfig
BUILD_ID: 1
BUILD_TIME: 1715538408
CPV: sys-power/acpid-2.0.34-r1
DEFINED_PHASES: install postinst pretend setup
DEPEND: >=sys-kernel/linux-headers-3
EAPI: 8
IUSE: selinux
KEYWORDS: amd64 ~arm arm64 ~ia64 ~loong ~riscv x86
LICENSE: GPL-2
MD5: 713a42e8447fb4a40e5c64468399700a
PATH: sys-power/acpid/acpid-2.0.34-r1-1.xpak
RDEPEND: >=sys-libs/glibc-2.39-r6
REQUIRES: x86_64: libc.so.6
SHA1: a438ed0eee5782c77329189d346f4a4c7be25b9b
SIZE: 85662
USE: abi_x86_64 amd64 elibc_glibc kernel_linux
MTIME: 1715538409
REPO: gentoo

BDEPEND: virtual/pkgconfig
BUILD_ID: 1
BUILD_TIME: 1714073397
CPV: sys-power/acpid-2.0.34-r1
DEFINED_PHASES: install postinst pretend setup
DEPEND: >=sys-kernel/linux-headers-3
EAPI: 8
IUSE: selinux
KEYWORDS: amd64 ~arm arm64 ~ia64 ~loong ~riscv x86
LICENSE: GPL-2
MD5: ab29fa2fa5c9dc1951e69afc7c57130c
PATH: sys-power/acpid/acpid-2.0.34-r1-2.xpak
RDEPEND: >=sys-libs/glibc-2.39-r3
REQUIRES: x86_64: libc.so.6
SHA1: 1ee8a61ac7cb9553ab617fff39e4d5c775af95b1
SIZE: 82116
USE: abi_x86_64 amd64 elibc_glibc kernel_linux
MTIME: 1715553439
REPO: gentoo

BDEPEND: virtual/pkgconfig
BUILD_ID: 1
BUILD_TIME: 1714073397
CPV: sys-power/acpid-2.0.34-r1
DEFINED_PHASES: install postinst pretend setup
DEPEND: >=sys-kernel/linux-headers-3
EAPI: 8
IUSE: selinux
KEYWORDS: amd64 ~arm arm64 ~ia64 ~loong ~riscv x86
LICENSE: GPL-2
MD5: d2a06651fac5070871b32cb95695c627
PATH: sys-power/acpid/acpid-2.0.34-r1-3.xpak
RDEPEND: >=sys-libs/glibc-2.39-r3
REQUIRES: x86_64: libc.so.6
SHA1: 4fb0fbf184a2c7804ef1d79badf2fc08678c49a7
SIZE: 82131
USE: abi_x86_64 amd64 elibc_glibc kernel_linux
MTIME: 1715553443
REPO: gentoo