Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 934784 - PKGDIR should distinguish between the cache for remotely downloaded binaries vs locally built binaries
Summary: PKGDIR should distinguish between the cache for remotely downloaded binaries ...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 948643 933499
Blocks: 930730 945384
  Show dependency tree
 
Reported: 2024-06-23 19:28 UTC by Eli Schwartz
Modified: 2025-01-24 21:20 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eli Schwartz gentoo-dev 2024-06-23 19:28:27 UTC
When creating your own binhost downstream of another binhost -- e.g. you use Gentoo's official binhost described at https://wiki.gentoo.org/wiki/Project:Binhost but also build your own packages for things not covered by the official one -- it is disruptive to end up with binary packages that are duplicates of your upstream binhost, copied into your own binhost.

I'd like to be able to specify a PKGDIR that is used whenever I run emerge --buildpkg, and a different directory which is used by --getbinpkg. This would allow me to re-host only the former. Users would be expected to enable multiple binhosts and get packages from whoever originally created them; I wouldn't be responsible for monitoring the server footprint of files I didn't create.

This would also mean that it's trivially possible to rm -rf the remote downloads bincache to recover space, without running the risk of deleting binaries that will suddenly no longer be available anywhere.
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-23 19:29:23 UTC
Fixing this would also make it easier for us to default-on signature verification (binpkg-request-signature) because we don't suddenly require sigs from locally built binpkgs.
Comment 2 Zac Medico gentoo-dev 2024-07-13 23:13:39 UTC
It seems like we should be able to implement the bincaches as an extension of the  "remote" packages handing in binarytree. We don't necessarily need to maintain a local Packages file to track the content of each local bincache, since if the binarytree treats them as "remote" then it will get their metadata from the remote Packages file (which is cached in /var/cache/edb/binhost and can be frozen via binrepos.conf).
Comment 3 Eli Schwartz gentoo-dev 2024-07-14 06:10:15 UTC
That sounds plausible.

I kind of intended to mention this but I think I forgot, but as mentioned in the title, remotely downloaded packages are a *cache*, whereas locally built packages aren't a cache -- they are user data. I think this is an important distinction.


It logically follows that we don't need a Packages index for the cache as we use the edb index (also a cache!), whereas we do need a Packages index for the user data.
Comment 4 Zac Medico gentoo-dev 2024-07-15 00:09:35 UTC
The binarytree.isremote method can return True for these packages regardless of whether or not they've been downloaded, and we can add a new method to check if a remote package is cached locally or not.

We can make the behavior vary depending on whether or not the binrepo has a local cache directory defined in binrepos.conf. We might name the cache directory "location" the same as we do for source repositories.
Comment 5 Larry the Git Cow gentoo-dev 2024-07-17 02:26:11 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=20cd76664c11991e59b7d72b782fea96259ff9af

commit 20cd76664c11991e59b7d72b782fea96259ff9af
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2024-07-17 02:14:34 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2024-07-17 02:14:34 +0000

    binrepos.conf: Support custom download location
    
    Download packages to a custom location if it is configured
    in binrepos.conf, instead of PKGDIR. If a custom download
    location is not configured then inject downloaded packages
    into PKGDIR as usual.
    
    The binarytree download_required method should now be used
    instead of the isremote method to check if download is
    required, since a remote package may or may not be cached
    in the custom location. The get_local_repo_location method
    should be used to check if there is a custom download
    location, and if there is then downloaded packages must
    not be injected into PKGDIR.
    
    If any packages from a repo with a custom download location
    were injected into PKGDIR in the past, their identity will
    be recognized and will not be re-downloaded to the custom
    location.
    
    Bug: https://bugs.gentoo.org/934784
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/Binpkg.py           |  7 +++--
 lib/_emerge/BinpkgFetcher.py    |  8 ++++-
 lib/_emerge/BinpkgPrefetcher.py |  7 +++++
 lib/_emerge/Scheduler.py        |  8 +++--
 lib/_emerge/actions.py          |  2 +-
 lib/portage/binrepo/config.py   |  3 ++
 lib/portage/dbapi/bintree.py    | 70 +++++++++++++++++++++++++++++++++++++++--
 lib/portage/versions.py         |  6 +++-
 man/portage.5                   |  7 ++++-
 9 files changed, 106 insertions(+), 12 deletions(-)
Comment 6 Larry the Git Cow gentoo-dev 2024-09-11 01:30:49 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=02d0e00a1ba811b39140d10e17488f7fc3916534

commit 02d0e00a1ba811b39140d10e17488f7fc3916534
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-09-11 01:30:10 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-09-11 01:30:30 +0000

    sys-apps/portage: add 3.0.66
    
    Closes: https://bugs.gentoo.org/435066
    Closes: https://bugs.gentoo.org/907061
    Closes: https://bugs.gentoo.org/910560
    Closes: https://bugs.gentoo.org/933433
    Closes: https://bugs.gentoo.org/934220
    Closes: https://bugs.gentoo.org/934514
    Closes: https://bugs.gentoo.org/934784
    Closes: https://bugs.gentoo.org/935830
    Closes: https://bugs.gentoo.org/936273
    Closes: https://bugs.gentoo.org/937384
    Closes: https://bugs.gentoo.org/937485
    Closes: https://bugs.gentoo.org/937740
    Closes: https://bugs.gentoo.org/937888
    Closes: https://bugs.gentoo.org/937891
    Closes: https://bugs.gentoo.org/938127
    Closes: https://bugs.gentoo.org/933499
    Signed-off-by: Sam James <sam@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.66.ebuild | 227 +++++++++++++++++++++++++++++++++
 2 files changed, 228 insertions(+)
Comment 7 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-09-11 01:31:57 UTC
Is this one done or just partly? Please reopen if not.
Comment 8 Eli Schwartz gentoo-dev 2024-09-11 01:43:55 UTC
I think that we are good. I was test-running this patch for a while and it seems to do everything I want from it.