When creating your own binhost downstream of another binhost -- e.g. you use Gentoo's official binhost described at https://wiki.gentoo.org/wiki/Project:Binhost but also build your own packages for things not covered by the official one -- it is disruptive to end up with binary packages that are duplicates of your upstream binhost, copied into your own binhost. I'd like to be able to specify a PKGDIR that is used whenever I run emerge --buildpkg, and a different directory which is used by --getbinpkg. This would allow me to re-host only the former. Users would be expected to enable multiple binhosts and get packages from whoever originally created them; I wouldn't be responsible for monitoring the server footprint of files I didn't create. This would also mean that it's trivially possible to rm -rf the remote downloads bincache to recover space, without running the risk of deleting binaries that will suddenly no longer be available anywhere.
Fixing this would also make it easier for us to default-on signature verification (binpkg-request-signature) because we don't suddenly require sigs from locally built binpkgs.
It seems like we should be able to implement the bincaches as an extension of the "remote" packages handing in binarytree. We don't necessarily need to maintain a local Packages file to track the content of each local bincache, since if the binarytree treats them as "remote" then it will get their metadata from the remote Packages file (which is cached in /var/cache/edb/binhost and can be frozen via binrepos.conf).
That sounds plausible. I kind of intended to mention this but I think I forgot, but as mentioned in the title, remotely downloaded packages are a *cache*, whereas locally built packages aren't a cache -- they are user data. I think this is an important distinction. It logically follows that we don't need a Packages index for the cache as we use the edb index (also a cache!), whereas we do need a Packages index for the user data.
The binarytree.isremote method can return True for these packages regardless of whether or not they've been downloaded, and we can add a new method to check if a remote package is cached locally or not. We can make the behavior vary depending on whether or not the binrepo has a local cache directory defined in binrepos.conf. We might name the cache directory "location" the same as we do for source repositories.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=20cd76664c11991e59b7d72b782fea96259ff9af commit 20cd76664c11991e59b7d72b782fea96259ff9af Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2024-07-17 02:14:34 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2024-07-17 02:14:34 +0000 binrepos.conf: Support custom download location Download packages to a custom location if it is configured in binrepos.conf, instead of PKGDIR. If a custom download location is not configured then inject downloaded packages into PKGDIR as usual. The binarytree download_required method should now be used instead of the isremote method to check if download is required, since a remote package may or may not be cached in the custom location. The get_local_repo_location method should be used to check if there is a custom download location, and if there is then downloaded packages must not be injected into PKGDIR. If any packages from a repo with a custom download location were injected into PKGDIR in the past, their identity will be recognized and will not be re-downloaded to the custom location. Bug: https://bugs.gentoo.org/934784 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/Binpkg.py | 7 +++-- lib/_emerge/BinpkgFetcher.py | 8 ++++- lib/_emerge/BinpkgPrefetcher.py | 7 +++++ lib/_emerge/Scheduler.py | 8 +++-- lib/_emerge/actions.py | 2 +- lib/portage/binrepo/config.py | 3 ++ lib/portage/dbapi/bintree.py | 70 +++++++++++++++++++++++++++++++++++++++-- lib/portage/versions.py | 6 +++- man/portage.5 | 7 ++++- 9 files changed, 106 insertions(+), 12 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=02d0e00a1ba811b39140d10e17488f7fc3916534 commit 02d0e00a1ba811b39140d10e17488f7fc3916534 Author: Sam James <sam@gentoo.org> AuthorDate: 2024-09-11 01:30:10 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2024-09-11 01:30:30 +0000 sys-apps/portage: add 3.0.66 Closes: https://bugs.gentoo.org/435066 Closes: https://bugs.gentoo.org/907061 Closes: https://bugs.gentoo.org/910560 Closes: https://bugs.gentoo.org/933433 Closes: https://bugs.gentoo.org/934220 Closes: https://bugs.gentoo.org/934514 Closes: https://bugs.gentoo.org/934784 Closes: https://bugs.gentoo.org/935830 Closes: https://bugs.gentoo.org/936273 Closes: https://bugs.gentoo.org/937384 Closes: https://bugs.gentoo.org/937485 Closes: https://bugs.gentoo.org/937740 Closes: https://bugs.gentoo.org/937888 Closes: https://bugs.gentoo.org/937891 Closes: https://bugs.gentoo.org/938127 Closes: https://bugs.gentoo.org/933499 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.66.ebuild | 227 +++++++++++++++++++++++++++++++++ 2 files changed, 228 insertions(+)
Is this one done or just partly? Please reopen if not.
I think that we are good. I was test-running this patch for a while and it seems to do everything I want from it.