At least on version 2.3.49, Portage no longer seems to honour the FETCHCOMMAND when fetching the "Packages" file(s) for binary packages, from PORTAGE_BINHOST URLs. This is what now happens on my (remote) systems: > # FETCHCOMMAND='wget --certificate /mcs/auth/auth.pem --private-key /mcs/auth/auth-key.pem -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"' emerge -uNDaG @world > !!! Error fetching binhost package info from 'https://update.[removed]/os/' > !!! HTTP Error 400: Bad Request > !!! Error fetching binhost package info from 'https://update.[removed]/app/' > !!! HTTP Error 400: Bad Request Looking at dbapi/bintree.py, I've noticed that it doesn't even attempt to use the FETCHCOMMAND, but outright uses portage.util._urlopen. This is a problem for me, since my update system relies on TLS client certificates not just for authentication, but also for determining what Packages need to be shipped to a system. The portage built in function does not support passing certificates. And thus, NGINX returns "400 Bad Request" since it expects client certificates. Changing this line: > if parsed_url.scheme not in ('https',) or _have_pep_476(): to "if False:" will make portage respect the FETCHCOMMAND again. zmedico told me on IRC that this is intentional behaviour, but it's surprising behaviour to me, and it's breaking the update workflow for (my) remote machines. I am hoping we can figure out a way to at least have an option to respect FETCHCOMMAND rather than unconditionally call portage.util._urlopen. I would prefer that to adding client certificate support to the portage.util._urlopen function, since FETCHCOMMAND seems more flexible.
On IRC it was brought up the current method supports HTTP 'If-Modified-Since'. I think that supporting a feature like HTTP 'If-Modified-Since' should be quite feasible with wget in a FETCHCOMMAND as well if this is particularly important.
In order to express things like this, we really need to introduce support for binary package repository configuration files. This will make it possible to specify a different FETCHCOMMAND for each binary package repository.
*** Bug 371447 has been marked as a duplicate of this bug. ***
Patches posted for review: https://archives.gentoo.org/gentoo-portage-dev/message/1a4794a59788885b513f58f8646e7eaf https://github.com/gentoo/portage/pull/618
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=5ebc8a249b08318da5a2ca89cee2eed604f7e639 commit 5ebc8a249b08318da5a2ca89cee2eed604f7e639 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-07 00:13:13 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-08 01:38:08 +0000 binrepos.conf: support fetchcommand customization (bug 668302) Support customization of fetchcommand and resumecommand in binrepos.conf, allowing customized authentication mechanisms for each repository. Bug: https://bugs.gentoo.org/668302 Signed-off-by: Zac Medico <zmedico@gentoo.org> lib/_emerge/BinpkgFetcher.py | 29 +++++++++++++++++++---------- lib/portage/binrepo/config.py | 2 ++ lib/portage/dbapi/bintree.py | 34 +++++++++++++++++++++++++--------- man/portage.5 | 14 ++++++++++++++ 4 files changed, 60 insertions(+), 19 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd840baeef963f5fe746f9167976ce0eed83a3b4 commit fd840baeef963f5fe746f9167976ce0eed83a3b4 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2020-09-08 03:44:51 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2020-09-08 03:51:23 +0000 sys-apps/portage: Bump to version 3.0.6 #668302 binrepos.conf: support fetchcommand customization #668334 Add binrepos.conf to replace PORTAGE_BINHOST #704416 env-update: create systemd user-session environment #737470 egencache --update-pkg-desc: emulate esync --verbose #737480 emerge --search: auto-detect regular expressions #739908 @change-deps: fix erroneous repeated rebuilds #740588 make.conf: Treat __* variables as local #740898 PORTAGE_BINHOST urlopen proxy support #740904 rsync and gemato proxy support Bug: https://bugs.gentoo.org/739718 Bug: https://bugs.gentoo.org/668302 Bug: https://bugs.gentoo.org/668334 Bug: https://bugs.gentoo.org/704416 Bug: https://bugs.gentoo.org/737470 Bug: https://bugs.gentoo.org/737480 Bug: https://bugs.gentoo.org/739908 Bug: https://bugs.gentoo.org/740588 Bug: https://bugs.gentoo.org/740898 Bug: https://bugs.gentoo.org/740904 Package-Manager: Portage-3.0.6, Repoman-3.0.1 Signed-off-by: Zac Medico <zmedico@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.6.ebuild | 265 ++++++++++++++++++++++++++++++++++ 2 files changed, 266 insertions(+)