Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 668302 - sys-apps/portage does not honour FETCHCOMMAND{,_HTTPS} while fetching "Packages" file over https
Summary: sys-apps/portage does not honour FETCHCOMMAND{,_HTTPS} while fetching "Packag...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - External Interaction (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 371447 (view as bug list)
Depends on: 625246 668334
Blocks: 739718
  Show dependency tree
 
Reported: 2018-10-11 11:01 UTC by Merlijn Wajer
Modified: 2021-01-18 02:42 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Merlijn Wajer 2018-10-11 11:01:18 UTC
At least on version 2.3.49, Portage no longer seems to honour the FETCHCOMMAND when fetching the "Packages" file(s) for binary packages, from PORTAGE_BINHOST URLs. This is what now happens on my (remote) systems:


> # FETCHCOMMAND='wget --certificate /mcs/auth/auth.pem --private-key /mcs/auth/auth-key.pem -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}"' emerge -uNDaG @world
> !!! Error fetching binhost package info from 'https://update.[removed]/os/'
> !!! HTTP Error 400: Bad Request
> !!! Error fetching binhost package info from 'https://update.[removed]/app/'
> !!! HTTP Error 400: Bad Request

Looking at dbapi/bintree.py, I've noticed that it doesn't even attempt to use the FETCHCOMMAND, but outright uses portage.util._urlopen.

This is a problem for me, since my update system relies on TLS client certificates not just for authentication, but also for determining what Packages need to be shipped to a system. The portage built in function does not support passing certificates. And thus, NGINX returns "400 Bad Request" since it expects client certificates.

Changing this line:
> if parsed_url.scheme not in ('https',) or _have_pep_476():
to "if False:" will make portage respect the FETCHCOMMAND again.

zmedico told me on IRC that this is intentional behaviour, but it's surprising behaviour to me, and it's breaking the update workflow for (my) remote machines. I am hoping we can figure out a way to at least have an option to respect FETCHCOMMAND rather than unconditionally call portage.util._urlopen. I would prefer that to adding client certificate support to the portage.util._urlopen function, since FETCHCOMMAND seems more flexible.
Comment 1 Merlijn Wajer 2018-10-11 11:05:11 UTC
On IRC it was brought up the current method supports HTTP 'If-Modified-Since'.

I think that supporting a feature like HTTP 'If-Modified-Since' should be quite feasible with wget in a FETCHCOMMAND as well if this is particularly important.
Comment 2 Zac Medico gentoo-dev 2018-10-11 17:28:22 UTC
In order to express things like this, we really need to introduce support for binary package repository configuration files. This will make it possible to specify a different FETCHCOMMAND for each binary package repository.
Comment 3 Zac Medico gentoo-dev 2020-09-06 18:43:57 UTC
*** Bug 371447 has been marked as a duplicate of this bug. ***
Comment 5 Larry the Git Cow gentoo-dev 2020-09-08 02:35:09 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=5ebc8a249b08318da5a2ca89cee2eed604f7e639

commit 5ebc8a249b08318da5a2ca89cee2eed604f7e639
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-07 00:13:13 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-08 01:38:08 +0000

    binrepos.conf: support fetchcommand customization (bug 668302)
    
    Support customization of fetchcommand and resumecommand in
    binrepos.conf, allowing customized authentication mechanisms for
    each repository.
    
    Bug: https://bugs.gentoo.org/668302
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/_emerge/BinpkgFetcher.py  | 29 +++++++++++++++++++----------
 lib/portage/binrepo/config.py |  2 ++
 lib/portage/dbapi/bintree.py  | 34 +++++++++++++++++++++++++---------
 man/portage.5                 | 14 ++++++++++++++
 4 files changed, 60 insertions(+), 19 deletions(-)
Comment 6 Larry the Git Cow gentoo-dev 2020-09-08 03:51:35 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd840baeef963f5fe746f9167976ce0eed83a3b4

commit fd840baeef963f5fe746f9167976ce0eed83a3b4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-08 03:44:51 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-08 03:51:23 +0000

    sys-apps/portage: Bump to version 3.0.6
    
     #668302 binrepos.conf: support fetchcommand customization
     #668334 Add binrepos.conf to replace PORTAGE_BINHOST
     #704416 env-update: create systemd user-session environment
     #737470 egencache --update-pkg-desc: emulate esync --verbose
     #737480 emerge --search: auto-detect regular expressions
     #739908 @change-deps: fix erroneous repeated rebuilds
     #740588 make.conf: Treat __* variables as local
     #740898 PORTAGE_BINHOST urlopen proxy support
     #740904 rsync and gemato proxy support
    
    Bug: https://bugs.gentoo.org/739718
    Bug: https://bugs.gentoo.org/668302
    Bug: https://bugs.gentoo.org/668334
    Bug: https://bugs.gentoo.org/704416
    Bug: https://bugs.gentoo.org/737470
    Bug: https://bugs.gentoo.org/737480
    Bug: https://bugs.gentoo.org/739908
    Bug: https://bugs.gentoo.org/740588
    Bug: https://bugs.gentoo.org/740898
    Bug: https://bugs.gentoo.org/740904
    Package-Manager: Portage-3.0.6, Repoman-3.0.1
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest             |   1 +
 sys-apps/portage/portage-3.0.6.ebuild | 265 ++++++++++++++++++++++++++++++++++
 2 files changed, 266 insertions(+)