Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 745771 - app-portage/gemato-15.2, 16.1 + portage-3.0.x failover from wkd to keyserver not working
Summary: app-portage/gemato-15.2, 16.1 + portage-3.0.x failover from wkd to keyserver ...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Stabilization (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Michał Górny
URL:
Whiteboard:
Keywords: ALLARCHES, CC-ARCHES
Depends on:
Blocks:
 
Reported: 2020-09-30 18:12 UTC by Hank Leininger
Modified: 2020-10-02 20:42 UTC (History)
0 users

See Also:
Package list:
app-portage/gemato-16.2
Runtime testing required: ---
nattka: sanity-check+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Hank Leininger 2020-09-30 18:12:33 UTC
gemato 14.x and prior used to attempt WKD via https://gentoo.org/, and then fail over to hkps://keys.gentoo.org/ if that failed.

But the current versions in the tree - 15.2 (stable) and 16.1 (unstable) both abort immediately if they cannot connect to gentoo.org:443 - at least when behind a restrictive web proxy.

Discovered this in an environment where CONNECT gentoo.org:443 was not permitted, but CONNECT keys.gentoo.org:443 was - which worked fine previously.

The behavior with older gemato's:

# equery l portage gemato
 * Searching for portage ...
[IP-] [  ] sys-apps/portage-3.0.8:0

 * Searching for gemato ...
[I--] [??] app-portage/gemato-14.5:0

# . /etc/portage/make.conf && export http_proxy && /usr/bin/emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys via WKD ...                                                     [ !! ]
 * Refreshing keys from keyserver hkps://keys.gentoo.org ...                       [ ok ]
[SNIP]

[ The make.conf dance was required because old Gemato did not know how to find proxy settings from make.conf itself; it got support for that recently, see https://github.com/gentoo/portage/pull/607 ]

But versions 15.2 (stable) and 16.1 (unstable) both abort after the first attempt fails:

# equery l portage gemato
 * Searching for portage ...
[IP-] [  ] sys-apps/portage-3.0.8:0

 * Searching for gemato ...
[IP-] [  ] app-portage/gemato-16.1:0

# /usr/bin/emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys via WKD ...Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 317, in connect
    self._tunnel()
  File "/usr/lib/python3.7/http/client.py", line 927, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 403 Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gentoo.org', port=443): Max retries exceeded with url: /.well-known/openpgpkey/hu/9tik1tjkx1fe3wke63tpd7iikyogfbtq?l=repomirrorci (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/portage/util/_async/AsyncFunction.py", line 39, in _run
    result = self.target(*(self.args or []), **(self.kwargs or {}))
  File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 165, in sync
    taskmaster.run_tasks(tasks, func, status, options=task_opts)
  File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line 65, in run_tasks
    result = getattr(inst, func)(**kwargs)
  File "/usr/lib/python3.7/site-packages/portage/sync/modules/webrsync/webrsync.py", line 89, in sync
    self._refresh_keys(openpgp_env)
  File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 268, in _refresh_keys
    if openpgp_env.refresh_keys_wkd():
  File "/usr/lib/python3.7/site-packages/gemato/openpgp.py", line 419, in refresh_keys_wkd
    resp = requests.get(url, proxies=proxies)
  File "/usr/lib/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='gentoo.org', port=443): Max retries exceeded with url: /.well-known/openpgpkey/hu/9tik1tjkx1fe3wke63tpd7iikyogfbtq?l=repomirrorci (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

Action: sync for repo: gentoo, returned code = 1


#

gemato has an internal knob allow_wkd, but I can't figure out the right way to turn that when gemato is used by emerge, or to otherwise tell portage not to try WKD. If I hack up /usr/lib/python3.7/site-packages/portage/sync/syncbase.py to comment out the 'Refreshing keys via WKD' section entirely, it proceeds to refreshing from keyserver and works fine.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-10-01 11:13:15 UTC
My bad.  The code only accounts for non-200 status but I've completely forgotten to handle connect failures etc.
Comment 2 Larry the Git Cow gentoo-dev 2020-10-01 12:22:03 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=eeda16253769493db3bf50be19d7392302a5e89b

commit eeda16253769493db3bf50be19d7392302a5e89b
Author:     Michał Górny <mgorny@gentoo.org>
AuthorDate: 2020-10-01 12:17:09 +0000
Commit:     Michał Górny <mgorny@gentoo.org>
CommitDate: 2020-10-01 12:22:00 +0000

    app-portage/gemato: Bump to 16.2
    
    Bug: https://bugs.gentoo.org/745771
    Signed-off-by: Michał Górny <mgorny@gentoo.org>

 app-portage/gemato/Manifest           |  1 +
 app-portage/gemato/gemato-16.2.ebuild | 43 +++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-10-01 12:22:37 UTC
Could you confirm that 16.2 solves your issue?  If it does, I'll fast-stabilize it.
Comment 4 Hank Leininger 2020-10-01 21:55:45 UTC
Yes! Thanks @mgorny, with gemato-16.2 I once again get the expected behavior:

# equery l gemato portage
 * Searching for gemato ...
[I-O] [  ] app-portage/gemato-16.2:0

 * Searching for portage ...
[IP-] [  ] sys-apps/portage-3.0.8:0

# emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys via WKD ...                                     [ !! ]
 * Refreshing keys from keyserver hkps://keys.gentoo.org ...       [ ok ]
Fetching most recent snapshot ...
[snip]
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-10-02 17:44:27 UTC
Arch teams, please fast-stabilize this version.
Comment 6 Sam James archtester gentoo-dev Security 2020-10-02 20:42:35 UTC
amd64 arm arm64 hppa ppc ppc64 sparc x86 (ALLARCHES) done

all arches done