Bug 922727

Summary:	keys.gentoo.org issue - results in emerge --sync failing when gentoo.org is not accessible
Product:	Gentoo Infrastructure	Reporter:	Hank Leininger <hlein>
Component:	Other	Assignee:	Gentoo Infrastructure <infra-bugs>
Status:	RESOLVED FIXED
Severity:	normal	CC:	ajak, floppym, gentoo, hlein
Priority:	Normal
Version:	unspecified
Hardware:	All
OS:	Linux
See Also:	https://bugs.gentoo.org/show_bug.cgi?id=877791
Whiteboard:
Package list:		Runtime testing required:	---

Description Hank Leininger 2024-01-22 22:33:38 UTC

It looks like keys.gentoo.org is borked. When behind a proxy that allows CONNECT keys.gentoo.org:443 but not CONNECT gentoo.org:443, emerge --sync fails when it tries to refresh keys.

If I try manually pulling @gentoo.org keys from keys.gentoo.org I get an error, as well:

$ gpg --keyserver hkps://keys.gentoo.org --refresh-keys '@gentoo.org'
gpg: refreshing 165 keys from hkps://keys.gentoo.org
gpg: keyserver refresh failed: No data

https://infra-status.gentoo.org/ does not have an entry for keys.gentoo.org, but if it did it could well be fooled anyway, because https://keys.gentoo.org/ is reachable (but searching for a keyid results in an error).

The symptoms, for the sake of others searching for similar problems with emerge --sync:

On a host not permitted to access gentoo.org:

/var/tmp/portage/webrsync-Or 100%[=============================================>]  43.10M  2.62MB/s    in 16s     

2024-01-22 17:17:32 (2.61 MB/s) - ‘/var/tmp/portage/webrsync-OrL1sf/gentoo-20240121.tar.xz’ saved [45196692/45196692]

 * Checking digest ...
 * Checking signature ...
[   INFO] Refreshing keys...
[  ERROR] OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://keys.gentoo.org
gpg: keyserver refresh failed: No data

 * ERROR: /:: failed:
 *   signature verification failed
 * 
 * If you need support, post the output of `emerge --info '=/::'`,
 * the complete build log and the output of `emerge -pqv '=/::'`.
 * Working directory: '/var/tmp/portage/webrsync-OrL1sf'
!!! emerge-webrsync error in /usr/portage

I see successful CONNECT keys.gentoo.org:443 through the proxy corresponding to this, but of course I can't see inside the request+response.

When gentoo.org is permitted:

/var/tmp/portage/webrsync-gw 100%[=============================================>]  43.10M  2.65MB/s    in 17s     

2024-01-22 17:19:08 (2.49 MB/s) - ‘/var/tmp/portage/webrsync-gwBCh7/gentoo-20240121.tar.xz’ saved [45196692/45196692]

 * Checking digest ...
 * Checking signature ...
[   INFO] Refreshing keys...
[   INFO] Keys refreshed.
[   INFO] File /var/tmp/portage/webrsync-gwBCh7/gentoo-20240121.tar.xz verified successfully against the signature in /var/tmp/portage/webrsync-gwBCh7/gentoo-20240121.tar.xz.gpgsig:
[   INFO] - status: OpenPGPSignatureStatus.GOOD
[   INFO] - valid: True, trusted: True
[   INFO] - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[   INFO] - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[   INFO] - timestamp: 2024-01-22 00:56:32 UTC
 * Getting snapshot timestamp ...
 * Syncing local repository ...
...

Based on metadata/timestamp.x timestamps I think this just started - hosts behind a proxy last successfully pulled a snapshot 2024-01-20, those not behind one could pull yesterday/today no problem.

Comment 1 Mike Gilbert gentoo-dev

2024-01-23 19:48:36 UTC

It probably uses WKD against gentoo.org by default.

When you have that blocked, it probably falls back on hkps://keys.gentoo.org, which does not work.

It seems very strange to me that you would block access to gentoo.org.

Comment 2 Hank Leininger 2024-01-23 19:56:27 UTC

We block everything by default, and only allow what is needed.

Main gentoo.org has never(?) been needed (except for some specific gemato bugs a bit ago). Just whatever specific mirrors are used, and keys.gentoo.org.

It looks like keys.gentoo.org works now, though:

$ gpg --keyserver hkps://keys.gentoo.org --refresh-keys '@gentoo.org'
gpg: refreshing 165 keys from hkps://keys.gentoo.org
gpg: key 39EA32FE8222EEEC: "Ulrich Müller <ulm@gentoo.org>" 2 new user IDs
gpg: key 39EA32FE8222EEEC: "Ulrich Müller <ulm@gentoo.org>" 2 new signatures
...

And now emerge --sync that doesn't allow gentoo.org also works, so I'm marking this as fixed.

Comment 3 Mike Gilbert gentoo-dev

2024-01-23 21:38:34 UTC

I would still suggest that you allow access to gentoo.org to allow WKD key refreshes to work properly.

Comment 4 John Helmert III archtester

2024-01-24 04:26:38 UTC

I wonder if it's silently falling back to the apex domain in a fashion similar to what was seen in bug 877791 or something, then hits the different error on gentoo.org.