Until app-portage/gemato-20.2, it was possible to emerge --sync from behind a proxy that blocked CONNECT gentoo.org:443 but allowed CONNECT keys.gentoo.org:443. emerge --sync would first try CONNECT gentoo.org:443, get a 403, and immediately switch to CONNECT keys.gentoo.org:443 and all was fine. Starting with gemato-20.2, emerge --sync doesn't ever attempt keys.gentoo.org, it just keeps erroring and retrying gentoo.org: # emerge --sync >>> Syncing repository 'gentoo' into '/usr/portage'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD ... [ !! ] * Refreshing keys from keyserver hkps://keys.gentoo.org ...OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: General error OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: General error OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: General error OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: General error Rolled back to gemato-20.1 and it worked fine: # emerge --sync >>> Syncing repository 'gentoo' into '/usr/portage'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD ... [ !! ] * Refreshing keys from keyserver hkps://keys.gentoo.org ... [ ok ] * PGP verification method: legacy FEATURES=webrsync-gpg * Fetching most recent snapshot ... * Latest snapshot date: 20230519 * * Approximate snapshot timestamp: 1684543500 * Current local timestamp: 1684543201 * * The current local timestamp is possibly identical to the * timestamp of the latest snapshot. In order to force sync, use * the --revert option or remove the timestamp file located at * '/usr/portage/metadata/timestamp.x'. === Sync completed for gentoo Briefly looking over diffs between 20.1 and 20.2, nothing jumps out at me to explain the behavior change. But it's 100% consistent and follows flipping between 20.1 and 20.2.
I suspect "standard-resolver" setting may be relevant. Could you try editing gemato/openpgp.py and removing the "standard-resolver" line?
I hope it's *not* that (it might well be) because I suspect using standard-resolver fixes a bunch of issues we've had with the gai impl in gnupg..
Well, I have good news and bad news, and they are both that commenting out standard-resolver made gemato-20.2 work perfectly...
Are you able to check if the version with standard-resolver actually tries to CONNECT? I suspect it tries DNS lookup first for some reason.
Hm, yes and no. It tries the single proxy CONNECT gentoo.org:443, and once that fails it starts repeatedly looking up _pgpkey-https._tcp.keys.gentoo.org IN SRV. 4 requests, then about a minute, then another 4 requests, repeat. I think that request gets answered, but then it doesn't do anything with that answer - no attempt to connect to the IP it gets either directly or via another proxy CONNECT request. This also leaves lingering gpg-agent and dirmngr processes. It eventually gives up after 20 minutes of retries.
> It eventually gives up after 20 minutes of retries. (You had me confused for a moment wrt 20 minutes until I realised https://github.com/projg2/gemato/commit/a098d83e92fa807d95ee160b1506c4561b28eb2b was after the last tag.)
For comparison, would you be able also to monitor the variant without standard-resolver? I think we're seeing an upstream bug (sigh).
Sure, done. gemato 20.2 with standard-resolver commented out does: 1) proxied CONNECT gentoo.org:443 attempt (denied by proxy) 2) single SRV lookup for _pgpkey-https._tcp.keys.gentoo.org 3) single A lookup for keys.gentoo.org 4) several CONNECT keys.gentoo.org:443 (permitted by proxy) with standard-resolver enabled it never gets past #2, looping and retrying. It looks like there's some chatter back and forth between the moving parts: python, gpg-agent, and dirmngr, localhost ports 9050 and 9150. But no other network traffic is attempted other than the repeated SRV _pgpkey-https._tcp.keys.gentoo.org lookups every few seconds, then 10s, then 20s, 60s, 135s, etc.
Thanks a lot, I think that's enough for us to go on. I'm going to try to reproduce this without gemato being involved.
Ok, I have managed to reproduce with GPG alone. I'm going to file a bug upstream now.
That said, I'm not sure if my reproducer is correct. It's possible that GPG simply doesn't work at all without a working DNS server, and recursive-resolver somehow works around the limitations of your local resolver.
What resolver limitations, though? _pgpkey-https._tcp.keys.gentoo.org can be resolved just fine. It's just nothing is done with that information, it just goes into a loop repeating that lookup. Never issues another proxied https/hkps request, never attempts an outbound connection, etc.
(In reply to Hank Leininger from comment #12) > What resolver limitations, though? > > _pgpkey-https._tcp.keys.gentoo.org can be resolved just fine. > > It's just nothing is done with that information, it just goes into a loop > repeating that lookup. Never issues another proxied https/hkps request, > never attempts an outbound connection, etc. Ah, so my "reproducer" is different because it hits a problem when it can't be resolved.
Ah, yeah. Keep in mind also, in my setup keys.gentoo.org is permitted through the proxy, it's only gentoo.org that is not. That looks different from the scenario you reproduced in https://dev.gnupg.org/T6500, unless I'm reading it wrong.
(In reply to Hank Leininger from comment #14) > Ah, yeah. Keep in mind also, in my setup keys.gentoo.org is permitted > through the proxy, it's only gentoo.org that is not. That looks different > from the scenario you reproduced in https://dev.gnupg.org/T6500, unless I'm > reading it wrong. The gentoo.org fetching is done directly by gemato, so I've skipped that step. Just to confirm, you can try setting up a GNUPGHOME with a dirmngr.conf like mine (but with your proxy), and trying gpg call directly to confirm you get the same result. If it loops for you, logs from that could also be helpful to upstream.
We need to differentiate two cases: 1. WKD fetch: First and foremost the fetch over WKD is tried which apparently fails. I would try to look into why this is the case. But, it's difficult to debug. We had failing WKD in the past. Bug reports were openend on bgo. But, no real solution was found AFAIK. 2. HKPS fetch: As a fallback (see [1]), HKPS is tried thereafter. Here, the SRV DNS record for "_pgpkey-https._tcp.keys.gentoo.org" is looked up, although the RFC only knows of domains starting with "_hkp._tcp." (see [2]). If you have the time, you can delve into this weirdness. My suggestion: I suppose "_pgpkey-https._tcp.keys.gentoo.org" is treated the same way as the domain starting with "_hkp._tcp.". I suggest the removal of the following DNS record and test whether things work thereafter: ❯ dig +short srv _pgpkey-https._tcp.keys.gentoo.org _pgpkey-https._tcp.keys.geodns.gentoo.org. _pgpkey-https._tcp.keys.geodns-europe.gentoo.org. 20 100 443 kookaburra.keys.gentoo.org. 30 100 443 motmot.keys.gentoo.org. 10 100 443 trogan.keys.gentoo.org. The removal of the DNS record shouldn't affect things negatively, because the keyserver "hkps://keys.gentoo.org" is hardcoded in portage anyway (see [3]). And, it is passed to gemato (see [4]). [1] https://github.com/projg2/gemato/blob/a098d83e92fa807d95ee160b1506c4561b28eb2b/gemato/openpgp.py#L655-L658 [2] https://datatracker.ietf.org/doc/html/draft-shaw-openpgp-hkp-00#section-7 [3] https://github.com/gentoo/portage/blob/0181746a36f8ed9aa6610ab5ebdd0764c119c0df/cnf/repos.conf#L13 [4] https://github.com/projg2/gemato/blob/a098d83e92fa807d95ee160b1506c4561b28eb2b/gemato/openpgp.py#L645
@infra, would it be ok if I removed the SRV record, at least temporarily to see if that helps?
I noticed above mentioned weirdness with "_pgpkey-https._tcp.keys.gentoo.org" being looked up before (see [1]). Today, I was wondering why the SRV DNS record lookup wasn't shown. It turned out that you need "debug-all" instead of "debug-level guru" in ~/.gnupg/dirmngr.conf. Just mentioning in case you run in the same issue. I wonder why that is. The manpage states: > guru All of the debug messages you can get. [1] https://digitalcourage.social/@duxsco/110159374385071304
I don't use any proxy, but still get an error with "--refresh-keys": ❯ export GNUPGHOME="$(mktemp -d)" ❯ echo -e "standard-resolver\ndebug-all\nlog-file /tmp/meh.log" > $GNUPGHOME/dirmngr.conf ❯ gpg --import 13EBBDBEDE7A12775DFDB1BABB572E0E2D182910.asc gpg: keybox '/tmp/tmp.Bv7whCjqmD/pubring.kbx' created gpg: /tmp/tmp.Bv7whCjqmD/trustdb.gpg: trustdb created gpg: key BB572E0E2D182910: public key "Gentoo Linux Release Engineering (Automated Weekly Release Key) <releng@gentoo.org>" imported gpg: Total number processed: 1 gpg: imported: 1 ❯ gpg --keyserver hkps://keys.gentoo.org --refresh-keys gpg: refreshing 1 key from hkps://keys.gentoo.org gpg: keyserver refresh failed: General error ❯ cat /tmp/meh.log 2023-05-23 09:15:28 dirmngr[6549] listening on socket '/run/user/1000/gnupg/d.5d8zo7wutw3k6845mekh5ch1/S.dirmngr' 2023-05-23 09:15:28 dirmngr[6550.0] error loading certificate '/etc/ssl/certs/ca-certificates.crt': Certificate expired 2023-05-23 09:15:28 dirmngr[6550.0] error loading certificate '/etc/ssl/certs/ca-certificates.crt': Certificate expired 2023-05-23 09:15:28 dirmngr[6550.0] permanently loaded certificates: 140 2023-05-23 09:15:28 dirmngr[6550.0] runtime cached certificates: 0 2023-05-23 09:15:28 dirmngr[6550.0] trusted certificates: 140 (140,0,0,0) 2023-05-23 09:15:28 dirmngr[6550.0] failed to open cache dir file '/tmp/tmp.Bv7whCjqmD/crls.d/DIR.txt': No such file or directory 2023-05-23 09:15:28 dirmngr[6550.0] creating directory '/tmp/tmp.Bv7whCjqmD/crls.d' 2023-05-23 09:15:28 dirmngr[6550.0] new cache dir file '/tmp/tmp.Bv7whCjqmD/crls.d/DIR.txt' created 2023-05-23 09:15:28 dirmngr[6550.6] handler for fd 6 started 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> # Home: /tmp/tmp.Bv7whCjqmD 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> # Config: /tmp/tmp.Bv7whCjqmD/dirmngr.conf 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> OK Dirmngr 2.4.0 at your service 2023-05-23 09:15:28 dirmngr[6550.6] connection from process 6548 (1000:1000) 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 <- GETINFO version 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> D 2.4.0 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> OK 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 <- KEYSERVER --clear hkps://keys.gentoo.org 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> OK 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 <- KEYSERVER 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> S KEYSERVER hkps://keys.gentoo.org 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> OK 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 <- KS_GET -- 0x13EBBDBEDE7A12775DFDB1BABB572E0E2D182910 2023-05-23 09:15:28 dirmngr[6550.6] DBG: dns: getsrv(_pgpkey-https._tcp.keys.gentoo.org): General error 2023-05-23 09:15:28 dirmngr[6550.6] command 'KS_GET' failed: General error 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> ERR 167772161 General error <Dirmngr> 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 <- BYE 2023-05-23 09:15:28 dirmngr[6550.6] DBG: chan_6 -> OK closing connection 2023-05-23 09:15:28 dirmngr[6550.6] handler for fd 6 terminated
(In reply to David Sardari from comment #19) > I don't use any proxy, but still get an error with "--refresh-keys": > Errors happen, but given you were the person who committed this, did this work for you before or did you not test it?
For comparison: I personally don't use a SRV DNS record and don't run in such a problem: ❯ export GNUPGHOME="$(mktemp -d)" ❯ rm /tmp/meh.log rm: remove regular file '/tmp/meh.log'? y ❯ echo -e "standard-resolver\ndebug-all\nlog-file /tmp/meh.log" > $GNUPGHOME/dirmngr.conf ❯ gpg --import duxsco.asc gpg: keybox '/tmp/tmp.Ju3zhAY7Yq/pubring.kbx' created gpg: key 11BE5F68440E0758: 1 signature not checked due to a missing key gpg: /tmp/tmp.Ju3zhAY7Yq/trustdb.gpg: trustdb created gpg: key 11BE5F68440E0758: public key "David Sardari <d@duxsco.de>" imported gpg: Total number processed: 1 gpg: imported: 1 gpg: no ultimately trusted keys found ❯ gpg --keyserver hkps://revoked.duxsco.de --refresh-keys gpg: refreshing 1 key from hkps://revoked.duxsco.de gpg: key 11BE5F68440E0758: "David Sardari <d@duxsco.de>" 12 new signatures gpg: key 11BE5F68440E0758: "David Sardari <d@duxsco.de>" 6 new subkeys gpg: Total number processed: 1 gpg: new subkeys: 6 gpg: new signatures: 12 ❯ cat /tmp/meh.log 2023-05-23 09:33:42 dirmngr[8575] listening on socket '/run/user/1000/gnupg/d.y6kszefadcqaxqtq4uxyii95/S.dirmngr' 2023-05-23 09:33:42 dirmngr[8576.0] error loading certificate '/etc/ssl/certs/ca-certificates.crt': Certificate expired 2023-05-23 09:33:42 dirmngr[8576.0] error loading certificate '/etc/ssl/certs/ca-certificates.crt': Certificate expired 2023-05-23 09:33:42 dirmngr[8576.0] permanently loaded certificates: 140 2023-05-23 09:33:42 dirmngr[8576.0] runtime cached certificates: 0 2023-05-23 09:33:42 dirmngr[8576.0] trusted certificates: 140 (140,0,0,0) 2023-05-23 09:33:42 dirmngr[8576.0] failed to open cache dir file '/tmp/tmp.Ju3zhAY7Yq/crls.d/DIR.txt': No such file or directory 2023-05-23 09:33:42 dirmngr[8576.0] creating directory '/tmp/tmp.Ju3zhAY7Yq/crls.d' 2023-05-23 09:33:42 dirmngr[8576.0] new cache dir file '/tmp/tmp.Ju3zhAY7Yq/crls.d/DIR.txt' created 2023-05-23 09:33:42 dirmngr[8576.6] handler for fd 6 started 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> # Home: /tmp/tmp.Ju3zhAY7Yq 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> # Config: /tmp/tmp.Ju3zhAY7Yq/dirmngr.conf 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> OK Dirmngr 2.4.0 at your service 2023-05-23 09:33:42 dirmngr[8576.6] connection from process 8574 (1000:1000) 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 <- GETINFO version 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> D 2.4.0 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> OK 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 <- KEYSERVER --clear hkps://revoked.duxsco.de 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> OK 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 <- KEYSERVER 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> S KEYSERVER hkps://revoked.duxsco.de 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 -> OK 2023-05-23 09:33:42 dirmngr[8576.6] DBG: chan_6 <- KS_GET -- 0x3AAE5FC903BB199165D4C02711BE5F68440E0758 2023-05-23 09:33:42 dirmngr[8576.6] DBG: dns: getsrv(_pgpkey-https._tcp.revoked.duxsco.de) -> 0 records 2023-05-23 09:33:42 dirmngr[8576.6] DBG: dns: resolve_dns_name(revoked.duxsco.de): Success 2023-05-23 09:33:42 dirmngr[8576.6] resolve_dns_addr for 'revoked.duxsco.de': 'revoked.duxsco.de' [already known] 2023-05-23 09:33:42 dirmngr[8576.6] resolve_dns_addr for 'revoked.duxsco.de': 'revoked.duxsco.de' [already known] 2023-05-23 09:33:42 dirmngr[8576.6] number of system provided CAs: 142 2023-05-23 09:33:42 dirmngr[8576.6] DBG: Using TLS library: GNUTLS 3.7.8 2023-05-23 09:33:42 dirmngr[8576.6] DBG: http.c:connect_server: trying name='revoked.duxsco.de' port=443 2023-05-23 09:33:42 dirmngr[8576.6] DBG: dns: resolve_dns_name(revoked.duxsco.de): Success 2023-05-23 09:33:42 dirmngr[8576.6] DBG: http.c:1951:socket_new: object 0x00007fd1382e1220 for fd 7 created 2023-05-23 09:33:42 dirmngr[8576.6] DBG: http.c:request: 2023-05-23 09:33:42 dirmngr[8576.6] DBG: >> GET /pks/lookup?op=get&options=mr&search=0x3AAE5FC903BB199165D4C02711BE5F68440E0758 HTTP/1.0\r\n 2023-05-23 09:33:42 dirmngr[8576.6] DBG: >> Host: revoked.duxsco.de\r\n 2023-05-23 09:33:42 dirmngr[8576.6] DBG: http.c:request-header: 2023-05-23 09:33:42 dirmngr[8576.6] DBG: >> \r\n 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:response: 2023-05-23 09:33:43 dirmngr[8576.6] DBG: >> HTTP/1.1 301 Moved Permanently\r\n 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Date: Tue, 23 May 2023 07:33:42 GMT' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Content-Type: text/html' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Content-Length: 162' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Location: https://revoked.duxsco.de/3AAE5FC903BB199165D4C02711BE5F68440E0758.asc' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Connection: close' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Cache-Control: max-age=604800, must-revalidate, no-transform, public' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'referrer-policy: strict-origin-when-cross-origin' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-content-type-options: nosniff' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-frame-options: deny' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-robots-tag: noindex' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-xss-protection: 1; mode=block' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'expect-ct: enforce, max-age=31536000' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'strict-transport-security: max-age=31536000; includeSubDomains; preload' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'content-security-policy: default-src 'none';' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: '' 2023-05-23 09:33:43 dirmngr[8576.6] URL 'https://revoked.duxsco.de:443/pks/lookup?op=get&options=mr&search=0x3AAE5FC903BB199165D4C02711BE5F68440E0758' redirected to 'https://revoked.duxsco.de/3AAE5FC903BB199165D4C02711BE5F68440E0758.asc' (301) 2023-05-23 09:33:43 dirmngr[8576.6] DBG: Using TLS library: GNUTLS 3.7.8 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:connect_server: trying name='revoked.duxsco.de' port=443 2023-05-23 09:33:43 dirmngr[8576.6] DBG: dns: resolve_dns_name(revoked.duxsco.de): Success 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:1951:socket_new: object 0x00007fd138134870 for fd 7 created 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:request: 2023-05-23 09:33:43 dirmngr[8576.6] DBG: >> GET /3AAE5FC903BB199165D4C02711BE5F68440E0758.asc HTTP/1.0\r\n 2023-05-23 09:33:43 dirmngr[8576.6] DBG: >> Host: revoked.duxsco.de\r\n 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:request-header: 2023-05-23 09:33:43 dirmngr[8576.6] DBG: >> \r\n 2023-05-23 09:33:43 dirmngr[8576.6] DBG: http.c:response: 2023-05-23 09:33:43 dirmngr[8576.6] DBG: >> HTTP/1.1 200 OK\r\n 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Date: Tue, 23 May 2023 07:33:43 GMT' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Content-Type: application/pgp-keys' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Content-Length: 5539' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Last-Modified: Mon, 06 Feb 2023 22:21:56 GMT' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Connection: close' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Vary: Accept-Encoding' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'ETag: "63e17d84-15a3"' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Cache-Control: max-age=604800, must-revalidate, no-transform, public' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'referrer-policy: strict-origin-when-cross-origin' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-content-type-options: nosniff' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-frame-options: deny' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-robots-tag: noindex' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'x-xss-protection: 1; mode=block' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'expect-ct: enforce, max-age=31536000' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'strict-transport-security: max-age=31536000; includeSubDomains; preload' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'content-security-policy: default-src 'none';' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'content-disposition: attachment; filename=3AAE5FC903BB199165D4C02711BE5F68440E0758.asc' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: 'Accept-Ranges: bytes' 2023-05-23 09:33:43 dirmngr[8576.6] http.c:RESP: '' 2023-05-23 09:33:43 dirmngr[8576.6] DBG: chan_6 -> S SOURCE https://revoked.duxsco.de:443 2023-05-23 09:33:43 dirmngr[8576.6] DBG: (5539 bytes sent via D lines not shown) 2023-05-23 09:33:43 dirmngr[8576.6] DBG: chan_6 -> OK 2023-05-23 09:33:43 dirmngr[8576.6] DBG: chan_6 <- BYE 2023-05-23 09:33:43 dirmngr[8576.6] DBG: chan_6 -> OK closing connection 2023-05-23 09:33:43 dirmngr[8576.6] handler for fd 6 terminated
(In reply to Sam James from comment #20) > (In reply to David Sardari from comment #19) > > I don't use any proxy, but still get an error with "--refresh-keys": > > > > Errors happen, but given you were the person who committed this, did this > work for you before or did you not test it? I don't quite get your question. What do you mean with "this" and "it"? I tried with GnuPG only, and the SRV DNS record seems to break things.
For completeness sake: The keyserver hkps://revoked.duxsco.de is documented in my GitHub profile README: https://github.com/duxsco/duxsco
I only tried to reproduce the following line while keeping gemato out of the equation: https://github.com/projg2/gemato/blob/a098d83e92fa807d95ee160b1506c4561b28eb2b/gemato/openpgp.py#L648
(In reply to Sam James from comment #20) > (In reply to David Sardari from comment #19) > > I don't use any proxy, but still get an error with "--refresh-keys": > > > > Errors happen, but given you were the person who committed this, did this > work for you before or did you not test it? I think I get what you mean. While testing my PR (see [1]), WKD always worked on my machine. So, I never run in the use of HKPS. I should've covered this case in my tests. This was an error on my part. [1] https://github.com/projg2/gemato/pull/31
Ah, I see! Thanks for explaining.
But, this error caused by the SRV DNS record is independent of gemato. So, it should happen even without my patch. I just never noticed so far, because WKD always worked, and there was no need for my machines to fetch over HKPS.
In general, I prefer for HKPS to be dropped. I outlined my reasons in a previous bug (see [1]). Long story short: With HKPS, GLEP79 cannot be implemented, because 3rd-party sigs are dropped by GnuPG. Currently, L1 doesn't act as the sole root of trust on Gentoo machines. So, if the non-L1 and non-L2 keys get stolen, for example, it's difficult to fix things, because trust relies only on these "leaf" pubkeys. I am argumenting strictly from GLEP79 perspective. [1] https://bugs.gentoo.org/878479
And, if the fetch over WKD fails, a note should be printed by portage stating that access to "gentoo.org" or "openpgpkey.gentoo.org" over HTTPS is necessary: https://metacode.biz/openpgp/web-key-directory -> type in "releng@gentoo.org"
Hank, we've removed the SRV record now and it should have propagated to you already. Could you try the "broken" version again and see what happens now?
(In reply to Michał Górny from comment #30) > Hank, we've removed the SRV record now and it should have propagated to you > already. Could you try the "broken" version again and see what happens now? Yes, worked perfectly! - gemato 20.2 - restored/uncommented "standard-resolver" in openpgp.py # emerge --sync >>> Syncing repository 'gentoo' into '/usr/portage'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD ... [ !! ] * Refreshing keys from keyserver hkps://keys.gentoo.org ... [ ok ] [success continues] Meanwhile on the proxy server: 1.2.3.4 - - [24/May/2023:01:09:42 -0400] "CONNECT gentoo.org:443 HTTP/1.0" 403 3683 "-" "-" TCP_DENIED:HIER_NONE 1.2.3.4 - - [24/May/2023:01:09:44 -0400] "CONNECT keys.gentoo.org:443 HTTP/1.0" 200 27311 "-" "-" TCP_TUNNEL:HIER_DIRECT 1.2.3.4 - - [24/May/2023:01:09:44 -0400] "CONNECT keys.gentoo.org:443 HTTP/1.0" 200 35442 "-" "-" TCP_TUNNEL:HIER_DIRECT 1.2.3.4 - - [24/May/2023:01:09:44 -0400] "CONNECT keys.gentoo.org:443 HTTP/1.0" 200 37461 "-" "-" TCP_TUNNEL:HIER_DIRECT 1.2.3.4 - - [24/May/2023:01:09:44 -0400] "CONNECT keys.gentoo.org:443 HTTP/1.0" 200 46030 "-" "-" TCP_TUNNEL:HIER_DIRECT 1.2.3.4 - - [24/May/2023:01:09:45 -0400] "CONNECT gentoo.osuosl.org:443 HTTP/1.1" 200 6876 "-" "Wget/1.21.3" TCP_TUNNEL:HIER_DIRECT ... And on the DNS server: 24-May-2023 01:09:42.895 queries: info: client @0x7f787000d9f0 1.2.3.4#30292 (_pgpkey-https._tcp.keys.gentoo.org): query: _pgpkey-https._tcp.keys.gentoo.org IN SRV + (1.2.3.1) 24-May-2023 01:09:43.588 queries: info: client @0x7f7820009f10 1.2.3.4#51310 (keys.gentoo.org): query: keys.gentoo.org IN A + (1.2.3.1) So after CONNECT gentoo.org:443 fails, it does a lookup for _pgpkey-... but now that that fails, it quickly moves on to keys.gentoo.org lookup and CONNECT. This sounds like it's only a workaround, but it works now so good enough for me ;) Thanks!
Great! I never liked the SRV record. The RFC is unclear on this, e.g. it mentions only "_hkp._tcp.", but not "_hkps._tcp.". And, GnuPG isn't RFC compliant in this regard. Alternatively, the Gentoo project could make use of gpg's "--sig-keyserver-url" and "--sender" flags. With the public key not pre-imported, you get: ❯ gpg --verify asdf.txt.sig asdf.txt gpg: keybox '/tmp/tmp.pornVHpiZw/pubring.kbx' created gpg: Signature made Mi 24 Mai 2023 08:15:21 CEST gpg: using EDDSA key BA9783BF96DE7F9BFCEA0E369FA90B7D26619239 gpg: issuer "d@duxsco.de" <=== added by "--sender" gpg: Key available at: hkps://keys.example.org <=== added by "--sig-keyserver-url" gpg: Can't check signature: No public key With the e-mail address being known, you can use WKD and with the keyserver pointed out HKPS, of course. GnuPG can automatically make use of the embedded pointer to the keyserver with "gpg --keyserver-options honor-keyserver-url --refresh-keys".
Please, take note of the "man gpg" info on "--auto-key-retrieve", especially number 2. and 3.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=9268a92b9666eaaf263999b18220c0d56d8c476c commit 9268a92b9666eaaf263999b18220c0d56d8c476c Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-13 04:36:04 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-17 06:52:55 +0000 sync: rsync, git: respect --debug for gemato Respect --debug and pass it down to gemato so we get nice debugging output when e.g. 'refreshing keys' is stuck. Bug: https://bugs.gentoo.org/646194 Bug: https://bugs.gentoo.org/647696 Bug: https://bugs.gentoo.org/691666 Bug: https://bugs.gentoo.org/779766 Bug: https://bugs.gentoo.org/873133 Bug: https://bugs.gentoo.org/906875 Bug: https://github.com/projg2/gemato/issues/7 Bug: https://github.com/projg2/gemato/issues/25 Signed-off-by: Sam James <sam@gentoo.org> lib/portage/sync/modules/git/git.py | 15 +++++++++++++-- lib/portage/sync/modules/rsync/rsync.py | 11 +++++++++-- lib/portage/sync/syncbase.py | 12 ++++++++---- 3 files changed, 30 insertions(+), 8 deletions(-)
