Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 711766 - Cannot use keys.gentoo.org with some IP/ISP
Summary: Cannot use keys.gentoo.org with some IP/ISP
Status: RESOLVED INVALID
Alias: None
Product: Mirrors
Classification: Unclassified
Component: Server Problem (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Mirror Admins
URL: https://forums.gentoo.org/viewtopic-t...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-07 05:22 UTC by fiouzy
Modified: 2020-03-11 04:47 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info (file_711766.txt,6.51 KB, text/plain)
2020-03-07 05:22 UTC, fiouzy
Details

Note You need to log in before you can comment on or make changes to this bug.
Description fiouzy 2020-03-07 05:22:59 UTC
Created attachment 617360 [details]
emerge --info

https://forums.gentoo.org/viewtopic-t-1109330.html?sid=3c5a8675720ba966920320f675893716

Does keys.gentoo.org maintain an IP or ISP blacklist?

gpg key retrieval does not work with my fiber internet access (this used to work before I upgraded from xDSL a few days ago).

# gpg -v --debug-level=10 --keyserver hkps://keys.gentoo.org --recv-keys 18F703D702B1B9591373148C55D3238EC050396E
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: no running Dirmngr - starting '/usr/bin/dirmngr'
gpg: waiting for the dirmngr to come up ... (5s)
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.2.17 at your service
gpg: connection to dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.17
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://keys.gentoo.org
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
gpg: DBG: chan_3 <- ERR 219 Server indicated a failure <Unspecified source>
gpg: keyserver receive failed: Server indicated a failure
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks


DNS resolution:

# host keys.gentoo.org
keys.gentoo.org is an alias for keys.geodns.gentoo.org.
keys.geodns.gentoo.org is an alias for keys.geodns-asia.gentoo.org.
keys.geodns-asia.gentoo.org has address 140.211.166.190
keys.geodns-asia.gentoo.org has address 208.116.51.2
keys.geodns-asia.gentoo.org has address 89.238.71.4
keys.geodns-asia.gentoo.org has IPv6 address 2001:470:ea4a:1:230:48ff:fef8:9fdc
keys.geodns-asia.gentoo.org has IPv6 address 2a00:1828:a00d:ffff::4
keys.geodns-asia.gentoo.org has IPv6 address 2001:470:1f06:a91::2







When switching to another ISP (mobile phone tethering), this works as expected:

# gpg -v --debug-level=10 --keyserver hkps://keys.gentoo.org --recv-keys 18F703D702B1B9591373148C55D3238EC050396E
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.2.17 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.17
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://keys.gentoo.org
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
gpg: DBG: chan_3 <- S SOURCE https://140.211.166.190:443
gpg: DBG: chan_3 <- D -----BEGIN PGP PUBLIC KEY BLOCK-----%0AVersion: SKS 1.1.6%0AComment: Hostname: motmot.gentoo.org [truncated]


DNS resolution with mobile phone ISP:
# host keys.gentoo.org
keys.gentoo.org is an alias for keys.geodns.gentoo.org.
keys.geodns.gentoo.org is an alias for keys.geodns-asia.gentoo.org.
keys.geodns-asia.gentoo.org has address 208.116.51.2
keys.geodns-asia.gentoo.org has address 89.238.71.4
keys.geodns-asia.gentoo.org has address 140.211.166.190
keys.geodns-asia.gentoo.org has IPv6 address 2001:470:ea4a:1:230:48ff:fef8:9fdc
keys.geodns-asia.gentoo.org has IPv6 address 2001:470:1f06:a91::2
keys.geodns-asia.gentoo.org has IPv6 address 2a00:1828:a00d:ffff::4


As suggested in the aforementioned forum post, the only found workaround to perform portage sync is to switch from rsync to git.






(I can privately share the affected IP address upon request)
Comment 1 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-03-08 05:34:48 UTC
Can you narrow down which keys node you're actually connecting to?

You'll notice that the DNS serves up 3 different hosts, you can individually select them via:
hkps://trogan.keys.gentoo.org
hkps://motmot.keys.gentoo.org
hkps://kookaburra.keys.gentoo.org

There's no DNS records that let you select between IPv4 & IPv6 at that level, but I have seen prior IPv6 routing issues.
Comment 2 fiouzy 2020-03-10 08:20:12 UTC
Here are the attempts with the specific host names:


# gpg -v --debug-level=10 --keyserver hkps://trogan.keys.gentoo.org --recv-keys 18F703D702B1B9591373148C55D3238EC050396E
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.2.17 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.17
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://trogan.keys.gentoo.org
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
gpg: DBG: chan_3 <- ERR 219 Server indicated a failure <Unspecified source>
gpg: keyserver receive failed: Server indicated a failure
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks
# host trogan.keys.gentoo.org
trogan.keys.gentoo.org has address 89.238.71.4
trogan.keys.gentoo.org has IPv6 address 2a00:1828:a00d:ffff::4


# gpg -v --debug-level=10 --keyserver hkps://motmot.keys.gentoo.org --recv-keys 18F703D702B1B9591373148C55D3238EC050396E
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.2.17 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.17
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://motmot.keys.gentoo.org
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
gpg: DBG: chan_3 <- ERR 219 Server indicated a failure <Unspecified source>
gpg: keyserver receive failed: Server indicated a failure
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks
# host motmot.keys.gentoo.org
motmot.keys.gentoo.org has address 140.211.166.190
motmot.keys.gentoo.org has IPv6 address 2001:470:ea4a:1:230:48ff:fef8:9fdc


# gpg -v --debug-level=10 --keyserver hkps://kookaburra.keys.gentoo.org --recv-keys 18F703D702B1B9591373148C55D3238EC050396E
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.2.17 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.17
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://kookaburra.keys.gentoo.org
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
gpg: DBG: chan_3 <- ERR 219 Server indicated a failure <Unspecified source>
gpg: keyserver receive failed: Server indicated a failure
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks
# host kookaburra.keys.gentoo.org
kookaburra.keys.gentoo.org has address 208.116.51.2
kookaburra.keys.gentoo.org has IPv6 address 2001:470:1f06:a91::2
Comment 3 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-03-10 18:50:19 UTC
All 3 servers work fine for me.

Do you have broken IPv6 maybe?

If you inject a /etc/hosts entry with JUST the IPv4 address for the hostnames ({hostname}.keys.gentoo.org), does it work? You'll have to restart dirmngr due to the cache it has.
Comment 4 fiouzy 2020-03-11 02:02:32 UTC
Apologies, I got distracted by the client/server nature of the debug messages and I wrongly assumed the error response came from keys.gentoo.org: it is actually (and as hinted by your advice to restart dirmngr) the inter-process communication messages between gpg and dirmngr on my side that I was seeing in the debug output.

Enabling more verbose output from dirmngr (add verbose + debug-all in ~/.gnupg/dirmngr.conf) and directly querying it revealed the following two issues:
- inability to perform SRV record DNS query
- incomplete error feedback from dirmngr when handling DNS failure (at least when it comes to SRV record RFC2782[1])

# dirmngr
dirmngr[55635]: enabled debug flags: x509 crypto memory cache memstat hashing ipc dns network lookup extprog
dirmngr[55635.0]: permanently loaded certificates: 141
dirmngr[55635.0]:     runtime cached certificates: 0
dirmngr[55635.0]:            trusted certificates: 141 (140,0,0,1)
dirmngr[55635.0]: DBG: chan_3 -> # Home: /root/.gnupg
# Home: /root/.gnupg
dirmngr[55635.0]: DBG: chan_3 -> # Config: /root/.gnupg/dirmngr.conf
# Config: /root/.gnupg/dirmngr.conf
dirmngr[55635.0]: DBG: chan_3 -> OK Dirmngr 2.2.19 at your service
OK Dirmngr 2.2.19 at your service
GETINFO version
dirmngr[55635.0]: DBG: chan_3 <- GETINFO version
dirmngr[55635.0]: DBG: chan_3 -> D 2.2.19
D 2.2.19
dirmngr[55635.0]: DBG: chan_3 -> OK
OK
KEYSERVER --clear hkps://keys.gentoo.org
dirmngr[55635.0]: DBG: chan_3 <- KEYSERVER --clear hkps://keys.gentoo.org
dirmngr[55635.0]: DBG: chan_3 -> OK
OK
KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
dirmngr[55635.0]: DBG: chan_3 <- KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
dirmngr[55635.0]: DBG: dns: getsrv(_pgpkey-https._tcp.keys.gentoo.org): Try again later
dirmngr[55635.0]: command 'KS_GET' failed: Try again later
dirmngr[55635.0]: DBG: chan_3 -> ERR 167772472 Try again later <Dirmngr>
ERR 167772472 Try again later <Dirmngr>
dirmngr[55635.0]: DBG: chan_3 <- [eof]

(gnupg was also updated to 2.2.19 but it did not seem to be of importance here)

The DNS failure originates from the router (Netgear R7800) vendor firmware which does not understand SRV lookups [2]. Installing a custom firmware (such as OpenWRT) or bypassing the DHCP-provided embedded DNS server with an external one (such as CloudFlare 1.1.1.1) addresses the problem and let gpg retrieve the key:

dirmngr[55656.0]: DBG: chan_3 <- KS_GET -- 0x18F703D702B1B9591373148C55D3238EC050396E
dirmngr[55656.0]: DBG: dns: getsrv(_pgpkey-https._tcp.keys.gentoo.org) -> 0 records
dirmngr[55656.0]: DBG: dns: resolve_dns_name(keys.gentoo.org): Success
dirmngr[55656.0]: DBG: dns: resolve_dns_addr(): Success


Thank your for your time.

(I misdiagnosed the issue on the forum post where I initially stated that the issue happened with previous router, I guess it did not; I will amend the post [3] and publish this explanation)


[1] https://tools.ietf.org/html/rfc2782
[2] https://www.willglynn.com/2013/11/01/comcast-netgear-routers-eat-srv-records/
[3] https://forums.gentoo.org/viewtopic-t-1109330.html?sid=3c5a8675720ba966920320f675893716
Comment 5 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-03-11 04:47:19 UTC
fiouzy:
Thanks for the followup.

k_f:
I thought there was a GnuPG patch to treat resolver failure on SRV records differently than TRYAGAIN?