I have a lot of gentoo boxes connected to local networks. These boxes have no working name resolution to the internet. They only can reach the internet via a proxy. This proxy (squid) is capable of doing https, http and rsync. So it's not possible for them to refresh the gpg keys from the keyserver since they can not resolve it's IP nor can download keys via hkps. So what I need (and I think a lot others too) is a way to update the keys via proxy. I do not want to disable the verification since it is a security feature.
got the same problem over here. I would really appreciate a secure solution.
If we provide a way for you to override the keyserver, then you can host one for yourself. @mgorny, what do you think about supporting keyserver override in gemato?
I think it's a worthwhile goal but I'm not yet sure how to implement that nicely. In the meantime, I think setting an override in /etc/gnupg might work. CC-ing gnupg maintainers for wider review of proxy/etc. perspectives.
(In reply to Michał Górny from comment #3) > I think it's a worthwhile goal but I'm not yet sure how to implement that > nicely. In the meantime, I think setting an override in /etc/gnupg might > work. > > CC-ing gnupg maintainers for wider review of proxy/etc. perspectives. hkps is just a HTTP request over TLS, and already uses the default port of 443 so any proxy on the network should already pick it up. As for application specific proxies, see man dirmngr --honor-http-proxy If the environment variable ‘http_proxy’ has been set, use its value to access HTTP servers. --http-proxy host[:port] Use host and port to access HTTP servers. The use of this option overrides the environment variable ‘http_proxy’ regard‐ less whether --honor-http-proxy has been set. so one alternative would be for gemato to include honor-http-proxy in dirmngr.conf in the empheral homedir, that leaves the rest with users.
(In reply to Kristian Fiskerstrand from comment #4) > > so one alternative would be for gemato to include honor-http-proxy in > dirmngr.conf in the empheral homedir, that leaves the rest with users. Seems this is already done; 187 with open(os.path.join(self._home, 'dirmngr.conf'), 'w') as f: 188 f.write('''# autogenerated by gemato 189 190 # honor user's http_proxy setting 191 honor-http-proxy are you saying it isn't working?
(In reply to Kristian Fiskerstrand from comment #5) > (In reply to Kristian Fiskerstrand from comment #4) > > > > > > so one alternative would be for gemato to include honor-http-proxy in > > dirmngr.conf in the empheral homedir, that leaves the rest with users. > > Seems this is already done; > > 187 with open(os.path.join(self._home, 'dirmngr.conf'), 'w') as f: > 188 f.write('''# autogenerated by gemato > 189 > 190 # honor user's http_proxy setting > 191 honor-http-proxy > > are you saying it isn't working? No, it tries to resolve the name locally and to connect directly without using the proxy.
Are there any news on this? For me it still looks impossible that the gpg key refresh mechanism used within emerge --sync uses the proxy configured in http_proxy or https_proxy env vars. I use emerge-webrsync as a workaround but this is some kind of a pain in the a.. and as far as I see not secure since there is no tree verification process. For me as end user it looks like comment #4 is the solution but since now not implemented. If there is anything I can do to get this working, please let me know!
(In reply to tazinblack from comment #7) > I use emerge-webrsync as a workaround but this is some kind of a pain in the > a.. and as far as I see not secure since there is no tree verification > process. This one's easy, it's secure by default since bug 689506.
Doesn't it use the same key refresh mechanism, though?
True, emerge-webrsync uses gemato for key refresh since bug 689506.
Therefore, I don't see how one could work and the other would not.
You might be able to use proxychains as a workaround. With portage-2.3.76 (which has bug 693026 fix) you can sync via rsync with proxychains.
Doesn't look for me as if emerge-webrsync does a tree verification. # emerge-webrsync Fetching most recent snapshot ... Trying to retrieve 20191013 snapshot from http://distfiles.gentoo.org ... Fetching file portage-20191013.tar.xz.md5sum ... Fetching file portage-20191013.tar.xz.gpgsig ... Fetching file portage-20191013.tar.xz ... Checking digest ... Getting snapshot timestamp ... Syncing local tree ... Number of files: 161,499 (reg: 134,431, dir: 27,068) Number of created files: 1,160 (reg: 1,140, dir: 20) Number of deleted files: 1,742 (reg: 1,613, dir: 129) Number of regular files transferred: 17,251 Total file size: 216.79M bytes Total transferred file size: 46.20M bytes Literal data: 46.20M bytes Matched data: 0 bytes File list size: 1.77M File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 29.97M Total bytes received: 441.26K sent 29.97M bytes received 441.26K bytes 1.07M bytes/sec total size is 216.79M speedup is 7.13 Cleaning up ... # Also the time it needs is to short for that. I also can't see that emerge-webrsync tries to update any keys like emerge --sync does # emerge --sync >>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'... * Using keys from /usr/share/openpgp-keys/gentoo-release.asc * Refreshing keys via WKD ... [ !! ] * Refreshing keys from keyserver hkps://keys.gentoo.org ...OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: No such file or directory OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://keys.gentoo.org gpg: keyserver refresh failed: No such file or directory Also I don't understand the problems making the keyrefresh work with a proxy since Comment #4 in my eyes shows the solution. But I'm of course only a user not a developer. So I will give proxychains a try and hope that you will manage to integrate the use of a proxy with the envvars. Again the offer: If you need something, let me know. Maybe it would help to let you take a look on my system via teamviewer or something.
(In reply to tazinblack from comment #13) > Doesn't look for me as if emerge-webrsync does a tree verification. > > > # emerge-webrsync > Fetching most recent snapshot ... For sync-webrsync-verify-signature to work, you can't call emerge-webrsync directly. You have to set sync-type = webrsync in /etc/portage/repos.conf, and run emerge --sync. You want something like this in repos.conf: [gentoo] sync-type = webrsync sync-webrsync-verify-signature = yes sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc sync-openpgp-keyserver = hkps://keys.gentoo.org
(In reply to Zac Medico from comment #14) > (In reply to tazinblack from comment #13) > > Doesn't look for me as if emerge-webrsync does a tree verification. > > > > > > # emerge-webrsync > > Fetching most recent snapshot ... > > For sync-webrsync-verify-signature to work, you can't call emerge-webrsync > directly. You have to set sync-type = webrsync in /etc/portage/repos.conf, > and run emerge --sync. You want something like this in repos.conf: > > [gentoo] > sync-type = webrsync > sync-webrsync-verify-signature = yes > sync-openpgp-key-path = /usr/share/openpgp-keys/gentoo-release.asc > sync-openpgp-keyserver = hkps://keys.gentoo.org ok, that explains why I do not have the problems there. Also this means that it would be more secure to have key refresh working via proxy out of the box. I think there are a lot more with the same problem
Please try the following: export http_proxy=... umask 077 mkdir /tmp/gpg cat > /tmp/gpg/dirmngr.conf <-EOF honor-http-proxy EOF export GNUPGHOME=/tmp/gpg gpg --locate-key infrastructure@gentoo.org It works as expected for me (i.e. I see a CONNECT request in access log).
Or you could try the following first: export http_proxy=... gemato openpgp-verify -K /usr/share/openpgp-keys/gentoo-release.asc - It's going to refresh keys before waiting for input.
(In reply to Michał Górny from comment #17) > Or you could try the following first: > > export http_proxy=... > gemato openpgp-verify -K /usr/share/openpgp-keys/gentoo-release.asc - > > It's going to refresh keys before waiting for input. somebox / # export http_proxy="http://192.168.171.67:3128" somebox / # LANG=en_US.utf8 gemato openpgp-verify -K /usr/share/openpgp-keys/gentoo-release.asc - INFO:root:Refreshing keys... ERROR:root:OpenPGP keyring refresh failed: gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net gpg: keyserver refresh failed: No such file or directory somebox /etc/portage # That does not look very promising :-(
(In reply to Michał Górny from comment #16) > Please try the following: > > export http_proxy=... > umask 077 > mkdir /tmp/gpg > cat > /tmp/gpg/dirmngr.conf <-EOF > honor-http-proxy > EOF > export GNUPGHOME=/tmp/gpg > gpg --locate-key infrastructure@gentoo.org > > It works as expected for me (i.e. I see a CONNECT request in access log). As this one did not work for me I modified it a bit: somebox / # umask 077 somebox / # mkdir /tmp/gpg somebox / # echo " honor-http-proxy" >/tmp/gpg/dirmngr.conf somebox / # export GNUPGHOME=/tmp/gpg somebox / # gpg --locate-key infrastructure@gentoo.org gpg: keybox '/tmp/gpg/pubring.kbx' created gpg: /tmp/gpg/trustdb.gpg: trustdb created gpg: error retrieving 'infrastructure@gentoo.org' via WKD: No such file or directory gpg: error reading key: No such file or directory somebox / # And here your original: somebox / # rm -R /tmp/gpg/ somebox / # umask 077 somebox / # mkdir /tmp/gpg somebox / # cat > /tmp/gpg/dirmngr.conf <-EOF -bash: -EOF: Datei oder Verzeichnis nicht gefunden somebox / # honor-http-proxy -bash: honor-http-proxy: Kommando nicht gefunden. somebox / # EOF -bash: EOF: Kommando nicht gefunden. somebox / # export GNUPGHOME=/tmp/gpg somebox / # gpg --locate-key infrastructure@gentoo.org gpg: keybox '/tmp/gpg/pubring.kbx' created gpg: /tmp/gpg/trustdb.gpg: trustdb created gpg: error retrieving 'infrastructure@gentoo.org' via WKD: No such file or directory gpg: error reading key: No such file or directory somebox / # Same here. Not working
Do you have access to the proxy logs, or at least knowledge what software is it running? It seems that the proxy doesn't work properly.
You can also try adding --debug to gpg command but I'm not sure if it's going to produce anything useful.
(In reply to Michał Górny from comment #20) > Do you have access to the proxy logs, or at least knowledge what software is > it running? It seems that the proxy doesn't work properly. there is no connection to the proxy while running these commands. Neither in the proxy logs nor can I see any connection in the firewall logs. Otherwise when I emerge some new software or do a emerge --sync. I can see proxy access and connections going over the firewall to the proxy. So in my eyes it looks like it just doesn't use the proxy. In my installations I set the proxy env vars using /etc/env.d/99local with this content: somebox ~ # cat /etc/env.d/99local http_proxy="http://192.168.171.67:3128" https_proxy="http://192.168.171.67:3128" ftp_proxy="http://192.168.171.67:3128" RSYNC_PROXY="192.168.171.67:3128" somebox ~ # So maybe these tools use another environment? Is that possible?
Even this is not going via the proxy: somebox /tmp/temp # LANG=en_US gpg --keyserver-options http-proxy=192.168.171.67:3128 --locate-key infrastructure@gentoo.org gpg: directory '/root/.gnupg' created gpg: keybox '/root/.gnupg/pubring.kbx' created gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: error retrieving 'infrastructure@gentoo.org' via WKD: No such file or directory gpg: error reading key: No such file or directory somebox /tmp/temp #
What are your app-crypt/gnupg flags?
somebox /tmp/gpg # emerge -pv app-crypt/gnupg These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] app-crypt/gnupg-2.2.17::gentoo USE="bzip2 ldap nls readline smartcard ssl -doc (-selinux) -tofu -tools -usb -user-socket -wks-server" 0 KiB Total: 1 package (1 reinstall), Size of downloads: 0 KiB somebox /tmp/gpg #
Ok, please additionally add to dirmngr.conf the following lines: debug-level guru debug-all log-file /tmp/dirmngr.log and redo the rest. Then paste the log. If you reuse the old GNUPGHOME, you need to kill dirmngr so that it will reread config.
somebox /tmp # cat dirmngr.log 2019-10-17 16:23:30 dirmngr[17491] listening on socket '/tmp/gpg/S.dirmngr' 2019-10-17 16:23:30 dirmngr[17492.0] permanently loaded certificates: 141 2019-10-17 16:23:30 dirmngr[17492.0] runtime cached certificates: 0 2019-10-17 16:23:30 dirmngr[17492.0] trusted certificates: 141 (140,0,0,1) 2019-10-17 16:23:30 dirmngr[17492.6] handler for fd 6 started 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> # Home: /tmp/gpg 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> # Config: /tmp/gpg/dirmngr.conf 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> OK Dirmngr 2.2.17 at your service 2019-10-17 16:23:30 dirmngr[17492.6] connection from process 17489 (0:0) 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 <- GETINFO version 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> D 2.2.17 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> OK 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 <- WKD_GET -- infrastructure@gentoo.org 2019-10-17 16:23:30 dirmngr[17492.6] stat'ing '/etc/resolv.conf' failed: No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] stat'ing '/etc/resolv.conf' failed: No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] failed to load '/etc/resolv.conf': No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] DBG: dns: resolve_dns_name(openpgpkey.gentoo.org): No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] stat'ing '/etc/resolv.conf' failed: No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] stat'ing '/etc/resolv.conf' failed: No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] failed to load '/etc/resolv.conf': No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] DBG: dns: getsrv(_openpgpkey._tcp.gentoo.org): No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] command 'WKD_GET' failed: No such file or directory 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> ERR 167805009 No such file or directory <Dirmngr> 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 <- BYE 2019-10-17 16:23:30 dirmngr[17492.6] DBG: chan_6 -> OK closing connection 2019-10-17 16:23:30 dirmngr[17492.6] handler for fd 6 terminated somebox /tmp # Ok, now I see the problem. It tries to do a local DNS lookup which it shouldn't do. It should pass everything to the proxy.
Indeed. I can understand this to some degree as the new version of WKD optionally supports specifying some data in DNS, so you can't really get it via proxy (unless you reinvent DNS resolver). However, I think it should still fallback to normal requests rather than bailing out. Could you try if creating an empty /etc/resolv.conf helps? It seems that the lack of this file is making it run in circles.
(In reply to Michał Górny from comment #28) > Indeed. I can understand this to some degree as the new version of WKD > optionally supports specifying some data in DNS, so you can't really get it > via proxy (unless you reinvent DNS resolver). However, I think it should > still fallback to normal requests rather than bailing out. > > Could you try if creating an empty /etc/resolv.conf helps? It seems that > the lack of this file is making it run in circles. I just did that, same idea. Then the error message "file not found" is gone but the result is the same. I do not understand why DNS is needed here. The idea of a proxy is that some machines which can't access the internet directly use a proxy. Oy maybe an internal DNS can't resolve external DNS names.
(In reply to tazinblack from comment #29) > (In reply to Michał Górny from comment #28) > > Indeed. I can understand this to some degree as the new version of WKD > > optionally supports specifying some data in DNS, so you can't really get it > > via proxy (unless you reinvent DNS resolver). However, I think it should > > still fallback to normal requests rather than bailing out. > > > > Could you try if creating an empty /etc/resolv.conf helps? It seems that > > the lack of this file is making it run in circles. > > I just did that, same idea. Then the error message "file not found" is gone > but the result is the same. Could you include the full debug? > I do not understand why DNS is needed here. The idea of a proxy is that some > machines which can't access the internet directly use a proxy. Oy maybe an > internal DNS can't resolve external DNS names. Yeah, I get it. But HTTP proxy can't generally be used to query non-A/AAAA records. You'd have to reimplement DNS resolver and make it run via CONNECT.
(In reply to Michał Górny from comment #30) > (In reply to tazinblack from comment #29) > > (In reply to Michał Górny from comment #28) > > > Indeed. I can understand this to some degree as the new version of WKD > > > optionally supports specifying some data in DNS, so you can't really get it > > > via proxy (unless you reinvent DNS resolver). However, I think it should > > > still fallback to normal requests rather than bailing out. > > > > > > Could you try if creating an empty /etc/resolv.conf helps? It seems that > > > the lack of this file is making it run in circles. > > > > I just did that, same idea. Then the error message "file not found" is gone > > but the result is the same. > > Could you include the full debug? > > > I do not understand why DNS is needed here. The idea of a proxy is that some > > machines which can't access the internet directly use a proxy. Oy maybe an > > internal DNS can't resolve external DNS names. > > Yeah, I get it. But HTTP proxy can't generally be used to query non-A/AAAA > records. You'd have to reimplement DNS resolver and make it run via CONNECT. Sorry for the delay. Here you are: #cat dirmngr.log 2019-10-17 16:28:05 dirmngr[17548] listening on socket '/tmp/gpg/S.dirmngr' 2019-10-17 16:28:05 dirmngr[17549.0] permanently loaded certificates: 141 2019-10-17 16:28:05 dirmngr[17549.0] runtime cached certificates: 0 2019-10-17 16:28:05 dirmngr[17549.0] trusted certificates: 141 (140,0,0,1) 2019-10-17 16:28:05 dirmngr[17549.6] handler for fd 6 started 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> # Home: /tmp/gpg 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> # Config: /tmp/gpg/dirmngr.conf 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> OK Dirmngr 2.2.17 at your service 2019-10-17 16:28:05 dirmngr[17549.6] connection from process 17546 (0:0) 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 <- GETINFO version 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> D 2.2.17 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> OK 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 <- WKD_GET -- infrastructure@gentoo.org 2019-10-17 16:28:05 dirmngr[17549.6] DBG: dns: libdns initialized 2019-10-17 16:28:05 dirmngr[17549.6] DBG: dns: resolve_dns_name(openpgpkey.gentoo.org): Server indicated a failure 2019-10-17 16:28:05 dirmngr[17549.6] DBG: dns: getsrv(_openpgpkey._tcp.gentoo.org): Server indicated a failure 2019-10-17 16:28:05 dirmngr[17549.6] command 'WKD_GET' failed: Server indicated a failure <Unspecified source> 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> ERR 219 Server indicated a failure <Unspecified source> 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 <- BYE 2019-10-17 16:28:05 dirmngr[17549.6] DBG: chan_6 -> OK closing connection 2019-10-17 16:28:05 dirmngr[17549.6] handler for fd 6 terminated
Is there a problem with the DNS record itself? For example, the following shows a lookup failure : https://tools.dnsstuff.com/#dnsLookup|type=domain&&value=_openpgpkey._tcp.gentoo.org&&recordType=SRV&&displaytype=pretty