Today I run mirrorselect -HDs5 to fetch a latest fastest mirror list and encounter: _deeptime(): maxtime is 10 deeptime(): ip's for host mirror.dkm.cz: ['86.49.49.49', '[2a02:8300:8000:3::49]'] deeptime(): testing url: https://86.49.49.49/gentoo/distfiles/../snapshots/deltas/snapshot-20171127-20171128.patch.bz2 Traceback (most recent call last): File "/usr/lib/python-exec/python3.5/mirrorselect", line 61, in <module> MirrorSelect().main(sys.argv) File "/usr/lib64/python3.5/site-packages/mirrorselect/main.py", line 373, in main urls = self.select_urls(hosts, options) File "/usr/lib64/python3.5/site-packages/mirrorselect/main.py", line 322, in select_urls selector = Deep(hosts, options, self.output) File "/usr/lib64/python3.5/site-packages/mirrorselect/selectors.py", line 226, in __init__ self.deeptest() File "/usr/lib64/python3.5/site-packages/mirrorselect/selectors.py", line 253, in deeptest mytime, ignore = self.deeptime(host, maxtime) File "/usr/lib64/python3.5/site-packages/mirrorselect/selectors.py", line 338, in deeptime ip, ips[ips.index(ip):]) File "/usr/lib64/python3.5/site-packages/mirrorselect/selectors.py", line 421, in _test_connection f = url_open(test_url) File "/usr/lib64/python3.5/urllib/request.py", line 163, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.5/urllib/request.py", line 466, in open response = self._open(req, data) File "/usr/lib64/python3.5/urllib/request.py", line 484, in _open '_open', req) File "/usr/lib64/python3.5/urllib/request.py", line 444, in _call_chain result = func(*args) File "/usr/lib64/python3.5/urllib/request.py", line 1297, in https_open context=self._context, check_hostname=self._check_hostname) File "/usr/lib64/python3.5/urllib/request.py", line 1254, in do_open h.request(req.get_method(), req.selector, req.data, headers) File "/usr/lib64/python3.5/http/client.py", line 1107, in request self._send_request(method, url, body, headers) File "/usr/lib64/python3.5/http/client.py", line 1152, in _send_request self.endheaders(body) File "/usr/lib64/python3.5/http/client.py", line 1103, in endheaders self._send_output(message_body) File "/usr/lib64/python3.5/http/client.py", line 934, in _send_output self.send(msg) File "/usr/lib64/python3.5/http/client.py", line 877, in send self.connect() File "/usr/lib64/python3.5/http/client.py", line 1261, in connect server_hostname=server_hostname) File "/usr/lib64/python3.5/ssl.py", line 385, in wrap_socket _context=self) File "/usr/lib64/python3.5/ssl.py", line 760, in __init__ self.do_handshake() File "/usr/lib64/python3.5/ssl.py", line 996, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.5/ssl.py", line 646, in do_handshake match_hostname(self.getpeercert(), self.server_hostname) File "/usr/lib64/python3.5/ssl.py", line 309, in match_hostname % (hostname, dnsnames[0])) ssl.CertificateError: hostname '86.49.49.49' doesn't match 'mirror.dkm.cz' 1) For both http and https URLs, simply replace hostname with IP is not good. If target IP serves multiple virtual hosts, we may not get what we want. Also, for TLS connections, hostname is used in certificate verification. 2) mirrorselect.selectors doesn't catch ssl.CertificateError causing a crash. Besides, I found using signal.alarm to timeout a getaddrinfo() call is useless. But that's unrelated to this bug, perhaps. I'll try to give a patch to fix those，when i'm idle. Whatever, report it before I get off work today. :)
A patch would be very much appreciated.
Created attachment 510026 [details, diff] [PATCH 1/2] selectors.py: fix crash on certificate verification failure
Created attachment 510028 [details, diff] [PATCH 2/2] selectors.py: use original url for https sites
Patches uploaded. Also at https://github.com/Xiami2012/mirrorselect/commits/fix-ssl-cert Sorry for late reply. was busy and had a cold for a week.
This is a duplicate of bug 604968. It has a more lengthy discussion which comes to the same conclusion. Thanks for the patch Xiami. Would be nice to have the issue fixed.
I have tested the second patch and can confirm that it solves the problem: mirrorselect --deep completes successfully.
(In reply to Xiami from comment #3) > Created attachment 510028 [details, diff] [details, diff] > [PATCH 2/2] selectors.py: use original url for https sites We can handle it earlier, in the deeptime method like this: diff --git a/mirrorselect/selectors.py b/mirrorselect/selectors.py index cf70b21..722b266 100644 --- a/mirrorselect/selectors.py +++ b/mirrorselect/selectors.py @@ -329,6 +329,10 @@ class Deep(object): delta = 0 f = None + if url_parts.scheme == "https": + # ips don't work because the hostname must match the cert + ips = [url_parts.hostname] + for ip in ips: test_parts = url_parts._replace(netloc=ip) test_url = url_unparse(test_parts)
(In reply to Zac Medico from comment #7) > (In reply to Xiami from comment #3) > > Created attachment 510028 [details, diff] [details, diff] [details, diff] > > [PATCH 2/2] selectors.py: use original url for https sites > > We can handle it earlier, in the deeptime method like this: > > diff --git a/mirrorselect/selectors.py b/mirrorselect/selectors.py > index cf70b21..722b266 100644 > --- a/mirrorselect/selectors.py > +++ b/mirrorselect/selectors.py > @@ -329,6 +329,10 @@ class Deep(object): > delta = 0 > f = None > > + if url_parts.scheme == "https": > + # ips don't work because the hostname must match the cert > + ips = [url_parts.hostname] > + > for ip in ips: > test_parts = url_parts._replace(netloc=ip) > test_url = url_unparse(test_parts) That's great. Thanks. :)
Created attachment 511816 [details, diff] [PATCH 2/2] selectors.py: use original url for https sites Already tested by myself.
Are there any reasons to not apply the suggested patch?
The patches look good to me. It looks like bug 566778 is another good candidate to fix for the next release.
(In reply to Zac Medico from comment #11) > The patches look good to me. > > It looks like bug 566778 is another good candidate to fix for the next > release. Hmmm... Patch in Bug#566778 is fixing another rare situation and irrelevant to my patch. They fix different problems and can be merged. What I fix is at TLS-handshake stage, to make urlopen know the correct CN/SAN the certificate should have. However, Bug#566778's patch deals at http protocol. To connect a web server hosting multiple vhosts on one ip, changing domain part in URL may get data from a vhost we don't expect. _test_connection() in selectors.py shows a work around to this. In HTTPError handler, it restores url to the original url (with domain part not replaced) if it fails testing the last ip. This work around is a little disturbing in a situation where two vhost have the same directory layout (e.g. Both provides /distfiles/glibc.2.25.tar.xz). Setting request header Host solves this. It's great. If applied, the old work around can be removed. At least in nginx (which i had tested only), define two ssl sites, one with server_name a.example.com and the other b.example.com . And `curl -H 'Host: a.example.com' https://b.example.com` gets and validates cert of b.example.com, but returning web contents under a.example.com . Therefore, they are patches dealing different problems. It's good to apply both though for ssl sites adding a Host header is redundant.
just another traceback: clayman /home/emc # mirrorselect -s3 -b10 -D * Using url: https://api.gentoo.org/mirrors/distfiles.xml * Downloading a list of mirrors... Got 131 mirrors. * Downloading mirrorselect-test files from each mirror... [36 of 131]Traceback (most recent call last): File "/usr/lib/python-exec/python3.6/mirrorselect", line 61, in <module> MirrorSelect().main(sys.argv) File "/usr/lib64/python3.6/site-packages/mirrorselect/main.py", line 373, in main urls = self.select_urls(hosts, options) File "/usr/lib64/python3.6/site-packages/mirrorselect/main.py", line 322, in select_urls selector = Deep(hosts, options, self.output) File "/usr/lib64/python3.6/site-packages/mirrorselect/selectors.py", line 226, in __init__ self.deeptest() File "/usr/lib64/python3.6/site-packages/mirrorselect/selectors.py", line 253, in deeptest mytime, ignore = self.deeptime(host, maxtime) File "/usr/lib64/python3.6/site-packages/mirrorselect/selectors.py", line 338, in deeptime ip, ips[ips.index(ip):]) File "/usr/lib64/python3.6/site-packages/mirrorselect/selectors.py", line 421, in _test_connection f = url_open(test_url) File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.6/urllib/request.py", line 526, in open response = self._open(req, data) File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open '_open', req) File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain result = func(*args) File "/usr/lib64/python3.6/urllib/request.py", line 1361, in https_open context=self._context, check_hostname=self._check_hostname) File "/usr/lib64/python3.6/urllib/request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/usr/lib64/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 964, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1400, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 814, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 694, in do_handshake match_hostname(self.getpeercert(), self.server_hostname) File "/usr/lib64/python3.6/ssl.py", line 331, in match_hostname % (hostname, dnsnames[0])) ssl.CertificateError: hostname '2a02:8300:8000:3::49' doesn't match 'mirror.dkm.cz'
i just moved a server to a new location and while trying to update mirror list encountered the same problem $ mirrorselect -D -s3 -o * Using url: https://api.gentoo.org/mirrors/distfiles.xml * Downloading a list of mirrors... Got 147 mirrors. * Downloading mirrorselect-test files from each mirror... [6 of 147]Traceback (most recent call last): File "/usr/lib/python-exec/python2.7/mirrorselect", line 61, in <module> MirrorSelect().main(sys.argv) File "/usr/lib64/python2.7/site-packages/mirrorselect/main.py", line 373, in main urls = self.select_urls(hosts, options) File "/usr/lib64/python2.7/site-packages/mirrorselect/main.py", line 322, in select_urls selector = Deep(hosts, options, self.output) File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 226, in __init__ self.deeptest() File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 253, in deeptest mytime, ignore = self.deeptime(host, maxtime) File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 338, in deeptime ip, ips[ips.index(ip):]) File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 421, in _test_connection f = url_open(test_url) File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 447, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1241, in https_open context=self._context) File "/usr/lib64/python2.7/urllib2.py", line 1195, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib64/python2.7/httplib.py", line 1042, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1082, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1038, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 882, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 844, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 1263, in connect server_hostname=server_hostname) File "/usr/lib64/python2.7/ssl.py", line 369, in wrap_socket _context=self) File "/usr/lib64/python2.7/ssl.py", line 617, in __init__ self.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 854, in do_handshake match_hostname(self.getpeercert(), self.server_hostname) File "/usr/lib64/python2.7/ssl.py", line 288, in match_hostname % (hostname, ', '.join(map(repr, dnsnames)))) ssl.CertificateError: hostname '92.60.51.128' doesn't match either of '*.zeroone.sk', 'zeroone.sk' app-portage/mirrorselect-2.2.3
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=a6532e7c6b655ebba0dce53f92d9fca180b23be6 commit a6532e7c6b655ebba0dce53f92d9fca180b23be6 Author: Xiami <i@f2light.com> AuthorDate: 2017-12-14 10:30:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-02-13 08:06:57 +0000 selectors.py: handle ssl.CertificateError (bug 639156) Bug: https://bugs.gentoo.org/639156 Signed-off-by: Zac Medico <zmedico@gentoo.org> mirrorselect/selectors.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=856abee86416d4b2159f81d34cf28ef3422b92ec commit 856abee86416d4b2159f81d34cf28ef3422b92ec Author: Michel Ganguin <ganguin@romandie.com> AuthorDate: 2018-12-31 21:54:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-02-13 08:20:47 +0000 selectors.py: Give urllib hostname info (bug 604968) Give urllib hostname info such that: * it will not fail when using HTTPS because of hostname mismatch (CertificateError) * it will not fail when the server is a virtualhost * it will not fail when the server validates ssl SNI Bug: https://bugs.gentoo.org/566778 Bug: https://bugs.gentoo.org/604968 Bug: https://bugs.gentoo.org/639156 Signed-off-by: Zac Medico <zmedico@gentoo.org> mirrorselect/selectors.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cf18832afef56fa988b0291ec8877d739a8c6ba commit 8cf18832afef56fa988b0291ec8877d739a8c6ba Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-02-13 08:59:32 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-02-13 09:01:14 +0000 app-portage/mirrorselect: version bump to 2.2.4 Bug: https://bugs.gentoo.org/566778 Bug: https://bugs.gentoo.org/604968 Bug: https://bugs.gentoo.org/639156 Package-Manager: Portage-2.3.60, Repoman-2.3.12 Signed-off-by: Zac Medico <zmedico@gentoo.org> app-portage/mirrorselect/Manifest | 1 + app-portage/mirrorselect/mirrorselect-2.2.4.ebuild | 35 ++++++++++++++++++++++ 2 files changed, 36 insertions(+)
