Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 604968 - app-portage/mirrorselect gives SSL certificate error for mirror.dkm.cz
Summary: app-portage/mirrorselect gives SSL certificate error for mirror.dkm.cz
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: Normal normal with 4 votes (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords: InVCS
Depends on: 686446
Blocks:
  Show dependency tree
 
Reported: 2017-01-07 18:36 UTC by Joe Stroller
Modified: 2019-07-18 19:17 UTC (History)
11 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Patch to handle ssl.CertificateError (mirrorselect-9999-ssl-certificateerror.patch,728 bytes, patch)
2017-05-18 18:25 UTC, godlike64
Details | Diff
Simpler patch to handle ssl.CertificateError (mirrorselect.patch,738 bytes, patch)
2017-09-02 11:13 UTC, Daniel Harding
Details | Diff
Give urllib hostname info (fix_url_host.patch,1.22 KB, patch)
2018-12-31 21:54 UTC, Michel Ganguin
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Stroller 2017-01-07 18:36:24 UTC
$ mirrorselect -R Europe -o -D -s 8
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts. 
* Downloading a list of mirrors...
 Got 155 mirrors.
* Downloading mirrorselect-test files from each mirror... [57 of 87]Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/mirrorselect", line 61, in <module>
    MirrorSelect().main(sys.argv)
  File "/usr/lib64/python2.7/site-packages/mirrorselect/main.py", line 375, in main
    urls = self.select_urls(hosts, options)
  File "/usr/lib64/python2.7/site-packages/mirrorselect/main.py", line 324, in select_urls
    selector = Deep(hosts, options, self.output)
  File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 226, in __init__
    self.deeptest()
  File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 253, in deeptest
    mytime, ignore = self.deeptime(host, maxtime)
  File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 336, in deeptime
    ip, ips[ips.index(ip):])
  File "/usr/lib64/python2.7/site-packages/mirrorselect/selectors.py", line 419, in _test_connection
    f = url_open(test_url)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib64/python2.7/urllib2.py", line 1195, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1278, in connect
    server_hostname=server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/lib64/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 838, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 276, in match_hostname
    % (hostname, dnsnames[0]))
ssl.CertificateError: hostname '86.49.49.49' doesn't match 'mirror.dkm.cz'
$

Mirrorselect and certs are latest stable, I think:

$ eix -I mirrorselect
[I] app-portage/mirrorselect
     Available versions:  2.2.0.1 2.2.2 2.2.2-r2 **9999 {PYTHON_TARGETS="python2_7 python3_4"}
     Installed versions:  2.2.2-r2(02:27:40 25/04/16)(PYTHON_TARGETS="python2_7 python3_4 -python3_3")
     Homepage:            https://wiki.gentoo.org/wiki/Project:Mirrorselect
     Description:         Tool to help select distfiles mirrors for Gentoo

951 ~ $ eix -I cert
[I] app-misc/ca-certificates
     Available versions:  20151214.3.21 ~20160104.3.27.1-r2 20161102.3.27.2-r2 ~20161130.3.28 {(+)cacert insecure_certs}
     Installed versions:  20161102.3.27.2-r2(19:04:23 06/01/17)(-cacert -insecure_certs)
     Homepage:            http://packages.debian.org/sid/ca-certificates
     Description:         Common CA Certificates PEM files

[I] dev-python/certifi
     Available versions:  2015.11.20 ~2016.8.8 ~2016.8.31 2016.9.26 {PYTHON_TARGETS="pypy pypy3 python2_7 python3_4 python3_5"}
     Installed versions:  2016.9.26(19:24:15 06/01/17)(PYTHON_TARGETS="python2_7 python3_4 -pypy -pypy3 -python3_5")
     Homepage:            http://certifi.io/ https://pypi.python.org/pypi/certifi
     Description:         Python package for providing Mozilla's CA Bundle

Found 2 matches
$
Comment 1 Matthew Thode ( prometheanfire ) archtester Gentoo Infrastructure gentoo-dev Security 2017-01-10 03:02:33 UTC
A straight curl works here (didn't need to allow odd certs)

I have the same version of cacerts at least

I did get the same error, though on a diferent server/IP...

ssl.CertificateError: hostname 'IPV6_DNS_IP_GOES_HERE' doesn't match either of 'my.home.ssl.cert.domain.list'


Not sure why it was trying to test that, I imagine my dns server returned for the IPV6 IP of my dns server though...

For me it happened on gentoo.iteam.net.ua (checked via debug output).  Maybe that mirror needs to be purged?  Not your error though.
Comment 2 godlike64 2017-02-14 19:14:48 UTC
Happens to me too with a similar traceback, using mirrorselect-9999. Funny thing: on my local network (Argentina) it happens, but on a dedicated server in Europe, that mirror never gets into the list and does not trigger the issue.

There appears to be nothing wrong with the certificate from what I can see:

# openssl s_client -connect mirror.dkm.cz:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mirror.dkm.cz
verify return:1
---
Certificate chain
 0 s:/CN=mirror.dkm.cz
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE/jCCA+agAwIBAgISA5OPSW9MMmKXmcTgVWAQaMgXMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNjEyMDYxMzIyMDBaFw0x
NzAzMDYxMzIyMDBaMBgxFjAUBgNVBAMTDW1pcnJvci5ka20uY3owggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCjj7RhC6sibs1M+berxcwE5kp+oUfzuuuu
hjbpMxgJxn9wDYXykA4shrOGJsDwHcU0jeogRDJ7M5aM4FV/Hl1Xc3/kKLDby0kX
fqiOUgi9K1eNpaInApNN1YitnsH931XvpfO0C6iXNCnFnhpiWCKUp61yU2ssiXug
iA9fMks24i557zqcVP5pL0VwhMG89BCNjEjOP/0KVXP15htmckZUKKB5tAPie/sw
zWgKieGxqyFXu9NpG+Egyye1CtA5IS+h8BG8XJHuxiXxUeClQjc8mZOi4I3n7gIZ
eORtosvLTGdhcsAQfZNjB+RgdEXFqvkEsqt3QEiOKVkjerISfqp1AgMBAAGjggIO
MIICCjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFHBqZ+mn+8w5FXDRjmSda1NMbwdZ
MB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMHAGCCsGAQUFBwEBBGQw
YjAvBggrBgEFBQcwAYYjaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9y
Zy8wLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5v
cmcvMBgGA1UdEQQRMA+CDW1pcnJvci5ka20uY3owgf4GA1UdIASB9jCB8zAIBgZn
gQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRpZmlj
YXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGllcyBh
bmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBvbGlj
eSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5LzAN
BgkqhkiG9w0BAQsFAAOCAQEAiRDuoWUNClo2y5IIX5fzWmsUu3EZgGmYDtVlXU/P
w3U9KpxgkdQcJt5F6ylDV3rQ6O28BeqdTFAcYRfLLsFl5oYHn4xsdNO132d+skfV
mg2ZFAZ8QVUFYmTgknUjVkIK3cpudHWwcBOqIn7LzFiQAGDvNSdw7LuHj48EON1/
fc4vBT9UtR0dcqqrQGKWiul1dWPR61OdQfwJYsLix47UiVBmfLt7mlNQ4bJEQV19
4OBJi/fcy0UqtnlIf8BK+I6+kCVeoTA1+0HQz1Ffimp1T4JpjcgAwpi92eRFAwV7
4qb1UV9G39h9FLeDljhmSjfy9qWKzSydCs+CoXcGoa7B8w==
-----END CERTIFICATE-----
subject=/CN=mirror.dkm.cz
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2971 bytes and written 434 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: D6A3A37FFFB3B03703B56227E749E2F191EC049E1BCE7A8B898B18680469782F
    Session-ID-ctx: 
    Master-Key: 6D2E56669B853B5D575654FF2BE42E457A4929D2D8E33537C2A458105A7C05A0517BEE2D51F93C25A34F16EAFE10C00A
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1487099164
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

# ping -c1 mirror.dkm.cz
PING mirror.dkm.cz (86.49.49.49) 56(84) bytes of data.
64 bytes from mirror.dkm.cz (86.49.49.49): icmp_seq=1 ttl=50 time=297 ms
Comment 3 David Zaslavsky 2017-05-18 02:03:16 UTC
I'm getting the same error, though with a different IP address, when installing on a VPS. This is with mirrorselect 2.2.2-r2 (stable!)

# eix mirrorselect
[I] app-portage/mirrorselect
     Available versions:  2.2.0.1 2.2.2 2.2.2-r2{xpak} **9999 {PYTHON_TARGETS="python2_7 python3_4"}
     Installed versions:  2.2.2-r2{xpak}(08:21:32 PM 04/18/2017)(PYTHON_TARGETS="python2_7 python3_4")
     Homepage:            https://wiki.gentoo.org/wiki/Project:Mirrorselect
     Description:         Tool to help select distfiles mirrors for Gentoo

# mirrorselect -o -s3 -b10 -D
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Downloading a list of mirrors...
 Got 155 mirrors.
* Downloading mirrorselect-test files from each mirror... [19 of 155]Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.4/mirrorselect", line 61, in <module>
    MirrorSelect().main(sys.argv)
  File "/usr/lib64/python3.4/site-packages/mirrorselect/main.py", line 375, in main
    urls = self.select_urls(hosts, options)
  File "/usr/lib64/python3.4/site-packages/mirrorselect/main.py", line 324, in select_urls
    selector = Deep(hosts, options, self.output)
  File "/usr/lib64/python3.4/site-packages/mirrorselect/selectors.py", line 226, in __init__
    self.deeptest()
  File "/usr/lib64/python3.4/site-packages/mirrorselect/selectors.py", line 253, in deeptest
    mytime, ignore = self.deeptime(host, maxtime)
  File "/usr/lib64/python3.4/site-packages/mirrorselect/selectors.py", line 336, in deeptime
    ip, ips[ips.index(ip):])
  File "/usr/lib64/python3.4/site-packages/mirrorselect/selectors.py", line 419, in _test_connection
    f = url_open(test_url)
  File "/usr/lib64/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.4/urllib/request.py", line 464, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.4/urllib/request.py", line 482, in _open
    '_open', req)
  File "/usr/lib64/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.4/urllib/request.py", line 1226, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python3.4/urllib/request.py", line 1183, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib64/python3.4/http/client.py", line 1137, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python3.4/http/client.py", line 1182, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python3.4/http/client.py", line 1133, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python3.4/http/client.py", line 963, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.4/http/client.py", line 898, in send
    self.connect()
  File "/usr/lib64/python3.4/http/client.py", line 1287, in connect
    server_hostname=server_hostname)
  File "/usr/lib64/python3.4/ssl.py", line 362, in wrap_socket
    _context=self)
  File "/usr/lib64/python3.4/ssl.py", line 580, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.4/ssl.py", line 815, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib64/python3.4/ssl.py", line 286, in match_hostname
    % (hostname, dnsnames[0]))
ssl.CertificateError: hostname '2a02:8300:8000:3::49' doesn't match 'mirror.dkm.cz'

From what I can tell, this seems not to be a certificate error but a DNS propagation error. If I manually resolve mirror.dkm.cz using IPv6, I get the address shown (some output omitted below):

# dig AAAA mirror.dkm.cz

;; QUESTION SECTION:
;mirror.dkm.cz.                 IN      AAAA

;; ANSWER SECTION:
mirror.dkm.cz.          3600    IN      AAAA    2a02:8300:8000:3::49

;; AUTHORITY SECTION:
dkm.cz.                 3600    IN      NS      ns2.dkm.cz.
dkm.cz.                 3600    IN      NS      ns1.dkm.cz.

;; Query time: 330 msec
;; SERVER: 173.255.243.5#53(173.255.243.5)
;; WHEN: Thu May 18 01:48:49 UTC 2017
;; MSG SIZE  rcvd: 194

but if I look up the IP address I get no answer.

# dig -x 2a02:8300:8000:3::49

;; QUESTION SECTION:
;9.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.0.0.0.0.8.0.0.3.8.2.0.a.2.ip6.arpa. IN PTR

;; AUTHORITY SECTION:
0.3.8.2.0.a.2.ip6.arpa. 7200    IN      SOA     ns1.upc.biz. hostmaster.chello.at. 2015070102 86400 7200 2592000 7200

;; Query time: 702 msec
;; SERVER: 173.255.243.5#53(173.255.243.5)
;; WHEN: Thu May 18 01:49:12 UTC 2017
;; MSG SIZE  rcvd: 168

However, if I use the nameserver designated by the original DNS record I do get the correct result.

# dig @ns1.dkm.cz -x 2a02:8300:8000:3::49

;; QUESTION SECTION:
;9.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.0.0.0.0.8.0.0.3.8.2.0.a.2.ip6.arpa. IN PTR

;; ANSWER SECTION:
9.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.0.0.0.0.8.0.0.3.8.2.0.a.2.ip6.arpa. 86400 IN PTR mirror.dkm.cz.

;; AUTHORITY SECTION:
0.0.3.8.2.0.a.2.ip6.arpa. 86400 IN      NS      ns1.dkm.cz.
0.0.3.8.2.0.a.2.ip6.arpa. 86400 IN      NS      ns2.dkm.cz.

;; Query time: 162 msec
;; SERVER: 2a02:8301:0:10::2#53(2a02:8301:0:10::2)
;; WHEN: Thu May 18 01:53:57 UTC 2017
;; MSG SIZE  rcvd: 252

Ordinarily I would blame this on the DNS server being used by my VPS, but then again why is this happening to other people in other locations...? Anyway I hope the information will be helpful to someone.

I think it would be nice if mirrorselect had the ability to drop mirrors from the list and keep going when there is a failure in name resolution or something like that, but that's a bigger issue.
Comment 4 godlike64 2017-05-18 13:37:39 UTC
(In reply to David Zaslavsky from comment #3)
> I think it would be nice if mirrorselect had the ability to drop mirrors
> from the list and keep going when there is a failure in name resolution or
> something like that, but that's a bigger issue.

Actually, that would be the best way to fix this. The whole traceback looks like there's a try/except block that was not taken into account on the mirrorselect code. Such one would prevent mirrorselect from failing and just allow for silently (or verbosively) drop a mirror from the list and continue onwards.
Comment 5 godlike64 2017-05-18 18:25:51 UTC
Created attachment 473178 [details, diff]
Patch to handle ssl.CertificateError

I wrote this in a hurry when I saw the code and realized it was already handling some exceptions. This runs and does not fail, although it appears the offending mirror is no longer on the list.
Comment 6 Jesse Adelman 2017-07-24 16:20:43 UTC
+1 here.
Comment 7 Andriy Mykhaylyk 2017-08-02 15:44:50 UTC
I have same issue with the same mirror.
Comment 8 nic 2017-08-03 18:14:15 UTC
(In reply to godlike64 from comment #5)
> Created attachment 473178 [details, diff] [details, diff]
> Patch to handle ssl.CertificateError
> 
> I wrote this in a hurry when I saw the code and realized it was already
> handling some exceptions. This runs and does not fail, although it appears
> the offending mirror is no longer on the list.

Thanks. This patch handled the ssl errors with python 3.4 amd64.
Comment 9 Jesse Adelman 2017-08-03 18:49:20 UTC
Can we at least set this to CONFIRMED? :) https://paste.pound-python.org/show/PwmZiYJMXyYQejbf1nKC/

Pastebin output a result of:

# mirrorselect -DHs5

Cheers.
Comment 10 Daniel Harding 2017-09-02 11:13:17 UTC
Created attachment 491764 [details, diff]
Simpler patch to handle ssl.CertificateError

Hit this also for the same server (mirror.dkm.cz) - here is the patch I used to fix the issue for me.
Comment 11 Tamas Jantvik 2017-09-15 10:21:21 UTC
The patch seems to work for me. No idea about the security implications, though. :)
Comment 12 Jaroslav Rakhmatoullin 2017-10-31 11:08:43 UTC
The issue is not with exception handling. A certificate is issued for a domain name. Comparing that to an IP is a mistake.

I think that the url_open() code is not being called correctly (because the library changed?) or there is a mistake in the reverse DNS record for the host. 

Mirrorselect should handle the latter but not the former - or actually it should, but not by catching the exception but by calling url_open with correct parameters.

The issue lies in the "handle python 2 and 3 code" at the top `if sys.version_info[0] >= 3:` - it assumes these libraries work identically.
Perhaps they are identical APIs, i'm not sure..


this issue does not seem to affect mirrorselect when called with --ftp
Comment 13 Andrew 2017-12-26 19:45:15 UTC
Bug 639156 about the same problem has a working patch provided to fix the problem with HTTPS mirrors.
Comment 14 Armitage 2018-05-22 12:00:49 UTC
(In reply to Daniel Harding from comment #10)
> Created attachment 491764 [details, diff] [details, diff]
> Simpler patch to handle ssl.CertificateError
> 
> Hit this also for the same server (mirror.dkm.cz) - here is the patch I used
> to fix the issue for me.

I can confirm that the patch proposed by Daniel works.
Comment 15 Mario Bachmann 2018-08-02 18:57:20 UTC
Why don't You fix it? It is one line of code. 
The problem is obvious and reproducalbe.
Comment 16 Daniel Harding 2018-08-02 19:15:48 UTC
(In reply to Mario Bachmann from comment #15)
> Why don't You fix it? It is one line of code. 
> The problem is obvious and reproducalbe.

Hello Mario,

I'm not a developer for any Gentoo project, but as a developer who is occasionally active in the open source world, the kind of attitude expressed in your comment is in no way motivating for anyone to fix anything.  I know it can be frustrating to have to deal with software bugs, especially ones that seem like they should have an easy fix.  However, having a demanding or demeaning attitude certainly won't cause anyone to fix the problem any faster, and may actually cause it to take longer.  Repeated comments of this type can actually accelerate burnout and cause developers (who after all are volunteers) to disengage from the project.  Because this bug is marked as CONFIRMED with at least two potential approaches for addressing it, the best way to signal interest is probably to just add yourself to the CC list.  However, if you would still desire to add a comment, phrasing it more along the lines of “Hey, just wanted to mention that I am running into this as well.  Wanted to see if there has been any progress toward getting a fix applied.” would be much more well received.
Comment 17 Brian Dolbec (RETIRED) gentoo-dev 2018-08-03 02:51:36 UTC
Sorry, I missed seeing this one earlier.

This is not the fault of mirrorselect code, but one of the mirrors data getting out of sync with reality.

Re-assigning this one to the infra team who have access to the mirrors data.
Comment 18 Brian Dolbec (RETIRED) gentoo-dev 2018-08-03 02:55:15 UTC
man, I should not be trying to work on these things when I am so tired...

This is a code issue...
Comment 19 Michel Ganguin 2018-12-31 19:58:21 UTC
the try/except patch kind of works, but...

...there is another problem, when this patch is applied, https:// mirrors are all skipped. This is because mirrorselect (netselect as well) tests the server using the resolved IP address in the URL instead of the domain name. And therefore will always raise the CetificateError. Even worse, when the mirror is a virtualhost  (http or https), it may not hit the right server.

So the *real* fix should set the host header correctly when performing the HTTP request. I'll try to come up with a patch.
Comment 20 Michel Ganguin 2018-12-31 21:54:29 UTC
Created attachment 559198 [details, diff]
Give urllib hostname info

Give urllib hostname info such that:
* it will not fail when using HTTPS because of hostname mismatch (CertificateError)
* it will not fail when the server is a virtualhost
* it will not fail when the server validates ssl SNI
Comment 21 Michel Ganguin 2018-12-31 21:59:10 UTC
I've proposed a new patch to fix my mentioned HTTPS issue.

I think both patches should be applied:
* The one from Daniel for not crashing on *real* Certificate errors
* Mine for making HTTPS actually works

Hope this can be applied soon

Thanks
Comment 22 Oleg Gordienko 2019-02-12 21:53:38 UTC
@(In reply to Michel Ganguin from comment #21)
> I've proposed a new patch to fix my mentioned HTTPS issue.
> 
> I think both patches should be applied:
> * The one from Daniel for not crashing on *real* Certificate errors
> * Mine for making HTTPS actually works
> 
> Hope this can be applied soon
> 
> Thanks

Thank you, tested on 2.2.3
Comment 23 Larry the Git Cow gentoo-dev 2019-02-13 05:51:30 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=92d682ca0f8d173149df8511a06b0457ffbffa8d

commit 92d682ca0f8d173149df8511a06b0457ffbffa8d
Author:     Daniel Harding <dharding@living180.net>
AuthorDate: 2017-09-02 11:13:00 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-02-13 05:50:12 +0000

    selectors.py: handle ssl.CertificateError (bug 604968)
    
    Bug: https://bugs.gentoo.org/604968
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 mirrorselect/selectors.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 24 Larry the Git Cow gentoo-dev 2019-02-13 08:22:05 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=856abee86416d4b2159f81d34cf28ef3422b92ec

commit 856abee86416d4b2159f81d34cf28ef3422b92ec
Author:     Michel Ganguin <ganguin@romandie.com>
AuthorDate: 2018-12-31 21:54:29 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-02-13 08:20:47 +0000

    selectors.py: Give urllib hostname info (bug 604968)
    
    Give urllib hostname info such that:
    * it will not fail when using HTTPS because of hostname mismatch (CertificateError)
    * it will not fail when the server is a virtualhost
    * it will not fail when the server validates ssl SNI
    
    Bug: https://bugs.gentoo.org/566778
    Bug: https://bugs.gentoo.org/604968
    Bug: https://bugs.gentoo.org/639156
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 mirrorselect/selectors.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
Comment 25 Larry the Git Cow gentoo-dev 2019-02-13 09:01:25 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cf18832afef56fa988b0291ec8877d739a8c6ba

commit 8cf18832afef56fa988b0291ec8877d739a8c6ba
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-02-13 08:59:32 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-02-13 09:01:14 +0000

    app-portage/mirrorselect: version bump to 2.2.4
    
    Bug: https://bugs.gentoo.org/566778
    Bug: https://bugs.gentoo.org/604968
    Bug: https://bugs.gentoo.org/639156
    Package-Manager: Portage-2.3.60, Repoman-2.3.12
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 app-portage/mirrorselect/Manifest                  |  1 +
 app-portage/mirrorselect/mirrorselect-2.2.4.ebuild | 35 ++++++++++++++++++++++
 2 files changed, 36 insertions(+)