extract from mirrorselect -s3 -b10 -o -D -t200 -d2 * Downloading mirrorselect-test files from each mirror... [36 of 162] _deeptime(): maxtime is 0.44767165184020996 deeptime(): ip's for host gentoo.supp.name: ['81.91.84.194'] deeptime(): testing url: http://81.91.84.194/distfiles/mirrorselect-test deeptime(): timing url: http://81.91.84.194/distfiles/mirrorselect-test deeptime(): md5sum error for file: mirrorselect-test expected: bdf077b2e683c506bf9e8f2494eeb044 got.....: e323286f60abe8029c8c64d8707270e2 host....: gentoo.supp.name, 81.91.84.194 did try again * Downloading mirrorselect-test files from each mirror... [158 of 162] _deeptime(): maxtime is 0.25365185737609863 deeptime(): ip's for host gentoo.supp.name: ['81.91.84.194'] deeptime(): testing url: http://81.91.84.194/distfiles/mirrorselect-test deeptime(): timing url: http://81.91.84.194/distfiles/mirrorselect-test deeptime(): md5sum error for file: mirrorselect-test expected: bdf077b2e683c506bf9e8f2494eeb044 got.....: e323286f60abe8029c8c64d8707270e2 host....: gentoo.supp.name, 81.91.84.194 Reproducible: Always
The server you've tested seems to be using virtual hosts to serve different sites. Mirrorselect shouldn't replace the hostname with the IP as that renders the server unable to serve the correct contents. It seems though that host names are replaced on purpose to test each IP explicitly? If so, then adding a Host: header for HTTP mirrors should fix this, otherwise don't replace host names. ;)
It looks like we should be able to pass a Host header to urlopen using an instance of urllib.request.Request or urllib2.Request like this: f = url_open(Request(test_url, headers={'Host': url_parts.hostname})
Created attachment 514746 [details, diff] selectors.py: send Host header
I got this same issue with mirrorselect-2.2.3: # mirrorselect -s 4 GENTOO_MIRRORS="http://104.19.137.75/gentoo/ http://104.19.138.75/gentoo/ http://104.19.139.75/gentoo/ http://104.19.135.75/gentoo/" All those IPs are on Cloudflare and none works without a real hostname as Cloudflare doesn't allow direct IP access.
Created attachment 559210 [details, diff] set host in request the suggested patch still returns a CertificateError, in my testing the request host should be set to the real hostname and then it should work
(In reply to Michel Ganguin from comment #5) > Created attachment 559210 [details, diff] [details, diff] > set host in request > > the suggested patch still returns a CertificateError, in my testing the > request host should be set to the real hostname and then it should work This looks pretty close to the patch that I posted in comment #3. I see you used url_parts.netloc instead of url_parts.hostname. I think url_parts.hostname is more correct since netloc may also include a :port suffix. Anyway, I'd like to merge a patch for this very soon and make a release.
(In reply to Zac Medico from comment #6) > (In reply to Michel Ganguin from comment #5) > > Created attachment 559210 [details, diff] [details, diff] [details, diff] > > set host in request > > > > the suggested patch still returns a CertificateError, in my testing the > > request host should be set to the real hostname and then it should work > > This looks pretty close to the patch that I posted in comment #3. I see you > used url_parts.netloc instead of url_parts.hostname. I think > url_parts.hostname is more correct since netloc may also include a :port > suffix. > > Anyway, I'd like to merge a patch for this very soon and make a release. yeah url_parts.hostname should be better. Regarding patch in comment #3, it's very similar, but did not fix the issue for me: the CertficateError was still raised and the hostname was not in SNI in the SSL Hello message. But, using Request.host instead of directly changing the "Host:" header worked for me.
*** Bug 675198 has been marked as a duplicate of this bug. ***
I can confirm this bug. What's even more annoying is that when executing the command without --deep and without --interactive no error is raised. Only later, after switching with chroot to the new environment and trying to update the @world set, a request forbidden (HTTP 403) is returned from the server. $ mirrorselect --http --region Europe --servers 3 -o -d2 main(); config_path = /etc/portage/make.conf main(); rsync = False get_filesystem_mirrors(): config_path = /etc/portage/make.conf get_filesystem_mirrors(): mirrorlist = ['https://104.19.138.75/gentoo/', 'https://104.19.136.75/gentoo/', 'https://104.19.135.75/gentoo/', 'https://104.19.139.75/gentoo/', 'https://104.19.137.75/gentoo/', 'http://104.19.137.75/gentoo/', 'http://104.19.135.75/gentoo/', 'http://104.19.138.75/gentoo/', 'http://104.19.139.75/gentoo/', 'http://104.19.136.75/gentoo/', 'http://mirror.leaseweb.com/gentoo/', 'https://mirror.netcologne.de/gentoo/', 'http://mirror.netcologne.de/gentoo/', 'http://ftp.snt.utwente.nl/pub/os/linux/gentoo', 'http://gentoo.mirrors.ovh.net/gentoo-distfiles/', 'https://ftp.halifax.rwth-aachen.de/gentoo/', 'http://ftp.halifax.rwth-aachen.de/gentoo/', 'https://ftp.fau.de/gentoo', 'https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/', 'http://ftp.fau.de/gentoo'] get_filesystem_mirrors(): ignoring non-accessible mirror = https://104.19.138.75/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://104.19.136.75/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://104.19.135.75/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://104.19.139.75/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://104.19.137.75/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://mirror.netcologne.de/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://ftp.halifax.rwth-aachen.de/gentoo/ get_filesystem_mirrors(): ignoring non-accessible mirror = https://ftp.fau.de/gentoo get_filesystem_mirrors(): ignoring non-accessible mirror = https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ get_filesystem_mirrors(): fsmirrors = [] using url: https://api.gentoo.org/mirrors/distfiles.xml * Using url: https://api.gentoo.org/mirrors/distfiles.xml * Limiting test to "region=Europe" hosts. * Limiting test to http hosts. getlist(): fetching https://api.gentoo.org/mirrors/distfiles.xml * Downloading a list of mirrors... Enabled ssl certificate verification: True, for: https://api.gentoo.org/mirrors/distfiles.xml Connector.connect_url(); headers = {'Accept-Charset': 'utf-8', 'User-Agent': 'Mirrorselect-2.2.3'} Connector.connect_url(); connecting to opener Connector.connect_url() HEADERS = {'Server': 'nginx', 'Date': 'Wed, 23 Jan 2019 17:28:07 GMT', 'Content-Type': 'text/xml', 'Content-Length': '22566', 'Last-Modified': 'Mon, 10 Dec 2018 07:05:17 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=20', 'ETag': '"5c0e102d-5826"', 'X-Clacks-Overhead': 'GNU Terry Pratchett, Noirin Trouble Pluinceid', 'Accept-Ranges': 'bytes'} Connector.connect_url() Status_code = 200 New content downloaded for: https://api.gentoo.org/mirrors/distfiles.xml Got 147 mirrors. Extractor(): fetched mirrors, 41 hosts after filtering * Using netselect to choose the top 3 mirrors... netselect(): running "netselect -s3 http://mirrors.evowise.com/gentoo/ https://mirrors.evowise.com/gentoo/ http://ftp.fi.muni.cz/pub/linux/gentoo/ http://gentoo.mirror.web4u.cz/ http://mirror.dkm.cz/gentoo/ https://mirror.dkm.cz/gentoo/ http://gentoo.mirrors.ovh.net/gentoo-distfiles/ http://ftp.free.fr/mirrors/ftp.gentoo.org/ http://mirrors.soeasyto.com/distfiles.gentoo.org/ http://gentoo.modulix.net/gentoo/ https://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ https://ftp.fau.de/gentoo http://ftp.fau.de/gentoo https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ https://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ http://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ https://mirror.netcologne.de/gentoo/ http://mirror.netcologne.de/gentoo/ https://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.ntua.gr/pub/linux/gentoo/ http://ftp.snt.utwente.nl/pub/os/linux/gentoo https://mirror.leaseweb.com/gentoo/ http://mirror.leaseweb.com/gentoo/ http://ftp.vectranet.pl/gentoo/ https://ftp.rnl.tecnico.ulisboa.pt/pub/gentoo/gentoo-distfiles/ http://ftp.rnl.tecnico.ulisboa.pt/pub/gentoo/gentoo-distfiles/ http://ftp.dei.uc.pt/pub/linux/gentoo/ http://mirrors.xservers.ro/gentoo/ http://ftp.romnet.org/gentoo/ http://mirror.mdfnet.se/gentoo https://gentoo.wheel.sk/ http://gentoo.wheel.sk/ http://tux.rainside.sk/gentoo/ http://ftp.linux.org.tr/gentoo/ https://mirror.bytemark.co.uk/gentoo/ http://mirror.bytemark.co.uk/gentoo/ https://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/" Done. netselect(): returning [b'https://104.19.135.75/gentoo/', b'https://104.19.138.75/gentoo/', b'https://104.19.139.75/gentoo/'] and {b'0': b'https://104.19.139.75/gentoo/'} GENTOO_MIRRORS="https://104.19.135.75/gentoo/ https://104.19.138.75/gentoo/ https://104.19.139.75/gentoo/" Issuing `emerge --update --newuse --deep @world` in the new environment with the returned mirrors leads to the following error: >>> Emerging (1 of 52) dev-libs/ustr-1.0.4-r8::gentoo * Fetching files in the background. * To view fetch progress, run in another terminal: * tail -f /var/log/emerge-fetch.log !!! SELinux module not found. Please verify that it was installed. !!! SELinux module not found. Please verify that it was installed. >>> Downloading 'http://104.19.137.75/gentoo/distfiles/ustr-1.0.4.tar.bz2' --2019-01-23 17:24:21-- http://104.19.137.75/gentoo/distfiles/ustr-1.0.4.tar.bz2 Connecting to 104.19.137.75:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-01-23 17:24:21 ERROR 403: Forbidden. >>> Downloading 'http://104.19.135.75/gentoo/distfiles/ustr-1.0.4.tar.bz2' --2019-01-23 17:24:21-- http://104.19.135.75/gentoo/distfiles/ustr-1.0.4.tar.bz2 Connecting to 104.19.135.75:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-01-23 17:24:21 ERROR 403: Forbidden. >>> Downloading 'http://104.19.139.75/gentoo/distfiles/ustr-1.0.4.tar.bz2' --2019-01-23 17:24:21-- http://104.19.139.75/gentoo/distfiles/ustr-1.0.4.tar.bz2 Connecting to 104.19.139.75:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-01-23 17:24:21 ERROR 403: Forbidden. >>> Downloading 'ftp://ftp.and.org/pub/james/ustr/1.0.4/ustr-1.0.4.tar.bz2' --2019-01-23 17:24:21-- ftp://ftp.and.org/pub/james/ustr/1.0.4/ustr-1.0.4.tar.bz2 => '/usr/portage/distfiles/ustr-1.0.4.tar.bz2' Resolving ftp.and.org... 173.203.194.147 Connecting to ftp.and.org|173.203.194.147|:21... failed: No route to host. !!! Couldn't download 'ustr-1.0.4.tar.bz2'. Aborting. * Fetch failed for 'dev-libs/ustr-1.0.4-r8', Log file: * '/var/tmp/portage/dev-libs/ustr-1.0.4-r8/temp/build.log' >>> Failed to emerge dev-libs/ustr-1.0.4-r8, Log file: >>> '/var/tmp/portage/dev-libs/ustr-1.0.4-r8/temp/build.log'
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=856abee86416d4b2159f81d34cf28ef3422b92ec commit 856abee86416d4b2159f81d34cf28ef3422b92ec Author: Michel Ganguin <ganguin@romandie.com> AuthorDate: 2018-12-31 21:54:29 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-02-13 08:20:47 +0000 selectors.py: Give urllib hostname info (bug 604968) Give urllib hostname info such that: * it will not fail when using HTTPS because of hostname mismatch (CertificateError) * it will not fail when the server is a virtualhost * it will not fail when the server validates ssl SNI Bug: https://bugs.gentoo.org/566778 Bug: https://bugs.gentoo.org/604968 Bug: https://bugs.gentoo.org/639156 Signed-off-by: Zac Medico <zmedico@gentoo.org> mirrorselect/selectors.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cf18832afef56fa988b0291ec8877d739a8c6ba commit 8cf18832afef56fa988b0291ec8877d739a8c6ba Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2019-02-13 08:59:32 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2019-02-13 09:01:14 +0000 app-portage/mirrorselect: version bump to 2.2.4 Bug: https://bugs.gentoo.org/566778 Bug: https://bugs.gentoo.org/604968 Bug: https://bugs.gentoo.org/639156 Package-Manager: Portage-2.3.60, Repoman-2.3.12 Signed-off-by: Zac Medico <zmedico@gentoo.org> app-portage/mirrorselect/Manifest | 1 + app-portage/mirrorselect/mirrorselect-2.2.4.ebuild | 35 ++++++++++++++++++++++ 2 files changed, 36 insertions(+)
