mirrorselect -s3 -b10 -o -D takes 2 hours to complete Reproducible: Always Steps to Reproduce: emerge =app-portage/mirrorselect=1.4.1 mirrorselect -s3 -b10 -o -D
mirrorselect -s3 -b10 -o -D -d getlist(): fetching http://www.gentoo.org/main/en/mirrors.xml?passthru=1 * Downloading a list of mirrors... Got 258 mirrors. Extractor(): fetched mirrors.xml, 258 hosts before filtering * Downloading 100k files from each mirror... [1 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 1.47979307175 seconds for host ftp://gentoo.arcticnetwork.ca/pub/gentoo/distfiles/mirrorselect-test _list_add(): added host ftp://gentoo.arcticnetwork.ca/pub/gentoo/. with a time of 1.47979307175 _list_add(): host_dict is not full yet. reusing timeout of 10 sec. _list_add(): new max time is 10 seconds, and now len(host_dict)= 1 * Downloading 100k files from each mirror... [2 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 0.367179870605 seconds for host http://gentoo.arcticnetwork.ca/distfiles/mirrorselect-test _list_add(): added host http://gentoo.arcticnetwork.ca/. with a time of 0.367179870605 _list_add(): host_dict is not full yet. reusing timeout of 10 sec. _list_add(): new max time is 10 seconds, and now len(host_dict)= 2 * Downloading 100k files from each mirror... [3 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 1.70333886147 seconds for host ftp://mirrors.tera-byte.com/pub/gentoo/distfiles/mirrorselect-test _list_add(): added host ftp://mirrors.tera-byte.com/pub/gentoo. with a time of 1.70333886147 _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 1.70333886147 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [4 of 217] _deeptime(): maxtime is 1.70333886147 deeptime(): download completed. deeptime(): 0.366360902786 seconds for host http://gentoo.mirrors.tera-byte.com/distfiles/mirrorselect-test _list_add(): Adding host http://gentoo.mirrors.tera-byte.com/ with a time of 0.366360902786 _list_add(): removing ftp://mirrors.tera-byte.com/pub/gentoo _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 1.47979307175 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [5 of 217] _deeptime(): maxtime is 1.47979307175 deeptime(): download completed. deeptime(): 0.307488918304 seconds for host http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/distfiles/mirrorselect-test _list_add(): Adding host http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ with a time of 0.307488918304 _list_add(): removing ftp://gentoo.arcticnetwork.ca/pub/gentoo/ _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 0.367179870605 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [6 of 217] _deeptime(): maxtime is 0.367179870605 # And it hangs!
It sounds as if it got hung on one of the servers (probably ftp://gentoo.arcticnetwork.ca). That server seems to be responding now. Can you still reproduce it now? Anyway, mirrorselect is supposed to timeout if a server doesn't respond, so I'll have to audit the code to see if I can find a spot where it connects without a timeout.
hi, i don't think the problem is ftp://gentoo.arcticnetwork.ca/pub/gentoo/ because I access to it fine. but mirrorselect still hangs on the same point: mirrorselect -s3 -b10 -o -D -d getlist(): fetching http://www.gentoo.org/main/en/mirrors.xml?passthru=1 * Downloading a list of mirrors... Got 258 mirrors. Extractor(): fetched mirrors.xml, 258 hosts before filtering * Downloading 100k files from each mirror... [1 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 1.47625112534 seconds for host ftp://gentoo.arcticnetwork.ca/pub/gentoo/distfiles/mirrorselect-test _list_add(): added host ftp://gentoo.arcticnetwork.ca/pub/gentoo/. with a time of 1.47625112534 _list_add(): host_dict is not full yet. reusing timeout of 10 sec. _list_add(): new max time is 10 seconds, and now len(host_dict)= 1 * Downloading 100k files from each mirror... [2 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 0.366876125336 seconds for host http://gentoo.arcticnetwork.ca/distfiles/mirrorselect-test _list_add(): added host http://gentoo.arcticnetwork.ca/. with a time of 0.366876125336 _list_add(): host_dict is not full yet. reusing timeout of 10 sec. _list_add(): new max time is 10 seconds, and now len(host_dict)= 2 * Downloading 100k files from each mirror... [3 of 217] _deeptime(): maxtime is 10 deeptime(): download completed. deeptime(): 1.71619009972 seconds for host ftp://mirrors.tera-byte.com/pub/gentoo/distfiles/mirrorselect-test _list_add(): added host ftp://mirrors.tera-byte.com/pub/gentoo. with a time of 1.71619009972 _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 1.71619009972 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [4 of 217] _deeptime(): maxtime is 1.71619009972 deeptime(): download completed. deeptime(): 0.390762090683 seconds for host http://gentoo.mirrors.tera-byte.com/distfiles/mirrorselect-test _list_add(): Adding host http://gentoo.mirrors.tera-byte.com/ with a time of 0.390762090683 _list_add(): removing ftp://mirrors.tera-byte.com/pub/gentoo _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 1.47625112534 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [5 of 217] _deeptime(): maxtime is 1.47625112534 deeptime(): download completed. deeptime(): 0.244431018829 seconds for host http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/distfiles/mirrorselect-test _list_add(): Adding host http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/ with a time of 0.244431018829 _list_add(): removing ftp://gentoo.arcticnetwork.ca/pub/gentoo/ _list_add(): host_dict is full. Selecting the best timeout _list_add(): new max time is 0.390762090683 seconds, and now len(host_dict)= 3 * Downloading 100k files from each mirror... [6 of 217] _deeptime(): maxtime is 0.390762090683
Created attachment 195385 [details, diff] timeout when closing the initial "wakeup" connection If this patch is saved as /tmp/deeptime_close_timeout.patch, then it can be applied as follows: patch /usr/sbin/mirrorselect /tmp/deeptime_close_timeout.patch That should fix it, because that's the only network access I see in that area without a timeout.
that fixed it! good job!
Thanks for reporting/testing. This is fixed in mirrorselect-1.4.2.
