Example problematic session: $ rm oxygen-icons5-5.80.0.tar.xz $ LANG=C wget https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz --2021-06-03 08:40:22-- https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz Resolving distfiles.gentoo.org... 195.181.175.49, 195.181.175.52, 195.181.175.45, ... Connecting to distfiles.gentoo.org|195.181.175.49|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 236044748 (225M) [application/x-xz] Saving to: 'oxygen-icons5-5.80.0.tar.xz' oxygen-icons5-5.80.0.tar.xz 0%[ ] 1.07M 16.6KB/s eta 2h 5m ^C $ rm oxygen-icons5-5.80.0.tar.xz $ LANG=C wget https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz --2021-06-03 08:41:01-- https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz Resolving distfiles.gentoo.org... 195.181.175.52, 195.181.175.46, 195.181.175.54, ... Connecting to distfiles.gentoo.org|195.181.175.52|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 236044748 (225M) [application/x-xz] Saving to: 'oxygen-icons5-5.80.0.tar.xz' oxygen-icons5-5.80.0.tar.xz 0%[ ] 1.30M 21.0KB/s eta 1h 47m ^C $ rm oxygen-icons5-5.80.0.tar.xz $ LANG=C wget https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz --2021-06-03 08:41:43-- https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz Resolving distfiles.gentoo.org... 195.181.175.51, 195.181.175.48, 195.181.175.54, ... Connecting to distfiles.gentoo.org|195.181.175.51|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 236044748 (225M) [application/x-xz] Saving to: 'oxygen-icons5-5.80.0.tar.xz' oxygen-icons5-5.80.0.tar.xz 4%[====> ] 10.37M 5.38KB/s in 25s 2021-06-03 08:42:08 (430 KB/s) - Connection closed at byte 10878349. Retrying. --2021-06-03 08:42:09-- (try: 2) https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz Connecting to distfiles.gentoo.org|195.181.175.51|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 236044748 (225M), 225166399 (215M) remaining [application/x-xz] Saving to: 'oxygen-icons5-5.80.0.tar.xz' oxygen-icons5-5.80.0.tar.xz 5%[+++++> ] 12.48M 15.8KB/s in 16s 2021-06-03 08:42:26 (139 KB/s) - Connection closed at byte 13089498. Retrying. --2021-06-03 08:42:28-- (try: 3) https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz Connecting to distfiles.gentoo.org|195.181.175.51|:443... $ traceroute -n 195.181.175.49 traceroute to 195.181.175.49 (195.181.175.49), 30 hops max, 60 byte packets 1 192.168.1.254 3.338 ms 3.408 ms 4.253 ms 2 * * * 3 * * * 4 31.55.187.188 19.498 ms 23.358 ms 24.091 ms 5 195.99.127.178 24.087 ms 213.121.192.96 24.995 ms 213.121.192.112 25.574 ms 6 62.172.103.172 27.107 ms 109.159.252.184 25.106 ms 194.72.16.142 26.912 ms 7 185.229.189.58 28.018 ms 12.842 ms 10.435 ms 8 185.229.188.4 22.661 ms 185.229.188.12 23.500 ms 185.229.188.4 23.794 ms 9 185.229.188.155 24.434 ms 24.185 ms 185.229.188.157 24.170 ms 10 195.181.175.49 23.483 ms 23.378 ms 23.381 ms
For contrast here is UK->US example (should be far away fro me): $ LANG=C wget https://gentoo.osuosl.org//distfiles/45/oxygen-icons5-5.80.0.tar.xz --2021-06-03 08:46:32-- https://gentoo.osuosl.org//distfiles/45/oxygen-icons5-5.80.0.tar.xz Resolving gentoo.osuosl.org... 140.211.166.134, 64.50.233.100, 64.50.236.52, ... Connecting to gentoo.osuosl.org|140.211.166.134|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 236044748 (225M) [application/x-xz] Saving to: 'oxygen-icons5-5.80.0.tar.xz.1' oxygen-icons5-5.80.0.tar.xz.1 67%[===============================================================================> ] 152.91M 2.06MB/s eta 45s ^C $ traceroute -n 140.211.166.134 traceroute to 140.211.166.134 (140.211.166.134), 30 hops max, 60 byte packets 1 192.168.1.254 7.647 ms 7.703 ms 7.802 ms 2 * * * 3 * * * 4 31.55.187.176 25.230 ms 25.764 ms 27.508 ms 5 195.99.127.28 27.510 ms 213.121.192.128 27.511 ms 195.99.127.28 29.373 ms 6 62.172.103.166 29.283 ms 109.159.252.236 23.490 ms 194.72.16.136 23.425 ms 7 166.49.214.194 23.412 ms 8.602 ms 9.343 ms 8 * * * 9 154.54.58.173 11.529 ms 154.54.57.161 12.251 ms 154.54.58.173 11.821 ms 10 154.54.82.38 81.465 ms 82.444 ms 80.747 ms 11 154.54.0.221 80.835 ms 77.274 ms 66.28.4.237 78.316 ms 12 154.54.26.129 92.414 ms 154.54.29.173 87.723 ms 154.54.26.129 93.750 ms 13 154.54.7.129 95.915 ms 107.395 ms 154.54.6.221 94.576 ms 14 154.54.42.165 105.907 ms 106.179 ms 154.54.44.169 105.721 ms 15 154.54.5.89 132.282 ms 154.54.31.89 119.008 ms 154.54.5.89 131.187 ms 16 154.54.41.145 129.914 ms 129.875 ms 129.912 ms 17 154.54.44.73 147.436 ms 154.54.86.110 161.980 ms 154.54.2.197 164.780 ms 18 154.54.42.150 154.627 ms 156.198 ms 154.54.31.78 156.930 ms 19 38.142.108.50 161.180 ms 163.538 ms 165.019 ms 20 207.98.64.83 144.157 ms 143.916 ms 144.208 ms 21 207.98.64.45 163.439 ms 207.98.64.25 158.499 ms 207.98.64.45 158.470 ms 22 207.98.64.19 147.255 ms 146.393 ms 145.778 ms 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *
(In reply to Sergei Trofimovich from comment #0) > Example problematic session: > > $ rm oxygen-icons5-5.80.0.tar.xz > $ LANG=C wget > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > --2021-06-03 08:40:22-- > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > Resolving distfiles.gentoo.org... 195.181.175.49, 195.181.175.52, > 195.181.175.45, ... > Connecting to distfiles.gentoo.org|195.181.175.49|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: 236044748 (225M) [application/x-xz] > Saving to: 'oxygen-icons5-5.80.0.tar.xz' > > oxygen-icons5-5.80.0.tar.xz 0%[ > ] 1.07M 16.6KB/s eta 2h 5m ^C > > $ rm oxygen-icons5-5.80.0.tar.xz > $ LANG=C wget > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > --2021-06-03 08:41:01-- > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > Resolving distfiles.gentoo.org... 195.181.175.52, 195.181.175.46, > 195.181.175.54, ... > Connecting to distfiles.gentoo.org|195.181.175.52|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: 236044748 (225M) [application/x-xz] > Saving to: 'oxygen-icons5-5.80.0.tar.xz' > > oxygen-icons5-5.80.0.tar.xz 0%[ > ] 1.30M 21.0KB/s eta 1h 47m ^C > > $ rm oxygen-icons5-5.80.0.tar.xz > $ LANG=C wget > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > --2021-06-03 08:41:43-- > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > Resolving distfiles.gentoo.org... 195.181.175.51, 195.181.175.48, > 195.181.175.54, ... > Connecting to distfiles.gentoo.org|195.181.175.51|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: 236044748 (225M) [application/x-xz] > Saving to: 'oxygen-icons5-5.80.0.tar.xz' > > oxygen-icons5-5.80.0.tar.xz 4%[====> > ] 10.37M 5.38KB/s in 25s > > 2021-06-03 08:42:08 (430 KB/s) - Connection closed at byte 10878349. > Retrying. > > --2021-06-03 08:42:09-- (try: 2) > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > Connecting to distfiles.gentoo.org|195.181.175.51|:443... connected. > HTTP request sent, awaiting response... 206 Partial Content > Length: 236044748 (225M), 225166399 (215M) remaining [application/x-xz] > Saving to: 'oxygen-icons5-5.80.0.tar.xz' > > oxygen-icons5-5.80.0.tar.xz 5%[+++++> > ] 12.48M 15.8KB/s in 16s > > 2021-06-03 08:42:26 (139 KB/s) - Connection closed at byte 13089498. > Retrying. > > --2021-06-03 08:42:28-- (try: 3) > https://distfiles.gentoo.org/distfiles/45/oxygen-icons5-5.80.0.tar.xz > Connecting to distfiles.gentoo.org|195.181.175.51|:443... > > $ traceroute -n 195.181.175.49 > traceroute to 195.181.175.49 (195.181.175.49), 30 hops max, 60 byte packets > 1 192.168.1.254 3.338 ms 3.408 ms 4.253 ms > 2 * * * > 3 * * * > 4 31.55.187.188 19.498 ms 23.358 ms 24.091 ms > 5 195.99.127.178 24.087 ms 213.121.192.96 24.995 ms 213.121.192.112 > 25.574 ms > 6 62.172.103.172 27.107 ms 109.159.252.184 25.106 ms 194.72.16.142 > 26.912 ms > 7 185.229.189.58 28.018 ms 12.842 ms 10.435 ms > 8 185.229.188.4 22.661 ms 185.229.188.12 23.500 ms 185.229.188.4 23.794 > ms > 9 185.229.188.155 24.434 ms 24.185 ms 185.229.188.157 24.170 ms > 10 195.181.175.49 23.483 ms 23.378 ms 23.381 ms Can you run 'mtr' instead? I want to see if there is packet loss on the routes from you to the CDN. mtr <dns-name> (not a URL!) should be good. Also unclear if this is cache fill, so one test is to pick a CDN IP, fetch a single object completely, then fetch it again. Are they the same speed? If yes, then maybe a problem between you and the CDN, or the CDN is not caching properly. -A
or as another thing "is it only this file or all files are slow"?
With 20kb/s one needs a lot of patience to have 200MB file downloaded fully :) ETA claims to take a few hours, I'll try to. Anecdotally multiple files are affected. Trying to find another example: """ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e krita-4.4.3.ebuild clean fetch >>> Resuming download... >>> Downloading 'https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz' --2021-06-03 21:48:31-- https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz Resolving distfiles.gentoo.org... 185.59.220.18, 195.181.175.55, 195.181.174.6, ... Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 254855083 (243M), 205801877 (196M) remaining [application/x-gzip] Saving to: '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' entoo/distfiles/krita-4.4.3.ta 19%[+++++++++ ] 46.93M 11.5KB/s eta 4h 51m """ mtr output (only one row has 90% loss, probably router is aggressively cutting responses) """ My traceroute [v0.94] zn3 (192.168.1.201) -> 185.59.220.18 2021-06-03T21:52:23+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 192.168.1.254 0.0% 135 6.5 4.5 1.2 42.0 7.1 2. 172.16.11.2 0.0% 134 7.5 8.0 6.6 28.8 2.1 3. 31.55.187.181 96.2% 134 7.8 8.2 7.8 8.9 0.4 4. 31.55.187.188 0.0% 134 8.5 8.8 7.3 11.9 0.8 5. core2-hu0-8-0-1.southbank.ukcore.bt.net 0.0% 134 7.6 8.5 7.2 13.8 0.9 6. peer7-et-4-1-5.telehouse.ukcore.bt.net 0.0% 134 8.7 9.3 7.5 29.4 2.8 7. bt-lon.cdn77.com 0.0% 134 8.3 8.4 7.2 16.6 1.1 8. vl1101.fra-itx7-core-2.cdn77.com 0.0% 134 19.4 20.0 18.7 31.0 1.3 9. unassigned.cdn77.com 0.0% 134 20.0 20.0 18.7 25.7 0.9 10. frankfurt-10.cdn77.com 0.0% 134 19.3 20.3 18.8 51.1 3.0 """
Added an entry to /etc/hosts to pin to a particular IP: 185.59.220.18 distfiles.gentoo.org and trying to download php. Initially the transfer speed is very fast (about ~2MB/s), then it reconnects and drops down to 20KB/s. Fun fact: ongoing krita download for a #comment4 is not affected by this download (no reconnects or speed fluctuations) and is steady on 20KB/s. """ $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e php-8.0.6.ebuild clean fetch >>> Downloading 'https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz' --2021-06-03 21:56:16-- https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz Resolving distfiles.gentoo.org... 185.59.220.18 Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10764932 (10M) [application/x-xz] Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> ] 1.24M 202KB/s eta 44s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> ] 1.28M 4.92KB/s in 18s 2021-06-03 21:56:35 (71.4 KB/s) - Connection closed at byte 1342874. Retrying. --2021-06-03 21:56:36-- (try: 2) https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 10764932 (10M), 9422058 (9.0M) remaining [application/x-xz] Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 26%[++++++++++++++================> ] 2.69M 136KB/s eta 44s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 29%[++++++++++++++====================> ] 3.07M 26.0KB/s eta 89s """
(In reply to Sergei Trofimovich from comment #5) > Added an entry to /etc/hosts to pin to a particular IP: > 185.59.220.18 distfiles.gentoo.org > > and trying to download php. Initially the transfer speed is very fast (about > ~2MB/s), then it reconnects and drops down to 20KB/s. > > Fun fact: ongoing krita download for a #comment4 is not affected by this > download (no reconnects or speed fluctuations) and is steady on 20KB/s. > > """ > $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e php-8.0.6.ebuild > clean fetch > >>> Downloading 'https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz' > --2021-06-03 21:56:16-- > https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz > Resolving distfiles.gentoo.org... 185.59.220.18 > Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: 10764932 (10M) [application/x-xz] > Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' > > /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> > ] 1.24M 202KB/s eta 44s > /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> > ] 1.28M 4.92KB/s in 18s > > 2021-06-03 21:56:35 (71.4 KB/s) - Connection closed at byte 1342874. > Retrying. > > --2021-06-03 21:56:36-- (try: 2) > https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz > Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. > HTTP request sent, awaiting response... 206 Partial Content > Length: 10764932 (10M), 9422058 (9.0M) remaining [application/x-xz] After a successful php download refect did not make things faster: """ # previous session completion: $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e php-8.0.6.ebuild clean fetch >>> Downloading 'https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz' --2021-06-03 21:56:16-- https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz Resolving distfiles.gentoo.org... 185.59.220.18 Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10764932 (10M) [application/x-xz] Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> ] 1.24M 202KB/s eta 44s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 12%[=============> ] 1.28M 4.92KB/s in 18s 2021-06-03 21:56:35 (71.4 KB/s) - Connection closed at byte 1342874. Retrying. --2021-06-03 21:56:36-- (try: 2) https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 10764932 (10M), 9422058 (9.0M) remaining [application/x-xz] Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 26%[++++++++++++++================> ] 2.69M 136KB/s eta 44s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 80%[++++++++++++++================================================================================> ] 8.23M 29.9KB/s eta 77s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 87%[++++++++++++++=========================================================================================> ] 8.99M 17.7KB/s eta 50s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 89%[++++++++++++++===========================================================================================> ] 9.15M 13.3KB/s eta 46s /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 100%[++++++++++++++========================================================================================================>] 10.27M 15.1KB/s in 6m 21s 2021-06-03 22:02:57 (24.2 KB/s) - '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' saved [10764932/10764932] * php-8.0.6.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ] # next session start $ rm /gentoo/distfiles/php-8.0.6.tar.xz $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e php-8.0.6.ebuild clean fetch >>> Downloading 'https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz' --2021-06-03 22:03:07-- https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz Resolving distfiles.gentoo.org... 185.59.220.18 Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10764932 (10M) [application/x-xz] Saving to: '/gentoo/distfiles/php-8.0.6.tar.xz.__download__' /gentoo/distfiles/php-8.0.6.tar.xz.__download__ 15%[=================> ] 1.58M 23.1KB/s eta 5m 40s """
(In reply to Sergei Trofimovich from comment #4) > With 20kb/s one needs a lot of patience to have 200MB file downloaded fully > :) ETA claims to take a few hours, I'll try to. > > Anecdotally multiple files are affected. Trying to find another example: > > """ > LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e krita-4.4.3.ebuild > clean fetch > >>> Resuming download... > >>> Downloading 'https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz' Second fetch was fast: """ # first fetch $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e krita-4.4.3.ebuild clean fetch >>> Downloading 'https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz' --2021-06-03 21:48:04-- https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz Resolving distfiles.gentoo.org... 195.181.175.54, 195.181.174.6, 195.181.175.48, ... Connecting to distfiles.gentoo.org|195.181.175.54|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 254855083 (243M) [application/x-gzip] Saving to: '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' rita-4.4.3.tar.gz.__download__ 19%[========> ] 46.73M 1.94MB/s eta 98s ^CTask was destroyed but it is pending! task: <Task pending name='Task-5' coro=<PipeLogger._io_loop() done, defined at /usr/lib/python3.9/site-packages/portage/util/_async/PipeLogger.py:79> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f9b6e48a8b0>()]> cb=[PipeLogger._io_loop_done()]> Task was destroyed but it is pending! task: <Task pending name='Task-6' coro=<BuildLogger._main() done, defined at /usr/lib/python3.9/site-packages/portage/util/_async/BuildLogger.py:83> wait_for=<Future pending cb=[AsynchronousTask.async_wait.<locals>.<lambda>() at /usr/lib/python3.9/site-packages/_emerge/AsynchronousTask.py:42, <TaskWakeupMethWrapper object at 0x7f9b6db13040>()]> cb=[BuildLogger._main_exit()]> Task was destroyed but it is pending! task: <Task pending name='Task-7' coro=<PipeLogger._io_loop() done, defined at /usr/lib/python3.9/site-packages/portage/util/_async/PipeLogger.py:79> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f9b6e38d3d0>()]> cb=[PipeLogger._io_loop_done()]> [zn3] ~/portage/gentoo/media-gfx/krita:LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e krita-4.4.3.ebuild clean fetch>>> Resuming download... >>> Downloading 'https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz' --2021-06-03 21:48:31-- https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz Resolving distfiles.gentoo.org... 185.59.220.18, 195.181.175.55, 195.181.174.6, ... Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 254855083 (243M), 205801877 (196M) remaining [application/x-gzip] Saving to: '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' /gentoo/dis 24%[+++++++++==> ] 60.48M 20.4KB/s eta 2h 13m /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 30%[++++++++++++++++++++++=============> ] 73.79M 19.5KB/s eta 2h 12m /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 32%[++++++++++++++++++++++===============> ] 78.93M 37.9KB/s eta 2h 6m /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 57%[++++++++++++++++++++++============================================> ] 138.80M 35.5KB/s eta 73m 58s /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 64%[++++++++++++++++++++++=====================================================> ] 156.71M 28.7KB/s eta 60m 36s /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 77%[++++++++++++++++++++++=====================================================================> ] 188.92M 15.8KB/s in 99m 39s 2021-06-03 23:28:14 (24.3 KB/s) - Connection closed at byte 198097770. Retrying. --2021-06-03 23:28:15-- (try: 2) https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 254855083 (243M), 56757313 (54M) remaining [application/x-gzip] Saving to: '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++==========================>] 243.05M 772KB/s in 74s 2021-06-03 23:29:30 (746 KB/s) - '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' saved [254855083/254855083] * krita-4.4.3.tar.gz BLAKE2B SHA512 size ;-) ... [ ok ] >>> Downloading 'https://distfiles.gentoo.org/distfiles/28/krita-4.2.9-patchset.tar.xz' --2021-06-03 23:29:31-- https://distfiles.gentoo.org/distfiles/28/krita-4.2.9-patchset.tar.xz Resolving distfiles.gentoo.org... 185.59.220.18 Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7928 (7.7K) [application/x-xz] Saving to: '/gentoo/distfiles/krita-4.2.9-patchset.tar.xz.__download__' /gentoo/distfiles/krita-4.2.9-patchset.tar.xz.__downl 100%[======================================================================================================================>] 7.74K --.-KB/s in 0s 2021-06-03 23:29:31 (8.19 GB/s) - '/gentoo/distfiles/krita-4.2.9-patchset.tar.xz.__download__' saved [7928/7928] * krita-4.2.9-patchset.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ] # second fetch $ rm /gentoo/distfiles/krita-4.4.3.tar.gz $ LANG=C GENTOO_MIRRORS=https://distfiles.gentoo.org e krita-4.4.3.ebuild clean fetch >>> Downloading 'https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz' --2021-06-03 23:30:40-- https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz Resolving distfiles.gentoo.org... 185.59.220.18 Connecting to distfiles.gentoo.org|185.59.220.18|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 254855083 (243M) [application/x-gzip] Saving to: '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' /gentoo/distfiles/krita-4.4.3.tar.gz.__download__ 100%[======================================================================================================================>] 243.05M 2.07MB/s in 3m 19s 2021-06-03 23:34:00 (1.22 MB/s) - '/gentoo/distfiles/krita-4.4.3.tar.gz.__download__' saved [254855083/254855083] * krita-4.4.3.tar.gz BLAKE2B SHA512 size ;-) ... [ ok ] * krita-4.2.9-patchset.tar.xz BLAKE2B SHA512 size ;-) ... """
Ok so there is a lot going on here: (1) If mtr says you have 90% packet loss; you are just not going to get good speeds with that over TCP because TCP doesn't do well over high loss connections (congestion control over TCP will reduce the # of inflight packets until it hits a floor or the link appears less congested.) (2) Multiple files is good, it eliminates specific files. (3) You tried 2 files (php, krita) and for php the second download was not faster; but it was for krita. The idea for this test was to try to see if the cdn was serving from the origin (might be slower, the CDN has to fill its cache) or whether it was serving from cache. Because we didn't get a clear idea (e.g. ideally both files were faster the 2nd time, but we didn't see that) this is probably an inconclusive test. My focus is still on (1) here; as it doesn't matter how good the CDN is if you have 90% loss to it. Another thing to clarify is like "mtr -z -T <name>" to try to both print: the AS number of the router dropping your packets and to use TCP packets instead of ICMP (maybe the router handles the protocols differently.) I'd expect MTR loss output to correlate with bad throughput though; if it doesn't then we have to debug further.
(In reply to Alec Warner from comment #8) > Ok so there is a lot going on here: > > (1) If mtr says you have 90% packet loss; you are just not going to get good > speeds with that over TCP because TCP doesn't do well over high loss > connections (congestion control over TCP will reduce the # of inflight > packets until it hits a floor or the link appears less congested.) Note that according packet loss does not propagate past that router. To see the effect on end-to-end it should propagate. Or I'm misinterpreting mtr numbers? On the other hand if I look at the wireshark I do see TCP retransmits from 185.59.220.18 every 25-150 packets (1-4%?) on slow transfers (but not fast ones). As if ACKs from my machine would not reach the sender with that rate. Don't know if it's a lot for TCP (BBR) or normal for unusually slow transfers. > (2) Multiple files is good, it eliminates specific files. > (3) You tried 2 files (php, krita) and for php the second download was not > faster; but it was for krita. The idea for this test was to try to see if > the cdn was serving from the origin (might be slower, the CDN has to fill > its cache) or whether it was serving from cache. Because we didn't get a > clear idea (e.g. ideally both files were faster the 2nd time, but we didn't > see that) this is probably an inconclusive test. Maybe it takes a while to ingest files? Today both https://distfiles.gentoo.org/distfiles/bb/php-8.0.6.tar.xz : fetch time 5s https://distfiles.gentoo.org/distfiles/7d/krita-4.4.3.tar.gz : fetch time 2min are served fast at max 2.0 MB/s speed from 185.59.220.18. Trying to fetch a new file is slow again: http://distfiles.gentoo.org/distfiles/1c/go-linux-ppc64-bootstrap-1.13.6.tbz go-linux-ppc64-bootstrap-1.13.6.tbz 0%[ ] 784.09K 19.8KB/s eta 87m 7s With reconnects and average speed of 20KB/s. Once it fully fetched I can't reproduce slow transfers second time. Another attempt for: http://distfiles.gentoo.org/distfiles/1c/gpredict-2.3.tar.gz - first fetch is 20KB/s - instant subsequent fetch is 20KB/s - fetch after 10 minute pause is 2MB/s - instant subsequent fetch is 20KB/s (degraded again?) - instant subsequent fetch is 2MB/s - instant subsequent fetch is 2MB/s - instant subsequent fetch is 2MB/s > My focus is still on (1) here; as it doesn't matter how good the CDN is if > you have 90% loss to it. Another thing to clarify is like "mtr -z -T <name>" > to try to both print: the AS number of the router dropping your packets and > to use TCP packets instead of ICMP (maybe the router handles the protocols > differently.) I'd expect MTR loss output to correlate with bad throughput > though; if it doesn't then we have to debug further. I would expect both ICMP requests and TCP requests to have ICMP responses and both to have same aggressive filtering rules on the responder's side. `mtr` reports the same amount of packet loss. Note, packet loss is not propagated past `31.55.187.177`. """ $ mtr -z -T 195.181.175.54 My traceroute [v0.94] zn3 (192.168.1.201) -> 195.181.175.54 2021-06-04T07:55:31+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. AS??? 192.168.1.254 0.4% 465 3068. 104.0 1.2 7103. 625.6 2. (waiting for reply) 3. AS2856 31.55.187.177 65.3% 465 3107. 2734. 6.9 7150. 2690. 4. AS2856 31.55.187.176 0.0% 465 8.1 9.0 7.3 65.8 3.0 5. AS2856 core2-hu0-17-0-5.southbank.ukcore.bt.net 0.0% 465 9.3 9.3 7.2 117.2 6.8 AS2856 core1-hu0-15-0-8.ukcore.bt.net AS2856 host213-121-192-106.ukcore.bt.net AS2856 host213-121-192-98.ukcore.bt.net AS2856 core1-hu0-2-0-1.southbank.ukcore.bt.net AS2856 194.72.31.170 AS2856 core1-hu0-17-0-1.southbank.ukcore.bt.net AS2856 core2-hu0-8-0-5.southbank.ukcore.bt.net AS2856 core1-hu0-16-0-8.southbank.ukcore.bt.net 6. AS2856 peer7-et-7-0-2.telehouse.ukcore.bt.net 0.0% 465 17.6 10.8 7.2 66.1 6.9 AS2856 peer8-et0-1-5.telehouse.ukcore.bt.net AS2856 peer8-et3-1-6.telehouse.ukcore.bt.net AS2856 peer8-et-0-1-7.telehouse.ukcore.bt.net AS2856 peer8-et4-0-2.telehouse.ukcore.bt.net AS2856 peer7-et-4-0-5.telehouse.ukcore.bt.net AS2856 peer8-et-3-0-5.telehouse.ukcore.bt.net AS2856 peer7-et-3-0-2.telehouse.ukcore.bt.net AS2856 peer8-et-3-1-2.telehouse.ukcore.bt.net 7. AS60068 bt-lon.cdn77.com 0.0% 464 8.2 8.4 6.9 66.5 2.9 AS60068 bt-lon.cdn77.com 8. AS60068 vl1101.fra-itx7-core-2.cdn77.com 0.0% 464 19.3 20.0 18.5 35.2 1.5 AS60068 vl1101.fra-itx7-core-1.cdn77.com 9. AS60068 unassigned.cdn77.com 0.0% 464 19.9 20.0 18.4 72.2 2.7 AS60068 unassigned.cdn77.com AS60068 unassigned.cdn77.com AS60068 unassigned.cdn77.com 10. AS60068 frankfurt-53.cdn77.com 0.0% 464 19.8 20.3 18.9 102.6 4.0 """
I can't reproduce the slowness against distfiles.gentoo.org anymore. Tested on 100 random files with size of ~50MB or more. Download speed is normally around 2MB/s.
(In reply to Sergei Trofimovich from comment #10) > I can't reproduce the slowness against distfiles.gentoo.org anymore. Tested > on 100 random files with size of ~50MB or more. Download speed is normally > around 2MB/s. I did some more debugging on the CDN on an unrelated bug recently. Some things I learned for next time. - The CDN doesn't always use anycast, so distfiles.g.o will resolve to multiple IPs. These IPs correspond to POPs. It would be good to know if a specific POP is fast / slow for you, as we can escalate this better to the CDN provider. E.g. in your earlier paste on this topic we see: "Resolving distfiles.gentoo.org... 195.181.175.49, 195.181.175.52, 195.181.175.45, ..." 195.181.175.45 195.181.175.49 195.181.175.52 These are potentially different IP routes into frankfurt: antarus@antarus-h8-1437c:~$ host 195.181.175.49 49.175.181.195.in-addr.arpa domain name pointer frankfurt-47.cdn77.com. antarus@antarus-h8-1437c:~$ host 195.181.175.45 45.175.181.195.in-addr.arpa domain name pointer frankfurt-44.cdn77.com. antarus@antarus-h8-1437c:~$ host 195.181.175.52 52.175.181.195.in-addr.arpa domain name pointer frankfurt-50.cdn77.com. If a particular one is slow its better to say "hey this IP is slow" than to say "frankfurt is slow." In the request headers (curl -vvv shows htem, unsure what the wget command is) they set x-77-* response headers and having one of those with a bad download can help us submit the traceID upstream and correlate what happened to it. Not saying we did anything wrong here; but for next time I think we have additional steps we can take on this.
