Via NetCologne: =============== > # time wget -4 -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > --2020-09-21 15:33:48-- https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183 > Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Date: Mon, 21 Sep 2020 13:33:48 GMT > Server: Apache > Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT > ETag: "1581c6d-1adfb67c-5aca154dc7740" > Accept-Ranges: bytes > Content-Length: 450868860 > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Content-Type: application/x-xz > Length: 450868860 (430M) [application/x-xz] > Saving to: ‘libvpx-testdata-1.9.0.tar.xz’ > > libvpx-testdata-1.9.0.tar.xz 100%[=================================================>] 429.98M 4.04MB/s in 22m 23s > > 2020-09-21 15:56:11 (328 KB/s) - ‘libvpx-testdata-1.9.0.tar.xz’ saved [450868860/450868860] > > > real 22m23.704s > user 0m1.576s > sys 0m3.074s > > > # traceroute -4 dev.gentoo.org > traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets > 1 fb1.intern (10.1.0.254) 1.233 ms 1.386 ms 1.029 ms > 2 bras-vc1.netcologne.de (195.14.226.20) 18.118 ms 18.786 ms 19.850 ms > 3 ip-core-eup2-ae13.netcologne.de (89.1.16.145) 20.748 ms 25.351 ms ip-core-eup1-ae13.netcologne.de (89.1.16.141) 18.798 ms > 4 ip-core-sto2-et10-3-2.netcologne.de (87.79.17.30) 24.871 ms bdr-sto1-ae2.netcologne.de (81.173.192.118) 19.378 ms bdr-sto1-ae1.netcologne.de (81.173.192.114) 23.012 ms > 5 bdr-sto1-ae1.netcologne.de (81.173.192.114) 22.329 ms bdr-sto1-ae2.netcologne.de (81.173.192.118) 22.149 ms et-0-0-18-1000.edge6.Dusseldorf1.Level3.net (62.67.17.17) 26.873 ms > 6 et-0-0-18-1000.edge6.Dusseldorf1.Level3.net (62.67.17.17) 28.030 ms * 21.061 ms > 7 CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 169.535 ms * * > 8 * * CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 167.633 ms > 9 * * dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 169.765 ms > 10 dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80) 181.498 ms lon--cenic-100ge.cenic.net (137.164.3.103) 183.219 ms dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80) 181.451 ms > 11 lon--cenic-100ge.cenic.net (137.164.3.103) 185.657 ms 184.765 ms eugn-p1-gw.nero.net (207.98.64.196) 188.862 ms > 12 corv-p2-gw.nero.net (207.98.64.13) 190.442 ms 179.874 ms 207.98.64.47 (207.98.64.47) 188.221 ms > 13 corv-car1-gw.nero.net (207.98.64.17) 179.221 ms corv-p2-gw.nero.net (207.98.64.13) 182.154 ms 182.180 ms > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * Via Hetzner: ============ > $ time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.ta > r.xz > --2020-09-21 15:43:47-- https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183 > Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Date: Mon, 21 Sep 2020 13:43:48 GMT > Server: Apache > Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT > ETag: "1581c6d-1adfb67c-5aca154dc7740" > Accept-Ranges: bytes > Content-Length: 450868860 > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Content-Type: application/x-xz > Length: 450868860 (430M) [application/x-xz] > Saving to: ‘/dev/null’ > > /dev/null 100%[=================================================>] 429,98M 3,53MB/s in 10m 2s > > 2020-09-21 15:53:50 (732 KB/s) - ‘/dev/null’ saved [450868860/450868860] > > > real 10m2,601s > user 0m1,645s > sys 0m0,652s > > > $ traceroute -4 dev.gentoo.org > traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets > 1 static.17.200.243.136.clients.your-server.de (136.243.200.17) 0.314 ms 0.303 ms 0.312 ms > 2 * core24.fsn1.hetzner.com (213.239.229.53) 0.244 ms 0.516 ms > 3 juniper4.nbg1.hetzner.com (213.239.252.229) 2.700 ms 4.220 ms juniper5.nbg1.hetzner.com (213.239.252.245) 2.571 ms > 4 ae5-710.nbg40.core-backbone.com (81.95.15.5) 2.851 ms ae3-710.nbg40.core-backbone.com (5.56.20.253) 2.827 ms 2.838 ms > 5 ae1-2003.fra20.core-backbone.com (80.255.14.82) 6.137 ms 6.131 ms 6.100 ms > 6 195.122.181.129 (195.122.181.129) 6.295 ms 6.328 ms 6.326 ms > 7 * * * > 8 CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 152.465 ms 152.369 ms 152.436 ms > 9 dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 152.802 ms dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29) 153.484 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 152.762 ms > 10 dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80) 152.289 ms 152.259 ms 152.273 ms > 11 lon--cenic-100ge.cenic.net (137.164.3.103) 163.857 ms 163.510 ms 163.840 ms > 12 eugn-p1-gw.nero.net (207.98.64.196) 165.532 ms 165.740 ms 165.734 ms > 13 207.98.64.47 (207.98.64.47) 167.684 ms corv-p2-gw.nero.net (207.98.64.13) 167.636 ms 207.98.64.47 (207.98.64.47) 167.645 ms > 14 corv-car1-gw.nero.net (207.98.64.17) 164.789 ms 164.755 ms 164.766 ms > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * Via Manitu: =========== > # time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > --2020-09-21 15:35:35-- https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > Resolving dev.gentoo.org... 140.211.166.183 > Connecting to dev.gentoo.org|140.211.166.183|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Date: Mon, 21 Sep 2020 13:35:36 GMT > Server: Apache > Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT > ETag: "1581c6d-1adfb67c-5aca154dc7740" > Accept-Ranges: bytes > Content-Length: 450868860 > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Content-Type: application/x-xz > Length: 450868860 (430M) [application/x-xz] > Saving to: ‘/dev/null’ > > /dev/null 100%[=================================================>] 429,98M 299KB/s in 15m 43s > > 2020-09-21 15:51:19 (467 KB/s) - ‘/dev/null’ saved [450868860/450868860] > > > real 15m43,938s > user 0m10,716s > sys 0m1,901s > > > traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets > 1 1.64.238.89.in-addr.arpa.manitu.net (89.238.64.1) 0.111 ms 0.092 ms 0.080 ms > 2 * * * > 3 8-1-2.ear3.Frankfurt1.Level3.net (62.67.35.161) 5.355 ms 5.343 ms 5.406 ms > 4 * * * > 5 CENIC.ear1.SanJose1.Level3.net (4.15.122.46) 153.271 ms 153.341 ms 153.385 ms > 6 dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 153.491 ms dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29) 154.359 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 153.904 ms > 7 dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80) 152.960 ms 152.985 ms 152.984 ms > 8 lon--cenic-100ge.cenic.net (137.164.3.103) 164.412 ms 164.362 ms 164.636 ms > 9 eugn-p1-gw.nero.net (207.98.64.196) 166.527 ms 166.475 ms 166.483 ms > 10 207.98.64.47 (207.98.64.47) 167.438 ms 167.442 ms corv-p2-gw.nero.net (207.98.64.13) 167.367 ms > 11 corv-car1-gw.nero.net (207.98.64.17) 165.601 ms 165.547 ms 165.618 ms > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * Via DataPacket: =============== > # time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > --2020-09-21 15:34:40-- https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > Resolving dev.gentoo.org... 140.211.166.183 > Connecting to dev.gentoo.org|140.211.166.183|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Date: Mon, 21 Sep 2020 13:34:41 GMT > Server: Apache > Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT > ETag: "1581c6d-1adfb67c-5aca154dc7740" > Accept-Ranges: bytes > Content-Length: 450868860 > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Content-Type: application/x-xz > Length: 450868860 (430M) [application/x-xz] > Saving to: ‘/dev/null’ > > /dev/null 100%[=================================================>] 429,98M 5,34MB/s in 9m 35s > > 2020-09-21 15:44:16 (766 KB/s) - ‘/dev/null’ saved [450868860/450868860] > > > real 9m35,891s > user 0m1,180s > sys 0m1,852s > > > traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets > 1 redacted.datapacket.com (195.181.x.z) 0.176 ms unn-195-181-170-189.datapacket.com (195.181.170.189) 0.165 ms 0.147 ms > 2 unassigned.cdn77.com (185.229.188.156) 0.260 ms unassigned.cdn77.com (185.229.188.160) 0.232 ms unassigned.cdn77.com (185.229.188.156) 0.205 ms > 3 unassigned.cdn77.com (185.229.188.151) 0.277 ms 0.196 ms 0.210 ms > 4 de-fra02a-rc1-ae-0-0.aorta.net (84.116.137.78) 17.189 ms 19.427 ms 19.415 ms > 5 de-fra02a-rc1-ae-0-0.aorta.net (84.116.137.78) 19.434 ms nl-ams02a-rc2-lag-8-0.aorta.net (84.116.130.98) 12.235 ms 12.244 ms > 6 nl-ams02a-rc2-lag-8-0.aorta.net (84.116.130.98) 12.290 ms 12.399 ms 12.350 ms > 7 nl-sar.nordu.net (109.105.98.186) 9.069 ms nl-ams04a-ri3-ae-8-0.aorta.net (84.116.130.97) 14.421 ms 14.406 ms > 8 nl-sar.nordu.net (109.105.98.186) 9.042 ms 9.014 ms 8.991 ms > 9 us-man.nordu.net (109.105.97.64) 95.556 ms 99.83.66.169 (99.83.66.169) 100.116 ms us-man.nordu.net (109.105.97.64) 94.282 ms > 10 * 99.83.66.169 (99.83.66.169) 99.996 ms 99.966 ms > 11 * * * > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 ptck-p1-gw.nero.net (207.98.64.79) 160.211 ms * 160.488 ms > 20 corv-p1-gw.nero.net (207.98.64.25) 164.524 ms ptck-p1-gw.nero.net (207.98.64.79) 160.217 ms corv-p1-gw.nero.net (207.98.64.25) 165.439 ms > 21 207.98.64.45 (207.98.64.45) 163.627 ms corv-car1-gw.nero.net (207.98.64.19) 160.100 ms 160.804 ms > 22 * * corv-car1-gw.nero.net (207.98.64.19) 160.094 ms > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * This is not new. But now I lost patient and filed a bug.
For comparison, it maxes out my download bandwidth from my home in Detroit, MI, USA. > floppym@naomi tmp % time wget -4 -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > --2020-09-21 10:24:55-- https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz > Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183 > Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected. > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Date: Mon, 21 Sep 2020 14:24:56 GMT > Server: Apache > Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT > ETag: "1581c6d-1adfb67c-5aca154dc7740" > Accept-Ranges: bytes > Content-Length: 450868860 > Keep-Alive: timeout=15, max=100 > Connection: Keep-Alive > Content-Type: application/x-xz > Length: 450868860 (430M) [application/x-xz] > Saving to: ‘libvpx-testdata-1.9.0.tar.xz’ > > libvpx-testdata-1.9.0.ta 100%[==================================>] 429.98M 11.8MB/s in 35s > > 2020-09-21 10:25:32 (12.1 MB/s) - ‘libvpx-testdata-1.9.0.tar.xz’ saved [450868860/450868860] > > wget -4 -S 11.04s user 2.98s system 38% cpu 36.643 total > > floppym@naomi tmp % traceroute -4 dev.gentoo.org > traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets > 1 192.168.0.1 (192.168.0.1) 1.743 ms 2.051 ms 2.352 ms > 2 d149-67-1-56.try.wideopenwest.com (67.149.56.1) 144.304 ms 144.255 ms 144.863 ms > 3 d47-69-97-57.col.wideopenwest.com (69.47.57.97) 18.719 ms 18.772 ms 18.861 ms > 4 76-73-165-180.knology.net (76.73.165.180) 18.949 ms 21.080 ms 19.031 ms > 5 ip65-46-186-97.z186-46-65.customer.algx.net (65.46.186.97) 26.053 ms 25.849 ms 25.930 ms > 6 207.88.13.13.ptr.us.xo.net (207.88.13.13) 74.089 ms 72.215 ms 72.135 ms > 7 198.32.251.209 (198.32.251.209) 85.457 ms 79.763 ms 83.628 ms > 8 dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29) 84.404 ms 84.339 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31) 80.624 ms > 9 dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80) 81.499 ms * 76.697 ms > 10 lon--cenic-100ge.cenic.net (137.164.3.103) 71.977 ms 70.990 ms 70.944 ms > 11 eugn-p1-gw.nero.net (207.98.64.196) 80.457 ms 79.744 ms 77.812 ms > 12 corv-p2-gw.nero.net (207.98.64.13) 83.734 ms 82.461 ms 79.748 ms > 13 corv-car1-gw.nero.net (207.98.64.17) 70.789 ms 70.932 ms 75.366 ms > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * *
Created attachment 661789 [details] wget.log
(In reply to Thomas Deutschmann from comment #2) > Created attachment 661789 [details] > wget.log Can you do a tcp capture? I'm curious about the tcp window params for your slow link (receive and congestion window.) In my test: TCP segment sizes ranged from 1448 to 7420. My TCP window was 61 segments. My RTT is 54ms If we round: 1000ms in a second / 54ms ~= 20 complete RTTs in a second. 61 inflights per RTT * 20 = 1220 segments per second. When segments are 1448: 1220 * 1448 == 1766560 bytes / second (70mbps) When segments are 7420: 1220 * 7420 == 9052400 bytes / second (13mbps) Now these are all theoretical speeds (we don't actually sustain 61 inflghts consistently, rtt is not quite 54ms all the time, the connection does not sustain a particular segment size, etc.) Find me on IRC and we can do a test. But in theory: throughput = RTT * window_size * segment_size I'm curious to know what the bottleneck is here; is the window size just small? is the kernel picking small segments (e.g. because big segments are lost on the link somewhere, causing a stall and impacting window fill?)
(In reply to Alec Warner from comment #3) > (In reply to Thomas Deutschmann from comment #2) > > Created attachment 661789 [details] > > wget.log > > Can you do a tcp capture? I'm curious about the tcp window params for your > slow link (receive and congestion window.) > > In my test: > > TCP segment sizes ranged from 1448 to 7420. > My TCP window was 61 segments. > My RTT is 54ms > > If we round: > 1000ms in a second / 54ms ~= 20 complete RTTs in a second. > 61 inflights per RTT * 20 = 1220 segments per second. > > When segments are 1448: 1220 * 1448 == 1766560 bytes / second (70mbps) > When segments are 7420: 1220 * 7420 == 9052400 bytes / second (13mbps) > > Now these are all theoretical speeds (we don't actually sustain 61 inflghts > consistently, rtt is not quite 54ms all the time, the connection does not > sustain a particular segment size, etc.) > > Find me on IRC and we can do a test. But in theory: > > throughput = RTT * window_size * segment_size > > I'm curious to know what the bottleneck is here; is the window size just > small? is the kernel picking small segments (e.g. because big segments are > lost on the link somewhere, causing a stall and impacting window fill?) Testing today with whissi and others: - TCP window: 61 - segment size: 1440 - These two things result in the 'slow' path (< 1MB/s) due to unfill windows and longer RTTs for europeans. The cause appears to "the internet is bad" and various links are congested. This in turn causes loss, which causes tcp stack to reduce segment sizes (to 1440, the nominal smallest segment size; I haven't seen segments smaller than this.) Loss at some links was 20-80%; which..I mean..you can't get a good speed over that kind of link using standard control algorithms. FWIW I reported this upstream (to the OSL) but I don't expect a short term solution. -A