Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 743962 - Connection to dev.gentoo.org is very slow
Summary: Connection to dev.gentoo.org is very slow
Status: IN_PROGRESS
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Dev box issues (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-09-21 13:58 UTC by Thomas Deutschmann (RETIRED)
Modified: 2020-09-22 21:37 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
wget.log (wget.log,679.97 KB, text/plain)
2020-09-21 15:10 UTC, Thomas Deutschmann (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Deutschmann (RETIRED) gentoo-dev 2020-09-21 13:58:49 UTC
Via NetCologne:
===============
> # time wget -4 -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> --2020-09-21 15:33:48--  https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183
> Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Mon, 21 Sep 2020 13:33:48 GMT
>   Server: Apache
>   Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT
>   ETag: "1581c6d-1adfb67c-5aca154dc7740"
>   Accept-Ranges: bytes
>   Content-Length: 450868860
>   Keep-Alive: timeout=15, max=100
>   Connection: Keep-Alive
>   Content-Type: application/x-xz
> Length: 450868860 (430M) [application/x-xz]
> Saving to: ‘libvpx-testdata-1.9.0.tar.xz’
> 
> libvpx-testdata-1.9.0.tar.xz  100%[=================================================>] 429.98M  4.04MB/s    in 22m 23s
> 
> 2020-09-21 15:56:11 (328 KB/s) - ‘libvpx-testdata-1.9.0.tar.xz’ saved [450868860/450868860]
> 
> 
> real    22m23.704s
> user    0m1.576s
> sys     0m3.074s
> 
> 
> # traceroute -4 dev.gentoo.org
> traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets
>  1  fb1.intern (10.1.0.254)  1.233 ms  1.386 ms  1.029 ms
>  2  bras-vc1.netcologne.de (195.14.226.20)  18.118 ms  18.786 ms  19.850 ms
>  3  ip-core-eup2-ae13.netcologne.de (89.1.16.145)  20.748 ms  25.351 ms ip-core-eup1-ae13.netcologne.de (89.1.16.141)  18.798 ms
>  4  ip-core-sto2-et10-3-2.netcologne.de (87.79.17.30)  24.871 ms bdr-sto1-ae2.netcologne.de (81.173.192.118)  19.378 ms bdr-sto1-ae1.netcologne.de (81.173.192.114)  23.012 ms
>  5  bdr-sto1-ae1.netcologne.de (81.173.192.114)  22.329 ms bdr-sto1-ae2.netcologne.de (81.173.192.118)  22.149 ms et-0-0-18-1000.edge6.Dusseldorf1.Level3.net (62.67.17.17)  26.873 ms
>  6  et-0-0-18-1000.edge6.Dusseldorf1.Level3.net (62.67.17.17)  28.030 ms *  21.061 ms
>  7  CENIC.ear1.SanJose1.Level3.net (4.15.122.46)  169.535 ms * *
>  8  * * CENIC.ear1.SanJose1.Level3.net (4.15.122.46)  167.633 ms
>  9  * * dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  169.765 ms
> 10  dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80)  181.498 ms lon--cenic-100ge.cenic.net (137.164.3.103)  183.219 ms dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80)  181.451 ms
> 11  lon--cenic-100ge.cenic.net (137.164.3.103)  185.657 ms  184.765 ms eugn-p1-gw.nero.net (207.98.64.196)  188.862 ms
> 12  corv-p2-gw.nero.net (207.98.64.13)  190.442 ms  179.874 ms 207.98.64.47 (207.98.64.47)  188.221 ms
> 13  corv-car1-gw.nero.net (207.98.64.17)  179.221 ms corv-p2-gw.nero.net (207.98.64.13)  182.154 ms  182.180 ms
> 14  * * *
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  * * *
> 20  * * *
> 21  * * *
> 22  * * *
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * * *
> 28  * * *
> 29  * * *
> 30  * * *


Via Hetzner:
============
> $ time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.ta
> r.xz
> --2020-09-21 15:43:47--  https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183
> Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Mon, 21 Sep 2020 13:43:48 GMT
>   Server: Apache
>   Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT
>   ETag: "1581c6d-1adfb67c-5aca154dc7740"
>   Accept-Ranges: bytes
>   Content-Length: 450868860
>   Keep-Alive: timeout=15, max=100
>   Connection: Keep-Alive
>   Content-Type: application/x-xz
> Length: 450868860 (430M) [application/x-xz]
> Saving to: ‘/dev/null’
> 
> /dev/null                     100%[=================================================>] 429,98M  3,53MB/s    in 10m 2s
> 
> 2020-09-21 15:53:50 (732 KB/s) - ‘/dev/null’ saved [450868860/450868860]
> 
> 
> real    10m2,601s
> user    0m1,645s
> sys     0m0,652s
> 
> 
> $ traceroute -4 dev.gentoo.org
> traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets
>  1  static.17.200.243.136.clients.your-server.de (136.243.200.17)  0.314 ms  0.303 ms  0.312 ms
>  2  * core24.fsn1.hetzner.com (213.239.229.53)  0.244 ms  0.516 ms
>  3  juniper4.nbg1.hetzner.com (213.239.252.229)  2.700 ms  4.220 ms juniper5.nbg1.hetzner.com (213.239.252.245)  2.571 ms
>  4  ae5-710.nbg40.core-backbone.com (81.95.15.5)  2.851 ms ae3-710.nbg40.core-backbone.com (5.56.20.253)  2.827 ms  2.838 ms
>  5  ae1-2003.fra20.core-backbone.com (80.255.14.82)  6.137 ms  6.131 ms  6.100 ms
>  6  195.122.181.129 (195.122.181.129)  6.295 ms  6.328 ms  6.326 ms
>  7  * * *
>  8  CENIC.ear1.SanJose1.Level3.net (4.15.122.46)  152.465 ms  152.369 ms  152.436 ms
>  9  dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  152.802 ms dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29)  153.484 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  152.762 ms
> 10  dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80)  152.289 ms  152.259 ms  152.273 ms
> 11  lon--cenic-100ge.cenic.net (137.164.3.103)  163.857 ms  163.510 ms  163.840 ms
> 12  eugn-p1-gw.nero.net (207.98.64.196)  165.532 ms  165.740 ms  165.734 ms
> 13  207.98.64.47 (207.98.64.47)  167.684 ms corv-p2-gw.nero.net (207.98.64.13)  167.636 ms 207.98.64.47 (207.98.64.47)  167.645 ms
> 14  corv-car1-gw.nero.net (207.98.64.17)  164.789 ms  164.755 ms  164.766 ms
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  * * *
> 20  * * *
> 21  * * *
> 22  * * *
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * * *
> 28  * * *
> 29  * * *
> 30  * * *


Via Manitu:
===========
> # time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> --2020-09-21 15:35:35--  https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> Resolving dev.gentoo.org... 140.211.166.183
> Connecting to dev.gentoo.org|140.211.166.183|:443... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Mon, 21 Sep 2020 13:35:36 GMT
>   Server: Apache
>   Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT
>   ETag: "1581c6d-1adfb67c-5aca154dc7740"
>   Accept-Ranges: bytes
>   Content-Length: 450868860
>   Keep-Alive: timeout=15, max=100
>   Connection: Keep-Alive
>   Content-Type: application/x-xz
> Length: 450868860 (430M) [application/x-xz]
> Saving to: ‘/dev/null’
> 
> /dev/null                     100%[=================================================>] 429,98M   299KB/s    in 15m 43s
> 
> 2020-09-21 15:51:19 (467 KB/s) - ‘/dev/null’ saved [450868860/450868860]
> 
> 
> real    15m43,938s
> user    0m10,716s
> sys     0m1,901s
> 
> 
> traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets
>  1  1.64.238.89.in-addr.arpa.manitu.net (89.238.64.1)  0.111 ms  0.092 ms  0.080 ms
>  2  * * *
>  3  8-1-2.ear3.Frankfurt1.Level3.net (62.67.35.161)  5.355 ms  5.343 ms  5.406 ms
>  4  * * *
>  5  CENIC.ear1.SanJose1.Level3.net (4.15.122.46)  153.271 ms  153.341 ms  153.385 ms
>  6  dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  153.491 ms dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29)  154.359 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  153.904 ms
>  7  dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80)  152.960 ms  152.985 ms  152.984 ms
>  8  lon--cenic-100ge.cenic.net (137.164.3.103)  164.412 ms  164.362 ms  164.636 ms
>  9  eugn-p1-gw.nero.net (207.98.64.196)  166.527 ms  166.475 ms  166.483 ms
> 10  207.98.64.47 (207.98.64.47)  167.438 ms  167.442 ms corv-p2-gw.nero.net (207.98.64.13)  167.367 ms
> 11  corv-car1-gw.nero.net (207.98.64.17)  165.601 ms  165.547 ms  165.618 ms
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  * * *
> 20  * * *
> 21  * * *
> 22  * * *
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * * *
> 28  * * *
> 29  * * *
> 30  * * *


Via DataPacket:
===============
> # time wget -4 -O /dev/null -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> --2020-09-21 15:34:40--  https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> Resolving dev.gentoo.org... 140.211.166.183
> Connecting to dev.gentoo.org|140.211.166.183|:443... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Mon, 21 Sep 2020 13:34:41 GMT
>   Server: Apache
>   Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT
>   ETag: "1581c6d-1adfb67c-5aca154dc7740"
>   Accept-Ranges: bytes
>   Content-Length: 450868860
>   Keep-Alive: timeout=15, max=100
>   Connection: Keep-Alive
>   Content-Type: application/x-xz
> Length: 450868860 (430M) [application/x-xz]
> Saving to: ‘/dev/null’
> 
> /dev/null                     100%[=================================================>] 429,98M  5,34MB/s    in 9m 35s
> 
> 2020-09-21 15:44:16 (766 KB/s) - ‘/dev/null’ saved [450868860/450868860]
> 
> 
> real    9m35,891s
> user    0m1,180s
> sys     0m1,852s
> 
> 
> traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets
>  1  redacted.datapacket.com (195.181.x.z)  0.176 ms unn-195-181-170-189.datapacket.com (195.181.170.189)  0.165 ms  0.147 ms
>  2  unassigned.cdn77.com (185.229.188.156)  0.260 ms unassigned.cdn77.com (185.229.188.160)  0.232 ms unassigned.cdn77.com (185.229.188.156)  0.205 ms
>  3  unassigned.cdn77.com (185.229.188.151)  0.277 ms  0.196 ms  0.210 ms
>  4  de-fra02a-rc1-ae-0-0.aorta.net (84.116.137.78)  17.189 ms  19.427 ms  19.415 ms
>  5  de-fra02a-rc1-ae-0-0.aorta.net (84.116.137.78)  19.434 ms nl-ams02a-rc2-lag-8-0.aorta.net (84.116.130.98)  12.235 ms  12.244 ms
>  6  nl-ams02a-rc2-lag-8-0.aorta.net (84.116.130.98)  12.290 ms  12.399 ms  12.350 ms
>  7  nl-sar.nordu.net (109.105.98.186)  9.069 ms nl-ams04a-ri3-ae-8-0.aorta.net (84.116.130.97)  14.421 ms  14.406 ms
>  8  nl-sar.nordu.net (109.105.98.186)  9.042 ms  9.014 ms  8.991 ms
>  9  us-man.nordu.net (109.105.97.64)  95.556 ms 99.83.66.169 (99.83.66.169)  100.116 ms us-man.nordu.net (109.105.97.64)  94.282 ms
> 10  * 99.83.66.169 (99.83.66.169)  99.996 ms  99.966 ms
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  ptck-p1-gw.nero.net (207.98.64.79)  160.211 ms *  160.488 ms
> 20  corv-p1-gw.nero.net (207.98.64.25)  164.524 ms ptck-p1-gw.nero.net (207.98.64.79)  160.217 ms corv-p1-gw.nero.net (207.98.64.25)  165.439 ms
> 21  207.98.64.45 (207.98.64.45)  163.627 ms corv-car1-gw.nero.net (207.98.64.19)  160.100 ms  160.804 ms
> 22  * * corv-car1-gw.nero.net (207.98.64.19)  160.094 ms
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * * *
> 28  * * *
> 29  * * *
> 30  * * *


This is not new. But now I lost patient and filed a bug.
Comment 1 Mike Gilbert gentoo-dev 2020-09-21 14:30:17 UTC
For comparison, it maxes out my download bandwidth from my home in Detroit, MI, USA.

> floppym@naomi tmp % time wget -4 -S https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> --2020-09-21 10:24:55--  https://dev.gentoo.org/~whissi/dist/libvpx/libvpx-testdata-1.9.0.tar.xz
> Resolving dev.gentoo.org (dev.gentoo.org)... 140.211.166.183
> Connecting to dev.gentoo.org (dev.gentoo.org)|140.211.166.183|:443... connected.
> HTTP request sent, awaiting response...
>   HTTP/1.1 200 OK
>   Date: Mon, 21 Sep 2020 14:24:56 GMT
>   Server: Apache
>   Last-Modified: Tue, 11 Aug 2020 22:09:57 GMT
>   ETag: "1581c6d-1adfb67c-5aca154dc7740"
>   Accept-Ranges: bytes
>   Content-Length: 450868860
>   Keep-Alive: timeout=15, max=100
>   Connection: Keep-Alive
>   Content-Type: application/x-xz
> Length: 450868860 (430M) [application/x-xz]
> Saving to: ‘libvpx-testdata-1.9.0.tar.xz’
> 
> libvpx-testdata-1.9.0.ta 100%[==================================>] 429.98M  11.8MB/s    in 35s
> 
> 2020-09-21 10:25:32 (12.1 MB/s) - ‘libvpx-testdata-1.9.0.tar.xz’ saved [450868860/450868860]
> 
> wget -4 -S   11.04s user 2.98s system 38% cpu 36.643 total
> 
> floppym@naomi tmp % traceroute -4 dev.gentoo.org
> traceroute to dev.gentoo.org (140.211.166.183), 30 hops max, 60 byte packets
>  1  192.168.0.1 (192.168.0.1)  1.743 ms  2.051 ms  2.352 ms
>  2  d149-67-1-56.try.wideopenwest.com (67.149.56.1)  144.304 ms  144.255 ms  144.863 ms
>  3  d47-69-97-57.col.wideopenwest.com (69.47.57.97)  18.719 ms  18.772 ms  18.861 ms
>  4  76-73-165-180.knology.net (76.73.165.180)  18.949 ms  21.080 ms  19.031 ms
>  5  ip65-46-186-97.z186-46-65.customer.algx.net (65.46.186.97)  26.053 ms  25.849 ms  25.930 ms
>  6  207.88.13.13.ptr.us.xo.net (207.88.13.13)  74.089 ms  72.215 ms  72.135 ms
>  7  198.32.251.209 (198.32.251.209)  85.457 ms  79.763 ms  83.628 ms
>  8  dc-svl-agg8--svl-agg4-100ge-1.cenic.net (137.164.11.29)  84.404 ms  84.339 ms dc-svl-agg8--svl-agg4-100ge-2.cenic.net (137.164.11.31)  80.624 ms
>  9  dc-svl-agg10--svl-agg8-300g.cenic.net (137.164.11.80)  81.499 ms *  76.697 ms
> 10  lon--cenic-100ge.cenic.net (137.164.3.103)  71.977 ms  70.990 ms  70.944 ms
> 11  eugn-p1-gw.nero.net (207.98.64.196)  80.457 ms  79.744 ms  77.812 ms
> 12  corv-p2-gw.nero.net (207.98.64.13)  83.734 ms  82.461 ms  79.748 ms
> 13  corv-car1-gw.nero.net (207.98.64.17)  70.789 ms  70.932 ms  75.366 ms
> 14  * * *
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  * * *
> 20  * * *
> 21  * * *
> 22  * * *
> 23  * * *
> 24  * * *
> 25  * * *
> 26  * * *
> 27  * * *
> 28  * * *
> 29  * * *
> 30  * * *
Comment 2 Thomas Deutschmann (RETIRED) gentoo-dev 2020-09-21 15:10:16 UTC
Created attachment 661789 [details]
wget.log
Comment 3 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-09-21 19:16:36 UTC
(In reply to Thomas Deutschmann from comment #2)
> Created attachment 661789 [details]
> wget.log

Can you do a tcp capture? I'm curious about the tcp window params for your slow link (receive and congestion window.)

In my test:

TCP segment sizes ranged from 1448 to 7420.
My TCP window was 61 segments.
My RTT is 54ms

If we round:
 1000ms in a second / 54ms ~= 20 complete RTTs in a second.
 61 inflights per RTT * 20 = 1220 segments per second.

When segments are 1448: 1220 * 1448 == 1766560 bytes / second (70mbps)
When segments are 7420: 1220 * 7420 == 9052400 bytes / second (13mbps)

Now these are all theoretical speeds (we don't actually sustain 61 inflghts consistently, rtt is not quite 54ms all the time, the connection does not sustain a particular segment size, etc.)

Find me on IRC and we can do a test. But in theory:

throughput = RTT * window_size * segment_size

I'm curious to know what the bottleneck is here; is the window size just small? is the kernel picking small segments (e.g. because big segments are lost on the link somewhere, causing a stall and impacting window fill?)
Comment 4 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-09-22 21:37:33 UTC
(In reply to Alec Warner from comment #3)
> (In reply to Thomas Deutschmann from comment #2)
> > Created attachment 661789 [details]
> > wget.log
> 
> Can you do a tcp capture? I'm curious about the tcp window params for your
> slow link (receive and congestion window.)
> 
> In my test:
> 
> TCP segment sizes ranged from 1448 to 7420.
> My TCP window was 61 segments.
> My RTT is 54ms
> 
> If we round:
>  1000ms in a second / 54ms ~= 20 complete RTTs in a second.
>  61 inflights per RTT * 20 = 1220 segments per second.
> 
> When segments are 1448: 1220 * 1448 == 1766560 bytes / second (70mbps)
> When segments are 7420: 1220 * 7420 == 9052400 bytes / second (13mbps)
> 
> Now these are all theoretical speeds (we don't actually sustain 61 inflghts
> consistently, rtt is not quite 54ms all the time, the connection does not
> sustain a particular segment size, etc.)
> 
> Find me on IRC and we can do a test. But in theory:
> 
> throughput = RTT * window_size * segment_size
> 
> I'm curious to know what the bottleneck is here; is the window size just
> small? is the kernel picking small segments (e.g. because big segments are
> lost on the link somewhere, causing a stall and impacting window fill?)

Testing today with whissi and others:
 - TCP window: 61
 - segment size: 1440
 - These two things result in the 'slow' path (< 1MB/s) due to unfill windows and longer RTTs for europeans.

The cause appears to "the internet is bad" and various links are congested. This in turn causes loss, which causes tcp stack to reduce segment sizes (to 1440, the nominal smallest segment size; I haven't seen segments smaller than this.) Loss at some links was 20-80%; which..I mean..you can't get a good speed over that kind of link using standard control algorithms.

FWIW I reported this upstream (to the OSL) but I don't expect a short term solution.

-A