Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 772140 - New rsync mirror: IEWC - 1Gbps+ @ JHB, South Africa (ZA)
Summary: New rsync mirror: IEWC - 1Gbps+ @ JHB, South Africa (ZA)
Status: RESOLVED FIXED
Alias: None
Product: Mirrors
Classification: Unclassified
Component: New Server (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Mirror Admins
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-22 10:36 UTC by Jaco Kroon
Modified: 2024-04-18 15:51 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jaco Kroon 2021-02-22 10:36:58 UTC
Please fill in the information below.

 Sponsor Name:  Interexcel World Connection (Pty) Ltd / Ultimate Linux Solutions (Pty) Ltd
     Location:  South Africa, JHB
    Bandwidth:  Varies on path, from 200Mbps to 10Gbps.
Admin Contact:  support@iewc.co.za, escalation jaco@uls.co.za

The mirror can be accessed via the following IP addresses:

IPv4: 154.73.32.1 and 154.73.32.2
IPv6: 2c0f:f720::1 and 2c0f:f720::2

There is a limit on concurrent connections [amount/no]: 30

Additional comments:

Take note:  two physical hosts.
Comment 1 Jaco Kroon 2022-09-05 10:55:58 UTC
PING.

Just an update, bandwidth his been significantly upgraded over the last year or so, and the lowest capacity path is currently at 1Gbps.  From ZA though majority of should traverse one of the 10G links.  Obviously some capacity is being used for other purposes, but as a general rule I believe the rsync protocol itself, or possibly disk access speed, should at this point in time be the bottleneck, not the bandwidth.
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-09-06 14:09:35 UTC
I can't seem to actually reach either of those address (via IPv4 at least).

I've got the DNS commits ready though.
Comment 3 Jaco Kroon 2023-09-06 21:07:30 UTC
(In reply to Sam James from comment #2)
> I can't seem to actually reach either of those address (via IPv4 at least).
> 
> I've got the DNS commits ready though.

Interesting.

From where?  And could you possibly provide some mtr?

For the most part this has not affected us too badly, but we have had one or two sources coming to us via Cogent that bumped into this:

https://www.itweb.co.za/content/DZQ58MV8wj5vzXy2

I did see a notification from one of the consortium partners to ourselves in the last week indicating that repairs should have commenced today on one of the two cables, but I've not seen confirmation that this has actually happened.
Comment 4 Matt Jolly gentoo-dev 2023-09-06 23:06:33 UTC
I can't establish connectivity on port 22 to either of the ipv4 addresses:

```
kangie@monolith ~ $ ssh 154.73.32.1 -vvvv
OpenSSH_9.4p1, OpenSSL 3.1.2 1 Aug 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo-security.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo-security.conf
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo.conf
debug2: resolve_canonicalize: hostname 154.73.32.1 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/kangie/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/kangie/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to 154.73.32.1 [154.73.32.1] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
^C
Time: 0h:00m:40s                                                                                              
kangie@monolith ~ $ nc -z -n -v 154.73.32.1 22

^C
Time: 0h:00m:13s                                                                                              
kangie@monolith ~ $ ssh 154.73.32.2 -vvvv         
OpenSSH_9.4p1, OpenSSL 3.1.2 1 Aug 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo-security.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo-security.conf
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo.conf
debug2: resolve_canonicalize: hostname 154.73.32.2 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/kangie/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/kangie/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to 154.73.32.2 [154.73.32.2] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48

^C
Time: 0h:00m:25s
```

IPv6 seems fine:

```
kangie@monolith ~ $ ssh 2c0f:f720::1 -vvvv
OpenSSH_9.4p1, OpenSSL 3.1.2 1 Aug 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo-security.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo-security.conf
debug3: /etc/ssh/ssh_config line 47: Including file /etc/ssh/ssh_config.d/9999999gentoo.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/9999999gentoo.conf
debug2: resolve_canonicalize: hostname 2c0f:f720::1 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/kangie/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/kangie/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to 2c0f:f720::1 [2c0f:f720::1] port 22.
debug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48
debug1: Connection established.
. . .
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.3
debug1: compat_banner: match: OpenSSH_9.3 pat OpenSSH* compat 0x04000000
```
Comment 5 Matt Jolly gentoo-dev 2023-09-06 23:15:38 UTC
Hosts are contactable via ipv4 (pings), routing seems fine. Is sshd listening on ipv4? If so, is there a firewall in the way?
Comment 6 Matt Jolly gentoo-dev 2023-09-06 23:19:13 UTC
forgot to paste my mtr results:

```
kangie@monolith ~ $ mtr -c 3 -4 -r -w  154.73.32.1
Start: 2023-09-07T09:12:16+1000
HOST: monolith                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                                        0.0%     3    0.1   0.2   0.1   0.2   0.1
  2.|-- loop11918360.bng.bne.aussiebb.net               0.0%     3    7.6   8.1   6.5  10.3   1.9
  3.|-- 10.241.1.52                                     0.0%     3    5.8   6.3   5.8   7.3   0.9
  4.|-- be30.core2.equinix-sg1.sin.aussiebb.net         0.0%     3  109.2 108.9 108.7 109.2   0.3
  5.|-- ce-0-12-0-2.r00.sngpsi07.sg.bb.gin.ntt.net      0.0%     3  108.9 109.3 108.8 110.2   0.8
  6.|-- ???                                            100.0     3    0.0   0.0   0.0   0.0   0.0
  7.|-- ae-8.r21.vienat02.at.bb.gin.ntt.net             0.0%     3  366.3 364.7 363.5 366.3   1.4
  8.|-- ae-12.r20.amstnl07.nl.bb.gin.ntt.net            0.0%     3  279.0 278.9 278.7 279.1   0.2
  9.|-- ae-15.r20.londen12.uk.bb.gin.ntt.net           33.3%     3  281.1 280.9 280.8 281.1   0.2
 10.|-- ae-0.a02.londen12.uk.bb.gin.ntt.net             0.0%     3  294.5 284.7 278.9 294.5   8.5
 11.|-- dimensiondata-0.r02.londen03.uk.bb.gin.ntt.net  0.0%     3  274.7 275.4 273.6 277.9   2.3
 12.|-- core1b-pkl-ten-ge-0-6-0-1.ip.ddii.network       0.0%     3  452.7 452.6 452.6 452.7   0.1
 13.|-- mi-za-pkl-br7-be-302.ip.ddii.network            0.0%     3  447.3 447.3 447.1 447.5   0.2
 14.|-- 197.96.224.9                                   66.7%     3  443.1 443.1 443.1 443.1   0.0
 15.|-- 154.73.32.8                                     0.0%     3  442.0 442.4 442.0 442.7   0.4
 16.|-- kerberos.iewc.co.za                             0.0%     3  446.9 446.8 446.7 446.9   0.1
Time: 0h:00m:19s                                                                                              
kangie@monolith ~ $ mtr -c 3 -6 -r -w 2c0f:f720::1
Start: 2023-09-07T09:12:51+1000
HOST: monolith                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- <snip-my-gateway-static-ip6>.ip6.aussiebb.net  0.0%     3    0.1   0.1   0.1   0.1   0.0
  2.|-- ???                                                   100.0     3    0.0   0.0   0.0   0.0   0.0
  3.|-- be500-636.cfl5.nextdc-b1.bne.aussiebb.net              0.0%     3    8.2   6.8   6.1   8.2   1.2
  4.|-- 2403-5800-1700-5b--1.ip6.aussiebb.net                  0.0%     3    9.4   7.1   5.8   9.4   2.0
  5.|-- 2403-5800-1701-1b--2.ip6.aussiebb.net                  0.0%     3    5.8   5.7   5.6   5.8   0.1
  6.|-- ???                                                   100.0     3    0.0   0.0   0.0   0.0   0.0
  7.|-- ce-0-12-0-2.r00.sngpsi07.sg.bb.gin.ntt.net             0.0%     3  110.2 109.7 108.8 110.2   0.8
  8.|-- ???                                                   100.0     3    0.0   0.0   0.0   0.0   0.0
  9.|-- ae-8.r21.vienat02.at.bb.gin.ntt.net                    0.0%     3  354.1 357.2 352.8 364.7   6.5
 10.|-- ae-12.r20.amstnl07.nl.bb.gin.ntt.net                   0.0%     3  280.4 280.9 278.9 283.2   2.2
 11.|-- ae-15.r20.londen12.uk.bb.gin.ntt.net                   0.0%     3  280.5 281.2 280.5 281.6   0.6
 12.|-- ae-0.a02.londen12.uk.bb.gin.ntt.net                    0.0%     3  287.0 293.4 280.3 313.0  17.3
 13.|-- xe-1-3-2-0.a02.londen12.uk.ce.gin.ntt.net              0.0%     3  282.8 282.6 281.2 283.7   1.3
 14.|-- core1a-ctn-gi0-1-0-4-323.ipv6.isnet.net                0.0%     3  445.2 447.3 445.2 451.5   3.6
 15.|-- 2c0f:fc00:0:5::31                                      0.0%     3  447.9 447.5 447.3 447.9   0.3
 16.|-- 2c0f:fc10:2:19::1                                      0.0%     3  446.1 446.3 446.1 446.5   0.2
 17.|-- ???                                                   100.0     3    0.0   0.0   0.0   0.0   0.0
 18.|-- 2c0f:fc00:5003:8::857:2                                0.0%     3  446.9 446.9 446.8 446.9   0.1
 19.|-- kerberos.iewc.co.za                                    0.0%     3  446.7 447.3 446.7 447.8   0.5
Time: 0h:00m:18s                                                                                              
```
Comment 7 Jaco Kroon 2023-09-07 13:10:14 UTC
(In reply to Matt Jolly from comment #5)
> Hosts are contactable via ipv4 (pings), routing seems fine. Is sshd
> listening on ipv4? If so, is there a firewall in the way?

We're talking rsync here, surely ssh doesn't need to be accessible?  In fact, the fact that it is reachable via IPv6 is a firewall oversight which we'll be looking to fix.

We need port 873 (rsync) open to the world for this.  There was in fact an error on the firewall where this was restricted to our own ranges and an off-net monitoring server (which also happens to be from where I tested ...)

Seeing 154.73.32.[78] in the traceroutes indicates that routing is fine, so that's at least a small sigh of relieve.

Please retest and confirm all is in order now.
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-09-07 13:32:05 UTC
Looking good. On it...
Comment 9 Larry the Git Cow gentoo-dev 2023-09-07 13:34:48 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/data/api.git/commit/?id=71df092fb8792f1c423b9c68c144469528e82a37

commit 71df092fb8792f1c423b9c68c144469528e82a37
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-09-07 13:34:15 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-09-07 13:34:15 +0000

    mirrors: rsync: add Interexcel World Connection mirror in JHB, South Africa
    
    Many thanks to Jaco for bringing us our first mirror in ZA!
    
    Bug: https://bugs.gentoo.org/772140
    Signed-off-by: Sam James <sam@gentoo.org>

 files/mirrors/rsync.xml | 10 ++++++++++
 1 file changed, 10 insertions(+)
Comment 10 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-09-07 13:58:01 UTC
DNS done (rsync1.za), all done now I think. Thanks!

Please file a new bug if willing to be a distfile mirror as well.