Anycast¶
REANNZ conveniently provides us with BGP peerings and address space which we can announce down to single IPs.
Anycast prefixes¶
163.7.134.112/29
2404:138:4000::/48
Anycast groups¶
163.7.134.112
and2404:138:4000::
formirror.fsmg.org.nz
(mirrors, SSH push)163.7.134.113
and2404:138:4000::1
forfsmg.org.nz
(website)
Linknets¶
The fq_codel
qdisc is used to try and ensure fair distribution of bandwidth on the interface among all connected clients.
On wlglam
:
iface enp4s0f0 inet static
address 163.7.134.123/31
gateway 163.7.134.122
post-up ip -6 addr add 2404:138:134:122::2/112 dev enp4s0f0 || true
post-up ip -6 route replace default via 2404:138:134:122::1 || true
post-up tc qdisc add dev enp4s0f0 root fq_codel || true
On hlzmel
:
auto enp4s0f0
iface enp4s0f0 inet static
address 163.7.134.121/31
gateway 163.7.134.120
post-up ip -6 addr add 2404:138:134:120::2/112 dev enp4s0f0 || true
post-up ip -6 route replace default via 2404:138:134:120::1 || true
post-up tc qdisc add dev enp4s0f0 root sfq || true
On both nodes (loopback for anycast):
iface br-lo inet static
bridge-ports none
address 163.7.134.112/32
post-up ip addr add 163.7.134.113/32 dev br-lo || true
post-up ip addr add 2404:138:4000::/128 dev br-lo || true
post-up ip addr add 2404:138:4000::1/128 dev br-lo || true
Routing daemon¶
We’re using ExaBGP to advertise our anycast routes to REANNZ. There is a general plan to move to beka with the support of Sam.
ExaBGP is configured to speak both IPv4 and IPv6:
neighbor 163.7.134.120 {
router-id 163.7.134.121;
local-address 163.7.134.121;
local-as 65200;
peer-as 38022;
graceful-restart;
process health4 {
run /opt/fsmg/bgp/health /opt/fsmg/bgp/ipv4.yaml;
}
}
neighbor 2404:138:134:120::1 {
router-id 163.7.134.121;
local-address 2404:138:134:120::2;
local-as 65200;
peer-as 38022;
graceful-restart;
process health6 {
run /opt/fsmg/bgp/health /opt/fsmg/bgp/ipv6.yaml;
}
}
Health checks¶
Healthcheck daemons are used to check the /.fsmg/stats/healthcheck
path of the mirror. If this returns anything other than an HTTP 200 status code and the text “online” then the node will be considered “in maintenance” and the routes will be removed.
The healthcheck daemons also write their state to /run/exabgp/state-ip4
and /run/exabgp/state-ip6
to allow Ansible to do rolling restarts of service impacting daemons. These files will contain “online” if ExaBGP is currently expected to announce the node.
The health check scripts are in git at https://gitlab.com/fsmg/healthchecks.