Anycast

REANNZ conveniently provides us with BGP peerings and address space which we can announce down to single IPs.

Anycast prefixes

• 163.7.134.112/29
• 2404:138:4000::/48

Anycast groups

• 163.7.134.112 and 2404:138:4000:: for mirror.fsmg.org.nz (mirrors, SSH push)
• 163.7.134.113 and 2404:138:4000::1 for fsmg.org.nz (website)

Linknets

The fq_codel qdisc is used to try and ensure fair distribution of bandwidth on the interface among all connected clients.

On wlglam:

iface enp4s0f0 inet static
    address 163.7.134.123/31
    gateway 163.7.134.122
    post-up ip -6 addr add 2404:138:134:122::2/112 dev enp4s0f0 || true
    post-up ip -6 route replace default via 2404:138:134:122::1 || true
    post-up tc qdisc add dev enp4s0f0 root fq_codel || true

On hlzmel:

auto enp4s0f0
iface enp4s0f0 inet static
    address 163.7.134.121/31
    gateway 163.7.134.120
    post-up ip -6 addr add 2404:138:134:120::2/112 dev enp4s0f0 || true
    post-up ip -6 route replace default via 2404:138:134:120::1 || true
    post-up tc qdisc add dev enp4s0f0 root sfq || true

On both nodes (loopback for anycast):

iface br-lo inet static
    bridge-ports none
    address 163.7.134.112/32
    post-up ip addr add 163.7.134.113/32 dev br-lo || true
    post-up ip addr add 2404:138:4000::/128 dev br-lo || true
    post-up ip addr add 2404:138:4000::1/128 dev br-lo || true

Routing daemon

We’re using ExaBGP to advertise our anycast routes to REANNZ. There is a general plan to move to beka with the support of Sam.

ExaBGP is configured to speak both IPv4 and IPv6:

neighbor 163.7.134.120 {
    router-id 163.7.134.121;
    local-address 163.7.134.121;
    local-as 65200;
    peer-as 38022;
    graceful-restart;

    process health4 {
        run /opt/fsmg/bgp/health /opt/fsmg/bgp/ipv4.yaml;
    }
}

neighbor 2404:138:134:120::1 {
    router-id 163.7.134.121;
    local-address 2404:138:134:120::2;
    local-as 65200;
    peer-as 38022;
    graceful-restart;

    process health6 {
        run /opt/fsmg/bgp/health /opt/fsmg/bgp/ipv6.yaml;
    }
}

Health checks

Healthcheck daemons are used to check the /.fsmg/stats/healthcheck path of the mirror. If this returns anything other than an HTTP 200 status code and the text “online” then the node will be considered “in maintenance” and the routes will be removed.

The healthcheck daemons also write their state to /run/exabgp/state-ip4 and /run/exabgp/state-ip6 to allow Ansible to do rolling restarts of service impacting daemons. These files will contain “online” if ExaBGP is currently expected to announce the node.

The health check scripts are in git at https://gitlab.com/fsmg/healthchecks.