₹200 free credit when you verify after signup, plus your first recharge doubled. Terms
Engineering blog

Withdraw the anycast route when the service dies: health checks with hysteresis for BIRD 2 and VyOS

By Geet Mangal, Founder and CTO · Published · Updated · 9 min read

Anycast is easy to switch on and easy to get wrong. You announce the same prefix from several sites, and BGP sends each client to the nearest one. That is fine until the service behind one announcement stops answering and the router keeps announcing anyway. BGP has no idea that unbound crashed or nginx is returning 502. As long as the session is up and the route exists, that site keeps drawing its share of traffic into a dead end.

The fix is old and well known: tie the announcement to a health check. anycast-healthcheck.py is our version of it, written for the two routing stacks we spend the most time with, BIRD 2 and VyOS. (BookMyHost's own edge runs VyOS behind Cloudflare Magic Transit, which is why VyOS gets first-class support rather than a hook example.) This post covers the design, the timing you can expect from a given config, and how it fails.

What the script does

Every interval seconds it runs each configured check in turn:

  • tcp: a plain connect() to host and port.
  • http: GET a URL, check the status against expect_status, optionally match body_regex, optionally send a Host header, with TLS verification on or off.
  • dns: a hand-built UDP query (RD set, random transaction ID). It checks the reply's ID and QR bit, then compares the rcode with expect_rcode (default 0, NOERROR).

A round is healthy only if every check passes. A small state machine turns rounds into a decision, and a backend applies it:

  • bird: birdc disable <protocol> / birdc enable <protocol>, or include-file mode with birdc configure.
  • vyos: delete / set protocols static route <prefix> blackhole, then commit and save through vbash.
  • exec: your own --exec-up / --exec-down shell commands, with ANYCAST_PREFIX and ANYCAST_ACTION in the environment.

It uses only the standard library, so there is no virtualenv to keep alive on a router or a minimal VM.

Design decisions

Hysteresis instead of a single check

The state machine is the important part, and it is pure Python with no I/O, so the tests cover it directly:

def feed(self, healthy: bool) -> Optional[str]:
    if healthy:
        self.consecutive_ok += 1
        self.consecutive_fail = 0
        if self.state == self.WITHDRAWN and self.consecutive_ok >= self.ok_threshold:
            self.state = self.ANNOUNCED
            return "announce"
    else:
        self.consecutive_fail += 1
        self.consecutive_ok = 0
        if self.state == self.ANNOUNCED and self.consecutive_fail >= self.fail_threshold:
            self.state = self.WITHDRAWN
            return "withdraw"
    return None

You need fail_threshold bad rounds in a row to withdraw and ok_threshold good rounds in a row to announce. A single good round resets the failure count, and a single bad round resets the success count. The example config sets ok_threshold (5) higher than fail_threshold (3) on purpose. Pulling a sick site out quickly is cheap. Putting a flapping site back too early sends real users to it and invites route-flap damping upstream.

The router does the routing

In BIRD the anycast address sits in a static protocol, and the BGP export filter accepts only routes from that protocol:

protocol static anycast_static {
    ipv4;
    route 203.0.113.53/32 blackhole;   # the loopback-bound anycast address
}

filter export_anycast {
    if proto = "anycast_static" then accept;
    reject;
}

protocol bgp upstream {
    local as 65001;
    neighbor 192.0.2.1 as 65000;
    ipv4 { export filter export_anycast; import none; };
}

The route is a blackhole only in the routing table. The service itself listens on the address bound to a loopback interface, so local delivery still works. Disabling anycast_static removes the route, and BIRD withdraws it from the BGP session. The script never touches the BGP session itself, so a withdraw does not reset your other prefixes.

Include-file mode (bird_state_file) suits setups where disabling a protocol is not acceptable, for example a static protocol that also holds other routes. The script atomically rewrites a file that either contains route <prefix> blackhole; or only comments, then runs birdc configure. The static protocol pulls that file in with include.

On VyOS the same idea uses a static blackhole route that BGP redistributes. You are responsible for a route-map so that only this prefix is redistributed, not every static route on the box.

State that survives restarts

After every round the script writes its state (announced or withdrawn, plus both counters) atomically to state_file. On start it restores that state when start_state is auto, which is the default. With --enforce-on-start, which the shipped systemd unit uses, it then pushes the restored state to the router. A restart therefore converges the router to whatever the script last decided, instead of assuming everything is announced.

Timing, as a calculation

We have not published a measured failover number for this script, and you should not trust anyone's number for your network anyway. What the code does guarantee can be worked out from the config.

Checks run one after another inside a round. After a round the loop sleeps for max(0, interval - elapsed). So the round period is:

P = max(interval, sum of check durations in that round)

The withdraw decision is made at the end of the fail_threshold-th failing round in a row. Depending on where in the cycle the service dies, that lands between roughly (fail_threshold - 1) × P and fail_threshold × P after the failure.

Take the example config: interval: 2, fail_threshold: 3, ok_threshold: 5, with three checks (DNS with timeout: 1.5, HTTP and TCP at the default timeout of 2.0).

  • The service process dies and the port refuses connections. Each check fails within milliseconds, so P is about 2 s. Withdraw happens about 4 to 6 s after the failure.
  • The service hangs or packets are dropped, and every check runs to its timeout. One round takes up to 1.5 + 2.0 + 2.0 = 5.5 s, so P is about 5.5 s. Withdraw happens about 11 to 16.5 s after the failure.
  • Recovery. With fast passing checks, re-announcing needs 5 good rounds, about 10 s after the service is healthy again.

Then add the backend command (birdc returns quickly; a VyOS commit plus save is a heavier operation, so time yours) and BGP propagation to your upstreams, which depends on them and not on this script. If the hanging case matters to you, shorten the check timeouts before you shorten the interval.

Walkthrough

git clone https://github.com/geetmangal/bmh-infra-scripts.git
cd bmh-infra-scripts
./bgp/anycast-healthcheck.py --print-example-config | sudo tee /etc/anycast-healthcheck.yaml
sudo chmod 0600 /etc/anycast-healthcheck.yaml

Edit the prefix, the checks and the backend, then try a round that touches nothing:

./bgp/anycast-healthcheck.py -c /etc/anycast-healthcheck.yaml --check-once --dry-run -v

The output is one line per check (OK/FAIL, latency, detail) followed by healthy=... announced=... transition=.... Add --json for a machine-readable summary that includes both counters.

Using custom hooks instead of a built-in backend:

./bgp/anycast-healthcheck.py -c my.yaml --backend exec \
  --exec-up "birdc enable anycast4" --exec-down "birdc disable anycast4"

Command-line --prefix, --interval, --state-file, --metrics-file and --backend override the file, which is handy while testing.

Running it under systemd

The repo ships a unit (also printed by --print-systemd-unit):

[Service]
Type=simple
ExecStart=/usr/local/bin/anycast-healthcheck.py -c /etc/anycast-healthcheck.yaml --enforce-on-start
Restart=always
RestartSec=2
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/run /etc/bird /var/lib/node_exporter/textfile

sudo install -m 0755 bgp/anycast-healthcheck.py /usr/local/bin/
sudo install -m 0644 bgp/anycast-healthcheck.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now anycast-healthcheck
journalctl -u anycast-healthcheck -f

ProtectSystem=strict makes the filesystem read-only apart from the listed paths. Every path in ReadWritePaths has to exist, so create /var/lib/node_exporter/textfile first, or remove it from the unit if you do not use metrics. Otherwise the unit fails to start. SIGTERM is handled cleanly: the loop exits after the current round and does not change the announcement.

This is a daemon by design, not a cron job. A one-minute cron granularity would make any threshold meaningless.

Metrics

If metrics_file is set, every round writes a Prometheus textfile (atomically, through a temp file and rename) for node_exporter's textfile collector: anycast_healthy, anycast_announced, anycast_last_change_timestamp_seconds, anycast_backend_errors_total, and per check anycast_check_up and anycast_check_latency_ms. Alert on anycast_announced == 0 for longer than you expect a maintenance to take. Alert on any increase in anycast_backend_errors_total, for the reason in the next section.

Failure modes

A backend command fails during a transition. The state machine has already flipped, so the script believes the prefix is withdrawn even though birdc or the VyOS commit failed. The error is logged and counted in anycast_backend_errors_total, but nothing retries until the next transition. The fix is operational: alert on that counter and restart the service, and --enforce-on-start will push the intended state again.

Every site fails at once. If all your instances check something they share (an upstream resolver, a database, a public URL served through the anycast address itself), one outage of that dependency withdraws the prefix everywhere. The script has no "last site standing" guard. Point checks at the local instance on its local address.

--check-once is not read-only. A one-shot run loads the same state file, feeds the state machine and calls the backend if a threshold is crossed. For Nagios or Zabbix, add --dry-run (no backend calls, no state or metrics writes) or give the check its own --state-file. Exit codes are 0 healthy, 1 unhealthy, 2 config error and 3 backend failure.

The controller dies. The router keeps its last state. If the whole host dies, the BGP session drops and the route goes with it, which is what you want. If only the script dies while withdrawn, the site stays out until it restarts. Restart=always covers that.

The config is code. With the exec backend, anyone who can edit the config can run commands as root. Keep it owned by root with mode 0600.

Skipped TLS verification. verify_tls: false is fine against https://<local-address>/healthz with a certificate issued for your public name. Do not copy it to checks that cross a network you do not control.

We sell NVMe KVM VPS in India at BookMyHost. A VM with a BGP session is a reasonable place to try this before you put it on a router.

Get the script

MIT-licensed, standard library only, with tests for the state machine and the DNS packet code. If you run it against FRR or GoBGP through the exec backend, a pull request with your hook would be welcome.

Script on GitHub · Repository

FAQ

Questions on this post

How quickly does anycast-healthcheck.py withdraw a prefix?

The decision time follows from the config. The prefix is withdrawn after fail_threshold failing rounds in a row, and a round runs every interval seconds or takes as long as the checks themselves, whichever is longer. With interval 2 and fail_threshold 3 and fast-failing checks, that is about 4 to 6 seconds, plus the router command and BGP propagation.

Why use hysteresis instead of withdrawing on the first failed check?

One lost UDP packet or one slow HTTP response should not move traffic. Requiring several failures in a row before a withdraw, and several successes in a row before a re-announce, stops the prefix from flapping, which upstreams may penalise with route-flap damping.

Does the script need PyYAML or any pip packages?

No. It uses only the Python 3.8+ standard library. The config is JSON or a small YAML subset (maps, lists of maps, scalars and inline lists) parsed by the script itself.

How does the BIRD backend withdraw the route?

By default it runs birdc disable or birdc enable on a static protocol that holds the anycast route, and the BGP export filter only exports routes from that protocol. In include-file mode it rewrites a file that does or does not contain the route and runs birdc configure.

What happens if every site fails its health check at the same time?

Every site withdraws and the prefix disappears from the internet. The script has no last-site-standing guard, so make your checks test only the local instance and not a shared dependency.

Run it on hourly billing

NVMe cloud servers in India, resize in place, GST invoice either way. Every new account starts with ₹200 of free credit.

Build a server
WhatsApp