High Availability Without a Floating IP: Multi-A DNS, Three Traefiks and a Kill Drill

"If node1 dies, node2 takes over." Everyone on the team believed that about our three-node cluster. When I actually went through the live configuration, every part of the sentence was false. DNS pointed only at node1. Most services were pinned to node1. node2 had an orphaned Traefik with no route files. Two databases would have come back empty if Swarm had ever moved them. This post covers how I made that sentence true on a $0 budget, on VPS nodes that can't share an IP, and how I proved it by turning off a production node on purpose.

Why a floating IP was off the table

The textbook answer for "keep one public IP alive across machines" is keepalived/VRRP with a virtual IP. That needs a shared Layer-2 network. Our three nodes are VPS instances, each with a routed /32 public address and no shared broadcast domain. VRRP has nothing to float across.

The provider does sell movable "failover IPs" you can reassign through an API, but only on its dedicated-server and public-cloud ranges. The VPS range doesn't offer them. The nodes sit in the same datacenter (~0.5 ms apart), so the cluster itself was healthy. The front door was the problem.

I wrote down every option, mostly so nobody would spend a day re-deciding it later:

• •A managed, health-checked load balancer at the edge (e.g. Cloudflare LB). Failover in seconds, TLS and DDoS protection at the edge, and it hides the origin IPs. About US$5/month plus a DNS migration. Technically the best option.
• •DIY DNS failover. Low TTL, plus a health checker that rewrites A records through the DNS provider's API. Failover in 1–3 minutes. The catch: the watcher has to run off the cluster, or it dies with the node it's supposed to be watching.
• •Re-platform onto cloud instances with a real floating IP and run keepalived in unicast mode. Sub-30-second failover, but it means rebuilding all three nodes. That's a project, not a change.

The constraints were explicit: $0, no third-party front door, keep the existing DNS provider and Dokploy. That forced an honest admission. Failover would be DNS-bound and imperfect, and that's a platform limit, not a tuning problem.

What we built: four dumb layers

There's no clever coordinator. Resilience comes from four independent layers, each doing one boring job:

1. Orchestration: Docker Swarm with three managers. They tolerate one failure.

2. Shared storage: GlusterFS replicated three ways, so the same files are visible on every node.

3. Ingress: Traefik on every node, terminating HTTPS and routing by service name over the overlay network, so any node can reach an app wherever it runs.

4. Entry: multi-A DNS. Every hostname returns all three node IPs, and the browser does the failover. Happy Eyeballs means that if the first IP refuses or times out, the client tries the next.

The fourth layer's honest limit: it has no health awareness. A node that's up but returning errors still gets roughly a third of first connections. It's clean and fast only for a hard-down node. I took that trade knowingly, as the $0 choice.

Three Traefiks, one certificate issuer

Running Traefik on three nodes is easy. Running three ACME clients is not. With HTTP-01 challenges and multi-A DNS, Let's Encrypt's validation request lands on whichever node it resolves to, and three solvers writing one shared acme.json will race each other.

So node1 stays the only issuer, just as a single-node setup would be. Every minute a sync job publishes a standby bundle to the shared volume: route files with the ACME resolver stripped, certificates extracted to PEM files, a TLS file-provider config, and an ACME-challenge passthrough so any node forwards /.well-known/acme-challenge/ to node1. node2 and node3 run a standby Traefik with no ACME at all. They serve the shared certs by SNI. No API tokens and no race.

The bug that hid for two months

Standbys served random 404s for a few seconds every minute, and occasionally the default certificate. The cause was embarrassingly simple. The sync rewrote every file in place every minute (open-for-write truncates first), and the pull on the standbys ran in the same cron second. Standbys sometimes copied half-written certificates. The logs held about 21,800 failed to find any PEM data errors per standby. Because every file's mtime changed, Traefik also reloaded its whole config every minute.

The v2 fix is the classic publish pattern:

• •Build the bundle in a temp directory and refuse to publish if any cert fails openssl x509.
• •Publish only files whose content changed, via temp file plus mv: certs first, then routes, then an atomic MANIFEST of SHA-256 hashes.
• •On the standbys, wait 30 seconds, snapshot, install only if the snapshot matches the manifest, and use rsync -c so unchanged files aren't touched and Traefik doesn't reload.

After the fix: 480 requests through both standbys across five minute boundaries, zero failures.

The control plane stays warm, not floating

I deliberately did not make the Dokploy panel float. It bind-mounts its own state and the local Docker socket, builds images locally, and isn't designed to run active on two hosts. Auto-floating it would mean putting the panel's database on a network filesystem and inviting split-brain. Instead the panel database is dumped every 6 hours and its config directory is mirrored hourly to the shared volume. A promote-control-plane script waits on the other nodes. It's dry-run by default, and it has a guard that refuses to run while node1 still answers ping.

The result: running apps fail over automatically, and the panel is a minutes-scale manual promotion. That's the standard, sane trade-off for a single-panel PaaS.

The drill: proving it

A design document proves nothing. I wrote a runbook and ran it against production.

§1 The gate: is multi-A DNS actually live?

dig +short @<authoritative-ns> app.example.com A   # want all three IPs

Query the authoritative nameserver, not your resolver. Our records passed, but the TTL was 3600 seconds, not 60. With an hour-long TTL, a resolver that cached the old answer keeps sending people to a dead node, and the test looks broken for reasons that have nothing to do with the cluster. Lower the TTL and wait out the old one before the drill.

§2 Baseline, including a data fingerprint

Record where every task runs, check Gluster heal status (you want zero pending entries), and count the tables in every database. Most important of all, confirm every locally built image exists on all three nodes. The very first drill failed because a service rescheduled to a node that didn't have its image.

§3 Per-node ingress, non-disruptive

curl -s -o /dev/null --resolve "app.example.com:443:$NODE_IP" \
  -w '%{http_code} tls=%{ssl_verify_result}\n' https://app.example.com/

--resolve pins curl to one node while sending the real Host header and SNI. That separates "this node's Traefik and certs are fine" from "DNS is fine". The pass condition: every app on every node returns 200 with tls=0.

§4 An observer that isn't on the victim

A curl loop runs on my laptop every 5 seconds and logs the status code, the last octet of the IP it actually reached, and the response time. An observer running on the node you're about to kill tells you nothing.

§5–§6 Drains

Draining node3 moved two databases and Metabase to node2 with data intact (the fingerprint matched), and the apps stayed at HTTP 200 throughout. Rescheduling takes about 30–60 seconds. The worst blip in the observer log was one app returning 502 for 45 seconds while its moved container cold-started. That counts as a pass. Draining node1 behaved exactly as designed: the four app domains recovered within a minute, and the panel stayed down for 7 minutes 44 seconds, which is the warm-standby trade-off doing its job.

A warning I'd underline twice: a drain is an optimistic test. It leaves the node's IP and its Traefik alive, so the front door keeps answering. It proves rescheduling, not ingress failover.

§7 Kill the node

Only after the gate passes. I simulated a dead VPS without touching the provider console: on the victim I armed an auto-restore first (systemd-run --on-active=14min running a restore script), then blackholed ports 80/443 with an iptables DROP rule and stopped Docker. DROP, not REJECT, so clients see a timeout exactly like a vanished machine, while SSH stays up so I can watch. The failure lasted about seven minutes; everything public stayed reachable through the other two nodes, and the restore timer brought the node back on its own. A hard power-off from the provider's console is the harsher version of this test, and the one to run next, with console access ready because you lose SSH. Test in a real browser too.

The failure the drill didn't find

A month later, a rolling reboot for kernel updates took every site down for about 22 minutes. Traefik ran as a plain container on the Swarm overlay network. At boot the network attach timed out and Docker never retried it. The fix was a small systemd unit that waits for Swarm, starts Traefik with retries, restarts it once, and re-runs on Docker restarts. The drill had tested a node dying. It hadn't tested a node coming back.

Takeaways

• •Know what your provider can't do before you design. No shared L2 means no VRRP, whatever the blog posts say.
• •Multi-A DNS plus Happy Eyeballs is real, free failover for hard-down nodes, and useless for broken-but-up ones. Be honest about which one you bought.
• •One certificate issuer, many servers. Publish atomically with a manifest, never in place.
• •Gate your drill on DNS, observe from outside, and don't mistake a drain for a death.
• •Test the reboot too. Recovery paths fail more quietly than failure paths.