k8s on bare-metal - Part 14 : Multi-datacenter
Part fourteen of the bare-metal series. Thirteen parts assumed one building; this one adds a second. The appealing mistake is to melt the two sites into one big cluster, and it is a trap, because etcd cannot keep quorum across a WAN. The discipline is to keep them as two autonomous clusters and connect them: a KubeSpan WireGuard mesh, Cilium ClusterMesh, state replicated not stretched, and users routed to the nearest healthy site, so losing a datacenter is an incident, not an outage.
Thirteen parts, one building. Every design so far, the control plane, the storage, the mesh, the scheduling, has quietly assumed the machines are close enough to treat as one place. A second datacenter breaks that assumption, and it breaks it in a way that is easy to get catastrophically wrong. The appealing mistake is to make the two sites disappear into one big cluster; the discipline is to keep them as two and connect them. This part is about doing the second thing well: spanning sites without a shared fate, so losing a datacenter is an incident, not an outage. It leans on almost everything before it, Cluster API to build the clusters, Cilium to mesh them, Ceph and the databases to replicate the state, Cloudflare and BGP to route around a failure, and OpenBao to keep trust alive through a partition.
What happens with two
A single datacenter has a comforting property: everything in it can fail together or not at all, so the platform never has to reason about half of itself being alive. A second site removes that comfort and replaces it with a question that runs through every layer, what happens when the link between them dies. Answer it well and two datacenters make the platform more resilient than one. Answer it by pretending the two sites are one, and you have simply built a bigger single point of failure. So the whole part is really about resisting one tempting shortcut.
The last single-site layer. Everything through the accelerators assumed one building; here the platform learns to live in two.
The trap, and the fork
The decision the whole part turns on is whether there is one cluster or two, and it is tempting to choose one. Talos KubeSpan will happily connect nodes in both datacenters into a single flat network, and a single cluster is a simpler thing to reason about. The problem is etcd. The control plane's consensus needs low, stable latency to keep quorum, and a WAN is neither; stretch etcd across sites and an ordinary link flap can stall the whole cluster or, worse, split its brain. Unless the two datacenters are a metro pair a few milliseconds apart, the answer is two clusters, each with its own quorum. Toggle the two models and watch what cutting the WAN does to each:
Each cluster keeps its own quorum and keeps running; only cross-site failover pauses until the link returns. Two datacenters, two failure domains, no shared fate.
A fleet of autonomous clusters
Two clusters is not two of everything to operate, because the fleet is managed as one. Cluster API from part one provisions each datacenter's cluster the same way it provisioned the first, and once a cluster exists it registers itself so Argo CD can see it. Then a single ApplicationSet with a cluster generator fans the entire platform layer, the CNI config, the operators, the policies, out to every cluster in the fleet at once. Adding a third datacenter later is provisioning one more cluster and letting the generator find it; the platform definition does not change. This is the multi-primary shape, every site a peer, rebuilt on tooling the earlier parts already run:
# One ApplicationSet, every cluster in the fleet. CAPI registers each site's
# cluster as a secret; the generator fans the platform layer out to all of them.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: platform, namespace: argocd }
spec:
generators:
- clusters: {} # every cluster CAPI has registered
template:
metadata: { name: 'platform-{{name}}' }
spec:
destination: { server: '{{server}}' }
source:
repoURL: https://git.example.com/platform
path: base # the same platform layer, every siteThe fleet mechanics this builds on: CAPI provisioning clusters as cattle, and Argo CD deploying to all of them at once.
KubeSpan and ClusterMesh
With two autonomous clusters, they still need to act like one platform, and that is two layers. The transport is KubeSpan, Talos's built-in WireGuard mesh, which connects the sites over whatever link exists, encrypted, with no hand-built tunnels and no dependence on the underlay being private. On top of it, Cilium ClusterMesh makes services global: a Service marked global is reachable from either cluster, and Cilium prefers the local backend, crossing to the other site only when the local one is gone. That local-affinity default matters, because it means the WAN carries failover traffic, not everyday traffic. This is the all-Cilium answer to the multi-cluster problem a separate writeup solved with Istio:
# Talos KubeSpan: a WireGuard mesh across sites, no manual tunnels.
machine:
network:
kubespan: { enabled: true } # nodes find each other via a discovery service
---
# Cilium ClusterMesh makes a service global: prefer the local backend, fail
# over to the other site only when the local one is gone.
apiVersion: v1
kind: Service
metadata:
name: api
annotations:
service.cilium.io/global: "true"
service.cilium.io/affinity: "local" # stay in-site until you cannotThe multi-cluster problem solved the other way: Istio's multi-primary and primary-remote models, and how Cilium ClusterMesh differs.
State you replicate, not stretch
Networking across sites is the easy half; state is the hard half, and the rule is blunt: you replicate it, you do not stretch it. Ceph and a database both depend on low-latency consensus among their members, exactly what a WAN does not provide, so each site runs its own. Ceph replicates asynchronously between sites, RGW multi-site for objects and RBD mirroring for volumes; a database keeps its primary in one site and an async standby in the other, a CloudNativePG replica cluster streaming over the mesh. Asynchronous is the deliberate choice, because synchronous replication across a WAN would make every write hostage to the link. The cost is honest and bounded, a small recovery-point window and a clear owner per dataset:
# The database's second site is a replica cluster, streaming asynchronously
# from the primary over the mesh, ready to promote on failover.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: checkout-db, namespace: payments } # in datacenter B
spec:
instances: 3
replica:
enabled: true
source: checkout-db-dc-a # the primary, in datacenter A
externalClusters:
- name: checkout-db-dc-a
connectionParameters:
host: checkout-db.dc-a.mesh # reached over ClusterMesh
sslmode: verify-fullReplication is asynchronous, because synchronous across a WAN would put every write at the mercy of the link. That buys survival at the cost of a small recovery-point window: if the primary site is lost, the few seconds of writes that had not yet crossed are gone, and the standby is promoted to primary. A clear owner per dataset keeps that trade honest.
The stores this replicates: the XDatabase engines from part twelve, now each with a primary in one site and an async standby in the other.
Sending users to the nearest site
If either datacenter can serve a request, something has to send each user to the right one and route around a dead site. Two mechanisms do it, at two layers. At the DNS layer, Cloudflare acts as a global load balancer from part six, health-checking each site and geo-routing users to the nearest healthy one, withdrawing a datacenter from rotation the moment its checks fail. At the network layer, a service that can be anycast is advertised with the same IP from both sites' BGP fabrics from part two, and the network itself delivers each user to the closer one. Together they mean a datacenter can fail and the platform keeps answering from the other, with nobody editing a record by hand:
# Cloudflare as global load balancer: a pool per site, health-checked, so a
# dead datacenter is withdrawn and users geo-route to the nearest healthy one.
name: api.example.com
steering_policy: geo
pools:
- name: dc-a
origins: [{ address: 203.0.113.10 }] # the Cilium LB IP, advertised by BGP
monitor: https-health
- name: dc-b
origins: [{ address: 198.51.100.10 }]
monitor: https-health
# For anycast-able services, the same IP is advertised from both sites' BGP
# fabrics (part 2), and the network routes each user to the closer one.Where the routing came from: the Cilium gateway, external-dns, and Cloudflare, now doing double duty as the global load balancer across sites.
Trust through a partition
One quiet thing has to keep working through all of this: certificates. Every mutual-TLS handshake and every issued cert depends on the PKI from part four, and if that PKI lived in one datacenter, a partition would slowly strangle the other as its certificates expired with nothing to renew them. The fix is the same one a separate writeup used for a multi-cloud mesh: one root of trust, offline, and an intermediate CA in each datacenter signed by it. Each site issues its own certificates from its own intermediate, depending on nothing across the WAN, so a partition can sever the link for hours and both sites keep minting and renewing certs that still chain to the one root the whole platform trusts:
# One root, an intermediate per site. Each datacenter issues its own certs, so
# a partition never blocks certificate issuance in either one.
bao secrets enable -path=pki_int_a pki # datacenter A's intermediate
bao secrets enable -path=pki_int_b pki # datacenter B's intermediate
# Both intermediates are signed by the same offline root the platform trusts,
# so certs from either site chain to one root, and neither site depends on the
# other's PKI to keep issuing through a link failure.The PKI this partitions: OpenBao's engine from part four, now a shared root with a per-site intermediate so trust survives a severed link.
What this buys
The platform now lives in two places at once without pretending to be one. Two autonomous clusters, each surviving the loss of the other; two copies of the state, replicated not stretched; users routed to the nearest healthy site and around a dead one; and a chain of trust a partition cannot break. Losing a datacenter has gone from an outage to an incident, and on bare metal you own the whole of that incident, the WAN, the replication, the routing, the trust, rather than renting a cloud's global backbone. Fourteen parts in, the platform is complete in the sense that everything a workload needs is here and self-service. What remains is not another capability but the discipline of keeping all of it healthy over years: the day-two work of upgrades, drift, failure, and fleet-wide operation, which is the last part.
The finale: keeping it all alive over years. Self-healing down to the physical machine, and clusters rebuilt from git, cattle all the way down.