k8s on bare-metal - Part 15 : Fleet

Built, and now kept alive

Everything until now has been construction. Day two is different work, and it is where most platforms quietly rot, because the effort of keeping them current and healthy grows until it swallows the team. The way out is not more effort; it is a property, that every layer is disposable and reproducible, so keeping the platform alive is mostly the platform reconciling itself while humans supply intent and judgment. This part is that property made concrete: how a version bumps, how a dead node comes back, how a lost cluster is rebuilt, and how a whole fleet is run from one desk.

Cattle, all the way down

There is one idea underneath all of day-two operations here, and it is the oldest one in the field: cattle, not pets. Nothing on this platform is nursed back to health by hand, because nothing is irreplaceable; every layer can die and be replaced or rebuilt from a declaration. That is not a slogan, it is a property the earlier parts quietly built, and it holds all the way down, from a single pod to an entire datacenter. Tap each level to see what kills it and what brings it back:

Upgrades without a war room

The first day-two task is the one that used to require a war room: upgrading Kubernetes. Here it is a field. Because every cluster is an instance of a shared ClusterClass driven by Cluster API from part one, changing the version in the cluster's topology is enough; CAPI computes the whole plan, walking the control plane through every minor and letting the workers leap ahead where the kubelet skew policy allows, rolling one node at a time and never taking a service past its PodDisruptionBudget. The blue-green cutover a separate writeup describes was the hard way to do this before chained upgrades existed; now falling behind is a choice, not a fate:

# The whole cluster's version is a field. Change it and CAPI computes the plan:
# the control plane walks every minor, workers skip per the skew policy, rolling
# one node at a time and never past a PodDisruptionBudget.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata: { name: dc-a }
spec:
  topology:
    class: bare-metal-talos        # the shared ClusterClass
    version: v1.34.0               # bump this; the upgrade is declarative
    controlPlane: { replicas: 3 }
    workers:
      machineDeployments:
        - class: default-worker
          replicas: 20

Self-healing to the metal

The most satisfying day-two behavior is the one nobody has to trigger. A MachineHealthCheck watches every node, and when one goes NotReady for too long it does not file a ticket; it deletes the Machine, and Cluster API asks Metal3 to build a replacement, which powers a spare server on over Redfish and reinstalls Talos, exactly the sequence from part one. Node Problem Detector sharpens this by turning kernel deadlocks, failing disks, and hardware faults into node conditions that both the health check and the alerting from part eight can act on. This is where the series closes its loop: the path that bootstrapped the cluster is the path that heals it, run automatically instead of by hand:

# A MachineHealthCheck turns a dead node into a reprovisioning: if a Machine's
# node is NotReady too long, CAPI deletes it and Metal3 builds a fresh one.
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata: { name: workers }
spec:
  clusterName: dc-a
  selector: { matchLabels: { role: worker } }
  maxUnhealthy: 40%                # never remediate a majority at once
  unhealthyConditions:
    - { type: Ready, status: "False", timeout: 5m }
    - { type: Ready, status: Unknown, timeout: 5m }
# Node Problem Detector feeds extra conditions (kernel deadlock, bad disk,
# hardware error) that this and the part-eight alerting both act on.

Clusters are cattle too

Self-healing handles a node; a whole cluster needs a rebuild story, and it has one, because a cluster here is not a snowflake but the sum of a few declarations and backups. Cluster API and a ClusterClass describe its shape; Argo CD from part three holds everything running in it; Velero backs its resources and volumes up to the Ceph object store from part five; and Talos snapshots etcd on a timer. Lose a cluster entirely and it is reproducible: provision a new one from the class, let Argo CD re-apply git, restore etcd from the snapshot and the rest from Velero. A disaster-recovery drill stops being a dreaded quarterly ritual and becomes something you can run on a Tuesday:

# Velero backs up resources and volumes to the Ceph RGW from part five, so a
# cluster's state is portable and restorable.
apiVersion: velero.io/v1
kind: Schedule
metadata: { name: nightly, namespace: velero }
spec:
  schedule: "0 1 * * *"
  template:
    storageLocation: ceph-rgw      # the object store from part five
    ttl: 720h
# Plus a Talos etcd snapshot on a timer. A lost cluster is then rebuildable:
# CAPI re-provisions it, Argo CD re-applies git, the snapshot restores etcd.

Operating the whole fleet

One cluster you can operate by hand; a fleet you cannot, and by now there may be several, across the datacenters from part fourteen. They are run from one place. A management cluster holds the tools, Cluster API to provision, Argo CD to deploy, Kyverno to govern, the aggregated Mimir to watch, and HolmesGPT to heal, and every workload cluster is an entry it reconciles. Adding a datacenter is committing one more Cluster resource; the class gives it the same shape as every other, the generator fills it with the same platform, and the same policies and dashboards apply from the first minute. The fleet grows without the operating effort growing with it:

# One ClusterClass is the fleet's template: every datacenter's cluster is an
# instance of it, so a fix or an upgrade to the class reaches all of them.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata: { name: bare-metal-talos }
spec:
  controlPlane:
    ref: { kind: TalosControlPlaneTemplate, name: cp }
  infrastructure:
    ref: { kind: Metal3ClusterTemplate, name: metal3 }
  workers:
    machineDeployments:
      - class: default-worker
        template: { infrastructure: { ref: { kind: Metal3MachineTemplate } } }

The whole thing

That is the platform, fifteen parts of it. It began with a powered-off server and a BMC, and it ends with a fleet that provisions, secures, observes, scales, persists, serves models, spans datacenters, and heals itself, all from git, all cattle, none of it a pet. Every hard thing a cloud does for you and marks up, the control plane, the load balancer, the managed database, the global backbone, the replaced hardware, is owned here instead, and the through-line that made owning all of it sustainable was a single discipline: declare the intent, let the machinery reconcile the world to match, and keep the human in the loop for judgment, not toil.

That discipline is the real product. The hardware was always going to be commodity; what a platform like this buys is the ability to run frontier workloads on it as calmly as a cloud would, while owning every layer beneath them. Which, for the labs that run their models on metal, is rather the whole point.