Clusters Are Cattle Too: Fleet Management with Cluster API

The mental model: Deployments, one level down

The primitives map almost one to one. A Machine is to a host what a Pod is to a container. A MachineDeployment manages a group of worker Machines the way a Deployment manages Pods, rollouts included. KubeadmControlPlane does the same for control-plane Machines, with the etcd-aware ordering that makes control planes special. And ClusterClass is the template layer: define a cluster shape once, stamp out as many as you need, then manage them as a set called a managed topology.

The payoff is the same one Deployments gave us: you stop scripting the how and start declaring the what. A cluster stops being the output of a provisioning pipeline and becomes a reconciled resource with drift correction, rollout strategies, and history.

Why day 0, not day 100

The strongest argument for CAPI is not multi-cluster scale, it is the retrofit you will never do. Every fleet horror story starts the same way: 'we just need one prod and one staging cluster, let's keep it simple.' Six months later there are four clusters, each provisioned by a slightly different Terraform state and a README nobody trusts, and the upgrade test in staging proves nothing about prod because the two have quietly diverged. When the forced upgrade lands (a deprecated API removal, a CVE, an end-of-support date), the gap between clusters becomes the incident.

Adopting CAPI on day 0 costs almost nothing: your first cluster is one Cluster object referencing one ClusterClass. Adopting it on day 500 means importing pet clusters with hand-grown quirks into a system built on uniformity, which is exactly the kind of migration that never gets scheduled. The first cluster defines the last one. It is a contract, not a tool.

The pattern that makes it click is pairing CAPI with GitOps: cluster specs live in a repo, a management cluster runs the controllers, and creating a cluster is a pull request. The same review, promotion, and rollback muscle memory your app teams already have now applies to the infrastructure itself.

v1.12: in-place updates

CAPI has always been proudly immutable: change a Machine spec and the controller builds a replacement, drains the old node, deletes it. Predictable, easy to reason about, and independent of whatever OS or bootstrap mechanism is underneath. But immutability charges rent: rotating a node credential should not require draining every workload in the fleet.

v1.12 introduces update extensions, a pluggable way to change existing Machines in place, supported by both KubeadmControlPlane and MachineDeployments. The important design choice: this is not in-place versus rollouts. You still just change the spec; CAPI picks the safest mechanism that achieves it, and the maintainers' guidance is blunt: if the workload will be disrupted anyway, keep the rollout. In-place earns its keep on changes that never needed a drain in the first place.

v1.12: chained upgrades

Until now, CAPI enforced the same rule as everyone else: one minor version at a time. Falling three versions behind meant three full upgrade campaigns, each planned, executed, and babysat before the next could start, which is precisely why fleets fall behind in the first place.

Chained upgrades break the deadlock. You declare the target version on the Cluster topology, and CAPI computes an upgrade plan: the control plane walks through every intermediate minor in strict order, while the workers skip whatever the kubelet version-skew policy (n-3) allows, often rerolling once instead of four times. Upgrade-plan runtime extensions let you shape the plan, and lifecycle hooks let you automate the between-steps chores, like upgrading an addon right after the control plane lands.

# The whole upgrade campaign is this diff.
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: prod-eu-1
spec:
  topology:
    classRef:
      name: fleet-standard      # the ClusterClass every cluster stamps from
-   version: v1.31.4
+   version: v1.35.2            # CAPI computes and runs the chain
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 12

Play with the plan below. The asymmetry between the control-plane row and the worker row is the whole feature: control planes must walk every minor, workers ride the skew policy.

The upgrade we did the hard way

I have personal history with this problem. We once sat six minors behind on EKS extended support with a forced-upgrade deadline, and the least-bad option was a full blue-green cutover: build a parallel cluster on the target version, migrate everything, flip the traffic. It worked, and I wrote up the playbook, but it was weeks of engineering for what was, conceptually, a version field change.

That is exactly the ceremony chained upgrades absorb. Declare v1.36, let the control plane walk the chain, let the workers leap. The blue-green playbook still matters when you are changing more than the version (new CNI, new topology), but 'we fell behind' stops being a reason to build a second cluster.

Karpenter for the muscle

Fleet lifecycle solved, there is still capacity to manage inside each cluster. Cluster Autoscaler thinks in node groups: give it MachineDeployments and it nudges replica counts. Karpenter inverts this and thinks in pods: it watches what is Pending, simulates scheduling against requests, affinities, and taints, then provisions the cheapest machine shape that actually fits, no pre-sized groups involved.

Karpenter grew up welded to AWS, but since its donation to the CNCF the core went provider-neutral, and karpenter-provider-cluster-api is the glue that lets it drive any CAPI-compatible infrastructure. The translation is clean: Karpenter's NodePool holds the rules and limits, a ClusterAPINodeClass replaces the AWS-specific EC2NodeClass, and every NodeClaim materializes as a Cluster API Machine instead of a raw EC2 instance. One honest caveat: native Karpenter's party trick is needing no management cluster at all, and routing through CAPI reintroduces one. If you are running a CAPI fleet anyway, that cluster already exists and the trade is free.

The two CRDs that wire it together, trimmed to their essence:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.cluster.x-k8s.io
        kind: ClusterAPINodeClass
        name: fleet-workers
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]
  limits:
    cpu: "200"                 # the fleet-wide ceiling
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    expireAfter: 720h          # forced node rotation
---
apiVersion: karpenter.cluster.x-k8s.io/v1alpha1
kind: ClusterAPINodeClass
metadata:
  name: fleet-workers
spec:
  # Which CAPI MachineDeployments Karpenter may scale,
  # selected by label instead of hardcoded names.
  machineDeploymentSelector:
    matchLabels:
      node-pool: karpenter-managed

Where this leaves the fleet

Put together, the shape is pleasingly boring. ClusterClass stamps out uniform clusters; GitOps makes new ones a pull request; chained upgrades keep the version story to a single field edit; in-place updates stop credential rotations from draining fleets; and Karpenter right-sizes the machines underneath, deleting them again when the load goes home.

One warning from the release notes worth repeating: being able to jump three minors in one move is not a reason to fall three minors behind. Chained upgrades make catching up cheap; they make staying current cheaper still. The fleet you want is the one where an upgrade is so unremarkable that nobody schedules a war room for it.