k8s on bare-metal - Part 9 : Scaling
Part nine of the bare-metal series. The platform can see itself now; this part gives it reflexes. It scales pods out on the signals from part eight, right-sizes them in place, and grows the cluster itself when workloads go Pending. The twist is physical: a scale-up powers a real server on, a scale-down powers it off, and the ceiling is the iron you actually own.
Autoscaling in the cloud is a solved, almost invisible thing: a metric moves, a number changes, and capacity appears from an infinite pool you rent by the second. On bare metal every part of that sentence needs re-examining. The metric still moves, and the signals from part eight are exactly what to move on. The number still changes. But capacity does not appear from nowhere; it is a physical machine that has to be claimed from a finite pool and powered on, which takes minutes and draws real watts. That constraint is not a limitation to apologize for, it is the thing that makes the design interesting: scaling becomes a question of powering iron up and down against real demand, and the same observability that shows the load also shows the power bill.
Reflexes, not just senses
The last part gave the platform senses: metrics, logs, traces, and the rest, all queryable in one place. Senses without reflexes are just a dashboard, though. This part wires the two together, so the numbers the platform can now see become the trigger for what it does: add copies of a workload, resize them, or grow the cluster underneath them. Two of those three are ordinary Kubernetes. The third, on bare metal, is where it gets physical.
The senses this part reacts to: the Mimir metrics, from queue depth to power draw, that every autoscaler here reads.
Three ways to scale
There are three independent ways to give a workload more room, and a healthy platform uses all three. It can add more copies of a pod, make each pod bigger, or add machines for the pods to land on. They answer different questions and lean on different tools, and the third is where bare metal changes the story. Tap through them:
When a Mimir metric or an event source crosses a threshold, a queue filling, request rate climbing, KEDA adds replicas, and removes them again down to zero when the work stops. It reads the same signals part eight collects, so scaling is driven by what the platform already measures.
Out: KEDA on the signals we already have
Adding replicas is the reflex most reached for, and the modern way to do it is KEDA rather than a bare Horizontal Pod Autoscaler. KEDA scales on events and on any metric, not just CPU, which matters because the honest signal for a worker is rarely CPU; it is the depth of the queue it drains or the rate of requests it serves. Those numbers already live in the Mimir from part eight, so a ScaledObject can point straight at them, and unlike a plain HPA it can scale all the way to zero when there is no work, so an idle consumer costs nothing:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: worker }
spec:
scaleTargetRef: { name: worker }
minReplicaCount: 0 # scale to zero when the queue is empty
maxReplicaCount: 50
triggers:
- type: prometheus # query Mimir, the store from part eight
metadata:
serverAddress: http://mimir-query-frontend.observability.svc/prometheus
query: sum(rate(jobs_enqueued_total[2m]))
threshold: "100" # one replica per ~100 jobs/sec of backlogUp: VPA, without the restart
Adding replicas only helps if each replica is the right size, which is the job nobody does by hand well. The Vertical Pod Autoscaler watches actual usage against requested resources and corrects the request, so a pod that asked for far too much stops wasting a node's capacity and one that asked for too little stops being throttled. Historically the catch was that changing a pod's resources meant recreating it; in-place pod resize removes that, letting the VPA adjust a running pod without a restart. The one rule to remember is that vertical and horizontal scaling must not fight over the same signal: point the VPA at CPU and memory for right-sizing, and let KEDA scale replicas on the business metric.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: api }
spec:
targetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
updatePolicy:
updateMode: Auto # apply recommendations automatically...
minReplicas: 2 # ...but never drop below two live pods
resourcePolicy:
containerPolicies:
- containerName: api
controlledResources: ["cpu", "memory"]
# In-place resize (Kubernetes 1.33+) adjusts a running pod without recreating it.
# Never point a VPA and an HPA at the same resource metric; they will oscillate.When a number becomes a server
The first two axes are pods on machines that already exist. The third adds machines, and this is the sentence a cloud lets you forget: on bare metal a node is a physical server. Karpenter watches for pods that cannot be scheduled and provisions the cheapest machine that fits, and through karpenter-provider-cluster-api that provisioning is a Cluster API Machine rather than a cloud instance. From there the bare-metal path from part one takes over: Metal3 claims a spare BareMetalHost, Ironic writes the image, and the BMC powers the server on over Redfish. What was a Pending pod becomes, minutes later, a running workload on iron that was dark until it was needed:
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: workers }
spec:
template:
spec:
nodeClassRef: { kind: ClusterAPINodeClass, name: bare-metal }
limits: { cpu: "400" } # the ceiling is the rack inventory
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m # bin-pack and delete idle nodes
# ClusterAPINodeClass points at Metal3-backed MachineDeployments, so a NodeClaim
# becomes a CAPI Machine, which becomes a provisioned, powered-on server.Everything left of the line is software Karpenter and Cluster API already do in the cloud. Everything right of it is the bare-metal difference: the node is a machine that has to be powered on, which is why it takes minutes, not seconds.
The Karpenter-to-Cluster-API mechanism in full: NodePool, ClusterAPINodeClass, and NodeClaims materializing as CAPI Machines.
The pool is finite
This is where the analogy to cloud autoscaling finally breaks, and usefully so. Karpenter in the cloud draws from an effectively infinite pool; here it draws from the machines in your racks, and that changes two things. First, powering a machine off is a real saving, not a smaller invoice: a deprovisioned server draws no power, and the watts show up in the BMC metrics from part eight, so scale-down is visible as a lower power draw and not just a lower node count. Second, there is a ceiling, the inventory itself, so a spike beyond the pool cannot be absorbed by spending more, only by having planned for it. A couple of warm spares kept powered on absorb ordinary bursts instantly; beyond that, capacity is a purchasing decision. Drive the pool and watch it:
6 machines are powered off, saving about 2.4 kW that a cloud would have kept billing. The warm spares stay on so a burst schedules instantly while a cold machine takes minutes to boot.
Where the power control came from: the Redfish BMC that provisioned each machine is the same interface that now powers it on and off to scale.
Bursting when the iron is slow
The ceiling is one bare-metal cost; latency is the other. A cold machine takes minutes to power on and provision, so a workload that goes Pending and waits for a new server feels every second of that boot. The fix is to keep a little headroom always scheduled but discardable. A deployment of pause pods at a negative priority holds roughly a node's worth of space; when a real pod needs to land, it preempts the pause pods instantly and schedules now, and their eviction is what triggers the new machine to start powering on in the background. The burst is served from headroom while the iron catches up:
# A low-priority "pause" deployment holds headroom. Real pods preempt it
# instantly and schedule now, while a new server powers on in the background.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: overprovisioning }
value: -10 # below any real workload, so it yields first
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: overprovisioning }
spec:
replicas: 6 # about one node's worth of reserved space
template:
spec:
priorityClassName: overprovisioning
containers:
- name: pause
image: registry.k8s.io/pause:3.10
resources: { requests: { cpu: "1", memory: 2Gi } }Coming back down safely
Scaling up is the easy half; scaling down is where platforms hurt themselves. Karpenter's consolidation constantly looks for nodes it can empty and delete by repacking their pods elsewhere, which is exactly what you want on finite hardware, but repacking means evicting, and eviction without care is an outage. Two guardrails keep it safe. PodDisruptionBudgets set a floor no drain may cross, so consolidation can never take a service below its minimum, a lesson learned the hard way in a disruption-budget incident. And the descheduler periodically rebalances pods that consolidation left lopsided. The system add-ons get their own small autoscaler too: the cluster-proportional-autoscaler sizes things like CoreDNS to the node count, so the control plane's helpers grow and shrink with the cluster:
# Scale-down drains nodes; a PodDisruptionBudget keeps enough replicas up so
# consolidation can never take the service below its floor.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec:
minAvailable: 80%
selector: { matchLabels: { app: api } }The disruption-budget lesson behind this section: what happens when nodes go away faster than the PDBs allow.
Scaling the models
One workload deserves a note, because it is a large part of why this platform exists: model serving. An idle accelerator is the most expensive thing in the building to leave running, so the instinct to scale to zero is strongest here, and KEDA on queue depth or GPU utilization does exactly that, spinning an inference deployment up when requests arrive and down when they stop. The accelerator-specific parts, GPU and Trainium and Inferentia scheduling, node shapes, and warm-pool tradeoffs, are involved enough to have their own part later; here it is enough to say the same three axes apply, with the cost of getting them wrong an order of magnitude higher.
The accelerator workloads this scales: vLLM across three generations of silicon, spot-first, behind an Envoy AI Gateway.
What this buys
The platform now has reflexes to match its senses. It scales pods out on the metrics it collects, right-sizes them in place, and grows itself when workloads go Pending, turning spare servers on and powering them off again when the load goes home, all against real demand rather than a static guess. The bare-metal constraints that looked like disadvantages, finite capacity and slow physical provisioning, turned into a design that is honest about cost in a way a cloud never forces you to be. What is left is to make all of this safe from the inside. The platform can grow, but a growing platform is a growing attack surface, and the next part hardens what runs on it: admission policy, runtime security, and turning the Tetragon signal from part eight from something that watches into something that enforces.
Next: hardening what all this capacity runs. Admission policy, sandboxed runtimes, and promoting the Tetragon signal from watching to killing.