Anatomy of a 28-Minute Outage
A disruption budget set to 100%, a spot interruption, and an AI coding assistant: how a new service went partly dark, and the systemic fixes that followed.
The best incidents to write up are the ones where nothing was fundamentally wrong with the design. The platform did exactly what it was told, and what it was told was subtly wrong. This is one of those.
What happened
A new service had PodDisruptionBudgets set to minAvailable: 100% on spot-backed workloads. When AWS reclaimed a spot node, Karpenter couldn't drain it (a 100% budget permits zero voluntary disruptions), but AWS terminated the node anyway. The result was resources stuck Terminating: finalizers that couldn't complete because the kubelet was already gone, failed evictions, and replacement pods that wouldn't schedule because Kubernetes still believed the old ones existed.
For about 28 minutes, roughly 35% of the new service's APIs saw elevated latency and 5xx, and its background queues stopped consuming.
Detection and recovery
My LGTM stack caught it and paged through AlertManager and OpsGenie at the 8-minute mark; on-call diagnosed and escalated to me around 10 minutes. The fix was hands-on: deleting the stuck pods, PersistentVolumeAttachments, CSI cleanup objects, and custom resources whose finalizers could never complete on their own with the kubelet gone, while confirming there was no stateful impact before and after.
The root cause was an AI coding assistant
The developer had omitted a critical: true flag that would have placed the workload on on-demand capacity with correct disruption budgets. The composition allowed a PDB override, and their AI coding assistant had helpfully filled in minAvailable: 100%. Technically valid, operationally fatal on spot. It slipped past PR review and the Change Approval Board.
The fix was systemic, not a patch
- Removed the PDB override from the platform composition and exposed a single critical: true construct that sets on-demand placement and correct budgets automatically.
- Ran a fleet-wide audit for manual PDB overrides and fixed several, including a few in off-the-shelf Helm charts running on spot.
- Made a rule: any PDB implies at least three replicas plus topology spread, with an admission policy to block violations before they merge.
- HolmesGPT now front-runs this class of incident, cutting diagnosis from 10–15 minutes to 2–3.
The part I keep thinking about
The proximate cause was a well-meaning AI suggestion that was locally reasonable and globally wrong. As more infrastructure is written with AI assistance, the guardrails that catch that class of mistake (composition defaults, admission policies, review that a machine can't wave through) stop being nice-to-haves and become the platform's real job.