When the Gap Is Huge, Leap: A Blue-Green Kubernetes Upgrade
We were six minor versions behind, on EKS extended support, with a hard deadline before AWS force-upgraded the cluster out from under us. So we skipped the six in-place hops and did one blue-green cutover from 1.30 to 1.36. Here is how it went, what to watch for, and the boring habit that would have made the whole thing unnecessary.
Our production cluster was on Kubernetes 1.30, and 1.30 was already on EKS extended support. Extended support is not a comfortable place to sit: it costs more, and it ends. Ours ended on 23 July 2026, about three months out. What happens at that date is the part that concentrates the mind: if you have not upgraded, AWS auto-upgrades the control plane for you, on its schedule, not yours, and it moves only the control plane, leaving your node groups behind and your version skew widening. An uncontrolled minor bump landing on a live cluster whenever the vendor decides is a very efficient way to breach every SLA at once. So we chose to jump first, on our own terms, six versions in a single leap, and to build a habit so we never get cornered like this again.
Why blue-green, not six in-place hops
Kubernetes only lets you upgrade one minor version at a time, and EKS enforces it. From 1.30 to 1.36 that is six sequential in-place upgrades, each with its own control-plane bump, node-group roll, drain, validation, and soak. You can absolutely crawl it: take the weeks, do it carefully, one hop at a time. We did not want to. If I am honest, part of it was impatience: six careful hops is six chances to babysit a cluster at 2am.
But there was a real reason too. A blue-green cutover (stand up a brand-new 1.36 cluster beside the live 1.30 one and swap traffic across) turns the version gap into a single clean, reversible cut. And once you are building a fresh cluster anyway, you can batch every other disruptive change you have been deferring into the same window: the CNI swap to Cilium, the Istio-to-Cilium service mesh move, Cluster Mesh, and the Gateway API. One scary maintenance window instead of six, and the version leap stops mattering the moment you stop fighting it hop by hop.
The cutover, step by step
The shape of a blue-green cluster migration is always the same four beats: build the target idle, seed and warm it, flip the network, and (the beat everyone skips) keep a hot path back. The old cluster does not get torn down at the flip; it stays read-only-warm as the safety net, so rollback is the exact same DNS swap in reverse rather than a heroic restore from backup.
Six versions in one leap: stand up a fresh 1.36 cluster beside the live 1.30 one, warm it, and swap traffic. Walk the sequence.
- writes ✓ · reads ✓
- controllers running
- ~months of accumulated etcd state
- no writes · no reads
- controllers installed in NoOp
- etcd empty
Standard operating mode. The 1.36 target is built and validated, its controllers installed but held in NoOp, its etcd empty. The window is not open yet.
Audit the APIs before you flip
Here is the failure mode that turns a clean cutover into a bad afternoon: a fresh 1.36 apiserver silently rejects manifests written against APIs that died somewhere in the six-version gap. Your GitOps sync does not warn you politely. It fails, or worse, half-applies, and you are debugging a partially-reconciled cluster with live traffic already on it. Every dead API is a silent rejection waiting to happen, so the audit happens before the flip, not after.
The slide-deck classics (PodSecurityPolicy, dockershim, the old Ingress API) were all removed before 1.30, so they were not our problem. These were. Tap through the ones that actually bit in the 1.30-to-1.36 range:
A fresh 1.36 apiserver silently rejects manifests written against dead APIs, and your GitOps sync just fails, or half-applies. These are the ones that bit in our range. Tap one.
Permanently disabled on the 1.36 target. It cannot even be re-enabled by a flag. Any Pod that mounted a gitRepo volume simply will not schedule. It was a root-level RCE waiting to happen, so this one is a good riddance.
Replace with an initContainer running git-sync (or a bare clone) into an emptyDir. We had a couple of legacy jobs still doing this; they had to be rebuilt before the flip.
across six versions, 1.30 → 1.36. You will discover all of them eventually. The audit decides whether that is before the flip or during it.
How to find them before they find you
You do not audit 40-plus API changes by hand. Two tools do almost all of it: pluto scans static manifests and Helm charts, and kubent (kube-no-trouble) scans a live cluster. Run both against a 1.36 target version, feed them your GitOps repo as well as the running cluster, and fix every hit before the flip. One ecosystem change is worth calling out that no API scanner catches: Ingress-NGINX was retired in March 2026, which was one more nudge toward the Gateway API we were adopting in the same cutover anyway.
# Scan static manifests + Helm charts, and the live cluster, against 1.36.
pluto detect-files -d ./gitops/
pluto detect-helm --target-versions k8s=v1.36.0
kubent --target-version 1.36
# The removals/deprecations that land in the 1.30 -> 1.36 range:
# flowcontrol.apiserver.k8s.io/v1beta3 REMOVED 1.32 -> .../v1
# kubernetes.io/cephfs (in-tree) REMOVED 1.31 -> CephFS CSI driver
# status.nodeInfo.kubeProxyVersion REMOVED 1.33 -> node labels / DS image
# core/v1 Endpoints DEPRECATED 1.33 -> discovery.k8s.io/v1 EndpointSlice
# volume.gitRepo DISABLED 1.36 -> initContainer + git-sync
# Service.spec.externalIPs DEPRECATED 1.36 -> LoadBalancer / Gateway APITune the whole stack for the seed
However you seed the target (a Velero-style restore of state, a mass GitOps re-sync, or both), you are pushing a very large number of objects through the target's apiserver in a short window. A fresh cluster on default limits chokes on that. The restore path is a pipe: backup client, then apiserver, then admission webhooks, then etcd. Optimize one layer and you just move the bottleneck to the next. Tap through all three:
Every object you replay into the target flows through the same pipe. A fresh cluster on default limits chokes on a bulk restore. Tap a layer to squeeze it.
Reads and writes stop queuing behind tiny default limits. A short event TTL and more delete workers keep etcd quieter, leaving headroom for the bulk load. Priority & Fairness was reshuffling our restore traffic. On recent versions you widen it with a permissive FlowSchema on flowcontrol.apiserver.k8s.io/v1 rather than disabling it.
Tune every dial in isolation and the naive product is a fantasy: QPS×30 · workers×32 · GC×10 · shards×15. Reality lands near a single order of magnitude, because bottlenecks shift: lift one ceiling and the next one reveals itself. That is exactly why you tune all the layers, not the one you noticed first.
Every knob here is for the restore window only. Roll them all back the moment the load stabilizes. The steady-state defaults exist for good reasons, and you do not want a 3000-inflight apiserver with admission set to Ignore running your production week.
One caveat that matters on EKS
Two of those three layers are yours; one is not. On a managed control plane like EKS you do not hold the kube-apiserver flags (AWS sizes and scales the control plane for you), so in practice you lean on the restore client and the admission webhooks, which you do control, and let the managed control plane autoscale under the load. On a self-managed control plane (kubeadm, kOps, Cluster API) Layer 2 is back in your hands. Either way, treat all of it as strictly window-scoped: the defaults exist for good reasons, and a 3000-inflight apiserver with admission set to Ignore is not something you leave running through a normal production week.
# Layer 1: the restore/backup engine (yours everywhere). Indicative values;
# tune to your own cluster's headroom.
args:
- --client-qps=150 # was 5
- --client-burst=300 # was 10
- --item-block-worker-count=32 # was 1 (parallel, not one-at-a-time)
env:
- name: GOGC
value: "10" # was 100 (~90% less GC pause mid-restore)
# Layer 2: kube-apiserver (self-managed control planes only; NOT settable on EKS).
- --max-requests-inflight=3000 # was 400
- --max-mutating-requests-inflight=1500 # was 200
- --delete-collection-workers=10 # was 1
- --event-ttl=30m
# APF: v1beta3 is gone as of 1.32, widen flowcontrol.apiserver.k8s.io/v1 with a
# permissive FlowSchema for the window rather than disabling fairness outright.
# Layer 3: admission webhooks (yours everywhere), FOR THE RESTORE WINDOW ONLY.
webhooks:
- name: "*"
failurePolicy: Ignore # was Fail, a slow/unavailable webhook can no
# longer stall the bulk restore. Set back to Fail after.Four things to carry into the next one
- When the gap is huge, leap. Do not crawl. A clean, reversible, parallelizable cut beats six anxious in-place hops. The version gap stops mattering once you stop fighting it one version at a time.
- Tune the whole stack, not one layer. Backup engine plus apiserver plus admission compound; any one alone is a single-digit win, because the bottleneck just shifts to whatever you did not tune.
- Any partition is parallelism. Namespaces, labels, providers, owners: find a cleaving dimension and shard the restore across it instead of streaming it single-file.
- Plan the rollback first. A migration without a flip-back is a bet. Engineer reversibility into every step, because the path you never walked is the path that breaks.
The upgrade we should never have needed
This cutover was a success, and I am proud of how it went, precisely because we batched it. One disruptive window bought us the version leap, the CNI swap, the mesh migration, Cluster Mesh, and the Gateway API all at once. Blue-green earned its keep.
But I want to be honest about the framing, because the clean execution hides an own-goal. We would not have needed a six-version leap at all if we had not let the cluster drift onto extended support until a hard AWS deadline was the thing forcing our hand. A six-version jump is a great story and a bad habit. So the real deliverable was not the migration. It was the process we put around it afterwards: track the EKS version calendar, budget a routine in-place upgrade every cycle, and never again let the gap grow to the point where a vendor's auto-upgrade is what moves us. Leap when you genuinely must. The goal is to arrange your life so you almost never have to.