Service Mesh Evolution: Istio Sidecars → Cilium eBPF
Blue-green migrated a 300-node production mesh off Istio sidecars onto a Cilium eBPF dataplane, unifying CNI and mesh, cutting API latency ~5%, and stretching one fabric across two clouds.
Choosing a service mesh is a decision you should be willing to revisit as the ecosystem moves. I did: moving a 300-node production cluster off a full Istio stack onto Cilium's eBPF dataplane. Because swapping the CNI and the mesh at once is inherently unpredictable, I did it blue-green: stand up a brand-new cluster on the target stack and cut traffic over once it proved itself, with the old mesh serving live traffic the entire time.
Why Istio first, and what we were leaving
I started on Istio in sidecar mode because I needed L7 authorization policies the Gateway API didn't yet support. It was well-tested and did the job. Over time it grew into a full stack: sidecar Envoys handling both L4 and L7, the Istio CNI plugin wiring their traffic redirection, Kiali for mesh observability, and Istio Multi-Cluster joining our clusters, all driven by Istio's native VirtualService and Gateway resources. It worked. The costs were the latency every sidecar added to every hop, and the sheer number of moving parts.
The target stack replaced each of those pieces with a Cilium-native one. Here is the whole swap at a glance:
Every piece of the Istio stack had a Cilium-native replacement. Tap a layer to see the trade.
Istio puts a full Envoy next to every pod to handle both layers. Cilium keeps L4 in the kernel with eBPF and only reaches for a single per-node Envoy when an L7 policy actually applies. Fewer proxies, fewer hops.
L4 in the kernel, L7 only when you need it
The core difference is where traffic is handled. Istio injects a full Envoy sidecar into every pod, and that proxy terminates both L4 and L7, so every hop pays two userspace Envoy traversals, one on each end, whether or not any L7 rule is in play. Cilium splits the layers: eBPF enforces L4 forwarding and policy in the kernel, and L7 is handled by a single node-local Envoy that the Cilium agent starts per node, which only sees traffic that actually matches an L7 policy.
That is the whole latency story. One shared node-local proxy costs less than a sidecar on both ends, and pure L4 traffic never leaves the kernel for userspace at all, which is where our roughly 5% API-latency drop came from. Watch a single request travel each way:
Same two pods, same request. The difference is how many userspace proxies it has to cross.
L4 forwarding and policy stay in the kernel; traffic only detours to a single node-local Envoy when an L7 rule actually matches. Most packets never touch userspace at all.
The CNI switch was mostly a deletion
People expect the CNI change to be the scary part. It was the opposite. The Istio CNI plugin is not a full CNI. It chains onto your real one and exists only to set up sidecar traffic redirection without granting NET_ADMIN to every pod. Once there are no sidecars, there is nothing to redirect to, so that entire plugin simply goes away.
Cilium, by contrast, is a full eBPF CNI: on the new cluster it owns pod networking, IPAM, and L4 policy natively. So the move wasn't porting one CNI config to another. It was deleting a category of configuration and letting the kernel datapath do the job the sidecar redirection used to.
Observability: Kiali to Hubble
Kiali is a genuinely good Istio console. It draws the mesh graph and validates how VirtualServices and DestinationRules wire together. But it builds that picture from Envoy telemetry scraped into Prometheus, so it only ever sees what the sidecars measured. When the sidecars left, so did its data source.
Hubble reads flow events straight from eBPF in the kernel. It sees every flow on the node (meshed or not, L3/L4 and L7 alike) and, crucially, it shows the policy verdict behind each one: when a connection is dropped, it names the exact policy responsible. That is a different and, for us, more useful lens:
Both draw a service map. They are looking at completely different things to do it. Pick a dimension.
Shows what flowed. It does not tell you why a given connection was denied.
Shows the verdict for every flow, including which network policy dropped it and why.
One fabric across two clouds
We had joined our clusters with Istio Multi-Cluster, which routes cross-cluster traffic through an East-West gateway and loads remote endpoints into every sidecar. On the new stack we used Cilium ClusterMesh instead: it shares endpoint state in the kernel via clustermesh-apiserver and routes pod-to-pod directly, with any Service annotated global load-balancing and failing over across clusters on its own.
This is how we connected the primary EKS cluster to a burst cluster on Oracle OKE running some legacy workloads, one logical service fabric across two clouds, discoverable and policy-enforced on both sides. Istio has the richer cross-cluster L7 story; we valued the leaner, kernel-level connectivity and the far smaller pile of moving parts:
Two ways to make many clusters behave like one fabric. We joined EKS and Oracle OKE, pick a dimension.
Traffic to another cluster hairpins through an East-West gateway (an Envoy) in each cluster.
Pods route to remote pods directly at the eBPF datapath, with no central gateway hop in the middle.
The network underneath the mesh
ClusterMesh is only the Kubernetes layer of that cross-cloud link. Underneath it sits a real hybrid network: an IPSec site-to-site VPN between AWS and OCI, Transit Gateways stitching each region together, and two fully independent AWS regions fronted by Route 53 and Global Accelerator for disaster recovery. The mesh rides on that reachability. It doesn't create it. I built and wrote about the whole fabric separately.
Why the Gateway API
The last piece was routing. Istio's VirtualService and Gateway are powerful but not portable. They are Istio CRDs, and they do not survive a change of mesh. Moving our north-south routing to the Kubernetes Gateway API meant that config was vendor-neutral: the same routes run on Istio or Cilium, so they carried across the migration instead of being rewritten from scratch.
The Gateway API also has a cleaner shape. It splits one tangled config into three resources with three clear owners (GatewayClass, Gateway, and HTTPRoute) and gives each protocol its own typed resource instead of cramming everything into a single VirtualService:
The Gateway API splits one tangled config into three resources with three clear owners. Tap a layer.
Attaches routing to a Gateway: hostnames, paths, header matches, backends, weights. The developer ships routes without ever touching listener or infrastructure config.
The Gateway API is a vendor-neutral upstream standard, so the same routes run on Istio or Cilium. That portability is a big part of why our north-south config survived the mesh swap instead of being rewritten.
HTTPRoute, GRPCRoute, TCPRoute, TLSRoute: each protocol is its own typed resource, instead of an Istio VirtualService that crams every protocol into one object.
Why we did it blue-green
Swapping the CNI and the mesh in one move is exactly the kind of change that behaves unpredictably in ways no staging cluster fully reproduces. So rather than mutate a live 300-node cluster in place, I stood up a fresh cluster on the entire target stack (Cilium CNI, eBPF L4, node Envoy L7, Hubble, ClusterMesh, and Gateway API) and let the old Istio cluster keep serving production the whole time. We validated the new fabric under real conditions, cut traffic over with a network flip, and kept the old cluster warm as an instant rollback path.