Multi-Cluster, Multi-Cloud, Hybrid: One Network Fabric
Two independent AWS regions plus an Oracle OKE burst cluster, made to feel like one network: Route 53 and Global Accelerator for failover, an IPSec site-to-site VPN for the hybrid link, and Cilium ClusterMesh joining only the primary EKS to OKE.
Real platforms are never one tidy cluster. This one spans two AWS regions, a second cloud, and a pile of serverless: a primary EKS cluster, a full disaster-recovery mirror in a second region, an Oracle OKE burst cluster for heavy database and compute work, and the legacy AEC product still on Lambda. The engineering was in making all of it reachable as one network, while keeping the pieces that must fail independently genuinely independent.
The shape of the platform
Everything sits behind one global front door: Route 53 for latency- and health-based DNS, and AWS Global Accelerator for an anycast entry point that fails over between regions in seconds, without waiting out a DNS TTL. Behind it are two fully independent AWS regional stacks.
Each region is the same shape: a Primary VPC running Amazon EKS with Cilium and an Envoy Gateway, and a Legacy VPC holding the AEC product on Lambda and API Gateway, joined inside the region by a Transit Gateway. Hanging off the primary region is Oracle OKE, a cost-optimized burst cluster carrying heavy compute, ML/batch, and stateful databases. Keeping each AWS region self-contained is exactly what lets OCI act as an extension of the primary platform without entangling disaster recovery.
Follow it top to bottom: a global edge, two independent AWS regions, and OCI hanging off the primary. Tap any box.
The Primary VPC runs Amazon EKS with Cilium and an Envoy Gateway; the Legacy VPC holds the AEC product on Lambda and API Gateway. A Transit Gateway joins the two VPCs, and this region is the only one that anchors the OKE link.
- Primary VPC: EKS, Cilium, Envoy Gateway
- Legacy VPC: Lambda, API Gateway (AEC)
- Transit Gateway joins the two VPCs
- Anchors the VPN + ClusterMesh to OCI
ClusterMesh is the overlay, not the connection
The tempting mistake is to think Cilium ClusterMesh connects the two clouds. It doesn't. It rides on a connection you have to build underneath it first. The base layer is an IPSec site-to-site VPN between AWS and OCI: on the AWS side a Virtual Private Gateway and Customer Gateway terminate the tunnel and the Transit Gateway routes it into the EKS VPC; on the OCI side a Dynamic Routing Gateway and CPE do the mirror job into the OKE VCN. The two ends are near-perfect reflections of each other.
Only once nodes and pods can actually route to one another does ClusterMesh layer on top, sharing endpoint state through clustermesh-apiserver, routing pod-to-pod directly, with global Services discoverable and policy-enforced on both sides. It's a clean separation of concerns: the VPN and gateways provide raw reachability, and Cilium provides Kubernetes semantics over it.
Joining two clusters across clouds is built bottom-up. Tap a layer, top overlay to physical tunnel.
The Kubernetes layer: cross-cluster service discovery, direct pod-to-pod routing, and shared network policy. The thing to understand is that it does not create connectivity. It consumes it. Without L3 reachability from the layers below, there is nothing for ClusterMesh to route over.
Why the mesh stops at the primary
ClusterMesh joins the primary EKS cluster to OKE and nothing else. I deliberately did not extend it to the DR region, because a disaster-recovery stack that depends on the primary's cross-cluster state isn't a fallback. It shares fate with the very thing it's supposed to survive.
So Region 2 is a standalone mirror. On a Region 1 failure a Route 53 health check trips, Global Accelerator shifts to Region 2, and it serves immediately with nothing to rewire. OCI access during DR is already handled the same way: a second VPN/DRG is pre-established from Region 2 and kept dormant, activating only on failover. The middle of an outage is the worst possible time to be standing up new network connectivity, so that path is built ahead of time and simply waits.
The mesh reaches OCI from the primary only, so the DR region stays standalone. Trip Region 1.
Traffic lands in Region 1. Cilium ClusterMesh joins the primary EKS to OKE, so heavy compute and databases in OCI are reachable as if local. Region 2 sits warm and independent, carrying no live traffic and no mesh state.
One job per box
A hybrid network like this only stays debuggable if every component has a single, clear responsibility and the layers don't blur into one another. That discipline (Route 53 for DNS failover, Global Accelerator for the anycast edge, Transit Gateway for intra-region VPC stitching, the VPN and gateways for the hybrid link, Cilium for in-cluster networking, and ClusterMesh scoped to exactly one cross-cluster link) is what keeps a two-cloud, two-region topology from turning into a haunted house nobody wants to touch.
Every box in the diagram has exactly one job. Tap a component to see it.
Cross-cluster service discovery and connectivity, deliberately only between the primary EKS and OKE.