One Mesh, Two Clouds: A Deep Dive into Istio Multi-Cluster

Start with what istiod actually does

Everything in multi-cluster Istio follows from what the control plane actually does. One cluster, one istiod, and none of its four jobs needs a name. Add a second cluster to the mesh and a question appears that every deployment model is really an answer to, and Istio ships two answers, built by different pressures for different relationships between clusters. The pressures are worth more than the topologies, because they tell you which one you are under:

Multi-primary: built for independence

The shape: an istiod in every cluster, each watching all clusters' API servers (that is what remote secrets grant) but serving only the sidecars in its own. Injection, config pushes, and certificate signing are all local; only endpoint discovery, which tolerates loss, and the workload traffic itself cross the boundary. It was built for exactly that property: organizations running production in multiple regions refused to let cluster A's ability to function depend on anything in cluster B. When the link degrades, nothing important happens: both clusters keep injecting, configuring, and rotating certs; stale cross-cluster endpoints get ejected; traffic falls back local. Peers, with independent lives.

Where it gets used, and where it earns its cost:

• Active-active across regions: both clusters serve production all the time, locality routing keeps requests in-region, and a regional failure is not a failover event so much as the survivor simply continuing, at full capability, brain and all.
• Per-service failover, the case edge devices cannot see: a region can be perfectly healthy at its front door while one service inside it is down. A global load balancer keeps routing in. The mesh ejects that service's local endpoints via outlier detection and sends just those calls to the sister region, mid-call-graph, with mTLS and retries intact, while every other service stays local.
• Blast-radius partitioning and mesh canarying: with a control plane per cluster you can upgrade istiod in one cluster, watch it soak, then do the next. A shared control plane can never canary itself.
• Organizational autonomy: clusters owned by different teams or regions that share one trust domain but refuse shared fate.
• The honest cost: N istiods to operate, and mesh configuration consistency becomes your job. Istio does not replicate VirtualServices between primaries; GitOps does.

Primary-remote: built for economy

The shape: istiod in one cluster, the primary; every other cluster is a deliberately thin remote: workloads, sidecars, and a webhook pointing across the boundary. The satellite brings capacity, not a brain. It was built for the opposite pressure: running istiod is real operational work, and fleets kept growing with clusters that were small, subordinate, or short-lived, where a control plane each multiplied that work for nothing those workloads needed. There was a governance motive too: one place where mesh config and versions are decided, with no satellite able to drift.

The price is coupling, priced fairly: if the primary, or the link to it, is down, the remote's sidecars keep serving their last-pushed configuration; what pauses is change (injection, routing updates, cert rotation) until it returns. For satellite workloads that is a degraded control plane, not an outage. Which points at the test that picks the model: if the first cluster vanished, is the second supposed to carry on alone? Yes means peer, and peers get multi-primary. No means extension, and extensions look like this:

• Burst capacity: overflow scheduling into a second cluster, often a cheaper cloud. Same applications, same owners, placed elsewhere for capacity and cost. The satellite exists to absorb load, not to survive alone.
• Data gravity and legacy adjacency: workloads that belong physically next to a database estate or a system that is not moving. The mesh extends to where the data lives instead of the data moving to the mesh.
• Edge and branch clusters: dozens of small clusters in stores, factories, or POPs, with neither headroom nor headcount for a control plane each.
• Ephemeral clusters: CI, batch, seasonal capacity. Clusters that join the mesh in minutes precisely because they bring nothing but workloads.

Cilium ClusterMesh: the answer from a different layer

Cilium ClusterMesh solves the same sentence, services in one cluster can call services in another, without a mesh control plane at all, because it lives in the CNI. Its model fits neither Istio box, though it rhymes with one. Every cluster runs its own complete Cilium; there is no such thing as a remote Cilium cluster, so every member is structurally a primary. Each cluster exposes a clustermesh-apiserver, a small read-only endpoint publishing that cluster's identities, endpoints, and services, and every node in each cluster connects directly to the other clusters' apiservers and pulls that state into its local eBPF maps. Peer-to-peer state synchronization, not configuration distribution: nothing is ever pushed to anyone.

Services merge by convention: the same name and namespace in both clusters plus a service.cilium.io/global annotation, and the datapath load-balances across both clusters' endpoints. That is namespace sameness again, enforced at L4. Trust is network-level, WireGuard or IPsec between nodes, rather than per-workload certificates. So ClusterMesh is multi-primary-shaped, symmetric and autonomous with an even stronger link-failure story (a lost peer just means stale remote entries), and it achieves that by having dramatically less to distribute. What it does not carry across clusters is the L7 layer: SPIFFE workload identity, request-level routing, retries, mirroring, per-service locality failover. The comparison is not which is better but which sentence you need:

The second axis: one network or two

The control plane axis answers who thinks; the network axis answers how packets travel, and the network underneath decides it for you. If the clusters' networks are peered with non-overlapping CIDRs and pod IPs are routable end to end, the clusters share one Istio network: sidecars connect pod IP to pod IP directly, and the mesh needs no gateways in the data path. If pods cannot route to each other, or CIDRs overlap, each cluster is its own network: an east-west gateway in every cluster bridges them, endpoints in other networks are advertised as the remote gateway's address instead of unroutable pod IPs, and cross-network traffic arrives at port 15443 where the gateway forwards by SNI (AUTO_PASSTHROUGH) without terminating the workloads' mTLS.

This axis is also the entire difference between Istio's four install guides, which read as more choice than they are. Within each pair, the control-plane story is identical; the multi-network variants add exactly three things:

• A topology.istio.io/network label on each cluster's istio-system namespace, naming its network.
• An east-west gateway in every cluster (the same-network guides deploy one only in a primary, and only to expose istiod).
• The expose-services step: the 15443 AUTO_PASSTHROUGH gateway that lets workloads cross the network boundary, with istiod rewriting cross-network endpoints to gateway addresses.

The architecture: two primaries, one shared satellite

Now assemble it. Two EKS clusters, eks-east in us-east-1 and eks-west in us-west-2, are equal primaries in multi-primary mode, serving traffic active-active. The failure this design actually worries about is not a region disappearing, which is rare, but a single service degrading in one region, which is Tuesday. That is why there is no global load balancer doing failover at the edge: an edge device sees a healthy front door and keeps routing to it, while the mesh sees the failing service itself, ejects its local endpoints, and reroutes just those calls to the sister region automatically. Failover becomes a per-service, per-request decision made where the failure is visible.

Beneath them, one OKE cluster in OCI Ashburn attaches to both primaries in primary-remote mode. It exists for two reasons that both fit the extension test: burst capacity, and adjacency to a legacy Oracle database estate that is not moving. Workloads on OKE (the reporting service, overflow replicas) are full mesh citizens callable from either region; the databases themselves stay outside the mesh, reachable through ServiceEntries. All three networks are peered with non-overlapping CIDRs, pod IPs route end to end, so the whole mesh is one Istio network: no east-west gateways in any data path, sidecar-to-sidecar mTLS everywhere.

One satellite, two masters

Attaching one remote to two primaries is the piece the install guides do not spell out, and it is what makes the satellite survive a regional control-plane loss without ever owning a brain. Four things make it work. Both primaries hold a remote secret for the satellite, so workloads in either region can route to its pods. The satellite's istio-system namespace names both primaries in its topology.istio.io/controlPlaneClusters annotation, so both maintain its webhooks. The satellite's discovery address is one fronted name, a health-checked DNS record you own, resolving to the pinned control-plane addresses of both regions, so when eks-east goes dark the satellite's agents reconnect to eks-west on the next resolve. And both primaries must carry identical mesh configuration, which is a GitOps job, because a satellite whose behavior depends on which brain it happens to be talking to is a debugging story you do not want.

The fourth dependency, certificates, is removed rather than fronted: with certificate issuance handled locally on the satellite (the cert-manager section below), cert rotation does not depend on either primary at all.

graph LR
  PA["istiod<br/>primary, region A"] -.->|remote secret A:<br/>watches satellite| S["shared satellite<br/>burst capacity, no istiod"]
  PB["istiod<br/>primary, region B"] -.->|remote secret B:<br/>watches satellite| S
  S -->|xDS + injection| F["one fronted address<br/>health-checked DNS over<br/>both pinned gateway IPs"]
  F -->|healthy| PA
  F -->|healthy| PB

Wiring the primaries

The multi-primary half is refreshingly small: two nearly identical installs that agree on the mesh name and the network name but differ on the cluster name, then remote secrets exchanged both ways so each istiod can watch the other's endpoints. On one Istio network there are no gateways to deploy for data traffic at all:

# eks-east (repeat on eks-west with clusterName: eks-west):
cat <<EOF | istioctl install --context "$CTX_EAST" -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: eks-east
      network: cloud-net        # one network: everything is peered
EOF

# Cross-watch, both directions:
istioctl create-remote-secret --context "$CTX_EAST" --name eks-east \
  | kubectl apply -f - --context "$CTX_WEST"
istioctl create-remote-secret --context "$CTX_WEST" --name eks-west \
  | kubectl apply -f - --context "$CTX_EAST"

# From here the two regions are one mesh: a sidecar in us-east-1 sees
# endpoints in us-west-2 and can reach them directly, pod IP to pod IP.

The EKS wrinkle

Before the satellite can attach, each primary's istiod must be reachable at a stable address (xDS on 15012, the injection webhook on 15017), and right here the official primary-remote guide posts a warning: these instructions are not suitable for an AWS EKS primary. The reason is small and infuriating. AWS load balancers surface in Kubernetes as DNS names (the Service status carries a hostname, not an IP), while the remote cluster's plumbing for remotePilotAddress builds a Service whose endpoints must be IP addresses. An FQDN where an IP belongs, and the attachment never assembles.

The fix is to stop letting AWS choose the address. The AWS Load Balancer Controller can provision an internal network load balancer with private IPv4 addresses you specify per subnet. Pin them, in both regions, and the control-plane addresses become numbers you chose, reachable across the peering, with a DNS record you own fronting the pair:

# In BOTH primaries: the Service exposing istiod gets an internal NLB
# with pinned private IPs (AWS Load Balancer Controller annotations).
apiVersion: v1
kind: Service
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-0aaa, subnet-0bbb
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.10.1.15, 10.10.2.15
spec:
  type: LoadBalancer
  # ports 15012 (xDS) and 15017 (webhook), per gen-eastwest-gateway.sh
---
# eks-west: same Service, its own subnets, e.g. 10.20.1.15, 10.20.2.15.
# Then one record you own fronts both regions, health-checked:
#   istiod.mesh.internal -> 10.10.1.15, 10.20.1.15

Attaching the satellite

The OKE side is deliberately thin: one annotation naming both managers, a remote profile pointing at the fronted address, and two remote secrets so both primaries can fold the satellite's endpoints into the mesh:

# 1. The satellite is managed by BOTH primaries:
kubectl --context "$CTX_OKE" create namespace istio-system
kubectl --context "$CTX_OKE" annotate namespace istio-system \
  topology.istio.io/controlPlaneClusters=eks-east,eks-west

# 2. Remote profile: no istiod, just pointers at the fronted address.
cat <<EOF | istioctl install --context "$CTX_OKE" -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: remote
  values:
    istiodRemote:
      injectionPath: /inject/cluster/oke-1/net/cloud-net
    global:
      remotePilotAddress: istiod.mesh.internal   # both regions behind it
EOF

# 3. BOTH primaries watch the satellite's endpoints:
istioctl create-remote-secret --context "$CTX_OKE" --name oke-1 \
  | kubectl apply -f - --context "$CTX_EAST"
istioctl create-remote-secret --context "$CTX_OKE" --name oke-1 \
  | kubectl apply -f - --context "$CTX_WEST"

# A pod scheduled on oke-1 is injected by whichever primary is healthy,
# dials the fronted address for config, and joins the mesh.

DNS does not follow the mesh

Here is the part that surprises everyone once: endpoint discovery crossing clusters does not make names resolve across clusters. Kubernetes DNS answers from the local cluster's Service objects, full stop. A pod on eks-east calling the reporting service that only runs on oke-1 gets NXDOMAIN before Istio's beautifully merged endpoint table is ever consulted, because the request dies at name resolution, inside the client, before the sidecar sees a single byte. Every multi-cluster mesh answers the DNS question explicitly, and there are four honest answers, from mirrored stub Services (the documented recommendation, and the formal version of the namespace sameness convention), through the sidecar answering DNS itself, up to the ecosystem's two heavyweights: the Multi-Cluster Services API, which makes exporting services Kubernetes' own problem via ServiceExport and clusterset.local, and Admiral, which watches the whole fleet and writes the Istio config you would otherwise maintain by hand:

# checkout runs only on oke-1. From a pod on eks-1:
$ nslookup checkout.shop.svc.cluster.local
** server can't find checkout.shop.svc.cluster.local: NXDOMAIN

# istiod knows the oke-1 endpoints (remote secret) and would route
# there happily. The request never leaves the client: kube-dns answers
# from LOCAL Service objects only, and there is nothing to resolve.

One root, zero key copies

Everything so far assumed the three clusters trust each other, and in a mesh trust means one thing: every workload certificate, in all three clusters, chains to the same root. The stock way to arrange that is the cacerts secret: generate a root, mint an intermediate per cluster, and place each intermediate, private key included, into every cluster as a Kubernetes Secret for istiod's built-in CA to sign with. It works, and it should make you slightly uncomfortable: long-lived CA signing keys, sitting in etcd in three clusters across two clouds, readable by anyone who can read Secrets in istio-system, rotated by hand-distributing new files.

cert-manager's istio-csr replaces that arrangement. It is a small agent that implements the same gRPC certificate API istiod's CA speaks; istiod runs with its CA server disabled, every istio-agent sends its CSR to the cluster-local istio-csr instead (the caAddress setting), and istio-csr turns each CSR into a cert-manager CertificateRequest, signed by whatever Issuer you configured. Run it in all three clusters, back each cluster's Issuer with an intermediate chaining to one root held in Vault, ACM Private CA, or an offline box, and the trust story changes shape:

• No signing key ever enters any cluster. The intermediates live in Vault or PCA; the root can live in a safe. The cacerts pattern of CA private keys as etcd rows disappears entirely.
• One PKI, three clusters, two clouds: workloads verify each other because the chains meet at the root, not because key material was copied anywhere. A fourth cluster is another intermediate, not another key ceremony.
• Issuance stays local everywhere, which quietly upgrades the satellite: its certs come from its own istio-csr, so rotation keeps working even with both primaries unreachable. The one control-plane dependency the fronted address could not remove, removed.
• Rotation, renewal windows, and audit come from cert-manager, the same machinery already running the rest of your PKI, and istiod's own serving certificate is cert-manager-managed too.
• The signer is swappable: start on a CA issuer, move to Vault or ACM PCA later by changing the Issuer, not the mesh.

# In EACH of the three clusters: istio-csr against an intermediate whose
# chain ends at the shared root (Vault, ACM Private CA, or offline).
helm upgrade --install cert-manager-istio-csr \
  oci://quay.io/jetstack/charts/cert-manager-istio-csr \
  --namespace cert-manager \
  --set app.certmanager.issuer.name=istio-intermediate \
  --set app.certmanager.issuer.kind=Issuer \
  --set app.tls.trustDomain=cluster.local \
  --set app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem

# In the primaries' IstioOperator (folded into the config from earlier):
# CA server off, and every sidecar's CSRs go to its local istio-csr.
#   spec:
#     components:
#       pilot:
#         k8s:
#           env:
#             - name: ENABLE_CA_SERVER
#               value: "false"
#     values:
#       global:
#         caAddress: cert-manager-istio-csr.cert-manager.svc:443
#
# caAddress is a cluster-local name: every pod reaches its own cluster's
# istio-csr. Signing never crosses a cluster boundary, let alone a cloud.

graph LR
  W["istio-agent<br/>reporting pod, oke-1"] -->|1. CSR over gRPC<br/>caAddress| CSR["istio-csr<br/>local, cert-manager ns"]
  CSR -->|2. authenticates the pod,<br/>creates CertificateRequest| CM[cert-manager]
  CM -->|3. sign| ISS["Issuer<br/>intermediate CA"]
  ISS -.->|chains to| ROOT["Shared root<br/>Vault / ACM PCA"]
  ISS -->|4. signed chain| CM
  CM -->|5. SPIFFE cert<br/>+ trust bundle| W
  W ===|6. mTLS verified<br/>across clouds| P["istio-agent<br/>payments pod, eks-east"]

Teach the mesh where it is

The last move turns the topology into the behavior the architecture was chosen for. By default Istio load-balances evenly across all endpoints, a cloud away included. What active-active actually wants is: serve local, and fail over per service the moment local is unhealthy. Locality load balancing does exactly this, keyed off the region labels the clusters already carry, with one catch that reads like a riddle until you know it: locality-aware routing only activates when outlier detection is configured, because failover needs a health signal to decide that local has failed. This DestinationRule is the entire no-edge-failover story:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: checkout-locality
  namespace: shop
spec:
  host: checkout.shop.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        failover:
          - from: us-east-1        # each region prefers itself...
            to: us-west-2          # ...and fails over to its sister
          - from: us-west-2
            to: us-east-1
    outlierDetection:              # the health signal failover requires
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 60s
# checkout failing in us-east-1: its local endpoints are ejected and only
# checkout's calls shift west. Every healthy service stays in-region.
# The satellite joins the ladder with failoverPriority or a weighted
# distribute block when a service should burst into oke-1 as well.

What you end up with

Three clusters that behave like one platform, with each model doing the job it was built for. The two regions are peers: active-active, independently operable, failing over per service and per request at the point where failure is actually visible, no edge device guessing from outside. The satellite is an extension: burst capacity and legacy-adjacent workloads, full mesh citizens with no control plane to run, managed by whichever region is healthy and degrading to frozen-but-serving if both are gone. Names resolve because DNS was answered explicitly instead of discovered in production. And every certificate in all three clusters chains to a root that none of them has ever held, issued and rotated locally by the same cert-manager machinery that runs the rest of your PKI. One mesh, two clouds, and every piece chosen for the relationship it serves: that is what the deployment models are for.