k8s on bare-metal - Part 6 : Traffic
Part six of the bare-metal series. The platform can store things now, but nothing outside can reach it and nothing inside is governed. This piece handles traffic in both directions: north-south through a Cilium gateway on an LB-IPAM IP, east-west through a sidecar-less Cilium mesh, with cert-manager minting the certificates and external-dns publishing the names. One dataplane does all of it, because the cluster already runs Cilium for everything underneath.
So far Cilium has been the plumbing: the CNI, the load-balancer IPs, the node encryption, the network policy. This part promotes it to the whole traffic story. The temptation on a platform like this is to reach for a service mesh out of habit, stand up Istio, and run a second dataplane alongside the one already carrying every packet. The better move, when the CNI is Cilium, is to notice how much of the job it already does and let it do the rest. North-south, the way requests arrive from outside, and east-west, the way services talk to each other, both become configuration on a dataplane that is already there, with certificates and DNS wired in from the parts before this one.
One dataplane, both directions
There are two flows of traffic to handle and they are usually treated as two products. North-south is ingress: a request from the internet arriving at a service. East-west is the mesh: one service calling another inside the cluster. The reflex is a gateway or ingress controller for the first and Istio for the second, which is two more moving parts on top of the CNI. But Cilium, already installed and already carrying every packet, does both, so the whole of this part is teaching one dataplane a bit more rather than bolting on two new ones.
The previous layer: Rook Ceph. The gateway will expose the platform's UIs; the mesh will govern the services that use those Ceph volumes.
The L7 decision, made once
Before any YAML, the one decision everything hangs on: Cilium's own service mesh, or Istio in ambient mode alongside it. Istio is more capable at L7, genuinely so, with fine-grained per-request routing and a deeper authorization model. But most of what a platform actually uses, Gateway API routing, identity-based policy, encryption, and observability, Cilium already provides, and the parts Istio adds are not needed here yet. Weighed against a second dataplane to run and upgrade forever, the choice is Cilium:
| Cilium (chosen) | Istio ambient | |
|---|---|---|
| Dataplane | eBPF plus a per-node Envoy, no sidecars, and it is already running here for the CNI | ztunnel plus waypoint proxies (sidecar-less now), but a second dataplane to run |
| North-south | Cilium Gateway API on an LB-IPAM IP over BGP, same tool as everything else | Istio gateway, a separate Envoy fleet from the CNI underneath |
| L7 features | Gateway API routing plus L7-aware network policy: covers the common cases | Deeper: fine-grained per-request routing, mirroring, richer AuthorizationPolicy |
| Identity | Mutual authentication with SPIFFE identities (SPIRE-backed), for authorization | Per-workload SPIFFE mTLS as a first-class primitive |
| Encryption | WireGuard at the node level: identity and encryption are deliberately separate | mTLS encrypts and authenticates in one step |
| The trade | One dataplane, least machinery, everything it does not do we do not need yet | More capability, more to run: worth it only once the extra L7 depth is actually required |
One tool, six jobs
Calling it a mesh undersells it. The same Cilium install covers the front door and the internal fabric with one set of primitives: the gateway and its IP for north-south, mutual authentication and encryption and L7 policy for east-west, and Hubble watching all of it. Tap through what each does and which direction it serves:
The front door. A Gateway with gatewayClassName cilium takes a LoadBalancer IP from LB-IPAM, advertised to the fabric over BGP, and HTTPRoutes send traffic to backends. No separate ingress controller, just eBPF intercepting and forwarding to a per-node Envoy.
North-south: the front door
The front door is a Gateway with gatewayClassName cilium. It asks LB-IPAM for a specific external IP, which the BGP control plane from the network part advertises to the top-of-rack switches, and HTTPRoutes attach to send hostnames to backends. There is no ingress controller pod to run: eBPF intercepts traffic on the way in and hands it to a per-node Envoy. The Gateway API mechanics themselves, listeners and routes and TLS, are the same across implementations, so the details live in a separate writeup; here the point is only that the class is cilium and the IP is one you already own:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: edge, namespace: edge }
spec:
gatewayClassName: cilium
addresses: # ask LB-IPAM for a specific IP
- type: IPAddress
value: 10.0.10.1 # advertised to the fabric by Cilium BGP
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "*.example.com"
tls: { mode: Terminate, certificateRefs: [{ name: wildcard-tls }] }
- name: http # port 80: the ACME HTTP-01 driveway
protocol: HTTP
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: app, namespace: apps }
spec:
parentRefs: [{ name: edge, namespace: edge }]
hostnames: ["app.example.com"]
rules:
- backendRefs: [{ name: app, port: 8080 }]The Gateway API depth this part builds on: listeners, HTTPRoutes, ListenerSets, and the cert-manager gateway shim, in full.
Turning the gateway on
Two things have to be true for that Gateway to work, and both are easy to forget. The Cilium install gains gatewayAPI.enabled, which needs the Gateway API CRDs present and an operator and agent restart to take effect. And the gateway's ingress traffic runs under Cilium's special ingress identity, so the host firewall from the network part must admit the world to it, and every backend must admit the ingress identity to itself. Skip the second and the front door is open to the internet but closed to your own services, which is a confusing afternoon:
# Added to the Cilium Helm values (operator + agent restart after):
gatewayAPI:
enabled: true
enableAlpn: true # HTTP/2, required for gRPC over the gateway
---
# The ingress identity has to be allowed to reach a backend, or the route
# resolves to nothing. This is the rule people miss:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: allow-from-gateway, namespace: apps }
spec:
endpointSelector: { matchLabels: { app: app } }
ingress:
- fromEntities: [ingress]East-west, without sidecars
Inside the cluster, the mesh is not a fleet of sidecars but the same eBPF datapath enforcing identity and policy between services. Each workload has a SPIFFE identity issued by a SPIRE server Cilium runs, and a policy can require a caller to prove that identity before a connection is allowed, then narrow it further to specific HTTP methods and paths. What matters is the order of the checks and where encryption sits, which is deliberately not in this diagram:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: b-requires-auth, namespace: apps }
spec:
endpointSelector: { matchLabels: { app: service-b } }
ingress:
- fromEndpoints: [{ matchLabels: { app: service-a } }]
authentication:
mode: required # service-a must prove its SPIFFE identity
toPorts:
- ports: [{ port: "8080" }]
rules:
http: [{ method: "GET", path: "/api/.*" }] # L7: only GET /api/*graph TD
A["Pod A"] -->|SPIFFE identity<br/>from Cilium's SPIRE| B{"Mutual auth:<br/>is the peer allowed?"}
B -->|no| X["dropped"]
B -->|yes| C["L7 policy:<br/>method + path allowed?"]
C -->|traffic| D["WireGuard encrypts<br/>node to node"]
D --> E["Pod B"]Identity is not encryption
That last policy is worth pausing on, because it is exactly the trap a whole other writeup is about. Cilium's mutual authentication proves who the peer is; it does not, by itself, encrypt the payload. Treating the two as one is how a team can believe they have mutual TLS and actually be sending plaintext. On this platform the split is on purpose: mutual authentication decides who may talk, and WireGuard, turned on back in the network part, makes what they say ciphertext on the wire. Two jobs, two mechanisms, and no illusion that one is the other.
The cautionary tale behind this section: how a feature branded mTLS turned out to be authentication without encryption, and why keeping them separate is the fix.
Certificates, public and private
Both directions need certificates, and they come from two different authorities depending on who the name is for. Public hostnames, the ones the gateway serves to the world, get certificates from Let's Encrypt through cert-manager, solved either over HTTP-01 through the gateway's port-80 listener or, for wildcards, over DNS-01 against Cloudflare. Private hostnames, the ones that never leave the cluster, get certificates from cert-manager's Vault issuer pointed at OpenBao's PKI engine from the secrets part, using the same Kubernetes auth as everything else so no token is stored anywhere. Toggle the two tiers:
# Public: ACME, with two solvers on one issuer.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt }
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: platform@example.com
privateKeySecretRef: { name: letsencrypt-acc }
solvers:
- http01: { gatewayHTTPRoute: {} } # normal hosts, via the gateway
- dns01: # wildcards, via Cloudflare
cloudflare:
apiTokenSecretRef: { name: cloudflare-token, key: token }
selector: { dnsNames: ["*.example.com"] }
---
# Private: cert-manager's Vault issuer against OpenBao's PKI engine.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: openbao-pki }
spec:
vault:
server: https://openbao.internal:8200
path: pki_int/sign/internal
caBundle: <base64 of the OpenBao root>
auth:
kubernetes: # same k8s auth as the secrets operator
role: cert-manager
mountPath: /v1/auth/kubernetes
serviceAccountRef: { name: cert-manager }- For
- Names the outside world reaches: the gateway's external hostnames
- Authority
- cert-manager ACME against Let's Encrypt, a public CA browsers already trust
- Challenge
- HTTP-01 through the gateway's port-80 listener for normal hosts; DNS-01 via Cloudflare for wildcards and names not yet routable
- Where it lands
- A TLS Secret the Cilium Gateway serves on its HTTPS listener
Names from annotations
The last piece is DNS. external-dns watches the Gateway and its HTTPRoutes, and for every hostname it finds it writes a record in Cloudflare pointing at the gateway's LB-IPAM IP. The Cloudflare API token it needs is itself an external secret from OpenBao, so even the DNS credential stays inside the secrets model. Put it together and a service is reachable over HTTPS the moment its route exists: cert-manager mints the certificate, external-dns publishes the name, and the LB IP is already advertised by BGP. No cloud load balancer, no hand-edited DNS:
# external-dns Helm values.
provider: { name: cloudflare }
env:
- name: CF_API_TOKEN
valueFrom: # the token is itself an ExternalSecret
secretKeyRef: { name: cloudflare-token, key: token }
sources: [gateway-httproute, service] # read Gateway API routes + Services
domainFilters: [example.com]graph TD
A["Apply a Gateway + HTTPRoute<br/>hostname app.example.com"] --> B["cert-manager mints the cert<br/>(ACME via the gateway / Cloudflare)"]
A --> C["external-dns publishes the name<br/>to Cloudflare -> the LB-IPAM IP"]
D["Client resolves app.example.com"] --> C
C --> E["LB IP, advertised by Cilium BGP"]
E --> F["Cilium Gateway<br/>class: cilium"]
F --> G["HTTPRoute -> service"]
B --> FThe whole path
Standing back, the traffic layer is one picture: a name resolving through Cloudflare to a Cilium LB IP the fabric knows by BGP, into a Cilium Gateway, routed to a service, which then talks east-west to other services under mutual authentication and WireGuard, all on the single dataplane that was already carrying every packet before this part started.
What can finally be reached
The platform now has a front door and an internal nervous system. A service gets an external HTTPS name from a route and an annotation; services inside prove who they are to each other and talk in ciphertext; and it is all one dataplane, one tool to operate. Which means the next thing the platform builds can actually be seen and used: a container registry with a UI to expose and images to serve. That is where this goes next, and it leans on almost everything so far, the object storage for its backend, the gateway for its front end, the secrets model for its credentials, and, most of all, a hard look at the software supply chain feeding the cluster.
Next: Harbor on the Ceph object store, fronted by this gateway, and the chain that decides which images are even allowed to run.