k8s on bare-metal - Part 2 : CNI

Still NotReady

Cilium is not just an add-on here; it is the first workload the cluster ever runs. Nothing else can go on before it, not even the GitOps controller, because a pod cannot be scheduled onto a node that has no network, and Argo CD is made of pods. So the CNI is installed by the Cluster API add-on provider the moment the control plane exists, and only once it turns the nodes Ready does the rest of the platform become possible. That ordering is the spine of this article, and the last section comes back to it.

How a packet crosses

Every other decision follows from one: does a pod packet travel as itself, or wrapped in an overlay? Native routing sends it unchanged and asks the physical fabric to route it; tunnel mode wraps it so the fabric can stay ignorant. On a cloud you never see this choice. On bare metal with a fabric you control, it is the difference between paying an encapsulation tax on every packet or not. Toggle the two:

Speaking eBGP to the rack

Native routing only works if the switches know how to reach pods, and the clean way to tell them is BGP. Cilium has a BGP control plane built in, so every node speaks eBGP directly to its own top-of-rack switch, with a unique ASN per rack and no route reflector in the middle. Each node advertises its slice of the pod address space, and later its LoadBalancer service IPs, up into the fabric. These are the same sessions that already carry the API VIP from part one, so pod routing costs no new infrastructure, just more routes on a session that exists.

# One config per rack: nodes in rack 1 run AS 65001, peering to their ToR.
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: rack-1
spec:
  nodeSelector:
    matchLabels: { rack: "1" }
  bgpInstances:
    - name: rack-1
      localASN: 65001               # unique per rack
      peers:
        - name: tor-1
          peerASN: 65101            # the ToR's ASN: eBGP, so it differs
          peerAddress: 10.0.0.1     # the top-of-rack switch
          peerConfigRef: { name: tor }
---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: pods-and-services
  labels: { advertise: bgp }
spec:
  advertisements:
    - advertisementType: PodCIDR                 # native routing needs this
    - advertisementType: Service
      service: { addresses: [LoadBalancerIP] }   # and the service IPs
      selector:
        matchLabels: { advertise: bgp }

No kube-proxy, no iptables

With the fabric routing pods, kube-proxy can go. Cilium replaces it entirely: service load balancing moves into the eBPF datapath, so there are no iptables chains whose length grows with every Service you add. A request to a LoadBalancer lands on any node by ECMP, the kernel picks a backend with Maglev consistent hashing, and Direct Server Return sends the reply straight back to the client without a return-path hairpin and without rewriting the source IP. Follow one request through:

graph TD
  C["Client"] -->|1. dst = service IP<br/>learned via BGP| F["Fabric / ToR<br/>ECMP across nodes"]
  F -->|2. lands on any node| N["Node<br/>eBPF datapath"]
  N -->|3. Maglev picks a backend<br/>no kube-proxy, no iptables| P["Backend pod<br/>maybe another node"]
  P -->|4. DSR: reply goes<br/>straight to the client| C

Everything switched on

An install is a set of decisions, not a wall of YAML. Here is the set, each with the switch behind it: native routing plus the BGP control plane for the datapath, full kube-proxy replacement with DSR and Maglev, cluster-pool IPAM so Cilium owns pod addressing, WireGuard for encryption, a host firewall, and Hubble for visibility. Two are worth dwelling on. WireGuard is one flag and genuinely encrypts pod traffic on the wire, and because the fabric runs jumbo 9000 frames, the pod MTU is simply reduced by WireGuard's overhead rather than fragmenting anything. And because kube-proxy is gone, the agents cannot rely on it to reach the apiserver, so they are pointed at KubePrism, Talos's built-in local API proxy on every node, rather than at the external VIP, which keeps the datapath from ever depending on the VIP being reachable. Tap through the switches:

# The load-bearing Helm values (via the add-on provider, see below).
kubeProxyReplacement: true
cgroup: { autoMount: { enabled: false }, hostRoot: /sys/fs/cgroup }  # Talos already mounts cgroupv2 + bpffs
routingMode: native
ipv4NativeRoutingCIDR: 10.244.0.0/16
autoDirectNodeRoutes: false        # BGP carries the routes, not L2 direct routes
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
    clusterPoolIPv4MaskSize: 24     # a /24 per node, the CIDR it advertises
bgpControlPlane: { enabled: true }
loadBalancer:
  mode: dsr                         # hybrid is the safe fallback under strict uRPF
  algorithm: maglev
encryption: { enabled: true, type: wireguard }
mtu: 8920                           # 9000 fabric minus WireGuard overhead
bpf: { masquerade: true, hostLegacyRouting: false }   # needs Talos forwardKubeDNSToHost: false (below)
hostFirewall: { enabled: true }
hubble:
  relay: { enabled: true }
  ui: { enabled: true }
k8sServiceHost: localhost            # KubePrism: Talos's local API proxy, not the VIP
k8sServicePort: 7445

Tuned for Talos

Cilium on an immutable OS needs a handful of settings that are easy to miss and painful to debug, because Talos is not a normal Linux host. It already provides the cgroupv2 and bpffs mounts, so Cilium must be told not to mount them itself (cgroup.autoMount.enabled: false), or the agent ends up fighting the operating system. Talos also refuses to let workloads load kernel modules, so the SYS_MODULE capability is dropped from the agent and anything Cilium needs is loaded by the machine config instead. And the apiserver endpoint is KubePrism rather than the VIP: Talos runs a local, load-balanced API proxy on every node at localhost:7445, so a kube-proxy-free Cilium reaches the control plane through it and never depends on the external VIP being up.

One known issue is worth naming because this exact build walks into it. Talos forwards cluster DNS to the host by default, and that does not get along with Cilium's eBPF masquerade; CoreDNS starts misbehaving in ways that are miserable to trace back. The fix is a single machine-config line, forwardKubeDNSToHost: false, and it comes right. A note for later in the series, too: when Multus is added for the storage network, Cilium must run with cni.exclusive: false so it does not claim the CNI configuration directory as its own.

Encryption that actually encrypts

It is worth being precise about what WireGuard here does, because Cilium's security surface has a famous trap. This is transparent, node-to-node encryption of the actual pod payload: turn it on and traffic between nodes is ciphertext on the wire. That is a different guarantee from Cilium's mutual-authentication feature, which authenticates identities without necessarily encrypting the data, a distinction subtle enough to have fooled people who thought they had mTLS and did not. For a fleet that spans racks, or one where compliance asks the plain question of whether traffic is encrypted in transit, WireGuard answers it in one line.

Locking the host

The same eBPF that routes pods can firewall the nodes. With the host firewall enabled, a CiliumClusterwideNetworkPolicy becomes the node's real ingress filter, and the only open ports are the ones named: the apiserver, the Talos API, etcd between control-plane nodes, BGP to the ToR, WireGuard, the kubelet, and Hubble. The one rule of rolling this out safely is to start in audit mode, where violations are logged rather than dropped, watch what a normal day actually needs, and only then switch to enforce. Locking a node's firewall while blind is how you fence yourself out of your own cluster.

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: host-firewall
spec:
  nodeSelector: {}                  # every node
  enableDefaultDeny:
    ingress: true                   # flip this on only after audit mode is clean
  ingress:
    - fromEntities: [remote-node, health]   # nodes and health checks talk freely
    - toPorts:
        - ports:
            - { port: "6443" }      # kube-apiserver
            - { port: "50000" }     # talos apid
            - { port: "2379" }      # etcd (control-plane only, tighten by selector)
            - { port: "179" }       # BGP to the ToR
            - { port: "51871" }     # WireGuard (Cilium default)
            - { port: "10250" }     # kubelet

External IPs at last

This is the third of the three address layers from part one coming live: service LoadBalancer IPs. A CiliumLoadBalancerIPPool defines the routable range, any Service of type LoadBalancer draws from it automatically, and the BGP advertisement from earlier carries that IP into the fabric so it is reachable from outside the cluster. No MetalLB, no second BGP speaker, no cloud. The IP a Service gets is announced by the same Cilium that routes its pods.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: prod
spec:
  blocks:
    - cidr: 10.0.10.0/24            # routable, advertised via BGP
---
apiVersion: v1
kind: Service
metadata:
  name: web
  labels: { advertise: bgp }       # matches the BGP advertisement selector
spec:
  type: LoadBalancer               # gets 10.0.10.x, announced to the ToR
  selector: { app: web }
  ports:
    - { port: 443, targetPort: 8443 }

Installed before everything

Back to the ordering, because it is the part that makes the whole platform bootstrap cleanly. Cilium is delivered by the Cluster API add-on provider as a HelmChartProxy that targets the cluster the instant it is created. The add-on provider watches for clusters with a label and lays the chart down before anything else is scheduled, which is the only way to break the deadlock: the CNI cannot wait for GitOps, because GitOps cannot run without the CNI. Once Cilium is in and the nodes go Ready, Argo CD can be installed the same way, and from there the platform installs itself.

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: cilium
spec:
  clusterSelector:
    matchLabels: { cni: cilium }   # every cluster labelled this gets Cilium, first
  repoURL: https://helm.cilium.io/
  chartName: cilium
  namespace: kube-system
  valuesTemplate: |
    # the values from earlier, templated per cluster

graph TD
  A["Control plane up, nodes NotReady<br/>(end of part one)"] --> B["CAPI add-on provider applies<br/>a Cilium HelmChartProxy"]
  B --> C["cilium-agent DaemonSet starts<br/>on every node"]
  C --> D["CNI ready: kubelet flips<br/>the nodes to Ready"]
  D --> E["eBGP sessions come up to the ToRs"]
  E --> F["PodCIDRs + LoadBalancer IPs<br/>advertised into the fabric"]
  F --> G["cluster is now a place<br/>Argo CD can be installed"]

Ready

Apply it and the nodes that sat NotReady at the end of part one turn Ready within a minute: pods route to each other natively across the fabric, every hop between nodes is WireGuard ciphertext, kube-proxy is nowhere to be found, and a Service of type LoadBalancer gets a real IP that the top-of-rack switches already know how to reach. The cluster is now a functioning network that happens to run on metal you can touch. What it does not yet have is a way to install anything without a human running helm, which is the next piece: part three hands the cluster its GitOps brain, Argo CD bootstrapped by the same add-on provider, then driving everything else through ApplicationSets.