k8s on bare-metal - Part 1 : Bootstrap

Cattle, all the way down to the iron

The goal sets the tooling. We want to describe a cluster in YAML, apply it, and have physical machines provision themselves into it, then scale, heal, and upgrade the same declarative way. That is precisely what Cluster API was built for, and its great insight is that it does not try to be one monolith that knows how to do everything. It shatters cluster creation into orthogonal concerns, each owned by a provider that knows nothing about the others. Get that model straight and the rest of this article is just filling in slots. Tap through the provider types:

The pairing: Metal3 for the metal, Talos for the OS

Two slots decide the character of a bare-metal build: the infrastructure provider that commands the hardware, and the OS that lands on it. For the hardware, Metal3 is the strongest choice, and not by a little. It is a CNCF project built on Ironic, the bare-metal provisioning engine that has run OpenStack fleets for a decade, and it is the same machinery underneath OpenShift's bare-metal installer, so it is proven at exactly the scale that matters. It speaks Redfish to the baseboard management controllers, does image-based provisioning (it writes a whole disk image, it never installs packages one by one), and handles the unglamorous but essential parts of real fleets: firmware, BIOS, and RAID configuration before the OS ever boots. Its Cluster API face is CAPM3.

For the OS, Talos Linux is the natural partner. It is immutable and API-driven: no shell, no SSH, no package manager, no config drift, a machine you talk to only through a gRPC API and a single declarative config. Image-based provisioning and an image-based OS are the same idea at two layers, which is why the two fit so cleanly. The other bare-metal options are real but wrong here: Sidero Metal is elegant and Talos-native, but its makers have shifted momentum to a non-Cluster-API product; MAAS is mature but Ubuntu-and-Juju shaped, and fights an immutable OS. Metal3 plus Talos, glued by Cluster API, is the combination I would reach for.

Cluster API stays honest about the division of labour: CAPM3 owns the hardware and the OS image, and Talos's own providers own the Kubernetes layer. The bootstrap provider (CABPT) emits the immutable Talos machine config instead of shell scripts; the control-plane provider (CACPPT) brings up etcd and rolls the control plane. Three providers, one management cluster, each replaceable on its own.

Three addresses, three tools

Before any YAML, the single most conflated topic in bare-metal Kubernetes: IP addresses. There is no cloud load balancer to paper over it, so you own the addressing, and "the IP" is actually three different jobs at three layers, done by three different tools. Confusing them is how people end up with a control plane that will not bootstrap or a load balancer that fights the CNI. Tap each layer:

The shape of it

Here is the whole of part one in one picture. A management cluster runs Cluster API and the three providers. It reaches the physical servers over two deliberately separate networks: the BMC network, where it speaks Redfish out-of-band to power boxes on and set their boot order, and a dedicated provisioning VLAN, where PXE and the Talos image and the static node IPs live, isolated from anything that will later carry workload traffic. What comes out the far side is a running control plane that is, on purpose, not yet ready.

The chicken and the egg

Cluster API runs inside a Kubernetes cluster, which is awkward when your goal is to create your first Kubernetes cluster. The standard escape is a two-step bootstrap. A throwaway kind cluster on a laptop gets the providers installed, and its only job is to provision the real, highly-available management cluster onto three bare-metal nodes. Then clusterctl move pivots every Cluster API and Metal3 object from the kind cluster onto the new one, which from that point manages itself and every workload cluster you build after. The laptop cluster is deleted; nothing of consequence ever lived there.

# On the throwaway kind cluster: install core CAPI + the three providers.
clusterctl init \
  --infrastructure metal3 \
  --bootstrap talos \
  --control-plane talos \
  --ipam metal3

# ...apply the management-cluster manifests, wait for it to come up, then
# relocate all Cluster API state onto the freshly built management cluster:
clusterctl move --to-kubeconfig=./mgmt.kubeconfig

# The kind cluster has done its job and can be deleted.

graph TD
  A["kind cluster<br/>throwaway, on a laptop"] -->|clusterctl init:<br/>Metal3 + Talos + IPAM| B["providers installed<br/>in kind"]
  B -->|apply the mgmt cluster spec| C["3 bare-metal nodes<br/>provisioned as the real<br/>management cluster"]
  C -->|clusterctl move| D["pivot: all CAPI + Metal3<br/>objects relocate onto it"]
  D --> E["management cluster<br/>now manages itself"]
  E -->|and provisions| F["workload clusters"]
  A -.->|deleted| G["kind thrown away"]

Enrolling the iron

A physical server enters the system as a BareMetalHost: the one thing you cannot infer, its BMC address and credentials, plus the MAC it will PXE-boot from. The Bare Metal Operator takes it from there, powering the box just far enough to run an inspection that reads its real CPU, memory, NICs, and disks into the resource's status, then parking it available in the pool. Nothing is provisioned yet; you have simply told the system this hardware exists and how to reach its out-of-band controller.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: node-cp-01
  namespace: metal3
spec:
  online: true
  bootMACAddress: "e4:11:5b:aa:bb:01"   # NIC on the provisioning VLAN
  bmc:
    address: redfish://10.0.0.11/redfish/v1/Systems/1
    credentialsName: node-cp-01-bmc      # Secret: username + password
  rootDeviceHints:
    deviceName: /dev/nvme0n1              # be explicit; device paths drift
---
apiVersion: v1
kind: Secret
metadata:
  name: node-cp-01-bmc
  namespace: metal3
type: Opaque
stringData:
  username: admin
  password: <from your secrets store, not git>

What one apply sets in motion

This is the step that trips up newcomers to Cluster API: "install the OS" and "join the control plane" are not two things you do in sequence. They are two providers reconciling one declarative spec. When you apply a control-plane that wants three replicas, the BareMetalHost walks a state machine, and the OS install, config delivery, and cluster join fall out of it automatically:

graph TD
  A["BareMetalHost registered<br/>BMC address + credentials"] --> B["Inspecting<br/>Ironic reads CPU / RAM / NICs / disks"]
  B --> C["Available<br/>host waits in the pool"]
  C -->|CAPI wants a control-plane Machine| D["Provisioning<br/>Redfish powers on, PXE boots the<br/>Ironic Python Agent"]
  D --> E["IPA writes the Talos disk image<br/>+ injects the machine config from CABPT"]
  E --> F["Provisioned<br/>reboots into Talos, applies the config"]
  F --> G["CACPPT bootstraps etcd on node 1,<br/>rolls to 3 control-plane replicas"]
  G --> H["Node NotReady<br/>cni: none, nothing routes pods yet"]
  H -.->|next article| I["Cilium → Ready"]

Declaring the control plane

The control plane is three resources working together. The Cluster ties an infrastructure reference (Metal3Cluster, which carries the API endpoint) to a control-plane reference (TalosControlPlane). The TalosControlPlane says how many replicas, at what version, on what machine template, and carries the Talos config patches. The Metal3MachineTemplate says which Talos disk image to write and which hardware to select. The two patches that define a bare-metal Talos node are here: the CNI is set to none because Cilium comes later, and kube-vip is dropped in as a static pod so the API VIP is up before the CNI is.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod
  namespace: metal3
spec:
  controlPlaneRef:
    kind: TalosControlPlane
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    name: prod-cp
  infrastructureRef:
    kind: Metal3Cluster
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    name: prod
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Cluster
metadata:
  name: prod
  namespace: metal3
spec:
  controlPlaneEndpoint:
    host: 10.0.1.1        # the kube-vip VIP, advertised by BGP
    port: 6443
  noCloudProvider: true
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: TalosControlPlane
metadata:
  name: prod-cp
  namespace: metal3
spec:
  version: v1.34.0
  replicas: 3
  infrastructureTemplate:
    kind: Metal3MachineTemplate
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    name: prod-cp
  controlPlaneConfig:
    controlplane:
      generateType: controlplane
      talosVersion: v1.11
      configPatches:
        - op: add
          path: /cluster/network/cni
          value: { name: none }          # Cilium installs in part two
        - op: add
          path: /cluster/proxy
          value: { disabled: true }       # Cilium replaces kube-proxy
        - op: add
          path: /machine/pods             # kube-vip as a Talos static pod
          value:
            - apiVersion: v1
              kind: Pod
              metadata: { name: kube-vip, namespace: kube-system }
              spec:
                # ...kube-vip container, BGP mode, address 10.0.1.1
                hostNetwork: true

Node IPs without a DHCP lease

The last piece is deterministic addressing. Rather than let DHCP hand out whatever it likes, Metal3's IP-address-manager owns a static IPPool, and a Metal3DataTemplate wires each provisioned machine's network to it. CAPM3 claims an address from the pool at provision time, so a host knows its IP before it boots. The Metal3MachineTemplate points at the Talos raw image to write and selects hardware from the enrolled pool:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: prod-cp
  namespace: metal3
spec:
  template:
    spec:
      image:
        url: https://factory.talos.dev/image/<schematic>/v1.11.0/metal-amd64.raw.zst
        checksum: https://factory.talos.dev/image/<schematic>/v1.11.0/sha256sum.txt
        checksumType: sha256
        format: raw.zst
      dataTemplate:
        name: prod-cp-metadata      # ties the machine to the IPPool below
      hostSelector:
        matchLabels: { role: control-plane }
---
apiVersion: ipam.metal3.io/v1alpha1
kind: IPPool
metadata:
  name: provisioning
  namespace: metal3
spec:
  pools:
    - start: 10.0.2.20
      end: 10.0.2.60
  prefix: 24
  gateway: 10.0.2.1

NotReady, on purpose

Apply all of it and, minutes later, three physical servers you never plugged a monitor into report in as a Kubernetes control plane: etcd bootstrapped, three apiservers behind a BGP-advertised VIP, every node a byte-identical immutable Talos image. And every one of them NotReady. That is not a failure; it is the seam this design planned for. The machine config said cni: none, so nothing routes pod traffic yet, and the whole cluster sits waiting for a network. Scaling is now a replica count, upgrades are a version bump the control-plane provider rolls out, and a dead node is a BareMetalHost the system re-provisions from scratch. The pets are gone; what is left is cattle that happen to weigh forty kilos each. Part two gives them a network: Cilium, kube-proxy-free, with its own BGP and load balancing, and the moment the nodes turn Ready.