k8s on bare-metal - Part 5 : Storage

The cluster with no memory

A pod that writes to its local disk loses everything when it moves, which is most of the time, so real workloads need a PersistentVolume backed by storage that outlives any node. The platform has none yet. What it does have is disks: NVMe drives in every storage node across every rack, currently doing nothing. Ceph turns a pile of disks like that into a single distributed storage system that replicates data, survives hardware failure, and grows by adding more disks, and Rook runs Ceph as a set of Kubernetes resources so the whole thing is declarative like everything else in this series.

Three storage types from one cluster

The reason to run Ceph rather than something simpler is that one cluster serves all three shapes of storage a platform needs, from the same pool of disks. Block for single-writer volumes, a shared filesystem for the read-write-many case, and an S3-compatible object store for everything that speaks buckets. Each is a Rook custom resource that produces a StorageClass, and a workload just asks for the class it needs. Tap through them:

Raw disks on an OS that won't install anything

This is where Ceph meets Talos, and it is the section that earns the part, because Ceph expects to install packages and load modules on a normal host and Talos allows neither. Everything Ceph needs is arranged declaratively in the machine config instead: the kernel modules it depends on are loaded at boot, its per-node state directory is bind-mounted so it survives reboots, and, the counterintuitive bit, the data disks are handled by leaving them entirely alone. Tap through the four pieces:

machine:
  kernel:
    modules:                       # Ceph cannot load these itself on Talos
      - name: rbd
      - name: ceph
  kubelet:
    extraMounts:
      - destination: /var/lib/rook  # Rook's per-node state, kept across reboots
        source: /var/lib/rook
        type: bind
        options: [bind, rshared]
# machine.disks lists ONLY the OS disk. The NVMe data disks are deliberately
# absent, so Talos never formats or mounts them and Rook can claim them raw.

machine:
  kernel:
    modules:
      - name: rbd
      - name: ceph

The shape on the rack

Before the YAML, the picture. The Rook operator sits to the side and orchestrates; it is never in the data path. On each storage node the Ceph daemons run, the monitors that hold cluster state, the managers, the metadata servers for the filesystem, the gateways for object, and one OSD per whole NVMe disk, each holding a shard of the data. A Multus split keeps the heavy replication traffic on its own storage VLAN, away from client I/O. And the failure domain is the rack:

From raw disk to OSD

With the disks left raw and the operator installed, turning a disk into usable capacity is Rook's job, not yours. It discovers the NVMe devices by a stable path filter, wipes and claims each one, and brings up an OSD per disk that joins the CRUSH map at its rack. A CephCluster resource describes the whole thing: three monitors for quorum, the Multus networks, msgr2 encryption on the wire, and an explicit device filter so a typo can never hand Ceph the OS disk:

One caveat earns a mention, because it is the most common reason an OSD never appears: Rook only claims a disk that is genuinely blank. A drive with a leftover partition table, an old filesystem, or a previous cluster's signature is silently skipped, and since Talos has no shell to wipe it in place, a reused disk has to be zapped out of band before its node is enrolled. It is worth pinning a known-good Ceph image too, because a few point releases have misread hot-swap NVMe as removable and refused perfectly good drives.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata: { name: rook-ceph, namespace: rook-ceph }
spec:
  dataDirHostPath: /var/lib/rook
  mon: { count: 3 }                 # odd quorum; loses one, stays up
  mgr: { count: 2 }
  network:
    provider: multus
    selectors:
      public:  rook-ceph/public-net
      cluster: rook-ceph/cluster-net   # storage VLAN, jumbo 9000
    connections:
      requireMsgr2: true
      encryption: { enabled: true }
  storage:
    useAllNodes: false               # never auto-claim across the fleet
    useAllDevices: false
    devicePathFilter: "^/dev/disk/by-path/.*-nvme-.*"   # stable across reboots
  placement:
    all:                             # OSDs only on labelled storage nodes
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - { key: role.storage, operator: In, values: ["true"] }
  disruptionManagement: { managePodBudgets: true }

A rack can burn

How the data is protected is a per-pool choice, and it comes down to failure domain and redundancy scheme. Because every rack carries storage, the failure domain is the rack: Ceph places copies so that no two live in the same one, and a whole rack can lose power without the cluster losing data. The redundancy scheme is where the hot path and the bulk path diverge. Toggle the two:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata: { name: replicapool, namespace: rook-ceph }
spec:
  failureDomain: rack               # no two copies share a rack
  replicated: { size: 3 }           # three copies, three racks

Block and file

The two replicated pools cover the everyday cases. The block pool backs the default StorageClass, ceph-block: a PersistentVolumeClaim gets an RBD image, ReadWriteOnce, replicated across three racks, the disk a database or any single-writer workload wants. The filesystem adds the read-write-many case that block cannot do, a shared POSIX mount many pods can write to at once, coordinated by the metadata server and exposed as the ceph-filesystem class. Both classes keep reclaimPolicy set to Retain, so deleting a claim leaves the data in place for a deliberate cleanup rather than reclaiming it on the spot, and the block class turns on the modern RBD image features the recent Talos kernel supports, fast-diff, object-map, and deep-flatten, for cheaper snapshots and clones. That is the block-and-file path in the explorer above.

# The shared, read-write-many filesystem alongside the default block class.
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata: { name: shared, namespace: rook-ceph }
spec:
  metadataPool: { replicated: { size: 3 } }
  dataPools:
    - name: data0
      failureDomain: rack
      replicated: { size: 3 }
  metadataServer: { activeCount: 1, activeStandby: true }
# -> StorageClass "ceph-filesystem", provisioner rook-ceph.cephfs.csi.ceph.com,
#    which hands out ReadWriteMany volumes.

Object: the S3 that isn't a cloud

The third type is the one that changes what the rest of the platform can assume, the object path in the explorer above. The RADOS Gateway serves an S3-compatible API from an erasure-coded pool, and a workload gets a bucket by creating an ObjectBucketClaim, which hands back an endpoint and credentials in a Secret. That is the internal S3 the later parts depend on: Harbor stores registry layers in it, the LGTM stack keeps its long-term metrics, logs, and traces there, backups land there, and none of it touches a cloud. The bucket credentials flow to consumers through OpenBao and the external-secrets operator from the previous part, so even the S3 keys stay in the secrets model:

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata: { name: s3, namespace: rook-ceph }
spec:
  metadataPool: { failureDomain: rack, replicated: { size: 3 } }
  dataPool:
    failureDomain: rack
    erasureCoded: { dataChunks: 4, codingChunks: 2 }   # ~1.5x, needs >=6 racks
  gateway: { instances: 3, port: 80 }
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata: { name: harbor }
spec:
  generateBucketName: harbor
  storageClassName: ceph-bucket

Installed by git, upgraded with care

All of it arrives the way everything else does: the Rook operator and the CephCluster are Argo CD applications, ordered by sync waves so the operator is healthy before the cluster asks for OSDs, and the pools and StorageClasses land after the cluster is up. Two operational notes matter more on Ceph than elsewhere. Upgrades are done one node at a time, waiting for the cluster to report HEALTH_OK before touching the next, because Ceph is actively rebalancing data as nodes come and go, and pod disruption budgets (managePodBudgets) let Rook hold the line during maintenance. And because Talos never formatted the data disks, tearing a node down for reprovision means wiping them deliberately, since Rook left its own signature on them.

What the cluster can finally remember

The platform now has a memory. A PersistentVolumeClaim gets a replicated block device, a read-write-many workload gets a shared filesystem, and anything that speaks S3 gets a bucket, all from the NVMe in the rack, all placed so a rack can fail, all declared in git. Every stateful thing the rest of the series stands up, the registry, the observability stack, the databases, now has somewhere durable to live. The next piece is the front door: how traffic from outside reaches all of this, with Gateway API and Cilium load balancing, cert-manager minting certificates, and external-dns publishing the names, so the services the platform is about to grow are actually reachable.