k8s on bare-metal - Part 13 : GPU
Part thirteen of the bare-metal series, the accelerators the whole platform exists to serve. A GPU is where a cloud hides the most and bare metal exposes the most: the driver on an immutable OS, the partitioning of a card that costs as much as a car, and the interconnect fabric that lets GPUs in different boxes train as one. This part owns all of it, and a developer still just asks for a GPU.
Every part so far has quietly assumed the same kind of machine: a server with CPUs and disks and NICs. The whole reason this platform exists undermines that assumption, because the models it serves and trains do not run on CPUs, they run on accelerators, and an accelerator is where a cloud hides the most and bare metal exposes the most. A cloud sells you a GPU as an instance type with the driver, the sharing, and the interconnect already solved. Here you own all of it: the kernel module on an immutable OS, the partitioning of a card that cost as much as a car, and the high-speed fabric that lets GPUs in different boxes train as one. To be clear about scope, the accelerator here is the GPU you can actually rack, NVIDIA in the main, with AMD Instinct and Intel Gaudi as real alternatives. The Trainium and Inferentia silicon from a certain cloud is rented, not owned, so it has no place in a datacenter you built.
The silicon the models run on
This is the layer the previous twelve parts were quietly building toward. A platform that provisions, secures, scales, and persists is table stakes for running services; the reason to build all of it on bare metal in the first place is to run the models, and models run on GPUs. So this part is about owning the accelerator end to end, every piece of which a cloud would have handed you pre-solved and marked up. The interesting thing is how much there is to own, and how much of it reuses machinery the earlier parts already built.
The last of the ordinary-hardware layers. From here the platform stops assuming CPUs and starts owning accelerators.
The anatomy of a GPU node
A GPU node is a stack, and on bare metal you assemble every layer of it yourself. At the bottom is the hardware you racked; at the top is a pod that just asked for a GPU; and in between is everything that makes the first serve the second. It is worth seeing the whole stack once before pulling any layer apart, because each layer is a thing a cloud instance type hands you for free and you now own. Tap through it:
The hurdle a cloud AMI hides. There is no shell to install a driver on, so the NVIDIA kernel module and container toolkit are baked into the immutable Talos image as system extensions, delivered by the Image Factory from part one.
A driver on an immutable OS
Start at the layer that breaks first, the driver. On an ordinary node you install the NVIDIA driver with a package manager. Talos, from part one, has no package manager, no shell, and an immutable root, so there is nothing to install onto. The answer is to bake the driver into the OS image itself: the NVIDIA kernel module and the container toolkit are Talos system extensions, added to the boot image by the Image Factory as a schematic. Once the node boots with the driver already present, the NVIDIA GPU Operator does the Kubernetes half, but told explicitly not to manage the driver, since Talos already has it. What is left for the operator is the device plugin that advertises the GPUs, DCGM for metrics, and the MIG manager for partitioning:
# The driver arrives as Talos system extensions, baked into the boot image by
# the Image Factory from part one (a schematic, not a package install).
customization:
systemExtensions:
officialExtensions:
- siderolabs/nonfree-kmod-nvidia-production
- siderolabs/nvidia-container-toolkit-production
---
# The NVIDIA GPU Operator, told NOT to manage the driver (Talos already has it).
driver: { enabled: false } # use the host driver from the extension
toolkit: { enabled: false } # same: provided by the extension
devicePlugin: { enabled: true } # advertise GPUs to the kubelet
dcgmExporter: { enabled: true } # metrics into Mimir (part 8)
migManager: { enabled: true } # apply MIG partition profilesWhere the Image Factory came from: the schematic that builds a custom Talos image now bakes the GPU driver into the immutable OS.
The fabric you own
The single biggest thing a cloud hides behind a GPU instance is the network between the GPUs, and it is the single biggest thing you own on bare metal. Inside a box, GPUs talk over NVLink through an NVSwitch at terabytes per second. Across boxes, which is what large training needs, they talk over a separate high-speed fabric, InfiniBand or RoCE, that you design and cable yourself, ideally rail-optimized so every GPU has a clean path to its peers. GPUDirect RDMA lets a GPU move data straight to the NIC without troubling the CPU. Kubernetes reaches this fabric the same way it reached the Ceph storage network in part five, through Multus, with the NVIDIA Network Operator and SR-IOV provisioning the RDMA devices. The Cilium pod network is still there for ordinary traffic; this is a second, faster network laid alongside it for the GPUs:
# The InfiniBand network is a second attachment, wired by the NVIDIA Network
# Operator and reached through Multus, the same mechanism the Ceph net used.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata: { name: ib-rdma, namespace: training }
spec:
config: '{ "type": "ib-sriov", "ipam": { "type": "whereabouts" } }'
---
# A training pod asks for GPUs and a rail on the fabric at once.
apiVersion: v1
kind: Pod
metadata:
annotations: { k8s.v1.cni.cncf.io/networks: ib-rdma } # attach the IB rail
spec:
containers:
- name: trainer
resources:
limits:
nvidia.com/gpu: 8
rdma/rdma_shared_device_a: 1 # a GPUDirect RDMA deviceWhere Multus first split a second network off the pod network: the same mechanism that gave Ceph its storage VLAN now attaches the InfiniBand rail.
Two workloads, two schedulers
Two very different workloads share these cards, and they want opposite things from the scheduler. Inference wants to be elastic and cheap when idle, so it scales to zero on KEDA from part nine and comes back on demand, often on a fraction of a card. Training wants the opposite: many whole GPUs at once, placed close together on the fabric, held for hours or days. The danger with training is partial scheduling, a distributed job that gets half its GPUs and deadlocks while holding them, so it is gang-scheduled through Kueue, which admits a job only when all of its GPUs are free. Underneath both, Dynamic Resource Allocation is the modern way a pod requests a GPU, expressing the sharing and topology the old device-plugin resource could not:
# The modern request: Dynamic Resource Allocation, which can express sharing
# and topology the old device-plugin resource could not.
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata: { name: one-h100, namespace: inference }
spec:
devices:
requests:
- name: gpu
deviceClassName: gpu.nvidia.com # a whole H100, or a MIG slice
---
# Training queues through Kueue, gang-admitted only when ALL its GPUs are free,
# so a half-scheduled distributed job never deadlocks holding idle cards.
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: { name: training }
spec:
resourceGroups:
- coveredResources: ["nvidia.com/gpu"]
flavors:
- name: h100
resources: [{ name: nvidia.com/gpu, nominalQuota: 64 }]- Scheduler
- KEDA, scale to zero
- GPUs
- fractional, MIG, one node
- Fabric
- single node
- Shape
- long-lived, bursty
- Wants
- low latency per request
- When idle
- scaled to zero, nothing runs
- Scheduler
- Kueue, gang scheduled
- GPUs
- many whole cards, topology-tight
- Fabric
- multi-node, RDMA over InfiniBand
- Shape
- batch, hours to days
- Wants
- raw throughput to finish sooner
- When idle
- queued; runs when its GPUs free up
The same GPUs serve both, but the scheduler treats them as opposites: inference is elastic and latency-bound, so it scales to zero and back; training is a batch that needs all its GPUs together or it deadlocks, so it queues until they are free. One cluster, two disciplines.
The model-serving stack this schedules: vLLM behind an Envoy AI Gateway, the runtime layer that sits on top of the GPUs this part provisions.
Watching the watts
Two signals matter more on GPUs than anywhere else on the platform: health and heat. DCGM, running under the GPU Operator, exports each card's utilization, memory, ECC error counts, temperature, and power draw into the Mimir from part eight, so a card that is throttling or throwing correctable errors is visible before a training run silently slows or corrupts. And power is not a footnote here: an H100 pulls around 700 watts, so a rack of them is a serious electrical and cooling load, which the BMC watt metrics track and the cost model from part eleven turns into by far the largest line on any team's bill:
# DCGM exports each card's vitals into Mimir: utilization, memory, ECC errors,
# temperature, and power. On GPUs, heat and health lead the dashboard.
- alert: GpuThermalThrottle
expr: DCGM_FI_DEV_GPU_TEMP > 87 # celsius; sustained heat is a throttle
for: 5m
- alert: GpuEccErrors
expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0 # uncorrectable memory
# And the OpenCost model from part eleven gains a GPU rate, the largest by far.Where these land: DCGM ships GPU utilization, temperature, and power into the same Mimir as every other signal, and onto the same dashboards.
What this buys
The platform can now run the thing it was always for. The models it serves and the jobs that train them land on accelerators that were bought, racked, driven, partitioned, wired, scheduled, and watched by the same platform that handles everything else, and a developer still just asks for a GPU. Every hard part a cloud hides behind an instance type, the driver on an immutable OS, the sharing of an expensive card, the fabric between the boxes, is owned here, and owning it is the point of building on bare metal in the first place. There is one assumption left to challenge, though. All of this has lived in a single datacenter. The next part asks what happens when there are two, and how a cluster stretches across sites without losing the properties the last thirteen parts worked to build.
Next: two sites. Why you must not stretch one cluster across them, and how a fleet of autonomous clusters spans a WAN instead.