k8s on bare-metal - Part 10 : Runtime security
Part ten of the bare-metal series. The platform can grow itself now, and a bigger platform is a bigger attack surface. This part hardens what runs on it in layers: admit only safe configuration, wrap untrusted code in its own kernel, and promote the Tetragon signal from part eight from watching a forbidden action to killing it in the kernel before it completes. Defense in depth, with the honest gaps named.
Everything up to here made the platform capable: it provisions, reconciles, stores, routes, sees, and scales itself. Capability is attack surface. A cluster that can turn on new servers, run any signed image, and serve models to the internet is a cluster worth attacking, and the honest response is not one security product but layers, each assuming the one outside it will sometimes fail. This part builds four: admission policy so unsafe configuration never becomes a pod, sandboxing so untrusted code cannot reach the host, runtime enforcement so a compromised workload is stopped mid-action, and a baseline check so none of it silently drifts. Two of these reuse tools already installed, Kyverno from part seven and Tetragon from part eight, which is the point: hardening is mostly a matter of turning existing capability from advisory into binding.
A bigger surface to defend
Security that lives in one place is security with a single point of failure. The model that holds up is defense in depth: concentric layers around a workload, where each layer does one job and assumes the others will occasionally be bypassed. If admission misses something, isolation contains it; if isolation is escaped, runtime enforcement stops it; and underneath, a baseline check makes sure the walls are still standing. Tap through the layers before the detail:
The capability this hardens: a platform that can now grow itself and serve models is a platform worth defending in depth.
Defense in depth at runtime
The four layers are not alternatives, they are a stack you pass through. Most of what stops an attack happens at the outermost one, admission, because a manifest that never becomes a pod cannot do anything at all. The inner layers exist for what gets past it. Here is the shape:
The outermost gate. Only configuration that clears policy is allowed to become a pod at all: no privilege, no host access, resource limits set, a signed image. Most attacks are stopped here, before anything runs, because the manifest never gets in.
Admit: Kyverno beyond image signing
Part seven installed Kyverno to verify image signatures, which turns out to be one of four things it does at admission. Runtime security uses the other three too. It validates, rejecting any pod that runs privileged, mounts a host path, asks for host networking, omits resource limits, or runs as root. It mutates, quietly adding secure defaults like a dropped capability set and a read-only root filesystem so teams do not have to remember them. And it generates, giving every new namespace a default-deny NetworkPolicy and a quota so nothing is ever open by omission. The image-verification rule from part seven sits alongside these as one policy among many. Tap through the four:
The workhorse: reject any pod that breaks a rule. No privileged containers, no hostPath or hostNetwork, resource limits required, must run as non-root, a seccomp profile set. A manifest that fails never becomes a pod.
validate:
failureAction: Enforce
pattern:
spec:
=(securityContext):
runAsNonRoot: true
containers:
- securityContext:
privileged: "false"
allowPrivilegeEscalation: "false"Where Kyverno arrived: the VerifyImages rule that admits only signed, attested images is one of the four rule types this part builds on.
The floor, and the native path
Kyverno is the expressive layer, but two lighter mechanisms sit with it. Pod Security Admission is the built-in floor: a label on each namespace enforces one of three Pod Security Standards, and the platform default is restricted, the strictest, with a named exception only where a component genuinely needs it, the way rook-ceph needed a privileged label back in part five. And ValidatingAdmissionPolicy is the newer in-tree option, a CEL expression evaluated inside the API server with no webhook to run or keep available, ideal for a simple invariant you want enforced even if Kyverno is down. The three compose: PSA sets the coarse floor, VAP catches simple invariants natively, and Kyverno expresses everything richer.
# The floor: every namespace is restricted unless it says otherwise.
apiVersion: v1
kind: Namespace
metadata:
name: apps
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
---
# The native path: a CEL invariant in the API server, no webhook.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: no-latest-tag }
spec:
validations:
- expression: "object.spec.containers.all(c, !c.image.endsWith(':latest'))"
message: "images must be pinned, not :latest"Isolate: a sandbox for code you don't trust
Admission judges configuration; it cannot judge intent. Some workloads on an AI platform are things the platform did not write and has no reason to trust: an agent running its own tools, code a model generated, a tenant's job. For those, container isolation on a shared kernel is too thin, because a single kernel bug is a path to every other pod on the node. The answer is a RuntimeClass backed by Kata Containers, which gives such a workload its own kernel inside a lightweight VM. A breakout then lands on a guest kernel and a virtualization boundary instead of the host, and on bare metal you own the hardware virtualization, so the cost is small. Trusted platform services keep running on plain runc; only what needs the wall asks for it:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: kata }
handler: kata-qemu
---
# An untrusted workload opts into the VM boundary by naming the RuntimeClass.
apiVersion: v1
kind: Pod
metadata: { name: agent-sandbox, namespace: untrusted }
spec:
runtimeClassName: kata # its own kernel; a breakout stays in the VM
containers:
- name: tool
image: harbor.example.com/agents/tool@sha256:...A container breakout lands on the host kernel every other pod shares. Fine for code you wrote and trust.
A breakout hits the guest kernel and the VM boundary first, not the host. Stronger isolation for a small overhead.
A RuntimeClass picks which one a pod gets. Platform services run on runc; anything running code the platform did not write, an agent's tools, a model's output, a tenant's job, gets the VM boundary by asking for the Kata RuntimeClass.
Exactly the workload this isolates: an autonomous agent running a real offensive toolkit, code you must run but cannot trust.
The network stays shut by default
Isolation is not only about the kernel; it is about what a workload can reach. The Cilium dataplane from parts two and six already enforces network policy in eBPF, and the hardening posture is simply to make the default deny. Every namespace gets a default-deny policy at creation, generated by the Kyverno rule above, so a pod can talk to nothing until an explicit CiliumNetworkPolicy allows it, ideally scoped to L7. A compromised workload behind that default is loud and stuck: its first unexpected connection is denied and visible in Hubble. Calico offers a comparable policy model for clusters not on Cilium, but here the CNI is already the enforcement point, so there is nothing extra to run.
The enforcement point this leans on: Cilium network policy and Hubble, already in the dataplane, now set to deny by default.
Enforce: from watching to killing
This is the promise part eight deferred. Tetragon has been observing from the kernel, turning syscalls, process launches, and file access into an audit signal. Promoting it to enforcement is a single field. The same TracingPolicy that recorded a process opening a sensitive file gains a matchAction, and now the kernel sends SIGKILL before the read returns, so the action is not logged after the fact, it is prevented. Because it runs on the same eBPF datapath as the CNI, there is no sidecar and no userspace round trip; the decision happens in the hook. Toggle the one field and watch the outcome change:
selectors:
- matchArgs:
- index: 0
operator: Prefix
values: ["/etc/shadow"]The read succeeds and an event lands in Loki. You have a perfect record of the breach, which you read after the secret is already gone.
Same hook, same policy, one added field. That is the promotion this part is about: the signal from part eight becomes a control.
The reasoning behind choosing Tetragon: the same eBPF datapath that observes can enforce, and the honest gap in what translates.
What Tetragon can't see
Honesty is part of a security design, so here is the gap. Tetragon enforces on things the kernel can see: a syscall, a process, a file, a socket. It cannot see actions that never touch the kernel of a node, because they happen at the Kubernetes API instead, someone with a stolen token creating a RoleBinding, exec-ing into a pod, or reading a Secret. Those have no kernel hook to attach to, so a second source covers them: the Kubernetes API audit log, shipped to the Loki from part eight, where a policy flags the sensitive verbs and an alert fires. Admission stops bad configuration, Tetragon stops bad behavior on a node, and the audit log catches bad behavior against the control plane. The three together have far fewer blind spots than any one of them.
Where the audit log lands: the Kubernetes API audit stream ships to the same Loki, so control-plane actions are searchable and alertable.
Verify the baseline
Layers assume a hardened foundation, so the last job is to keep checking that it is. kube-bench runs the CIS Kubernetes benchmark against the cluster, and the pleasant surprise on this platform is how much already passes, because Talos from part one is hardened by construction: no shell, no SSH, an immutable read-only root, a locked-down kubelet, and a minimal attack surface with nothing extra installed to misconfigure. Most CIS control-plane and node checks are green out of the box, which turns kube-bench from a remediation project into a regression test. kubescape adds the broader posture view for the NSA and CISA hardening guidance on top. The point is not a one-time score; it is a scheduled check that fails loudly the day something drifts.
Why most of the CIS benchmark is already green: Talos ships hardened by construction, with no shell, no SSH, and an immutable root.
What this buys
The platform is now hard to misuse from any single angle. Unsafe configuration never becomes a pod; untrusted code runs boxed in its own kernel and can reach nothing by default; a workload that turns malicious at runtime is killed in the kernel mid-action; control-plane abuse is caught in the audit log; and the whole hardened baseline is checked on a schedule against drift. None of it is a product bolted on the side. It is the admission engine from part seven, the eBPF dataplane from part six, and the runtime signal from part eight, each turned from advisory into binding, plus a sandbox for the code that deserves one. The platform can build, store, route, gate, see, scale, and now defend itself. What it still cannot do is let the humans who use it move quickly, and that is the next thing to build: a developer platform, a self-service front door onto everything these ten parts assembled.
Next: the front door. One short spec becomes the whole hardened stack, a portal to ship it, and cost and access for a cluster you own.