Writeups

A complete production platform, from a powered-off server to a self-healing, multi-datacenter AI platform, owning every layer a cloud would rent.

We were six minor versions behind, on EKS extended support, with a hard deadline before AWS force-upgraded the cluster out from under us. So we skipped the six in-place hops and did one blue-green cutover from 1.30 to 1.36. Here is how it went, what to watch for, and the boring habit that would have made the whole thing unnecessary.

We left Istio's sidecar mesh for Cilium and assumed we kept mTLS along the way. We did not. The feature branded "mTLS-based" is mutual authentication, not encryption, and the difference is the whole point of mTLS. Here is how the name fooled us, why the maintainers agreed it was wrong, and why the real fix is still out of reach.

Cluster API is to clusters what Deployments are to Pods: declare the fleet, let controllers reconcile it. With v1.12 adding in-place updates and chained upgrades, the case for running CAPI from day 0, and pairing it with Karpenter, got a lot stronger.

Gateway API is the Kubernetes-native successor to Istio's Gateway and VirtualService, and for a single service the migration is a mechanical, two-resource translation. The interesting part comes after: ListenerSets, standard in v1.5, let every team bring their own listeners and certificates to one shared gateway, and cert-manager makes those certificates issue and renew themselves. One front door, many owners.

Multi-primary was built for independence, primary-remote was built for economy, and Cilium ClusterMesh answers the same question from a different layer entirely. A deep dive into how Istio spans clusters, and then a full working architecture: two region primaries serving traffic active-active with per-service failover and no edge device, plus one shared satellite in a second cloud for burst capacity and legacy data, all with a trust chain no cluster ever holds.

Moving our dataplane to Cilium made Tetragon the obvious next step for runtime security. Then a feasibility pass showed why a clean cutover was not possible yet. Here is what we found, and why I still want to make the move.

Long-lived secrets are a liability that only grows. Chain OpenBao, the External Secrets Operator, and Reloader, and rotation becomes something that happens on its own, at three in the morning, with no human and no downtime.

A real Kubernetes service is thousands of lines of YAML nobody wants to own. So I hid all of it behind twenty fields. Here is what that does for developers, and why the YAML engineer isn't quite dead yet.

A platform is only as good as the window developers see it through. Ours is Backstage: one catalog, one search, one place where every service, its docs, its running state, and its infrastructure spec all live.

Anthropic's Responsible Scaling Policy decides when a model needs heavier guardrails. Constitutional Classifiers are the guardrail that actually holds the deployment line, and their next generation makes them cheap enough to leave on.

I spent a while running PentAGI, an autonomous AI agent for penetration testing. The capabilities are real. What stuck with me is where the only real safeguard actually lives, and how easily self-hosting removes it.

A disruption budget set to 100%, a spot interruption, and an AI coding assistant: how a new service went partly dark, and the systemic fixes that followed.

Part one of a series on running Kubernetes on bare metal the way serious compute fleets do: declaratively, from an API, with no SSH and no snowflakes. This piece is the foundation: how a physical server goes from a powered-off box to a Kubernetes control-plane node, using Cluster API to orchestrate Metal3 for the hardware and Talos for the OS. It stops the moment the nodes come up, deliberately, NotReady.

Part two of the bare-metal series. Part one left three control-plane nodes running and deliberately NotReady, because the machine config said cni: none. This piece gives them a network: Cilium, in eBPF, with native routing advertised over BGP, WireGuard on the wire, kube-proxy deleted, and LoadBalancer IPs handed out by the same BGP that carries the API VIP. The moment it lands, the nodes turn Ready.

Part three of the bare-metal series. Part two left a cluster that is Ready but empty, waiting for a human to run helm. This piece gives it a brain: Argo CD, installed first by the add-on provider and then managing itself, delivering the whole platform from git, and turning a labelled pull request into a preview, a promotion, and a self-healing production release. No pipeline pushes to the cluster ever again.

Part four of the bare-metal series. Part three made the cluster install and run itself from a git repository anyone can read, which is exactly why secrets cannot live there. This piece gives the platform a secrets layer that keeps the GitOps model intact: OpenBao unsealed by a hardware security module, configured from git by Crossplane, and feeding values to workloads two different ways depending on how sensitive they are. Nothing sensitive is ever committed.

Part five of the bare-metal series. So far the cluster provisions, networks, delivers, and secures itself, but it cannot remember anything: it has no storage of its own. This piece turns the raw NVMe disks in the rack into a resilient, replicated storage layer with Rook and Ceph, one cluster serving block, file, and object, tuned for an immutable OS and spread so a whole rack can fail without losing a byte.

Part six of the bare-metal series. The platform can store things now, but nothing outside can reach it and nothing inside is governed. This piece handles traffic in both directions: north-south through a Cilium gateway on an LB-IPAM IP, east-west through a sidecar-less Cilium mesh, with cert-manager minting the certificates and external-dns publishing the names. One dataplane does all of it, because the cluster already runs Cilium for everything underneath.

Part seven of the bare-metal series. The platform can store, expose, and govern traffic now, but it still pulls its images from the public internet and trusts whatever turns up. This piece closes that gap: Harbor running on the Ceph object store from part five, fronted by the gateway from part six, every node pulling through it, and a signing-and-verification chain that ends in a hard admission gate. Nothing runs unless the platform built it, scanned it, and signed it.

Part eight of the bare-metal series. The platform can build, store, route, gate, and sign itself now, but it cannot see itself. This piece gives it sight: four telemetry signals on Grafana's LGTM stack plus Pyroscope, one eBPF-aware agent shipping all of it, the mesh and the models and the storage and the iron underneath all made legible, every backend stored on the Ceph object store from part five, and an alert that finally pages carrying its own root cause.

Part nine of the bare-metal series. The platform can see itself now; this part gives it reflexes. It scales pods out on the signals from part eight, right-sizes them in place, and grows the cluster itself when workloads go Pending. The twist is physical: a scale-up powers a real server on, a scale-down powers it off, and the ceiling is the iron you actually own.

Part ten of the bare-metal series. The platform can grow itself now, and a bigger platform is a bigger attack surface. This part hardens what runs on it in layers: admit only safe configuration, wrap untrusted code in its own kernel, and promote the Tetragon signal from part eight from watching a forbidden action to killing it in the kernel before it completes. Defense in depth, with the honest gaps named.

Part eleven of the bare-metal series. Ten parts built a machine of real power, and power that only its builders can operate is not a platform. This part is the front door: Crossplane turning one short spec into the whole hardened stack, Backstage to ship it, OpenCost pricing a cluster with no bill, and Teleport reaching a cluster with no shells. The platform becomes a product, and the fast path becomes the safe path.

Part twelve of the bare-metal series, redeeming the promise from part eleven that a database is a field, not a project. Behind one engine flag sits a shelf of ten very different data stores, from Postgres to ClickHouse to a Ceph bucket, each a purpose-built operator you run yourself, on Ceph, with dynamic credentials and backups to your own object store. The self-hosted managed service, and the operational knowledge it hides.

Part thirteen of the bare-metal series, the accelerators the whole platform exists to serve. A GPU is where a cloud hides the most and bare metal exposes the most: the driver on an immutable OS, the partitioning of a card that costs as much as a car, and the interconnect fabric that lets GPUs in different boxes train as one. This part owns all of it, and a developer still just asks for a GPU.

Part fourteen of the bare-metal series. Thirteen parts assumed one building; this one adds a second. The appealing mistake is to melt the two sites into one big cluster, and it is a trap, because etcd cannot keep quorum across a WAN. The discipline is to keep them as two autonomous clusters and connect them: a KubeSpan WireGuard mesh, Cilium ClusterMesh, state replicated not stretched, and users routed to the nearest healthy site, so losing a datacenter is an incident, not an outage.

Part fifteen, and the finale. A platform is not finished when it is built but when it can be run for years without heroics. This part is day two: upgrades that are a field, not an event; nodes that heal themselves by reprovisioning the physical machine; clusters rebuilt from git and a snapshot; and a fleet run from one place. It closes the loop the series opened, because the machinery that first brought a server to life is the machinery that keeps it alive. Cattle, all the way down.