k8s on bare-metal - Part 11 : IDP
Part eleven of the bare-metal series. Ten parts built a machine of real power, and power that only its builders can operate is not a platform. This part is the front door: Crossplane turning one short spec into the whole hardened stack, Backstage to ship it, OpenCost pricing a cluster with no bill, and Teleport reaching a cluster with no shells. The platform becomes a product, and the fast path becomes the safe path.
Ten parts built a machine of real power: it provisions its own hardware, reconciles from git, keeps its secrets, stores its data, routes and secures its traffic, watches itself, scales itself, and defends itself. And almost none of that power is any use if only the people who built it can operate it. A platform that requires platform expertise to use is not a platform, it is a second job for every team. This part is the front door: the layer that turns ten layers of capability into a product a developer consumes without knowing it is there. It has four faces, one for each thing a human needs from the platform, to describe what they want, to do it through an interface, to know what it costs, and to get in.
The platform becomes a product
There is a failure mode that quietly ruins good infrastructure: it works, it is powerful, and using it requires a member of the team who built it. Every new service becomes a conversation, every config a ticket, and the platform team becomes the bottleneck they were trying to remove. The fix is not more infrastructure; it is an interface over the infrastructure that already exists, one where a developer expresses intent and the platform does the rest. That interface is what this part builds, and it rests entirely on the ten parts beneath it.
The last layer of the machine: the hardening a developer will now inherit by default, without choosing or even seeing it.
Four faces of self-service
The developer experience is not one tool but four, each answering a different need, and together they are the whole of it. Abstract turns intent into infrastructure, interface makes it approachable, account makes it accountable, and access makes it reachable. Tap each:
An XApplication of about twenty fields expands, through a Crossplane composition, into every resource a real service needs on this platform: the route, the secret, the scaling, the storage, the policy. The developer declares intent; the platform emits the ten layers.
One spec, the whole stack
Start with the face that matters most, because it is where the ten parts collapse into a single object. A service on this platform is not a folder of Deployments and Services and policies; it is one XApplication of about twenty fields. A Crossplane composition reads those fields and emits everything a real service needs, and on this platform that everything is the primitives the earlier parts built: a route on the Cilium gateway, a secret from OpenBao, scaling on a Mimir metric, a volume on Ceph, a sandbox from Kata. The developer writes intent; the composition writes the stack. Tap a field to see what one line becomes:
# The entire developer-facing surface: one XApplication, about twenty fields.
apiVersion: platform.example.com/v1
kind: XApplication
metadata: { name: checkout, namespace: payments }
spec:
image: harbor.example.com/payments/checkout:1.4.2
routes: ["checkout.example.com"] # -> Cilium gateway + cert + DNS
secrets: ["stripe-key"] # -> ExternalSecret from OpenBao
scaling:
metric: sum(rate(http_requests_total{app="checkout"}[2m]))
threshold: 200 # -> KEDA on a Mimir query
storage: { size: 20Gi, class: ceph-block }
database: { engine: postgres } # -> XDatabase (part twelve)
critical: true # off preemptible capacityOne list of hostnames becomes a route on the shared Cilium gateway, a certificate minted by cert-manager, and a DNS record published to Cloudflare. The service is reachable over HTTPS the moment it exists.
What this much abstraction does for productivity: thousands of lines of YAML collapsed behind a twenty-field spec.
The composition, on this ground
The magic is not magic, it is a program. In production these compositions are function-pythonic, Python that reads the simple spec and emits the resource graph, and the entry point reads like a table of contents for a production service. The interesting thing about moving it to bare metal is how little the shape changes and how much the substrate does. Where the cloud version emitted a Pod Identity role, an Istio VirtualService, and an S3 bucket, this one emits an OpenBao ExternalSecret, a Cilium HTTPRoute, and a Ceph PVC. Same abstraction, different ground underneath:
def compose(self):
self._namespace() # PSA restricted + default-deny NetworkPolicy
self._service_account()
self._external_secret() # OpenBao, via Kubernetes auth (part 4)
self._rollout() # Argo Rollout, non-root, caps dropped (3, 10)
if self._spec.routes:
self._http_route() # Cilium gateway + cert + external-dns (part 6)
if self._spec.scaling:
self._scaled_object() # KEDA on a Mimir metric (parts 8, 9)
self._pod_disruption_budget()
if self._spec.storage:
self._ceph_pvc() # Rook Ceph (part 5)
if self._spec.untrusted:
self._kata_runtime_class() # sandbox untrusted code (part 10)
if self._spec.database:
self._xdatabase() # one claim per engine (part 12)
self._service_monitor() # observable by default (part 8)The production platform this mirrors: the XApplication and XDatabase compositions that ~60 engineers ship on without touching YAML.
A database from one flag
Persistence is a second, smaller abstraction, XDatabase, driven by one engine flag. A developer asks for the datastore they need and gets a correctly-operated one: the right operator, the right topology, and the unglamorous parts, backups to a Ceph bucket and least-privilege access, provisioned for them. The per-engine operational knowledge behind that flag is enough to fill its own part, which is the next one. Here the point is only that a database is a field, not a project:
# Persistence is one flag; the engine picks the operator and topology.
apiVersion: platform.example.com/v1
kind: XDatabase
metadata: { name: checkout-db, namespace: payments }
spec:
engine: postgres # cnpg | dragonfly | clickhouse | valkey | ...
size: small
backup: true # a Ceph RGW bucket + schedule, provisioned for youThe front door: Backstage
A spec you can write is only self-service if there is somewhere to write it. Backstage is that place: a catalog of every service and its live cluster state, software templates that scaffold a brand-new service into a pull request, docs, and a Kubernetes view. The loop closes with the GitOps from part three. The template opens a PR containing the XApplication, a human approves it, and Argo CD applies it, so no platform-team ticket sits in the middle. The only manual step is a code review; everything after it is machinery from the ten parts below:
# A software template scaffolds a new service into a PR, no ticket required.
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata: { name: service }
spec:
parameters:
- title: New service
properties:
name: { type: string }
routes: { type: array, items: { type: string } }
steps:
- id: scaffold # render the repo and the XApplication
action: fetch:template
- id: pr # open a PR; Argo CD applies it (part 3)
action: publish:github:pull-requestNo ticket, no platform-team handoff. The only human step is the pull request; everything right of it is the machinery from the previous ten parts, triggered by one short spec.
The portal in depth: the Software Catalog, templates, TechDocs, and the Kubernetes plugin that make it the single pane.
A price without a bill
Owning your hardware saves the cloud premium, but it also loses the one thing a cloud gives you for free: a bill. Without per-team cost, capacity becomes a tragedy of the commons, so the platform derives what the cloud would have charged. OpenCost allocates resources per namespace, and a custom pricing model supplies the rates: capex amortized over the machines for the compute, and the measured power from part eight for the electricity. The result is a real showback number for every team, on iron that never sent an invoice. Accountability is not something you give up by leaving the cloud; it is something you compute:
apiVersion: v1
kind: ConfigMap
metadata: { name: opencost-custom-pricing }
data:
# capex-amortized rates; kWh is fed the measured power from Mimir (part 8)
default.json: |
{ "CPU": "0.011", "RAM": "0.0015", "GPU": "1.20", "kWh": "0.12" }The number a cloud hands you for free, a per-team cost, is derived here instead: what the iron cost to buy, plus what it drew to run, split by who used it. Owning the hardware does not mean losing accountability.
Where the power number comes from: the Redfish/BMC watt metrics in Mimir, which feed the electricity half of the cost model.
Into a cluster with no shells
The last face is getting in, and it is shaped by a decision from part one: Talos has no SSH, no shell, no node login at all. That is a real security win that would be an operational nightmare with static kubeconfigs and a VPN. Teleport resolves it. Access to the cluster, to databases, and to dashboards goes through one identity-aware proxy: a developer signs in once, receives a short-lived certificate scoped by RBAC, and every session is recorded. Nothing is standing, nothing is shared, and there is an audit trail for every action against a cluster you otherwise could not log into at all:
# Access to a shell-less cluster: short-lived certs, RBAC, recorded sessions.
kind: role
version: v7
metadata: { name: developer }
spec:
allow:
kubernetes_labels: { env: "dev" }
kubernetes_resources:
- { kind: pod, namespace: payments, verbs: ["get", "list"] }
options:
max_session_ttl: 8h # the certificate expires; nothing stands
record_session: { k8s: true }Why access has to be identity-aware: Talos ships with no SSH and no shell, so there is no node to log into in the old sense.
The golden path is the secure path
Step back and the deepest property of the platform is not any one tool, it is that the easy path and the correct path have been made identical. A developer who writes the shortest possible XApplication still gets a workload that is Pod Security restricted, network default-deny, running a signed image as non-root, scaled with a disruption budget, observable from the first minute, and sandboxed if it asked to be. They did not choose any of that, and they could not opt out of it if they tried. Every hard-won property from the ten parts below is inherited by construction, so the platform's security and reliability do not rest on every team remembering to configure them. That is what a paved road is: not a suggestion, a default.
What this buys
The platform is finally a product. A developer ships a fully wired, secure, observable, scalable service from a twenty-field spec and a portal, sees a real cost for it, and reaches it through an audited front door, without ever meeting the ten layers that make it true. Everything since part one has been in service of this: infrastructure that disappears behind intent. One promise in that spec is still unredeemed, though, the database field that turns an engine name into a running, backed-up, correctly-operated datastore. Making that real, across a shelf of very different engines, is the next part.
Next: redeeming the database field. One engine flag over ten very different data stores, each an operator you run yourself, on Ceph, backed up to your own object store.