k8s on bare-metal - Part 12 : Databases
Part twelve of the bare-metal series, redeeming the promise from part eleven that a database is a field, not a project. Behind one engine flag sits a shelf of ten very different data stores, from Postgres to ClickHouse to a Ceph bucket, each a purpose-built operator you run yourself, on Ceph, with dynamic credentials and backups to your own object store. The self-hosted managed service, and the operational knowledge it hides.
Give me a database is one of the most loaded requests in infrastructure. It hides the choice of engine, the operator that runs it, a replication topology, a backup and restore story, credentials, monitoring, and a dozen failure modes specific to that one system. A cloud answers it with a managed service and a monthly bill. On bare metal there is no managed service to rent, so the platform becomes the managed service: an XDatabase composition that turns one engine flag into a correctly-operated store, run by a purpose-built operator, on the Ceph from part five, with credentials from part four and observability from part eight. This part opens the shelf and walks all ten engines, then the pattern they share, because the value is not any single database, it is that asking for one is a field.
A database is a field
Part eleven ended on an unredeemed promise: the XApplication had a database field, and behind it was supposed to be a running, backed-up, correctly-operated datastore. This part is that promise made real. Persistence is its own composition, XDatabase, driven by a single engine flag, and the reason it deserves a part of its own is that the flag hides an enormous amount. Ten engines, ten operators, ten different ideas of what durable means, all reached by changing one string.
Where the promise was made: the XApplication's database field, forward-referenced to here, now turns an engine name into a real store.
The shelf
Here is the whole shelf. Each engine is a different operator with its own topology, storage layout, and backup mechanism, and the developer sees none of that; they name the engine and receive a store. The three Rook entries are deliberately separate, because Ceph serves three genuinely different kinds of data store and picking between a disk, a shared filesystem, and a bucket is a real decision. Tap through all ten:
The default relational store. CloudNativePG runs a primary with hot-standby replicas on streaming replication, one per rack, each on its own Ceph RBD volume. WAL archiving and base backups stream to Ceph RGW for point-in-time recovery. This is what most services mean when they say database.
One pattern, every engine
The engines could not be more different, but the composition treats them the same way: it wraps whichever one you chose in a set of guarantees that apply regardless of type. Placement on a dedicated database node pool with a replica per rack, storage on replicated Ceph, credentials that expire, backups to the object store, a policy that refuses plaintext connections, and monitoring from the first minute. The developer opts into none of these and cannot opt out of them, which is the whole point: the unglamorous, easy-to-forget parts of running a datastore are the parts the platform guarantees.
# One flag, dispatched to one method per engine, wrapped in the guarantees
# every store gets regardless of type.
ENGINES = {
"postgres": _cloudnative_pg,
"dragonfly": _dragonfly,
"valkey": _valkey,
"clickhouse": _clickhouse, # + ClickHouse Keeper for coordination
"mongodb": _mongodb, # MongoDBCommunity via the MCK operator
"opensearch": _opensearch,
"rabbitmq": _rabbitmq,
"rook-block": _ceph_rbd,
"rook-file": _ceph_fs,
"rook-object": _ceph_bucket,
}
def compose(self):
self._place_on_db_pool() # dedicated node pool, one replica per rack
self._deny_non_tls() # reject any plaintext connection
self._dynamic_credentials() # OpenBao dynamic login, no static password
self.ENGINES[self._spec.engine](self) # the engine-specific resources
if self._spec.backup:
self._backup_to_rgw() # scheduled, to Ceph RGW (part 5)
self._service_monitor() # into Mimir (part 8)
self._pod_disruption_budget()a dedicated database node pool, with one replica per rack so a whole rack can fail
Ceph RBD, replicated three ways, so the disk survives a node loss under the engine's own replication
OpenBao dynamic logins, short-lived and per-app, so there is no static database password anywhere
to Ceph RGW on a schedule, with the engine's native tooling and a tested restore
a policy that rejects any non-TLS connection, so nothing talks to the store in the clear
a ServiceMonitor and a dashboard into Mimir, so the store is watched from the first minute
The production XDatabase this generalizes: ten engines behind one flag, with backups and least privilege provisioned for free.
The relational archetype
Postgres is worth expanding because it is the shape most of the shelf follows. The postgres engine becomes a CloudNativePG Cluster: a primary and hot-standby replicas on streaming replication, one per rack so a rack can fail without losing the database, each instance on its own Ceph RBD volume. Backups are not an afterthought bolted on later; the same resource declares WAL archiving and base backups to the Ceph RGW object store, which is what makes point-in-time recovery possible. The developer asked for postgres; this is what postgres means when it is done properly:
# The postgres engine, expanded: a CloudNativePG Cluster, HA on Ceph RBD,
# with WAL and base backups streamed to the Ceph RGW object store.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: checkout-db, namespace: payments }
spec:
instances: 3 # one primary, two hot standbys
storage: { size: 20Gi, storageClass: ceph-block }
topologySpreadConstraints: # one replica per rack
- maxSkew: 1
topologyKey: topology.kubernetes.io/rack
whenUnsatisfiable: DoNotSchedule
backup:
barmanObjectStore:
destinationPath: s3://db-backups/checkout
endpointURL: http://rook-ceph-rgw-store.rook-ceph.svc:80
# credentials come from OpenBao via an ExternalSecret
retentionPolicy: "30d"Rook, three ways
The three Rook engines are where a data store stops being a database and becomes raw storage, and splitting them is deliberate. Ceph from part five already serves block, file, and object, so the shelf exposes each as its own engine rather than hiding them behind one confusing rook flag. A developer picks rook-block for a private replicated disk, rook-file for a shared filesystem many pods can mount, or rook-object for an S3 bucket. Same cluster underneath, three very different stores on top:
One writer, replicated three ways. The disk a single-writer workload wants when it manages its own state.
Many pods mounting the same POSIX tree at once. Shared uploads, checkpoints, anything several replicas read and write together.
An S3 endpoint and keys from one claim. Application blobs, and the store every other engine backs up into.
Three engines, one Ceph cluster from part five. The developer picks block, file, or object by name; the composition turns it into the right Ceph resource with the same backups, credentials, and monitoring every other engine gets.
The cluster these three engines sit on: Rook Ceph serving block, file, and object, now exposed as self-service data stores.
Credentials that expire
A datastore's password is the thing most likely to leak and least likely to be rotated, so this platform does not have one. Instead of generating a static password and storing it, the composition wires each app to OpenBao's database secrets engine, which mints a fresh login on request, scoped to that app, and revokes it when the lease ends. The application never holds a credential longer than an hour, and there is no shared password sitting in a secret for an attacker to find. It is the rotation story from a separate writeup, applied by default to every store on the shelf:
# No static password. OpenBao's database engine mints a short-lived login
# on request, scoped to this app, and revokes it when the lease ends.
apiVersion: vault.example.com/v1 # Crossplane provider-vault (part 4)
kind: DatabaseSecretRole
metadata: { name: checkout }
spec:
backend: database
dbName: checkout-db
creationStatements: # SQL run to create the ephemeral role
- CREATE ROLE "{{name}}" LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'
- GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO "{{name}}"
defaultTtl: 1h # the login expires in an hour; ESO fetches a fresh one
maxTtl: 24hThe dynamic-credential mechanism this uses: OpenBao leases, External Secrets, and Reloader, chained so nothing static is ever held.
On Ceph, under replication
There is a subtle question about running databases on Ceph: the database replicates its own data across instances, and Ceph replicates every block three ways underneath, so is that redundant? It is layered, not redundant, and the layers do different jobs. Ceph replication keeps a volume alive when a disk or a node dies, so a replica does not lose its data and have to rebuild from scratch; the database's own replication keeps the service alive and consistent when an instance dies, and handles failover and reads. Each instance takes a ReadWriteOnce RBD volume, anti-affinity spreads the instances across racks, and the failure domain from part five means one rack can burn without taking a quorum with it. The cost is a little write latency from the network hop to Ceph, which for all but the most latency-sensitive stores is a fair trade for never rebuilding a replica by hand.
Backups, and getting them back
A backup you have never restored is a rumor, so backups here are uniform and testable. Every engine writes to the same Ceph RGW object store, using its own native tooling, CloudNativePG's Barman, mongodump for MongoDB, clickhouse-backup, RabbitMQ definition exports, on a schedule the composition sets. Because the target is one S3-compatible store, a restore drill is the same shape for every engine, and because the store is the Ceph the cluster already runs, backups never leave the building. The point is not that backups exist; it is that they are one habit across ten different systems instead of ten separate ones:
# Every engine backs up to the same Ceph RGW, on a schedule.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata: { name: checkout-nightly, namespace: payments }
spec:
schedule: "0 0 2 * * *" # 02:00 every day
cluster: { name: checkout-db }
method: barmanObjectStore # to the RGW path declared on the ClusterWhat this buys
The data layer is now self-service, and it is the same to ask for as anything else on the platform: a field, a pull request, a running store. Behind that field is a shelf of ten operators, each running a database the way its own experts would, on replicated Ceph, with logins that expire and backups that land in the cluster's own object store. The hard, specialized, easy-to-get-wrong work of operating stateful systems has been done once, in a composition, and handed to every team as a flag. The platform can now provision, reconcile, secure, observe, scale, and persist, all self-service. What it has been quietly assuming this whole time is ordinary hardware, and the next part changes that: the accelerators, the GPUs that the models this platform exists to serve actually run on.
Next: the silicon the models run on. Getting a driver onto an immutable OS, sharing a $30k card, and the interconnect fabric you own.