k8s on bare-metal - Part 4 : Secrets
Part four of the bare-metal series. Part three made the cluster install and run itself from a git repository anyone can read, which is exactly why secrets cannot live there. This piece gives the platform a secrets layer that keeps the GitOps model intact: OpenBao unsealed by a hardware security module, configured from git by Crossplane, and feeding values to workloads two different ways depending on how sensitive they are. Nothing sensitive is ever committed.
GitOps has one honest limitation: it makes the repository the source of truth, and a repository is readable. That is a feature for manifests and a disaster for secrets. The answer is not to encrypt values into git and hope, it is to keep the reference in git and the value somewhere built to hold it. On this cluster that somewhere is OpenBao, the open-source fork of Vault, and the interesting parts are all bare-metal specific: how it unseals with no cloud key service to lean on, how its own configuration stays declarative, and how a value gets from OpenBao into a running pod without ever becoming a plaintext liability. This is the layer that lets the cluster keep a secret.
The one thing git can't hold
The principle carries straight over from the GitOps part: git holds the reference, OpenBao holds the value. An object in git says materialize this secret from this path; the value itself never appears in the repository. That is the whole design in a sentence, and everything below is the machinery that makes it true on bare metal, starting with the hardest bit, because OpenBao is useless until it is unsealed, and unsealing on metal is where the cloud tutorials stop being any help.
Why secrets need their own home: Argo CD made the cluster run itself from a readable git repo, and this is the state that repo cannot hold.
OpenBao, unsealed by hardware
OpenBao runs in-cluster, highly available on Raft integrated storage, installed by Argo CD like everything else. It boots sealed: the key that decrypts its storage has to come from somewhere, and on a cloud you would point it at a managed key service. There is none here, so the choice is among OpenBao's native seal methods, and it matters more than any other decision in this article because it decides where the root of the entire secret system lives. Tap through them:
# The key lives in the HSM and is used there over PKCS#11.
# OpenBao's OSS build ships this; Vault gated it behind Enterprise.
seal "pkcs11" {
lib = "/usr/lib/softhsm/libsofthsm2.so" # a real HSM in prod
token_label = "openbao"
pin = "env://BAO_HSM_PIN" # PIN grants use, not export
key_label = "openbao-unseal"
mechanism = "0x1087" # AES-GCM
}
# OpenBao will NOT create the key material; it is provisioned in the HSM first.- How
- The unseal key lives inside a hardware security module and is used there over PKCS#11; it never leaves.
- Posture
- The root of the whole secret system is anchored in tamper-resistant hardware. The only at-rest credential is the HSM PIN, which grants use, not export.
- Verdict
- The ideal, and the reason OpenBao on metal beats Vault on metal: HSM auto-unseal ships in the open-source build. Real HSM in prod, Nitrokey NetHSM as the open-source option, SoftHSM in the lab.
Unsealed without a human
With the seal configured, a fresh OpenBao pod comes up already unsealed and so does every HA replica, no key shares, no operator at 3am. The unseal key is unwrapped inside the HSM and the plaintext key never crosses the boundary; the only thing the pod holds is the PIN, which lets it ask the HSM to do the unwrapping. Follow it:
graph TD
A["OpenBao pod starts<br/>sealed"] -->|reads seal stanza<br/>lib + slot + PIN + key_label| B["PKCS#11 to the HSM"]
B -->|unwrap the root key<br/>inside the HSM| C["key never leaves<br/>the hardware"]
C -->|unwrapped root key<br/>returned to the pod| D["OpenBao unsealed<br/>no human, no share"]
D --> E["HA replicas each<br/>unseal the same way"]Config as code, the Crossplane way
OpenBao needs configuring: secret engines mounted, a Kubernetes auth backend enabled, policies and roles written. Doing that with a pile of imperative bao commands would sit outside GitOps entirely, so instead the Crossplane provider-vault expresses every piece as a managed resource. Because the cluster already runs Crossplane for its internal developer platform, this is one control plane, not a new one, and Argo CD syncs the config from git and corrects drift like anything else. The one step that cannot be declarative is the very first: the bao operator init ceremony that produces the recovery keys, done once and stored offline or in the HSM. Everything after it is a git commit.
apiVersion: vault.vault.upbound.io/v1alpha1
kind: Mount # a KV v2 secret engine
metadata: { name: kv }
spec:
forProvider: { path: kv, type: kv-v2 }
---
apiVersion: vault.vault.upbound.io/v1alpha1
kind: Policy
metadata: { name: app-read }
spec:
forProvider:
name: app-read
policy: |
path "kv/data/apps/*" { capabilities = ["read"] }
---
apiVersion: kubernetes.vault.upbound.io/v1alpha1
kind: AuthBackendRole # bind a ServiceAccount to a policy
metadata: { name: external-secrets }
spec:
forProvider:
backend: kubernetes
roleName: external-secrets
boundServiceAccountNames: [external-secrets]
boundServiceAccountNamespaces: [external-secrets]
tokenPolicies: [app-read]How the operator gets in without a secret
The External Secrets Operator has to authenticate to OpenBao, and the trap would be to hand it a token stored in a Kubernetes Secret, a secret to fetch secrets. Kubernetes auth removes the bootstrap credential entirely: OpenBao validates the operator's own ServiceAccount token against the cluster's TokenReview API and issues a short-lived, policy-scoped token in return. There is no secret zero for the operator, only an identity the cluster already vouches for.
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: { name: openbao }
spec:
provider:
vault:
server: "https://openbao.internal:8200"
path: kv
version: v2
auth:
kubernetes: # no static token anywhere
mountPath: kubernetes
role: external-secrets
serviceAccountRef:
name: external-secrets
namespace: external-secretsTwo modes for two sensitivities
Not every secret deserves the same handling. A third-party API key and a signing private key are both secrets, but leaking the first is a bad afternoon and leaking the second is a company-ending event, so the platform runs two modes that differ end to end: how the value is created, whether it ever touches git, how the app reads it, and whether a change needs a restart. Toggle between them:
- In git?
- Yes, as a Crossplane MR encrypted with KSOPS (SOPS using an OpenBao transit key, so no static key sits in the repo-server)
- Into OpenBao
- Crossplane provider-vault writes the static secret from the decrypted MR
- Consumed by
- External Secrets Operator syncs it to a native Kubernetes Secret
- App reads it as
- Environment variables
- On change
- Stakater Reloader restarts the workload so the new env is picked up
- At rest
- A Kubernetes Secret exists in etcd (encrypt etcd at rest)
The rule falls out of the last row: if an app needs env vars, it is a secret; if it can read a file, it can be a super-secret.
secrets: encrypted in git, picked up on a restart
The less-sensitive mode keeps everything in git, because for these values the audit trail and the review flow are worth more than hiding the ciphertext. The static secret is a Crossplane managed resource, encrypted with KSOPS before it is committed. The clever part is what SOPS encrypts with: not a static age key living in the Argo CD repo-server, but an OpenBao transit key, so decryption is a call to OpenBao that is audited and revocable rather than a file someone can steal. Crossplane writes the decrypted value into OpenBao, the External Secrets Operator syncs it out to a native Kubernetes Secret, and because that Secret becomes environment variables, a change needs Stakater Reloader to restart the workload.
# .sops.yaml: encrypt secrets-mode files with an OpenBao transit key,
# so the master key is OpenBao itself, not a file in the repo-server.
creation_rules:
- path_regex: secrets/.*\.yaml$
hc_vault_transit_uri: "https://openbao.internal:8200/v1/transit/keys/sops"
---
# The consumer side: a reference in git, never a value.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: stripe }
spec:
secretStoreRef: { name: openbao, kind: ClusterSecretStore }
target: { name: stripe }
data:
- secretKey: api_key
remoteRef: { key: apps/stripe, property: api_key }
# The Deployment carries reloader.stakater.com/auto: "true" to restart on change.graph TD
A["Crossplane MR in git<br/>KSOPS-encrypted<br/>(SOPS via OpenBao transit key)"] -->|Argo CD repo-server<br/>decrypts at render| B["decrypted MR"]
B -->|provider-vault applies it| C["static secret written<br/>into OpenBao"]
C -->|ExternalSecret pulls it| D["native k8s Secret"]
D -->|env vars| E["app pod"]
C -.->|value changes| F["ESO re-syncs the Secret"]
F -.->|Stakater Reloader| G["workload restarts to pick it up"]The day-2 half of this mode in depth: OpenBao leases, the External Secrets refresh, and Reloader turning a rotated value into a rolling restart.
super-secrets: never in git, refreshed in place
The highly-sensitive mode refuses the compromise of the first. Putting a value in git encrypted and then having Crossplane write it in creates a sprawl: the same secret exists as ciphertext in the repo and as a resource in the cluster, two copies to reason about. For super-secrets that is not worth it, so they skip git entirely and are seeded out-of-band, written straight into OpenBao over a secure channel, once. They are consumed through the Secret Store CSI driver, which mounts the value as a tmpfs file inside the pod, so it never becomes a Kubernetes Secret and never lands in etcd. And because the app reads a file rather than an environment variable, the driver can refresh that file in place when the value rotates, with no restart at all.
# Seeded once, out of band, over a secure channel. Never in git:
# bao kv put kv/apps/signing-key private_key=@signing.pem
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: signing-key }
spec:
provider: vault
parameters:
vaultAddress: "https://openbao.internal:8200"
roleName: app-signing
objects: |
- objectName: signing-key
secretPath: kv/data/apps/signing-key
secretKey: private_key
# The pod mounts a CSI volume for this class and reads a file; the driver,
# with rotation enabled, updates the file in place. No k8s Secret, no restart.graph TD
A["Seeded out-of-band<br/>straight into OpenBao<br/>(nothing in git)"] --> B["OpenBao KV / dynamic engine"]
B -->|Secret Store CSI Driver<br/>+ OpenBao provider| C["tmpfs file<br/>mounted in the pod"]
C -->|app reads the file| D["app pod"]
B -.->|value changes| E["CSI refreshes the file in place"]
E -.->|no restart| DPrefer dynamic, seed nothing
Above both modes sits the option that avoids the whole question: dynamic secret engines. A database credential, a cloud role, an internal certificate, none of these needs to be created and stored at all, because OpenBao mints them on request and revokes them on a lease. Most of what an application calls a secret is really a credential OpenBao can generate, so the two static modes end up covering only the irreducible remainder, the external values OpenBao cannot produce itself. Declared through Crossplane like the rest of the config, a database engine hands out a short-lived login per pod that expires on its own, which is a stronger posture than any static secret, rotated or not.
apiVersion: database.vault.upbound.io/v1alpha1
kind: SecretBackendRole # short-lived DB creds, minted per request
metadata: { name: app }
spec:
forProvider:
backend: database
name: app
dbName: app-postgres
defaultTtl: 3600 # a one-hour login, then gone
maxTtl: 7200
creationStatements:
- "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';"
- "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"The other ways in, and why not
The two consumption paths chosen here are ESO for secrets and the CSI driver for super-secrets, but they are not the only options, and it is worth being explicit about what the alternatives trade away rather than pretending they do not exist:
| Option | Mechanism | Secret at rest | Refresh | Here |
|---|---|---|---|---|
| External Secrets Operator | Syncs OpenBao to a native k8s Secret | k8s Secret in etcd | Re-sync on interval; env needs a restart | used |
| Secret Store CSI + OpenBao provider | Mounts OpenBao secrets as tmpfs files | None (file only) | File refreshed in place, no restart | used |
| CSI with syncK8sSecret: true | CSI, but also mirrors to a k8s Secret for env use | k8s Secret in etcd | Env needs a restart | no |
| Bank-Vaults secrets-webhook | Injects into process memory at pod start | None (memory only) | None; a change needs a pod restart | no |
| Vault Secrets Operator | Syncs to a k8s Secret, strong dynamic-lease handling | k8s Secret in etcd | Rotates and restarts workloads | no |
Why not the others: CSI (syncK8sSecret), Bank-Vaults, Vault each trade away something this split already gives you. See the notes above.
The whole layer
Put together, the secrets layer is one picture: an HSM-unsealed OpenBao at the centre, configured from git by Crossplane, fed values two ways by sensitivity and consumed two ways to match, with dynamic engines quietly handling most of it so little needs storing at all.
Where the turtles stop
Every secrets system bottoms out somewhere, and the honest thing is to say where. This one bottoms out in an HSM that holds the unseal key and a one-time init ceremony whose recovery keys live offline, plus the handful of super-secrets seeded out-of-band by a human who is trusted once. Nothing else is a turtle: the operator authenticates by identity, most credentials are dynamic and expire on their own, and the only values in git are ciphertext whose master key is OpenBao. The cluster now provisions itself, networks itself, delivers itself, and keeps its own secrets, all from a repository that is safe to read. What it still cannot do is remember anything, because it has no storage of its own yet. That is the next piece: persistent storage on bare metal, where the disks in the rack become a resilient, replicated volume the whole platform can rely on.
Give the cluster a memory: Rook and Ceph turning the rack's raw NVMe into replicated block, file, and object storage, tuned for Talos.