Rotate Your Secrets When You Sleep
Long-lived secrets are a liability that only grows. Chain OpenBao, the External Secrets Operator, and Reloader, and rotation becomes something that happens on its own, at three in the morning, with no human and no downtime.
A static credential is a bet that it will never leak, and that is a bad bet at scale. 'We should rotate our secrets' is one of those tasks that lives on the board forever, because doing it by hand is tedious and slightly scary. The fix is to stop doing it by hand. Chain three tools together and rotation becomes a property of the system: it happens on a schedule, while you sleep, with no human and no downtime.
The problem with secrets that don't move
A database password that never changes is valid the day it leaks and every day after. The usual mitigation, rotating it periodically, runs straight into human nature: manual rotation is disruptive, easy to get wrong, and therefore rarely done. So credentials sit unchanged for years and the blast radius of any single leak is, effectively, forever.
The goal here is to make rotation boring and automatic, so that short-lived is the default and a leaked secret is useful for minutes rather than years. Three pieces get us there: OpenBao to rotate the secret, the External Secrets Operator to carry it into Kubernetes, and Reloader to make the app actually pick it up.
OpenBao: secrets that expire on their own
OpenBao is an open-source, Linux Foundation fork of HashiCorp Vault, and it does secrets management, key management, and encryption as a service. Its use cases are the familiar Vault ones: secure secret storage, dynamic secrets, encryption and signing as a service (the Transit engine, so applications and the supply chain use keys they never actually hold), identity-based access, and, the part that matters most here, leasing and revocation. Every secret it hands out can carry a lease, and when that lease expires OpenBao revokes it automatically. And the master key that protects all of it never sits in the cluster: it lives in a hardware security module and is unsealed over PKCS#11, the same hardware key-custody and key-management guarantee you would otherwise reach for a cloud KMS to get.
That lease model is what makes rotation native rather than bolted on. The strongest form is dynamic secrets: instead of storing a shared password, OpenBao generates a unique, short-lived credential on demand and deletes it when the lease ends. The work happens in secrets engines, each mounted at a path, each with its own rotation story:
A secrets engine is mounted at a path and either stores, generates, or encrypts. Click one to see how it makes rotation automatic.
Generates a unique database username and password on demand, scoped to a role, instead of a shared static password.
Dynamic creds are leased and revoked when the lease ends. For static roles, OpenBao rotates the password itself on a rotation_period, so nothing long-lived is ever handed out.
External Secrets Operator: into Kubernetes
OpenBao holds the secret; your application runs in Kubernetes and just wants a plain Secret it can mount. The External Secrets Operator (ESO) bridges the two. It is a Kubernetes operator that reads from an external secret manager and writes the value into a native Kubernetes Secret, keeping the two in sync.
You describe the connection once with a SecretStore or ClusterSecretStore, then an ExternalSecret says which key to pull and which Secret to write. The OpenBao and HashiCorp Vault providers are the same code, and they read the KV engine. The load-bearing field is refreshInterval: ESO re-reads on that cadence, so when OpenBao rotates the value, the Kubernetes Secret follows on its own.
# 1. Point ESO at OpenBao (the HashiCorp Vault provider works for both).
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: openbao
spec:
provider:
vault:
server: "https://openbao.internal:8200"
path: "secret" # the KV v2 mount
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso"
serviceAccountRef:
name: external-secrets
---
# 2. Sync one key into a native Secret, re-checking every minute.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: "1m"
secretStoreRef:
name: openbao
kind: ClusterSecretStore
target:
name: db-credentials # the Secret the app mounts
data:
- secretKey: password
remoteRef:
key: apps/orders/db
property: passwordReloader: restarting the app so it notices
This is the gap almost everyone hits: Kubernetes does not restart pods when a Secret they reference changes. Your Secret is now v2, but the process is still holding v1, from an env var or a file it read at startup. Reloader closes that gap. It is a small controller that watches ConfigMaps and Secrets and performs a rolling restart of the workloads that use them.
You opt a Deployment in with an annotation. reloader.stakater.com/auto: "true" reloads on any referenced ConfigMap or Secret, or you can name a specific one:
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
annotations:
# When db-credentials changes, roll the pods so they pick it up.
secret.reloader.stakater.com/reload: "db-credentials"
spec:
template:
spec:
containers:
- name: orders
envFrom:
- secretRef:
name: db-credentialsThe whole chain, while you sleep
Put the three together and a rotation propagates on its own. OpenBao rotates the credential; ESO notices within a refreshInterval and updates the Kubernetes Secret; Reloader sees the Secret change and rolls the Deployment; the new pods come up on the new secret. Step through it:
Step through a single rotation as it propagates from OpenBao to the running pods. No human required.
Why this matters
The point is not the individual tools; it is what falls out of chaining them. Rotation stops being a project you schedule and dread, and becomes a property of the system that is simply always true. The window in which a leaked credential is useful shrinks from 'until someone remembers' to 'one refreshInterval,' and on an ongoing basis it costs you nothing.
That is the version of security I like best: the kind that keeps working while you sleep, precisely because no step of it depends on a human remembering to do it.