Agent-Driven Operations: ~70% of SRE, Run by an AI Operator
HolmesGPT in operator mode as a member of the on-call rotation: it runs 24/7, investigates alert to resolution across the whole estate, verifies every deploy, and opens the PR to fix what it finds.
I run HolmesGPT in operator mode as a genuine member of the on-call rotation, not a chatbot someone opens when they remember to. It works about 70% of our SRE load: the full alert-to-resolution loop, deployment verification, scheduled health checks, and opening PRs on GitHub to fix what it finds. It is useful because I wired it into everything we run, so it can reason about an incident the way an experienced operator would, only faster and around the clock.
Most AI agents wait to be asked
A troubleshooting agent that only runs when a human types a question is still bottlenecked on the human. Someone has to notice the problem, decide it is worth investigating, and press go. Operator mode removes that. HolmesGPT runs continuously in the cluster, spots problems on its own, and comes to you with the answer.
That is the shift that actually changes reliability: the agent stops being a tool you reach for during an incident and becomes the thing that catches the incident first.
A troubleshooting agent still needs a human to notice something is wrong and press go. Operator mode removes that. Toggle the two worlds.
A scheduled health check or an alert catches it first, usually before customers notice.
HolmesGPT starts immediately, pulling from every connected data source at once.
The agent correlates the whole estate and returns a root cause in 2 to 3 minutes.
The agent proposes the fix in Slack and opens a GitHub PR to apply it.
Reviews and merges. The human checks the agent's work instead of starting from a blank page.
In operator mode the agent is the trigger, the investigator, and the fixer. The human moves to the end of the loop, reviewing findings and merging PRs.
How it works one incident
When an alert fires, the agent does what a good on-call engineer does, in one pass instead of five tools. It pulls the relevant metrics, logs, traces, Kubernetes state, and recent deploys, lines them up in time, and returns a grounded root cause with the fix. Access is read-only and respects existing RBAC, and it reasons over raw data rather than summaries, so there is nothing for it to hallucinate.
How the agent works one incident. Click a step.
- An AlertManager alert (a 5xx spike, a crashloop, saturated connections) reaches Robusta, or a scheduled health check turns something up on its own.
- The alert's labels and annotations become the starting context: alertname, namespace, pod, severity.
Connected to the whole estate
The agent is only as good as what it can see, so I connected it to every layer we operate: Prometheus and the full Grafana LGTM stack (Loki, Tempo, Mimir, Pyroscope, and an Alloy collector with Beyla), Grafana dashboards and a Grafana MCP, OpenSearch, Kubernetes, AWS, Argo CD, Crossplane, Cilium, Postgres via CloudNativePG, ClickHouse, RabbitMQ, MongoDB, and GitHub and Confluence MCPs.
The agent is only as good as what it can see. I wired it into every layer we run. Click a category.
The live state and the recent change: pod events, the last rollout, the composition that was applied, the network policy in the way.
That breadth is the point. A human operator carries this map in their head and pays for it in years of experience. The agent queries all of it in one pass, which is why it can reason about an incident end to end.
The operator: reliability as Kubernetes resources
The best part, as a platform engineer, is that all of this is declarative. The operator runs in Kubernetes and defines checks as CRDs. A ScheduledHealthCheck is a 24/7 watch on the things that matter, and a TriggeredHealthCheck verifies every rollout automatically, whoever shipped it, diffing the estate before and after.
# Every 15 minutes, check the critical namespaces; page Slack only when something is wrong.
apiVersion: holmesgpt.dev/v1alpha1
kind: ScheduledHealthCheck
metadata:
name: critical-namespaces
spec:
schedule: "*/15 * * * *"
query: "Are the critical namespaces healthy? Look for crashloops, OOMKills, and pods stuck not-ready."
mode: alert
destinations:
- type: slack
config: { channel: "#sre-oncall" }Reliability defined declaratively. Three CRDs, each a different kind of watch. Click one.
- Runs an investigation on a cron schedule and spawns a HealthCheck each time, tracking recent history.
- This is the 24/7 watch: "every fifteen minutes, are the critical namespaces healthy?" It catches drift long before it becomes a page.
Any check can run in monitor mode (store the result) or alert mode (page a destination like Slack or PagerDuty on failure), and through the GitHub MCP the agent can open a PR with the fix rather than just reporting it.
Every deploy, verified after it ships
The one I lean on most is deployment verification. It fires on any Deployment rollout, from CI, Argo CD, kubectl, or a rollback, waits for the new version to settle, and compares error rate, latency, restarts, and logs before versus after. A regression gets caught minutes after it ships instead of on the next page, and every rollout leaves an audit trail.
# Verify every rollout, whoever shipped it, and diff before vs after.
apiVersion: holmesgpt.dev/v1alpha1
kind: TriggeredHealthCheck
metadata:
name: verify-rollouts
spec:
deploymentRollout:
selector:
matchLabels: {} # empty matches every Deployment in the namespace
delaySeconds: 300 # wait for the rollout to settle, then check
cooldownSeconds: 600 # don't re-fire for the same Deployment within 10m
query: |
Compare error rate, latency, restarts, and logs before and after this rollout.
Rolled out to {{ .new.image }} (was {{ .old.image }}). Is this a regression?
mode: alert
destinations:
- type: slack
config: { channel: "#deploys" }Bedrock, and the one line I hold
The infrastructure agents run on AWS Bedrock, which is cheaper and scales better for ops workloads than serving my own model for this. The self-hosted models are reserved for anything that touches customer data, and that is a line our GRC team drew, not a technical one. Operations telemetry goes to Bedrock; customer data does not leave our own inference.
What it changed
Robusta enriches every alert with HolmesGPT's findings before it pages anyone, so on-call opens an incident that has already been investigated rather than a blank one. That compressed diagnosis from 10 to 15 minutes down to 2 or 3. And because so much of the investigation time simply disappears, the outages that used to run 25 minutes or more now resolve in 15 to 20.
Add the scheduled watches and the per-deploy verification on top, and roughly 70% of the SRE work is now done by the agent. The team's time moved from digging to reviewing.
Redefining reliability
This is what agent-driven operations actually means: not a smarter chatbot, but a change in who does the work. Detection, investigation, and the first draft of the fix move to an agent that never sleeps and never has to remember where the logs live. The humans move to the end of the loop, reviewing findings and merging PRs, which is exactly where scarce, senior judgment belongs.