AI for Cyber Defence: Phase Two
Offense-informed defense: autonomous penetration testing with PentAGI, and breach-and-attack simulation with OpenAEV, wired so a plain-English threat compiles into a controlled Caldera simulation and a graded verdict. Detection-as-code for the SOC.
Phase one made the SOC faster at reading incidents. Phase two goes on the offensive to keep it honest: autonomous penetration testing to find the holes, and breach-and-attack simulation to continuously prove that the controls meant to catch them actually do. Two open-source foundations, PentAGI and OpenAEV, wired together with Claude.
PentAGI: autonomous penetration testing
The offensive half runs on PentAGI, an open-source platform that turns a language model into an autonomous pentester: a multi-agent system with a full suite of real security tools (nmap, Metasploit, sqlmap, and more) in sandboxed containers, a browser, search backends, and long-term memory. It plans an engagement, runs the tools, reads the output, and decides what to do next.
PentAGI ships as a Docker Compose stack, which isn't how we run things. So I wrote a Helm chart for the whole system and deployed it on Kubernetes, with the network segmentation the tool needs to keep offensive traffic isolated from everything else. I've open-sourced the chart. The deeper story, including why the only real safeguard against misuse lives in the model rather than the tool, is in a separate writeup.
OpenAEV: breach & attack simulation, validated
The defensive half is about proof. OpenAEV (Open Adversary Exposure Validation) runs controlled simulations of real adversary behavior on the MITRE Caldera framework and its injector, then validates whether your controls actually detected or prevented each technique. On top of it I built a wrapper that turns intent into a test: describe a threat in plain English, or drop in a CTI report, a CVE, or an ATT&CK ID, and it compiles that into an OpenAEV scenario, runs it against a Caldera lab, and grades the outcome.
The mental model is detection-as-code: pytest for your SOC. Every control gets a pass or fail against a real adversary ability, and the verdict is tracked over time, so a regression in detection shows up like a broken test.
The stack
The stack is deliberately small. Claude (Opus and Sonnet 4.x) does the compile and judge steps, the OpenAEV REST API is wrapped as an MCP server to give the agent its tools, the Claude Agent SDK is the orchestration harness, and a Caldera lab (a Caldera instance plus two or three endpoints running agents) is the execution target.
Compile, execute, validate
The pipeline is three stages, and the interesting engineering is in treating the first one like a compiler: ground the model in the platform's real, installed abilities before it generates anything, type-check the result against a live catalog, and only then run it. Execution drives Caldera's async lifecycle behind safety rails, and validation grades the run both deterministically and with an LLM-judge that catches scenarios which passed while testing the wrong thing.
Detection-as-code: pytest for your SOC. Click a stage for what it does.
- Ground before you generate. Pull the real injector contracts and installed Caldera abilities, the endpoints and their platforms, and the asset groups via REST, and index them (RAG). This is what stops the model emitting a fake ability ID, or a Windows ability against a Linux host.
- Emit a typed Scenario spec: metadata, ordered injects each bound to a real contract that fits the target, expectations (detection or prevention, plus the validation mode), and ATT&CK / Domain tags.
- Type-check the spec against a JSON schema and the live catalog before anything is created, and auto-repair unknown IDs or platform mismatches. Treat it like a compiler front-end.
- Materialize via REST: create the scenario, its injects, and the inject-expectations.
Validated the way the labs do it
The approach isn't speculative. Anthropic's engagement with Pacific Northwest National Laboratory did almost exactly this at the frontier: they built a scaffold that translated natural-language prompts into full attack chains against a high-fidelity water treatment plant simulation (run by PNNL for CISA), with code-based tools that let Claude act on the networked systems. Attack reconstruction that normally takes multiple weeks was done in about three hours, and when a UAC bypass failed, the model recognized it and found another route on its own.
That's the same shape as what I built: natural language in, grounded tools in the middle, a controlled target, and a graded result. Seeing a national lab and a frontier lab validate the pattern on critical infrastructure was reassuring, and it reinforces the reason to do this at all: used well, AI lets defenders find and close the gaps faster than attackers can reach them.