One Week to One Hour: Automated IaC Delivery to 150+ SIEM Tenants

The stack I was deploying, ~150 times over

To monitor a customer's SIEM effectively, Armor's SOC needs a full Sentinel stack standing in the customer's own tenant. I built that as Terraform modules: the resource group and Log Analytics workspace, Sentinel itself, data connectors, analytic and threat-hunting rules, KQL functions and parsers, watchlists, playbooks and automation rules, diagnostic settings, DCRs, workbooks, and dashboards. Azure Lighthouse delegation is what lets the SOC operate the tenant without ever holding the customer's credentials.

The layout is a two-repo Terragrunt pattern: an infra-live repo (the Terragrunt configuration) that references an infra-modules repo (the Terraform). Crucially, each customer owns their copy: the destination repos live in the customer's own GitHub, because letting them own their SIEM analytics and stack is part of the offering. So there is an Armor source pair and, per customer, a destination pair.

The tax on every change

When I joined, rolling a change out was entirely manual. Detection engineering would change the source repo and raise a ticket. The Support team, who held the customer credentials, would sign in to each Azure environment and the customer's GitHub, pull the change across, run a template-to-environment migration by hand, then plan, eyeball the diff, and apply. One environment at a time.

That worked out to about ten environments a day, so a single release was roughly a week of work for three engineers. The second-order costs were worse than the hours. Customers had almost no visibility into what was changing in their own tenant. And because every release was so expensive to ship, detection engineering became cautious about changing anything, which is exactly the wrong incentive for a team whose job is to keep detections sharp.

Atlantis, and one deliberate inversion

Atlantis was the right foundation. Its locking stops two changes racing on the same state, its autoplanning reacts to a PR on its own, and its automerging closes the loop once an apply succeeds. Those three behaviors mapped cleanly onto what I needed.

The inversion was refusing its default interface. Atlantis is built around pull requests: engineers comment 'atlantis apply' on a PR and read plan output in the thread. I did not want our customers living in GitHub PRs. So I wrote a middleware, the updater, that wraps Atlantis and drives the entire flow, with nothing controlled from the PR and everything surfaced on Nexus, our MDR product. The PR still exists; it is just plumbing the customer never has to see.

How Atlantis learns a Terragrunt repo

Atlantis understands Terraform, not Terragrunt, and our environments are Terragrunt through and through. The bridge is terragrunt-atlantis-config. It walks the repo for every terragrunt.hcl, evaluates each unit's dependency, terraform, locals, and source blocks to work out what it needs, and assembles those relationships into a directed acyclic graph.

That graph is then rendered as an atlantis.yaml, one project per unit, with the dependency order baked in. It is what lets Atlantis plan and apply the workspace before the rules that live in it, and the rules before the playbooks they trigger, across dozens of small units, without anyone hand-maintaining that ordering.

projects:
- name: analytic-rules
  dir: live/analytic-rules
  autoplan:
    when_modified:
    - "*.hcl"
  # applied after: workspace

The generator that quietly holds it together

It is worth pausing on terragrunt-atlantis-config, because it does more than a format conversion. Atlantis needs an atlantis.yaml that spells out which projects exist and how a change in one should trigger plans in the others. Terragrunt already knows all of that, in its dependency and terraform blocks. This tool is the bridge: it reads the Terragrunt tree and writes the Atlantis config so the two agree. I run it as a pre-workflow hook, so it regenerates on every clone, which means nobody on our side, and certainly no customer, ever writes or maintains an atlantis.yaml.

That is the whole reason it is load-bearing. A Sentinel environment is dozens of small, interdependent units, and there are ~150 of them. Hand-authoring a correct project graph per repo, and keeping it right as detection engineering adds a parser here or a playbook there, is not a job a human should ever hold. Generating it from the source of truth means the plan order is correct by construction rather than by vigilance. A few of its features did real work for us:

• Cascade dependencies: because it walks the graph transitively, a change to a shared function or the workspace re-plans everything downstream, so a dependent is never applied against a stale parent
• The pre-workflow hook regenerates the config on every clone, so the project graph can never quietly drift from the actual Terragrunt tree
• A --filter glob scopes generation to the live environment directories and skips the template directory that seeds them, so the template is never planned as if it were real
• Per-module locals like atlantis_skip and extra_atlantis_dependencies let a single unit opt out, or pull in a non-HCL dependency such as a shared file, without special-casing the pipeline
• It caches dependency resolution as it walks, which keeps generation fast even when the same modules are evaluated all over a large tree

Driving Atlantis from the outside

Because the updater is the only thing that talks to Atlantis, a few pieces of wiring do the real work. The updater relays each customer PR webhook to Atlantis's /events endpoint itself, rather than letting GitHub reach Atlantis directly. When a plan or apply finishes, its PR comment is turned off at the source with silence_pr_comments and the output is shipped back to the updater instead, so the thread stays empty and the record of truth lives in our system.

The apply trigger is my favourite detail. Rather than call an API, the updater sends Atlantis a synthetic, HMAC-signed issue_comment webhook that looks exactly like someone typing 'atlantis apply' on the PR. It is the programmatic equivalent of the human gesture Atlantis already knows how to handle, which keeps the whole integration on one well-worn path. A pre-workflow hook generates the config from the Terragrunt graph, and server flags like automerge, branch checkout, and discarding a stale approval on re-plan handle the rest.

# Representative, sanitized. The flags and hooks that turn a
# PR-driven tool into an unattended delivery plane.

automerge: true                 # merge the PR once apply succeeds
autodiscover: { mode: enabled } # find projects without hand-written config
checkout-strategy: branch       # check out the PR branch, not the merge ref
discard-approval-on-plan: true  # a re-plan invalidates a stale approval

repos:
- id: /.*/
  apply_requirements: [undiverged]      # never apply a branch behind its base
  silence_pr_comments: [plan, apply]    # no plan/apply comments on the PR at all
  pre_workflow_hooks:
  # render the Terragrunt DAG into atlantis.yaml projects
  - run: terragrunt-atlantis-config generate --autoplan --output atlantis.yaml
  workflows:
    terragrunt:
      plan:
        steps:
        - run: terragrunt plan -out=$PLANFILE
        - run: ship-output-to-updater "$PLANFILE"   # not a PR comment
      apply:
        steps:
        - run: terragrunt apply "$PLANFILE"
        - run: ship-output-to-updater --apply

Built for partial failure

Fanning one change across ~150 tenants means partial failure is not an edge case, it is Tuesday. A provider throttles, a connector is mid-migration, one customer's repo has drifted. So the updater tracks every environment through a state machine in DynamoDB: queued, synced, planning, plan ready, approved, applying, done. Each of the ~150 rows advances on its own.

Because the state is the source of truth rather than the PR, a failed step retries from the last good point instead of restarting the whole rollout, and every step is written to be idempotent so a replay is safe. There are manual retrigger APIs for anything that gets genuinely stuck, and an auto-approve setting for customers who would rather their changes roll straight through without a per-release click.

What it unlocked

The headline is a week down to about an hour to roll a change across every customer environment, with no support engineer touching credentials. But the change I am proudest of is the incentive it fixed. Detection engineering can now make a change, test it, and ship it the same day, so the detections stay sharp instead of being rationed by deployment cost.

Customers gained something they never had: a clear view of exactly what will change in their tenant, in full or as an LLM summary that turns a wall of plan output into a couple of readable sentences, and the choice to approve it or let it flow. A tool built around pull requests became a delivery plane the people using it never have to see.