A Distributed DAG Scheduler for Landing Zones
Onboarding and landing-zone provisioning modeled as a dependency graph in ArangoDB, executed by a Celery worker pool: every task runs the moment its dependencies are done.
Creating one of LetsBloom's 18 compliant landing zones is not a script you run top to bottom. It is dozens of steps with real dependencies: the network has to exist before workloads land on it, identity and logging before the guardrails that depend on them, and the compliance scan only makes sense once everything it checks is in place. I built a distributed work scheduler that treats provisioning as what it actually is, a directed graph, and executes it as one.
Provisioning is a dependency graph
A shell script provisions a landing zone in one fixed order, serially, and if step 30 fails you are often re-running steps 1 through 29 to get back. That is the wrong shape. The real structure is a DAG: most steps depend on a few others, and independent branches (DNS and identity, say) have no reason to wait for each other.
So each landing-zone type is defined as a dependency graph. A node is a unit of provisioning work; an edge means 'this must finish before that can start.' The 18 zones share a lot of structure, so they are built from overlapping subgraphs rather than 18 copy-pasted scripts.
The graph lives in ArangoDB
The graphs live in ArangoDB, a graph database, which is the natural home for this. Tasks are vertices and dependencies are edges, so the questions the scheduler asks are graph questions. 'What can run right now' is just: every node all of whose predecessors are done. That is a short AQL traversal, not bookkeeping I have to maintain in application code.
Storing the graph, rather than encoding it in code, also makes a run inspectable and resumable. The state of a provisioning job is the state of its graph: which nodes are done, running, or still blocked. Restarting a failed job means picking up the nodes that are not done yet, and nothing more.
Celery runs the tasks
Execution is Celery's job. Celery is a distributed task queue: you define units of work as tasks, a broker (I used RabbitMQ) holds the queue of task messages, and a pool of workers pulls from it and runs them. Add workers and you get more parallelism for free. The scheduler never runs provisioning work itself; it only decides what is ready and hands it to the pool.
The loop is small. The scheduler asks the graph for ready nodes and dispatches each as a Celery task. A worker runs the step (often a terraform apply), and on success writes the node's completion back to the graph, which unblocks its dependents. Those get enqueued next. The whole run is that cycle, draining the graph wave by wave.
# Each provisioning step is a Celery task, run by any worker in the pool.
@app.task(bind=True, max_retries=3, default_retry_delay=30)
def provision(self, zone_id, node_id):
run_step(zone_id, node_id) # e.g. terraform apply for this node
graph.mark_done(zone_id, node_id) # write completion back to ArangoDB
schedule_ready(zone_id) # enqueue whatever this unblocked
# The scheduler: find every node whose dependencies are satisfied, and dispatch it.
def schedule_ready(zone_id):
for node in graph.ready_nodes(zone_id): # a short AQL traversal over the graph
provision.delay(zone_id, node.id) # -> broker -> a free workerThe graph is the source of truth. Everything else is a loop that drains it.
Watch the scheduler run
This part is easier to see than to describe. Below is a landing-zone graph you can run. Press Step to advance one wave, or Run to let it flow. Watch how independent branches light up together, and how a node stays blocked until every one of its dependencies is done.
Each node is a Celery task. A task runs the moment its dependencies are done. Step through it, or press Run.
Create the AWS account or Azure subscription, baseline tags, and budgets.
nothing, it is a root task
Why a graph beats a script
Modeling provisioning as a graph bought several things at once. Independent work runs in parallel instead of waiting in line, so a zone comes up faster. A failure is contained: the graph knows exactly which nodes finished, so a retry resumes rather than restarts. Common structure is shared across the 18 zone types as reused subgraphs instead of duplicated scripts. And the whole thing is observable, because 'where is this provisioning job' has a precise answer you can query.
The general lesson I took from it: when work has real dependencies, make the dependencies a first-class, queryable thing. Once the graph is explicit, scheduling it is almost mechanical, and the distributed execution is a solved problem you can hand to Celery.