Eight Arms, One Source of Truth

From pipelines to a living system

A one-off CI/CD pipeline runs, pushes to the cluster, and forgets. It holds credentials, it fires on a trigger, and between runs nobody is checking whether the cluster still matches what you intended. I wanted the opposite: a system that is continuously alive, always comparing the running state against Git and correcting the difference.

So the pipeline now stops at the registry. Everything after that is pull-based GitOps: Argo agents inside the cluster watch Git and reconcile toward it. That single shift buys a lot:

• Git is the single source of truth. If it is not in Git, it is not in the cluster.
• State is declarative, not a script of imperative steps that can half-succeed.
• Promotion history lives in GitHub as immutable commit history, not in a CI tool's logs.
• Environment gating is native GitHub: branch protection and environments do the gating.
• Rollback is trivial: revert a commit, or let Rollouts revert the canary for you.
• Manual changes to the cluster are automatically reconciled away.
• Production is gated by branch protection that checks automated gate statuses plus a manual PR approval.
• No environment credentials or cluster access live in GitHub. A push-based pipeline needs them; pull-based GitOps does not.
• Environment parity is enforced natively, because every environment is the same declared state at a different point on the promotion path.

The Argo ecosystem

Four tools from the Argo project do the work, each owning one job. Argo CD reconciles the cluster to Git; Argo Rollouts turns a deploy into a gated canary; Argo Image Updater automates image bumps; and the Argo GitOps Promoter moves a change from Dev to Prod through Git.

Apps that generate themselves

I never hand-write an Argo CD Application. An ApplicationSet with an SCM generator scans the GitHub org and generates an app for every repo that matches, so onboarding a service is mostly a matter of the repo existing.

Each generated app deploys the Crossplane composition that ships inside the application's own repo, so a team declares its infrastructure (the app, its database, and the rest) right next to its code, and Argo CD renders it into the cluster.

Preview a PR on Dev, from a label

The nicer trick is per-PR preview environments, and developers get them by merging two generators into one ApplicationSet. The shape is exactly MERGE(SCM + MATRIX(SCM x PR)): a plain SCM generator finds each repo's stable Dev branch, and a matrix of that same SCM generator crossed with a Pull Request generator finds the labelled PRs. Because the merge keys on the repository, each repo resolves to a single Application: with no labelled PR it tracks the Dev branch, and when a labelled PR appears, the merge folds in that PR's commit and a templatePatch flips the same app over to the PR's image. There is no separate preview app to name or garbage-collect.

Add the genivi-deploy label to a PR and a GitHub Action builds an ephemeral image (tagged test-<commit>, garbage-collected after 7 days) and the preview deploys to Dev. Remove the label and the preview is torn down; the app reverts to the last stable state pointing at master. The whole preview lifecycle is a label.

A bit of trivia: genivi-deploy is not a convention anyone will find in the docs. genivi is Ge-rard + Ni-cholas + Vi-kash, the initials of the colleagues I work with the most. It stuck.

# One ApplicationSet, two intents, merged on the repository key:
#   MERGE( SCM  +  MATRIX( SCM x PR ) )
generators:
  - merge:
      mergeKeys: [repository]
      generators:
        # (a) every repo with an environment/dev branch -> a stable Dev app
        - scmProvider:
            github: { organization: my-org, allBranches: true }
            filters:
              - repositoryMatch: ^(app|service)-
                branchMatch: ^environment/dev$
        # (b) PRs carrying the preview label -> one ephemeral app per PR
        - matrix:
            generators:
              - scmProvider:
                  github: { organization: my-org }
                  filters:
                    - repositoryMatch: ^(app|service)-
              - pullRequest:
                  github:
                    owner: my-org
                    repo: '{{ .repository }}'
                    labels: [genivi-deploy]
                  filters:
                    - branchMatch: '^feature/[A-Z]+-[0-9]+'
                      targetBranchMatch: '^master$'

Two ways to write a tag back

When a PR merges, semantic-release computes the version, GitHub Actions builds the image and pushes the semver tag to ECR, and Argo Image Updater notices. What it does next depends on the tag:

• A released semver image gets the git write-back method: Image Updater commits the new tag into the repo, and the GitOps Promoter takes over from there.
• An ephemeral test-<commit> image gets the argocd write-back method: Image Updater updates the running app directly, so Argo CD stays pinned to the branch's config and the image built from that branch for as long as the preview label lives.

A self-heal conflict, and a fix I upstreamed

The argocd write-back path has a catch. Argo CD self-heal does not know the test tag was placed on purpose; it reads it as drift and tries to revert it. The cure is ignoreApplicationDifferences, which tells Argo CD to leave the image field alone. But upstream, that rule could only select applications by name, and preview apps are generated with names I do not choose.

So I select them by label. Every PR app is stamped deploy: pr by a templatePatch, and the ignore rule targets exactly those. The one problem: labelSelector did not exist on ignoreApplicationDifferences. So I added it to Argo CD. The PR is linked below, and it is why the config just above works at all.

# Stamp every PR-generated app so it can be selected by label,
# and pin it to the PR's own commit.
templatePatch: |
  {{- if index . "number" }}
  metadata:
    labels: { deploy: pr }
  spec:
    source:
      targetRevision: '{{ .head_short_sha_7 }}'
    syncPolicy:
      automated: { prune: true, selfHeal: true }
  {{- end }}

# Then stop self-heal from reverting the ephemeral test image.
# Selecting apps by *label* (not name) is what my PR #26473 adds.
ignoreApplicationDifferences:
  - labelSelector:
      matchLabels: { deploy: pr }
    jsonPointers:
      - /spec/source/kustomize/images

Self-healing by default

Inside the cluster I use Argo Rollouts instead of plain Deployments. A release goes out as a canary: it takes a step, an AnalysisRun checks the criteria, and only then does it proceed. I wire the analysis to real signals, so if something like 5xx crosses a threshold mid-rollout, Rollouts automatically rolls back to the last stable revision. A bad deploy undoes itself.

Underneath all of it, Argo CD is always reconciling. A manual change in the cluster is rare and tightly controlled, but if one happens, Argo CD sees the drift from Git and reverts it. The declared state wins, every time. That is what makes the environment self-healing rather than merely automated.

Develop → Promote, end to end

Put together, the pieces form one continuous loop. A change starts as a labelled PR and ends live in Prod, gated and versioned, without anyone pushing to the cluster or promoting by hand.

The octopus on my desk

One last thing. I've got the Argo mascot octopus staring up at me from under the monitors. It is a fitting mascot: a handful of arms, each doing its own job, all coordinated toward the same thing. That is roughly what this system is, and having it watch over the desk while I build the pull-based version of it never stopped being funny.