k8s on bare-metal - Part 3 : GitOps

The pipeline that forgot

The model most teams arrive with is a push pipeline: continuous integration builds an artifact and then reaches into the cluster and applies it. It works, and it quietly rots. The pipeline holds cluster credentials, it only runs on a trigger, and between runs nobody is comparing what is deployed against what was intended. Pull-based GitOps inverts every one of those: agents live inside the cluster and continuously reconcile it toward git, so git is the single source of truth, no credentials ever leave, and drift is corrected rather than accumulated. Toggle the two:

Argo CD arrives first, and installs itself

The bootstrap has to break a small loop: Argo CD manages the cluster from git, but something has to install Argo CD. The same Cluster API add-on provider that laid down Cilium does it, with one HelmChartProxy for the argo-cd chart. The trick is the chart's extraObjects field, which renders arbitrary manifests alongside the install, so the very same resource that installs Argo CD also seeds an AppProject and a root ApplicationSet. From that seed, Argo CD discovers every app in the repo, including its own manifests, and takes over. One resource in, a self-managing GitOps controller out.

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: argo-cd
spec:
  clusterSelector:
    matchLabels: { gitops: argo }
  repoURL: https://argoproj.github.io/argo-helm
  chartName: argo-cd
  namespace: argocd
  valuesTemplate: |
    extraObjects:                     # arbitrary manifests, same install
      - apiVersion: argoproj.io/v1alpha1
        kind: AppProject
        metadata: { name: platform, namespace: argocd }
        spec:
          sourceRepos: ["*"]
          destinations: [{ server: "*", namespace: "*" }]
      - apiVersion: argoproj.io/v1alpha1
        kind: ApplicationSet          # the root: discovers everything else
        metadata: { name: bootstrap, namespace: argocd }
        spec:
          generators:
            - git:
                repoURL: https://github.com/my-org/platform
                directories: [{ path: "apps/*" }]
          template:
            metadata: { name: "{{path.basename}}" }
            spec:
              project: platform
              source:
                repoURL: https://github.com/my-org/platform
                path: "{{path}}"       # Kustomize per directory
              destination: { server: https://kubernetes.default.svc, namespace: "{{path.basename}}" }
              syncPolicy:
                automated: { prune: true, selfHeal: true }

graph TD
  A["Cluster Ready (end of part two)"] --> B["CAPI add-on provider applies a<br/>HelmChartProxy: argo-cd chart"]
  B --> C["extraObjects seed an AppProject<br/>+ a root ApplicationSet"]
  C --> D["Argo CD is up and self-seeded"]
  D --> E["root ApplicationSet: Git generator<br/>discovers every app in the repo"]
  E --> F["platform apps sync<br/>cert-manager, monitoring, ..."]
  E --> G["Argo CD's own manifests are<br/>just another app: it manages itself"]
  E --> H["app-delivery ApplicationSet:<br/>SCM + MERGE/MATRIX previews"]

The octopus and its arms

Argo CD reconciles, but the delivery story needs three more tools from the same project, each owning one job. Argo Rollouts turns a deploy into a gated canary; Argo Image Updater automates image bumps with two different write-back methods; and the Argo GitOps Promoter moves a change from Dev to Prod through git. The mascot is an octopus, and between the four tools it has about enough arms to run the whole thing from one source of truth. Tap each:

Apps that generate themselves

You never hand-write an Argo CD Application. A generator makes them, and which generator you pick is the whole design. The platform layer uses a Git directory generator (one app per folder: cert-manager, monitoring, and the rest). The application layer uses an SCM Provider generator that scans the GitHub org and produces one app per repo that matches a filter, so onboarding a service is mostly a matter of the repo existing. App definitions are Kustomize, which keeps overlays and image pins first-class. Tap through the generators this setup uses, and the one it deliberately does not:

A preview from a label

The nicest trick is per-PR preview environments, and it falls out of merging two generators into one ApplicationSet. The shape is MERGE(SCM + MATRIX(SCM x PR)): a plain SCM generator finds each repo's stable Dev branch, and a matrix of that same SCM generator crossed with a Pull Request generator finds the labelled PRs. Because the merge keys on the repository, each repo resolves to a single Application. With no labelled PR it tracks the Dev branch; when a labelled PR appears, the merge folds in that PR's commit and a templatePatch flips the same app to the PR's ephemeral image. There is no separate preview app to name or garbage-collect. Toggle the label and watch one app change identity:

# One ApplicationSet, two intents, merged on the repository key:
#   MERGE( SCM  +  MATRIX( SCM x PR ) )
generators:
  - merge:
      mergeKeys: [repository]
      generators:
        # (a) every app repo with a Dev branch -> a stable Dev app
        - scmProvider:
            github: { organization: my-org, allBranches: true }
            filters:
              - repositoryMatch: ^(app|service)-
                branchMatch: ^environment/dev$
        # (b) PRs carrying the preview label -> one ephemeral app per PR
        - matrix:
            generators:
              - scmProvider:
                  github: { organization: my-org }
                  filters:
                    - repositoryMatch: ^(app|service)-
              - pullRequest:
                  github: { owner: my-org, repo: "{{ .repository }}", labels: [preview] }

Two ways to write a tag back

When a PR merges, a release tool computes the version, CI builds the image and pushes a semver tag to the registry, and Argo Image Updater notices. What it does next depends on the tag, and the split matters:

• A released semver image gets git write-back: Image Updater commits the new tag into the repo, which makes the change a reviewable git event and hands off to the GitOps Promoter.
• An ephemeral preview image (test-<commit>) gets argocd write-back: Image Updater updates the running app directly, so a preview stays pinned to its own build without ever touching git.

# Annotations on the generated app decide the write-back path.
# Released apps, tracked on git:
argocd-image-updater.argoproj.io/image-list: app=registry.internal/app
argocd-image-updater.argoproj.io/app.update-strategy: semver
argocd-image-updater.argoproj.io/write-back-method: git

# Preview apps, pinned in place (set on the PR-generated apps instead):
argocd-image-updater.argoproj.io/write-back-method: argocd

Self-heal versus the preview tag

The argocd write-back path has a catch worth knowing, because it is a genuine GitOps corner. Argo CD self-heal does not know the preview tag was placed on purpose; it reads the running image differing from git as drift and tries to revert it. The cure is ignoreApplicationDifferences, which tells Argo CD to leave a field alone. But that rule historically could only select applications by name, and preview apps are generated with names you do not choose. The fix is to select them by label instead: stamp every PR app with a label in the templatePatch, and target exactly those. Selecting ignoreApplicationDifferences by label is a capability that had to be added to Argo CD upstream, which is what makes this pattern work at all.

# Stamp every PR-generated app so it can be selected by label,
# and pin it to the PR's own commit.
templatePatch: |
  {{- if index . "number" }}          # only PR params carry a PR number
  metadata:
    labels: { deploy: pr }
  spec:
    source: { targetRevision: "{{ .head_short_sha_7 }}" }
    syncPolicy: { automated: { prune: true, selfHeal: true } }
  {{- end }}
---
# Then stop self-heal reverting the ephemeral image, by LABEL not name:
ignoreApplicationDifferences:
  - labelSelector: { matchLabels: { deploy: pr } }
    jsonPointers: [/spec/source/kustomize/images]

A deploy that undoes itself

Inside the cluster, workloads run as Argo Rollouts rather than plain Deployments. A release goes out as a canary: it shifts a slice of traffic, an AnalysisRun checks real signals, and only then does it proceed. Wire the analysis to something that actually means failure, like a 5xx rate crossing a threshold, and a bad deploy rolls itself back to the last stable revision with no human in the loop. Combined with Argo CD reverting drift underneath, the environment is self-healing rather than merely automated.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: checkout }
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - analysis: { templates: [{ templateName: error-rate }] }
        - setWeight: 50
        - analysis: { templates: [{ templateName: error-rate }] }
        - setWeight: 100
  # a failed AnalysisRun aborts and rolls back to the stable ReplicaSet

graph TD
  A["New revision"] --> B["Canary: shift 10% of traffic"]
  B --> C{"AnalysisRun<br/>5xx, latency in bounds?"}
  C -->|pass| D["Shift more: 25, 50, 100%"]
  C -->|fail| E["Auto-rollback<br/>to last stable revision"]
  D --> F["Promoted: canary becomes stable"]

Develop to promote, end to end

Put the arms together and they form one continuous loop. A change starts as a labelled PR and ends live in Prod, gated and versioned, without anyone pushing to the cluster or promoting by hand. Preview on Dev from a label, merge, let the release tool version it, let Image Updater write the tag back, let the Promoter carry it to Prod behind branch protection and a human approval, and let Rollouts guard the actual cutover. Every arrow below is a git operation or an agent reacting to one:

graph TD
  A["Labelled PR<br/>preview label added"] --> B["GitHub Action builds<br/>an ephemeral test-&lt;commit&gt; image"]
  B --> C["Preview app on Dev<br/>(argocd write-back pins the image)"]
  C --> D["PR reviewed and merged"]
  D --> E["semantic-release computes<br/>the version, CI builds + pushes semver"]
  E --> F["Argo Image Updater:<br/>git write-back commits the new tag"]
  F --> G["GitOps Promoter:<br/>Dev to Prod along gated branches"]
  G -->|branch protection:<br/>gate statuses + human approval| H["Live in Prod"]
  H --> I["Argo Rollouts canary<br/>auto-rolls back if analysis fails"]

Self-healing by default

Underneath all of it, Argo CD is always reconciling. Every app syncs with prune and self-heal on, so a manual change in the cluster is seen as drift from git and reverted; the declared state wins, every time. That is the property that makes the whole platform trustworthy: not that it was automated once, but that it is continuously held to git. Two ApplicationSets do the delivering, the Git directory generator for the platform and the SCM generator for the apps, and both are just more state in the repo, which means Argo CD reconciles the delivery machinery itself. There is no privileged layer sitting outside the system.

What still cannot live in git

The cluster now installs itself, runs itself, previews changes, promotes them, and heals from drift, all from a repository anyone can read. Which is exactly the problem the next part solves, because there is one kind of state that must never live in that readable repository: secrets. Git holds the reference; it cannot hold the value. Part four gives the platform a secrets layer that keeps the GitOps model intact, external-secrets pulling values from OpenBao into the cluster, with the reference in git and the secret never anywhere near it.