k8s on bare-metal - Part 7 : Supply chain

A registry, and everything around it

The obvious job here is to run a registry so the cluster stops depending on Docker Hub. The real job is bigger. A registry is where the supply chain becomes concrete: it is the one place every image passes through, so it is the natural place to scan, to sign, to cache, and to enforce. Treating it as just a bucket for images misses the point. Treated as the choke point it actually is, it becomes the seam where a hard look at what the cluster runs finally has somewhere to live, and it does so on top of the storage, gateway, and secrets layers already standing.

The chain, source to pod

Before choosing tools, it helps to see the whole path a piece of code takes from a commit to a running pod, because every tool is just one station on it. The source is scanned for secrets and statically analyzed, its dependencies and manifests are checked, an image is built, that image is described by a bill of materials and scanned for known vulnerabilities, then signed along with those facts as attestations, pushed to the registry, and finally checked at the cluster door before it is allowed to start. The whole build half runs on the cluster's own CI, and the last station refuses anything the chain did not produce:

Choosing the chain, not the pile

There is a large pile of tools that all claim a spot on that chain, and the temptation is to run several of each and feel thorough. The better instinct is to pick one owner per job and stop. This platform drives the whole thing from GitHub Actions on self-hosted ARC runners, and runs Harbor as the registry, Trivy as the scanner with Grype as an occasional second opinion, Syft for the bill of materials, Cosign for signing, in-toto and SLSA for provenance, and Kyverno as the admission gate. On the source side, SonarQube does static analysis, Gitleaks catches secrets, OpenSSF Scorecard scores repository posture, and Renovate keeps base images current. Two more extend the same idea past the container: OpenSSF Model Signing covers the model weights, and GUAC gathers every SBOM and attestation into one queryable graph. Much of this is OpenSSF stock, the same foundation OpenBao comes from. The others are not wrong; they are just redundant here. Tap each role to see what does the job and what got left on the shelf, and why:

CI that runs on the cluster

Every gate on that chain has to run somewhere, and where it runs matters. The usual answer is a hosted runner: a machine outside the cluster that clones the repo, builds the image, and has to be handed the registry password and the signing key to do its job. That is a lot of trust placed outside the platform. The better answer here is to bring CI onto the cluster with the Actions Runner Controller, which turns a GitHub Actions job into an ephemeral pod. A runner spins up when a job is queued, runs it, and is destroyed, scaling to zero when nothing is building, so idle CI costs nothing.

Running on-cluster changes the trust story completely. The build sits next to Harbor, so pushes never leave the network and pulls come through the same cache as everything else. And because the runner is a pod with a service account, it authenticates to OpenBao with Kubernetes auth, exactly like every other workload, so the signing step asks the vault to sign without any static token ever living in a GitHub secret. The one credential ARC itself needs, a GitHub App key to register runners, is an external secret from OpenBao like all the rest.

# gha-runner-scale-set Helm values: ephemeral runners as pods, scale to zero.
githubConfigUrl: https://github.com/example/platform
githubConfigSecret: arc-github-app   # GitHub App creds, an ExternalSecret from OpenBao
minRunners: 0                        # no idle runners, no idle cost
maxRunners: 20
containerMode:
  type: dind                         # docker-in-docker, for image builds
template:
  spec:
    serviceAccountName: ci-signer    # the SA OpenBao trusts via Kubernetes auth
    containers:
      - name: runner
        image: harbor.example.com/ci/actions-runner:2.320.0   # from our own Harbor

graph TD
  A["Workflow job queued<br/>on GitHub"] --> B["ARC listener<br/>sees the job"]
  B --> C["Scale up: one<br/>ephemeral runner Pod"]
  C --> D["Runner registers,<br/>runs the pipeline"]
  D --> E["Job done:<br/>Pod destroyed"]
  E --> F["Idle again:<br/>scaled to zero"]
  F -.->|next job| B

The pipeline those runners execute

With the runners in place, the pipeline itself is one GitHub Actions workflow, and it is worth reading in full because it is the chain made concrete. It runs on the self-hosted scale set and walks every gate in order: scan the source for secrets, run static analysis, check dependencies and manifests, build, describe the image with an SBOM, scan the image, push, then sign and attest. Any gate that fails stops the run, so a dirty change never reaches the registry, let alone the cluster. The signing step is the payoff of running on-cluster: it logs in to OpenBao with the runner's own service account, so the key is used without a token ever being stored.

name: build-sign-push
on:
  push: { branches: [main] }
env:
  IMAGE: harbor.example.com/apps/api
jobs:
  supply-chain:
    runs-on: bare-metal-runners        # the ARC scale set, on our own cluster
    steps:
      - uses: actions/checkout@v4

      - name: Secret scan                # no hardcoded credentials
        run: gitleaks detect --no-banner --redact

      - name: SAST + quality gate        # SonarQube; blocks on the quality gate
        uses: sonarsource/sonarqube-scan-action@v3
        env:
          SONAR_HOST_URL: https://sonarqube.example.com   # token mounted from OpenBao

      - name: Dependencies + IaC         # vulnerable libs, misconfigured manifests
        run: |
          trivy fs --scanners vuln,license --exit-code 1 .
          trivy config --exit-code 1 ./deploy

      - name: Build
        run: docker build -t "$IMAGE:$GITHUB_SHA" .

      - name: SBOM                        # Syft; attached as an attestation below
        run: syft "$IMAGE:$GITHUB_SHA" -o spdx-json=sbom.spdx.json

      - name: Image scan                  # fail on fixable highs and criticals
        run: |
          trivy image --severity HIGH,CRITICAL --ignore-unfixed \
            --exit-code 1 "$IMAGE:$GITHUB_SHA"

      - name: Push
        run: |
          docker push "$IMAGE:$GITHUB_SHA"
          echo "DIGEST=$(crane digest "$IMAGE:$GITHUB_SHA")" >> "$GITHUB_ENV"

      - name: Sign + attest               # key stays in OpenBao; SA auth, no token
        env:
          VAULT_ADDR: https://openbao.internal:8200
        run: |
          export VAULT_TOKEN=$(bao write -field=token auth/kubernetes/login \
            role=ci-signer jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token)
          REF="$IMAGE@$DIGEST"
          cosign sign   --key hashivault://cosign "$REF"
          cosign attest --key hashivault://cosign --type spdxjson \
            --predicate sbom.spdx.json "$REF"
          cosign attest --key hashivault://cosign --type slsaprovenance \
            --predicate provenance.json "$REF"

Before the build, the source gates

The cheapest problem to fix is the one caught before an image even exists, so several checks run on the source first. Secret scanning with Gitleaks rejects a commit that carries a hardcoded credential, because a leaked key is worth catching in seconds, not after it ships. Static analysis with SonarQube reads the code for bug patterns and enforces a quality gate, self-hosted so the platform's own source never leaves it. And dependency and manifest scanning, Trivy over the lockfiles for vulnerable libraries and over the Kustomize and Terraform for misconfigurations, catches the risk that comes from what the code pulls in and how it is deployed, not just what it does.

A fourth check is different in kind. OpenSSF Scorecard does not fail on one finding; it scores the security posture of a repository across a set of automated checks, whether branch protection is on, whether releases are signed, whether dependencies are pinned, whether workflow tokens are over-scoped. Run on the platform's own repos it is a standing report card, and run against a third-party repository before a dependency on it is taken, it is a way to judge a project's hygiene rather than just its code. It answers a question the other gates do not: not is this artifact clean, but is this project one to trust.

One last thing keeps this from being a losing battle: currency. A base image left alone only accumulates vulnerabilities. Renovate runs on a schedule, opening pull requests that bump base images and dependencies, so fixes land continuously and the scanners have less to complain about. The earliest gate is not a tool at all; it is not falling behind in the first place.

{
  "extends": ["config:recommended"],
  "packageRules": [
    { "matchDatasources": ["docker"], "groupName": "base images" }
  ],
  "vulnerabilityAlerts": { "enabled": true }
}

Harbor, on the storage it already has

Harbor is the registry, and the reason it fits here is that it needs nothing the platform has not already built. Its image blobs go to the Ceph object store from part five over the S3 API, so there is no special disk for images; its small stateful pieces, the metadata database and the cache, sit on Ceph block volumes; its UI is exposed through an HTTPRoute on the gateway from part six with a cert-manager certificate; and its backend credentials for the object store arrive as an external secret from OpenBao. The scanner ships inside it. What comes out is one registry that is also a scanning gate and a signing store, running entirely on the layers beneath it. Two details save an afternoon of debugging: the database and cache sit on ReadWriteOnce Ceph volumes, which attach to only one pod at a time, so those stateful components take a Recreate update strategy, since a rolling one would deadlock waiting on a volume the old pod still holds; and because part ten made the platform refuse anything else, Harbor runs as non-root with its capabilities dropped, passing the same restricted Pod Security and Kyverno policies as every other workload:

# Harbor Helm values: image blobs on Ceph RGW, not on a node disk.
externalURL: https://harbor.example.com
expose:
  type: ingress                    # realized as a Gateway API HTTPRoute
  tls: { certSource: secret, secret: { secretName: harbor-tls } }
persistence:
  imageChartStorage:
    type: s3
    s3:
      region: us-east-1
      bucket: harbor
      regionendpoint: http://rook-ceph-rgw-store.rook-ceph.svc:80
      existingSecret: harbor-rgw   # RGW keys, an ExternalSecret from OpenBao
  persistentVolumeClaim:           # only the small stateful bits use RBD
    database: { storageClass: ceph-block }
    redis: { storageClass: ceph-block }
trivy: { enabled: true }           # the scanner lives inside the registry

An SBOM and a provenance for every image

Two documents travel with every image, and both are signed. The first is a software bill of materials, produced by Syft: a full list of every package and version inside the image, in SPDX or CycloneDX form. It is what lets the platform answer, months later and without taking the image apart, whether a newly disclosed vulnerability is present, and it is what a license or VEX check reads. The second is provenance: an in-toto attestation in SLSA form recording how the image was built and from what, the commit, the builder, the steps, so a running image can be traced back to a pipeline rather than taken on faith.

Neither is useful if it can be swapped, so both are attached to the image as Cosign attestations, signed with the same OpenBao key as the image itself. That turns them from claims into evidence: the admission gate later does not just check that the image is signed, it checks that a signed SBOM and a signed provenance are present. Verification is the mirror of signing and needs only the public key:

# The SBOM and provenance ride with the image as signed attestations.
# Verifying them needs only the public key; the admission gate does this too.
cosign verify-attestation --key cosign.pub --type spdxjson \
  harbor.example.com/apps/api@sha256:9f2b...        # the bill of materials

cosign verify-attestation --key cosign.pub --type slsaprovenance \
  harbor.example.com/apps/api@sha256:9f2b...        # how it was built, and from what

Scanning, before it lands and after

Scanning is not a single event. Trivy runs in the pipeline and fails the build on a fixable high or critical, which stops known-bad images at the door. But the vulnerability database moves faster than releases: an image that was clean on Tuesday has a critical on Friday because something new was disclosed, not because the image changed. So Harbor re-scans what it already holds against fresh feeds, and a report stays current without a rebuild. Grype is kept as an occasional cross-check, because two scanners rarely agree exactly and the disagreement is sometimes the interesting part.

The failure mode of scanning is noise: a wall of criticals that are not actually reachable here drowns the ones that are. Two habits keep it honest. Ignoring unfixable findings by default, because a CVE with no available fix is a fact to track, not a build to block. And VEX, a signed statement that a given CVE is not exploitable in this artifact, so a triaged false positive stays quiet on the next scan instead of being re-litigated every run.

# A signed VEX statement says a present CVE is not exploitable here, so a
# triaged false positive stays quiet on the next scan instead of blocking it.
trivy image --vex vex.openvex.json \
  --severity HIGH,CRITICAL --ignore-unfixed \
  harbor.example.com/apps/api@sha256:9f2b...

Every pull through the front desk

Once the registry exists, the next move is to stop the nodes from ever going to the public internet directly. Harbor runs a proxy cache: a project that mirrors an upstream registry, fetching an image once on a miss and serving it from Ceph forever after. Point every node at it through the Talos machine config, and every pull in the cluster passes the front desk. The payoff is threefold: images are cached, so a pull is fast and cheap; they are scanned on the way through, so nothing unknown lands; and an upstream outage no longer stops the cluster, because anything pulled before is still there. A few production touches make it robust: each upstream gets its own proxy-cache project, the node authenticates to Harbor with a read-only robot account, and the upstream itself is listed as a fallback endpoint, so a pull still succeeds even during a brief Harbor restart:

# Talos machine config: mirror every registry through Harbor's proxy caches,
# with the upstream listed as a fallback so a pull survives Harbor being down.
machine:
  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://harbor.example.com/v2/dockerhub-proxy   # the proxy-cache project
          - https://registry-1.docker.io                    # fallback to upstream
        overridePath: true         # use this exact path, Harbor's proxy project
      ghcr.io:
        endpoints:
          - https://harbor.example.com/v2/ghcr-proxy
          - https://ghcr.io
        overridePath: true
      registry.k8s.io:
        endpoints:
          - https://harbor.example.com/v2/k8s-proxy
          - https://registry.k8s.io
        overridePath: true
    config:
      harbor.example.com:          # authenticate the pull with a read-only robot
        auth: { username: robot$mirror, password: from-openbao-via-eso }

graph TD
  A["Talos node needs<br/>docker.io/library/x"] -->|machine.registries.mirrors| B["Harbor proxy-cache project"]
  B -->|cache hit| C["served from Ceph<br/>already scanned"]
  B -->|cache miss| D["fetch from upstream once"]
  D --> B
  E["Upstream down?"] -.->|cached images<br/>still pull| C

Sign here, verify there

The heart of the supply chain is a signature the cluster can trust, and the delicate part is the key. Cosign signs an image and its attestations, but the signing key must never sit on a build runner or in a repo. It does not have to: OpenBao's transit engine from part four holds the key and signs on request, so the private key never leaves it. Cosign speaks to it over the same Vault-compatible transit API, addressing the key with a hashivault URI. And because the build runs on an on-cluster runner, that runner authenticates to OpenBao with Kubernetes auth using its own service account, so there is not even a signing token stored in a GitHub secret. Signing, shown in the workflow above, asks the vault to sign; verification, which the admission gate does automatically, needs only the public half:

# The signing key lives in OpenBao and never leaves it; the CI step reaches it
# via Kubernetes auth, so no signing token is stored anywhere.
# Verification is the mirror image and needs only the public key.
cosign verify --key cosign.pub \
  harbor.example.com/apps/api@sha256:9f2b...   # digest, never a tag

graph TD
  A["cosign sign<br/>--key hashivault://cosign"] -->|signing happens in OpenBao's<br/>transit engine; key never exported| B["signature + attestations"]
  B --> C["stored in Harbor<br/>beside the image"]
  D["Pod admission"] --> E["Kyverno verifyImages<br/>with the Cosign public key"]
  C -.->|reads the signature| E
  E -->|valid| F["admit, pin to digest"]
  E -->|missing or invalid| G["deny"]

The same signature, for models

Container images are not the only artifact this cluster runs. On an AI platform the model weights are supply-chain artifacts too, often larger and longer-lived than any image, and usually pulled from a hub with far less scrutiny than a container registry gets. They deserve the same treatment, and they can have it. OpenSSF Model Signing signs a manifest of a model's files with the same Sigstore tooling, so a set of weights can be verified before it is loaded into a serving pod, the exact trust decision made for an image before it runs.

The point is that it reuses everything already standing rather than inventing a parallel world for models. The signing is backed by the same key material OpenBao already holds, verification is the same public-key check, and a model can live as an OCI artifact in the same Harbor that holds the images. One signing story then covers both halves of what an AI platform actually runs, the code and the models it serves:

# OpenSSF Model Signing signs a manifest of the model's files, using the
# same Sigstore tooling and the same PKI material as the container images.
model_signing sign ./llama-3-70b --signature model.sig

# Verify before the weights are ever loaded into a serving pod.
model_signing verify ./llama-3-70b --signature model.sig

Nothing runs unless it's signed

All of it converges on one control: Kyverno at admission, verifying signatures before a pod is allowed to start. The policy holds the Cosign public key and checks that the image is signed by it, that the required attestations are present and signed, and that the reference points at Harbor and not somewhere off-platform. When it passes, Kyverno rewrites the tag to the exact digest it verified, so the pod is pinned to the bytes that were checked. When it fails, admission is denied and the workload never starts. There is no runtime exception. Step through what the gate checks:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-images }
spec:
  validationFailureAction: Enforce   # deny, do not just warn
  background: false
  rules:
    - name: signed-and-attested
      match:
        any: [{ resources: { kinds: [Pod] } }]
      verifyImages:
        - imageReferences: ["harbor.example.com/*"]   # only from our registry
          mutateDigest: true          # pin the tag to the verified digest
          required: true
          attestors:
            - entries:
                - keys:
                    publicKeys: |-     # the Cosign public half; private stays in OpenBao
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----
          attestations:
            - type: https://spdx.dev/Document   # an SBOM must be present + signed
              attestors:
                - entries: [{ keys: { publicKeys: "…same key…" } }]

Somewhere for the evidence to go

By now the pipeline emits a lot of paper: an SBOM and a provenance per image, VEX statements, scan reports, Scorecard scores, signatures. Scattered, each is a file next to an artifact answering one question about one thing. Gathered, they answer questions about everything. GUAC ingests all of it, SBOMs, SLSA attestations, OSV vulnerability data, VEX, and Scorecard results, into one graph, so the platform can ask fleet-wide questions it otherwise could not: which running images contain a newly disclosed library, what the blast radius of a given CVE is, which artifacts are missing provenance, what depends on the package that just got yanked.

That turns the supply chain from a stack of checks that ran once at build time into something queryable at any time. It is also the natural bridge to what comes next: the instinct that a platform should be able to answer what it is running and what it is exposed to is the same instinct behind observability, giving the platform eyes on itself.

What this buys

The platform no longer trusts the internet, and it no longer trusts a build it cannot account for. CI runs on the cluster as ephemeral pods, so the build sits beside the registry and the vault instead of outside them. Every commit passes a wall of gates before an image exists; every image, and every model, ships with a signed bill of materials and a signed record of how it was built; every pull comes through the platform's own cache; the door verifies all of it before a pod starts; and the evidence collects in a graph the platform can query. The registry is not a bucket bolted on the side; it is the choke point where CI, storage, gateway, secrets, and admission control all meet and do one job together. What is missing now is sight. The platform can build, store, route, and gate, but it cannot yet see itself: no metrics, no logs, no traces, no way to answer what is slow or what just broke. That is the next thing to build, an observability stack on the same storage and the same gateway, so the platform can finally watch itself run.