k8s on bare-metal - Part 8 : Observability

Blind, but not for long

A platform you cannot see is a platform you operate by superstition. Every layer built so far emits something, a latency, a log line, a dropped packet, a hot CPU, but until it is collected, stored, and made queryable, none of it exists in any useful sense. The goal of this part is not dashboards; it is legibility. By the end the platform can be asked what is slow and answer with a number, a log, a trace, and a flame graph, and it can raise its hand before a human notices at all.

Four signals, one place

Observability gets flattened to a word for dashboards, but it is really four distinct signals, each answering a question the others cannot. Metrics tell you something is wrong; logs tell you what the app said; traces tell you where across services the time went; profiles tell you which function burned the CPU. Grafana's stack is one purpose-built backend per signal, Mimir, Loki, Tempo, and Pyroscope, unified in a single pane so you can pivot from a spike to the log line to the trace to the exact function behind it. They share one design goal that makes them affordable: keep the index tiny, and push the bulk to object storage.

sum(rate(http_requests_total{status="500"}[5m])) by (service)

On the storage it already has

Here is the bare-metal payoff. Every one of those backends is built to keep a small local index and write the heavy data to an S3-compatible object store, and the platform already has one: the Ceph RGW from part five. So the telemetry needs no cloud bucket and no second storage system. Mimir writes its metric blocks, Loki its log chunks, Tempo its trace blocks, and Pyroscope its profiles, all into Ceph, with bucket credentials arriving as external secrets from OpenBao and Grafana exposed through the Cilium gateway. The object floor that started out holding container images now holds the platform's entire memory of itself.

# The object-storage block every LGTM backend shares: Ceph RGW, not a cloud.
# (Loki, Mimir, Tempo, and Pyroscope all take the same shape.)
common:
  storage:
    backend: s3
    s3:
      endpoint: rook-ceph-rgw-store.rook-ceph.svc:80   # the RGW from part five
      bucket_name: mimir-blocks                         # via an ObjectBucketClaim
      access_key_id: ${S3_ACCESS_KEY}                   # injected by ESO from OpenBao
      secret_access_key: ${S3_SECRET_KEY}
      insecure: true                                    # in-cluster traffic

Metrics, and where they come from

Metrics are the signal you alert on, and Mimir is the store: horizontally scalable, multi-tenant, and long-term on Ceph, so retention is a question of buckets, not of disks filling on a Prometheus node. The more interesting question is where the numbers come from, because a Kubernetes cluster has no single source of them. Eight independent producers each see a slice, and Alloy scrapes them all into Mimir. Two of those producers only exist because this is bare metal, and they are the ones a cloud quietly kept from you: node-exporter for the host, and a Redfish exporter reading the baseboard controller, the same interface part one used to provision the machine, now reporting inlet temperature, fan speed, power draw, and drive health as metrics you can alert on.

The one worth naming out loud, because it trips everyone, is metrics-server. It is not a monitoring backend and never was. It holds a few seconds of live CPU and memory in memory to feed the autoscaler and kubectl top, and it stores nothing. Mimir is the history; metrics-server is just the live reading. Keep them straight and the rest falls into place.

Logs, traces, and profiles

The other three signals share the same cost model and differ only in what they index. Loki does not full-text index logs; it indexes the labels on a stream and stores the compressed bodies on Ceph, so a query selects streams by label and scans just those chunks, and because the labels are Prometheus-shaped, the selector that finds a metric finds its logs. Tempo indexes only the trace ID and keeps the spans on Ceph, then TraceQL queries them by attribute, so a p99 spike in a metric jumps straight to the one trace where the 400ms actually lived. Pyroscope goes inside the function the trace stops at, sampling stacks continuously at low overhead and rendering them as flame graphs where the widest self-time frame is the thing to fix. The unifying trick is correlation: an exemplar on a metric links to the trace, the trace links to its logs, and the trace links to its profile, so one pane walks the whole causal chain.

Collection: one agent, and eBPF for the rest

All of it is shipped by one agent. Grafana Alloy is not a monolith but a graph of components, receivers that gather telemetry, processors that batch and relabel it, and exporters that fan each signal to its backend, wired by referencing one component's output as the next one's input. It speaks OpenTelemetry end to end and runs as a DaemonSet, one per node. The gap it leaves is code you cannot or will not instrument, and Beyla, which Alloy packages, closes it: eBPF probes at the socket derive rate, errors, and duration, plus distributed traces, from HTTP, gRPC, and HTTP2 with no code change, on the same eBPF dataplane Cilium already runs. Watching the socket rather than a library, it even captures the queue time a real user feels.

// Alloy config is a component graph: each block's output feeds the next.
prometheus.scrape "kube" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.trim.receiver]
}

prometheus.relabel "trim" {                 // shed high-cardinality labels early
  rule { action = "labeldrop", regex = "pod_template_hash" }
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint { url = "http://mimir-distributor.observability.svc/api/v1/push" }
}

The mesh, from the kernel

Service-to-service traffic needs its own view, and because the dataplane is Cilium there is no sidecar to ask and no Kiali to run. Hubble reads flows straight from eBPF: L3 and L4 connections, L7 protocols with HTTP status and latency and DNS, and the network-policy verdict behind every allowed or dropped flow, all with no application change. Turned into metrics it feeds Mimir, so request rates and drops by route sit in the same dashboards as everything else; the Hubble UI, exposed through the gateway, draws the live service map from the kernel rather than from proxies. The mesh observability is not a new system, it is the CNI telling you what it already sees.

# Cilium values: Hubble flows become metrics in Mimir, the map on the gateway.
hubble:
  enabled: true
  metrics:
    enabled: [dns, drop, tcp, flow, "httpV2:exemplars=true"]  # scraped into Mimir
  relay: { enabled: true }
  ui:    { enabled: true }        # exposed via an HTTPRoute on the Cilium gateway

The storage, watching itself

There is a pleasing loop to close. The Ceph cluster stores every signal above, so it had better report its own health into the same place, and it does. Ceph's manager ships a Prometheus module exposing OSD state, pool capacity, placement-group health, recovery throughput, and client latency, and Rook wires it in with one flag that creates the ServiceMonitor Alloy scrapes. The storage layer the entire stack depends on now writes its own vital signs into a Mimir whose blocks live on that very storage. If Ceph starts to struggle, the metric that says so arrives before the dashboards it backs go dark.

# Rook CephCluster: turn on the mgr Prometheus module and its ServiceMonitor.
apiVersion: ceph.rook.io/v1
kind: CephCluster
spec:
  monitoring:
    enabled: true            # creates the ServiceMonitor that Alloy scrapes
    metricsDisabled: false

Runtime security as a signal

Observability is not only performance. What syscalls, process executions, and network connections actually happen inside a running container is a signal too, and on this platform it comes from the same eBPF dataplane as everything else. Tetragon attaches to kernel hooks and emits process, file, and network events described by TracingPolicies, so opening a sensitive file, spawning an unexpected shell, or making an egress connection becomes an observable event streamed to Loki and counted in Mimir. In this part Tetragon only watches; the same TracingPolicy can also enforce, killing the process in the kernel, but that decision belongs to the runtime-security part later. Here it earns its place as the fifth thing worth seeing: not is the app slow, but did anything run that should not have.

# Tetragon TracingPolicy: observe now, enforce later (part on runtime security).
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata: { name: watch-sensitive-files }
spec:
  kprobes:
    - call: security_file_open
      syscall: false
      args: [{ index: 0, type: file }]
      selectors:
        - matchArgs:
            - { index: 0, operator: Prefix, values: ["/etc/shadow"] }
          # a matchActions: Sigkill here would BLOCK; part eight only records.

The LLM stack, its own lens

An AI platform runs a class of workload that infrastructure metrics are blind to. A generation can be slow or expensive or wrong while every CPU and pod looks healthy, so the model-serving layer needs a lens of its own, and that is Langfuse. Its data model is containment, not a pipeline: one request is a trace holding observations, and the observation that calls a model is a generation carrying the model name, the prompt and completion, the token counts, and the cost derived from them. That is what lets the platform watch tokens, latency, and spend per model and per feature, not just in aggregate. Self-hosted, it keeps its event blobs on the Ceph RGW like everything else, with its Postgres and ClickHouse coming from the databases part still ahead.

# Langfuse, self-hosted: blobs on Ceph now, databases forward-referenced.
env:
  LANGFUSE_S3_EVENT_UPLOAD_BUCKET:   langfuse
  LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: http://rook-ceph-rgw-store.rook-ceph.svc:80
  DATABASE_URL:    postgresql://langfuse@cnpg-langfuse:5432/langfuse   # part 12
  CLICKHOUSE_URL:  http://clickhouse-langfuse:8123                     # part 12
  # every credential here is an ExternalSecret from OpenBao

Dashboards as code, and an MCP

None of this is worth much if Grafana itself is a snowflake configured by hand, so it is not. The Grafana Operator turns the instance, its datasources, and its dashboards into Kubernetes resources, reconciled by Argo CD like everything else, which means a datasource is a CRD and a dashboard is a versioned JSON reference, not a click no one can reproduce. Grafana runs on the Cilium gateway with a certificate from cert-manager and a login backed by OpenBao. And because an operator on this platform reads dashboards too, a Grafana MCP exposes them to the AI operator, so the same panels a human reads are a tool the agent can query.

# Grafana Operator: datasources and dashboards are CRDs, reconciled by Argo CD.
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata: { name: mimir }
spec:
  instanceSelector: { matchLabels: { app: grafana } }
  datasource:
    name: Mimir
    type: prometheus
    url: http://mimir-query-frontend.observability.svc/prometheus
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata: { name: ceph-overview }
spec:
  instanceSelector: { matchLabels: { app: grafana } }
  url: https://raw.githubusercontent.com/rook/rook/master/.../ceph-cluster.json

A page that arrives with its context

The last mile is turning a signal into a page a human can act on, and the difference between a good platform and a punishing one is what the page carries. A Mimir rule fires on a threshold; Alertmanager groups, deduplicates, and routes it; Robusta runs a playbook that staples on the pod's logs, the relevant graph, and the owning workload; and HolmesGPT investigates the enriched alert and adds a probable root cause before on-call is ever paged. As a note on the rules themselves: rather than hand-writing thresholds, an SLO tool like Pyrra turns an availability target into the recording and alerting rules, so you page on burn rate, not on a raw number. Either way, nobody is woken by a bare threshold.

# SLOs as a note: Pyrra turns an objective into burn-rate rules for you.
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata: { name: checkout-availability }
spec:
  target: "99.9"          # of requests succeed
  window: 4w
  indicator:
    ratio:
      errors: { metric: 'http_requests_total{service="checkout",status=~"5.."}' }
      total:  { metric: 'http_requests_total{service="checkout"}' }

The operator on the other end

That root cause in the page is not a template; it is HolmesGPT reading raw signals. Running on the cluster, it is wired read-only into every backend built in this part, Mimir, Loki, Tempo, Pyroscope, the Grafana MCP, and Kubernetes itself, so it reasons over the actual data the way an experienced operator would, only faster and around the clock. It does more than answer pages: it verifies deploys by diffing the signals before and after a rollout, runs scheduled health checks, and opens a pull request to fix what it finds. It is the reason all of this collection is worth doing at three in the morning, when there is no human awake to correlate five tools by hand.

What this buys

The platform can finally see itself, and it sees itself completely: four signals for performance, a lens for the models, a map of the mesh, the temperature of the iron, a feed of what runs inside every container, and the storage layer reporting its own health, all shipped by one agent, all stored on the object floor it already owned, all in one pane, and all watched by an operator that pages with the answer attached. That is not a monitoring add-on; it is the platform gaining a nervous system. And it changes what comes next, because a platform that can measure its own load is a platform that can respond to it. The next part uses exactly these signals to scale: to add and remove capacity, right-size workloads, and grow the cluster itself against real demand rather than a guess.