Observability, LGTM
Four telemetry signals on Grafana's LGTM stack (metrics, logs, traces, profiles), shipped by one eBPF-aware agent, plus Langfuse for the LLM stack, Hubble for the mesh, and a HolmesGPT-enriched alert path.
Observability gets flattened to "dashboards," but it is really four distinct telemetry signals, each answering a question the others cannot, plus purpose-built lenses for the LLM stack, the mesh, runtime security, and the alert path. This is the whole apparatus that keeps a 300-node, 250-service platform legible: what each signal is for, why the tools are built the way they are, and how a raw threshold becomes a page a human can actually act on.
Four signals, not one dashboard
The four signals answer questions none of the others can. Metrics tell you something is wrong; logs tell you what the app said; traces tell you where across services the time went; profiles tell you which function burned the CPU. The Grafana LGTM stack is one storage-and-query backend per signal, unified in a single Grafana pane so you can pivot from a spike to the log to the trace behind it.
The backends share a design goal: keep the index small and push the bulk to object storage. Mimir is horizontally scalable, multi-tenant long-term storage for Prometheus and OpenTelemetry metrics; Loki indexes only labels; Tempo indexes only the trace ID; Pyroscope stores sampled profiles. Cheap to retain, fast to correlate.
Observability is not one thing. It is four telemetry signals, each answering a different question, unified in Grafana. Tap one.
One request as a tree of spans hopping between services. Tempo stores traces by trace ID in object storage and needs no index beyond it, so it scales cheaply while TraceQL still lets you find traces by span attributes, duration, or status.
Logs: index the labels, grep the rest
Loki's defining choice is that it does not full-text index your logs. It indexes only the labels on a stream (a set of lines sharing the same labels) and stores the bodies as compressed chunks in S3. A query selects streams by label first, then scans just those chunks. The index stays a rounding error next to the data, which is how the same design runs from a Raspberry Pi to petabytes a day.
Because the labels are Prometheus-shaped, the selector that finds a metric finds its logs too. LogQL then layers line filters and query-time parsing on top, so you get structured search without paying to index every field up front.
Loki's whole trick: it never full-text indexes your logs. Tap a piece of the query to see what is indexed and what is scanned.
This is the only part Loki indexes. Labels identify a stream, a set of log lines that share the same labels. Loki jumps straight to the matching streams' chunks without touching anything else, which is why the index stays tiny even at petabyte scale.
Traces: the anatomy of one request
A trace is telemetry structured as a tree. One request becomes a root span, and every service it touches adds child spans that point back at their parent, so the tree can be rebuilt from flat span data. Each span carries four intrinsics (name, duration, status, kind) plus its identity (trace ID, span ID, parent span ID) and a bag of attributes: HTTP method, model name, DB statement, whatever the instrumentation attached.
That structure is why a trace localizes latency the way a metric cannot. A p99 spike says the endpoint is slow; the trace says the 418ms lived inside the model call, not the database. Tempo stores traces by trace ID in object storage, needing no heavier index, and TraceQL still queries them by attribute, duration, or status.
One request across four services. Each bar is a span, offset by start time and sized by duration. Tap a span to read its fields.
- trace_id
- 4bf92f3577b34da6a3ce929d0e0e4736
- span_id
- b7ad6b7169203331
- parent_span_id
- a1b2c3d4e5f60718
- gen_ai.request.model
- llama-4
- gen_ai.usage.input_tokens
- 1180
- gen_ai.usage.output_tokens
- 512
Profiles: the fourth signal
Metrics, logs, and traces stop at the function boundary; profiling goes inside it. Pyroscope continuously samples the stack in production at low overhead and stores the result as profiles you read as flame graphs, where the width of a frame is its share of samples and stack depth runs top to bottom.
The number that matters is self time versus total: total is time in a function including its callees, self is time in that function alone, and you optimize the widest self frame. Pyroscope collects CPU, allocations, in-use memory, goroutines, mutex, and block profiles, via language SDKs or eBPF, in the pprof format, so a regression points at a line, not just a slow route.
Continuous profiling samples the stack in production. Width is the share of samples in a function; depth is the call stack. Tap a frame.
The true hotspot: 38% of all CPU samples, entirely self time, in the matrix multiply. This is what a flame graph exists to surface.
Collection: one agent, and eBPF for the rest
All of it is shipped by Grafana Alloy, which is not a monolithic collector but a graph of components: receivers that gather telemetry, processors that batch, filter, and relabel it, and exporters that fan each signal out to its backend. You wire them by referencing one component's output as the next one's input, so a pipeline is a config graph rather than a rigid file, and it speaks OpenTelemetry end to end.
The gap Alloy alone leaves is code you cannot or will not instrument. Beyla, which Alloy packages, closes it: it attaches eBPF probes at the kernel network layer and derives RED metrics and distributed traces from HTTP, gRPC, and HTTP2 traffic with no code change. Because it watches the socket rather than the library, it even captures the queue time a real user feels.
Alloy is a pipeline of components: receive, process, export. Toggle how a service gets instrumented.
Collection components pull or accept telemetry: an OTLP receiver for app-emitted traces and metrics, a Prometheus scraper for metrics endpoints, and Beyla feeding eBPF-derived spans and RED metrics. Each is a component with typed arguments.
AI observability: tracing the LLM stack
The LLM platform (the Envoy AI Gateway fronting Llama 4 and GPT-OSS-20B) needs its own lens, because a slow or expensive generation is invisible to infrastructure metrics. Langfuse provides it. It models each request as a trace holding nested observations; the observation that calls a model is a generation, and it captures the model, the input and output, token counts, and the cost derived from those tokens and the model's price.
That data model is what lets us watch the stack per harness. Every product drives the LLM through a different framework: Deepagents in AI for Triage, LangChain in AI for Chat, LangGraph in AI for Adversarial Emulation, and CopilotKit in AI for dashboards. Langfuse traces all of them and rolls generations up to traces, models, and users, so TPS, latency, cost, and failures are visible per tool and per function, not just in aggregate.
Langfuse models a request as a trace holding nested observations. The one that hits a model is a generation. Pick a harness to see its trace.
Cost is derived from token usage and the model's price, so it is exact per call. Langfuse rolls these up from each generation to the trace, then by model, product, and user, which is how we watch TPS, latency, cost, and failures per harness across the whole LLM stack behind the Envoy AI Gateway.
Mesh observability: Kiali, then Hubble
Service-to-service traffic needs its own view, and ours changed with the mesh. Under Istio we read the mesh with Kiali, which draws the graph from the sidecars' Envoy telemetry. When we moved to Cilium we moved to Hubble, which reads flows straight from eBPF in the kernel: L3/L4 connections and L7 protocols (HTTP status and latency, DNS, Kafka) alike, plus the network-policy verdict behind every allowed or dropped flow, transparently and with no app change.
Both draw a service map. They are looking at completely different things to do it. Pick a dimension.
Shows what flowed. It does not tell you why a given connection was denied.
Shows the verdict for every flow, including which network policy dropped it and why.
Runtime security observability: Falco, and the Tetragon plan
Runtime security is an observability signal too: what syscalls and process activity are actually happening inside a running container. Today that is Falco, watching kernel events against a ruleset, with falcosidekick-ui as the console. It answers "did anything suspicious just run," and it does that well.
The plan is to move to Tetragon, for one decisive reason: it reads from the same eBPF datapath as the rest of our Cilium stack, and it can enforce, not only audit. Falco can tell you a policy was violated; Tetragon can stop the action in the kernel. We have not migrated yet, and the trade-offs are written up separately.
Alerting: a page that arrives with its own context
The last mile is turning a signal into a page a human can act on. A Prometheus rule fires on a PromQL threshold; AlertManager groups, deduplicates, and routes it; and then Robusta Classic (the rule-based enrichment engine, not the SaaS platform) runs playbooks that attach the pod's logs, a relevant graph, and likely remediation. HolmesGPT investigates the enriched alert and adds a probable root cause before OpsGenie ever pages on-call.
The point of the whole chain is that nobody is woken by a bare threshold. By the time the phone buzzes, the alert already carries its logs, its graph, and a hypothesis, which is the difference between a five-minute triage and a thirty-minute one.
An alert should reach a human already carrying its own context. Tap a stage in the path.
Robusta Classic (the rule-based enrichment engine, not the SaaS platform) receives the alert as a trigger and runs playbooks: actions that attach the pod's logs, a relevant graph, recent events, and likely remediation. The page stops being a bare threshold and becomes evidence.