LLM Monitoring in Production: Metrics, Alerts and Drift

Key Takeaways

• LLM monitoring in production is the layer above observability: observability gives you scored traces, monitoring turns them into four metric families — latency, cost, quality, drift — with alert rules attached to each
• Use the OpenTelemetry GenAI semantic conventions as your metric naming baseline: gen_ai.client.operation.duration (histogram, seconds) and gen_ai.client.token.usage (histogram, filterable by gen_ai.token.type) are the two required primitives (OpenTelemetry)
• Alert on sustained budget burn, not single spikes: the SRE Workbook's multi-window multi-burn-rate pattern — a 14.4× burn over 1 hour confirmed by a 5-minute window — removes the false pages a naive threshold generates (Google SRE)
• Drift has three independent surfaces — input drift, prompt/config drift, and output quality drift — and each needs a different test: PSI or KL divergence on input distributions, version correlation on config, and scheduled golden-set replay on output
• Prodinit instruments monitoring as phase one of production AI engagements; on a voice AI platform running 10,000+ calls/day, the same Langfuse metric layer gated a five-stage progressive rollout with zero rollbacks

A fine-tuned model can degrade for three weeks without moving p95 latency, error rate, or throughput by a single percent. The request still returns 200 OK. What changed is the shape of the answers, and nothing in a standard Grafana dashboard is watching that. Most teams find out from a support ticket.

LLM monitoring in production means computing four families of metrics from your LLM telemetry — latency, cost, quality, and drift — and attaching alert rules to each one with thresholds baselined against your own stable period. Observability is the instrumentation that captures traces and scores. Monitoring is the operational layer that decides when a human gets paged.

What LLM Monitoring in Production Measures That Observability Does Not

Observability and monitoring are often used interchangeably, and the conflation costs teams real incidents. Observability is the capture layer: inputs, outputs, per-span latency, and quality scores written to a queryable store. Monitoring is the aggregation and alerting layer on top. Instrumenting Langfuse gives you the first and none of the second, which is why fully traced systems still miss regressions.

The practical difference is who looks at the data. Observability is pull-based: an engineer opens a trace because something is already suspected. Monitoring is push-based: a rule evaluates continuously and surfaces a problem nobody suspected. A team with full tracing and no alert rules has perfect hindsight and zero warning.

If you have not built the capture layer yet, start there: the LLM observability guide covers what to log per call and how to instrument it. This post assumes those traces exist and covers what to compute from them.

The Four Metric Families Worth Paging On

Four families cover the failure surface of a production LLM system; anything outside them is a dashboard nicety or belongs in standard APM. Name them using the OpenTelemetry GenAI semantic conventions so your metrics stay portable across Langfuse, Prometheus, and whatever you migrate to next.

1. Latency. gen_ai.client.operation.duration is a histogram in seconds, and the spec recommends bucket boundaries from 0.01s up to 81.92s — wide, because LLM latency distributions are nothing like HTTP. Track time-to-first-token separately from total completion time: TTFT is what a streaming user perceives, total duration is what your timeout budget consumes. Record each span of a multi-step pipeline independently — aggregate latency hides that a 400ms regression came from retrieval, not generation.

2. Cost. gen_ai.client.token.usage is a histogram you filter by gen_ai.token.type to split input from output tokens. Convert to currency per request and aggregate by route, tenant, and model version, not just globally: a global number tells you spend went up, per-route cost tells you which feature did it. Rising average completion tokens with no change in request mix is both a cost and a quality signal — the model is hedging more.

3. Quality. This is the family standard APM cannot produce. Track at least one automated score per sampled call: groundedness for RAG, a hallucination flag, or an LLM-as-judge rubric score. Monitor the 10th percentile, not the mean — mean quality holds steady while a new failure mode affects a small, growing slice of traffic, and p10 moves first. If you have no rubric yet, the LLM evaluation rubric template is a working starting point, and LLM-as-judge explains the scoring mechanism.

4. Drift. A derived family, computed on a schedule rather than per request: distribution distance on inputs, version correlation on prompts and configs, golden-set replay on outputs. It gets its own section below because it is the family teams most often skip and the one that produces the slowest, most expensive failures.

How to Set LLM Alert Thresholds That Don't Page on Noise

The failure mode of a first alerting pass is always the same: thresholds set on a single short window, firing on spikes that have already resolved, muted within two weeks. The fix is well established in SRE practice and transfers directly to LLM systems — alert on sustained error-budget burn, confirmed by two windows of different lengths.

Google's SRE Workbook describes multi-window, multi-burn-rate alerting: a page-tier rule fires only when a 1-hour window and a 5-minute window both exceed a 14.4× burn rate — roughly 2% of a 30-day error budget consumed in an hour. The long window proves the problem is sustained; the short one proves it is still happening. The Workbook's reference table adds a second page tier (6× over 6 hours, confirmed by 30 minutes) and a ticket tier (1× over 3 days, confirmed by 6 hours), with the short window set to roughly one twelfth of the long one.

The same structure, with LLM-specific signals:

Two rules about that table. Every threshold is a starting point to re-baseline against your own stable period, not a constant — quality score distributions are specific to your rubric and your traffic. And quality and drift signals need longer windows than latency signals: a 5-minute window on a score sampled at 3% of traffic is mostly sampling noise, which is why those rows pair 6 or 24 hours with a 30-minute or 2-hour confirmation.

Alerts also need enough context to act on. Every payload should carry the model and version, the prompt version hash, the route, and the last deploy timestamp. An alert that says "quality p10 dropped" starts an investigation; one that says "quality p10 dropped, prompt v2.4 deployed 90 minutes ago" starts a rollback.

How to Detect Drift in a Production LLM System

Drift in an LLM system is not one phenomenon. It has three independent surfaces — input drift, prompt and config drift, and output quality drift — each with a different detection method, a different cadence, and a different owner. Conflating them is why "the model drifted" so rarely leads anywhere useful.

Input drift — what users send you moves away from the distribution you built and evaluated against. Measure it with Population Stability Index or KL divergence over binned features: query length, language, intent class, retrieved-source mix, tenant share. PSI's conventional reading, attributed to Lewis, is that below 0.1 is no substantial change, 0.1–0.25 a small shift, 0.25 or above a substantial one. Treat those bands as convention rather than statistics: a critical review of population stability testing notes the cutoffs are not statistically derived. Bin edges must be identical across baseline and current samples or the number is meaningless.

Prompt and config drift — the system changed, not the world. System prompt edits, model version bumps by your provider, temperature changes, retrieval k changes, and index rebuilds all shift output behaviour. This surface needs no statistics, only discipline: log a prompt version hash and model version on every trace, then correlate every quality metric shift against the config change timeline before investigating anything else. In practice this resolves a large share of apparent model regressions, and it costs one metadata field.

Output quality drift — the answers themselves degrade. The only honest test is scheduled replay: hold a fixed golden set of 200–500 representative cases with known-good answers, run it nightly against production config, and score it with the same rubric you use online. A significant drop on a fixed input set isolates the model and pipeline from traffic changes, because the inputs did not move. Embedding-space distance on outputs (cosine distance against a stable-period centroid) is a cheap companion signal, but it detects change, not degradation — it fires on a legitimate prompt improvement just as readily.

Tooling splits along the same lines. Langfuse holds the scored traces; drift statistics are generally computed outside it, with a library such as Evidently AI or a scheduled job over exported traces writing results back as scores. Prometheus and Grafana stay the right home for latency and cost — ordinary time series. Expect two systems, not one.

Build prompt and config drift correlation first. It is the cheapest of the three, catches the most common cause, and the version metadata it needs is a prerequisite for interpreting the other two anyway.

What Monitoring Looks Like at 10,000+ Calls a Day

The argument for building this layer properly is that one metric set serves three jobs — live alerting, release gating, and training-data selection. Prodinit built exactly that architecture on a voice AI platform handling 10,000+ calls/day, documented in the model distillation case study.

The goal was a 70% inference cost reduction by distilling GPT-4.1 into a fine-tuned GPT-4o-mini, but the first phase was instrumentation and metrics, not training. Once quality scores, hallucination rates, and per-span latency were computed continuously, the same metric layer gated a five-stage progressive rollout — 10% → 25% → 50% → 75% → 90% of traffic to the student model — each gate evaluated against collected metrics before traffic advanced. Zero rollbacks across all five transitions. Those same scored traces supplied the 80,000–100,000 filtered training examples for each quarterly retraining run.

Prodinit treats this as phase one rather than a later hardening step, for a structural reason: you cannot safely ship a model change without a baseline, and a baseline takes a stable period of production traffic to establish. Teams that instrument monitoring after the first incident spend the next two weeks unable to tell whether their fix worked. For the surrounding practice, see the LLMOps guide; voice systems add failure modes generic LLM monitoring misses, covered in voice AI monitoring.