LLM Observability in Production: What to Log and Why

Key Takeaways

• LLM observability in production requires logging four things per call: the full input context, the model output, per-layer latency, and at least one quality signal — without all four, you are operating blind on one or more failure dimensions
• Langfuse is the production-tested instrumentation layer for LLM observability: it captures traces, spans, and custom scores in a queryable format, integrates with LangChain and direct API calls, and supports async score posting so quality checks do not add per-request latency
• Quality scores posted back to Langfuse traces are what make continuous evals possible — the observability layer doubles as the data collection pipeline for the next fine-tuning or distillation run
• A voice AI client Prodinit instrumented with Langfuse at 10,000+ calls/day uses the same observability layer for live A/B monitoring, hallucination rate tracking, and quarterly retraining data collection — one instrumentation pass, three production uses

Most teams instrument LLM applications the same way they instrument microservices: p50/p95 latency, error rate, requests per second. Those metrics will not tell you that your model started confabulating on a specific query type three days ago. They will not tell you that your RAG retrieval quality degraded after a corpus update. They will not tell you that the fine-tuned model version you shipped last week has higher hallucination rate on edge cases than the previous version. By the time those failures show up in error rate or user-reported issues, they have already been in production long enough to matter.

LLM observability in production means logging the input, the output, per-layer latency, and a quality signal for every call — then building a continuous scoring and alerting layer on top of that telemetry. Standard APM covers the infrastructure; LLM-specific observability covers the model behaviour. Both are necessary. Only one is typically missing.

What LLM Observability Covers That Standard APM Does Not

Standard application monitoring captures whether a request succeeded and how long it took. LLM observability captures whether the request produced the right answer — a fundamentally different question that requires different instrumentation. At 10,000 calls/day, you cannot read every response. You need a telemetry layer that surfaces behaviour patterns, quality trends, and model regressions automatically.

Three gaps in standard APM define what LLM observability adds:

Prompt and context visibility. Standard APM logs an HTTP request. LLM observability logs the full input context — system prompt version, user message, retrieved documents if RAG, conversation history if multi-turn. Without logging the full input, you cannot diagnose why a model response was wrong. Was it a retrieval failure? A prompt regression from a recent system prompt edit? An edge case in user phrasing? The answer lives in the logged input.

Output quality signals. HTTP success (200 OK) means the API responded. It does not mean the response was accurate, grounded, or on-policy. LLM observability attaches quality scores to every response — either inline (adding latency) or asynchronously (posting scores after the response is delivered). Quality score trends over time surface model drift before it becomes a user complaint.

Model-level spans for multi-step pipelines. A RAG call involves retrieval, embedding, and generation — each with its own latency and failure surface. Voice AI adds STT and TTS spans. Standard APM reports the total request time; LLM observability reports each span separately, so a latency increase is attributable to retrieval, not generation, before any debugging begins.

The Four Things to Log for Every LLM Call

These four fields form the minimum viable observability schema for a production LLM system. Missing any one creates a blind spot in a different failure dimension.

1. Full input context (prompt + metadata)

Log the complete system prompt (or a version hash if prompts are large), the user message, any retrieved context documents (for RAG), and the model identifier and version. Also log: session ID, user ID if available, and the caller (which part of the application triggered this call).

The prompt version hash is critical: when a hallucination cluster surfaces, you need to know whether it correlates with a recent prompt change or is independent of prompt version. Without it, you are debugging blind.

2. Full model output

Log the raw model completion, not a processed or truncated version. Downstream quality checks — groundedness scoring, hallucination detection, LLM-as-judge — run against the raw output. Logging a processed version breaks the eval pipeline.

Also log: finish reason (stop vs length truncation), token counts (prompt tokens, completion tokens), and model temperature if it varies per call.

3. Per-span latency

For a simple LLM API call: time-to-first-token (TTFT) and total completion latency. For a RAG pipeline: retrieval latency, embedding latency, and generation latency as separate spans. For voice AI: STT latency, LLM TTFT, TTS latency, and WebRTC transport latency as separate spans (covered in the voice agent latency guide).

Aggregate latency conceals where problems are. If total latency increases by 400ms, it could be retrieval quality degradation causing the model to process more tokens, or it could be a network path change affecting the LLM API. Separate spans tell you which.

4. At least one quality signal

The quality signal is the gap between standard APM and LLM observability. It can be: a groundedness score (for RAG), a hallucination flag (from a reference check or self-consistency sample), a rubric score from an LLM-as-judge call, or a human feedback label (thumbs up/down). At minimum, log one automated quality signal — even a simple consistency flag — per sampled call. The LLM hallucination detection guide covers the detection methods that plug into this signal layer.

How to Instrument LLM Observability with Langfuse

Langfuse is the production observability layer Prodinit uses across LLM engagements. It captures traces (full request lifecycle), spans (per-step timing within a trace), and custom scores (quality signals posted synchronously or asynchronously). It integrates with LangChain, LlamaIndex, and direct API calls, and exports data in a format compatible with fine-tuning pipelines.

Basic Langfuse trace setup for a direct OpenAI call:

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context
import openai
import time

langfuse = Langfuse()
client = openai.OpenAI()

@observe()
def run_llm_call(system_prompt: str, user_message: str, model: str = "gpt-4o") -> dict:
    langfuse_context.update_current_observation(
        input={"system": system_prompt, "user": user_message},
        metadata={"model": model, "prompt_version": "v2.3"},
    )

    t0 = time.perf_counter()
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
        temperature=0.2,
    )
    latency_ms = round((time.perf_counter() - t0) * 1000)

    output = response.choices[0].message.content
    langfuse_context.update_current_observation(
        output=output,
        usage={
            "input": response.usage.prompt_tokens,
            "output": response.usage.completion_tokens,
        },
        metadata={"latency_ms": latency_ms, "finish_reason": response.choices[0].finish_reason},
    )

    return {"output": output, "trace_id": langfuse_context.get_current_trace_id()}

The @observe() decorator automatically creates a Langfuse trace for each function call. langfuse_context.update_current_observation() attaches the input, output, token usage, and metadata. The trace ID is returned so quality scores can be posted back asynchronously after the response is delivered.

Posting quality scores asynchronously — so eval checks do not add per-request latency:

import threading

def post_quality_score_async(trace_id: str, output: str, context: str = None):
    def _score():
        # Run groundedness check (or hallucination check, LLM-as-judge, etc.)
        score = compute_quality_score(output, context)
        langfuse.score(
            trace_id=trace_id,
            name="groundedness",          # or "hallucination_flag", "quality_rubric"
            value=score["groundedness_score"],
            comment=str(score.get("ungrounded_claims", [])),
        )
    thread = threading.Thread(target=_score, daemon=True)
    thread.start()

# In your request handler:
result = run_llm_call(system_prompt, user_message)
if context_docs:  # RAG system — post groundedness score async
    post_quality_score_async(result["trace_id"], result["output"], context=context_docs)

Async score posting keeps request latency clean. Scores appear in the Langfuse trace within seconds of the primary response and are immediately queryable for dashboards and alerts.

Multi-span instrumentation for a RAG pipeline:

from langfuse.decorators import observe

@observe()
def retrieve(query: str) -> list[str]:
    # Retrieval logic here — Langfuse auto-times this span
    return vector_store.similarity_search(query, k=5)

@observe()
def generate(context: list[str], question: str) -> str:
    # LLM call here — Langfuse auto-times this span
    ...

@observe()  # Parent trace wraps both child spans
def rag_pipeline(question: str) -> dict:
    docs = retrieve(question)
    answer = generate(docs, question)
    return {"answer": answer, "trace_id": langfuse_context.get_current_trace_id()}

Each @observe()-decorated function becomes a span in the parent trace. Langfuse records start time, end time, and duration for each span independently. In the Langfuse UI, retrieval latency and generation latency are visible as separate segments within the same trace.

ConversAI: One Instrumentation Pass, Three Production Uses

The highest-ROI argument for investing in LLM observability infrastructure is that the same telemetry layer serves multiple production uses: real-time monitoring, quality gate evaluation, and retraining data collection. Prodinit built this architecture for a voice AI client running 10,000+ calls/day.

The engagement began as a model distillation project — migrating inference from GPT-4.1 to a fine-tuned GPT-4o-mini to reduce inference costs by 70% (documented in the model distillation case study). The first phase of that engagement — weeks 0–2 — was Langfuse instrumentation. Every production call: input, output, model metadata, latency. That instrumentation pass created three production capabilities that persisted after the distillation engagement ended:

Real-time A/B monitoring. During the progressive rollout (10% → 25% → 50% → 75% → 90% traffic to GPT-4o-mini), Langfuse provided live dashboards tracking quality scores, hallucination rates, and latency deltas between the GPT-4.1 control track and the fine-tuned student model treatment track. Each of five stage gates was evaluated against Langfuse-collected data before traffic advanced. Zero rollbacks across all five transitions.

Stage gate evaluation data. At each stage gate, the hallucination detection check ran against a held-out call set stored in Langfuse. The same trace data used for live dashboards was also the source for stage gate evaluation — no separate evaluation infrastructure, no data pipeline to maintain alongside the monitoring pipeline.

Quarterly retraining data collection. The distillation pipeline runs on a quarterly cycle: collect 80,000–100,000 high-quality training examples from Langfuse, filter low-quality examples, fine-tune the student model. Langfuse is the data flywheel. Without production observability, the data collection step would require a separate logging system. With Langfuse already instrumented for monitoring, the training dataset is generated from the same traces used for dashboards. For context on the distillation pipeline this feeds, the model distillation guide covers the full pipeline from data collection through progressive A/B rollout.

What to Alert On in a Production LLM System

Logging is necessary but not sufficient. The alerting layer converts telemetry into operational signal — the dashboards that page someone before a user reports a problem.

Quality score p10 (not just mean). Mean quality score is a lagging indicator — it stays stable until failures are numerous. The 10th-percentile quality score is a leading indicator: a drop in p10 signals that a new failure mode is affecting a small but growing subset of calls, before it reaches the mean. Alert when p10 drops more than 0.08 below its 7-day baseline.

Hallucination rate (sampled). Run hallucination detection on 3–5% of traffic asynchronously. Track the 7-day rolling hallucination rate as a primary dashboard metric. Alert when the rolling rate exceeds 1.5× the baseline rate established during your stable production period. For the detection methods that generate this metric, the LLM hallucination detection guide covers reference scoring, self-consistency, and groundedness approaches.

Retrieval latency spike (RAG systems). A retrieval latency spike above 2× p95 baseline usually indicates an index freshness problem or a query distribution shift. Because retrieval latency is a separate span, it is attributable independently — not masked by generation latency.

Prompt version correlation. When a quality metric shifts, correlate the timing against prompt version changes logged in trace metadata. A quality drop that started exactly when a prompt update was deployed is a prompt regression, not a model regression — two different investigations, two different fixes.

Token count drift. A trend toward higher average completion token counts often indicates the model is generating longer, more hedged responses — a quality signal in itself, and also a cost signal. Alert when 7-day average completion tokens increase more than 20% above baseline with no corresponding change in request type distribution.