Key Takeaways
- A voice AI agent can pass every pre-deployment eval and still fail in production silently — the conversation sounds successful while the underlying action never completes, which is exactly the class of failure monitoring catches and evals don't
- Dead air is a monitoring problem, not just a UX one: LiveKit's own engineering guidance frames it directly — a tool call blocks the conversation while it runs, the agent goes silent, and the caller assumes the call dropped (LiveKit)
- Genuine WebRTC connectivity failure rates run close to 0.2% at platform scale, and sub-1% ICE failure is achievable in production with a properly configured TURN relay — track disconnect rate against that baseline, not against 0
- Prodinit's self-hosted LiveKit stack for Cuebo runs Celery tasks specifically for stuck-call cleanup and health monitoring — session health is a first-class monitoring surface, not an afterthought
- Real-time monitoring and pre-deployment evals should share one instrumentation layer: Prodinit's Langfuse setup for a 10,000+ calls/day voice AI client feeds live dashboards, CI quality gates, and retraining data collection from the same traces
Voice AI monitoring is the real-time layer that watches live production calls for the failure modes a pre-deployment eval suite structurally cannot catch — dead air, hung sessions, provider outages, and disconnects that happen only under real network and traffic conditions. Evals score quality against a rubric before or during rollout; monitoring catches what breaks after, on calls no test scenario simulated. Voice AI needs both, and they are not the same layer.
Why Evals Aren't Enough
A voice agent can score well on every eval dimension — faithfulness, latency, WER — and still fail in a way none of those metrics catch. A healthcare booking agent can confirm an appointment in conversation while the backend write silently fails; the transcript looks perfect, the faithfulness score is high, and the appointment never gets booked. This is a silent failure: the conversation sounds successful while the action underneath it doesn't complete, and it's invisible to any eval that only scores what was said.
Production is adversarial in ways a test harness isn't. Accents, background noise, mid-call interruptions, a third-party STT provider degrading for twenty minutes, a tool timeout under real load — these surface on live calls, not synthetic ones. The voice AI evaluation framework covers how to score quality; this is about catching the failures that happen regardless of how well the system scored going in.
The Voice AI Failure Taxonomy
Five failure categories account for most of what breaks a voice agent in production, and each needs a distinct detection signal — a single "error rate" metric catches almost none of them.
| Failure | What it looks like to the caller | Detection signal |
|---|---|---|
| Dead air / silent tool failure | Agent goes silent mid-call, or confirms an action that didn't happen | No audio output for N seconds after a tool call starts; tool call succeeds but downstream write fails |
| Stuck / hung session | Call never ends; room stays open with no active exchange | Session duration exceeds expected max; no turn activity within a heartbeat window |
| STT/TTS provider degradation | Garbled responses, agent misunderstanding everything, or no audio at all | Per-provider error rate or confidence distribution shifts from baseline |
| Disconnect / call-drop spike | Call ends abruptly, caller has to call back | ICE connection failures, WebRTC negotiation failures, above-baseline drop rate |
| Agent repetition loop | Agent says the same thing twice, or gets stuck in a response cycle | Consecutive-turn text similarity above threshold |
Dead Air and Silent Tool Failures
Dead air is what happens when a function tool call blocks the conversation while it runs — the LLM decides to call a tool, the runtime waits for it to return, and for however long that takes, the agent says nothing. LiveKit's own engineering guidance describes exactly this failure: a tool hits a backend that takes fifteen seconds to respond, the agent goes silent for the full fifteen seconds, and the caller assumes the call dropped and hangs up before the tool even finishes (LiveKit). The fix at the architecture level is decoupling execution from speech — acknowledging immediately, narrating progress, and letting the model keep talking while the tool runs.
Monitoring's job is catching the cases where that architecture isn't in place yet, or fails anyway: instrument every tool call with a start timestamp and alert when no audio output has been generated N seconds after the call began. A more dangerous variant is the silent tool failure — the model correctly decides to call a tool and formats the request properly, but the serving layer fails to parse or execute it, and the model hallucinates a plausible-sounding response instead of surfacing the failure. The conversation transcript reads as normal; only the missing downstream side effect (the booking that never wrote, the record that never updated) reveals the failure. Catching this requires correlating the agent's claimed action against the backend system's actual state — a reconciliation check, not a conversation-level metric.
Session Health: Stuck Calls and Disconnects
A voice agent's session lifecycle is a monitoring surface in its own right, separate from anything said during the call. A crashed agent worker can leave a LiveKit room open with no active exchange — the caller has hung up, but the session never formally closes, consuming resources and skewing concurrency metrics until something cleans it up. Prodinit's self-hosted LiveKit stack for Cuebo runs dedicated Celery tasks for exactly this: stuck-call cleanup and health monitoring, listening for participant_joined and room_finished webhook events and closing sessions that have gone stale rather than letting them accumulate.
Disconnects are a different signal — a call that ends because the underlying WebRTC connection failed, not because the conversation concluded. This is where a real baseline matters more than an intuition about what "good" looks like: genuine WebRTC connectivity failure rates run close to 0.2% at large platform scale, and production deployments with a properly configured TURN relay report sub-1% ICE failure rates as achievable, not aspirational (industry analysis of large-scale WebRTC connectivity data, webrtcHacks). Track disconnect rate against that baseline — a spike from 0.3% to 1.5% is a real signal worth paging on; treating any non-zero disconnect rate as an incident produces alert fatigue that trains the on-call rotation to ignore the channel.
Provider-Level Degradation: STT, TTS, and LLM
Third-party providers degrade independently of your own code, and that's a monitoring category the eval framework's per-call WER tracking doesn't cover — WER catches accuracy drift on a sampled validation set, not a live provider outage happening right now. Track error rate and latency per provider, per pipeline variant, as a real-time signal distinct from the quality scoring layer: a spike in STT error rate or a confidence-score distribution shift on one provider, isolated from the others, is a provider-side incident, not a model or prompt regression.
This distinction matters operationally because the fix is different. A provider degradation is a routing decision — fail over to a backup provider or pipeline variant — where a model or prompt regression requires rolling back a deployment. Prodinit's five-pipeline architecture for Cuebo (Deepgram, Azure OpenAI, ElevenLabs, Claude Sonnet, Sarvam, Gemini Live across variants) makes this distinction actionable: per-pipeline error rate tracking isolates whether a spike is provider-specific or systemic, and traffic can shift to an unaffected pipeline variant without a full deployment.
What Should Actually Page Someone
Not every metric belongs on a pager. Dashboard-only metrics — 7-day quality score trends, WER drift, token cost — inform decisions on a normal working cadence. Paging metrics are the ones where every minute of delay costs live calls: dead air rate crossing a threshold, disconnect rate spiking above baseline, a single provider's error rate jumping in the last five minutes. Static thresholds work poorly here because call volume itself is not static — a fixed error-count alert fires constantly at peak hours and misses real degradation during off-peak traffic. Anomaly detection relative to a rolling baseline, not a fixed number, is what keeps the alert channel meaningful instead of noise the team learns to ignore.
The instrumentation layer that makes this practical doesn't need to be separate from the eval and observability layers already in place. Prodinit's Langfuse setup for a voice AI client running 10,000+ calls/day feeds live dashboards, CI quality gates, and quarterly retraining data collection from the same underlying traces — one instrumentation pass serving real-time monitoring, evaluation, and the distillation pipeline that retrains on production data. Monitoring and evals read the same telemetry; they just answer different questions on different timescales.
Get Prodinit's AI engineering guides in your inbox
Deep-dives on production LLMs, voice AI, and MLOps — published weekly. No sales emails.
Frequently Asked Questions
Voice AI monitoring is the real-time layer that watches live production calls for failures — dead air, stuck sessions, provider outages, disconnects — as they happen. Evals score conversation quality against a rubric, typically pre-deployment or in CI. The distinction matters because a voice agent can pass every eval and still fail silently in production: the conversation sounds correct while the underlying action doesn't complete, which only live monitoring catches.
Dead air happens when a function tool call blocks the conversation while it runs — the agent waits for the tool to return before it can speak again, and the caller hears silence. LiveKit's engineering guidance documents this directly: a slow backend tool call can leave a caller in silence long enough that they assume the call dropped and hang up before the response arrives (LiveKit). The fix is decoupling tool execution from speech — acknowledging immediately and narrating progress instead of going silent.
Genuine connectivity failure rates run close to 0.2% at large platform scale, and production deployments with a properly configured TURN relay report sub-1% ICE failure rates as an achievable target (webrtcHacks analysis). Track disconnect rate against your own established baseline rather than treating any non-zero rate as an incident — a meaningful spike (for example, 0.3% jumping to 1.5%) is the signal worth paging on.
A silent tool failure happens when the model correctly calls a tool but the serving infrastructure fails to execute it, and the model hallucinates a plausible response instead of surfacing the error — the transcript looks normal, but the underlying action never happened. Catching it requires reconciling what the agent claimed happened against the backend system's actual state, not just monitoring the conversation transcript, since the failure is invisible at the conversation level by design.