Testing Voice Agents: Barge-In, Latency and Structured Eval Logs

Key Takeaways

• Barge-in testing requires two specific failure modes to cover: false barge-in (too sensitive, stops on noise) and missed barge-in (too slow, agent keeps talking); each needs a different test scenario type
• The three latency segments to instrument separately are STT (end-of-utterance to transcript), LLM TTFT (first token), and TTS streaming start — end-to-end target is p50 < 300ms, p95 < 700ms
• Structured eval logs per call — with barge-in events, latency segments, and quality scores as structured fields — are the prerequisite for any percentile analysis or regression detection at scale
• At 90K+ calls/month across 5 pipeline variants, per-call eval logs made it possible to isolate barge-in miss rates and latency regressions to specific pipelines without reviewing individual transcripts

Most voice agent problems are invisible without deliberate instrumentation. A 12% barge-in miss rate does not announce itself — it shows up as users complaining that the agent "doesn't listen." A p95 latency of 1.4 seconds is not obvious from demos — it appears in churn data. Neither problem surfaces until you have structured tests that look for it.

Testing a voice agent is not a unit test problem. It is a measurement problem: you need to instrument the right moments in a real-time audio pipeline, define what "acceptable" means as numeric thresholds, and build a harness that can tell you when a code change breaks one of those thresholds across a population of test calls — not just on a single happy-path demo.

Testing voice agents for barge-in and latency means defining three things: what to measure (barge-in lag, STT latency, LLM TTFT, TTS streaming start), what thresholds to enforce (p50 and p95 targets per segment), and what structured data to emit per call so you can run that analysis automatically. This post covers all three.

The Three Things That Break in Voice Agents (and Why They Are Hard to Catch)

Barge-in detection, end-to-end latency, and eval logging failures are the three categories that most commonly cause voice agent regressions to slip past demo-and-ship teams. Each is invisible without explicit instrumentation.

Barge-in detection is the mechanism by which the agent stops talking when the user starts speaking. It relies on VAD (Voice Activity Detection) detecting the user's audio stream while the agent's TTS is playing, and triggering an interruption signal that cancels the in-flight TTS output. Two things go wrong: the agent is too sensitive (background noise or a user's "mm-hmm" cancels the response prematurely — a false barge-in), or too slow (the user clearly starts a new sentence but the agent keeps talking for another 400ms — a missed barge-in). Both are failure modes with different root causes and different test scenarios.

End-to-end latency is the gap between the last word of the user's utterance and the first audio byte of the agent's response. Unlike barge-in, latency failures accumulate across three independently variable pipeline segments — STT endpointing, LLM first-token time, and TTS streaming start — so a single "end-to-end latency" metric masks which segment degraded after a change. When you instrument each segment separately, a deploy that slows down only LLM TTFT (because you switched models or changed the prompt) is immediately visible without hours of call review.

Missing eval logs is not a voice AI problem in itself — but it is the reason the other two problems go undetected. A voice agent that emits no structured per-call data cannot be compared before and after a deploy. You cannot compute p95 latency, cannot calculate barge-in miss rates, cannot diff pipeline variants. At 90K+ calls/month across 5 AI pipeline variants, per-call eval logs are the only way to isolate whether a barge-in regression belongs to one pipeline or all of them.

How to Test Barge-In Detection

Barge-in tests require scripted scenarios that trigger the two specific failure modes. Unlike latency tests, which can be measured on any call, barge-in tests need to be constructed: you need to inject user audio at a controlled point in the agent's response.

Test scenario 1: Mid-response interruption. The agent is mid-sentence (at least 3 seconds into a response). The user starts speaking a full utterance. The correct behaviour: the agent stops within 80–150ms of the STT detecting the interruption signal. Measure from the VAD/STT interruption event timestamp to the timestamp of the last TTS audio chunk sent. If the gap exceeds 200ms, flag the test.

Test scenario 2: Short interjection (false barge-in risk). The agent is mid-sentence. Inject a short non-interruption sound (ambient noise, 50ms audio spike, a brief "mm"). The correct behaviour: the agent does not stop. If the agent cancels its response on this signal, it is a false barge-in failure. The threshold: the agent should not barge-in on inputs shorter than 200ms of continuous VAD activity.

Test scenario 3: Rapid back-and-forth. The conversation is in a fast exchange — short agent responses, user immediately responding. Verify that barge-in sensitivity does not degrade during multi-turn short exchanges (some pipelines desensitise barge-in detection after a certain number of turns to avoid noise interference).

For LiveKit-based voice agents, the barge-in signal flows through the agent's on_user_speech_committed / interruption handler. The instrumentation point is the delta between when the LiveKit agents SDK fires the interruption event and when the TTS track is actually cancelled:

import time
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class BargeInEvent:
    detected_at: float = 0.0
    stopped_at: Optional[float] = None

    @property
    def lag_ms(self) -> Optional[float]:
        if self.stopped_at is None:
            return None
        return (self.stopped_at - self.detected_at) * 1000

# In your agent's interrupt handler:
class MyVoiceAgent:
    def __init__(self):
        self._barge_in_events: list[BargeInEvent] = []
        self._current_barge_in: Optional[BargeInEvent] = None

    async def on_interruption_detected(self):
        self._current_barge_in = BargeInEvent(detected_at=time.monotonic())

    async def on_tts_cancelled(self):
        if self._current_barge_in:
            self._current_barge_in.stopped_at = time.monotonic()
            self._barge_in_events.append(self._current_barge_in)
            self._current_barge_in = None

Threshold targets: barge-in lag p50 < 80ms, p95 < 150ms. A p95 above 250ms is a regression that will be felt by users as the agent "ignoring" interruptions.

How to Measure Latency in Production

End-to-end latency for a voice agent has three independently variable segments. Measuring only the total hides which segment is responsible when the number changes.

Segment 1 — STT latency: From the moment the user's final audio chunk reaches the STT service to the moment the transcript (with end-of-speech signal) arrives at the agent. Deepgram Aura and Nova-2 in streaming mode with endpointing: 300 (ms of silence before end-of-utterance) typically deliver transcripts in 150–350ms. Whisper via a REST call runs 400–900ms and is unsuitable for real-time voice AI at p95.

Segment 2 — LLM TTFT (time to first token): From the STT transcript arriving at the agent to the LLM streaming the first token of its response. This is the segment most affected by model changes (GPT-4o vs GPT-4o-mini), prompt length, and API provider load. Target: p50 < 300ms, p95 < 600ms on a pre-built voice-optimised prompt. Long system prompts (>2,000 tokens) add measurable TTFT.

Segment 3 — TTS streaming start: From the first LLM token arriving at the TTS service to the first audio byte being available for playback. For streaming TTS (ElevenLabs streaming, Azure TTS, Cartesia), the first audio chunk is available after the provider processes the first sentence boundary. Target: p50 < 150ms, p95 < 300ms.

End-to-end: Sum of segments 1–3. Target: p50 < 550ms, p95 < 1,000ms for a voice agent that does not feel sluggish. Sub-300ms total is achievable on the fast path (Deepgram + GPT-4o-mini + Cartesia) and the target for latency-sensitive use cases. The voice AI latency architecture post covers the stack choices that determine where each segment lands.

Instrument each segment with time.monotonic() spans in your agent code:

import time

async def handle_user_speech(transcript: str, agent_context) -> dict:
    t0 = time.monotonic()

    # LLM call
    response_stream = await llm_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=build_messages(transcript, agent_context),
        stream=True
    )
    first_token_at = None

    async for chunk in response_stream:
        if first_token_at is None and chunk.choices[0].delta.content:
            first_token_at = time.monotonic()
        # pipe tokens to TTS...

    tts_start_at = time.monotonic()  # stamp when first audio chunk dispatched

    return {
        "llm_ttft_ms": round((first_token_at - t0) * 1000, 1),
        "tts_start_ms": round((tts_start_at - t0) * 1000, 1),
    }

Combine this with the STT-to-transcript timestamp (available from Deepgram's streaming event metadata) to get all three segments per call.

Structured Eval Logs: The Schema That Makes Testing Useful

A structured eval log is a per-call JSON record written at call end that contains all the measurements taken during the call. It is the output of your instrumentation and the input to every analysis you run: percentile computation, regression detection, per-pipeline comparison, barge-in miss rate trends.

The minimum schema for a voice agent eval log:

{
  "call_id": "uuid",
  "pipeline_variant": "deepgram-gpt4o-mini-elevenlabs",
  "session_duration_s": 142,
  "latency": {
    "stt_ms_p50": 228,
    "llm_ttft_ms_p50": 312,
    "tts_start_ms_p50": 148,
    "e2e_ms_p50": 688,
    "e2e_ms_p95": 980
  },
  "barge_in": {
    "events": 3,
    "false_barge_in_count": 0,
    "missed_barge_in_count": 0,
    "lag_ms_p95": 112
  },
  "transcript_turns": 18,
  "quality_score": 0.87,
  "flags": []
}

At 90K+ calls/month across 5 pipeline variants (Deepgram, Azure OpenAI, ElevenLabs, Claude Sonnet, Sarvam, Gemini Live in different combinations), a schema like this makes it possible to query: "which pipeline variant had the highest barge-in miss rate in the last 7 days, and on which turn count range?" — without touching individual call transcripts. When one variant's barge-in p95 lag drifts from 112ms to 310ms after a Deepgram SDK update, the eval log surfaces it across the population before it reaches a support ticket.

Write these logs to a queryable store: Langfuse (as trace metadata), S3 with Athena, or Postgres with a jsonb column. The choice matters less than having a consistent schema and writing it on every call, not just flagged ones. Sampling loses the tail-distribution data that surfaces the worst barge-in and latency cases.

Building a Continuous Eval Harness

A one-time test is not a harness. A continuous eval harness runs a fixed suite of synthetic test calls on every deploy, compares results against thresholds, and fails the deploy if a regression is detected. It is the difference between "we tested it once" and "we know the barge-in miss rate did not change this sprint."

The synthetic test suite should cover at minimum:

• 5 barge-in test scenarios (mid-response interruption, short interjection, rapid exchange, long overlap, no interruption — baseline)
• 5 latency measurement calls (normal conversation, long user utterance, short user utterance, first turn, late turn)
• 2 edge-case calls (very short response, error path)

Run these 12 calls against your staging agent on every PR merge. Compare against the last passing baseline:

THRESHOLDS = {
    "e2e_ms_p95": 1000,      # fail if p95 latency exceeds 1 second
    "barge_in_lag_ms_p95": 150,  # fail if barge-in stops within 150ms at p95
    "false_barge_in_rate": 0.02, # fail if >2% of calls have false barge-in
    "missed_barge_in_rate": 0.05  # fail if >5% of barge-in events are missed
}

def check_thresholds(eval_results: list[dict]) -> list[str]:
    failures = []
    e2e_p95 = compute_percentile([r["latency"]["e2e_ms_p50"] for r in eval_results], 95)
    barge_in_p95 = compute_percentile(
        [e["lag_ms"] for r in eval_results for e in r["barge_in"]["events"]], 95
    )
    false_barge_in_rate = sum(
        r["barge_in"]["false_barge_in_count"] for r in eval_results
    ) / len(eval_results)

    if e2e_p95 > THRESHOLDS["e2e_ms_p95"]:
        failures.append(f"e2e_p95={e2e_p95}ms exceeds {THRESHOLDS['e2e_ms_p95']}ms")
    if barge_in_p95 > THRESHOLDS["barge_in_lag_ms_p95"]:
        failures.append(f"barge_in_lag_p95={barge_in_p95}ms exceeds {THRESHOLDS['barge_in_lag_ms_p95']}ms")
    if false_barge_in_rate > THRESHOLDS["false_barge_in_rate"]:
        failures.append(f"false_barge_in_rate={false_barge_in_rate:.1%}")

    return failures

The broader eval framework — quality scoring, WER, response coherence, call outcome rates — is covered in the voice AI evaluation framework. Barge-in and latency are the mechanical correctness layer; that framework covers the quality and outcome layer on top.