Real-Time AI Streaming Architecture for Low-Latency Apps

Key Takeaways

• A real-time AI streaming architecture is bounded by its transport, not its model: over HTTP/1.1 browsers cap open connections at roughly six per domain, and each EventSource holds one for the life of the stream — a limit MDN notes is marked "Won't fix" in Chrome and Firefox (MDN)
• For audio, TCP is the wrong substrate: a single lost packet makes the receiver hold back audio that has already arrived, and WebSockets supply no jitter buffer and no playout timing — you build both yourself (LiveKit)
• Latency is a per-stage budget, not a total. Retrieval, first token, inter-token delay, and client paint each need their own number, because only the first two respond to model choice
• Dropped streams are routine, not exceptional. OpenAI's background mode resumes with GET /v1/responses/{id}?stream=true&starting_after=N against a tracked sequence_number (OpenAI); the Vercel AI SDK does it with a resume option plus Redis-backed resumable-stream (AI SDK)
• Prodinit rebuilt Cuebo's real-time voice path off a Django WebSocket server onto a self-hosted LiveKit stack with 5 selectable AI pipelines, eliminated every database query over 500 ms, and took the platform to 10x peak load (case study)

Most AI features do not feel slow because the model is slow. They feel slow because the first byte arrives after everything has finished — retrieval, generation, serialization, validation — and the user spends four seconds looking at a spinner that could have been looking at text. Streaming is not a UI polish step you add at the end. It is an architectural decision that determines what your transport, your error handling, and your observability have to look like.

Real-time AI streaming architecture is the design of the path between a model's first output token and the user's screen: the transport that carries incremental results, the latency budget allocated across each stage, the resumption strategy for when the connection drops, and the instrumentation that tells you which stage regressed. Get the transport wrong and no model upgrade will fix the experience.

What a Real-Time AI Streaming Architecture Has to Do

A real-time AI streaming architecture has four jobs: deliver the first useful token fast, deliver subsequent tokens at a steady cadence, survive a connection that drops mid-generation, and let the user cancel work that is already running. Most implementations do the first well and the other three by accident, which is why they fail in production rather than in staging.

Those four jobs pull in different directions. Delivering the first token fast pushes you toward starting generation before upstream stages finish. Steady cadence pushes you toward buffering, which delays the first token. Surviving a dropped connection means the stream has to exist somewhere other than the open socket. And cancellation means the server has to learn that a client is gone, which neither HTTP nor a long-lived socket tells you promptly or reliably. If partial output is a new primitive for your team, our what is streaming and why LLMs use partial results answer covers the basics first.

The failure mode is always the same shape: a demo that works on a laptop over Wi-Fi, against one user, on a stable network, with a short response. Production is mobile radios, backgrounded tabs, proxies that buffer, load balancers with idle timeouts, and responses long enough that something will interrupt them. The architecture has to assume the stream breaks, because at scale it does.

Choose the Transport Before You Choose the Model

Transport selection bounds every latency number you can later achieve, and is near-irreversible once clients ship. The choice is three-way: Server-Sent Events for one-directional token streaming over plain HTTP, WebSocket for bidirectional text and control messages, and WebRTC for anything carrying live audio or video. Picking by familiarity rather than by payload is the most expensive mistake in this layer.

SSE is the right default for text. It is plain HTTP, it reconnects automatically, and the browser sends a Last-Event-ID header on reconnect so a server can replay what was missed — though that replay is explicitly best-effort and not a delivery guarantee. The constraint to design around is connection count: MDN documents the per-browser, per-domain cap and notes the issue was marked "Won't fix" in Chrome and Firefox, with HTTP/2 as the practical fix, since the number of simultaneous streams is then negotiated between client and server and defaults to 100 (MDN). If your app opens a stream per panel and the user keeps five tabs open, you will starve ordinary requests to the same origin before you notice.

Audio is where the choice stops being a preference. LiveKit's analysis of the failure is precise: when a packet is lost, TCP pauses the stream and retransmits before delivering anything after it, so the receiver holds back audio that has already arrived — producing silence followed by a burst of buffered sound — and WebSockets provide no jitter buffer or playout timing, so a developer has to build both (LiveKit). GetStream estimates that at a 100 ms round-trip time, a single packet loss introduces at least 100 ms of delay, and congestion backoff can stretch that to seconds (GetStream). WebRTC's media path prefers dropping or concealing late packets over pausing playback, which is the correct trade for a conversation and the wrong one for a file transfer.

The vendor APIs encode the same split. OpenAI's Realtime API is reached over WebRTC from browser and mobile clients and over WebSocket for server-to-server integration, which is why proxy implementations expose both surfaces — a token and SDP relay for the WebRTC path, a socket for the server path (LiteLLM). When Prodinit rebuilt Cuebo's voice platform, the migration was off a Django WebSocket server that handled real-time AI sessions directly and onto a self-hosted LiveKit server for WebRTC signalling and media routing, with standalone Python agent workers behind it. That separation is what let voice scale independently of the API tier at all, and the self-hosted LiveKit guide covers the deployment shape in detail.

Budget Latency by Stage, Not by Total

A single end-to-end latency number tells you nothing actionable, because the stages respond to different interventions. Split the budget into four: time to retrieval complete, time to first token, inter-token interval, and client render delay. Only the middle two move when you change models — which is why model swaps so often fail to fix a slow-feeling feature.

Retrieval sits entirely in front of generation and adds to time-to-first-token one-for-one, because the model cannot start until the context is assembled. That makes it the first place to look, and the interventions are infrastructural rather than model-level: precomputed embeddings, a cache in front of frequent queries, approximate nearest-neighbour search instead of exact, and a hard timeout that degrades to a smaller context rather than blocking the stream. A retrieval stage with no timeout is a streaming architecture with no floor on its first-token latency.

Inter-token interval is a cadence problem, and it is where serving configuration dominates. Continuous batching raises throughput, but queue depth adds directly to per-token latency, so a batch setting tuned for aggregate tokens per second can make every individual response feel worse. The honest framing is that throughput and interactivity are traded against each other at the serving layer, and the trade has to be made deliberately per workload rather than inherited from a default. Prodinit treats these as separate SLOs on separate dashboards for exactly this reason.

The largest wins usually come from overlapping stages rather than from speeding any one of them up. In a voice pipeline, starting the LLM on a partial transcript instead of waiting for final ASR output removes a serial dependency entirely; routing generated tokens straight into streaming TTS removes another. Each Cuebo pipeline runs a VAD → STT → LLM → TTS chain with stopword detection and reconnect timeout monitoring, built so the stages overlap rather than queue. For the voice-specific numbers and the per-layer breakdown, see our production voice AI latency architecture guide.

The Dropped Stream Is the Default Failure Mode

Treat a mid-stream disconnect as expected traffic, not an error path. A user switches tabs, a phone drops to a weaker radio, a proxy hits an idle timeout — and generation continues on the server, billed, with nowhere to go. Mid-stream failures cannot be retried transparently, so the stream must live somewhere independent of its socket.

The provider-level answer is a background run with a replayable event log. OpenAI's background mode creates the response with background and stream both true, has the client track the sequence_number of each event, and resumes with GET /v1/responses/{id}?stream=true&starting_after=N, returning only events after that sequence number (OpenAI). Azure's Responses surface exposes the same pattern with a StartingAfter option (Microsoft). One caveat worth testing before you depend on it: a developer-forum report describes an error stating a response can no longer be streamed once it is more than five minutes old, and that window does not appear in the official guide — so verify the retention behaviour against your own traffic rather than assuming it.

The application-level answer is the same idea with your own storage. The Vercel AI SDK exposes a resume option on useChat that reconnects to an active stream on mount, plus a consumeSseStream callback that registers the outgoing stream with the resumable-stream package in Redis; on a page reload the client issues a GET to the resume endpoint and picks the stream back up. Their docs note that wrapping the work in Next.js's after keeps the stream in Redis after the initial response returns, which is what makes a later reconnect possible at all (AI SDK). The resumable-stream package supports character-offset resumption via skipCharacters, so a reconnecting client fetches only what it missed.

Cancellation is the mirror image and gets forgotten more often. If a user navigates away, something has to tell the server to stop generating, or you pay for tokens nobody will read and hold a worker slot that another request needs. Wire an abort signal from the client through every layer of your own stack to the provider's cancel endpoint, and make the generation path cancellable rather than fire-and-forget. Whether a provider bills output tokens produced before an abort is a question to settle against their billing documentation rather than assume — but the worker slot is unambiguously yours.

Streaming Structured Output Follows Different Rules

Token-level streaming is a UI technique, and applying it to structured output is where correctness bugs enter. A JSON object is not meaningful until it is closed, so code acting on partial structure acts on something the model may still contradict. The rule: stream text to humans, buffer structure for machines.

If you do stream structured output to a UI, use a real incremental parser rather than re-parsing the accumulated buffer on each delta. One streaming-JSON library documents the cost of the naive approach plainly: re-parsing the whole buffer per delta is quadratic in response length, so a stream twice as long costs roughly four times as much (stream_struct). That cost is invisible on demo-length responses and very visible on a long tool-call argument.

The sharper risk is silent truncation. If a provider signals a mid-stream failure with an in-band event and your parser skips non-content frames, the stream ends normally with a partial value and no error raised — a truncated answer that looks finished. Anthropic's Messages API documents that the stream may occasionally carry errors in-band, alongside the normal message_start → content_block_start → content_block_delta → content_block_stop → message_delta → message_stop sequence and interspersed ping events (Claude docs). Handle the error event type explicitly, and treat a stream that ends without its terminal event as a failure rather than a completion.

What to Instrument on a Streaming Path

Instrument the stages, not the request. A streaming endpoint with a single duration metric is unobservable by construction: the number blends retrieval, first token, generation length, and network conditions into one figure that moves for reasons you cannot distinguish. The minimum useful set is five metrics at p50 and p95, because streaming regressions show up in the tail first.

Record time-to-first-token, retrieval duration, inter-token interval, total stream duration, and a disconnect-reason tag on every stream. The disconnect reason is the one teams skip and the one that pays: it separates client navigation from network failure from provider error from your own idle timeout, and those four have nothing in common operationally. Pair it with a resume-success rate once you have resumable streams, since a resume path that silently fails looks identical to a working one in aggregate latency charts.

Watch for the metrics that only exist on a streaming path. Tokens generated after client disconnect tells you what cancellation is costing. Stream completion rate — the share of streams that reach their terminal event — catches the silent-truncation class of bug that duration metrics never will. Our LLM observability in production guide covers the trace instrumentation underneath these, including keeping score posting off the request path so quality checks never add to first-token latency.

The instrumentation also has to cover the infrastructure that carries the streams, because that is where the ceiling sits. On Cuebo's platform, Prodinit scaled the WebSocket tier on active connection count and connection establishment rate rather than CPU, since a box full of idle long-lived connections looks unloaded to a CPU-based policy right up until it stops accepting new ones. Connection-aware autoscaling and queue-depth-aware worker scaling are what made 10x peak load survivable.