Back

Tools & Frameworks

Voice AI Infrastructure

Voice AI spans two layers: transport infrastructure that makes latency invisible, and model architecture that makes conversation natural. OpenAI's relay-plus-transceiver split solves WebRTC-in-Kubernetes at scale, while GPT-Live's full-duplex architecture and delegation pattern decouple continuous interaction from deep reasoning. In production, the cascaded pipeline (STT → LLM → TTS) remains dominant because it lets teams enforce guardrails, swap components per-market, and shave latency at each stage independently — but speech-to-speech models are closing the gap and hybrid architectures are emerging as the midterm answer.

Created May 26, 2026·Updated Oct 1, 2026

Recent Updates

WebRTC Foundations

Real-time voice AI has three hard infrastructure requirements: global reach, fast connection setup, and low, stable media round-trip time. WebRTC is the practical foundation because it standardizes ICE for NAT traversal, DTLS/SRTP for encrypted transport, codec negotiation, RTCP for quality control, and client-side echo cancellation and jitter buffering. The critical property for AI is that audio arrives as a continuous stream, enabling a model to transcribe, reason, and generate speech while the user is still talking.

The architectural challenge is deploying WebRTC inside Kubernetes at scale. The conventional one-port-per-session model requires exposing large public UDP port ranges that are difficult to secure, load-balance, and reconcile with pod autoscaling. ICE and DTLS are stateful protocols — the process that created a session must keep receiving that session's packets for connectivity checks, DTLS handshakes, and SRTP decryption.

Media Architecture Choices

Three main options exist for terminating WebRTC connections:

SFU (Selective Forwarding Unit) — a media server that terminates a separate WebRTC connection per participant and selectively forwards streams. Good for multiparty scenarios (group calls, classrooms) where signaling, media routing, recording, and observability live in one system. Discord uses this approach for concurrent voice.

Transceiver — a WebRTC edge service terminates the client connection and converts media into simpler internal protocols for inference, transcription, and speech generation. Backend services scale like ordinary services instead of acting as WebRTC peers. Best for 1:1 workloads (one user to one model) where latency matters on every turn.

TURN relay — an edge relay terminates client allocations and forwards traffic. Clients only need to reach the TURN address, but allocations add setup round-trips and are difficult to recover across servers.

OpenAI chose the transceiver model for its predominantly 1:1, latency-sensitive sessions.

Relay + Transceiver Architecture

The shipped architecture splits packet routing from protocol termination into two layers:

Relay — a lightweight, stateless UDP forwarding layer with a small public footprint. It parses just enough of each STUN packet header to read the ICE username fragment (ufrag), decode a routing hint, and forward to the correct transceiver. It does not decrypt media, run ICE state machines, or participate in codec negotiation.

Transceiver — the stateful WebRTC endpoint that owns all protocol state: ICE, DTLS, SRTP, and session lifecycle. It listens on a shared UDP socket (one OS endpoint, not one socket per session). From the client's perspective, the WebRTC session behaves exactly as standard.

The key insight is routing on the ICE ufrag — a protocol-native field already exchanged during session setup. OpenAI generates the server-side ufrag to contain routing metadata (destination cluster and owning transceiver), giving deterministic first-packet routing without any hot-path external lookup. If a relay restarts and loses its in-memory session map, the next STUN packet rebuilds the route from the ufrag. A Redis cache provides faster recovery before the next STUN packet arrives.

Global Relay and Geo-Steering

With the public UDP surface reduced to a small number of stable addresses and ports, the relay pattern deploys globally. Geographic ingress shortens the first client-to-infrastructure hop — lower latency, less jitter, fewer loss bursts before traffic reaches the backbone.

Cloudflare geo and proximity steering routes the initial signaling request (HTTP/WebSocket) to a nearby transceiver cluster. The SDP answer provides the Global Relay address, while the ufrag encodes enough information for relay to forward media to the designated cluster and transceiver.

Implementation Details

The relay is written in Go, kept deliberately narrow:

  • SO_REUSEPORT — multiple relay workers on the same machine bind the same UDP port; the kernel distributes incoming packets across workers, avoiding a single read-loop bottleneck.
  • runtime.LockOSThread — pins each UDP-reading goroutine to a specific OS thread, improving cache locality and reducing context switching when combined with SO_REUSEPORT.
  • Pre-allocated buffers and minimal copying — keeps parsing overhead and GC pressure low.
  • No kernel bypass — a userspace Go process reading from a standard socket was sufficient for global real-time media traffic, avoiding the operational complexity of kernel-bypass frameworks.

The relay maintains only ephemeral state: a small, short-timeout, in-memory map of client address to transceiver destination for forwarding and observability.

Key Design Principles

  • Preserve protocol semantics at the edge — clients speak standard WebRTC; browser and mobile interoperability stays intact.
  • Concentrate hard state in one place — transceiver owns all WebRTC session state; relay only forwards.
  • Route on information already present in setup — the ICE ufrag provides a first-packet routing hook with no added dependency.
  • Optimize for the common case first — careful use of SO_REUSEPORT, thread pinning, and low-allocation parsing before reaching for kernel bypass.
  • Add complexity in a thin routing layer — not in every backend service and not in custom client behavior.

Production Cascaded Pipeline

The cascaded architecture (STT → LLM → TTS) is the dominant production pattern across voice AI vendors. In practice, the pipeline is more complex than three boxes. Voice input arrives over a transport (WebRTC, phone call, or WebSocket), then passes through pre-processing models for background noise removal and voice isolation (selecting one speaker from a noisy foreground). The cleaned audio feeds into speech-to-text transcription, where a turn detection model — typically voice activity detection plus a "smart turn" classifier — determines whether the speaker has finished. Unlike text chat, where pressing Enter signals completion, voice has no explicit delimiter: a mid-sentence pause is different from a finished thought.

After transcription, the text goes to an LLM for inference, which may return a direct response, trigger tool calls, or both. The LLM output then feeds into text-to-speech, which is typically faster than real-time. The synthesized audio streams back over the same transport to the user's speakers.

The cascade's strength is composability: each stage can be independently swapped, benchmarked, and optimized. Teams report shaving 10 milliseconds at a time across individual pipeline stages to make the overall experience feel responsive. Open-source tooling (from projects like Pipecat/Daily) publishes per-stage benchmarks for STT, LLM inference, and TTS, allowing teams to test new models against their own conversation flows without going live.

Speech-to-Speech vs Cascaded

The original motivation for speech-to-speech models was preserving emotional information lost in the STT step — a sad tone and an excited tone both transcribe to the same text, so the LLM responds identically to both. Speech-to-speech models process audio natively, enabling tone-aware and more natural responses.

However, production deployments still overwhelmingly use cascaded pipelines for three reasons. First, the accuracy of a dedicated STT model typically exceeds the encoder of a speech-to-speech model. Second, cascaded pipelines allow inserting supervisor models at each junction — checking for prompt injection, social engineering, factual grounding, and intent classification between stages. Third, swapping individual components (a better TTS for Japanese, a cheaper LLM for simple intents) is straightforward in a cascade and nearly impossible in an end-to-end model.

Speech-to-speech models like Smallest AI's Hydra are evolving toward multimodal input and output (speech and text simultaneously), enabling tool calls and guardrails within the speech-to-speech paradigm. The likely midterm outcome is a hybrid: speech-to-speech handles the "autopilot" portions of a conversation — the 50% of interactions an agent can answer without deep reasoning — while cascaded pipelines handle complex lookups, multi-step workflows, and cases requiring explicit guardrails. A semaphore-like mechanism gates between the two paths, interrupting the speech-to-speech response when the system detects a question requiring the cascade's rigor.

Turn-Taking

Turn-taking — knowing when to respond versus when the user is pausing to think — is a deceptively hard problem in voice AI. In text, the user presses Enter. In voice, there is no explicit signal. Systems combine voice activity detection (VAD) with learned turn-detection models that distinguish a mid-sentence pause from an end-of-utterance silence. Getting this wrong in either direction is costly: responding too early creates interruptions, responding too late creates awkward dead air.

The human brain processes conversation asynchronously — forming thoughts while still listening, interrupting when ready, and taking background notes (analogous to tool calls) in parallel. Replicating this requires moving beyond synchronous cascade processing toward architectures where the system can listen, think, and act concurrently. Voice AI practitioners note they solved problems like context compaction, conversation-state tracking across turns, and graceful degradation under long contexts a generation before coding agents encountered the same challenges at scale.

Latency-Intelligence-Cost Tradeoffs

The fundamental tension in voice agents is a three-way tradeoff between response intelligence, latency, and cost. More capable models produce better answers but respond more slowly. Larger system prompts improve coverage but increase both latency and token cost — especially wasteful on calls where users hang up within 10 seconds.

Practitioners manage this tradeoff through several techniques:

Contextual fillers — short acknowledgment phrases ("just give me a sec," "looking at that now") that keep the conversation feeling alive while the system processes complex queries or waits for slow tool calls. Testing shows that zero fillers sometimes feels more robotic than natural filler usage, since humans routinely say filler words while thinking.

Parallel model execution — forking the audio or transcribed text to multiple models simultaneously and gating the output downstream. This mirrors computer architecture patterns: run several candidate responses in parallel, then select or merge the best result. The speech-to-speech path can fork separately from the cascade path, with a downstream selector choosing which to surface.

Specialized small models — fine-tuned small language models (like Smallest AI's Electron) paired with better knowledge bases and memory can outperform frontier models for specific voice use cases at lower cost and latency, with the added benefit of self-hosted scaling control. Frontier API latency spikes are unpredictable and uncontrollable; self-hosted models let teams scale to demand.

Multi-agent delegation — offloading specialized sub-tasks (e.g., credit card collection) to constrained agents with minimal prompts, then returning to the main conversational agent. This keeps the primary agent's context lean and reduces latency for the common path.

Prompt Architecture for Voice

Voice agent prompt design diverges from text-chat patterns because latency compounds with prompt size. Two main approaches exist:

Single comprehensive prompt — one large system prompt containing all rules, workflows, and knowledge. Works well for outbound use cases where the conversation's direction is unpredictable and the agent needs to draw on any part of its instructions at any moment. The downside is cost (paying for the full prompt on every turn, including calls that hang up immediately) and the well-known tendency of LLMs to attend primarily to the first and last ~4% of context while losing information in the middle.

Node/graph-based workflows — decomposing the conversation into states with dedicated, minimal prompts per state. Works well for inbound use cases where the call has a finite set of intents and predictable flow. Each node loads only the context relevant to that stage, reducing latency and improving reliability. The tradeoff is that erratic or off-script user behavior can fall between nodes.

In practice, the choice is customer-specific. Some teams use condition-checked prompt compilation — evaluating which prompt sections are relevant to the current customer type and conversation state before assembling the system prompt for each turn. This was essential when models struggled with long prompts, but as frontier models improve at context utilization, some teams are swinging back toward larger single prompts and letting the model figure out relevance.

Inbound vs Outbound

Inbound and outbound voice agents face different constraint profiles that shape their architecture:

Outbound (the agent initiates the call) — the agent knows its objective and can predict most conversation flows. Guardrails are simpler because the agent can drop calls that go off-script. The main challenge is the first few seconds: users frequently hang up as soon as they realize they're talking to a bot. This makes prompt cost and first-response latency critical, since every hung-up call still pays for the full context window.

Inbound (the user calls in) — the agent has some context about its purpose but no knowledge of why this specific person is calling. For simple businesses (a flower shop, a barbershop), the intent space is small. For complex enterprises (Amazon-scale product catalogs with different return policies per category), the agent must dynamically retrieve and reason over relevant policy without stuffing everything into context. This is where retrieval quality, context compaction, and multi-agent delegation become essential — you cannot put a billion-product catalog into a single prompt without the model hallucinating on the policy buried in the middle.

Voice Architecture Evolution

OpenAI's voice systems have progressed through three architectural generations, each removing a layer of interaction friction:

Cascaded voice — the original ChatGPT Voice chained three separate models: speech-to-text, a language model (GPT-5.5), and text-to-speech. This enabled talking to frontier models for the first time but at the cost of information loss across model boundaries and high latency.

Turn-based voice — Advanced Voice Mode (GPT-4o era) processed and generated audio within a single model, reducing latency. But it still operated through discrete turns: the model waited for silence before responding, and silence-based turn detection caused false interruptions from pauses or background noise.

Full-duplex voice (GPT-Live) — the current generation, which processes input continuously while generating output. See GPT-Live: Full-Duplex Voice below.

GPT-Live: Full-Duplex Voice

GPT-Live (launched July 2026) introduces two architectural changes that move voice AI from turn-based to continuous interaction.

Continuous Interaction via Full-Duplex

Instead of processing a sequence of separate messages, GPT-Live continuously processes input while generating output. The model makes interaction decisions many times per second — whether to speak, continue listening, pause, interrupt, or invoke a tool. This enables natural back-and-forth: the model can acknowledge with phrases like "mhmm" or "got it," wait when the user pauses to think, and handle interruptions gracefully rather than relying on silence-based turn detection.

Delegation for Deep Work

GPT-Live decouples continuous interaction from deep reasoning. When a question requires web search, extended reasoning, or agentic capabilities, GPT-Live delegates to a frontier model (GPT-5.5 at launch) running in the background. The voice model keeps the conversation going while the delegated task completes, then brings the result back into the conversation. This separation means GPT-Live can continuously benefit from the latest frontier models without retraining the voice model itself.

Delegation maps to three reasoning effort levels: Instant (GPT-5.5 Instant — fast responses for simple lookups), Medium, and High (GPT-5.5 Thinking with medium or high reasoning effort for multi-step analysis). The split matters because in text chat, users tolerate waiting for a reasoning model, but in voice a long silence feels broken. Delegation lets the voice layer fill the gap — asking follow-up questions, confirming constraints, or simply acknowledging that deeper work is running — while the background model takes whatever time the problem requires source(https://blog.risingstack.com/chatgpt-live-voice-ai-architecture/).

Two Model Tiers

GPT-Live ships in two variants: GPT-Live-1 (for Plus, Go, and Pro users) and GPT-Live-1 mini (for Free users). In evaluations, GPT-Live-1 is strongly preferred over Advanced Voice Mode across overall preference, turn-taking, interruptions, conversational flow, and naturalness in matched 5–10 minute conversations. It also substantially outperforms on GPQA (expert-level scientific reasoning) and BrowseComp (agentic web search).

Voice Safety Architecture

GPT-Live adds real-time safeguards that can act while the model is speaking — steering toward safer responses, surfacing safety messaging, or ending voice conversations in higher-risk cases. Audio output includes SynthID watermarking for provenance verification, with both a public verification tool and API access for developers source(https://openai.com/index/introducing-gpt-live/). Age-appropriate behavior is trained directly into the model for teen users, with parental controls and linked-parent notifications for high-risk situations.

Voice as Orchestration Layer

The deeper implication of GPT-Live's architecture is that voice becomes a coordination surface rather than a speech-enabled chatbot. The voice model manages the conversation while other models, tools, and services perform specialized work behind the scenes — a pattern that mirrors distributed-system design where a responsive front-end accepts requests and delegates slower work to backend components.

This extends to output format. Voice is the control layer, but the system can choose the most suitable response format: speech for short explanations, text for exact wording, visual cards for structured data (weather, stocks, sports), maps for locations, or files for finished artifacts. Forcing every answer into spoken form would reproduce the limitations of old telephone menus where users listened through long lists that would be easier to scan visually.

Five design principles emerge for building on this pattern:

  • Separate interaction from execution — the component talking to the user does not need to perform every task itself. A fast interaction model handles turn-taking and acknowledgement; other models or services handle search, reasoning, and tool use.
  • Treat interruption as data — an interruption may signal a correction, a changed mind, a new constraint, or simply that the user already understands. Systems should preserve and interpret interruptions rather than just stopping output.
  • Design for asynchronous work — the interface should remain useful while background tasks run, allowing clarifying questions, progress updates, or work on a parallel sub-task.
  • Make state visible — when multiple models and tools work behind the scenes, users need to know whether the system is listening, searching, delegating, or waiting. Natural conversation should not completely hide system state.
  • Choose output modality per answer — not every response belongs in audio; structured results are often better displayed visually.

Sources

  • ChatGPT Live and the New Architecture of Voice AI — RisingStack Engineering — Reasoning effort levels (Instant/Medium/High), voice-as-orchestration-layer framing, developer design principles for full-duplex voice systems, multimodal output selection
  • Introducing GPT-Live — GPT-Live full-duplex architecture, delegation to frontier models, voice architecture evolution (cascaded → turn-based → full-duplex), evaluation results, SynthID audio watermarking, voice safety design
  • Forward Deployed: Voice AI on what works in 2026 — Latent Space / Basil (video, 2026) — Production cascaded pipeline anatomy, speech-to-speech vs cascaded tradeoffs, turn-taking challenges, latency-intelligence-cost tradeoffs, prompt architecture (single vs graph), inbound vs outbound constraint profiles, multilingual deployment (Japan, Arabic), self-hosted small models. Featuring practitioners from Decagon, Vapy, Daily, Smallest AI.
  • How OpenAI delivers low-latency voice AI at scale — Full architecture walkthrough of the relay + transceiver split, ICE ufrag routing, Global Relay, and Go implementation details
  • Andon FM — AI agents running autonomous radio stations 24/7
  • "Real-time semantic search for Conversational AI" — Moss (article, May 2026) — real-time semantic search (<10ms) for voice AI and conversational systems
  • "Moss adoption is growing 200% MoM" — Moss (tweet, May 2026) — YC F25; 50+ companies, 130+ projects, voice AI and multimodal agent teams