GPT-Live Full-Duplex API: Voice-Agent Architecture

A voice agent that waits for silence before it thinks can be fast and still feel robotic. GPT-Live attacks that constraint at the architecture level: it keeps listening and speaking on a continuous path, while search, tools, and deeper reasoning run beside—not inside—the audio loop. The result is more natural interaction, but also a much harder system to operate.

GPT-Live full-duplex API architecture in one minute

GPT-Live is not simply a faster speech-to-speech endpoint. OpenAI describes it as a full-duplex voice system that can process incoming audio while generating outgoing audio, make interaction decisions many times per second, and delegate deeper work to a frontier model. OpenAI’s engineering write-up says the system was built around streaming inference, a stateful conversation, WebRTC transport, and asynchronous work outside the media path.

The practical model is:

The important change is ownership of time. In a conventional voice agent, the application waits for a user turn, sends it to a model, then plays a response. In GPT-Live’s design, the voice session remains active while several kinds of work happen concurrently.

The old turn-based assumption belongs in the background

Cascaded voice systems run speech-to-text, a language model, and text-to-speech in sequence. Native speech-to-speech models remove some handoffs, but a separate voice-activity detector can still decide when the user has finished before inference begins. A short thinking pause may look like the end of a turn; background sound may look like a new one.

GPT-Live’s full-duplex approach moves that timing problem into the voice model. The model can keep listening while it speaks, notice an interruption, pause, continue, or produce a brief acknowledgement. That does not eliminate turn boundaries everywhere. It means turn boundaries are no longer allowed to block the live audio loop.

What GPT-Live changes on the live path

Continuous inference replaces turn gating

In a full-duplex session, input and output are streams rather than alternating audio blobs. The model can receive new speech while its previous response is still being rendered. It can decide whether the incoming audio is a meaningful interruption, a short acknowledgement, or background noise.

This changes client logic. A client should be prepared to send, receive, cancel, and replace audio events concurrently. A single await response() abstraction is a poor fit for this behavior because it hides the events that matter most: speech started, assistant audio started, interruption detected, tool requested, response cancelled, and session ended.

Developers should still retain voice activity signals for UI, analytics, and safety. The architectural mistake is using VAD as the only authority that decides when model inference may start.

Keep media fast; move work off the path

OpenAI’s engineering account separates the dedicated audio path from application logic. Audio travels directly between the client and voice model, while tool calls, policy checks, persistence, and backend operations cross an asynchronous boundary.

That boundary gives the system a hard rule: a slow CRM lookup may delay its own answer, but it should not stop audio frames from arriving on time. WebRTC supplies the low-latency media transport; application services should not sit synchronously between every microphone frame and the model.

The voice layer can say something brief while a delegated task runs, but filler speech is not a substitute for a bounded job. Set deadlines, cancellation rules, and a safe result state for every tool.

Delegation keeps responsiveness and intelligence separate

GPT-Live can delegate search, deeper reasoning, or complex work to a frontier model. OpenAI’s launch and engineering posts identify GPT-5.5 as the delegated model at launch. The voice model remains responsible for the immediate interaction; the frontier model handles work that does not fit comfortably inside a low-latency speaking loop.

A production implementation should treat delegation as its own pipeline:

1. Detect that the request needs search, reasoning, or a tool.
2. Acknowledge or pause without blocking the media path.
3. Start the background job with the relevant conversation context.
4. Cancel it if the user changes direction or ends the session.
5. Validate the result in the application.
6. Inject a concise result back into the live session.

Pre-initializing the delegated inference session, keeping session affinity, and caching repeated context can reduce the delay before useful output arrives. The end-to-end budget includes routing, prompt processing, model inference, tool calls, and every model-to-tool round trip—not only model token latency.

Stateful sessions need a second architecture

A long voice call is not a sequence of disposable requests. Context grows, model workers change, and the session may need compaction. OpenAI describes warming a replacement model instance, prefilling it with the current context, and switching only after it is ready. That avoids making an infrastructure transition audible to the caller.

Context compaction creates a similar problem. Summarizing earlier turns changes the context that supports the model’s key-value cache. Rebuilding that cache in the foreground would create a pause. A safer design compacts context in parallel, prepares a replacement instance, and keeps the old instance serving until the handoff is ready.

For a voice-agent backend, session state should therefore include more than a transcript:

• Current audio and response state
• Active tool calls and cancellation tokens
• Model-instance or worker affinity
• Provisional and finalized messages
• Context-compaction status
• Reconnect and recovery state
• Safety and confirmation state

The API contract is an event system, not request-response

Full duplex changes the internal protocol even if the external API eventually offers familiar SDK methods. The application needs explicit distinctions between events that are often conflated:

GPT-Live can operate continuously, but the rest of a product still needs messages for the UI, analytics, and safety systems. OpenAI describes maintaining a speculative view that can be revised as transcripts arrive and an authoritative record that is finalized later. This is a useful pattern: show responsive captions without treating every partial transcript as an immutable fact.

What voice-agent teams must redesign

Separate the media adapter from agent orchestration

Put provider-specific transport and event handling behind an adapter. The application should consume normalized events such as user_audio_started, assistant_interrupted, tool_requested, confirmation_required, and response_completed.

Keep model ID, voice, prompts, tool schemas, and cost limits in configuration. This is not only migration insurance. It lets a team test a documented Realtime model today while preserving an explicit target for GPT-Live semantics later.

For tools, the model proposes and the application validates. Payments, account changes, cancellations, address edits, medical triage, financial actions, and identity workflows need confirmation rules outside the model’s spoken confidence.

Choose transport by where audio is controlled

WebRTC is the natural fit for browser and mobile clients that capture and play audio directly. WebSocket may remain useful for server-controlled media pipelines, but teams should not assume that every realtime model accepts the same session shape over every transport.

An OpenClaw integration issue documents the practical failure mode: treating gpt-live-1 as an ordinary GA Realtime WebSocket session produced an invalid_model response, while the proposed GPT-Live browser flow used a distinct WebRTC session shape. The issue is an implementation report, not an OpenAI API contract, but it reinforces the design rule: detect the model family and negotiate its supported session type explicitly.

Measure on-time frames, not only token latency

OpenAI’s engineering post reports that a supporting stream component saturated before GPU capacity in production testing. The useful capacity unit was concurrent sustainable sessions with on-time frame delivery, not requests per GPU.

Track at least:

• Audio-frame lateness and drops
• Time to first playable audio
• Interruption-to-stop time
• Concurrent sessions by region
• Reconnects and duplicate tool calls
• Delegation completion time
• Tool timeout and cancellation rates
• Provisional-to-final transcript corrections
• Abandoned sessions and spend per session

Naturalness also has a control problem. Real users may like interruption and acknowledgement behavior in one context and find it intrusive in another. One early user report summarized the risk bluntly: “It's literally cutting her off constantly lmao” (@AutismCapital). Treat that as a reminder to tune barge-in policy against real conversations, not scripted demos.

GPT-Live versus today’s Realtime design decision

The official OpenAI model catalog now positions GPT-Live 1 for natural, expressive voice conversations and highlights smooth interruption handling. That catalog is not the same thing as a complete integration contract: the separate GPT-Live API page reviewed here remains a notification form without endpoint, rate, or limit details. Teams should verify the current developer documentation and account entitlement before committing to a launch plan.

The architecture is worth adopting even before the model is available to every account. Continuous media, normalized events, asynchronous tools, and explicit cancellation all improve a voice agent built on a conventional realtime model.

GPT-Live full-duplex API FAQ

Is GPT-Live the same as GPT-Realtime?

No. OpenAI presents GPT-Live as a distinct voice-conversation model family, while GPT-Realtime is the documented realtime API family. Similar audio capabilities do not guarantee identical session semantics, transports, or model IDs.

Does full duplex mean the model never waits?

No. It means the system can listen and speak concurrently. The model can still pause, remain quiet, wait for clarification, or delay a delegated result when that is safer or more useful.

Do developers still need VAD?

Yes, for media UX, analytics, captions, and safety signals. VAD should not be the sole gate that forces the model into rigid user-turn/assistant-turn sequencing.

Which transport should a voice agent use?

Use the transport supported for the specific client and model. WebRTC is generally suited to direct browser or mobile audio; backend media pipelines may use WebSocket where documented. Do not infer transport support from the model name.

What should be built before access is confirmed?

Build the adapter, normalized event schema, tool-validation layer, cancellation model, cost telemetry, fallbacks, and long-session recovery. Those components remain useful if the final GPT-Live API contract changes.

Choose the architecture, not the model name

The durable decision is to stop treating voice as a request-response wrapper around a text model. Keep the audio path continuously available, move slow work behind asynchronous boundaries, make interruptions and cancellations first-class, and maintain a transcript that can be revised before it becomes authoritative.

GPT-Live’s trade-off is clear: more natural overlap and delegation require more state, more observability, and less control through simple turn boundaries. Teams that accept that complexity can design for the full-duplex contract now; teams that need a documented production endpoint should ship on Realtime with the same event-driven seams.