Skip to main content

Quick Overview

In this developer Q&A session, Aarthi from OpenAI addresses questions submitted by the community across Reddit and the OpenAI Developer Community regarding GPT-Live. The video follows recent platform releases and covers model architecture, acoustic handling, tool delegation, and best practices for production voice implementations.

Key Points

  • 1.GPT-Live introduces full duplex architecture, enabling models to listen and speak at the same time and naturally discern interruptions from acknowledgments.
  • 2.Unlike GPT-Realtime, GPT-Live separates conversation handling from heavy execution by delegating complex tasks to backend models or systems.
  • 3.Handling noisy settings or multiple speakers can be guided via prompting to instruct the model to respond only when spoken to directly.
  • 4.New multilingual voices have been released, including French voices named Brise and Sillage, with additional custom voice generation possible through text prompts or recorded audio.
  • 5.Delegation can be managed natively using Decisions API session triggers or through client delegation using session commentary and thinking updates.

Summary

Aarthi opens the session by addressing questions submitted by the developer community regarding GPT-Live. In response to a query comparing GPT-Live to GPT-Realtime 2.1, she traces the architectural history of OpenAI's voice models. The earliest generation relied on a cascaded system composed of three chained components: speech-to-text, a language model such as 6.1 Sol for reasoning, and text-to-speech. While functional, each step introduced latency, resulting in sluggish and stilted speech. The subsequent GPT-Realtime series unified speech processing, generation, reasoning, and tool execution into a single speech-to-speech model. However, GPT-Realtime remained turn-based, requiring the model to determine when a speaker had stopped before generating a reply. GPT-Live addresses this through full duplex operation, which allows simultaneous listening and speaking to distinguish simple acknowledgments from actual conversational interruptions. It also introduces delegation, splitting real-time conversational responsibilities from backend execution models.

Addressing an inquiry about handling background noise and multiple voices in busy environments, Aarthi explains that GPT-Live leverages conversation context rather than solely relying on raw speech detection. This context helps the model determine when to continue, interject, or pause. Developers can further control this behaviour through prompting, such as giving instructions to ignore side conversations and reply only when directly addressed. Commercial adoption is highlighted through production use cases: Yelp processes reservation calls and food orders, Fin handles customer support inquiries, Speak delivers interactive language tutoring, and Cognition integrates voice into the Devin AI software engineering assistant.

When asked about language pronunciation, accents, and custom voices for non-English agents, Aarthi presents new voice releases with regional nuances, playing French audio samples of the voices Brise and Sillage. Developers seeking bespoke options can generate custom voices via prompt descriptions that define accent, tone, and pacing, or generate voices from audio recordings with speaker consent. Additional prompt adjustments allow teams to tune personality, formality, and phonetics.

On the technical challenge of managing delegation reliably over WebSockets, Aarthi advises keeping GPT-Live prompts short and clear regarding when to delegate versus when to stay in conversation. For GPT-Live 1, generating a specialized spawn thinking token improves delegation performance. Alternatively, developers can configure Decisions API triggers to automatically evaluate conversation transcripts and enforce delegation rules. To collect complex tool parameters without premature delegation, Aarthi recommends client delegation. Developers use the session commentary field to guide the immediate conversational response, while leveraging the thinking update field to track parameter checklists, ensuring all necessary fields are captured before executing a backend task.

Evolution of Voice Models to GPT-Live

The presentation outlines the progression from three-model cascaded systems (speech-to-text, language model reasoning, and text-to-speech) to unified speech-to-speech models like GPT-Realtime. While GPT-Realtime improved latency, it remained turn-based. GPT-Live adds full duplex capability to process and generate audio concurrently, alongside a delegation model that splits conversational management from backend task execution.

Handling Noisy and Multi-Speaker Environments

Older turn-based systems often struggle in busy environments trying to decide if someone has finished speaking. GPT-Live uses conversational context to distinguish conversational acknowledgments from interruptions. Developers can also steer behaviour through prompt engineering, instructing the model to ignore side conversations and respond solely when addressed.

Real-World Adoption and New Voice Options

Companies such as Yelp, Fin, Speak, and Cognition have implemented GPT-Live across reservations, customer support, language lessons, and AI software engineering. For regional and language support, OpenAI introduced new voices, including French voices Brise and Sillage, while allowing developers to generate custom voices using descriptive prompts or audio recordings.

Managing Delegation and Context

To handle complex tool calls and prevent over-delegation or missed delegations, developers can employ client delegation or native Decisions API triggers. In client delegation, backend instructions steer conversational queries in session commentary and manage parameter checklists within thinking updates, while automatic context compaction helps manage long conversations without losing critical details.

The Bottom Line

The video clarifies how GPT-Live differentiates itself from earlier voice architectures through full duplex audio and modular task delegation. It demonstrates concrete configuration paths for handling speech, voices, and tool calling via client delegation and API triggers. While clear best practices are established for prompt structuring and parameter gathering, managing edge cases in complex multistep workflows remains dependent on developer-configured policies and context compaction strategies.

FAQ

What is GPT-Live and how does it differ from previous voice models?

GPT-Live is OpenAI's voice model architecture featuring full duplex capabilities, allowing it to listen and speak simultaneously. Unlike turn-based cascaded or unified real-time models, it separates conversation management from heavy backend computation through task delegation.

How does GPT-Live handle noisy environments with multiple people speaking?

GPT-Live uses conversational context rather than simple voice activity detection to assess whether speech is an interruption or background noise. Developers can also guide this behaviour in system prompts by instructing the model to ignore side dialogue and only reply when directly addressed.

Which companies are currently using GPT-Live in their production products?

The video highlights Yelp for restaurant reservations and food orders, Fin for customer support, Speak for live language tutoring, and Cognition to add voice interaction to their AI software engineer, Devin.

How can developers create or customize French voices in GPT-Live?

Developers can use prebuilt multilingual voices such as Brise and Sillage, or create custom voices either by describing the voice's accent, pacing, and tone in a natural language prompt or by uploading a consent-backed audio recording.

What approach is recommended to prevent GPT-Live from over-delegating tool calls?

Developers can implement client delegation by using the session commentary field to instruct the model on the immediate next question to ask the user, while tracking required parameters in the thinking update field until all necessary information is collected.

How can developers ensure GPT-Live 1 reliably commits to backend delegation?

Developers should keep the main GPT-Live prompt concise, use the updated prompting guidance with the specialized SpawnThinking token, or integrate Decisions API triggers to monitor transcripts and force delegation programmatically.

Worth watching for

Developers and technical product teams building or migrating voice agents and interactive speech systems using OpenAI models.

  • gpt-live
  • voice-models
  • openai
  • full-duplex
  • speech-to-speech
  • developer-tools