Quick Overview
This video is a technical developer guide presented by Developer Relations Engineer Katie Nguyen for Google Cloud Tech. It follows up on an earlier overview video by explaining the code architecture, API endpoints, and configuration parameters for Gemini 3.5 Transcribe.
Key Points
- 1.Gemini 3.5 Transcribe is a specialized speech-to-text model designed for audio inputs such as customer calls, meetings, voice notes, and dictation.
- 2.Developers can integrate transcription via the Google Gen AI SDK using two distinct APIs: synchronous generateContent and the real-time streaming Live API.
- 3.Synchronous transcription supports audio inputs via direct byte arrays or Google Cloud Storage URIs.
- 4.Configuration options in the AudioTranscriptionConfig include word-level timestamps, pinned or auto-detected language codes, speaker diarization, and custom vocabulary lists.
- 5.Real-time streaming over WebSockets uses concurrent asynchronous tasks to stream raw PCM audio chunks while receiving interim and finalized transcription message types.
Summary
Katie Nguyen introduces Gemini 3.5 Transcribe, a specialized speech-to-text model built to transcribe audio files accurately across multiple speakers, background noise, and specialized domain jargon. The model provides dedicated configuration controls for speaker diarization, word-level timestamps, and custom vocabulary tailored for enterprise audio inputs like customer calls, meetings, voice notes, and dictation.
Developers access the transcription capabilities through the Google Gen AI Python SDK. Initialization requires installing the google-genai library and instantiating a client pointing to the Gemini Enterprise Agent Platform with specified project and location identifiers. The SDK supports two distinct transcription APIs that correspond to two separate model IDs: the batch model gemini-3.5-transcribe-preview used with the generateContent API for pre-recorded files, and the live model gemini-3.5-transcribe-live-preview used with the live.connect Live API for streaming real-time audio.
For synchronous batch processing, audio data can be passed as raw bytes from local files loaded in memory or directly as Google Cloud Storage URI paths using types.Part.from_uri with no separate upload step. Behavior is managed inside AudioTranscriptionConfig nested within GenerateContentConfig. Setting word_timestamp to true returns individual word boundaries with precise start and end offsets. The model auto-detects spoken languages by default, but developers can pin specific language codes, such as Korean and English, to support multilingual transcripts. Enabling diarization attaches speaker labels to each segment to distinguish conversational turns, while custom vocabulary allows passing domain terms or product names to improve transcription accuracy.
For real-time transcription, developers open a persistent session over WebSockets with the Live API using live.connect. The session configuration specifies the response modality as text along with an input audio transcription configuration supporting language codes and custom vocabulary. The client architecture runs two asynchronous coroutines concurrently: one coroutine pushes raw PCM audio byte blocks in real time tagged with the appropriate MIME type and sample rate before signaling audio stream end, while the receiving coroutine processes incoming server content. The receiver separates messages into interim input transcriptions for dynamic, in-place caption updates and final input transcriptions to append permanent segments to the completed transcript.
Overview and Client Setup
Gemini 3.5 Transcribe is introduced as a specialized speech-to-text model capable of handling domain jargon, background noise, and multi-speaker audio across both pre-recorded files and real-time streams. Developers access the model through the Google Gen AI SDK by initializing a client configured for the Gemini Enterprise Agent Platform with specific project and location parameters. Depending on whether the use case involves batch processing or live audio streaming, separate model IDs are selected.
Synchronous Transcription and Configurations
Synchronous transcription runs through the generateContent API using the batch model ID for pre-recorded audio files. Audio data can be supplied either directly as in-memory bytes or referenced from Cloud Storage by URI without requiring an explicit upload. Developers can configure specific features within AudioTranscriptionConfig, including word-level start and end timestamps, auto-detected or pinned multilingual language codes, speaker diarization labels, and custom vocabulary lists to bias recognition toward specific terms and product names.
Real-Time Streaming with the Live API
Streaming transcription uses the Live API over WebSockets via live.connect to process continuous audio input in a persistent session. The implementation uses two asynchronous coroutines running concurrently, where one captures and sends raw PCM audio chunks while the other listens for responses. The server returns interim transcription messages for real-time draft captioning as well as finalized transcription segments meant for the permanent transcript record.
The Bottom Line
The video establishes the technical implementation workflow for Gemini 3.5 Transcribe across both batch audio processing and live streaming transcription using the Google Gen AI SDK. It demonstrates the precise code structures and configuration parameters required to enable diarization, timestamping, custom vocabulary, and multilingual detection. The presentation concludes by directing developers to the documentation and sample notebook for full code execution.
FAQ
What is Gemini 3.5 Transcribe and what are its primary transcription capabilities?
Gemini 3.5 Transcribe is a specialized speech-to-text model designed for transcribing customer calls, meetings, voice notes, and dictation with domain jargon understanding, noise tolerance, speaker diarization, word timestamps, and custom vocabulary support.
Which Python SDK and client configuration are required to access Gemini 3.5 Transcribe?
Transcription runs through the Google Gen AI SDK (google-genai), initialized by creating a client configured with client_enterprise=True, a Google Cloud project ID, and a location.
What are the differences between synchronous transcription and real-time streaming transcription in the API?
Synchronous transcription uses the generateContent API with the model gemini-3.5-transcribe-preview for complete pre-recorded files, while real-time streaming uses the Live API (live.connect) over WebSockets with the model gemini-3.5-transcribe-live-preview for continuous audio input.
How can audio files stored in Google Cloud Storage be passed to the synchronous transcription API?
Audio in Cloud Storage can be passed by referencing its Cloud Storage GS path directly using types.Part.from_uri without needing a local download or file upload.
What is the function of the custom vocabulary parameter in AudioTranscriptionConfig?
The custom vocabulary parameter accepts a list of phrases or terms to bias the model toward recognizing specific product names, technical jargon, or uncommon spellings.
How do interim input transcription messages differ from final input transcription messages during Live API streaming?
Interim input transcription messages represent live, updating draft guesses suitable for updating captions in place, whereas final input transcription messages represent finalized segments to be permanently recorded.
Worth watching for
Software engineers and cloud developers looking to integrate speech-to-text, speaker diarization, and real-time streaming audio transcription into their applications using the Google Gen AI SDK.
- gemini-3-5-transcribe
- speech-to-text
- google-gen-ai-sdk
- audio-transcription
- live-api