Give your voice AI agent real abilities (tool calling)

Summary

Annie Wang explains how to equip a Gemini Live voice agent with external capabilities through real-time tool calling. On its own, a multimodal language model can only produce words and cannot directly operate software controls like playing music, skipping tracks, or adjusting playback. To grant real abilities, developers provide the model with a menu of function declarations detailing each tool name, its parameters, and a natural language description explaining when it should be used.

During a live conversation, the interaction operates within a four-step loop: declare, decide, execute, and return. First, the application declares available tools during session configuration. When a user issues a voice prompt, the model decides to emit a structured tool call containing the appropriate arguments rather than generating plain speech. The application backend intercepts this call and executes the local code. Once complete, the backend returns the result to the model, which smoothly incorporates the outcome and continues speaking without breaking conversational flow.

Two methods exist for writing tools. The raw Google Gen AI SDK requires writing dictionary schemas and manually handling every dispatch, catch, execution, and return step. In contrast, the Agent Development Kit, or ADK, streamlines the implementation. Developers write standard Python functions with docstrings, and ADK automatically parses the signature schema and manages the execution loop. Adding a new ability, such as a sleep timer that automatically stops playback after a set duration, simply requires defining the function and appending it to the agent tool list.

Maintaining a natural conversational feel requires following two critical operational rules. First, because live voice models pause speech while waiting for tool responses, every tool must execute instantly. If an action requires extensive time, the tool should fire the task in the background and return an immediate success status so speech continues uninterrupted. Second, applications should use intercepting checkpoints like the before_tool_callback in ADK. This callback inspects every tool call before execution, enabling systems to observe logs, modify arguments, or block prohibited operations, such as preventing loud music during late-night quiet hours, while allowing the agent to maintain its conversational persona.