How to monitor & optimize Google Gemini agents with Grafana Cloud

Summary

Shawn Pitts from Grafana Labs demonstrates how to monitor and optimize an AI financial assistant built on Google Gemini using Grafana Cloud AI Observability. While local development allows engineers to manually inspect individual LLM prompts and tweak system instructions, this approach breaks down when applications reach production and serve thousands of concurrent requests. Managing production workloads requires scalable telemetry to continuously track performance, errors, and costs.

The demonstration begins with the Google Agent Development Kit and the financial assistant application codebase in the Cursor editor. To capture agent execution, developers instantiate the Grafana Sigil client using Grafana Cloud endpoint settings and API keys. The SDK serves as an OpenTelemetry-native layer. Within the code, streaming generation blocks and tool execution functions are wrapped with Sigil method calls. These lightweight wrappers capture start times, parameters, input payloads, return values, and execution duration without modifying the underlying agent logic.

Once instrumented, agent metrics flow into Grafana Cloud. The AI Observability overview dashboard displays top-level telemetry, including total requests, average and P95 latency, error rates, time to first token, and total cost breakdown. Engineers can drill down into specific tabs such as Performance, Errors, Usage, and Tools. The Tools view lists specific tool executions, such as transaction queries, showing call frequency, latency percentiles, and associated conversation threads. The conversation inspection panel displays the exact prompt, assistant output, token usage, and AI judge quality evaluations marked with status indicators.

To eliminate manual dashboard investigations, Pitts showcases an automated diagnostic workflow inside a Slack operations channel. By tagging the Grafana bot with an investigation request, the assistant automatically reviews agent documentation, discovers infrastructure components, and analyzes telemetry data sources. The bot returns a structured summary highlighting active bottlenecks, including rate-limit exhaustion errors, token inflation caused by large transaction lookups, and latency spikes on Gemini 2.5 Pro.

The remediation cycle finishes by tagging Claude Code directly in the same Slack thread. Referencing the Grafana diagnostic report, Claude generates an action plan with code fixes, such as implementing client-side retry handling with exponential backoff and capping transaction query return sizes. This workflow establishes an automated loop of observing, analyzing, improving, and repeating across production AI agent deployments.