Skip to main content
AI Learning Resources

Quick Overview

This video is a hands-on technical demonstration from Google Cloud's AI Agent Clinic series featuring Dani Zamora, an Incubation Engineer at Google Cloud, and Matthew Feroz, Developer Advocate at Merge. The session walks through the process of building an automated evaluation pipeline for a LangGraph-based documentation agent within a 60-minute time constraint.

Key Points

  • 1.Evaluating AI agents with measurable metrics prevents silent failures and performance regressions before deployment to production environments.
  • 2.OpenTelemetry and OpenInference standards enable uniform capture of agent traces, spans, and metadata across different agentic frameworks like LangGraph and Google ADK.
  • 3.Antigravity can inspect agent source code and runtime traces to generate architecture diagrams and reproducible evaluation datasets from existing GitHub repositories.
  • 4.Evaluation suites require combining deterministic metrics such as latency and token costs with LLM-as-a-judge criteria like markdown formatting, citation fidelity, and grounding.
  • 5.Visual evaluation reports provide actionable score heatmaps that reveal hidden flaws in documentation generation and guide targeted prompt improvements.

Summary

In this episode of the AI Agent Clinic, host Dani Zamora from Google Cloud and guest Matthew Feroz from Merge set a challenge to build, configure, and execute an automated evaluation pipeline for an AI agent in under sixty minutes. The target agent is DocsHound, a LangGraph application designed to scan GitHub repositories, identify merged pull requests or closed issues lacking documentation, and draft Markdown pull requests to update project docs.

To begin the evaluation process, the hosts use Antigravity to analyze the DocsHound source code and its live execution traces on Google Cloud Run. Antigravity generates an architectural breakdown of the agent, mapping out its five workflow stages: routing, multi-source repository ingestion, gap detection and semantic clustering, grounding against official documentation, and documentation synthesis. It also sets up reproducible test scenarios by pulling historical pull requests and issue data from repositories like Pi Agent, T3 Code, and OpenCode.

To ensure compatibility across different agent frameworks, the workflow standardizes tracing using OpenTelemetry and OpenInference. While OpenTelemetry captures execution spans and telemetry pipelines, OpenInference standardizes semantic tags for agent tools, LLM calls, and retrievers. This enables the Google Cloud open-source agent-eval toolkit to ingest LangGraph traces without requiring the agent to be rebuilt on a native framework like the Agent Development Kit.

Next, the hosts translate subjective developer expectations into concrete metrics. Antigravity helps construct an evaluation configuration containing managed Vertex AI metrics such as Question Answering Quality alongside custom LLM-as-a-judge prompts. These prompts evaluate structure, actionability, technical precision, and LangGraph trajectory fidelity. Deterministic checkers are also configured to verify token consumption, wall-clock latency, and Python execution output.

Finally, the evaluation suite runs against the test dataset, outputting a detailed interactive HTML dashboard. The overview report details cost per question, wall-clock time, cache hit rate, and total token usage, alongside an LLM-judge radar chart and per-question score heatmaps. The evaluation reveals that DocsHound scored low on documentation quality due to missing code blocks and formatting issues, illustrating how automated evals provide clear data to guide prompt engineering and system updates.

Analyzing Agent Architecture with Antigravity

The demonstration begins by introducing DocsHound, a LangGraph agent created to turn GitHub pull requests and issues into documentation updates. Dani Zamora and Matthew Feroz connect an assistant called Antigravity to the DocsHound source code and execution logs. Antigravity parses the execution traces, maps the five-stage workflow architecture into a visual diagram, and extracts reproducible test scenarios from historical repository data.

Standardizing Evaluation Toolsets with OpenInference

To evaluate agents built across varying frameworks without rewriting testing pipelines, the hosts use OpenTelemetry and OpenInference standards. OpenTelemetry captures execution spans and operational telemetry, while OpenInference adds standardized semantic tags for agent tools, retrievers, and LLM calls. The open source agent-eval toolkit from Google Cloud uses these trace converters to connect LangGraph agent executions directly to Vertex AI evaluation services.

Translating Quality Definitions into Automated Metrics

Subjective human expectations of quality are translated into structured criteria for automated evaluation. Using Gemini as a judge alongside deterministic Python function checkers, Antigravity generates an evaluation configuration file containing rules for structural formatting, technical precision, confidence calibration, and trajectory validity. This configuration also incorporates managed metrics like question answering quality and grounding directly from the platform API.

Visualizing Results and Identifying Blind Spots

The evaluation pipeline executes against test datasets derived from repositories such as Pi Agent, T3 Code, and OpenCode. The generated HTML dashboard visualizes runtime metrics alongside an LLM-as-a-judge score heatmap. The results uncover an unexpected failure in DocsHound documentation quality due to missing code blocks, demonstrating how visual evals quickly pinpoint production blind spots that require prompt and code refinements.

The Bottom Line

The episode demonstrates that evaluating complex agentic workflows across different frameworks can be standardized and automated using open telemetry protocols and LLM judges. By executing an end-to-end evaluation run, the hosts successfully identify specific documentation formatting deficiencies in DocsHound that manual inspections missed. While the automated tooling streamlines metric generation and test execution, fine-tuning custom prompts and resolving platform-specific API error renderings remain ongoing tasks for production readiness.

FAQ

What is AI agent evaluation and why is it necessary for production deployment?

AI agent evaluation is the systematic measurement of an agent's reasoning, tool use, output quality, and operational cost against defined benchmarks. It is necessary because agent behavior can vary across prompts and models, making automated checks essential to prevent silent regressions and failures before shipping to production.

How does OpenInference extend OpenTelemetry for AI agent tracing and evaluation?

OpenTelemetry provides the underlying pipeline and structural span blocks to capture system behavior, while OpenInference extends it by adding standardized semantic tags for agent-specific actions, including tools, retrievers, and LLM calls.

What is DocsHound and how does it use LangGraph for documentation updates?

DocsHound is an AI agent built with LangGraph that analyzes open issues and merged pull requests in GitHub repositories, searches existing documentation to find information gaps, and drafts reviewable documentation pull requests.

How does the agent-eval toolkit bridge LangGraph agents with Gemini Enterprise Agent Platform?

The agent-eval toolkit uses trace converters to translate OpenInference and OpenTelemetry traces from frameworks like LangGraph into canonical data models accepted by Vertex AI and Gemini evaluation services.

What specific metrics are displayed in the Agent Evaluation dashboard generated during the demonstration?

The dashboard displays deterministic metrics such as total tokens, estimated cost, wall-clock latency, and cache hit rates, alongside LLM-as-a-judge scores for documentation quality, confidence calibration, trajectory validity, and citation fidelity.

Worth watching for

Software engineers, AI developers, and technical leads looking to implement robust testing, tracing, and automated evaluation pipelines for LLM agents before deploying them to production.

  • ai-agents
  • langgraph
  • vertex-ai
  • opentelemetry
  • openinference
  • llm-evaluation
Watch on YouTubeMore videos from Google Cloud Tech