wehub-resource-sync
516 lines
24 KiB
Plaintext
516 lines
24 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Copyright_01"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Header_01"
|
|
},
|
|
"source": [
|
|
"# ⚖️ Agent-as-a-Judge: Autonomous Trajectory Auditing for Tourism Swarms\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/agent_as_a_judge_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fagent_as_a_judge_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/agent_as_a_judge_eval.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/agent_as_a_judge_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Author_Architecture"
|
|
},
|
|
"source": [
|
|
"| Author | Core Framework | Evaluator Model | Target Worker Swarm |\n",
|
|
"| --- | --- | --- | --- |\n",
|
|
"| [Aniket Agrawal](https://github.com/aniketagrawal2012) | **Google ADK + Agent Platform Agent Runtime** | **Gemini 2.5 Pro** | **Gemini 2.5 Flash** |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Overview_02"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"Evaluating multi-step agentic systems requires moving past simple string comparisons of final outputs. Static approaches like *LLM-as-a-Judge* rely on zero-shot model inferences, forcing an LLM to count, divide, and assess complicated call trails strictly using raw text context—a method highly prone to grading inaccuracies.\n",
|
|
"\n",
|
|
"This notebook establishes an industry **Agent-as-a-Judge** implementation built via the **Google Agent Development Kit (ADK)** and hosted on **Agent Platform**. Unlike static configurations, our evaluating judge is itself an autonomous system equipped with a dedicated **Code Execution / Mathematical Evaluation tool**. It observes a worker tourism agent's actions, calculates operational metrics programmatically, and applies rule-based logic to grade agent performance.\n",
|
|
"\n",
|
|
"### Architectural Schema & Trajectory Tracking\n",
|
|
"\n",
|
|
"1. **Worker Swarm Execution (`gemini-2.5-flash`):** Fulfills a complex global tourism itinerary utilizing 100% real REST APIs (RESTCountries, Open-Meteo, Frankfurter, Deep-Translator).\n",
|
|
"2. **State Interception:** A runtime listener transparently logs each tool call, capturing raw inputs and schema outputs into an execution history trace.\n",
|
|
"3. **Agentic Judge Loop (`gemini-2.5-pro`):** Evaluates the entire trace autonomously, invoking its mathematical calculation tool to enforce path metrics strictly.\n",
|
|
"\n",
|
|
"### Metric Formulations\n",
|
|
"\n",
|
|
"* **Task Completion Rate ($TC$):** Ratio of successfully completed customer constraints:\n",
|
|
" $$TC = \\frac{\\sum_{i=1}^{N} C_i}{N}$$\n",
|
|
" Where $N$ is total prompt demands and $C_i \\in \\{0, 1\\}$ is verification of tool fulfillment.\n",
|
|
"* **Step Efficiency ($SE$):** Path optimization metric minimizing structural bloat or infinite tool looping:\n",
|
|
" $$SE = \\frac{U_{\\text{optimal}}}{U_{\\text{actual}} + R_{\\text{redundant}}}$$\n",
|
|
" Where $U_{\\text{optimal}}$ is the minimal necessary steps, $U_{\\text{actual}}$ is the tool volume used, and $R_{\\text{redundant}}$ flags errors or duplicate executions."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "1PQwMrdPyfFW"
|
|
},
|
|
"source": [
|
|
"## Before you begin\n",
|
|
"\n",
|
|
"1. In the Google Cloud console, on the project selector page, select or [create a Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).\n",
|
|
"2. [Make sure that billing is enabled for your Google Cloud project](https://cloud.google.com/billing/docs/how-to/verify-billing-enabled#console).\n",
|
|
"3. [Make sure the Agent Platform API is enabled](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"### Required roles\n",
|
|
"To get the permissions that you need to complete the tutorial, ask your administrator to grant you the [Agent Platform User](https://cloud.google.com/iam/docs/understanding-roles#aiplatform.user) (`roles/aiplatform.user`) IAM role on your project. For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Prerequisites_01"
|
|
},
|
|
"source": [
|
|
"### 1. Installation & Prerequisites\n",
|
|
"We establish environment configurations and install necessary packages for agent execution, geocoding, currency lookup, and language translations."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "PipInstall_01",
|
|
"outputId": "dd2e3e2a-24e1-46a1-c629-25cc14004725"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --quiet google-adk>=1.0.0\n",
|
|
"%pip install --quiet google-cloud-aiplatform>=1.97.0\n",
|
|
"%pip install --quiet langchain-google-vertexai\n",
|
|
"%pip install --quiet langchain-community requests deep-translator"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "1wc7ab-Wyn0X"
|
|
},
|
|
"source": [
|
|
"#### Authenticating your notebook environment\n",
|
|
"* If you are using **Colab** to run this notebook, run the cell below and continue.\n",
|
|
"* If you are using **Agent Platform Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "Authentication_01",
|
|
"outputId": "f3c6fa0d-28df-4b5e-a72f-2ab3bc0d5e1b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import sys\n",
|
|
"import vertexai\n",
|
|
"\n",
|
|
"if 'google.colab' in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
" auth.authenticate_user()\n",
|
|
" print('✅ Authenticated')\n",
|
|
"\n",
|
|
"# --- CONFIGURATION ---\n",
|
|
"PROJECT_ID = 'your-project-id' # @param {type:\"string\"}\n",
|
|
"REGION = 'us-central1' # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"os.environ['GOOGLE_CLOUD_PROJECT'] = PROJECT_ID\n",
|
|
"os.environ['GOOGLE_CLOUD_LOCATION'] = REGION\n",
|
|
"\n",
|
|
"vertexai.init(project=PROJECT_ID, location=REGION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Worker_Build"
|
|
},
|
|
"source": [
|
|
"### 2. Building the Worker Environment with Active Trajectory Interception\n",
|
|
"We wrap our live external tourism tools inside a programmatic `TrajectoryTracker` instance. As the target worker runs, it logs detailed histories for every API hit."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Worker_Tools_01"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import requests\n",
|
|
"import json\n",
|
|
"from deep_translator import GoogleTranslator\n",
|
|
"\n",
|
|
"class TrajectoryTracker:\n",
|
|
" def __init__(self):\n",
|
|
" self.history = []\n",
|
|
"\n",
|
|
" def log_step(self, tool_name, inputs, output):\n",
|
|
" self.history.append({\n",
|
|
" 'tool': tool_name,\n",
|
|
" 'inputs': inputs,\n",
|
|
" 'output': str(output)\n",
|
|
" })\n",
|
|
"\n",
|
|
" def clear(self):\n",
|
|
" self.history = []\n",
|
|
"\n",
|
|
"tracker = TrajectoryTracker()\n",
|
|
"\n",
|
|
"# --- REAL WORLD LIVE TOURISM SWARM TOOLS ---\n",
|
|
"\n",
|
|
"def query_tourism_graph(question: str) -> str:\n",
|
|
" '''Queries the Tourism Knowledge Graph to locate routes, prices, and target destinations.'''\n",
|
|
" res = 'Route: Tokyo to Kyoto via Shinkansen (Bullet Train). Duration: 2.2 hrs. Ticket Cost: 100 USD. Country: Japan. Attraction: Fushimi Inari Shrine.'\n",
|
|
" tracker.log_step('query_tourism_graph', {'question': question}, res)\n",
|
|
" return res\n",
|
|
"\n",
|
|
"def get_country_demographics(country: str) -> str:\n",
|
|
" '''Fetches live demographic metrics and currency mappings for a sovereign nation.'''\n",
|
|
" try:\n",
|
|
" url = f'https://restcountries.com/v3.1/name/{country}?fullText=true'\n",
|
|
" response = requests.get(url, timeout=5).json()[0]\n",
|
|
" currencies = list(response.get('currencies', {}).keys())\n",
|
|
" res = f'Country: {country}. Official Currency Codes: {currencies}.'\n",
|
|
" tracker.log_step('get_country_demographics', {'country': country}, res)\n",
|
|
" return res\n",
|
|
" except Exception as e:\n",
|
|
" return f'Error in demographics lookup: {e}'\n",
|
|
"\n",
|
|
"def live_currency_conversion(amount: float, from_currency: str, to_currency: str) -> str:\n",
|
|
" '''Converts fiat currencies using live European Central Bank indices.'''\n",
|
|
" try:\n",
|
|
" url = f'https://api.frankfurter.app/latest?amount={amount}&from={from_currency}&to={to_currency}'\n",
|
|
" data = requests.get(url, timeout=5).json()\n",
|
|
" converted = data.get('rates', {}).get(to_currency)\n",
|
|
" res = f'{amount} {from_currency} = {converted} {to_currency} (Rate Date: {data.get(\"date\")}).'\n",
|
|
" tracker.log_step('live_currency_conversion', {'amount': amount, 'from': from_currency, 'to': to_currency}, res)\n",
|
|
" return res\n",
|
|
" except Exception as e:\n",
|
|
" return f'Forex API lookup error: {e}'\n",
|
|
"\n",
|
|
"def get_live_weather(city: str) -> str:\n",
|
|
" '''Fetches actual environmental conditions and real-time temperatures for a global city.'''\n",
|
|
" try:\n",
|
|
" geo_url = f'https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1&format=json'\n",
|
|
" geo_data = requests.get(geo_url, timeout=5).json()\n",
|
|
" lat, lon = geo_data['results'][0]['latitude'], geo_data['results'][0]['longitude']\n",
|
|
"\n",
|
|
" weather_url = f'https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lon}¤t_weather=true'\n",
|
|
" weather_data = requests.get(weather_url, timeout=5).json()\n",
|
|
" cw = weather_data.get('current_weather', {})\n",
|
|
" res = f'Live Weather in {city}: {cw.get(\"temperature\")}°C.'\n",
|
|
" tracker.log_step('get_live_weather', {'city': city}, res)\n",
|
|
" return res\n",
|
|
" except Exception as e:\n",
|
|
" return f'Weather API mapping error: {e}'\n",
|
|
"\n",
|
|
"def translate_text(text: str, target_language_code: str) -> str:\n",
|
|
" '''Translates customer statements into target localized language codes natively.'''\n",
|
|
" try:\n",
|
|
" translated = GoogleTranslator(source='auto', target=target_language_code).translate(text)\n",
|
|
" res = f'Translated text to {target_language_code}: {translated}'\n",
|
|
" tracker.log_step('translate_text', {'text': text, 'target_lang': target_language_code}, res)\n",
|
|
" return res\n",
|
|
" except Exception as e:\n",
|
|
" return f'Translation layer error: {e}'\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Worker_Instantiation"
|
|
},
|
|
"source": [
|
|
"### 3. Initialize Worker Swarm Environment\n",
|
|
"We spin up our worker agent, powered by the fast and context-efficient `gemini-2.5-flash` model, ensuring tools are mapped exactly to task flows."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "Worker_Agent_Init",
|
|
"outputId": "31e0e053-8056-4a95-e0f4-359d27009f4d"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from google.adk.agents import Agent\n",
|
|
"\n",
|
|
"worker_travel_agent = Agent(\n",
|
|
" name='TourismOrchestratorSwarm',\n",
|
|
" model='gemini-2.5-flash',\n",
|
|
" description='Processes complex, multi-layered global travel requests using real world APIs.',\n",
|
|
" instruction='''\n",
|
|
" Fulfill user travel planning objectives systematically:\n",
|
|
" 1. Query the knowledge base via 'query_tourism_graph' to parse routing vectors, pricing, and country information.\n",
|
|
" 2. Execute 'get_country_demographics' to find the 3-letter currency code of the destination.\n",
|
|
" 3. Pass the currency code and matching traveler budget to 'live_currency_conversion' for real-time forex analysis.\n",
|
|
" 4. Use 'get_live_weather' to get current climate parameters.\n",
|
|
" 5. Handle localized translations via 'translate_text'.\n",
|
|
" Provide a polished, final summary breakdown mapping all data points accurately.\n",
|
|
" ''',\n",
|
|
" tools=[\n",
|
|
" query_tourism_graph,\n",
|
|
" get_country_demographics,\n",
|
|
" live_currency_conversion,\n",
|
|
" get_live_weather,\n",
|
|
" translate_text\n",
|
|
" ]\n",
|
|
")\n",
|
|
"print('✅ Worker Swarm Environment Initialized.')\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Generate_Trajectory_Data"
|
|
},
|
|
"source": [
|
|
"### 4. Execute Travel Request & Intercept Trajectory\n",
|
|
"We inject an enterprise travel prompt requiring multi-hop analysis. The local ADK application layer processes the request, while our listener records the sequential execution traces."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "Execute_Swarm_Run",
|
|
"outputId": "e2f148d6-7fe0-4d5b-81b3-910421bd5452"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from vertexai.preview import reasoning_engines\n",
|
|
"\n",
|
|
"tracker.clear() # Wipe clean before logging\n",
|
|
"worker_app = reasoning_engines.AdkApp(agent=worker_travel_agent)\n",
|
|
"\n",
|
|
"sample_user_prompt = '''\n",
|
|
"I am planning a trip from Tokyo to Kyoto to visit the Fushimi Inari Shrine.\n",
|
|
"Please look up the train ticket price and verify the current weather conditions in Kyoto right now.\n",
|
|
"Additionally, convert my $800 USD spending budget into the native currency of Japan using live exchange rates.\n",
|
|
"Finally, translate the statement 'Where is the train station?' into Japanese.\n",
|
|
"'''\n",
|
|
"\n",
|
|
"print('🏃♂️ Running Worker Swarm Pipeline...')\n",
|
|
"worker_response = ''\n",
|
|
"for event in worker_app.stream_query(user_id='client_traveler_101', message=sample_user_prompt):\n",
|
|
" if 'content' in event and 'parts' in event['content']:\n",
|
|
" part = event['content']['parts'][0]\n",
|
|
" if 'text' in part:\n",
|
|
" worker_response += part['text']\n",
|
|
"\n",
|
|
"print('\\n=== [WORKER AGENT FINAL OUTPUT] ===')\n",
|
|
"print(worker_response)\n",
|
|
"\n",
|
|
"print('\\n=== [CAPTURED INTERMEDIATE TRAJECTORY LOGS] ===')\n",
|
|
"captured_log_trail = json.dumps(tracker.history, indent=2)\n",
|
|
"print(captured_log_trail)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Judge_Implementation"
|
|
},
|
|
"source": [
|
|
"### 5. Architecting the \"Agent-as-a-Judge\" System with Math Execution Tools\n",
|
|
"We now build the autonomous evaluation layer. Instead of prompting an LLM to guess scores, our **Agentic Judge** runs on **Gemini 2.5 Pro** and is armed with a mathematical computing tool. The judge agent parses the raw json strings, executes the math tool to get precise metric valuations, audits response faithfulness, and returns a verified scorecard."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "Judge_Agent_Setup",
|
|
"outputId": "72aafd68-664d-411e-802f-ff90a8f79553"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def compute_trajectory_efficiency(target_optimal_steps: int, raw_trajectory_json: str) -> str:\n",
|
|
" '''Programmatic tool utilized by the Judge Agent to compute Step Efficiency (SE) without model hallucination.'''\n",
|
|
" try:\n",
|
|
" logs = json.loads(raw_trajectory_json)\n",
|
|
" actual_steps = len(logs)\n",
|
|
"\n",
|
|
" # Enforce mathematical Step Efficiency definition\n",
|
|
" se_score = min(1.0, float(target_optimal_steps) / max(1, actual_steps))\n",
|
|
" tools_used = list(set([step['tool'] for step in logs]))\n",
|
|
"\n",
|
|
" metrics_packet = {\n",
|
|
" 'mathematical_step_efficiency': round(se_score, 3),\n",
|
|
" 'total_invocations_observed': actual_steps,\n",
|
|
" 'unique_tools_accessed': tools_used,\n",
|
|
" 'contains_redundant_loops': actual_steps > target_optimal_steps\n",
|
|
" }\n",
|
|
" return json.dumps(metrics_packet)\n",
|
|
" except Exception as e:\n",
|
|
" return f'Error in mathematical parsing tool: {str(e)}'\n",
|
|
"\n",
|
|
"# Instantiate our autonomous Agentic Auditor\n",
|
|
"trajectory_judge_agent = Agent(\n",
|
|
" name='TrajectoryAuditorJudge',\n",
|
|
" model='gemini-2.5-pro', # Utilizing 2.5 Pro for multi-step data auditing and tool usage\n",
|
|
" description='An autonomous evaluation agent that calculates path quality and response faithfulness using custom execution tools.',\n",
|
|
" instruction='''\n",
|
|
" You are an autonomous Agent-as-a-Judge system tasked with evaluating worker agent run trajectories.\n",
|
|
"\n",
|
|
" Follow these execution steps:\n",
|
|
" 1. Call the 'compute_trajectory_efficiency' tool, passing it the target_optimal_steps (which is 5) and the raw trajectory logs to extract exact path efficiency statistics.\n",
|
|
" 2. Parse the User Prompt constraints. Compare the values returned by the tools (e.g. currency rates, weather temperatures)\n",
|
|
" against what the worker printed in the Final Response text to calculate Reasoning Faithfulness (detect any manufactured values).\n",
|
|
" 3. Format your final answer as a neat, comprehensive verification markdown scorecard summarizing the math tool results and your evaluation findings.\n",
|
|
" ''',\n",
|
|
" tools=[compute_trajectory_efficiency]\n",
|
|
")\n",
|
|
"print('✅ Tool-Equipped Agent-as-a-Judge initialized successfully.')\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Run_Evaluation_Section"
|
|
},
|
|
"source": [
|
|
"### 6. Execute the Evaluation Loop\n",
|
|
"We pipe the user query, final output, and trace history into our Judge Agent framework to observe the autonomous evaluation trace."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "Execute_Judge_Evaluation",
|
|
"outputId": "03ff4441-1391-4036-cf8b-587f0e4fa92a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"judge_app = reasoning_engines.AdkApp(agent=trajectory_judge_agent)\n",
|
|
"\n",
|
|
"judge_payload = f'''\n",
|
|
"[USER PROMPT CONSTRAINTS]\n",
|
|
"{sample_user_prompt}\n",
|
|
"\n",
|
|
"[AGENT FINAL RESPONSE TEXT]\n",
|
|
"{worker_response}\n",
|
|
"\n",
|
|
"[RAW TRAJECTORY LOGS TO PROCESS]\n",
|
|
"{captured_log_trail}\n",
|
|
"'''\n",
|
|
"\n",
|
|
"print('🕵️♂️ Activating Agent-as-a-Judge Execution Loop...')\n",
|
|
"print('=' * 60)\n",
|
|
"\n",
|
|
"for event in judge_app.stream_query(user_id='lead_qa_auditor', message=judge_payload):\n",
|
|
" if 'content' in event and 'parts' in event['content']:\n",
|
|
" part = event['content']['parts'][0]\n",
|
|
" if 'text' in part:\n",
|
|
" print(part['text'], end='')\n",
|
|
" elif 'function_call' in part:\n",
|
|
" print(f\"\\n[⚙️ JUDGE ENGINE -> Executing Trajectory Math Tool: {part['function_call']['name']}]\\n\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Conclusion_Markdown"
|
|
},
|
|
"source": [
|
|
"### 🏁 Conclusion\n",
|
|
"You now have a robust framework for auditing multi-step agents:\n",
|
|
"* **Trajectory Visibility:** By instrumenting the tool layer, we capture the actual trace rather than just guessing based on text output.\n",
|
|
"* **Autonomous Validation:** Enforcing programmatic validation via an independent agent ensures that automated software systems or monitoring dashboards can parse these performance records programmatically to flag loops, errors, or hallucinations before they reach production customers."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|