wehub-resource-sync
434 lines
17 KiB
Plaintext
434 lines
17 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Create a Gen AI Agent Evaluation for a Deployed Agent\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fcreate_genai_agent_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| Author(s) |\n",
|
|
"| --- |\n",
|
|
"| [Kelsi Lakey](https://github.com/lakeyk) |\n",
|
|
"| [Bo Zheng](https://github.com/coolalexzb) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This Colab notebook demonstrates how to use the Gen AI Eval SDK to evaluate a deployed Agent.\n",
|
|
"\n",
|
|
"- **Run Agent Inference:** First run agent inference to retrieve real responses and traces from the deployed agent.\n",
|
|
"- **Create Evaluation Run:** Then create an Evaluation Run to perform the Gen AI Agent Evaluation. This Evaluation will be persisted and accessible later.\n",
|
|
"\n",
|
|
"If you do not have a deployed Agent, please see:\n",
|
|
"- [Create & Deploy Agent and Run Gen AI Agent Evaluation]()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK and other required packages\n",
|
|
"Restart runtime after installation to load latest packages\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --force-reinstall -q google-cloud-aiplatform[evaluation]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B",
|
|
"cellView": "form"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import vertexai\n",
|
|
"from google.cloud import storage\n",
|
|
"from google.genai import types as genai_types\n",
|
|
"from vertexai import Client\n",
|
|
"\n",
|
|
"def get_or_create_gcs_bucket(project_id: str, gcs_dest: str) -> str:\n",
|
|
" \"\"\"Retrieves GCS bucket or creates a default.\"\"\"\n",
|
|
" full_path = gcs_dest or f\"{project_id}/agent-evaluation\"\n",
|
|
" path_no_prefix = full_path.removeprefix(\"gs://\")\n",
|
|
" bucket_name = path_no_prefix.split(\"/\")[0]\n",
|
|
"\n",
|
|
" storage_client = storage.Client(project=project_id)\n",
|
|
" if storage_client.lookup_bucket(bucket_name) is None:\n",
|
|
" print(f\"Creating bucket: {bucket_name}\")\n",
|
|
" storage_client.create_bucket(bucket_name)\n",
|
|
"\n",
|
|
" return f\"gs://{path_no_prefix}\"\n",
|
|
"\n",
|
|
"\n",
|
|
"# Configuration\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"LOCATION = \"\" # @param {type: \"string\", placeholder: \"us-central1\", isTemplate: true}\n",
|
|
"GCS_DEST = \"\" # @param {type: \"string\", placeholder: \"[your-gcs-bucket]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"GCS_DEST = get_or_create_gcs_bucket(PROJECT_ID, GCS_DEST)\n",
|
|
"AGENT = \"\" # @param {type: \"string\", placeholder: \"[your-agent]\", isTemplate: true}\n",
|
|
"\n",
|
|
"# Initialize SDK\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
|
|
"\n",
|
|
"client = Client(\n",
|
|
" project=PROJECT_ID,\n",
|
|
" location=LOCATION,\n",
|
|
" http_options=genai_types.HttpOptions(api_version=\"v1beta1\"),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5303c05f7aa6"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6fc324893334"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"from vertexai import types"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "OG6a0FJxNyxJ"
|
|
},
|
|
"source": [
|
|
"# Step 1: Prepare Agent Dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "T8p5TKiqQlSx"
|
|
},
|
|
"source": [
|
|
"## Define Agent Info\n",
|
|
"You will need to manually update the below `AgentInfo` definition with the input data specific to your agent. \\\n",
|
|
"An example definition is provided below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kjAldqqhQkrR"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define agent functions\n",
|
|
"search_products = genai_types.FunctionDeclaration(\n",
|
|
" description=\"Searches for products based on a query.\",\n",
|
|
" name=\"search_products\",\n",
|
|
" parameters={\n",
|
|
" \"properties\": {\"query\": {\"type\": \"string\"}},\n",
|
|
" \"required\": [\"query\"],\n",
|
|
" \"type\": \"object\",\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"# Define agent info\n",
|
|
"agent_info = types.evals.AgentInfo(\n",
|
|
" agents={\n",
|
|
" \"ecommerce_agent\": types.evals.AgentConfig(\n",
|
|
" agent_id=\"ecommerce_agent\",\n",
|
|
" instruction=\"You are an ecommerce expert\",\n",
|
|
" tools=[genai_types.Tool(function_declarations=[search_products])],\n",
|
|
" )\n",
|
|
" },\n",
|
|
" root_agent_id=\"ecommerce_agent\",\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "zpGx1vx_GisH"
|
|
},
|
|
"source": [
|
|
"## Define Agent Dataset\n",
|
|
"Define a dataset that is specific to your agent. \\\n",
|
|
"`agent_prompts` should consist of prompts or requests to be made to your agent. A few example prompts are shown below. \\\n",
|
|
"`session_inputs` are required for traces. For more information see [Session](https://google.github.io/adk-docs/sessions/session/)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "o5_PPZEyGjMV"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"session_inputs = types.evals.SessionInput(\n",
|
|
" user_id=\"user_123\",\n",
|
|
" state={},\n",
|
|
")\n",
|
|
"\n",
|
|
"agent_prompts = [\n",
|
|
" \"I'm looking for some good noise-cancelling headphones. Can you find a pair for me, check the details, and if they have a long battery life, add them to my cart?\",\n",
|
|
" \"Find me a product with ID 'B08H8H8H8H', and add two of them to my cart.\",\n",
|
|
" \"I want to buy some wireless ear buds. Please find a good pair and add it to my cart.\",\n",
|
|
" \"Search for a laptop with 16GB of RAM, and then find a cheaper one with similar specs.\",\n",
|
|
" \"I need a new laptop for work, can you find one with at least 16GB of RAM?\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"agent_dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"prompt\": agent_prompts,\n",
|
|
" \"session_inputs\": [session_inputs] * len(agent_prompts),\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Hq-aQXPlRCRG"
|
|
},
|
|
"source": [
|
|
"# Step 2: Run Agent Inference\n",
|
|
"\n",
|
|
"Run inference using your deployed agent. This will add `intermediate_events` and `response` columns to your dataset to be evaluated in the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LWy9VOHeQvGh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"agent_dataset_with_inference = client.evals.run_inference(\n",
|
|
" agent=AGENT,\n",
|
|
" src=agent_dataset,\n",
|
|
")\n",
|
|
"# Display agent inference results\n",
|
|
"agent_dataset_with_inference.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Hw9Kd5pGQw9b"
|
|
},
|
|
"source": [
|
|
"# Step 3: Run Gen AI Agent Evaluation\n",
|
|
"\n",
|
|
"Run Gen AI Agent Evaluation using the Evaluation Management Service. \\\n",
|
|
"This will persist your dataset and evaluation results which can be retrieved via the Agent Engine UI."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "BfjPgdSEG5ZI"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"evaluation_run = client.evals.create_evaluation_run(\n",
|
|
" dataset=agent_dataset_with_inference,\n",
|
|
" agent=AGENT,\n",
|
|
" agent_info=agent_info,\n",
|
|
" metrics=[\n",
|
|
" types.RubricMetric.FINAL_RESPONSE_QUALITY,\n",
|
|
" types.RubricMetric.TOOL_USE_QUALITY,\n",
|
|
" types.RubricMetric.HALLUCINATION,\n",
|
|
" types.RubricMetric.SAFETY,\n",
|
|
" ],\n",
|
|
" dest=GCS_DEST,\n",
|
|
")\n",
|
|
"# Display the Evaluation Run status and results\n",
|
|
"evaluation_run.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "QdwLb05_HD35"
|
|
},
|
|
"source": [
|
|
"### Poll Evaluation Run for Completion and Display Results\n",
|
|
"Retrieve the Evaluation Run and directly display the results using the .show() command. If the Evaluation Run failed the error message will be displayed. Otherwise the following results data will be displayed in an embedded report.\n",
|
|
"\n",
|
|
"- **Summary metrics:** An aggregated view of all metrics, showing the mean score and standard deviation across the entire dataset.\n",
|
|
"- **Agent info:** Information describing the evaluated agent, including developer instruction, agent description, tool definitions, etc. Applied for agent evaluation only.\n",
|
|
"- **Detailed results:** A case-by-case breakdown, allowing you to inspect the prompt, reference, candidate response, and the specific score and explanation for each metric. For agent evaluation, detailed results will also include traces showing the agent interactions.\n",
|
|
"\n",
|
|
"For more information, see [Visualizing Evaluation Reports](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/view-evaluation#visualizing-evaluation-reports)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "_dUt_MRJHEWB"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"while evaluation_run.state not in {\"SUCCEEDED\", \"FAILED\", \"CANCELLED\"}:\n",
|
|
" evaluation_run.show()\n",
|
|
" evaluation_run = client.evals.get_evaluation_run(name=evaluation_run.name)\n",
|
|
" time.sleep(5)\n",
|
|
"\n",
|
|
"\n",
|
|
"evaluation_run = client.evals.get_evaluation_run(\n",
|
|
" name=evaluation_run.name, include_evaluation_items=True\n",
|
|
")\n",
|
|
"\n",
|
|
"# Display the Evaluation Run status and results\n",
|
|
"evaluation_run.show()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "create_genai_agent_evaluation.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|