wehub-resource-sync
634 lines
22 KiB
Plaintext
634 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Create & Deploy Agent and Run Gen AI Agent Evaluation\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fcreate_agent_and_run_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_agent_and_run_evaluation.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| Author(s) |\n",
|
|
"| --- |\n",
|
|
"| [Kelsi Lakey](https://github.com/lakeyk) |\n",
|
|
"| [Bo Zheng](https://github.com/coolalexzb) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This Colab notebook demonstrates how to create and deploy an Agent and then use the Gen AI Eval SDK to evaluate it.\n",
|
|
"\n",
|
|
"- **Define an Agent:** Define a 'Hello World' Agent Development Kit agent with a few basic tool functions.\n",
|
|
"- **Deploy an Agent to Agent Engine:** Deploy the agent to Agent Engine.\n",
|
|
"- **Run Agent Inference:** Define an Evaluation Dataset and run agent inference to retrieve real responses.\n",
|
|
"- **Create Evaluation Run:** Create an Evaluation Run to perform Gen AI Agent Evaluation. This Evaluation Run will be persisted and accessible later.\n",
|
|
"\n",
|
|
"If you already have a deployed Agent, please see:\n",
|
|
"- [Create a Gen AI Agent Evaluation for a Deployed Agent](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/create_genai_agent_evaluation.ipynb)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK and other required packages\n",
|
|
"Restart runtime after installation to load latest packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install -q google-cloud-aiplatform[adk,agent_engines]\n",
|
|
"%pip install --upgrade --force-reinstall -q google-cloud-aiplatform[evaluation]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dmWOrTJ3gx13"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook in **Google Colab**, run the cell below to authenticate your account."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"cellView": "form",
|
|
"id": "WX-2IIgIgIlB"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"from urllib.parse import urlparse\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"from vertexai import Client\n",
|
|
"from google.cloud import storage\n",
|
|
"from google.genai import types as genai_types\n",
|
|
"\n",
|
|
"def get_config_value(initial_value: str, placeholder: str, env_var_name: str) -> str:\n",
|
|
" \"\"\"Gets a configuration value or environment variable if unspecified.\"\"\"\n",
|
|
" if not initial_value or initial_value == placeholder:\n",
|
|
" return os.environ.get(env_var_name)\n",
|
|
" return initial_value\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_or_create_gcs_bucket(project_id: str, gcs_dest: str) -> str:\n",
|
|
" \"\"\"Retrieves GCS bucket or creates a default.\"\"\"\n",
|
|
" gcs_dest = gcs_dest or f\"{project_id}/agent-evaluation\"\n",
|
|
" storage_client = storage.Client(project=project_id)\n",
|
|
" bucket = gcs_dest.replace(\"gs://\", \"\").split(\"/\")[0]\n",
|
|
" if not storage_client.lookup_bucket(bucket):\n",
|
|
" print(f\"Creating bucket: {bucket}\")\n",
|
|
" storage_client.create_bucket(bucket)\n",
|
|
" if not gcs_dest.startswith(\"gs://\"):\n",
|
|
" return f\"gs://{gcs_dest}\"\n",
|
|
" return gcs_dest\n",
|
|
"\n",
|
|
"\n",
|
|
"# Configuration\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"PROJECT_ID = get_config_value(PROJECT_ID, \"[your-project-id]\", \"GOOGLE_CLOUD_PROJECT\")\n",
|
|
"assert PROJECT_ID, \"Please specify a valid project\"\n",
|
|
"\n",
|
|
"LOCATION = \"\" # @param {type: \"string\", placeholder: \"[us-central1]\", isTemplate: true}\n",
|
|
"LOCATION = get_config_value(LOCATION, None, \"GOOGLE_CLOUD_REGION\")\n",
|
|
"assert LOCATION, \"Please specify a valid location\"\n",
|
|
"\n",
|
|
"GCS_DEST = \"\" # @param {type: \"string\", placeholder: \"[your-gcs-bucket]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"GCS_DEST = get_config_value(\n",
|
|
" GCS_DEST, \"[your-gcs-bucket]\", \"GOOGLE_CLOUD_STORAGE_BUCKET\"\n",
|
|
")\n",
|
|
"GCS_DEST = get_or_create_gcs_bucket(PROJECT_ID, GCS_DEST)\n",
|
|
"assert GCS_DEST, \"Please specify a valid GCS destination\"\n",
|
|
"STAGING_BUCKET = f\"gs://{urlparse(GCS_DEST).netloc}\"\n",
|
|
"\n",
|
|
"# Initialize SDK\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
|
|
"client = Client(\n",
|
|
" project=PROJECT_ID,\n",
|
|
" location=LOCATION,\n",
|
|
" http_options=genai_types.HttpOptions(api_version=\"v1beta1\"),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "F9-QJHBwUc8X"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AwLg3h-X71yf"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"from google.adk import Agent\n",
|
|
"from vertexai import types"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "j7nn1NUklwQD"
|
|
},
|
|
"source": [
|
|
"# Step 1: Create and Deploy an Agent"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TRmadZczmisn"
|
|
},
|
|
"source": [
|
|
"## Define Agent\n",
|
|
"Develop an Agent Development Kit agent by defining the model, instruction, and set of tools. \\\n",
|
|
"For more information see [Develop an Agent Development Kit agent](https://cloud.google.com/agent-builder/agent-engine/develop/adk)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Tu3v_or3kdwS"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define Agent Tools\n",
|
|
"def search_products(query: str):\n",
|
|
" \"\"\"Searches for products based on a query.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" query: The search query.\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" A list of products that match the query.\n",
|
|
" \"\"\"\n",
|
|
" # Mock response for demonstration\n",
|
|
" if \"headphones\" in query.lower():\n",
|
|
" return {\"products\": [{\"name\": \"Wireless Headphones\", \"id\": \"B08H8H8H8H\"}]}\n",
|
|
" return {\"products\": []}\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_product_details(product_id: str):\n",
|
|
" \"\"\"Gets the details for a given product ID.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" product_id: The ID of the product.\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" The details of the product.\n",
|
|
" \"\"\"\n",
|
|
" if product_id == \"B08H8H8H8H\":\n",
|
|
" return {\"details\": \"Noise-cancelling, 20-hour battery life.\"}\n",
|
|
" return {\"error\": \"Product not found.\"}\n",
|
|
"\n",
|
|
"\n",
|
|
"def add_to_cart(product_id: str, quantity: int):\n",
|
|
" \"\"\"Adds a specified quantity of a product to the cart.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" product_id: The ID of the product.\n",
|
|
" quantity: The quantity to add to the cart.\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" A status message indicating the addition to the cart.\n",
|
|
" \"\"\"\n",
|
|
" return {\"status\": f\"Added {quantity} of {product_id} to cart.\"}\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define Agent\n",
|
|
"ecommerce_agent = Agent(\n",
|
|
" model=\"gemini-2.5-flash\",\n",
|
|
" name=\"ecommerce_agent\",\n",
|
|
" instruction=\"You are an ecommerce expert\",\n",
|
|
" tools=[search_products, get_product_details, add_to_cart],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "X0wB9M_Wl1cU"
|
|
},
|
|
"source": [
|
|
"## Deploy Agent to Agent Engine\n",
|
|
"Create an Agent Development Kit agent and deploy to Agent Engine. \\\n",
|
|
"For more information on deploying an agent, see [Deploy an Agent](https://cloud.google.com/agent-builder/agent-engine/deploy). \\\n",
|
|
"This process may take up to 10 minutes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Bg5q_SAjkydq"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Deploy Agent\n",
|
|
"app = vertexai.agent_engines.AdkApp(\n",
|
|
" agent=ecommerce_agent,\n",
|
|
")\n",
|
|
"agent_engine = client.agent_engines.create(\n",
|
|
" agent=app,\n",
|
|
" config={\n",
|
|
" \"staging_bucket\": STAGING_BUCKET,\n",
|
|
" \"requirements\": [\"google-cloud-aiplatform[adk,agent_engines]\"],\n",
|
|
" \"env_vars\": {\"GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY\": \"true\"},\n",
|
|
" },\n",
|
|
")\n",
|
|
"agent_engine_resource_name = agent_engine.api_resource.name"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7WIiXNxNmD7u"
|
|
},
|
|
"source": [
|
|
"# Step 2: Run Agent Inference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7vUWiEFnlODO"
|
|
},
|
|
"source": [
|
|
"## Define Agent Dataset\n",
|
|
"Define a dataset that is specific to your agent. \\\n",
|
|
"`agent_prompts` should consist of prompts or requests to be made to your agent. A few example prompts are shown below. \\\n",
|
|
"`session_inputs` are required for traces. For more information see [Session](https://google.github.io/adk-docs/sessions/session/)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "0LRtzCRQkiqX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"session_inputs = types.evals.SessionInput(\n",
|
|
" user_id=\"user_123\",\n",
|
|
" state={},\n",
|
|
")\n",
|
|
"ecommerce_prompts = [\n",
|
|
" \"Search for 'noise-cancelling headphones'.\",\n",
|
|
" \"Show me the details for product 'B08H8H8H8H'.\",\n",
|
|
" \"Add one pair of 'B08H8H8H8H' to my shopping cart.\",\n",
|
|
" \"Find 'wireless ear buds' and then add the first result to my cart.\",\n",
|
|
" \"I need a new laptop for work, can you find one with at least 16GB of RAM?\",\n",
|
|
"]\n",
|
|
"ecommerce_dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"prompt\": ecommerce_prompts,\n",
|
|
" \"session_inputs\": [session_inputs] * len(ecommerce_prompts),\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TDLf9as_oIGy"
|
|
},
|
|
"source": [
|
|
"## Run Agent Inference\n",
|
|
"\n",
|
|
"Run inference using your deployed agent. This will add `intermediate_events` and `response` columns to your dataset to be evaluated in the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "timr1DkXn65s"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Run inference\n",
|
|
"agent_dataset_with_inference = client.evals.run_inference(\n",
|
|
" agent=agent_engine_resource_name,\n",
|
|
" src=ecommerce_dataset,\n",
|
|
")\n",
|
|
"# Display inference results\n",
|
|
"agent_dataset_with_inference.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No0mcgOPhHML"
|
|
},
|
|
"source": [
|
|
"# Step 3: Run Gen AI Agent Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "79RQWFWMk7_R"
|
|
},
|
|
"source": [
|
|
"## Option 1: Run Gen AI Evaluation with Evaluation Management Service\n",
|
|
"\n",
|
|
"Run Gen AI Agent Evaluation using the Evaluation Management Service. \\\n",
|
|
"This will persist your dataset and evaluation results which can be retrieved via the Agent Engine UI."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "N66n0rYspNbG"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create agent_info from Agent definition and deployed resource name\n",
|
|
"ecommerce_agent_info = types.evals.AgentInfo.load_from_agent(\n",
|
|
" ecommerce_agent, agent_engine_resource_name\n",
|
|
")\n",
|
|
"\n",
|
|
"# Evaluate Agent Dataset\n",
|
|
"evaluation_run = client.evals.create_evaluation_run(\n",
|
|
" dataset=agent_dataset_with_inference,\n",
|
|
" agent_info=ecommerce_agent_info,\n",
|
|
" metrics=[\n",
|
|
" types.RubricMetric.FINAL_RESPONSE_QUALITY,\n",
|
|
" types.RubricMetric.TOOL_USE_QUALITY,\n",
|
|
" types.RubricMetric.HALLUCINATION,\n",
|
|
" types.RubricMetric.SAFETY,\n",
|
|
" ],\n",
|
|
" dest=GCS_DEST,\n",
|
|
")\n",
|
|
"\n",
|
|
"# Display status and results\n",
|
|
"evaluation_run.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ZjBu72ciT32N"
|
|
},
|
|
"source": [
|
|
"### Poll Evaluation Run for Completion and Display Results\n",
|
|
"Retrieve the Evaluation Run and directly display the results using the .show() command. If the Evaluation Run failed the error message will be displayed. Otherwise the following results data will be displayed in an embedded report.\n",
|
|
"\n",
|
|
"- **Summary metrics:** An aggregated view of all metrics, showing the mean score and standard deviation across the entire dataset.\n",
|
|
"- **Agent info:** Information describing the evaluated agent, including developer instruction, agent description, tool definitions, etc. Applied for agent evaluation only.\n",
|
|
"- **Detailed results:** A case-by-case breakdown, allowing you to inspect the prompt, reference, candidate response, and the specific score and explanation for each metric. For agent evaluation, detailed results will also include traces showing the agent interactions.\n",
|
|
"\n",
|
|
"For more information, see [Visualizing Evaluation Reports](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/view-evaluation#visualizing-evaluation-reports)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "wlcjd6AlpeTF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"completed_states = set(\n",
|
|
" [\n",
|
|
" \"SUCCEEDED\",\n",
|
|
" \"FAILED\",\n",
|
|
" \"CANCELLED\",\n",
|
|
" ]\n",
|
|
")\n",
|
|
"\n",
|
|
"while evaluation_run.state not in completed_states:\n",
|
|
" evaluation_run.show()\n",
|
|
" evaluation_run = client.evals.get_evaluation_run(name=evaluation_run.name)\n",
|
|
" time.sleep(5)\n",
|
|
"evaluation_run = client.evals.get_evaluation_run(\n",
|
|
" name=evaluation_run.name, include_evaluation_items=True\n",
|
|
")\n",
|
|
"\n",
|
|
"evaluation_run.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BQ0LD9VDkzuZ"
|
|
},
|
|
"source": [
|
|
"## [Optional] Option 2: Run Gen AI Evaluation Locally\n",
|
|
"\n",
|
|
"Run Gen AI Agent Evaluation locally. \\\n",
|
|
"This will run the same evaluation as `Option 1` but results will not be available outside of this colab instance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "5q2HyJGCr4kU"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create agent_info from Agent definition and deployed resource name\n",
|
|
"ecommerce_agent_info = types.evals.AgentInfo.load_from_agent(\n",
|
|
" ecommerce_agent, agent_engine_resource_name\n",
|
|
")\n",
|
|
"\n",
|
|
"# Evaluate Agent Dataset\n",
|
|
"eval_result = client.evals.evaluate(\n",
|
|
" dataset=agent_dataset_with_inference,\n",
|
|
" agent_info=ecommerce_agent_info,\n",
|
|
" metrics=[\n",
|
|
" types.RubricMetric.FINAL_RESPONSE_QUALITY,\n",
|
|
" types.RubricMetric.TOOL_USE_QUALITY,\n",
|
|
" types.RubricMetric.HALLUCINATION,\n",
|
|
" types.RubricMetric.SAFETY,\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"# Display results\n",
|
|
"eval_result.show()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "create_agent_and_run_evaluation.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"environment": {
|
|
"kernel": "python3",
|
|
"name": "tf2-gpu.2-6.m108",
|
|
"type": "gcloud",
|
|
"uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m108"
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.16"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|