wehub-resource-sync
771 lines
35 KiB
Plaintext
771 lines
35 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ur8xi4C7S06n"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Copyright 2024 Google LLC\n",
|
||
"#\n",
|
||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||
"# you may not use this file except in compliance with the License.\n",
|
||
"# You may obtain a copy of the License at\n",
|
||
"#\n",
|
||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||
"#\n",
|
||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||
"# See the License for the specific language governing permissions and\n",
|
||
"# limitations under the License."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "JAPoU8Sm5E6e"
|
||
},
|
||
"source": [
|
||
"# Evaluating multimodal task"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "b67322628c5f"
|
||
},
|
||
"source": [
|
||
" <table align=\"left\">\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaltask_approach%2Fevaluate_multimodal_task_image.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\">\n",
|
||
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
" <td style=\"text-align: center\">\n",
|
||
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\">\n",
|
||
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||
" </a>\n",
|
||
" </td>\n",
|
||
"</table>\n",
|
||
"\n",
|
||
"<div style=\"clear: both;\"></div>\n",
|
||
"\n",
|
||
"<b>Share to:</b>\n",
|
||
"\n",
|
||
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
||
"</a>\n",
|
||
"\n",
|
||
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_multimodal_task_image.ipynb\" target=\"_blank\">\n",
|
||
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
||
"</a>\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "84f0f73a0f76"
|
||
},
|
||
"source": [
|
||
"| | |\n",
|
||
"|-|-|\n",
|
||
"| Author(s) |[Ivan Nardini](https://github.com/inardini) |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "tvgnzT1CKxrO"
|
||
},
|
||
"source": [
|
||
"## Overview\n",
|
||
"\n",
|
||
"This notebook shows how to use Vertex AI Python SDK for Gen AI Evaluation Service for evaluating multimodal task with your locally-defined `CustomMetric`, and use your own autorater model to perform model-based metric evaluation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "7fRop-TJ5iZW"
|
||
},
|
||
"source": [
|
||
"<div class=\"alert alert-block alert-warning\">\n",
|
||
"<b>⚠️ This notebook shows an experiment approach to evaluate multimodal task using Vertex AI Python SDK for Gen AI Evaluation Service. The result of the evaluation depends on the autorater's capabilities of handling multimodal inputs with evaluation criteria. ⚠️</b>\n",
|
||
"</div>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "3XZf_4VEOvFo"
|
||
},
|
||
"source": [
|
||
"## Getting Started"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "kE20na1OOvFo"
|
||
},
|
||
"source": [
|
||
"### Install Vertex AI SDK for Gen AI Evaluation Service"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "abLuRgBzOvFp"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install -U -q google-cloud-aiplatform[evaluation] google-genai"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "R5Xep4W9lq-Z"
|
||
},
|
||
"source": [
|
||
"### Restart runtime\n",
|
||
"\n",
|
||
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
|
||
"\n",
|
||
"The restart might take a minute or longer. After it's restarted, continue to the next step."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "XRvKdaPDTznN"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import IPython\n",
|
||
"\n",
|
||
"app = IPython.Application.instance()\n",
|
||
"app.kernel.do_shutdown(True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "SbmM4z7FOBpM"
|
||
},
|
||
"source": [
|
||
"<div class=\"alert alert-block alert-warning\">\n",
|
||
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
||
"</div>\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "dmWOrTJ3gx13"
|
||
},
|
||
"source": [
|
||
"### Authenticate your notebook environment (Colab only)\n",
|
||
"\n",
|
||
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "NyKGtVQjgx13"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import sys\n",
|
||
"\n",
|
||
"if \"google.colab\" in sys.modules:\n",
|
||
" from google.colab import auth\n",
|
||
"\n",
|
||
" auth.authenticate_user()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "DF4l8DTdWgPY"
|
||
},
|
||
"source": [
|
||
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
|
||
"\n",
|
||
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||
"\n",
|
||
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "GTL_YzF9OvFq"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
||
"import os\n",
|
||
"\n",
|
||
"import vertexai\n",
|
||
"\n",
|
||
"# fmt: off\n",
|
||
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
||
"# fmt: on\n",
|
||
"\n",
|
||
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
||
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
||
"\n",
|
||
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
|
||
"\n",
|
||
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "EdvJRUWRNGHE"
|
||
},
|
||
"source": [
|
||
"## Evaluate your multimodal use case with custom metric\n",
|
||
"\n",
|
||
"Imagine you have a customer insurance app which provides several services including live incident support using an LLM. Given a conversation about a car accident and an image of the damaged car, you want to evaluate the coherence of the generated responses.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "5303c05f7aa6"
|
||
},
|
||
"source": [
|
||
"### Import libraries"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "6fc324893334"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# General"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "tfQ7sPtOjZOw"
|
||
},
|
||
"source": [
|
||
"### Library settings"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"cellView": "form",
|
||
"id": "RjWUgU1TjZOw"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# @title\n",
|
||
"import logging\n",
|
||
"import warnings\n",
|
||
"\n",
|
||
"import pandas as pd\n",
|
||
"from IPython.display import HTML, Markdown, display\n",
|
||
"from google import genai\n",
|
||
"from google.genai import types\n",
|
||
"from vertexai.evaluation import CustomMetric, EvalTask\n",
|
||
"\n",
|
||
"logging.getLogger(\"urllib3.connectionpool\").setLevel(logging.ERROR)\n",
|
||
"warnings.filterwarnings(\"ignore\")\n",
|
||
"\n",
|
||
"# pd.set_option('display.max_colwidth', None)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "QfaFPmm7gC2V"
|
||
},
|
||
"source": [
|
||
"### Helpers"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"cellView": "form",
|
||
"id": "2kyy7mGhgEj9"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def display_eval_result(\n",
|
||
" eval_result: dict | object,\n",
|
||
" title: str | None = None,\n",
|
||
" metrics: list[str] | None = None,\n",
|
||
") -> None:\n",
|
||
" \"\"\"Display the evaluation results.\"\"\"\n",
|
||
" summary_metrics, metrics_table = (\n",
|
||
" eval_result.summary_metrics,\n",
|
||
" eval_result.metrics_table,\n",
|
||
" )\n",
|
||
"\n",
|
||
" metrics_df = pd.DataFrame.from_dict(summary_metrics, orient=\"index\").T\n",
|
||
" if metrics:\n",
|
||
" metrics_df = metrics_df.filter(\n",
|
||
" [\n",
|
||
" metric\n",
|
||
" for metric in metrics_df.columns\n",
|
||
" if any(selected_metric in metric for selected_metric in metrics)\n",
|
||
" ]\n",
|
||
" )\n",
|
||
" metrics_table = metrics_table.filter(\n",
|
||
" [\n",
|
||
" metric\n",
|
||
" for metric in metrics_table.columns\n",
|
||
" if any(selected_metric in metric for selected_metric in metrics)\n",
|
||
" ]\n",
|
||
" )\n",
|
||
"\n",
|
||
" if title:\n",
|
||
" # Display the title with Markdown for emphasis\n",
|
||
" display(Markdown(f\"## {title}\"))\n",
|
||
" # Display the summary metrics DataFrame\n",
|
||
" display(Markdown(\"### Summary Metrics\"))\n",
|
||
" display(metrics_df)\n",
|
||
" # Display the metrics table DataFrame\n",
|
||
" display(Markdown(\"### Row-based Metrics\"))\n",
|
||
" display(metrics_table)\n",
|
||
"\n",
|
||
"\n",
|
||
"def display_explanations(\n",
|
||
" eval_result: dict | object, metrics: list[str] | None = None, n: int = 1\n",
|
||
") -> None:\n",
|
||
" \"\"\"Display the explanations.\"\"\"\n",
|
||
" style = \"white-space: pre-wrap; width: 1500px; overflow-x: auto;\"\n",
|
||
" metrics_table = eval_result.metrics_table\n",
|
||
" df = metrics_table.sample(n=n)\n",
|
||
"\n",
|
||
" if metrics:\n",
|
||
" df = df.filter(\n",
|
||
" [\"response\", \"baseline_model_response\"]\n",
|
||
" + [\n",
|
||
" metric\n",
|
||
" for metric in df.columns\n",
|
||
" if any(selected_metric in metric for selected_metric in metrics)\n",
|
||
" ]\n",
|
||
" )\n",
|
||
" for index, row in df.iterrows():\n",
|
||
" for col in df.columns:\n",
|
||
" display(HTML(f\"<div style='{style}'><h4>{col}:</h4>{row[col]}</div>\"))\n",
|
||
" display(HTML(\"<hr>\"))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "17XhFaeuQu31"
|
||
},
|
||
"source": [
|
||
"### Load your evaluation dataset\n",
|
||
"\n",
|
||
"Depending on your task and metrics you are planning to calculate, you need to collect your evaluation dataset.\n",
|
||
"\n",
|
||
"In this scenario, you have a `context` column which contains both the conversation and the location of the image for a specific car accident scenario.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "_o2YjZvbQ1dH"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"instruction = \"\"\"\n",
|
||
"You are an insurance agent specializing in car accident assessments.\n",
|
||
"You will be provided with a conversation about a car accident and an image of the damaged car.\n",
|
||
"Your task is to analyze the image and identify the primary type of damage visible.\n",
|
||
"Use the conversation for context, but prioritize the visual evidence from the image.\n",
|
||
"\n",
|
||
"Categorize the primary damage as one of the following:\n",
|
||
"\n",
|
||
"* bumper\n",
|
||
"* engine_compartment\n",
|
||
"* hood\n",
|
||
"* lateral\n",
|
||
"* windshield\n",
|
||
"\n",
|
||
"If the image is unclear or the damage is not visible, respond with \"Unable to determine damage type from the provided image.\"\n",
|
||
"If the primary damage is something other than the listed categories, respond with \"Damage type not listed in available categories.\"\n",
|
||
"\n",
|
||
"Conversation:\n",
|
||
"{conversation}\n",
|
||
"\n",
|
||
"Image:\n",
|
||
"{image_of_car_accident}\n",
|
||
"\n",
|
||
"Provide your assessment of the primary damage type based on the image.\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"context = [\n",
|
||
" {\n",
|
||
" \"conversation\": '''AI insurance app: \"Hello, I'm the AI assistant for your car insurance. It looks like you've been in an accident. Could you please tell me what happened?\" App user: \"Yeah, I was just rear-ended while waiting at a red light.\" AI insurance app: \"I'm sorry to hear that. Could you take some pictures of the damage to your vehicle, including the license plate of the other car if possible?\" App user: (uploads the provided image) \"Here's the damage to my bumper.\" AI insurance app: \"Thank you. Can you describe any injuries to yourself or any passengers?\" App user: \"No, thankfully everyone is okay. Just a bit shaken up.\" AI insurance app: \"That's good to hear. I've created a claim based on the information and photo you provided. A representative will be in touch with you shortly to gather further details and discuss the next steps.\" App user: \"Okay, thank you.\" AI insurance app: \"You're welcome. Please don't hesitate to contact us if you have any questions.\"''',\n",
|
||
" \"image_of_car_accident\": \"gs://cloud-samples-data/generative-ai/evaluation/use_cases/car_assessment/bumper.jpg\",\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"conversation\": '''AI insurance app: \"Hi there! I'm Amelia, your AI assistant for [Insurance company name]. I see you've been in an accident. I'm so sorry to hear that. Are you okay?\" Driver: \"I'm a little shaken up, but I'm okay. My car isn't so lucky, though.\" AI insurance app: \"Oh no, I'm so sorry to hear that. Can you tell me what happened?\" Driver: \"I was stopped at a red light when I was rear-ended by another car. The damage to my car is pretty bad.\" AI insurance app: \"I understand. Can you take some pictures of the damage to your car, including the license plate of the other car if possible?\" Driver: \"Sure, here you go.\" (uploads the provided image) AI insurance app: \"Thank you. Is there anyone else involved in the accident?\" Driver: \"No, just me and the driver of the other car.\" AI insurance app: \"Okay. Do you need medical attention?\" Driver: \"No, I'm fine.\" AI insurance app: \"I'm glad to hear that. I've created a claim based on the information and photo you provided. A representative will be in touch with you shortly to gather further details and discuss the next steps.\" Driver: \"Okay, thank you.\" AI insurance app: \"You're welcome. Please don't hesitate to contact us if you have any questions.\"''',\n",
|
||
" \"image_of_car_accident\": \"gs://cloud-samples-data/generative-ai/evaluation/use_cases/car_assessment/engine_compartment.jpg\",\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"conversation\": '''**AI insurance app:** \"Hello, it appears you've been in an accident. Are you alright?\" **App user:** \"Yes, I'm okay. Just a bit shaken up.\" **AI insurance app:** \"I'm glad to hear you're physically unharmed. Could you please describe what happened?\" **App user:** \"Someone ran a red light and hit the front of my car.\" **AI insurance app:** \"I understand. To help assess the damage, could you please take some photos of your vehicle, especially the impacted areas? If possible, include a photo of the other vehicle's license plate.\" **App user:** (uploads the provided image) \"Here's the damage to my car.\" **AI insurance app:** \"Thank you for providing that. Were there any other vehicles involved, or was it just the two cars?\" **App user:** \"No, it was just us.\" **AI insurance app:** \"Okay. And to confirm, you don't require any medical assistance at this time?\" **App user:** \"No, I don't think so. Thankfully.\" **AI insurance app:** \"Alright. I've created an accident claim with the information and photos you've provided. One of our representatives will contact you soon to gather more details and guide you through the next steps.\" **App user:** \"Thank you, I appreciate the help.\" **AI insurance app:** \"You're very welcome. Please don't hesitate to reach out through the app if you have any further questions.\"''',\n",
|
||
" \"image_of_car_accident\": \"gs://cloud-samples-data/generative-ai/evaluation/use_cases/car_assessment/hood.jpg\",\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"conversation\": '''AI insurance app: \"Hi there! I'm Amelia, your AI assistant for [Insurance company name]. I see you've been in an accident. I'm so sorry to hear that. Are you okay?\" Driver: \"I'm a little shaken up, but I'm okay. My car isn't so lucky, though.\" AI insurance app: \"Oh no, I'm so sorry to hear that. Can you tell me what happened?\" Driver: \"I was stopped at a red light when I was rear-ended by another car. The damage to my car is pretty bad.\" AI insurance app: \"I understand. Can you take some pictures of the damage to your car, including the license plate of the other car if possible?\" Driver: \"Sure, here you go.\" (uploads the provided image) AI insurance app: \"Thank you. Is there anyone else involved in the accident?\" Driver: \"No, just me and the driver of the other car.\" AI insurance app: \"Okay. Do you need medical attention?\" Driver: \"No, I'm fine.\" AI insurance app: \"I'm glad to hear that. I've created a claim based on the information and photo you provided. A representative will be in touch with you shortly to gather further details and discuss the next steps.\" Driver: \"Okay, thank you.\" AI insurance app: \"You're welcome. Please don't hesitate to contact us if you have any questions.\"\"''',\n",
|
||
" \"image_of_car_accident\": \"gs://cloud-samples-data/generative-ai/evaluation/use_cases/car_assessment/lateral.jpg\",\n",
|
||
" },\n",
|
||
" {\n",
|
||
" \"conversation\": '''AI insurance app: \"Hello. I've received an alert that you may have been involved in an accident. Can you confirm and tell me if you're okay?\" App user: \"Yes, I was just in an accident. I'm okay, just a little shaken.\" AI insurance app: \"I'm relieved to hear you're not hurt. Can you tell me what happened?\" App user: \"A rock flew up from a truck in front of me and cracked my windshield.\" AI insurance app: \"I understand. To assess the damage, could you please take a photo of the damage?\" App user: (uploads the provided image) \"Here's a photo of the crack.\" AI insurance app: \"Thank you for providing that. Were there any other vehicles involved?\" App user: \"No, just my car.\" AI insurance app: \"Okay. And you didn't sustain any injuries?\" App user: \"No, thankfully not.\" AI insurance app: \"That's good to hear. I've created a claim for you based on the information and photo you provided. A representative will be in touch shortly to gather more details and guide you through the next steps.\" App user: \"Okay, thank you.\" AI insurance app: \"You're welcome. Please don't hesitate to contact us if you have any questions.\"''',\n",
|
||
" \"image_of_car_accident\": \"gs://cloud-samples-data/generative-ai/evaluation/use_cases/car_assessment/windshield.jpg\",\n",
|
||
" },\n",
|
||
"]\n",
|
||
"\n",
|
||
"generated_response = [\n",
|
||
" \"bumper\",\n",
|
||
" \"engine_compartment\",\n",
|
||
" \"lateral\",\n",
|
||
" \"lateral\",\n",
|
||
" \"windshield\",\n",
|
||
"]\n",
|
||
"\n",
|
||
"reference = [\"bumper\", \"engine_compartment\", \"hood\", \"lateral\", \"windshield\"]\n",
|
||
"\n",
|
||
"eval_dataset = pd.DataFrame(\n",
|
||
" {\n",
|
||
" \"instruction\": instruction,\n",
|
||
" \"context\": context,\n",
|
||
" \"response\": generated_response,\n",
|
||
" \"reference\": reference,\n",
|
||
" }\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "icfOEQopmpky"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"eval_dataset.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "EFUFtKRIUpAe"
|
||
},
|
||
"source": [
|
||
"### Bring your own autorater\n",
|
||
"\n",
|
||
"Gen AI Evaluation Service allows you to bring any autorater you prefer. In this case, you use `gemini-2.5-flash` with a typical structured output to evaluate your task."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "oV6ZI1E5Uu1c"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def get_autorater_response(metric_prompt: list) -> dict:\n",
|
||
" client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\n",
|
||
"\n",
|
||
" response = client.models.generate_content(\n",
|
||
" model=\"gemini-2.5-flash\",\n",
|
||
" contents=metric_prompt,\n",
|
||
" config=types.GenerateContentConfig(\n",
|
||
" response_mime_type=\"application/json\",\n",
|
||
" response_schema={\n",
|
||
" \"type\": \"OBJECT\",\n",
|
||
" \"properties\": {\n",
|
||
" \"score\": {\"type\": \"NUMBER\"},\n",
|
||
" \"explanation\": {\"type\": \"STRING\"},\n",
|
||
" },\n",
|
||
" \"required\": [\"score\", \"explanation\"],\n",
|
||
" },\n",
|
||
" safety_settings=[\n",
|
||
" types.SafetySetting(\n",
|
||
" category=\"HARM_CATEGORY_HATE_SPEECH\",\n",
|
||
" threshold=\"BLOCK_NONE\",\n",
|
||
" ),\n",
|
||
" types.SafetySetting(\n",
|
||
" category=\"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
|
||
" threshold=\"BLOCK_NONE\",\n",
|
||
" ),\n",
|
||
" types.SafetySetting(\n",
|
||
" category=\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
|
||
" threshold=\"BLOCK_NONE\",\n",
|
||
" ),\n",
|
||
" types.SafetySetting(\n",
|
||
" category=\"HARM_CATEGORY_HARASSMENT\",\n",
|
||
" threshold=\"BLOCK_NONE\",\n",
|
||
" ),\n",
|
||
" ],\n",
|
||
" ),\n",
|
||
" )\n",
|
||
"\n",
|
||
" return response.parsed"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "88_KslQFVKme"
|
||
},
|
||
"source": [
|
||
"### Build your custom metric\n",
|
||
"\n",
|
||
"To evaluate a multimodal task, you define a custom metric which takes any multimodal content (image, pdf, video and more) and returns a score and an explanation according to some criteria, rating rubric, and evaluation steps using the defined autorater.\n",
|
||
"\n",
|
||
"In this scenario, you provide define a coherence metric in this car accident assessment scenario."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ngpoc0KjVNF0"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def custom_coherence_fn(instance):\n",
|
||
"\n",
|
||
" conversation = instance[\"context\"][\"conversation\"]\n",
|
||
" image_of_car_accident = instance[\"context\"][\"image_of_car_accident\"]\n",
|
||
" response = instance[\"response\"]\n",
|
||
"\n",
|
||
" eval_instruction_template = \"\"\"\n",
|
||
"\n",
|
||
" # Instruction\n",
|
||
" You are an insurance agent specializing in evaluating car accident assessments.\n",
|
||
" You will be provided with a conversation about a car accident and an image of the damaged car.\n",
|
||
" You should first read the conversation and look at the image carefully, and then evaluate the coherence of the generated responses based on the Criteria provided in the Evaluation section below.\n",
|
||
" You will assign the response a rating following the Rating Rubric and Evaluation Steps. Give step-by-step explanations for your rating, and only choose ratings from the Rating Rubric.\n",
|
||
"\n",
|
||
" # Evaluation\n",
|
||
" ## Metric Definition\n",
|
||
" You will be assessing coherence, which measures the ability to provide a coherent response based on the conversation and car accident image.\n",
|
||
"\n",
|
||
" ## Criteria\n",
|
||
" Coherence: It is the quality of being logical and consistent.\n",
|
||
" In the context of conversation, it refers to the way that ideas and information are presented in a way that is easy to understand and follow.\n",
|
||
" A coherent conversation will have a clear flow and will not jump around from topic to topic.\n",
|
||
" The user will also use language that is appropriate for the audience and will avoid making claims that are not supported by evidence.\n",
|
||
"\n",
|
||
" ## Rating Rubric\n",
|
||
" 5: (Perfectly Aligned) The image precisely matches the damage described in the conversation, and the response accurately reflects the damaged car part.\n",
|
||
" 4: (Highly Aligned) The image generally supports the conversation's description of the damage, and the response is a suitable representation of the affected area.\n",
|
||
" 3: (Moderately Aligned) The image shows damage that is plausibly related to the accident described, but there might be minor inconsistencies, and the response is broadly relevant but not entirely specific.\n",
|
||
" 2: (Poorly Aligned) The image and/or the response have significant inconsistencies with the described accident in the conversation, raising doubts about the claim's validity.\n",
|
||
" 1: (Misaligned) The image, response, and conversation have major contradictions or are completely unrelated, making the claim appear illogical or fraudulent.\n",
|
||
"\n",
|
||
" ## Evaluation Steps\n",
|
||
" STEP 1: Assess Claim Consistency: Carefully read the conversation to understand the user's description of the accident and the claimed damage.\n",
|
||
" STEP 2: Analyze Image Relevance: Examine the image to determine if the depicted damage aligns with the user's account. Pay attention to the location and type of damage.\n",
|
||
" STEP 3: Evaluate Label Accuracy: Check if the generated label correctly identifies the damaged car part as described in the conversation and shown in the image.\n",
|
||
" STEP 4: Identify Inconsistencies: Look for any discrepancies between the conversation, image, and label. For example, does the image show damage not mentioned in the conversation, or is the label incorrect for the damaged part?\n",
|
||
" STEP 5: Determine Overall Coherence: Based on the previous steps, assign a coherence score using the 1-5 rubric. Consider the severity of any inconsistencies and their potential impact on the claim's validity.\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" # read image from uri\n",
|
||
" image_file = types.Part.from_uri(\n",
|
||
" file_uri=image_of_car_accident, mime_type=\"image/jpeg\"\n",
|
||
" )\n",
|
||
"\n",
|
||
" # generate the eval\n",
|
||
" evaluation_prompt = [\n",
|
||
" eval_instruction_template,\n",
|
||
" \"CONVERSATION: \",\n",
|
||
" conversation,\n",
|
||
" \"IMAGE: \",\n",
|
||
" image_file,\n",
|
||
" \"GENERATED RESPONSE: \",\n",
|
||
" response,\n",
|
||
" ]\n",
|
||
"\n",
|
||
" evaluation_response = get_autorater_response(evaluation_prompt)\n",
|
||
" return {\n",
|
||
" \"custom_coherence\": evaluation_response.get(\"score\", \"\"),\n",
|
||
" \"explanation\": evaluation_response.get(\"explanation\", \"\"),\n",
|
||
" }"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "_7Vhz16neast"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"custom_coherence_metric = CustomMetric(\n",
|
||
" name=\"custom_coherence\",\n",
|
||
" metric_function=custom_coherence_fn,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "A9-m7tmlb9mt"
|
||
},
|
||
"source": [
|
||
"### Run Evaluation\n",
|
||
"\n",
|
||
"With the autorater, and the custom metric, you run your evaluation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "nkiqjPHhcWo6"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"metrics = [\"exact_match\", custom_coherence_metric]\n",
|
||
"\n",
|
||
"experiment_name = \"eval-multimodal-metric\"\n",
|
||
"\n",
|
||
"eval_task = EvalTask(\n",
|
||
" dataset=eval_dataset,\n",
|
||
" metrics=metrics,\n",
|
||
" experiment=experiment_name,\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "Sp-TlIORdyFb"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"eval_result = eval_task.evaluate()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "vKFOB6y_yh3w"
|
||
},
|
||
"source": [
|
||
"### Visualize and validate your evaluation results.\n",
|
||
"\n",
|
||
"Using some helpers, you can both evaluation results (scores, and explanations)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "oHEblJYvgSpU"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"display_eval_result(eval_result, title=\"Evaluation Results\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "kdTt7ETMgfhR"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"display_explanations(eval_result, metrics=[\"custom_coherence\"])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "2a4e033321ad"
|
||
},
|
||
"source": [
|
||
"## Cleaning up"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "35VcF-W5peq5"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"delete_experiment = True\n",
|
||
"\n",
|
||
"if delete_experiment:\n",
|
||
" from google.cloud import aiplatform\n",
|
||
"\n",
|
||
" aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
|
||
" experiment = aiplatform.Experiment(experiment_name)\n",
|
||
" experiment.delete()"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"colab": {
|
||
"name": "evaluate_multimodal_task_image.ipynb",
|
||
"toc_visible": true
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"name": "python3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|