wehub-resource-sync
876 lines
33 KiB
Plaintext
876 lines
33 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Evaluate Gemini Structured Output\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_gemini_structured_output.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| Author |\n",
|
|
"| --- |\n",
|
|
"| [Steve Phillips](https://github.com/stevie-p) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook uses the [**Gen AI Evaluation Service**](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation) to evaluate and compare the performance of Gemini models for an extraction task.\n",
|
|
"\n",
|
|
"The task is to accurately extract information from a scanned, handwritten order form for \"Acme Corporation\".\n",
|
|
"\n",
|
|
"Within this notebook, we:\n",
|
|
"\n",
|
|
"* Use Gemini models with [structured output](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) to ensure well-structured JSON output\n",
|
|
"* Extract the data using Gemini models\n",
|
|
"* Define two custom metrics: `valid_schema` using the [`jsonschema`](https://pypi.org/project/jsonschema/) library, and `accuracy` using the [`deepdiff`](https://github.com/seperman/deepdiff) library\n",
|
|
"* Use the **Gen AI Evaluation service** to run the evaluation experiments\n",
|
|
"\n",
|
|
"The [models](https://cloud.google.com/vertex-ai/generative-ai/docs/models) under test are:\n",
|
|
"\n",
|
|
"* Gemini 2.0 Flash\n",
|
|
"* Gemini 2.5 Flash\n",
|
|
"* Gemini 2.5 Pro"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK and other required packages\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai jsonschema IPython==7.34.0 google-cloud-aiplatform 'pybind11>=2.12' deepdiff"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "795uBvHqqy4V"
|
|
},
|
|
"source": [
|
|
"Restart the runtime to use the newly installed packages."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dmWOrTJ3gx13"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create your own project and insert the project ID here ---->\n",
|
|
"\n",
|
|
"import os\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"LOCATION = \"us-central1\" # @param {type: \"string\"}\n",
|
|
"\n",
|
|
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"if not LOCATION:\n",
|
|
" LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
|
|
"\n",
|
|
"EXPERIMENT_NAME = \"eval-gemini-structured\"\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"from google import genai\n",
|
|
"\n",
|
|
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5303c05f7aa6"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6fc324893334"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"from IPython.display import Image, Markdown, display\n",
|
|
"from deepdiff import DeepDiff\n",
|
|
"from google.genai.types import (\n",
|
|
" GenerateContentConfig,\n",
|
|
" Part,\n",
|
|
")\n",
|
|
"from jsonschema import validate\n",
|
|
"from vertexai.evaluation import CustomMetric, EvalTask, notebook_utils"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "B9Q3T_1ox99_"
|
|
},
|
|
"source": [
|
|
"## View the images\n",
|
|
"\n",
|
|
"Let's have a look at the images we want to extract data from."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "q7u9dNhytp5s"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"images = [\n",
|
|
" {\n",
|
|
" \"image_url\": \"https://storage.googleapis.com/github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/AcmeOrderForm.jpg\",\n",
|
|
" \"image_uri\": \"gs://github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/AcmeOrderForm.jpg\",\n",
|
|
" \"image_type\": \"image/jpeg\",\n",
|
|
" \"image_name\": \"Acme Order Form.jpg\",\n",
|
|
" \"reference\": { # The Ground Truth\n",
|
|
" \"order_number\": \"98-X42-77A\",\n",
|
|
" \"order_date\": \"2025-09-01\",\n",
|
|
" \"customer_name\": \"WILE E. COYOTE (ESQ., PH.D, S.G.)\",\n",
|
|
" \"customer_address\": \"HIGH MESA, CORNER OF X-MARK AND DETONATION CANYON, ANVIL FALLS, AZ\",\n",
|
|
" \"line_items\": [\n",
|
|
" {\n",
|
|
" \"item_description\": \"Jet Propelled Unicycle\",\n",
|
|
" \"quantity\": 1,\n",
|
|
" \"unit_price\": 99.99,\n",
|
|
" \"delivery_option\": \"Next Day\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Instant Hole Kit\",\n",
|
|
" \"quantity\": 3,\n",
|
|
" \"unit_price\": 45.00,\n",
|
|
" \"delivery_option\": \"Standard\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"TNT High Explosives x24\",\n",
|
|
" \"quantity\": 1,\n",
|
|
" \"unit_price\": 120.00,\n",
|
|
" \"delivery_option\": \"Fast\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Super Magnet (XL)\",\n",
|
|
" \"quantity\": 1,\n",
|
|
" \"unit_price\": 150.00,\n",
|
|
" \"delivery_option\": \"Fast\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Rocket-Powered Roller skates\",\n",
|
|
" \"quantity\": 2,\n",
|
|
" \"unit_price\": 79.99,\n",
|
|
" \"delivery_option\": \"Next Day\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"image_url\": \"https://storage.googleapis.com/github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/EF0004.jpg\",\n",
|
|
" \"image_uri\": \"gs://github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/EF0004.jpg\",\n",
|
|
" \"image_type\": \"image/jpeg\",\n",
|
|
" \"image_name\": \"EF0004.jpg\",\n",
|
|
" \"reference\": { # The Ground Truth\n",
|
|
" \"order_number\": \"EF0004\",\n",
|
|
" \"order_date\": \"2025-10-26\",\n",
|
|
" \"customer_name\": \"Elmer J. Fudd\",\n",
|
|
" \"customer_address\": \"Happy Hunter's Hollow, Looney Tune Forest, CA\",\n",
|
|
" \"line_items\": [\n",
|
|
" {\n",
|
|
" \"item_description\": \"Silent Sneak Shoes\",\n",
|
|
" \"quantity\": 1,\n",
|
|
" \"unit_price\": 35.99,\n",
|
|
" \"delivery_option\": \"Standard\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Invisible Rabbit Trap\",\n",
|
|
" \"quantity\": 2,\n",
|
|
" \"unit_price\": 75.00,\n",
|
|
" \"delivery_option\": \"Standard\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Giant Butterfly Net\",\n",
|
|
" \"quantity\": 1,\n",
|
|
" \"unit_price\": 49.50,\n",
|
|
" \"delivery_option\": \"Fast\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Instant Camouflage Kit\",\n",
|
|
" \"quantity\": 3,\n",
|
|
" \"unit_price\": 65.00,\n",
|
|
" \"delivery_option\": \"Next Day\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"item_description\": \"Repellent Spray\",\n",
|
|
" \"quantity\": 4,\n",
|
|
" \"unit_price\": 29.99,\n",
|
|
" \"delivery_option\": \"Next Day\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" },\n",
|
|
" },\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mCjahTXbx9c2"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Display the images using their public URLs\n",
|
|
"\n",
|
|
"for image in images:\n",
|
|
" print(image[\"image_name\"])\n",
|
|
" display(Image(url=image[\"image_url\"], height=800))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "f8gOpub_1GYB"
|
|
},
|
|
"source": [
|
|
"These are mock order forms for *Acme Corporation*, for customers to order various products, and select a delivery option for each; either \"Standard\", \"Fast\" or \"Next Day\".\n",
|
|
"\n",
|
|
"We will use this form to evaluate the performance of Gemini."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TWKVr7mmjxy0"
|
|
},
|
|
"source": [
|
|
"## Extract the data using Gemini\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "X0UJShmGMv3h"
|
|
},
|
|
"source": [
|
|
"### Select the models"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3FWWciVnjxy0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define which models to compare\n",
|
|
"\n",
|
|
"models = [\n",
|
|
" # Gemini 2.0 family\n",
|
|
" \"gemini-2.0-flash\",\n",
|
|
" # Gemini 2.5 family\n",
|
|
" \"gemini-2.5-flash\",\n",
|
|
" \"gemini-2.5-pro\",\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "qiQ4jCWNM3lj"
|
|
},
|
|
"source": [
|
|
"### Define the prompt and schema"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cQScA9Wujxy0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the prompt and the structured output schema\n",
|
|
"\n",
|
|
"prompt = \"\"\"\n",
|
|
" Analyze the attached scanned form and extract the information in the table in accordance with the schema.\n",
|
|
"\n",
|
|
" Provide the output in a clean JSON format.\n",
|
|
"\n",
|
|
" If any date field is formatted ambiguously, assume the dates are in dd/mm/yyyy format.\n",
|
|
"\n",
|
|
" If a field is blank, illegible, or cannot be found, return null for its value.\n",
|
|
"\n",
|
|
" If there are blank rows, do not include them in the output.\n",
|
|
"\n",
|
|
" If there is no image attached, return null for all fields.\n",
|
|
"\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Use structured output to ensure well formatted and consistent JSON output\n",
|
|
"\n",
|
|
"schema = {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"order_number\": {\"type\": \"string\"},\n",
|
|
" \"order_date\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"format\": \"date\", # Note: Enforces a full date output in the RFC 3339 format (\"YYYY-MM-DD\")\n",
|
|
" },\n",
|
|
" \"customer_name\": {\"type\": \"string\"},\n",
|
|
" \"customer_address\": {\"type\": \"string\"},\n",
|
|
" \"line_items\": {\n",
|
|
" \"type\": \"array\",\n",
|
|
" \"items\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"item_description\": {\"type\": \"string\"},\n",
|
|
" \"quantity\": {\"type\": \"integer\"},\n",
|
|
" \"unit_price\": {\"type\": \"number\"},\n",
|
|
" \"delivery_option\": { # Note: We do not tell Gemini how to interpret the checkboxes as \"Standard\", \"Fast\" or \"Next Day\"\n",
|
|
" \"type\": \"string\"\n",
|
|
" },\n",
|
|
" },\n",
|
|
" },\n",
|
|
" },\n",
|
|
" },\n",
|
|
"}\n",
|
|
"\n",
|
|
"generate_content_config = GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=schema,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "F57ILmc0M8ZE"
|
|
},
|
|
"source": [
|
|
"### Run the prompt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cCqG3xE_jxy0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Run the prompt for each model in `models` and each image in `images`, storing the output in `gemini_response`\n",
|
|
"\n",
|
|
"gemini_response = {}\n",
|
|
"run_id = notebook_utils.generate_uuid(8)\n",
|
|
"\n",
|
|
"\n",
|
|
"for image_info in images:\n",
|
|
" image = Part.from_uri(\n",
|
|
" file_uri=image_info[\"image_uri\"], mime_type=image_info[\"image_type\"]\n",
|
|
" )\n",
|
|
" image_name = image_info[\"image_name\"]\n",
|
|
"\n",
|
|
" gemini_response[image_name] = {}\n",
|
|
"\n",
|
|
" for model in models:\n",
|
|
" run_name = f\"{run_id}-{model}-{image_name}\"\n",
|
|
"\n",
|
|
" response = client.models.generate_content(\n",
|
|
" model=model, contents=[prompt, image], config=generate_content_config\n",
|
|
" )\n",
|
|
"\n",
|
|
" response_json = json.dumps(response.parsed, indent=4)\n",
|
|
"\n",
|
|
" print(\"----------------------------------\")\n",
|
|
" print(f\"{run_name}: \\n{response_json}\")\n",
|
|
" gemini_response[image_name][model] = response_json"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "oUNVUz6nNHQE"
|
|
},
|
|
"source": [
|
|
"## Perform the Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "EdvJRUWRNGHE"
|
|
},
|
|
"source": [
|
|
"### Prepare the evaluation dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rajaEV5i6auy"
|
|
},
|
|
"source": [
|
|
"Now we have the outputs from the Gemini models we can run the evaulation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "hUGFuYawEJYX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create the Evaluation Dataset\n",
|
|
"\n",
|
|
"eval_dataset_rows = []\n",
|
|
"for image_info in images:\n",
|
|
" image_name = image_info[\"image_name\"]\n",
|
|
" image_uri = image_info[\"image_uri\"]\n",
|
|
" image_type = image_info[\"image_type\"]\n",
|
|
" reference_str = json.dumps(\n",
|
|
" image_info[\"reference\"], indent=4\n",
|
|
" ) # Convert the reference (ground truth) to pretty-printed JSON\n",
|
|
"\n",
|
|
" if image_name in gemini_response:\n",
|
|
" models_data = gemini_response[image_name]\n",
|
|
" for model_name, response_text in models_data.items():\n",
|
|
" eval_dataset_rows.append(\n",
|
|
" {\n",
|
|
" \"model\": model_name,\n",
|
|
" \"prompt\": prompt, # The same prompt is used for all Gemini calls\n",
|
|
" \"image\": image_name,\n",
|
|
" \"reference\": reference_str,\n",
|
|
" \"response\": response_text,\n",
|
|
" \"differences\": DeepDiff(\n",
|
|
" json.loads(reference_str), json.loads(response_text)\n",
|
|
" ).pretty(), # Uses the `deepdiff` library for identifying the differences between the response and the references\n",
|
|
" }\n",
|
|
" )\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame(eval_dataset_rows)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yrTqbS6NAKHk"
|
|
},
|
|
"source": [
|
|
"This evaluation data set now contains the reference (ground truth) and response for each combination of model and image."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "E6ByFZhujxy1"
|
|
},
|
|
"source": [
|
|
"### Define custom metrics for JSON schema validation and accuracy"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "W2v2WdYbjxy1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define a custom evaluation metric to assess whether the response complies with the schema\n",
|
|
"\n",
|
|
"\n",
|
|
"def is_valid_schema(instance: dict[str, str]) -> dict[str, bool]:\n",
|
|
" \"\"\"Return 1 if the response complies with the schema, 0 if not.\"\"\"\n",
|
|
" response = instance[\"response\"]\n",
|
|
"\n",
|
|
" try:\n",
|
|
" validate(instance=json.loads(response), schema=schema)\n",
|
|
" except Exception:\n",
|
|
" return {\"valid_schema\": False}\n",
|
|
"\n",
|
|
" return {\"valid_schema\": True}\n",
|
|
"\n",
|
|
"\n",
|
|
"valid_schema = CustomMetric(name=\"valid_schema\", metric_function=is_valid_schema)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "H76MFhlcy5lA"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define a `CustomMetric` for accuracy using the `deepdiff` library\n",
|
|
"\n",
|
|
"\n",
|
|
"def calculate_accuracy(instance: dict[str, str]) -> dict[str, float]:\n",
|
|
" ref_json_string = instance[\"reference\"]\n",
|
|
" resp_json_string = instance[\"response\"]\n",
|
|
"\n",
|
|
" try:\n",
|
|
" reference_data = json.loads(ref_json_string)\n",
|
|
" response_data = json.loads(resp_json_string)\n",
|
|
" except json.JSONDecodeError:\n",
|
|
" # If JSON is invalid or parsing fails, return 0 accuracy\n",
|
|
" return {\"accuracy\": 0.0}\n",
|
|
"\n",
|
|
" # Use the deepdiff library to calculate the difference between the response and the reference (0 = exact match, 1 = )\n",
|
|
" deep_distance = DeepDiff(\n",
|
|
" reference_data, response_data, ignore_order=True, get_deep_distance=True\n",
|
|
" ).get(\"deep_distance\")\n",
|
|
"\n",
|
|
" accuracy = 1.0 - deep_distance\n",
|
|
"\n",
|
|
" return {\"accuracy\": accuracy}\n",
|
|
"\n",
|
|
"\n",
|
|
"accuracy = CustomMetric(name=\"accuracy\", metric_function=calculate_accuracy)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "fXdy-2UVjxy1"
|
|
},
|
|
"source": [
|
|
"### Define `EvalTask` & Experiment"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "gRoVmM2gjxy1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the evaluation task\n",
|
|
"\n",
|
|
"extraction_eval_task = EvalTask(\n",
|
|
" dataset=eval_dataset,\n",
|
|
" metrics=[\n",
|
|
" \"exact_match\", # Exact match will only be 1 if the response is perfectly accurate, with no allowance for inconsistent JSON formatting. Hence, the custom `accuracy` metric is the better metric.\n",
|
|
" valid_schema,\n",
|
|
" accuracy,\n",
|
|
" ],\n",
|
|
" experiment=EXPERIMENT_NAME,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Pj69SXdqNdLC"
|
|
},
|
|
"source": [
|
|
"### Run the Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "e6KhR-uGjxy1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the experiment & experiment run\n",
|
|
"run_id = notebook_utils.generate_uuid(8)\n",
|
|
"\n",
|
|
"experiment_run_name = f\"eval-{run_id}\"\n",
|
|
"\n",
|
|
"eval_result = extraction_eval_task.evaluate(experiment_run_name=experiment_run_name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "LXmjQ4vkNikd"
|
|
},
|
|
"source": [
|
|
"### Display the results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "dQvVqjfrjxy1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"notebook_utils.display_eval_result(eval_result=eval_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tQMjObguZA-I"
|
|
},
|
|
"source": [
|
|
"## Analyse the results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8-Lj96MvOqW4"
|
|
},
|
|
"source": [
|
|
"### Use Gemini to analyse the results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "oS2LVREz1zOF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Let's get Gemini to analyse the results.\n",
|
|
"\n",
|
|
"# Prepare the prompt for Gemini 2.5 Flash to summarize and analyze the results\n",
|
|
"summary_prompt = \"\"\"\n",
|
|
"Analyze the following experiment results comparing Gemini models for extracted data from a scanned form.\n",
|
|
"The results include a summary table with overall metrics and row-based metrics, as well as the specific differences between the extracted data and the reference (ground truth).\n",
|
|
"\n",
|
|
"Summarize the performance of each model based on the metrics provided (valid_schema, accuracy) from the summary table.\n",
|
|
"Analyze the detailed differences to understand the *types* of errors and mismatches occurring for each model.\n",
|
|
"Identify which models performed best and worst for each metric and based on the detailed error analysis.\n",
|
|
"Draw conclusions about the strengths and weaknesses of Gemini models for this specific tabular data extraction task, considering both the overall accuracy and the nature of the errors.\n",
|
|
"Consider the different versions of Gemini and how their performance varies.\n",
|
|
"Provide a clear and concise summary of the overall results, followed by key conclusions supported by observations from the detailed comparison.\n",
|
|
"\n",
|
|
"Experiment Results Summary Table:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Convert the evaluation results summary and row-based metrics to a string format\n",
|
|
"# Assuming eval_result has a structure that can be converted to a readable string\n",
|
|
"try:\n",
|
|
" # This will likely involve converting the DataFrames within eval_result to string\n",
|
|
" eval_result_string = str(eval_result)\n",
|
|
"except Exception as e:\n",
|
|
" eval_result_string = f\"Could not convert evaluation results to string: {e}\"\n",
|
|
" print(eval_result_string)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Concatenate the prompt and the summary table results\n",
|
|
"full_prompt = summary_prompt + eval_result_string\n",
|
|
"\n",
|
|
"# Use Gemini 2.5 Flash to analyze the results\n",
|
|
"try:\n",
|
|
" # Generate the response\n",
|
|
" response = client.models.generate_content(\n",
|
|
" model=\"gemini-2.5-flash\", contents=full_prompt\n",
|
|
" )\n",
|
|
"\n",
|
|
" # Display the summary and analysis from Gemini\n",
|
|
" display(Markdown(response.text))\n",
|
|
"\n",
|
|
"except Exception as e:\n",
|
|
" print(f\"An error occurred while calling Gemini: {e}\")\n",
|
|
" print(\n",
|
|
" \"Please ensure you have access to Gemini 2.5 Flash and your project/location settings are correct.\"\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5xRAqAOCpk3C"
|
|
},
|
|
"source": [
|
|
"## Conclusions\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TXVk6VeKpolc"
|
|
},
|
|
"source": [
|
|
"This notebook has shown how to use the Gen AI Evaluation Service to evaluate Gemini's Structured Output, for a document processing task.\n",
|
|
"\n",
|
|
"It uses a \"bring your own response\" approach and uses custom `valid_schema` and `accuracy` metrics as well as the `exact_match` metric.\n",
|
|
"\n",
|
|
"It also does a deep \"field-wise\" comparison of the responses to understand inaccuracies, and uses Gemini to summarise and analyse the results."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "evaluate_gemini_structured_output.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|