wehub-resource-sync
436 lines
19 KiB
Plaintext
436 lines
19 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Evaluate videos with predefined Gecko\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_videos_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td> \n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_videos_with_predefined_gecko.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "UBAgFfnqvxy0"
|
|
},
|
|
"source": [
|
|
" | | | | |\n",
|
|
" |-|-|-|-|\n",
|
|
" |Author(s): | [Greg Breard](https://github.com/gregbreard) | Anant Nawalgaria | Olivia Wiles |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This CoLAB shows how to leverage the Vertex AI evaluation service in order to run [Gecko](https://arxiv.org/abs/2404.16820).\n",
|
|
"\n",
|
|
"As with a more standard rubric approach, Gecko proceeds in two stages: a rubric generation step followed by a validator step. The key difference is that the rubric is generated based on the prompt.\n",
|
|
"This allows for a more fine-grained metric that can be customized to prompts with differing challenges.\n",
|
|
"\n",
|
|
"In more detail, Gecko proceeds as follows, with two key steps: the QA generation step (ie the rubric generation step) and then the VQA step (ie the validator step).\n",
|
|
"\n",
|
|
"## The rubric generation step\n",
|
|
"Given a prompt, such as `A teddy bear riding a skateboard`, we prompt the Gemini model to generate a set of questions, answer choices and corresponding ground truth (GT) answer. The question is also tagged with a question type. Depending on the prompt, these questions can either be `yes`/`no` questions or multiple choice ones.\n",
|
|
"\n",
|
|
"`A teddy bear riding a skateboard` -->\n",
|
|
"\n",
|
|
"- `Q1: Is there a teddy bear? Choices: [yes, no]. GT Answer: yes. Tag: Object.`\n",
|
|
"- `Q2: Is there a skateboard? Choices: [yes, no]. GT Answer: yes. Tag: Object.`\n",
|
|
"- `Q3: Is the teddy bear riding a skateboard? Choices: [yes, no]. GT Answer: yes. Tag: Action.`\n",
|
|
"\n",
|
|
"## The validator step\n",
|
|
"Given a generated image and the questions above, we query the Gemini model for each question to give an answer. We then check if it matches the GT answer, with a result of 1 if it matches and 0 if it does not. We aggregate these results to give a final overall score, which can be broken down into scores per question. We can also aggregate scores based on tags.\n",
|
|
"\n",
|
|
"For example, imagine we have a generated image `<image1>` which includes a teddy bear but no skateboard, and Gemini outputs the following results:\n",
|
|
"\n",
|
|
"- `<image1> Is there a teddy bear? GT Answer: yes. Result: 1.`\n",
|
|
"- `<image1> Is there a skateboard? GT Answer: no. Result: 0.`\n",
|
|
"- `<image1> Is the teddy bear riding a skateboard? GT Answer: no. Result: 0.`\n",
|
|
"\n",
|
|
"The final score will be `0.33` with a score of `0.5` for the question tag and `0.0` for the action tag.\n",
|
|
"\n",
|
|
"## Further exploration\n",
|
|
"We provide two predefined metrics, engineered for video and image generation tasks. Below, we show how to run Gecko for the video modality on a set of generations.\n",
|
|
"\n",
|
|
"The quality can be analysed by exploring what questions are generated as well as the reliability of the validator step. Questions can also be manually added as desired for an application.\n",
|
|
"\n",
|
|
"## Steps\n",
|
|
"\n",
|
|
"1. Set up the environment.\n",
|
|
"2. Prepare the dataset for evaluation.\n",
|
|
"3. Run the evaluation.\n",
|
|
"\n",
|
|
"## Costs\n",
|
|
"This tutorial uses billable components of Google Cloud:\n",
|
|
"\n",
|
|
"- Vertex AI\n",
|
|
"\n",
|
|
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"# Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Install Vertex AI SDK for Python and other required packages\n",
|
|
"\n",
|
|
"%pip install --upgrade --quiet \"google-cloud-aiplatform[evaluation]>=1.122.0\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6q613INA6KwZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Authenticate your notebook environment (Colab only)\n",
|
|
"# @markdown If you're running this notebook on Google Colab, run the cell below to authenticate your environment.\n",
|
|
"\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "i7hQwDYs6LiI"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Set Google Cloud project information\n",
|
|
"# @markdown To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"# @markdown Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
|
|
"\n",
|
|
"# @markdown ---\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"LOCATION= \"us-central1\" # @param {type: \"string\", placeholder: \"us-central1\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"\n",
|
|
"from vertexai import Client, types\n",
|
|
"\n",
|
|
"client = Client(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nHjCRree99Px"
|
|
},
|
|
"source": [
|
|
"# Prepare the dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "HFJr-xZMaaUH"
|
|
},
|
|
"source": [
|
|
"In the following dataset, two prompts are used for each generated video. The first is the prompt that corresponds to the generated content. The second is a counterexample that is similar but does not exactly match the generated content. This is done to demonstrate the difference in the Gecko evaluation for high quality and low quality responses."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "U37FNLD8-Lp5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"prompts = [\n",
|
|
" \"Snow blanketed rocky mountains surround and shadow deep canyons. the canyons bend through the high elevated mountain peaks. black and white\",\n",
|
|
" \"Lush green valley is carved between rocky cliffs. the valley winds through the high elevated rock faces. misty morning\",\n",
|
|
" \"A couple in formal evening wear going home get caught in a heavy downpour with umbrellas\",\n",
|
|
" \"Two friends, dressed in casual summer clothes, are caught in a light summer rain while running home\",\n",
|
|
" \"A tranquil tableau of in the heart of the Utah desert, a massive sandstone arch spanned the horizon\",\n",
|
|
" \"A eerie panorama of the Arizona desert, with ancient ruins silhouetted against the setting sun\",\n",
|
|
" \"Few big purple plums rotating on the turntable. water drops appear on the skin during rotation. isolated on the white background. close-up\",\n",
|
|
" \"A large red apple rotating on the turntable. water drops appear on the skin during rotation. isolated on the black background. close-up\",\n",
|
|
" \"A boat sailing leisurely along the Seine River with the Eiffel Tower in background\",\n",
|
|
" \"A boat cruising rapidly along the Thames River with Big Ben behind\",\n",
|
|
"]\n",
|
|
"responses = [\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/mountain.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/mountain.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/couple.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/couple.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/desert.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/desert.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/plum.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/plum.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/boat.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"parts\": [\n",
|
|
" {\n",
|
|
" \"file_data\": {\n",
|
|
" \"mime_type\": \"video/mp4\",\n",
|
|
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/evaluation/videos/boat.mp4\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" \"role\": \"model\",\n",
|
|
" },\n",
|
|
"]\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"prompt\": prompts,\n",
|
|
" \"response\": responses,\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "qm6mHoPc-DS0"
|
|
},
|
|
"source": [
|
|
"# Run evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "5FAwPvYf7U7S"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Generate rubrics\n",
|
|
"# @markdown First we generate rubrics for the user prompts.\n",
|
|
"\n",
|
|
"data_with_rubrics = client.evals.generate_rubrics(\n",
|
|
" src=eval_dataset,\n",
|
|
" rubric_group_name=\"gecko_video_rubrics\",\n",
|
|
" predefined_spec_name=types.RubricMetric.GECKO_TEXT2VIDEO,\n",
|
|
")\n",
|
|
"\n",
|
|
"data_with_rubrics.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mMuDxiwy-MTt"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# @title ### Evaluate with rubrics\n",
|
|
"# @markdown Then we use the generated rubrics to evaluate the quality of the responses.\n",
|
|
"\n",
|
|
"eval_result = client.evals.evaluate(\n",
|
|
" dataset=data_with_rubrics,\n",
|
|
" metrics=[types.RubricMetric.GECKO_TEXT2VIDEO],\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_result.show()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "evaluate_videos_with_predefined_gecko.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|