{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "EElURkc9VGpZ" }, "outputs": [], "source": [ "# Copyright 2025 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "l7hzJLswRlgJ" }, "source": [ "# Getting Started: Quick Gen AI Evaluation\n" ] }, { "cell_type": "markdown", "metadata": { "id": "U_zRPTySRjHX" }, "source": [ " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Open in Colab\n", " \n", " 		\n", " \n", " \"Google
Open in Colab Enterprise\n", " \n", " 		\n", " \n", " \"Vertex
Open in Vertex AI Workbench\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", "
\n", "\n", "
\n", "\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "YK2R3uVZR1RD" }, "source": [ "| Author(s) |\n", "| --- |\n", "| [Jason Dai](https://github.com/jsondai) |" ] }, { "cell_type": "markdown", "metadata": { "id": "1ijw7-AyR344" }, "source": [ "## Overview\n", "\n", "This notebook shows the quickest way to evaluate a single generative model using the Vertex AI SDK for Gen AI Eval Service.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "5E1mN9wWT-Ol" }, "source": [ "### Costs\n", "\n", "This tutorial uses billable components of Google Cloud:\n", "\n", "- Vertex AI\n", "\n", "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "9yt-EPacUD4j" }, "source": [ "## Getting Started\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-zTjlxYMRAog" }, "outputs": [], "source": [ "# @title ### Install Vertex AI SDK for Gen AI Evaluation Service\n", "\n", "%pip install --upgrade \"google-cloud-aiplatform[evaluation]>=1.111.0\" --force-reinstall --quiet --no-warn-conflicts" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "cellView": "form", "id": "AWP0pW9hUOBR" }, "outputs": [], "source": [ "# @title ### Authenticate your notebook environment (Colab only)\n", "# @markdown If you're running this notebook on Google Colab, run the cell below to authenticate your environment.\n", "\n", "import sys\n", "\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "cellView": "form", "id": "LHpeZ2pWUQag" }, "outputs": [], "source": [ "# @title ### Set Google Cloud project information\n", "# @markdown To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", "# @markdown Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n", "\n", "import os\n", "PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", "LOCATION= \"us-central1\" # @param {type: \"string\", placeholder: \"us-central1\", isTemplate: true}\n", "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", LOCATION)\n", "\n", "\n", "from vertexai import Client, types\n", "client = Client(project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1951 }, "id": "lDZ6j_p3Vct9", "outputId": "612eb15a-2af2-4dba-9f1f-9c2ff23cb014" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Gemini Inference: 100%|██████████| 5/5 [00:17<00:00, 3.42s/it]\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\n", " \n", " Evaluation Dataset\n", " \n", " \n", " \n", "\n", "\n", "
\n", "

Evaluation Dataset

\n", "
\n", "
\n", " \n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# @title ### Generate Responses\n", "# @markdown The eval workflow starts with `run_inference()` to generate model responses for your dataset. The SDK can automatically detects and handles several common data formats. This means you can often use your data as-is without needing to perform manual conversions\n", "\n", "\n", "import pandas as pd\n", "\n", "eval_df = pd.DataFrame({\n", " \"prompt\": [\n", " \"Explain software 'technical debt' using a concise analogy of planting a garden.\",\n", " \"Write a Python function to find the nth Fibonacci number using recursion with memoization, but without using any imports.\",\n", " \"Write a four-line poem about a lonely robot, where every line must be a question and the word 'and' cannot be used.\",\n", " \"A drawer has 10 red socks and 10 blue socks. In complete darkness, what is the minimum number of socks you must pull out to guarantee you have a matching pair?\",\n", " \"An AI discovers a cure for a major disease, but the cure is based on private data it analyzed without consent. Should the cure be released? Justify your answer.\"\n", " ]\n", "})\n", "\n", "eval_dataset = client.evals.run_inference(\n", " model=\"gemini-2.5-flash\",\n", " src=eval_df,\n", ")\n", "\n", "eval_dataset.show()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 4383 }, "id": "55AVmz7yVpPF", "outputId": "7655fc51-204d-4fcc-9201-7356444894fd" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Computing Metrics for Evaluation Dataset: 100%|██████████| 5/5 [00:18<00:00, 3.69s/it]\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\n", " \n", " Evaluation Report\n", " \n", " \n", " \n", "\n", "\n", "
\n", "

Evaluation Report

\n", "
\n", "
\n", "
\n", " \n", "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# @title ### Run Evaluation\n", "# @markdown Evaluate the responses using the GENERAL_QUALITY adaptive rubric-based metric by default.\n", "\n", "eval_result = client.evals.evaluate(dataset=eval_dataset)\n", "eval_result.show()" ] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }