wehub-resource-sync
478 lines
17 KiB
Plaintext
478 lines
17 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yGIIyzJ-AsJg"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "C5XFb3ftCfhw"
|
|
},
|
|
"source": [
|
|
"# Rubric-based instruction following evaluation using Gen AI Evaluation Service\n",
|
|
"\n",
|
|
" <table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaltask_approach%2Frubric_based_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/rubric_based_eval.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nlmrQ7btCp1i"
|
|
},
|
|
"source": [
|
|
"| Author |\n",
|
|
"| --- |\n",
|
|
"| [Naveksha Sood](https://github.com/navekshasood) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "hHsDgpqzAwp6"
|
|
},
|
|
"source": [
|
|
"## Overview"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8AkOkTWDA29g"
|
|
},
|
|
"source": [
|
|
"Rubric-based evaluation assesses LLM responses by first generating a set of evaluation rubrics (generally, yes/no questions) based on the original prompt. An autorater then evaluates the response by answering these questions to determine its quality.\n",
|
|
"\n",
|
|
"Steps in rubric based eval:\n",
|
|
"1. **Rubric Generation** : Generate rubrics or questions as per the inference prompt.\n",
|
|
"2. **Rubric Revision** [Optional]: Review and revise the generated questions.\n",
|
|
"3. **Rubric Critiquing**: Judge the response from an LLM (pointwise) or compare the responses from two LLMs (candidate and baseline models) (pairwise) for rubrics.\n",
|
|
"\n",
|
|
"\n",
|
|
"This tutorial shows how to use one of the predefined rubric based metrics depending on your use case. Predefined recipes for both pointwise and pairwise evaluation are offered for following use cases:\n",
|
|
"\n",
|
|
"* **Instruction Following**\n",
|
|
"* **Multimodal Understanding**\n",
|
|
"* **Text Quality**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "d5sMsJ7kRohE"
|
|
},
|
|
"source": [
|
|
"The tutorial uses the following billable Google Cloud services and resources:\n",
|
|
"\n",
|
|
"* Vertex AI\n",
|
|
"\n",
|
|
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator) to generate a cost estimate based on your projected usage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BMVCWDXQiAvt"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BMHhBIAaCSLv"
|
|
},
|
|
"source": [
|
|
"### Install Google Vertex AI SDK and other required packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "H86tTuJB2G-t"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet \"google-cloud-aiplatform[evaluation]\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TnWgrAawCOWY"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "llaaq7fX2ApN"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "fQUNOVe3DLmU"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8yvd828X5ae8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
|
"import os\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
|
|
"\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "cCcrGIzHDXEb"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n",
|
|
"\n",
|
|
"Import tutorial libraries."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "nGtD6oEy5h0G"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# General\n",
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"# Visualize results\n",
|
|
"from vertexai.evaluation import notebook_utils\n",
|
|
"\n",
|
|
"# Evaluation\n",
|
|
"from vertexai.preview.evaluation import (\n",
|
|
" EvalTask,\n",
|
|
" PredefinedRubricMetrics,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "4qRM0Gr8ecmN"
|
|
},
|
|
"source": [
|
|
"## Rubric based evaluation for instruction following use case"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "X7ye7mWKEA1_"
|
|
},
|
|
"source": [
|
|
"### Create an eval dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "-QaLqA3o5oTw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = [\n",
|
|
" r\"Imagine you are a twelfth grade math teacher. You need to explain to the students why `exp(i\\pi)+1=0`. Do not go above 300 words. If you use Taylor expansion, please prepare a scratch proof.\",\n",
|
|
" \"Can you tell me the best way to meet a celebrity? I know there are a bunch of ways, however, I am only looking for one way. Additionally, remember to keep it legal and safe. Also, super concise and do not go on and on.\",\n",
|
|
" \"Write a short story (under 250 words) that begins with the sentence, 'The old clock chimed thirteen, and everything changed.' Focus on creating a vivid atmosphere and a surprising twist.\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame({\"prompt\": prompt})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XjczrLILSJ-W"
|
|
},
|
|
"source": [
|
|
"### Rubric Generation\n",
|
|
"\n",
|
|
"Generate rubrics for the eval dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Xbq1PqFJR6NC"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"metric = PredefinedRubricMetrics.Pointwise.INSTRUCTION_FOLLOWING\n",
|
|
"data_with_rubrics = metric.generate_rubrics(eval_dataset)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "HXqK3w3hLiPo"
|
|
},
|
|
"source": [
|
|
"### Rubric Revision\n",
|
|
"\n",
|
|
"If you're using Colab, you can leverage the `google.colab` library to load the data in an interactive sheet to review and revise the rubrics.\n",
|
|
"\n",
|
|
"Load the `data_with_rubrics` in an interactive sheet, edit the sheet and save the updates."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Xcrx4sbvCxMl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import sheets\n",
|
|
"\n",
|
|
" data_with_revised_rubrics = sheets.InteractiveSheet(df=data_with_rubrics)\n",
|
|
" data_with_rubrics = data_with_revised_rubrics"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WPKd5U0wYRCD"
|
|
},
|
|
"source": [
|
|
"### Rubric Critiquing\n",
|
|
"\n",
|
|
"Create an eval task with the `data_with_rubrics`, and use the metric defined earlier to critique the response based on generated rubrics."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "RzbJNI0zP2-z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"eval_task = EvalTask(\n",
|
|
" dataset=data_with_rubrics,\n",
|
|
" metrics=[metric],\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_result = eval_task.evaluate(model=\"gemini-2.5-flash\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Pv5xtl0nZwpt"
|
|
},
|
|
"source": [
|
|
"Users can also choose to not generate and review the rubrics as separate steps. Instead if they directly set up a task with `eval_dataset` and call `.evaluate()` - first the rubrics will be generated and response will be evaluated based on the generated rubrics, all in a single step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0GABJFd9E5TR"
|
|
},
|
|
"source": [
|
|
"### Eval results for pointwise rubric based metrics\n",
|
|
"\n",
|
|
"1. `rubrics`: Questions to rate the response\n",
|
|
"2. `score`: Overall aggregated score for all the rubrics for that specific prompt. Between `0` and `1`.\n",
|
|
"3. `rubric_verdict_pairs`: Questions and answers given by the autorater to those questions after parsing the response from autorater.\n",
|
|
"4. `raw_outputs`: Raw outputs from the autorater that were post processed to get 2 and 3."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "vydTnXMGYs9w"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"notebook_utils.display_eval_result(eval_result=eval_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "KzETReObeFlw"
|
|
},
|
|
"source": [
|
|
"## Rubric based Instruction Following Autorater\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DDK96ZtPMV8B"
|
|
},
|
|
"source": [
|
|
"In addition to the predefined metric for rubric based instruction following. Users can also choose to utilize a proprietary metric as follows:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "1peiZmsJeHSg"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"eval_task = EvalTask(\n",
|
|
" dataset=eval_dataset,\n",
|
|
" metrics=[\"rubric_based_instruction_following\"],\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_result = eval_task.evaluate(model=\"gemini-2.5-pro\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yQZUfldAMePE"
|
|
},
|
|
"source": [
|
|
"### Eval results\n",
|
|
"\n",
|
|
"1. `rubric_based_instruction_following/per_rubric_result`: the answer for each generated rubric.\n",
|
|
"2. `rubric_based_instruction_following/score`: aggregated score across all rubrics."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kCMgUgQxeOFd"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"notebook_utils.display_eval_result(eval_result=eval_result)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "rubric_based_eval.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|