wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
googlecloudplatform--genera…/gemini/tuning/sft_gemini_custom_metric_evaluation.ipynb
T
wehub-resource-sync 914fea506e chore: import upstream snapshot with attribution
2026-07-13 13:30:30 +08:00

703 lines
28 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Integrate Custom Metrics into Gemini Supervised Fine-Tuning\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/tuning/sft_gemini_custom_metric_evaluation.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ftuning%2Fsft_gemini_custom_metric_evaluation.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/tuning/sft_gemini_custom_metric_evaluation.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/tuning/sft_gemini_custom_metric_evaluation.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/sft_gemini_custom_metric_evaluation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/sft_gemini_custom_metric_evaluation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/sft_gemini_custom_metric_evaluation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/sft_gemini_custom_metric_evaluation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/sft_gemini_custom_metric_evaluation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| Jessica Wang |\n",
"| [Ivan Nardini](https://github.com/inardini) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This tutorial shows you how to integrate custom Python evaluation metrics into Gemini supervised fine-tuning (SFT) workflows using Vertex AI Gen AI Evaluation service.\n",
"\n",
"### Why Custom Metrics for Tuning?\n",
"\n",
"When fine-tuning Gemini, training loss doesn't tell you if your model is improving on **your specific quality criteria**. Custom metrics let you track what matters:\n",
"\n",
"- **Summary quality**: Is the model generating concise, accurate summaries?\n",
"- **Content coverage**: Does the summary capture the key points from the source text?\n",
"- **Writing style**: Is the summary following your preferred format (bullet points, sentences, etc.)?\n",
"\n",
"By integrating custom metrics into tuning, you can **measure model improvement on the criteria you care about** as it trains.\n",
"\n",
"### What You'll Learn\n",
"\n",
"In this tutorial, you will:\n",
"1. **Write a custom evaluation function** that scores summary quality\n",
"2. **Submit a tuning job** with your custom metric integrated via REST API\n",
"3. **Monitor the custom metric** as your model trains\n",
"\n",
"### Prerequisites\n",
"\n",
"- A Google Cloud project with billing enabled\n",
"- The Vertex AI API enabled ([enable it here](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com))\n",
"- A Google Cloud Storage bucket\n",
"- Training and validation datasets in supervised tuning format\n",
"- **No SDK required**—we use the Vertex AI REST API directly!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment\n",
"\n",
"If you are running this notebook in **Google Colab**, run the cell below to authenticate your account."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()\n",
" print(\"✅ Authentication successful!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"import json\n",
"\n",
"# TODO: Replace with your actual project ID\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"LOCATION = \"us-central1\" # @param {type: \"string\"}\n",
"# fmt: on\n",
"\n",
"# Auto-detect from environment if not set\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\", \"\"))\n",
"\n",
"if not PROJECT_ID:\n",
" raise ValueError(\"❌ Please set your PROJECT_ID above\")\n",
"\n",
"# Define GCS paths\n",
"BUCKET_NAME = f\"{PROJECT_ID}-gemini-sft-eval\"\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\"\n",
"\n",
"print(f\"📦 Creating GCS bucket: {BUCKET_NAME}...\")\n",
"\n",
"# Create the bucket\n",
"!gcloud storage buckets create {BUCKET_URI} --location {LOCATION} --project {PROJECT_ID}\n",
"\n",
"print(f\"✅ Using project: {PROJECT_ID}\")\n",
"print(f\"✅ Using region: {LOCATION}\")\n",
"print(f\"✅ Bucket: {BUCKET_NAME}\")\n",
"print(f\"✅ Bucket URI: {BUCKET_URI}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AgKr6gF8xZ7P"
},
"source": [
"## Step 1: Write Your Custom Evaluation Function\n",
"\n",
"Let's start creating a custom evaluation function that measures how well the model generates summaries.\n",
"\n",
"### What Makes a Good Custom Metric?\n",
"\n",
"For this summarization task, we want to measure:\n",
"- **Content overlap**: Does the summary include the key information?\n",
"- **Word-level accuracy**: How many words from the reference appear in the prediction?\n",
"- **Completeness**: Does the prediction cover all the main points?\n",
"\n",
"We'll use an **F1 score** approach based on word overlap:\n",
"- **Precision**: What fraction of the predicted words appear in the reference?\n",
"- **Recall**: What fraction of the reference words appear in the prediction?\n",
"- **F1**: The harmonic mean of precision and recall (0.0 to 1.0)\n",
"\n",
"This is a simple but effective metric for evaluating summary quality.\n",
"\n",
"### Define the evaluation function\n",
"\n",
"Your evaluation function **must**:\n",
"1. Be named `evaluate`\n",
"2. Accept one parameter: `instance` (a dictionary with `prediction` and `reference` fields)\n",
"3. Return a number (the score)\n",
"\n",
"**Important:** The function is defined as a string because it will be sent to the Vertex AI API and executed in a secure sandbox environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Aj1FscMYxgzZ"
},
"outputs": [],
"source": [
"# Define the custom evaluation function as a string\n",
"# This function compares summary text using word overlap F1 score\n",
"\n",
"evaluation_function = '''def evaluate(instance):\n",
" \"\"\"\n",
" Evaluate summary quality by comparing prediction to reference.\n",
"\n",
" Args:\n",
" instance: Dict with 'prediction' (model output) and 'reference' (ground truth)\n",
"\n",
" Returns:\n",
" F1 score between 0.0 and 1.0 based on word overlap\n",
" \"\"\"\n",
" # Get prediction and reference texts\n",
" prediction = instance.get(\"prediction\", \"\").strip().lower()\n",
" reference = instance.get(\"reference\", \"\").strip().lower()\n",
"\n",
" # If either is empty, return 0\n",
" if not prediction or not reference:\n",
" return 0.0\n",
"\n",
" # Exact match gets perfect score\n",
" if prediction == reference:\n",
" return 1.0\n",
"\n",
" # Calculate word-level overlap (F1-like metric)\n",
" pred_words = set(prediction.split())\n",
" ref_words = set(reference.split())\n",
"\n",
" if not pred_words or not ref_words:\n",
" return 0.0\n",
"\n",
" # Calculate overlap\n",
" overlap = pred_words.intersection(ref_words)\n",
"\n",
" # Precision: what fraction of predicted words are in reference\n",
" precision = len(overlap) / len(pred_words) if pred_words else 0.0\n",
"\n",
" # Recall: what fraction of reference words are in prediction\n",
" recall = len(overlap) / len(ref_words) if ref_words else 0.0\n",
"\n",
" # F1 score\n",
" if precision + recall == 0:\n",
" return 0.0\n",
"\n",
" f1 = 2 * (precision * recall) / (precision + recall)\n",
" return f1\n",
"'''\n",
"\n",
"print(\"✅ Custom evaluation function defined\")\n",
"print(\"\\nFunction summary:\")\n",
"print(\" - Compares prediction text to reference text\")\n",
"print(\" - Calculates word-level F1 score\")\n",
"print(\" - Returns 1.0 for exact match\")\n",
"print(\" - Returns 0.0 for no overlap\")\n",
"print(\" - Returns F1 score (0.0 to 1.0) based on word overlap\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wNHSPPI0xqgE"
},
"source": [
"## Step 2: Integrate Custom Metric into Tuning Job\n",
"\n",
"Now comes the exciting part: integrating your custom metric into a Gemini tuning job!\n",
"\n",
"### How This Works\n",
"\n",
"When you submit a tuning job with `evaluationConfig`, Vertex AI will:\n",
"1. Train the model on your training data\n",
"2. Periodically generate predictions on your validation data\n",
"3. For each prediction, run your custom evaluation function\n",
"4. Aggregate the scores (e.g., compute average)\n",
"5. Report the metrics so you can track improvement\n",
"\n",
"### Prerequisites for This Step\n",
"\n",
"You'll need:\n",
"- **Training dataset**: JSONL file with examples in supervised tuning format\n",
"- **Validation dataset**: JSONL file with validation examples (also in SFT format)\n",
"- Both datasets uploaded to Google Cloud Storage\n",
"\n",
"**Note:** For this tutorial, we'll use placeholder GCS paths. In production, replace these with paths to your actual training/validation datasets."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4QgPjY9IxwCZ"
},
"source": [
"### Step 2.1: Configure dataset paths\n",
"\n",
"Define the GCS paths for your training, and validation datasets.\n",
"\n",
"**Important:**\n",
"- Your **training and validation datasets** should be in standard supervised tuning format\n",
"- All files must be uploaded to GCS before submitting the tuning job"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Pm1FRd9kx1NU"
},
"outputs": [],
"source": [
"# Configure dataset paths\n",
"# TODO: Replace these with your actual dataset GCS paths\n",
"\n",
"# Training and validation datasets (standard SFT format)\n",
"TRAINING_DATASET_URI = \"gs://cloud-samples-data/ai-platform/generative_ai/gemini-2_0/text/sft_train_data.jsonl\"\n",
"VALIDATION_DATASET_URI = \"gs://cloud-samples-data/ai-platform/generative_ai/gemini-2_0/text/sft_validation_data.jsonl\"\n",
"\n",
"# Where to save evaluation results\n",
"EVAL_OUTPUT_URI = f\"{BUCKET_URI}/evaluation_results\"\n",
"\n",
"print(\"✅ Dataset paths configured\")\n",
"print(f\"\\nTraining data: {TRAINING_DATASET_URI}\")\n",
"print(f\"Validation data: {VALIDATION_DATASET_URI}\")\n",
"print(f\"\\nEvaluation results will be saved to: {EVAL_OUTPUT_URI}\")\n",
"print(\"\\n💡 For production: Replace training/validation paths with your own datasets\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1hSjwIuSyD50"
},
"source": [
"### Step 2.2: Build the tuning job request\n",
"\n",
"Now let's construct the REST API request for the tuning job with integrated custom evaluation.\n",
"\n",
"**Key sections in the request are**:\n",
"\n",
"| Section | Purpose |\n",
"|---------|---------|\n",
"| `base_model` | The foundation model to fine-tune (e.g., gemini-2.5-flash) |\n",
"| `supervisedTuningSpec` | Configuration for supervised fine-tuning |\n",
"| `trainingDatasetUri` | Your training examples |\n",
"| `validationDatasetUri` | Your validation examples |\n",
"| **`evaluationConfig`** | **This is where we integrate the custom metric!** |\n",
"| `metrics.custom_code_execution_spec` | Your custom evaluation function |\n",
"| `metrics.aggregation_metrics` | How to aggregate scores (AVERAGE, MAXIMUM, etc.) |\n",
"| `outputConfig` | Where to save detailed evaluation results |\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ujetjIgkyJes"
},
"outputs": [],
"source": [
"# Build the tuning job request with custom evaluation\n",
"tuning_request = {\n",
" \"description\": \"Gemini tuning with custom summary evaluation metric\",\n",
" \"base_model\": \"gemini-2.5-flash\",\n",
" \"supervisedTuningSpec\": {\n",
" # Standard tuning configuration\n",
" \"trainingDatasetUri\": TRAINING_DATASET_URI,\n",
" \"validationDatasetUri\": VALIDATION_DATASET_URI,\n",
"\n",
" # ============================================================\n",
" # THIS IS THE KEY PART: Custom evaluation configuration\n",
" # ============================================================\n",
" \"evaluationConfig\": {\n",
" \"metrics\": {\n",
" # Request AVERAGE score across all evaluation examples\n",
" \"aggregation_metrics\": [\"AVERAGE\"],\n",
"\n",
" # Provide our custom evaluation function\n",
" \"custom_code_execution_spec\": {\n",
" \"evaluation_function\": evaluation_function\n",
" }\n",
" },\n",
" # Save detailed evaluation results to GCS\n",
" \"outputConfig\": {\n",
" \"gcs_destination\": {\n",
" \"output_uri_prefix\": EVAL_OUTPUT_URI\n",
" }\n",
" }\n",
" }\n",
" }\n",
"}\n",
"\n",
"# Save the request to a JSON file\n",
"with open(\"tuning_request.json\", \"w\") as f:\n",
" json.dump(tuning_request, f, indent=2)\n",
"\n",
"print(\"✅ Tuning job request created\")\n",
"print(\"\\nRequest configuration:\")\n",
"print(\" ✓ Base model: gemini-2.5-flash\")\n",
"print(f\" ✓ Training dataset: {TRAINING_DATASET_URI}\")\n",
"print(f\" ✓ Validation dataset: {VALIDATION_DATASET_URI}\")\n",
"print(\" ✓ Custom metric: Summary Word Overlap F1 Score\")\n",
"print(\" ✓ Aggregation: AVERAGE\")\n",
"print(f\" ✓ Results output: {EVAL_OUTPUT_URI}\")\n",
"print(\"\\nSaved to: tuning_request.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_2kYi8o7yN4G"
},
"source": [
"### Step 2.3: Submit the tuning job\n",
"\n",
"Now we'll submit the tuning job using the Vertex AI REST API with `curl`.\n",
"\n",
"**What happens when you run this cell:**\n",
"1. The API creates a new tuning job\n",
"2. Returns immediately with a job ID\n",
"3. Training starts in the background (takes 30-60 minutes)\n",
"4. Your custom metric will be evaluated periodically during training\n",
"\n",
"**Expected output:** You'll receive a JSON response containing:\n",
"- `name`: The full tuning job resource name\n",
"- `state`: Should be `JOB_STATE_PENDING` initially\n",
"- `tunedModelDisplayName`: The name of your tuned model\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NGmRlEEuyRa3"
},
"outputs": [],
"source": [
"# Build the Vertex AI tuning jobs API endpoint\n",
"API_ENDPOINT = f\"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/{LOCATION}/tuningJobs\"\n",
"\n",
"print(\"🚀 Submitting tuning job with custom evaluation metric...\")\n",
"print(f\"\\nAPI Endpoint: {API_ENDPOINT}\\n\")\n",
"\n",
"# Submit the tuning job using curl\n",
"!curl -X POST \\\n",
" -H \"Content-Type: application/json\" \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" {API_ENDPOINT} \\\n",
" -d @tuning_request.json\n",
"\n",
"print(\"\\n\" + \"=\"*80)\n",
"print(\"✅ Tuning job submitted successfully!\")\n",
"print(\"=\"*80)\n",
"print(\"\\n📋 Next steps:\")\n",
"print(\" 1. Copy the 'name' field from the response above\")\n",
"print(\" 2. Run the next cell to monitor the job status\")\n",
"print(\" 3. Your custom metric will be evaluated during training\")\n",
"print(\"\\n⏱️ Expected training time: 30-60 minutes\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n26eKAvZyW0N"
},
"source": [
"## Step 3: Monitor Your Custom Metric\n",
"\n",
"Now that your tuning job is running, let's check its status and see your custom metric results. Below you have a quick overview to understanding Tuning Job States:\n",
"\n",
"| State | Meaning |\n",
"|-------|---------|\n",
"| `JOB_STATE_PENDING` | Waiting for resources |\n",
"| `JOB_STATE_RUNNING` | Training in progress |\n",
"| `JOB_STATE_SUCCEEDED` | Training complete! |\n",
"| `JOB_STATE_FAILED` | Something went wrong - check error message |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o3SCooxKygFZ"
},
"source": [
"### Step 3.1: Check tuning job status\n",
"\n",
"Paste the tuning job name from the previous cell's output to check its status.\n",
"\n",
"**How to use this cell:**\n",
"1. Find the `\"name\"` field in the response above (looks like `projects/.../tuningJobs/...`)\n",
"2. Copy the full path\n",
"3. Paste it in the `TUNING_JOB_NAME` field below\n",
"4. Run the cell\n",
"\n",
"**What you'll see:**\n",
"- Current job state\n",
"- Tuned model details (when complete)\n",
"- Any error messages (if failed)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eGL2r-AiytMS"
},
"outputs": [],
"source": [
"# TODO: Paste your tuning job name from the previous cell\n",
"# fmt: off\n",
"TUNING_JOB_NAME = \"projects/YOUR_PROJECT/locations/us-central1/tuningJobs/YOUR_JOB_ID\" # @param {type:\"string\"}\n",
"TUNING_JOB_NAME = \"projects/541923329259/locations/us-central1/tuningJobs/2125697426391040000\" # @param {type:\"string\"}\n",
"# fmt: on\n",
"\n",
"if \"YOUR_PROJECT\" in TUNING_JOB_NAME or \"YOUR_JOB\" in TUNING_JOB_NAME:\n",
" print(\"⚠️ Please paste your tuning job name from the cell above\")\n",
" print(\" It should look like: projects/12345/locations/us-central1/tuningJobs/67890\")\n",
"else:\n",
" # Build the status check URL\n",
" STATUS_URL = f\"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/{TUNING_JOB_NAME}\"\n",
"\n",
" print(f\"📊 Checking tuning job status...\\n\")\n",
"\n",
" # Get the job status\n",
" !curl -s \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" {STATUS_URL}\n",
"\n",
" print(\"\\n\" + \"=\"*80)\n",
" print(\"💡 Job Status Tips:\")\n",
" print(\"=\"*80)\n",
" print(\" - PENDING: Job is queued, waiting for resources\")\n",
" print(\" - RUNNING: Training is in progress\")\n",
" print(\" - SUCCEEDED: Training complete! Check evaluationConfig results in GCS\")\n",
" print(\" - FAILED: Check the error message\")\n",
" print(\"\\n Run this cell again to refresh the status\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bXb3pAa8zNPS"
},
"source": [
"### Step 3.2: View custom metric results\n",
"\n",
"Once training completes, your custom metric evaluation results will be saved to Google Cloud Storage.\n",
"\n",
"**What gets saved:**\n",
"- Detailed per-example evaluation scores\n",
"- Aggregate statistics (AVERAGE in our case)\n",
"- Timestamp information\n",
"\n",
"**To view your results:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GmNI6gGSzQ5v"
},
"outputs": [],
"source": [
"# List evaluation result files in GCS\n",
"print(f\"📂 Looking for evaluation results in: {EVAL_OUTPUT_URI}\\n\")\n",
"\n",
"!gcloud storage ls --recursive {EVAL_OUTPUT_URI}\n",
"\n",
"print(\"\\n\" + \"=\"*80)\n",
"print(\"📊 Viewing Custom Metric Results\")\n",
"print(\"=\"*80)\n",
"print(f\"\\nEvaluation results are saved in: {EVAL_OUTPUT_URI}\")\n",
"print(\"\\nTo download and view the results:\")\n",
"print(f\"\\n gcloud storage cp --recursive {EVAL_OUTPUT_URI}/* ./eval_results/\")\n",
"print(\"\\nThe results will include:\")\n",
"print(\" - Individual summary evaluation scores\")\n",
"print(\" - Aggregate metrics (AVERAGE F1 score)\")\n",
"print(\" - Model-generated summaries vs. reference summaries\")\n",
"print(\"\\n💡 Use these metrics to track if your summary quality improves during training!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bU_NrFeJN1mz"
},
"source": [
"## Congratulations!\n",
"\n",
"You've successfully integrated a custom evaluation metric into Gemini supervised fine-tuning!\n",
"\n",
"### What You Accomplished\n",
"\n",
"1. **Wrote a custom metric**: Implemented a word overlap F1 score evaluator for summaries\n",
"2. **Integrated into tuning**: Added the custom metric to a tuning job configuration\n",
"3. **Submitted via REST API**: Used curl to submit the tuning job (no SDK required!)\n",
"4. **Monitored results**: Learned how to check job status and view metric outputs\n",
"\n",
"### Key Takeaways\n",
"\n",
"- **Custom metrics provide visibility**: You can now track summary quality metrics that matter for your specific use case during training\n",
"- **REST API is powerful**: No SDK required—curl gives you full control\n",
"- **Results are stored in GCS**: Detailed per-example scores help you understand model behavior\n",
"\n",
"### Next Steps\n",
"\n",
"**Customize for your use case:**\n",
"- **Multiple aggregations**: Add `MAXIMUM`, `MINIMUM`, `PERCENTILE_P99` to track different statistics\n",
"- **Real datasets**: Replace the sample data with your actual production examples\n",
"- **Compare models**: Run multiple tuning jobs with different configurations and compare custom metrics\n",
"\n",
"**Learn more:**\n",
"- [Vertex AI Tuning Documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/models/tune-models)\n",
"- [Vertex AI Gen AI Evaluation Documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)"
]
}
],
"metadata": {
"colab": {
"name": "sft_gemini_custom_metric_evaluation.ipynb",
"toc_visible": true
},
"environment": {
"kernel": "python3",
"name": "workbench-notebooks.m132",
"type": "gcloud",
"uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m132"
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": ""
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink