{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "w2Nnpp1a3oKr" }, "outputs": [], "source": [ "# Copyright 2024 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "mSdMTAZgF_M2" }, "source": [ "# Evaluate Generative Model Tool Use with Custom Code Execution" ] }, { "cell_type": "markdown", "metadata": { "id": "b9ziZDnkF_j4" }, "source": [ "\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Open in Colab\n", " \n", " 		\n", " \n", " \"Google
Open in Colab Enterprise\n", " \n", " 		\n", " \n", " \"Vertex
Open in Vertex AI Workbench\n", " \n", " 		\n", " \n", " \"BigQuery
Open in BigQuery Studio\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", "
\n", "\n", "
\n", "\n", "

\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", "\n", "

" ] }, { "cell_type": "markdown", "metadata": { "id": "psD4Tv-Z3oKt" }, "source": [ "| | |\n", "|-|-|\n", "| Author | [Jessica Wang](https://github.com/wjess) |\n", "\n", "## Overview\n", "\n", "This notebook showcases how to use the `remote_custom_function` parameter in the Vertex AI Python SDK for Gen AI Evaluation Service to evaluate generative model tool use. Specifically, it demonstrates:\n", "* Defining custom python methods to validate tool calls, tool names, and tool parameters.\n", "* Executing these methods through custom remote functions against an evaluation dataset.\n", "* Leveraging `remote_custom_function` to mathematically calculate overall Precision, Recall, and F1 scores for tool usage.\n", "\n", "See also:\n", "* Learn more about [Vertex Gen AI Evaluation Service SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)." ] }, { "cell_type": "markdown", "metadata": { "id": "XLf9nKQ53oKu" }, "source": [ "## Getting Started\n", "\n", "### Install Vertex AI Python SDK for Gen AI Evaluation Service" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cCO6VFRB3oKu" }, "outputs": [], "source": [ "%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]" ] }, { "cell_type": "markdown", "metadata": { "id": "CqdvHNKR3oKv" }, "source": [ "### Authenticate your notebook environment (Colab only)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "kYyLF5Vq3oKv" }, "outputs": [], "source": [ "import sys\n", "\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "p83W-zR93oKv" }, "source": [ "### Set Google Cloud project information and initialize Vertex AI SDK" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8bRTXOY-3oKv" }, "outputs": [], "source": [ "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", "LOCATION = \"us-central1\" # @param {type:\"string\"}\n", "\n", "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", " raise ValueError(\"Please set your PROJECT_ID\")\n", "\n", "import vertexai\n", "\n", "vertexai.init(project=PROJECT_ID, location=LOCATION)\n", "client = vertexai.Client(project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "markdown", "metadata": { "id": "x9GuVowZ3oKw" }, "source": [ "### Import libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "dMmsCPdE3oKw" }, "outputs": [], "source": [ "import json\n", "\n", "import pandas as pd\n", "import vertexai\n", "from vertexai import types" ] }, { "cell_type": "markdown", "metadata": { "id": "ocrqPtj_3oKw" }, "source": [ "## Evaluate Tool use and Function Calling quality for Gemini\n", "\n", "### 1. Define Custom Metrics for Tool Use\n", "\n", "We will use the `remote_custom_function` parameter in the `Metric` class to define how our evaluation service should compute tool use matches remotely.\n", "\n", "#### Tool Call Valid Metric" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "42hMilzU3oKw" }, "outputs": [], "source": [ "tool_call_valid_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " # Negative example: no tool calls expected\n", " if not target.get(\"tool_calls\"):\n", " if not response.get(\"tool_calls\"):\n", " return 1.0 # True negative\n", " return 0.0 # False positive\n", "\n", " # Tool call expected but not provided\n", " if not response.get(\"tool_calls\"):\n", " return 0.0 # False negative\n", "\n", " target_call = target[\"tool_calls\"][0]\n", " prediction_call = response[\"tool_calls\"][0]\n", "\n", " if \"name\" in target_call and \"name\" in prediction_call:\n", " return 1.0\n", " return 0.0\n", "\"\"\"\n", "\n", "tool_call_valid_metric = types.Metric(\n", " name=\"tool_call_valid\", remote_custom_function=tool_call_valid_code\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PrmvQ2ki3oKw" }, "outputs": [], "source": [ "tool_name_match_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " if not target.get(\"tool_calls\"):\n", " return 1.0 if not response.get(\"tool_calls\") else 0.0\n", "\n", " if not response.get(\"tool_calls\"):\n", " return 0.0\n", "\n", " target_call = target[\"tool_calls\"][0]\n", " prediction_call = response[\"tool_calls\"][0]\n", "\n", " if target_call.get(\"name\") == prediction_call.get(\"name\"):\n", " return 1.0\n", " return 0.0\n", "\"\"\"\n", "\n", "tool_name_match_metric = types.Metric(\n", " name=\"tool_name_match\", remote_custom_function=tool_name_match_code\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "6w5g3yRn3oKw" }, "source": [ "#### Tool Parameter Match Metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jhw16VVr3oKw" }, "outputs": [], "source": [ "tool_parameter_key_match_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " if not target.get(\"tool_calls\"):\n", " return 1.0 if not response.get(\"tool_calls\") else 0.0\n", "\n", " if not response.get(\"tool_calls\"):\n", " return 0.0\n", "\n", " target_call = target[\"tool_calls\"][0]\n", " prediction_call = response[\"tool_calls\"][0]\n", "\n", " target_args = target_call.get(\"arguments\", {})\n", " prediction_args = prediction_call.get(\"arguments\", {})\n", "\n", " num_k_matches = sum(1 for k in target_args if k in prediction_args)\n", " num_unique_keys = len(set(target_args.keys()) | set(prediction_args.keys()))\n", "\n", " return float(num_k_matches) / float(num_unique_keys) if num_unique_keys > 0 else 1.0\n", "\"\"\"\n", "\n", "tool_parameter_kv_match_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " if not target.get(\"tool_calls\"):\n", " return 1.0 if not response.get(\"tool_calls\") else 0.0\n", "\n", " if not response.get(\"tool_calls\"):\n", " return 0.0\n", "\n", " target_call = target[\"tool_calls\"][0]\n", " prediction_call = response[\"tool_calls\"][0]\n", "\n", " target_args = target_call.get(\"arguments\", {})\n", " prediction_args = prediction_call.get(\"arguments\", {})\n", "\n", " num_kv_matches = 0\n", " for k, v in target_args.items():\n", " if k in prediction_args and prediction_args[k] == v:\n", " num_kv_matches += 1\n", "\n", " num_unique_keys = len(set(target_args.keys()) | set(prediction_args.keys()))\n", "\n", " return float(num_kv_matches) / float(num_unique_keys) if num_unique_keys > 0 else 1.0\n", "\"\"\"\n", "\n", "tool_parameter_key_match_metric = types.Metric(\n", " name=\"tool_parameter_key_match\",\n", " remote_custom_function=tool_parameter_key_match_code,\n", ")\n", "\n", "tool_parameter_kv_match_metric = types.Metric(\n", " name=\"tool_parameter_kv_match\", remote_custom_function=tool_parameter_kv_match_code\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "rpcB08iZ3oKx" }, "source": [ "### 2. Run Evaluation" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "KX0CkUa5Yprx" }, "outputs": [], "source": [ "# Create a sample evaluation dataset\n", "dataset = pd.DataFrame(\n", " {\n", " \"prompt\": [\n", " \"What is the weather in Tokyo?\",\n", " \"Tell me a joke.\",\n", " \"Book a flight to Paris tomorrow.\",\n", " ],\n", " \"reference\": [\n", " json.dumps(\n", " {\n", " \"content\": \"\",\n", " \"tool_calls\": [\n", " {\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n", " ],\n", " }\n", " ),\n", " json.dumps({\"content\": \"Here is a joke...\", \"tool_calls\": []}),\n", " json.dumps(\n", " {\n", " \"content\": \"\",\n", " \"tool_calls\": [\n", " {\n", " \"name\": \"book_flight\",\n", " \"arguments\": {\"destination\": \"Paris\", \"date\": \"tomorrow\"},\n", " }\n", " ],\n", " }\n", " ),\n", " ],\n", " \"response\": [\n", " json.dumps(\n", " {\n", " \"content\": \"\",\n", " \"tool_calls\": [\n", " {\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n", " ],\n", " }\n", " ),\n", " json.dumps({\"content\": \"Why did the chicken...\", \"tool_calls\": []}),\n", " json.dumps(\n", " {\n", " \"content\": \"\",\n", " \"tool_calls\": [\n", " {\"name\": \"book_flight\", \"arguments\": {\"destination\": \"Paris\"}}\n", " ],\n", " }\n", " ),\n", " ],\n", " }\n", ")\n", "\n", "eval_dataset = types.EvaluationDataset(eval_dataset_df=dataset)\n", "\n", "# Evaluate using our custom metrics\n", "eval_result = client.evals.evaluate(\n", " dataset=eval_dataset,\n", " metrics=[\n", " tool_call_valid_metric,\n", " tool_name_match_metric,\n", " tool_parameter_key_match_metric,\n", " tool_parameter_kv_match_metric,\n", " ],\n", ")\n", "\n", "eval_result.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "sG744cTa3oKx" }, "source": [ "### 3. Calculate Precision, Recall, and F1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Mdb_Mg7A3oKx" }, "outputs": [], "source": [ "# 1. Define Remote Custom Functions for Classification\n", "tool_call_tp_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n", " pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n", "\n", " if ref_has_tool and pred_has_tool:\n", " return 1.0\n", " return 0.0\n", "\"\"\"\n", "\n", "tool_call_fp_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n", " pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n", "\n", " if not ref_has_tool and pred_has_tool:\n", " return 1.0\n", " return 0.0\n", "\"\"\"\n", "\n", "tool_call_fn_code = \"\"\"\n", "import json\n", "\n", "def evaluate(instance: dict) -> float:\n", " ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n", " pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n", "\n", " target = json.loads(ref_text)\n", " response = json.loads(pred_text)\n", "\n", " ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n", " pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n", "\n", " if ref_has_tool and not pred_has_tool:\n", " return 1.0\n", " return 0.0\n", "\"\"\"\n", "\n", "metric_tp = types.Metric(name=\"tool_call_tp\", remote_custom_function=tool_call_tp_code)\n", "metric_fp = types.Metric(name=\"tool_call_fp\", remote_custom_function=tool_call_fp_code)\n", "metric_fn = types.Metric(name=\"tool_call_fn\", remote_custom_function=tool_call_fn_code)\n", "\n", "# 2. Evaluate the dataset remotely\n", "classification_eval_result = client.evals.evaluate(\n", " dataset=eval_dataset, metrics=[metric_tp, metric_fp, metric_fn]\n", ")\n", "\n", "# 3. Extract the Mean Scores\n", "mean_tp = next(\n", " m.mean_score\n", " for m in classification_eval_result.summary_metrics\n", " if m.metric_name == \"tool_call_tp\"\n", ")\n", "mean_fp = next(\n", " m.mean_score\n", " for m in classification_eval_result.summary_metrics\n", " if m.metric_name == \"tool_call_fp\"\n", ")\n", "mean_fn = next(\n", " m.mean_score\n", " for m in classification_eval_result.summary_metrics\n", " if m.metric_name == \"tool_call_fn\"\n", ")\n", "\n", "# 4. Calculate overall Precision, Recall, and F1 mathematically\n", "precision = mean_tp / (mean_tp + mean_fp) if (mean_tp + mean_fp) > 0 else 0.0\n", "recall = mean_tp / (mean_tp + mean_fn) if (mean_tp + mean_fn) > 0 else 0.0\n", "f1_score = (\n", " 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0\n", ")\n", "\n", "print(\"Overall Tool Usage Metrics (Calculated via Remote Custom Functions):\")\n", "print(f\"Precision: {precision}\")\n", "print(f\"Recall: {recall}\")\n", "print(f\"F1 Score: {f1_score}\")" ] } ], "metadata": { "colab": { "toc_visible": true, "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }