wehub-resource-sync
606 lines
24 KiB
Plaintext
606 lines
24 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "w2Nnpp1a3oKr"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "mSdMTAZgF_M2"
|
|
},
|
|
"source": [
|
|
"# Evaluate Generative Model Tool Use with Custom Code Execution"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "b9ziZDnkF_j4"
|
|
},
|
|
"source": [
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_with_your_python_code.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/bigquery/v1/32px.svg\" alt=\"BigQuery Studio logo\"><br> Open in BigQuery Studio\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/generative-ai/logos/GitHub_Invertocat_Dark.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_with_your_python_code.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "psD4Tv-Z3oKt"
|
|
},
|
|
"source": [
|
|
"| | |\n",
|
|
"|-|-|\n",
|
|
"| Author | [Jessica Wang](https://github.com/wjess) |\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook showcases how to use the `remote_custom_function` parameter in the Vertex AI Python SDK for Gen AI Evaluation Service to evaluate generative model tool use. Specifically, it demonstrates:\n",
|
|
"* Defining custom python methods to validate tool calls, tool names, and tool parameters.\n",
|
|
"* Executing these methods through custom remote functions against an evaluation dataset.\n",
|
|
"* Leveraging `remote_custom_function` to mathematically calculate overall Precision, Recall, and F1 scores for tool usage.\n",
|
|
"\n",
|
|
"See also:\n",
|
|
"* Learn more about [Vertex Gen AI Evaluation Service SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XLf9nKQ53oKu"
|
|
},
|
|
"source": [
|
|
"## Getting Started\n",
|
|
"\n",
|
|
"### Install Vertex AI Python SDK for Gen AI Evaluation Service"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cCO6VFRB3oKu"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "CqdvHNKR3oKv"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kYyLF5Vq3oKv"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "p83W-zR93oKv"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information and initialize Vertex AI SDK"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8bRTXOY-3oKv"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
|
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" raise ValueError(\"Please set your PROJECT_ID\")\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
|
|
"client = vertexai.Client(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "x9GuVowZ3oKw"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "dMmsCPdE3oKw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"import vertexai\n",
|
|
"from vertexai import types"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ocrqPtj_3oKw"
|
|
},
|
|
"source": [
|
|
"## Evaluate Tool use and Function Calling quality for Gemini\n",
|
|
"\n",
|
|
"### 1. Define Custom Metrics for Tool Use\n",
|
|
"\n",
|
|
"We will use the `remote_custom_function` parameter in the `Metric` class to define how our evaluation service should compute tool use matches remotely.\n",
|
|
"\n",
|
|
"#### Tool Call Valid Metric"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "42hMilzU3oKw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_call_valid_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" # Negative example: no tool calls expected\n",
|
|
" if not target.get(\"tool_calls\"):\n",
|
|
" if not response.get(\"tool_calls\"):\n",
|
|
" return 1.0 # True negative\n",
|
|
" return 0.0 # False positive\n",
|
|
"\n",
|
|
" # Tool call expected but not provided\n",
|
|
" if not response.get(\"tool_calls\"):\n",
|
|
" return 0.0 # False negative\n",
|
|
"\n",
|
|
" target_call = target[\"tool_calls\"][0]\n",
|
|
" prediction_call = response[\"tool_calls\"][0]\n",
|
|
"\n",
|
|
" if \"name\" in target_call and \"name\" in prediction_call:\n",
|
|
" return 1.0\n",
|
|
" return 0.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_call_valid_metric = types.Metric(\n",
|
|
" name=\"tool_call_valid\", remote_custom_function=tool_call_valid_code\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "PrmvQ2ki3oKw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_name_match_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" if not target.get(\"tool_calls\"):\n",
|
|
" return 1.0 if not response.get(\"tool_calls\") else 0.0\n",
|
|
"\n",
|
|
" if not response.get(\"tool_calls\"):\n",
|
|
" return 0.0\n",
|
|
"\n",
|
|
" target_call = target[\"tool_calls\"][0]\n",
|
|
" prediction_call = response[\"tool_calls\"][0]\n",
|
|
"\n",
|
|
" if target_call.get(\"name\") == prediction_call.get(\"name\"):\n",
|
|
" return 1.0\n",
|
|
" return 0.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_name_match_metric = types.Metric(\n",
|
|
" name=\"tool_name_match\", remote_custom_function=tool_name_match_code\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6w5g3yRn3oKw"
|
|
},
|
|
"source": [
|
|
"#### Tool Parameter Match Metrics"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "jhw16VVr3oKw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_parameter_key_match_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" if not target.get(\"tool_calls\"):\n",
|
|
" return 1.0 if not response.get(\"tool_calls\") else 0.0\n",
|
|
"\n",
|
|
" if not response.get(\"tool_calls\"):\n",
|
|
" return 0.0\n",
|
|
"\n",
|
|
" target_call = target[\"tool_calls\"][0]\n",
|
|
" prediction_call = response[\"tool_calls\"][0]\n",
|
|
"\n",
|
|
" target_args = target_call.get(\"arguments\", {})\n",
|
|
" prediction_args = prediction_call.get(\"arguments\", {})\n",
|
|
"\n",
|
|
" num_k_matches = sum(1 for k in target_args if k in prediction_args)\n",
|
|
" num_unique_keys = len(set(target_args.keys()) | set(prediction_args.keys()))\n",
|
|
"\n",
|
|
" return float(num_k_matches) / float(num_unique_keys) if num_unique_keys > 0 else 1.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_parameter_kv_match_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" if not target.get(\"tool_calls\"):\n",
|
|
" return 1.0 if not response.get(\"tool_calls\") else 0.0\n",
|
|
"\n",
|
|
" if not response.get(\"tool_calls\"):\n",
|
|
" return 0.0\n",
|
|
"\n",
|
|
" target_call = target[\"tool_calls\"][0]\n",
|
|
" prediction_call = response[\"tool_calls\"][0]\n",
|
|
"\n",
|
|
" target_args = target_call.get(\"arguments\", {})\n",
|
|
" prediction_args = prediction_call.get(\"arguments\", {})\n",
|
|
"\n",
|
|
" num_kv_matches = 0\n",
|
|
" for k, v in target_args.items():\n",
|
|
" if k in prediction_args and prediction_args[k] == v:\n",
|
|
" num_kv_matches += 1\n",
|
|
"\n",
|
|
" num_unique_keys = len(set(target_args.keys()) | set(prediction_args.keys()))\n",
|
|
"\n",
|
|
" return float(num_kv_matches) / float(num_unique_keys) if num_unique_keys > 0 else 1.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_parameter_key_match_metric = types.Metric(\n",
|
|
" name=\"tool_parameter_key_match\",\n",
|
|
" remote_custom_function=tool_parameter_key_match_code,\n",
|
|
")\n",
|
|
"\n",
|
|
"tool_parameter_kv_match_metric = types.Metric(\n",
|
|
" name=\"tool_parameter_kv_match\", remote_custom_function=tool_parameter_kv_match_code\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rpcB08iZ3oKx"
|
|
},
|
|
"source": [
|
|
"### 2. Run Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "KX0CkUa5Yprx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create a sample evaluation dataset\n",
|
|
"dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"prompt\": [\n",
|
|
" \"What is the weather in Tokyo?\",\n",
|
|
" \"Tell me a joke.\",\n",
|
|
" \"Book a flight to Paris tomorrow.\",\n",
|
|
" ],\n",
|
|
" \"reference\": [\n",
|
|
" json.dumps(\n",
|
|
" {\n",
|
|
" \"content\": \"\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n",
|
|
" ],\n",
|
|
" }\n",
|
|
" ),\n",
|
|
" json.dumps({\"content\": \"Here is a joke...\", \"tool_calls\": []}),\n",
|
|
" json.dumps(\n",
|
|
" {\n",
|
|
" \"content\": \"\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\n",
|
|
" \"name\": \"book_flight\",\n",
|
|
" \"arguments\": {\"destination\": \"Paris\", \"date\": \"tomorrow\"},\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" }\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" \"response\": [\n",
|
|
" json.dumps(\n",
|
|
" {\n",
|
|
" \"content\": \"\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n",
|
|
" ],\n",
|
|
" }\n",
|
|
" ),\n",
|
|
" json.dumps({\"content\": \"Why did the chicken...\", \"tool_calls\": []}),\n",
|
|
" json.dumps(\n",
|
|
" {\n",
|
|
" \"content\": \"\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\"name\": \"book_flight\", \"arguments\": {\"destination\": \"Paris\"}}\n",
|
|
" ],\n",
|
|
" }\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_dataset = types.EvaluationDataset(eval_dataset_df=dataset)\n",
|
|
"\n",
|
|
"# Evaluate using our custom metrics\n",
|
|
"eval_result = client.evals.evaluate(\n",
|
|
" dataset=eval_dataset,\n",
|
|
" metrics=[\n",
|
|
" tool_call_valid_metric,\n",
|
|
" tool_name_match_metric,\n",
|
|
" tool_parameter_key_match_metric,\n",
|
|
" tool_parameter_kv_match_metric,\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_result.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "sG744cTa3oKx"
|
|
},
|
|
"source": [
|
|
"### 3. Calculate Precision, Recall, and F1"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Mdb_Mg7A3oKx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Define Remote Custom Functions for Classification\n",
|
|
"tool_call_tp_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n",
|
|
" pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n",
|
|
"\n",
|
|
" if ref_has_tool and pred_has_tool:\n",
|
|
" return 1.0\n",
|
|
" return 0.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_call_fp_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n",
|
|
" pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n",
|
|
"\n",
|
|
" if not ref_has_tool and pred_has_tool:\n",
|
|
" return 1.0\n",
|
|
" return 0.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"tool_call_fn_code = \"\"\"\n",
|
|
"import json\n",
|
|
"\n",
|
|
"def evaluate(instance: dict) -> float:\n",
|
|
" ref_text = instance['reference']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
" pred_text = instance['response']['contents']['gemini_contents'][0]['parts'][0]['text']\n",
|
|
"\n",
|
|
" target = json.loads(ref_text)\n",
|
|
" response = json.loads(pred_text)\n",
|
|
"\n",
|
|
" ref_has_tool = len(target.get(\"tool_calls\", [])) > 0\n",
|
|
" pred_has_tool = len(response.get(\"tool_calls\", [])) > 0\n",
|
|
"\n",
|
|
" if ref_has_tool and not pred_has_tool:\n",
|
|
" return 1.0\n",
|
|
" return 0.0\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"metric_tp = types.Metric(name=\"tool_call_tp\", remote_custom_function=tool_call_tp_code)\n",
|
|
"metric_fp = types.Metric(name=\"tool_call_fp\", remote_custom_function=tool_call_fp_code)\n",
|
|
"metric_fn = types.Metric(name=\"tool_call_fn\", remote_custom_function=tool_call_fn_code)\n",
|
|
"\n",
|
|
"# 2. Evaluate the dataset remotely\n",
|
|
"classification_eval_result = client.evals.evaluate(\n",
|
|
" dataset=eval_dataset, metrics=[metric_tp, metric_fp, metric_fn]\n",
|
|
")\n",
|
|
"\n",
|
|
"# 3. Extract the Mean Scores\n",
|
|
"mean_tp = next(\n",
|
|
" m.mean_score\n",
|
|
" for m in classification_eval_result.summary_metrics\n",
|
|
" if m.metric_name == \"tool_call_tp\"\n",
|
|
")\n",
|
|
"mean_fp = next(\n",
|
|
" m.mean_score\n",
|
|
" for m in classification_eval_result.summary_metrics\n",
|
|
" if m.metric_name == \"tool_call_fp\"\n",
|
|
")\n",
|
|
"mean_fn = next(\n",
|
|
" m.mean_score\n",
|
|
" for m in classification_eval_result.summary_metrics\n",
|
|
" if m.metric_name == \"tool_call_fn\"\n",
|
|
")\n",
|
|
"\n",
|
|
"# 4. Calculate overall Precision, Recall, and F1 mathematically\n",
|
|
"precision = mean_tp / (mean_tp + mean_fp) if (mean_tp + mean_fp) > 0 else 0.0\n",
|
|
"recall = mean_tp / (mean_tp + mean_fn) if (mean_tp + mean_fn) > 0 else 0.0\n",
|
|
"f1_score = (\n",
|
|
" 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\"Overall Tool Usage Metrics (Calculated via Remote Custom Functions):\")\n",
|
|
"print(f\"Precision: {precision}\")\n",
|
|
"print(f\"Recall: {recall}\")\n",
|
|
"print(f\"F1 Score: {f1_score}\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"toc_visible": true,
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
} |