wehub-resource-sync
635 lines
22 KiB
Plaintext
635 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "OsXAs2gcIpbC"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WKiAD3qCCtOu"
|
|
},
|
|
"source": [
|
|
" # Evaluate Generative Model Tool Use"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "629723340bea"
|
|
},
|
|
"source": [
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaltask_approach%2Fevaluate_gemini_tool_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0922hY8RI11o"
|
|
},
|
|
"source": [
|
|
"| | |\n",
|
|
"|-|-|\n",
|
|
"|Author(s) | [Jason Dai](https://github.com/jsondai), [Bo Zheng](https://github.com/coolalexzb) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "fmNXwtYGCtOv"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"* Define an API function and a Tool for Gemini model, and evaluate the Gemini model tool use quality with *Vertex AI Python SDK for Gen AI Evaluation Service*.\n",
|
|
"\n",
|
|
"See also: \n",
|
|
"\n",
|
|
"- Learn more about [Vertex Gen AI Evaluation Service SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "58zvbIBKCtOv"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rHKucTJ9CtOv"
|
|
},
|
|
"source": [
|
|
"### Install Vertex AI Python SDK for Gen AI Evaluation Service"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6DnneQRilTFW"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation] google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nWC8oDQCCtOw"
|
|
},
|
|
"source": [
|
|
"### Restart runtime\n",
|
|
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
|
|
"\n",
|
|
"The restart might take a minute or longer. After it's restarted, continue to the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "dCkOvWuOCtOw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import IPython\n",
|
|
"\n",
|
|
"app = IPython.Application.instance()\n",
|
|
"app.kernel.do_shutdown(True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "h4pNZoFdCtOw"
|
|
},
|
|
"source": [
|
|
"<div class=\"alert alert-block alert-warning\">\n",
|
|
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
|
"</div>\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NdCE9EV5CtOw"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ZiX59CNjCtOx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "X4FOrb_sCtOx"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information and initialize Vertex AI SDK"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "G2LhRUspCtOx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
|
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" raise ValueError(\"Please set your PROJECT_ID\")\n",
|
|
"\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "1EvqYQT2CtOx"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "eHDPrnbQlTFX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"\n",
|
|
"import pandas as pd\n",
|
|
"from google import genai\n",
|
|
"from google.genai import types\n",
|
|
"from vertexai.evaluation import EvalTask\n",
|
|
"from vertexai.preview.evaluation import notebook_utils"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "RPBjzk96w9gV"
|
|
},
|
|
"source": [
|
|
"## Evaluate Tool use and Function Calling quality for Gemini"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "iY5hG6VMnS_s"
|
|
},
|
|
"source": [
|
|
"#### Tool evaluation metrics\n",
|
|
"\n",
|
|
"* `tool_call_valid`\n",
|
|
"* `tool_name_match`\n",
|
|
"* `tool_parameter_key_match`\n",
|
|
"* `tool_parameter_kv_match`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "vcZ2UikInZe0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_metrics = [\n",
|
|
" \"tool_call_valid\",\n",
|
|
" \"tool_name_match\",\n",
|
|
" \"tool_parameter_key_match\",\n",
|
|
" \"tool_parameter_kv_match\",\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dJgpakfIw9gc"
|
|
},
|
|
"source": [
|
|
"### 1. Evaluate a Bring-Your-Own-Prediction dataset\n",
|
|
"\n",
|
|
"Generative model's tool use quality can be evaluated if the eval dataset contains saved model tool call responses, and expected references."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d6Euv-SQw9gc"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = [\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Cinemark\", \"location\": \"Mountain View CA\", \"showtime\": \"5:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
"]\n",
|
|
"\n",
|
|
"reference = [\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Godzilla\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"9:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
" '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
|
|
"]\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame(\n",
|
|
" {\n",
|
|
" \"response\": response,\n",
|
|
" \"reference\": reference,\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "PyUj7kOnnNtH"
|
|
},
|
|
"source": [
|
|
"#### Define EvalTask"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tYl6A1zInObM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"experiment_name = \"eval-saved-llm-tool-use\" # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"tool_use_eval_task = EvalTask(\n",
|
|
" dataset=eval_dataset,\n",
|
|
" metrics=tool_metrics,\n",
|
|
" experiment=experiment_name,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "l8U4KVzAhsP2"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"run_id = notebook_utils.generate_uuid(8)\n",
|
|
"\n",
|
|
"experiment_run_name = f\"eval-{run_id}\"\n",
|
|
"\n",
|
|
"eval_result = tool_use_eval_task.evaluate(experiment_run_name=experiment_run_name)\n",
|
|
"notebook_utils.display_eval_result(\n",
|
|
" title=\"Tool Use Quality Evaluation Metrics\",\n",
|
|
" eval_result=eval_result,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3WXnsT2qo-OD"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"tool_use_eval_task.display_runs()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "oxow7GrIRgxo"
|
|
},
|
|
"source": [
|
|
"## 2. Tool Use and Function Calling with Gemini\n",
|
|
"\n",
|
|
"[Function Calling Documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Ew82QX0QWxYa"
|
|
},
|
|
"source": [
|
|
"### Define a function and tool\n",
|
|
"\n",
|
|
"Define an API specification and register the function in a tool with the latest version of [Vertex AI SDK for Python](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LiuafKieQgjF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from vertexai.generative_models import FunctionDeclaration, Tool\n",
|
|
"\n",
|
|
"book_tickets_func = FunctionDeclaration(\n",
|
|
" name=\"book_tickets\",\n",
|
|
" description=\"Book movie tickets\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"movie\": {\"type\": \"string\", \"description\": \"The title of the movie.\"},\n",
|
|
" \"theater\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The name of the movie theater.\",\n",
|
|
" },\n",
|
|
" \"location\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The location of the movie theater.\",\n",
|
|
" },\n",
|
|
" \"showtime\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The showtime of the movie in ISO 8601 format.\",\n",
|
|
" },\n",
|
|
" \"date\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The date of the movie in ISO 8601 format.\",\n",
|
|
" },\n",
|
|
" \"num_tix\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The integer number of tickets to book.\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" \"required\": [\n",
|
|
" \"movie\",\n",
|
|
" \"theater\",\n",
|
|
" \"location\",\n",
|
|
" \"showtime\",\n",
|
|
" \"date\",\n",
|
|
" \"num_tix\",\n",
|
|
" ],\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"book_tickets_tool = Tool(\n",
|
|
" function_declarations=[book_tickets_func],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "QQgreOFGXJNF"
|
|
},
|
|
"source": [
|
|
"### Generate a function call\n",
|
|
"\n",
|
|
"Prompt the Gemini model and include the tool that you defined."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "pNevmS2ngB0Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"I'd like to book 2 tickets for the movie \\\"Mission Impossible Dead Reckoning Part 1\\\"\n",
|
|
"at the Regal Edwards 14 theater in Mountain View, CA. The showtime is 7:30 PM on March 30th, 2024.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\n",
|
|
"\n",
|
|
"gemini_response = client.models.generate_content(\n",
|
|
" model=\"gemini-2.5-flash\",\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(tools=[book_tickets_tool]),\n",
|
|
")\n",
|
|
"\n",
|
|
"gemini_response.candidates[0].candidate.content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "lOAbbxFFXkdn"
|
|
},
|
|
"source": [
|
|
"### Unpack the Gemini response into a Python dictionary"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "x84XHpuWXmu6"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def unpack_response(response):\n",
|
|
" fc = response.candidates[0].content.parts[0].function_call\n",
|
|
" output = {\"content\": \"\", \"tool_calls\": [{\"name\": fc.name, \"arguments\": fc.args}]}\n",
|
|
" return json.dumps(output)\n",
|
|
"\n",
|
|
"\n",
|
|
"response = unpack_response(gemini_response)\n",
|
|
"response"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "fsLT0B75aJYz"
|
|
},
|
|
"source": [
|
|
"### Evaluate the Gemini's Function Call Response"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kgzK90mybqbK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"reference_str = json.dumps(\n",
|
|
" {\n",
|
|
" \"content\": \"\",\n",
|
|
" \"tool_calls\": [\n",
|
|
" {\n",
|
|
" \"name\": \"book_tickets\",\n",
|
|
" \"arguments\": {\n",
|
|
" \"movie\": \"Mission Impossible Dead Reckoning Part 1\",\n",
|
|
" \"theater\": \"Regal Edwards 14\",\n",
|
|
" \"location\": \"Mountain View CA\",\n",
|
|
" \"showtime\": \"7:30\",\n",
|
|
" \"date\": \"2024-03-30\",\n",
|
|
" \"num_tix\": \"2\",\n",
|
|
" },\n",
|
|
" }\n",
|
|
" ],\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"eval_dataset = pd.DataFrame({\"response\": [response], \"reference\": [reference_str]})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ttRycKaCg6GG"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Expected Tool Call Response\n",
|
|
"json.loads(eval_dataset.reference[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "k5SmLtFAdVCh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Actual Gemini Tool Call Response\n",
|
|
"json.loads(eval_dataset.response[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "r3MfX9QCZxHW"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"experiment_name = \"eval-gemini-model-function-call\" # @param {type:\"string\"}\n",
|
|
"run_id = notebook_utils.generate_uuid(8)\n",
|
|
"\n",
|
|
"eval_result = EvalTask(\n",
|
|
" dataset=eval_dataset,\n",
|
|
" metrics=tool_metrics,\n",
|
|
" experiment=experiment_name,\n",
|
|
").evaluate(experiment_run_name=f\"eval-{run_id}\")\n",
|
|
"\n",
|
|
"notebook_utils.display_eval_result(\n",
|
|
" title=\"Gemini Tool Use Quality Evaluation Metrics\",\n",
|
|
" eval_result=eval_result,\n",
|
|
")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "evaluate_gemini_tool_use.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|