wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gemini/function-calling/README.md
+28
View File
@@ -0,0 +1,28 @@
# Function Calling in Gemini
<!-- markdownlint-disable MD036 -->
**YouTube Video: AI + your code: Function Calling**
<!-- markdownlint-enable MD036 -->
<!-- markdownlint-disable MD033 -->
<a href="https://www.youtube.com/watch?v=NbAGbXr4DME&list=PLIivdWyY5sqLvGdVLJZh2EMax97_T-OIB" target="_blank">
<img src="https://img.youtube.com/vi/NbAGbXr4DME/maxresdefault.jpg" alt="AI + your code: Function Calling" width="500">
</a>
<!-- markdownlint-enable MD033 -->
[Function Calling in Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling)
lets developers create a description of a function in their code, then pass that
description to a language model in a request. The response from the model
includes the name of a function that matches the description and the arguments
to call it with.
## Sample notebooks and apps
| Description | Sample Name |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| Intro to Function Calling in Gemini | [intro_function_calling.ipynb](intro_function_calling.ipynb) |
| Working with Parallel Function Calls and Multiple Function Responses in Gemini | [parallel_function_calling.ipynb](parallel_function_calling.ipynb) |
| Working with Data Structures and Schemas in Gemini Function Calling | [function_calling_data_structures.ipynb](function_calling_data_structures.ipynb) |
| Forced Function Calling with Tool Configurations in Gemini | [forced_function_calling.ipynb](forced_function_calling.ipynb) |
gemini/function-calling/forced_function_calling.ipynb
+857
View File
@@ -0,0 +1,857 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Forced Function Calling with Tool Configurations in Gemini\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ffunction-calling%2Fforced_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/function-calling/forced_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/forced_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Kristopher Overholt](https://github.com/koverholt) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates the use of forced Function Calling in the Gemini model.\n",
"\n",
"### Gemini\n",
"\n",
"Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases.\n",
"\n",
"### Function Calling in Gemini\n",
"\n",
"[Function Calling in Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling) lets developers create a description of a function in their code, then pass that description to a language model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
"\n",
"### Forced Function Calling\n",
"\n",
"[Forced Function Calling in Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#tool-config) allows you to place constraints on how the model should use the function declarations that you provide it with. Using tool configurations, you can force the Gemini model to only predict function calls. You can also choose to provide the model with a full set of function declarations, but restrict its responses to a subset of these functions.\n",
"\n",
"## Objectives\n",
"\n",
"In this tutorial, you will learn how to use the Vertex AI SDK for Python to use different function calling modes, including forced function calling, via the Gemini model.\n",
"\n",
"You will complete the following tasks:\n",
"\n",
"- Read through an overview of forced function calling and when to use it\n",
"- Use the default function calling behavior in `AUTO` mode\n",
"- Enable forced function calling using the `ANY` mode\n",
"- Disable function calling using the `NONE` mode"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai arxiv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"global\"\n",
"\n",
"from google import genai\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "23a24049e443"
},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "cc6278ff6e55"
},
"outputs": [],
"source": [
"import arxiv\n",
"from IPython.display import Markdown, display\n",
"from google.genai.types import (\n",
" Content,\n",
" FunctionCallingConfig,\n",
" FunctionCallingConfigMode,\n",
" FunctionDeclaration,\n",
" GenerateContentConfig,\n",
" Part,\n",
" Schema,\n",
" Tool,\n",
" ToolConfig,\n",
" Type,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d091edc81048"
},
"source": [
"### Choose a model\n",
"\n",
"For more information about all AI models and APIs on Vertex AI, see [Google Models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models) and [Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models).\n",
"\n",
"Refer to the [Gemini Function Calling documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling) for more information on which models and model versions support forced function calling and tool configurations."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "c8922eacdb2d"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0505b45fa754"
},
"source": [
"## Define a function to search for scientific papers in arXiv\n",
"\n",
"Since this notebook focuses on using different tool configurations and modes in Gemini Function Calling, you'll define a function declaration to use throughout the examples.\n",
"\n",
"The purpose of this function is to extract a parameter to send as a query to search for relevant papers in [arXiv](https://arxiv.org/).\n",
"\n",
"arXiv is an open-access repository of electronic preprints and postprints that consists of scientific papers in various fields."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "72440503b923"
},
"outputs": [],
"source": [
"search_arxiv = FunctionDeclaration(\n",
" name=\"search_arxiv\",\n",
" description=\"Search for articles and publications in arXiv\",\n",
" parameters=Schema(\n",
" type=Type.OBJECT,\n",
" properties={\n",
" \"query\": Schema(\n",
" type=Type.STRING, description=\"Query to search for in arXiv\"\n",
" )\n",
" },\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "17f31eb30c46"
},
"source": [
"Define a tool that wraps the above function:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "e4d7150e5b03"
},
"outputs": [],
"source": [
"search_tool = Tool(\n",
" function_declarations=[\n",
" search_arxiv,\n",
" ],\n",
")\n",
"config = GenerateContentConfig(temperature=0, tools=[search_tool])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6a12e1e637c7"
},
"source": [
"You'll use this function declaration and tool throughout the next few sections of the notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3a043e118735"
},
"source": [
"## Overview of Forced Function Calling in Gemini"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0d17a45fcea5"
},
"source": [
"The default behavior for Function Calling allows the Gemini model to decide whether to predict a function call or a natural language response. This is because the default Function Calling mode in Gemini is set to `AUTO`.\n",
"\n",
"In most cases this is the desired behavior when you want the Gemini model to use information from the prompt to determine if it should call a function, and which function it should call. However, you might have specific use cases where you want to **force** the Gemini model to call a function (or a set of functions) in a given model generation request.\n",
"\n",
"Tool configurations in the Gemini API allow you to specify different Function Calling modes in Gemini. Refer to the [Gemini Function Calling documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling) for more information on forced function calling and tool configurations.\n",
"\n",
"The following code example for `tool_config` shows various modes that you can set and pass to the Gemini model either globally when you initialize the model or for a given model generation request:\n",
"\n",
"```py\n",
"tool_config = ToolConfig(\n",
" function_calling_config=ToolConfig.FunctionCallingConfig(\n",
" mode=ToolConfig.FunctionCallingConfig.Mode.AUTO, # The default model behavior. The model decides whether to predict a function call or a natural language response.\n",
" mode=ToolConfig.FunctionCallingConfig.Mode.ANY, # ANY mode forces the model to predict a function call from a subset of function names.\n",
" mode=ToolConfig.FunctionCallingConfig.Mode.NONE, # NONE mode instructs the model to not predict function calls. Equivalent to a model request without any function declarations.\n",
" allowed_function_names=[\n",
" \"function_to_call\"\n",
" ], # Allowed functions to call when mode is ANY, if empty any one of the provided functions will be called.\n",
" )\n",
")\n",
"```\n",
"\n",
"Using these Function Calling modes, you can configure the model to behave in one of the following ways:\n",
"\n",
"- Allow the model to choose whether to predict a function call or natural language response (`AUTO` mode)\n",
"- Force the model to predict a function call on one function or a set of functions (`ANY` mode)\n",
"- Disable function calling and return a natural language response as if no functions or tools were defined (`NONE` mode)\n",
"\n",
"In the following sections, you'll walk through examples and sample code for each Function Calling mode."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ab06d57a134"
},
"source": [
"## Example: Default Function Calling mode (`AUTO`)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3a3b8124f0fc"
},
"source": [
"In this example, you'll specify the function calling mode as `AUTO`. Note that `AUTO` mode is the default model behavior, therefore the Gemini model will also use this mode when there is no `tool_config` specified:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "50b1009e342b"
},
"outputs": [],
"source": [
"config.tool_config = ToolConfig(\n",
" function_calling_config=FunctionCallingConfig(\n",
" mode=FunctionCallingConfigMode.AUTO, # The default model behavior. The model decides whether to predict a function call or a natural language response.\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a1c7e020ee38"
},
"source": [
"Ask a question about a topic related to publications in arXiv and include the `tool_config` kwarg. Note that you can also set the `tool_config` kwarg globally in the model rather than with every request to generate content:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "0c22b8e9a99a"
},
"outputs": [
{
"data": {
"text/markdown": [
"**Reinforcement Learning (RL)** is a type of machine learning where an \"agent\" learns to make decisions by interacting with an environment. Instead of being told the correct answer, the agent performs actions and receives feedback in the form of rewards or penalties. Its goal is to develop a strategy, called a policy, that maximizes the total cumulative reward over time through trial and error.\n",
"\n",
"Here are four essential papers from arXiv to help you learn more, ranging from introductory overviews to foundational algorithms:\n",
"\n",
"### 1. The Best Starting Point (Survey)\n",
"**[Deep Reinforcement Learning: An Overview](https://arxiv.org/abs/1701.07274)**\n",
"* **Authors:** Yuxi Li\n",
"* **Why read it:** This is a comprehensive technical summary of the field. It covers the core concepts, the history of RL, and the various sub-fields like Deep Q-Networks (DQN), Policy Gradients, and Multi-Agent RL. It is an excellent \"map\" of the landscape.\n",
"\n",
"### 2. The Modern Foundation\n",
"**[Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602)**\n",
"* **Authors:** Volodymyr Mnih, et al. (DeepMind)\n",
"* **Why read it:** This is the landmark paper that started the \"Deep RL\" revolution. It describes how DeepMind trained an AI to play Atari games using only the pixels on the screen as input, outperforming humans in several games. It introduces the **Deep Q-Network (DQN)**.\n",
"\n",
"### 3. A Practical Deep RL Review\n",
"**[A Brief Survey of Deep Reinforcement Learning](https://arxiv.org/abs/1708.05866)**\n",
"* **Authors:** Kai Arulkumaran, et al.\n",
"* **Why read it:** This paper is slightly more concise than the first survey and focuses specifically on the \"Deep\" aspect of RL. It explains the transition from \"classical\" RL to using neural networks to solve complex problems.\n",
"\n",
"### 4. The Industry Standard Algorithm\n",
"**[Proximal Policy Optimization Algorithms (PPO)](https://arxiv.org/abs/1707.06347)**\n",
"* **Authors:** John Schulman, et al. (OpenAI)\n",
"* **Why read it:** PPO is currently one of the most popular and widely used RL algorithms because it is stable and relatively easy to tune. If you want to understand how modern RL agents (like those used in robotics or game AI) are actually trained today, this is the paper to read.\n",
"\n",
"***\n",
"\n",
"**Pro Tip:** If you are a complete beginner, you should also look into the textbook **\"Reinforcement Learning: An Introduction\"** by Richard Sutton and Andrew Barto. While not an arXiv paper, it is considered the \"Bible\" of the field and is available for free online."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"prompt = \"Explain reinforcement learning in a few sentences and give me papers from arXiv to learn more\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=config,\n",
")\n",
"\n",
"display(Markdown(response.candidates[0].content.parts[0].text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "63ee50104549"
},
"source": [
"The response includes a natural language summary to the prompt. However, you were probably hoping to make a function call along the way to search for actual papers in arXiv and return them to the end user!\n",
"\n",
"We'll make that happen in the next section by using the forced function calling mode."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ec647eed5130"
},
"source": [
"## Example: Using Forced Function Calling mode (`ANY`)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f76d1bb4b4c9"
},
"source": [
"In this example, you'll set the tool configuration to `ANY`, and specify one or more `allowed_function_names` that will force Gemini to make a function call against a function or subset of functions:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "992c70f9a9b8"
},
"outputs": [],
"source": [
"config.tool_config = ToolConfig(\n",
" function_calling_config=FunctionCallingConfig(\n",
" mode=FunctionCallingConfigMode.ANY, # ANY mode forces the model to predict a function call from a subset of function names.\n",
" allowed_function_names=[\n",
" \"search_arxiv\"\n",
" ], # Allowed functions to call when mode is ANY, if empty any one of the provided functions will be called.\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "60a33d7fff56"
},
"source": [
"Now you can ask the same question publications in arXiv with our newly defined `tool_config` that is set to `ANY` function calling mode, which will force the Gemini model to call our search function."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "753d7d208f77"
},
"outputs": [
{
"data": {
"text/plain": [
"FunctionCall(\n",
" args={\n",
" 'query': 'Deep Reinforcement Learning survey overview introduction'\n",
" },\n",
" name='search_arxiv'\n",
")"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"Explain reinforcement learning in a few sentences and give me papers from arXiv to learn more\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=config,\n",
")\n",
"\n",
"model_response_content = response.candidates[0].content # Save for thought signature\n",
"response.function_calls[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4f4715146eca"
},
"source": [
"You can extract the parameters from the model response so that we can use them to make an API call to search papers in arXiv:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "10754b0a94ba"
},
"outputs": [
{
"data": {
"text/plain": [
"{'query': 'Deep Reinforcement Learning survey overview introduction'}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"params = {}\n",
"for key, value in response.function_calls[0].args.items():\n",
" params[key] = value\n",
"params"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "ffc6cc2033f3"
},
"outputs": [],
"source": [
"if response.function_calls[0].name == \"search_arxiv\":\n",
" arxiv_client = arxiv.Client()\n",
"\n",
" search = arxiv.Search(\n",
" query=params[\"query\"], max_results=3, sort_by=arxiv.SortCriterion.Relevance\n",
" )\n",
"\n",
" results = arxiv_client.results(search)\n",
" results = str(list(results))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1f4e8482dcc2"
},
"source": [
"Print a sample of the API response from arXiv:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "d9ef5bebc0eb"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"\"[arxiv.Result(entry_id='http://arxiv.org/abs/1701.07274v6', updated=datetime.datetime(2018, 11, 26, 4, 56, 31, tzinfo=datetime.timezone.utc), published=datetime.datetime(2017, 1, 25, 11, 52, 11, tzinfo=datetime.timezone.utc), title='Deep Reinforcement Learning: An Overview', authors=[arxiv.Result.Author('Yuxi Li')], summary='We give an overview of recent exciting achievements of deep reinforcement learning (RL). We discuss six core elements, six important mechanisms, and twelve applications. We start with background of machine learning, deep learning and reinforcement learning. Next we discuss core RL elements, including value function, in particular, Deep Q-Network (DQN), policy, reward, model, planning, and exploration. After that, we discuss important mechanisms for RL, including attention and memory, unsupervised learning, transfer learning, multi-agent RL, hierarchical RL, and learning to learn. Then we discuss various applications of RL, including games, in particular, AlphaGo, r\""
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results[:1000]"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "fae1aaee9fe3"
},
"outputs": [
{
"data": {
"text/markdown": [
"**Reinforcement Learning (RL)** is a field of machine learning where an autonomous agent learns to make decisions by interacting with an environment. Instead of being told the \"correct\" answer, the agent receives feedback in the form of **rewards or penalties** based on its actions. Its ultimate goal is to develop a strategy, called a policy, that maximizes the total cumulative reward it receives over time.\n",
"\n",
"Here are four essential papers from arXiv to help you learn more, ranging from broad overviews to foundational algorithms:\n",
"\n",
"### 1. The Comprehensive Overview\n",
"* **Title:** [Deep Reinforcement Learning: An Overview](https://arxiv.org/abs/1701.07274)\n",
"* **Author:** Yuxi Li\n",
"* **Why read it:** This is an excellent starting point. It provides a massive bird's-eye view of the field, covering core elements (value functions, policies, rewards), key mechanisms (attention, memory, transfer learning), and real-world applications like robotics and games.\n",
"\n",
"### 2. The Modern Survey\n",
"* **Title:** [A Brief Survey of Deep Reinforcement Learning](https://arxiv.org/abs/1708.05866)\n",
"* **Authors:** Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath\n",
"* **Why read it:** This paper bridges the gap between traditional RL and \"Deep\" RL (using neural networks). It explains central algorithms like Deep Q-Networks (DQN) and Actor-Critic methods in a way that is accessible to those familiar with general machine learning.\n",
"\n",
"### 3. The Breakthrough (DQN)\n",
"* **Title:** [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602)\n",
"* **Authors:** Volodymyr Mnih, et al. (DeepMind)\n",
"* **Why read it:** This is a historic paper. It describes the first deep learning model to successfully learn control policies directly from high-dimensional sensory input (pixels) using reinforcement learning, famously outperforming humans at several Atari 2600 games.\n",
"\n",
"### 4. The Industry Standard (PPO)\n",
"* **Title:** [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)\n",
"* **Authors:** John Schulman, et al. (OpenAI)\n",
"* **Why read it:** PPO has become one of the most popular \"default\" algorithms in RL because it is easier to tune and more stable than previous methods. If you want to understand how modern AI (like ChatGPT's fine-tuning process) is trained, this is a key algorithm to know.\n",
"\n",
"### Bonus Tip for Learning\n",
"If you are new to the math behind RL, the most recommended textbook is **\"Reinforcement Learning: An Introduction\"** by Richard Sutton and Andrew Barto. While not an arXiv paper, it is the \"bible\" of the field and is [available for free online](http://incompleteideas.net/book/the-book-2nd.html)."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"config.tool_config = None\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[\n",
" prompt,\n",
" model_response_content, # Function call with thought signature\n",
" Content(\n",
" role=\"tool\",\n",
" parts=[\n",
" Part.from_function_response(\n",
" name=\"search_arxiv\",\n",
" response={\n",
" \"content\": results,\n",
" },\n",
" ),\n",
" ],\n",
" ),\n",
" ],\n",
" config=config,\n",
")\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a97b13c2a45b"
},
"source": [
"In this case, the natural language response contains information about relevant papers based on our function call to the arXiv API."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a64301e40297"
},
"source": [
"## Example: Disabling Function Calling (`NONE`)\n",
"\n",
"In this example, you'll set the tool configuration to `NONE`, which will instruct the Gemini model to behave as if no tools or functions were defined."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "cc0c43b9a696"
},
"outputs": [],
"source": [
"config.tool_config = ToolConfig(\n",
" function_calling_config=FunctionCallingConfig(\n",
" mode=FunctionCallingConfigMode.NONE, # NONE mode instructs the model to not predict function calls. Equivalent to a model request without any function declarations.\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "7678f426d195"
},
"outputs": [
{
"data": {
"text/markdown": [
"**Reinforcement Learning (RL)** is a branch of machine learning where an autonomous agent learns to make decisions by interacting with an environment. Instead of being told the \"correct\" answer, the agent performs actions and receives feedback in the form of rewards or penalties. Its ultimate goal is to develop a strategy, called a policy, that maximizes the total cumulative reward over time.\n",
"\n",
"Here are four essential papers from arXiv to help you learn more, ranging from foundational overviews to modern breakthroughs:\n",
"\n",
"### 1. The Comprehensive Overview\n",
"**[Deep Reinforcement Learning: An Overview](https://arxiv.org/abs/1701.07274)**\n",
"* **Author:** Yuxi Li\n",
"* **Why read it:** This is an excellent starting point. It provides a broad survey of the field, explaining the transition from \"classical\" RL to \"Deep\" RL, and covers major algorithms and applications like games, robotics, and healthcare.\n",
"\n",
"### 2. The Deep RL Breakthrough\n",
"**[Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602)**\n",
"* **Authors:** Volodymyr Mnih et al. (DeepMind)\n",
"* **Why read it:** This is the seminal paper that introduced the **Deep Q-Network (DQN)**. It demonstrated for the first time that an AI could learn to play complex video games directly from raw pixels, outperforming human experts in many cases.\n",
"\n",
"### 3. The Modern Standard\n",
"**[Proximal Policy Optimization Algorithms (PPO)](https://arxiv.org/abs/1707.06347)**\n",
"* **Authors:** John Schulman et al. (OpenAI)\n",
"* **Why read it:** PPO has become one of the most popular \"default\" algorithms in RL because it is more stable and easier to tune than previous methods. If you want to understand how modern agents (like those used in robotics or even fine-tuning LLMs) are trained, this is a key paper.\n",
"\n",
"### 4. The Future: Learning from Data\n",
"**[Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://arxiv.org/abs/2005.00451)**\n",
"* **Authors:** Sergey Levine et al.\n",
"* **Why read it:** Traditional RL requires constant interaction with an environment, which can be dangerous or expensive (e.g., in self-driving cars). This paper explains \"Offline RL,\" where agents learn from pre-existing datasets without needing to interact with the world in real-time.\n",
"\n",
"***\n",
"\n",
"**Pro Tip:** If you are a complete beginner, you should also look for the book *Reinforcement Learning: An Introduction* by **Sutton and Barto**. While not an arXiv paper, it is considered the \"Bible\" of the field and is available for free online."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"prompt = \"Explain reinforcement learning in a few sentences and give me papers from arXiv to learn more\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=config,\n",
")\n",
"\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7dc4ee73d814"
},
"source": [
"Note that the model returned a natural language response without making a function call, even though the prompt asked for arXiv papers. In `NONE` mode, the model generates its response entirely from its training data rather than calling the `search_arxiv` function to retrieve real-time results."
]
}
],
"metadata": {
"colab": {
"name": "forced_function_calling.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/function-calling/function_calling_data_structures.ipynb
+677
View File
@@ -0,0 +1,677 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Working with Data Structures and Schemas in Gemini Function Calling\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ffunction-calling%2Ffunction_calling_data_structures.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZNJC1SkrsJY3"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Kristopher Overholt](https://github.com/koverholt) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini models.\n",
"\n",
"[Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) in Gemini lets you create a description of a function in your code, then pass that description to a language model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
"\n",
"In this tutorial, you'll learn how to work with various data structures within Gemini Function Calling, including:\n",
" \n",
"- Single parameter\n",
"- Multiple parameters\n",
"- Lists of parameters\n",
"- Nested parameters and data structures"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Getting Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"global\"\n",
"\n",
"from google import genai\n",
"\n",
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXHfaVS66_01"
},
"source": [
"## Code Examples\n",
"\n",
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "lslYAvw37JGQ"
},
"outputs": [],
"source": [
"from google.genai.types import FunctionDeclaration, GenerateContentConfig, Tool"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Bfy0nopcsJY5"
},
"source": [
"### Initialize model\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "hvQy2EObsJY5"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j3KHAr6BsJY6"
},
"source": [
"### Example: Single parameter\n",
"\n",
"Let's say that you want to extract a location from a prompt to help a user navigate to their desired destination.\n",
"\n",
"You can build out a simple schema for a function that takes a single parameter as an input:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "ElbXcrGWsJY6"
},
"outputs": [],
"source": [
"get_destination = FunctionDeclaration(\n",
" name=\"get_destination\",\n",
" description=\"Get directions to a destination\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"destination\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Destination that the user wants to go to\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"destination_tool = Tool(\n",
" function_declarations=[get_destination],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T_u_cPW7sJY6"
},
"source": [
"Now you can send a prompt with a destination, and the model will return structured data with a single key/value pair:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "aD4UJ6BcsJY6"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'destination': 'Paris'\n",
" },\n",
" name='get_destination'\n",
" )]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"I'd like to travel to Paris\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=GenerateContentConfig(temperature=0, tools=[destination_tool]),\n",
")\n",
"\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q7jimk65sJY7"
},
"source": [
"### Example: Multiple parameters\n",
"\n",
"What if you want the function call to return more than one parameter?\n",
"\n",
"You can build out a simple schema for a function that takes multiple parameters as an input:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "OTaA258isJY7"
},
"outputs": [],
"source": [
"get_destination_params = FunctionDeclaration(\n",
" name=\"get_destination_params\",\n",
" description=\"Get directions to a destination\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"destination\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Destination that the user wants to go to\",\n",
" },\n",
" \"mode_of_transportation\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Mode of transportation to use\",\n",
" },\n",
" \"departure_time\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Time that the user will leave for the destination\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"destination_tool = Tool(\n",
" function_declarations=[get_destination_params],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xS-qg3udsJY7"
},
"source": [
"Now you can send a prompt with a destination, and the model will return structured data with multiple key/value pairs:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "2gm8YuoesJY7"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'departure_time': '9:00 am',\n",
" 'destination': 'Paris',\n",
" 'mode_of_transportation': 'train'\n",
" },\n",
" name='get_destination_params'\n",
" )]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"I'd like to travel to Paris by train and leave at 9:00 am\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=GenerateContentConfig(temperature=0, tools=[destination_tool]),\n",
")\n",
"\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y-ZP3TKrsJY7"
},
"source": [
"### Example: Lists of parameters\n",
"\n",
"What if you want the function call to return an array or list of parameters within an object?\n",
"\n",
"For example, you might want to call an API to get the geocoded coordinates of several different locations within a single prompt.\n",
"\n",
"In that case, you can build out a schema for a function that takes an array as an input:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "jGMeCeftsJY7"
},
"outputs": [],
"source": [
"get_multiple_location_coordinates = FunctionDeclaration(\n",
" name=\"get_location_coordinates\",\n",
" description=\"Get coordinates of multiple locations\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"locations\": {\n",
" \"type\": \"array\",\n",
" \"description\": \"A list of locations\",\n",
" \"items\": {\n",
" \"description\": \"Components of the location\",\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"point_of_interest\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Name or type of point of interest\",\n",
" },\n",
" \"city\": {\"type\": \"string\", \"description\": \"City\"},\n",
" \"country\": {\"type\": \"string\", \"description\": \"Country\"},\n",
" },\n",
" \"required\": [\n",
" \"point_of_interest\",\n",
" \"city\",\n",
" \"country\",\n",
" ],\n",
" },\n",
" }\n",
" },\n",
" },\n",
")\n",
"\n",
"geocoding_tool = Tool(\n",
" function_declarations=[get_multiple_location_coordinates],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p2X7KXuYsJY8"
},
"source": [
"Now we'll send a prompt with a few different locations and points of interest:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "8YtBm7-XsJY8"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'locations': [\n",
" {\n",
" 'city': 'Paris',\n",
" 'country': 'France',\n",
" 'point_of_interest': 'Eiffel Tower'\n",
" },\n",
" {\n",
" 'city': 'New York',\n",
" 'country': 'USA',\n",
" 'point_of_interest': 'Statue of Liberty'\n",
" },\n",
" {\n",
" 'city': 'Port Douglas',\n",
" 'country': 'Australia',\n",
" 'point_of_interest': 'Port Douglas'\n",
" },\n",
" ]\n",
" },\n",
" name='get_location_coordinates'\n",
" )]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"\n",
" I'd like to get the coordinates for\n",
" the Eiffel tower in Paris,\n",
" the statue of liberty in New York,\n",
" and Port Douglas near the Great Barrier Reef.\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=GenerateContentConfig(temperature=0, tools=[geocoding_tool]),\n",
")\n",
"\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D8giW_8OsJY8"
},
"source": [
"Note that the generative model populated values for all of the parameters for a given location since all three parameters are required."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jEHIc8x6sJY8"
},
"source": [
"### Example: Nested parameters and data structures\n",
"\n",
"What if you want the function call to include nested parameters or other complex data structures?\n",
"\n",
"You might want to send a command to create a product listing based on a few sentences that include the product details.\n",
"\n",
"In that case, you can build out a schema for a function that takes nested data structures as an input:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "27PuX0-fsJY8"
},
"outputs": [],
"source": [
"create_product_listing = FunctionDeclaration(\n",
" name=\"create_product_listing\",\n",
" description=\"Create a product listing using the details provided by the user.\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"product\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"name\": {\"type\": \"string\"},\n",
" \"price\": {\"type\": \"number\"},\n",
" \"category\": {\"type\": \"string\"},\n",
" \"description\": {\"type\": \"string\"},\n",
" },\n",
" }\n",
" },\n",
" },\n",
")\n",
"\n",
"product_listing_tool = Tool(\n",
" function_declarations=[create_product_listing],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k8sKF69fsJY8"
},
"source": [
"Now we'll send a prompt with product details and the model will extract the nested parameters:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "Pt-i-srosJY8"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'product': {\n",
" 'category': 'Electronics',\n",
" 'description': 'These headphones create a distraction-free environment.',\n",
" 'name': 'Noise-canceling headphones',\n",
" 'price': 149.99\n",
" }\n",
" },\n",
" name='create_product_listing'\n",
" )]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"Create a listing for noise-canceling headphones for $149.99.\n",
"These headphones create a distraction-free environment.\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=GenerateContentConfig(temperature=0, tools=[product_listing_tool]),\n",
")\n",
"\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GpDvGrmtsJY8"
},
"source": [
"And you're done! You successfully generated various types of data structures, including a single parameter, multiple parameters, a list of parameters, and nested parameters. Try another notebook to continue exploring other functionality in the Gemini API!"
]
}
],
"metadata": {
"colab": {
"name": "function_calling_data_structures.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/function-calling/intro_function_calling.ipynb
+953
View File
@@ -0,0 +1,953 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Intro to Function Calling with the Gemini API & Python SDK\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ffunction-calling%2Fintro_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/function-calling/intro_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://goo.gle/4jeQxBO\">\n",
" <img width=\"32px\" src=\"https://cdn.qwiklabs.com/assets/gcp_cloud-e3a77215f0b8bfa9b3f611c0d2208c7e8708ed31.svg\" alt=\"Google Cloud logo\"><br> Open in Cloud Skills Boost\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/intro_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84e7e432e6ff"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Kristopher Overholt](https://github.com/koverholt) |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"**YouTube Video: AI + your code: Function Calling**\n",
"\n",
"<a href=\"https://www.youtube.com/watch?v=NbAGbXr4DME&list=PLIivdWyY5sqLvGdVLJZh2EMax97_T-OIB\" target=\"_blank\">\n",
" <img src=\"https://img.youtube.com/vi/NbAGbXr4DME/maxresdefault.jpg\" alt=\"AI + your code: Function Calling\" width=\"500\">\n",
"</a>\n",
"\n",
"### Gemini\n",
"\n",
"Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases.\n",
"\n",
"### Calling functions from Gemini\n",
"\n",
"[Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) in Gemini lets developers create a description of a function in their code, then pass that description to a language model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
"\n",
"### Why function calling?\n",
"\n",
"Imagine asking someone to write down important information without giving them a form or any guidelines on the structure. You might get a beautifully crafted paragraph, but extracting specific details like names, dates, or numbers would be tedious! Similarly, trying to get consistent structured data from a generative text model without function calling can be frustrating. You're stuck explicitly prompting for things like JSON output, often with inconsistent and frustrating results.\n",
"\n",
"This is where Gemini Function Calling comes in. Instead of hoping for the best in a freeform text response from a generative model, you can define clear functions with specific parameters and data types. These function declarations act as structured guidelines, guiding the Gemini model to structure its output in a predictable and usable way. No more parsing text responses for important information!\n",
"\n",
"Think of it like teaching Gemini to speak the language of your applications. Need to retrieve information from a database? Define a `search_db` function with parameters for search terms. Want to integrate with a weather API? Create a `get_weather` function that takes a location as input. Function calling bridges the gap between human language and the structured data needed to interact with external systems."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DrkcqHrrwMAo"
},
"source": [
"### Objectives\n",
"\n",
"In this tutorial, you will learn how to use the Gemini API in Vertex AI with the Vertex AI SDK for Python to make function calls via the Gemini 3 Flash (`gemini-3.5-flash`) model.\n",
"\n",
"You will complete the following tasks:\n",
"\n",
"- Install the Google Gen AI SDK for Python\n",
"- Use the Gemini API in Vertex AI to interact with the Gemini model:\n",
"- Use Function Calling in a chat session to answer user's questions about products in the Google Store\n",
"- Use Function Calling to geocode addresses with a maps API\n",
"- Use Function Calling for entity extraction on raw logging data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C9nEPojogw-g"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"- Vertex AI\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Getting Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"global\"\n",
"\n",
"from google import genai\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "92e02c3e0375"
},
"source": [
"## Code Examples"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5671450907ec"
},
"source": [
"### Choose a model\n",
"\n",
"For more information about all AI models and APIs on Vertex AI, see [Google Models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models) and [Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "41e499d90618"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXHfaVS66_01"
},
"source": [
"### Import libraries\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "lslYAvw37JGQ"
},
"outputs": [],
"source": [
"import requests\n",
"from IPython.display import Markdown, display\n",
"from google.genai.types import FunctionDeclaration, GenerateContentConfig, Part, Tool"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "28f36bd968b4"
},
"source": [
"### Chat example: Using Function Calling in a chat session to answer user's questions about the Google Store"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2d28287bde87"
},
"source": [
"In this example, you'll use Function Calling along with the chat modality in the Gemini model to help customers get information about products in the Google Store.\n",
"\n",
"You'll start by defining three functions: one to get product information, another to get the location of the closest stores, and one more to place an order:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "3d4ed7ccc094"
},
"outputs": [],
"source": [
"get_product_info = FunctionDeclaration(\n",
" name=\"get_product_info\",\n",
" description=\"Get the stock amount and identifier for a given product\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"product_name\": {\"type\": \"string\", \"description\": \"Product name\"}\n",
" },\n",
" },\n",
")\n",
"\n",
"get_store_location = FunctionDeclaration(\n",
" name=\"get_store_location\",\n",
" description=\"Get the location of the closest store\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"Location\"}},\n",
" },\n",
")\n",
"\n",
"place_order = FunctionDeclaration(\n",
" name=\"place_order\",\n",
" description=\"Place an order\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"product\": {\"type\": \"string\", \"description\": \"Product name\"},\n",
" \"address\": {\"type\": \"string\", \"description\": \"Shipping address\"},\n",
" },\n",
" },\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e7d7319febd8"
},
"source": [
"Note that function parameters are specified as a Python dictionary in accordance with the [OpenAPI JSON schema format](https://spec.openapis.org/oas/v3.0.3#schemawr).\n",
"\n",
"Define a tool that allows the Gemini model to select from the set of 3 functions:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "4b2d1900730d"
},
"outputs": [],
"source": [
"retail_tool = Tool(\n",
" function_declarations=[\n",
" get_product_info,\n",
" get_store_location,\n",
" place_order,\n",
" ],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2b3781f6fd83"
},
"source": [
"Now you can initialize the Gemini model with Function Calling in a multi-turn chat session.\n",
"\n",
"You can specify the `tools` kwarg when initializing the chat session to avoid having to send it with every subsequent request:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "ef8c2d811321"
},
"outputs": [],
"source": [
"chat = client.chats.create(\n",
" model=MODEL_ID,\n",
" config=GenerateContentConfig(\n",
" temperature=0,\n",
" tools=[retail_tool],\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "adc8022b2461"
},
"source": [
"**Note:** The `temperature` parameter controls the degree of randomness in this generation. Lower temperatures are good for functions that require deterministic parameter values, while higher temperatures are good for functions with parameters that accept more diverse or creative parameter values. A temperature of `0` is deterministic. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.\n",
"\n",
"We're ready to chat! Let's start the conversation by asking if a certain product is in stock:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "9556d1ebcc1f"
},
"outputs": [
{
"data": {
"text/plain": [
"FunctionCall(\n",
" args={\n",
" 'product_name': 'Pixel 9'\n",
" },\n",
" name='get_product_info'\n",
")"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"\n",
"Do you have the Pixel 9 in stock?\n",
"\"\"\"\n",
"\n",
"response = chat.send_message(prompt)\n",
"response.function_calls[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3111780745fc"
},
"source": [
"The response from the Gemini API consists of a structured data object that contains the name and parameters of the function that Gemini selected out of the available functions.\n",
"\n",
"Since this notebook focuses on the ability to extract function parameters and generate function calls, you'll use mock data to feed synthetic responses back to the Gemini model rather than sending a request to an API server (not to worry, we'll make an actual API call in a later example!):"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "0c3f7b5474da"
},
"outputs": [],
"source": [
"# Here you can use your preferred method to make an API request and get a response.\n",
"# In this example, we'll use synthetic data to simulate a payload from an external API response.\n",
"\n",
"api_response = {\"sku\": \"GA04834-US\", \"in_stock\": \"yes\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3d86f58489be"
},
"source": [
"In reality, you would execute function calls against an external system or database using your desired client library or REST API.\n",
"\n",
"Now, you can pass the response from the (mock) API request and generate a response for the end user:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "5bbc8135093d"
},
"outputs": [
{
"data": {
"text/markdown": [
"Yes, the Pixel 9 is currently in stock."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"response = chat.send_message(\n",
" Part.from_function_response(\n",
" name=\"get_product_info\",\n",
" response={\n",
" \"content\": api_response,\n",
" },\n",
" ),\n",
")\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "186d7afafee9"
},
"source": [
"Next, the user might ask where they can buy a different phone from a nearby store:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "0258f7777226"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'product_name': 'Pixel 9 Pro XL'\n",
" },\n",
" name='get_product_info'\n",
" ),\n",
" FunctionCall(\n",
" args={\n",
" 'location': 'Mountain View, CA'\n",
" },\n",
" name='get_store_location'\n",
" )]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"\n",
"What about the Pixel 9 Pro XL? Is there a store in\n",
"Mountain View, CA that I can visit to try one out?\n",
"\"\"\"\n",
"\n",
"response = chat.send_message(prompt)\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "da19e8e5292c"
},
"source": [
"Again, you get a response with structured data, but notice that there are two function calls instead of one!\n",
"\n",
"The Gemini model identified that it needs both the `get_product_info` and `get_store_location` functions.\n",
"Look closely at the prompt that you used in this conversation turn a few cells up, and you'll notice that the user asked about a product -and- the location of a store.\n",
"\n",
"In cases like this when two or more functions are defined (or when the model predicts multiple function calls to the same function), the Gemini model might sometimes return back-to-back or parallel function call responses within a single conversation turn.\n",
"\n",
"This is expected behavior since the Gemini model predicts which functions it should call at runtime, what order it should call dependent functions in, and which function calls can be parallelized, so that the model can gather enough information to generate a natural language response.\n",
"\n",
"Not to worry! You can repeat the same steps as before and build synthetic payloads that would come from an external APIs:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "fba8fb03a8f9"
},
"outputs": [],
"source": [
"# Here you can use your preferred method to make an API request and get a response.\n",
"# In this example, we'll use synthetic data to simulate a payload from an external API response.\n",
"\n",
"product_info_api_response = {\"sku\": \"GA08475-US\", \"in_stock\": \"yes\"}\n",
"store_location_api_response = {\n",
" \"store\": \"2000 N Shoreline Blvd, Mountain View, CA 94043, US\"\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "adc1530ec2b1"
},
"source": [
"Again, you can pass the responses from the (mock) API requests back to the Gemini model:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "3d8728b830d0"
},
"outputs": [
{
"data": {
"text/markdown": [
"Yes, the Pixel 9 Pro XL is in stock. You can visit our store at 2000 N Shoreline Blvd, Mountain View, CA 94043, US to try one out."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"response = chat.send_message(\n",
" [\n",
" Part.from_function_response(\n",
" name=\"get_product_info\",\n",
" response={\n",
" \"content\": product_info_api_response,\n",
" },\n",
" ),\n",
" Part.from_function_response(\n",
" name=\"get_store_location\",\n",
" response={\n",
" \"content\": store_location_api_response,\n",
" },\n",
" ),\n",
" ]\n",
")\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "02f7d52fbe71"
},
"source": [
"Nice work!\n",
"\n",
"Within a single conversation turn, the Gemini model requested 2 function calls in a row before returning a natural language summary. In reality, you might follow this pattern if you need to make an API call to an inventory management system, and another call to a store location database, customer management system, or document repository.\n",
"\n",
"Finally, the user might ask to order a phone and have it shipped to their address:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "b430f3ea4f9a"
},
"outputs": [
{
"data": {
"text/plain": [
"[FunctionCall(\n",
" args={\n",
" 'address': '1155 Borregas Ave, Sunnyvale, CA 94089',\n",
" 'product': 'Pixel 9 Pro XL'\n",
" },\n",
" name='place_order'\n",
" )]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"\n",
"I'd like to order a Pixel 9 Pro XL and have it shipped to 1155 Borregas Ave, Sunnyvale, CA 94089.\n",
"\"\"\"\n",
"\n",
"response = chat.send_message(prompt)\n",
"response.function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6b0c9fc9d581"
},
"source": [
"Perfect! The Gemini model extracted the user's selected product and their address. Now you can call an API to place the order:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "55883a7238cf"
},
"outputs": [],
"source": [
"# This is where you would make an API request to return the status of their order.\n",
"# Use synthetic data to simulate a response payload from an external API.\n",
"\n",
"order_api_response = {\n",
" \"payment_status\": \"paid\",\n",
" \"order_number\": 12345,\n",
" \"est_arrival\": \"2 days\",\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "51376798e2d6"
},
"source": [
"And send the payload from the external API call so that the Gemini API returns a natural language summary to the end user."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "74f6d8722928"
},
"outputs": [
{
"data": {
"text/markdown": [
"Your order for a Pixel 9 Pro XL has been placed! Your order number is 12345 and it is estimated to arrive in 2 days."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"response = chat.send_message(\n",
" Part.from_function_response(\n",
" name=\"place_order\",\n",
" response={\n",
" \"content\": order_api_response,\n",
" },\n",
" ),\n",
")\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9df66c601c36"
},
"source": [
"And you're done!\n",
"\n",
"You were able to have a multi-turn conversation with the Gemini model using function calls, handling payloads, and generating natural language summaries that incorporated the information from the external systems."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "46b6be36bf79"
},
"source": [
"### Address example: Using Automatic Function Calling to geocode addresses with a maps API"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7845554ca0a5"
},
"source": [
"In this example, you'll define a function that takes multiple parameters as inputs. Then you'll use automatic function calling in the Gemini API to make a live API call to convert an address to latitude and longitude coordinates.\n",
"\n",
"Start by writing a Python function:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "cMLEbK60ZuZ6"
},
"outputs": [],
"source": [
"def get_location(\n",
" amenity: str | None = None,\n",
" street: str | None = None,\n",
" city: str | None = None,\n",
" county: str | None = None,\n",
" state: str | None = None,\n",
" country: str | None = None,\n",
" postalcode: str | None = None,\n",
") -> list[dict]:\n",
" \"\"\"Get latitude and longitude for a given location.\n",
"\n",
" Args:\n",
" amenity (str | None): Amenity or Point of interest.\n",
" street (str | None): Street name.\n",
" city (str | None): City name.\n",
" county (str | None): County name.\n",
" state (str | None): State name.\n",
" country (str | None): Country name.\n",
" postalcode (str | None): Postal code.\n",
"\n",
" Returns:\n",
" list[dict]: A list of dictionaries with the latitude and longitude of the given location.\n",
" Returns an empty list if the location cannot be determined.\n",
" \"\"\"\n",
" base_url = \"https://nominatim.openstreetmap.org/search\"\n",
" params = {\n",
" \"amenity\": amenity,\n",
" \"street\": street,\n",
" \"city\": city,\n",
" \"county\": county,\n",
" \"state\": state,\n",
" \"country\": country,\n",
" \"postalcode\": postalcode,\n",
" \"format\": \"json\",\n",
" }\n",
" # Filter out None values from parameters\n",
" params = {k: v for k, v in params.items() if v is not None}\n",
"\n",
" try:\n",
" response = requests.get(base_url, params=params, headers={\"User-Agent\": \"none\"})\n",
" response.raise_for_status()\n",
" return response.json()\n",
" except requests.RequestException as e:\n",
" print(f\"Error fetching location data: {e}\")\n",
" return []"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2dd17419f473"
},
"source": [
"In this example, you're asking the Gemini model to extract components of the address into specific fields within a structured data object. These fields are then passed to the function you defined and the result is returned to Gemini to make a natural language response.\n",
"\n",
"Send a prompt that includes an address, such as:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "715c7a7437e9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The coordinates for 1600 Amphitheatre Pkwy, Mountain View, CA 94043 are:\n",
"\n",
"* **Latitude:** 37.4224857\n",
"* **Longitude:** -122.0855846\n",
"\n",
"This location is identified as Google Building 41 within the Googleplex campus.\n"
]
}
],
"source": [
"prompt = \"\"\"\n",
"I want to get the coordinates for the following address:\n",
"1600 Amphitheatre Pkwy, Mountain View, CA 94043\n",
"\"\"\"\n",
"\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=prompt,\n",
" config=GenerateContentConfig(tools=[get_location], temperature=0),\n",
")\n",
"print(response.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ab8b57e204a6"
},
"source": [
"Great work! You were able to define a function that the Gemini model used to extract the relevant parameters from the prompt. Then you made a live API call to obtain the coordinates of the specified location.\n",
"\n",
"Here we used the [OpenStreetMap Nominatim API](https://nominatim.openstreetmap.org/ui/search.html) to geocode an address to keep the number of steps in this tutorial to a reasonable number. If you're working with large amounts of address or geolocation data, you can also use the [Google Maps Geocoding API](https://developers.google.com/maps/documentation/geocoding), or any mapping service with an API!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "47d9ae0b4b79"
},
"source": [
"## Conclusions\n",
"\n",
"You have explored the function calling feature through the Google Gen AI Python SDK.\n",
"\n",
"The next step is to enhance your skills by exploring this [documentation page](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling)."
]
}
],
"metadata": {
"colab": {
"name": "intro_function_calling.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/function-calling/multimodal_function_calling.ipynb
+1541
View File
File diff suppressed because it is too large Load Diff
gemini/function-calling/parallel_function_calling.ipynb
+813
View File
@@ -0,0 +1,813 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Working with Parallel Function Calls and Multiple Function Responses in Gemini\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ffunction-calling%2Fparallel_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/function-calling/parallel_function_calling.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/parallel_function_calling.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZNJC1SkrsJY3"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Kristopher Overholt](https://github.com/koverholt) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini models.\n",
"\n",
"[Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) in Gemini lets you create a description of a function in your code, then pass that description to a language model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
"\n",
"In this tutorial, you'll learn how to work with parallel function calling within Gemini Function Calling, including:\n",
" \n",
"- Handling parallel function calls for repeated functions\n",
"- Working with parallel function calls across multiple functions\n",
"- Extracting multiple function calls from a Gemini response\n",
"- Calling multiple functions and returning them to Gemini\n",
"\n",
"### What is parallel function calling?\n",
"\n",
"In previous versions of the Gemini models (prior to May 2024), Gemini would return two or more chained function calls if the model determined that more than one function call was needed before returning a natural language summary. Here, a chained function call means that you get the first function call response, return the API data to Gemini, get a second function call response, return the API data to Gemini, and so on.\n",
"\n",
"In recent versions of specific Gemini models (from May 2024 and on), Gemini has the ability to return two or more function calls in parallel (i.e., two or more function call responses within the first function call response object). Parallel function calling allows you to fan out and parallelize your API calls or other actions that you perform in your application code, so you don't have to work through each function call response and return one-by-one! Refer to the [Gemini Function Calling documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling) for more information on which Gemini model versions support parallel function calling.\n",
"\n",
"\n",
"<img src=\"https://storage.googleapis.com/github-repo/generative-ai/gemini/function-calling/parallel-function-calling-in-gemini.png\">"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Getting Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai wikipedia"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"global\"\n",
"\n",
"from google import genai\n",
"\n",
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXHfaVS66_01"
},
"source": [
"## Code Examples\n",
"\n",
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "lslYAvw37JGQ"
},
"outputs": [],
"source": [
"from typing import Any\n",
"\n",
"import wikipedia\n",
"from IPython.display import Markdown, display\n",
"from google.genai.types import (\n",
" FunctionDeclaration,\n",
" GenerateContentConfig,\n",
" GenerateContentResponse,\n",
" Part,\n",
" Tool,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b2acd610d52c"
},
"source": [
"### Define helper function"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "32c90d8c452a"
},
"outputs": [],
"source": [
"# Helper function to extract one or more function calls from a Gemini Function Call response\n",
"\n",
"\n",
"def extract_function_calls(response: GenerateContentResponse) -> list[dict]:\n",
" function_calls: list[dict] = []\n",
" for function_call in response.function_calls:\n",
" function_call_dict: dict[str, dict[str, Any]] = {function_call.name: {}}\n",
" for key, value in function_call.args.items():\n",
" function_call_dict[function_call.name][key] = value\n",
" function_calls.append(function_call_dict)\n",
" return function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j3KHAr6BsJY6"
},
"source": [
"## Example: Parallel function calls on the same function\n",
"\n",
"A great use case for parallel function calling is when you have a function that only accepts one parameter per API call and you need to make repeated calls to that function.\n",
"\n",
"With Parallel Function Calling, rather than having to send N number of API requests to Gemini for N number function calls, instead you can send a single API request to Gemini, receive N number of Function Call Responses within a single response, make N number of external API calls in your code, then return all of the API responses to Gemini in bulk. And you can do all of this without any extra configuration in your function declarations, tools, or requests to Gemini.\n",
"\n",
"In this example, you'll do exactly that and use Parallel Function Calling in Gemini to ask about multiple topics on [Wikipedia](https://www.wikipedia.org/). Let's get started!\n",
"\n",
"### Write function declarations and wrap them in a tool\n",
"\n",
"First, define your function declarations and tool using the Vertex AI Python SDK:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "31a4ae78030e"
},
"outputs": [],
"source": [
"search_wikipedia = FunctionDeclaration(\n",
" name=\"search_wikipedia\",\n",
" description=\"Search for articles on Wikipedia\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"query\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Query to search for on Wikipedia\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"wikipedia_tool = Tool(\n",
" function_declarations=[\n",
" search_wikipedia,\n",
" ],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "11c5abb4bbb1"
},
"source": [
"### Initialize the Gemini model\n",
"\n",
"Now you can initialize Gemini using a [model version that supports parallel function calling](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling):"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "521d7dbb44a7"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\"\n",
"\n",
"chat = client.chats.create(\n",
" model=MODEL_ID, config=GenerateContentConfig(temperature=0, tools=[wikipedia_tool])\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a8ac4fd2f45"
},
"source": [
"### Send prompt to Gemini\n",
"\n",
"Send a prompt to Gemini that includes a phrase that you expect to invoke two or more function calls.\n",
"\n",
"In this case we'll ask about three different article topics to search for on Wikipedia:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "aD4UJ6BcsJY6"
},
"outputs": [],
"source": [
"prompt = \"Search for articles related to solar panels, renewable energy, and battery storage and provide a summary of your findings\"\n",
"\n",
"response = chat.send_message(prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f8e67fe6825f"
},
"source": [
"### Extract function names and parameters\n",
"\n",
"Use the helper function that we created earlier to extract the function names and function parameters for each Function Call that Gemini responded with:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "468e7308ebb8"
},
"outputs": [
{
"data": {
"text/plain": [
"[{'search_wikipedia': {'query': 'solar panel'}},\n",
" {'search_wikipedia': {'query': 'renewable energy'}},\n",
" {'search_wikipedia': {'query': 'battery storage power station'}}]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"function_calls = extract_function_calls(response)\n",
"function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5d69b21870f1"
},
"source": [
"Note that the helper function is just one way to extract fields from the structured Function Call response. You can modify the helper function or write your own custom code to extract and get the fields into your preferred format or data structure!\n",
"\n",
"### Make external API calls\n",
"\n",
"Next, you'll loop through the Function Calls and use the `wikipedia` Python package to make an API call for each search query:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "754bfbf1864a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'search_wikipedia': {'query': 'solar panel'}}\n",
"{'search_wikipedia': {'query': 'renewable energy'}}\n",
"{'search_wikipedia': {'query': 'battery storage power station'}}\n"
]
}
],
"source": [
"api_response = []\n",
"\n",
"# Loop over multiple function calls\n",
"for function_call in function_calls:\n",
" print(function_call)\n",
"\n",
" # Make external API call\n",
" result = wikipedia.summary(function_call[\"search_wikipedia\"][\"query\"])\n",
"\n",
" # Collect all API responses\n",
" api_response.append(result)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "90340a8c2949"
},
"source": [
"### Get a natural language summary\n",
"\n",
"Now you can return all of the API responses to Gemini so that it can generate a natural language summary:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "3409d561a84b"
},
"outputs": [
{
"data": {
"text/markdown": "Based on recent research into solar panels, renewable energy, and battery storage, here is a summary of the current state and integration of these technologies.\n\n### 1. Solar Panels (Photovoltaics)\nSolar panels are devices that convert sunlight directly into electricity using **photovoltaic (PV) cells**. \n* **Mechanism:** When exposed to light, PV cells (typically made of silicon) produce excited electrons that create a direct current (DC). An **inverter** is then used to convert this into alternating current (AC) for use in homes or the grid.\n* **Applications:** They range from small-scale residential rooftop installations to massive \"solar farms.\"\n* **Benefits & Challenges:** They provide a clean, carbon-free energy source. However, their efficiency depends on sunlight intensity and weather, making them a \"variable\" energy source that requires management when the sun isn't shining.\n\n### 2. Renewable Energy Landscape\nRenewable energy (or \"green energy\") comes from natural resources that replenish on a human timescale, such as solar, wind, hydropower, and geothermal.\n* **Growth:** As of 2024, renewables account for over **30% of global electricity generation**, with a projected increase to 45% by 2030. Solar and wind have seen the most dramatic cost reductions, often becoming the cheapest options for new power generation.\n* **Environmental Impact:** The primary driver for the shift to renewables is the reduction of greenhouse gas emissions to combat climate change. The International Energy Agency (IEA) suggests that 90% of global electricity must be renewable by 2050 to reach net-zero goals.\n* **Variability:** A key distinction in this field is between **variable** sources (solar/wind) and **controllable** sources (hydro/geothermal), which can be adjusted based on demand.\n\n### 3. Battery Storage Systems\nBattery Energy Storage Systems (BESS) are critical for modernizing the electrical grid, acting as a bridge between variable energy production and consumer demand.\n* **Grid Stabilization:** Batteries are the fastest-responding power source on the grid. They can transition from standby to full power in under a second, helping to prevent blackouts and stabilize frequency.\n* **Energy Shifting:** They store excess energy produced during peak sunlight or wind hours and discharge it during periods of high demand or low production (e.g., at night).\n* **Cost Trends:** The cost of battery storage has plummeted, with the \"levelized cost of storage\" halving roughly every four years. By 2023, prices dropped to approximately $117 per MWh.\n* **Scale:** While individual battery stations are currently smaller than traditional pumped-hydro storage, they are being deployed rapidly in urban areas and near existing power plants due to their compact size and lack of emissions.\n\n### Summary of Findings\nThe synergy between these three areas is the foundation of the energy transition. **Solar panels** provide the clean generation, **renewable energy** policies drive the large-scale adoption, and **battery storage** solves the \"intermittency problem,\" ensuring that the clean energy generated during the day can reliably power the world 24/7. Together, they are rapidly displacing fossil fuels, which shrank from 68% to 62% of the global energy mix in the last decade.",
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Return the API response to Gemini\n",
"response = chat.send_message(\n",
" [\n",
" Part.from_function_response(\n",
" name=\"search_wikipedia\",\n",
" response={\n",
" \"content\": api_response[0],\n",
" },\n",
" ),\n",
" Part.from_function_response(\n",
" name=\"search_wikipedia\",\n",
" response={\n",
" \"content\": api_response[1],\n",
" },\n",
" ),\n",
" Part.from_function_response(\n",
" name=\"search_wikipedia\",\n",
" response={\n",
" \"content\": api_response[2],\n",
" },\n",
" ),\n",
" ],\n",
")\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6e55e0b45931"
},
"source": [
"And you're done with no extra configuration necessary!\n",
"\n",
"Note that Gemini will use the information in your `FunctionDeclarations` to determine if and when it should return parallel Function Call responses, or it will determine which Function Calls need to be made before others to get information that a subsequent Function Call depends on. So be sure to account for this behavior in your logic and application code!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YBnpF0Yl5exC"
},
"source": [
"## Example: Parallel function calls across multiple functions\n",
"\n",
"Another good fit for parallel function calling is when you have multiple, independent functions that you want to be able to call in parallel, which reduces the number of consecutive Gemini API calls that you need to make and (ideally) reduces the overall response time to the end user who is waiting for a natural language response.\n",
"\n",
"In this example, you'll use Parallel Function Calling in Gemini to ask about multiple aspects of topics and articles on [Wikipedia](https://www.wikipedia.org/).\n",
"\n",
"### Write function declarations and wrap them in a tool\n",
"\n",
"First, define your function declarations and tool using the Vertex AI Python SDK:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "577bd3ad36ed"
},
"outputs": [],
"source": [
"search_wikipedia = FunctionDeclaration(\n",
" name=\"search_wikipedia\",\n",
" description=\"Search for articles on Wikipedia\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"query\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Query to search for on Wikipedia\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"suggest_wikipedia = FunctionDeclaration(\n",
" name=\"suggest_wikipedia\",\n",
" description=\"Get suggested titles from Wikipedia for a given term\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"query\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Query to search for suggested titles on Wikipedia\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"summarize_wikipedia = FunctionDeclaration(\n",
" name=\"summarize_wikipedia\",\n",
" description=\"Get article summaries from Wikipedia\",\n",
" parameters={\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"topic\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Query to search for article summaries on Wikipedia\",\n",
" },\n",
" },\n",
" },\n",
")\n",
"\n",
"wikipedia_tool = Tool(\n",
" function_declarations=[\n",
" search_wikipedia,\n",
" suggest_wikipedia,\n",
" summarize_wikipedia,\n",
" ],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4884fb361482"
},
"source": [
"### Initialize the Gemini model\n",
"\n",
"Now you can initialize Gemini using a [model version that supports parallel function calling](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling):"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "470894a34e87"
},
"outputs": [],
"source": [
"chat = client.chats.create(\n",
" model=MODEL_ID, config=GenerateContentConfig(temperature=0, tools=[wikipedia_tool])\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4a7026e6a6a4"
},
"source": [
"### Send prompt to Gemini\n",
"\n",
"Send a prompt to Gemini that includes a phrase that you expect to invoke two or more function calls.\n",
"\n",
"In this case we'll ask about three types of details to look up for a given topic on Wikipedia:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "MVlbUOoe5exC"
},
"outputs": [],
"source": [
"prompt = \"Search for the solar system on Wikipedia, suggest related terms, and summarize the main article\"\n",
"\n",
"response = chat.send_message(prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "add3c553b773"
},
"source": [
"### Extract function names and parameters\n",
"\n",
"Use the helper function that we created earlier to extract the function names and function parameters for each Function Call that Gemini responded with:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "f02643f2344b"
},
"outputs": [
{
"data": {
"text/plain": [
"[{'search_wikipedia': {'query': 'Solar System'}},\n",
" {'suggest_wikipedia': {'query': 'Solar System'}},\n",
" {'summarize_wikipedia': {'topic': 'Solar System'}}]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"function_calls = extract_function_calls(response)\n",
"function_calls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0e86bbd26969"
},
"source": [
"### Make external API calls\n",
"\n",
"Next, you'll loop through the Function Calls and use the `wikipedia` Python package to make APIs calls and gather information from Wikipedia:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "b3ca4ba3118b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'search_wikipedia': {'query': 'Solar System'}}\n",
"{'suggest_wikipedia': {'query': 'Solar System'}}\n",
"{'summarize_wikipedia': {'topic': 'Solar System'}}\n"
]
}
],
"source": [
"api_response: dict[str, Any] = {} # type: ignore\n",
"\n",
"# Loop over multiple function calls\n",
"for function_call in function_calls:\n",
" print(function_call)\n",
" for function_name, function_args in function_call.items():\n",
" # Determine which external API call to make\n",
" if function_name == \"search_wikipedia\":\n",
" result = wikipedia.search(function_args[\"query\"])\n",
" if function_name == \"suggest_wikipedia\":\n",
" result = wikipedia.suggest(function_args[\"query\"])\n",
" if function_name == \"summarize_wikipedia\":\n",
" result = wikipedia.summary(function_args[\"topic\"], auto_suggest=False)\n",
"\n",
" # Collect all API responses\n",
" api_response[function_name] = result"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5edff88c747e"
},
"source": [
"### Get a natural language summary\n",
"\n",
"Now you can return all of the API responses to Gemini so that it can generate a natural language summary:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "0c94849ed303"
},
"outputs": [
{
"data": {
"text/markdown": "The **Solar System** is the gravitationally bound system of the Sun and the objects that orbit it. It formed approximately 4.6 billion years ago from the collapse of a giant interstellar molecular cloud.\n\n### **Summary of the Solar System**\n* **The Sun:** The central star that accounts for 99.86% of the system's total mass. It generates energy through nuclear fusion in its core.\n* **The Planets:** There are eight major planets divided into two groups:\n * **Inner Planets (Terrestrial):** Mercury, Venus, Earth, and Mars. These are rocky planets.\n * **Outer Planets (Giants):** Jupiter and Saturn (gas giants), and Uranus and Neptune (ice giants).\n* **Dwarf Planets:** Objects like Pluto, Ceres, Eris, Haumea, and Makemake that orbit the Sun but do not dominate their orbital paths.\n* **Small Bodies:** The system includes millions of asteroids (mostly in the Asteroid Belt), comets, and meteoroids.\n* **Moons:** Many planets and dwarf planets have natural satellites, such as Earth's Moon or Jupiter's Ganymede.\n* **Outer Reaches:** Beyond the planets lie the Kuiper Belt, the scattered disc, and the theorized Oort Cloud, which marks the outermost edge of the Solar System's gravitational influence.\n\n### **Related Terms and Topics**\nBased on Wikipedia's database, here are several related terms and areas of study:\n* **Formation and evolution of the Solar System:** The history of how the system began and how it will change over billions of years.\n* **Exoplanet:** Planets located outside of our Solar System orbiting other stars.\n* **List of Solar System objects by size:** A comparison of the various planets, moons, and dwarf planets.\n* **Kuiper Belt:** A region of the Solar System beyond the orbit of Neptune containing many small icy bodies.\n* **Heliopause:** The boundary where the solar wind is stopped by the interstellar medium.\n* **Celestial Mechanics:** The branch of astronomy that deals with the motions of objects in outer space.",
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Return the API response to Gemini\n",
"response = chat.send_message(\n",
" [\n",
" Part.from_function_response(\n",
" name=\"search_wikipedia\",\n",
" response={\n",
" \"content\": api_response.get(\"search_wikipedia\", \"\"),\n",
" },\n",
" ),\n",
" Part.from_function_response(\n",
" name=\"suggest_wikipedia\",\n",
" response={\n",
" \"content\": api_response.get(\"suggest_wikipedia\", \"\"),\n",
" },\n",
" ),\n",
" Part.from_function_response(\n",
" name=\"summarize_wikipedia\",\n",
" response={\n",
" \"content\": api_response.get(\"summarize_wikipedia\", \"\"),\n",
" },\n",
" ),\n",
" ],\n",
")\n",
"\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GpDvGrmtsJY8"
},
"source": [
"And you're done! You successfully made parallel function calls for a couple of different use cases. Feel free to adapt the code samples here for your own use cases and applications. Or try another notebook to continue exploring other functionality in the Gemini API.\n",
"\n",
"Happy parallel function calling!"
]
}
],
"metadata": {
"colab": {
"name": "parallel_function_calling.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}