{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "ijGzTHJJUCPY" }, "outputs": [], "source": [ "# Copyright 2024 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "VEqbX8OhE8y9" }, "source": [ "# Multimodal Function Calling with the Gemini API & Python SDK\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Run in Colab\n", " \n", " 		\n", " \n", " \"Google
Run in Colab Enterprise\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", " 		\n", " \n", " \"Vertex
Open in Vertex AI Workbench\n", " \n", "
\n", "\n", "
\n", "\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", " \n" ] }, { "cell_type": "markdown", "metadata": { "id": "84e7e432e6ff" }, "source": [ "| Author(s) |\n", "| --- |\n", "| [Kristopher Overholt](https://github.com/koverholt) |" ] }, { "cell_type": "markdown", "metadata": { "id": "CkHPv2myT2cx" }, "source": [ "## Overview\n", "\n", "### Introduction to Multimodal Function Calling with Gemini\n", "\n", "This notebook demonstrates a powerful [Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) capability of the Gemini model: support for multimodal inputs. With multimodal function calling, you can go beyond traditional text inputs, enabling Gemini to understand your intent and predict function calls and function parameters based on various inputs like images, audio, video, and PDFs. Function calling can also be referred to as *function calling with controlled generation*, which guarantees that output generated by the model always adheres to a specific schema so that you receive consistently formatted responses.\n", "\n", "Previously, implementing multimodal function calling required two separate calls to the Gemini API: one to extract information from media, and another to generate a function call based on the extracted text. This process was cumbersome, prone to errors, and resulted in the loss of detail in valuable contextual information. Gemini's multimodal function calling capability streamlines this workflow, enabling a single API call that efficiently processes multimodal inputs for accurate function predictions and structured outputs.\n", "\n", "### How It Works\n", "\n", "1. **Define Functions and Tools:** Describe your functions, then group them into `Tool` objects for Gemini to use.\n", "2. **Send Inputs and Prompt:** Provide Gemini with multimodal input (image, audio, PDF, etc.) and a prompt describing your request.\n", "3. **Gemini Predicts Action:** Gemini analyzes the multimodal input and prompt to predict the best function to call and its parameters.\n", "4. **Execute and Return:** Use Gemini's prediction to make API calls, then send the results back to Gemini.\n", "5. **Generate Response:** Gemini uses the API results to provide a final, natural language response to the user.\n", "\n", "This notebook will guide you through practical examples of using Gemini's multimodal function calling to build intelligent applications that go beyond the limitations of text-only interactions." ] }, { "cell_type": "markdown", "metadata": { "id": "DrkcqHrrwMAo" }, "source": [ "### Objectives\n", "\n", "In this tutorial, you will learn how to use the Gemini API in Vertex AI with the Vertex AI SDK for Python to make function calls with multimodal inputs, using the Gemini 3 model. You'll explore how Gemini can process and understand various input types — including images, video, audio, and PDFs — to predict and execute functions.\n", "\n", "You will complete the following tasks:\n", "\n", "- Install the Vertex AI SDK for Python.\n", "- Define functions that can be called by Gemini.\n", "- Package functions into tools.\n", "- Send multimodal inputs (images, video, audio, PDFs) and prompts to Gemini.\n", "- Extract predicted function calls and their parameters from Gemini's response.\n", "- Use the predicted output to make API calls to external systems (demonstrated with an image input example).\n", "- Return API responses to Gemini for natural language response generation (demonstrated with an image input example)." ] }, { "cell_type": "markdown", "metadata": { "id": "C9nEPojogw-g" }, "source": [ "### Costs\n", "\n", "This tutorial uses billable components of Google Cloud:\n", "\n", "- Vertex AI\n", "\n", "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "r11Gu7qNgx1p" }, "source": [ "## Getting Started\n" ] }, { "cell_type": "markdown", "metadata": { "id": "No17Cw5hgx12" }, "source": [ "### Install Google Gen AI SDK" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "tFy3H3aPgx12" }, "outputs": [], "source": [ "%pip install --upgrade --quiet google-genai wikipedia" ] }, { "cell_type": "markdown", "metadata": { "id": "dmWOrTJ3gx13" }, "source": [ "### Authenticate your notebook environment (Colab only)\n", "\n", "If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "NyKGtVQjgx13" }, "outputs": [], "source": [ "import sys\n", "\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "DF4l8DTdWgPY" }, "source": [ "### Set Google Cloud project information and initialize Vertex AI SDK\n", "\n", "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", "\n", "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "Nqwi-5ufWp_B" }, "outputs": [], "source": [ "import os\n", "\n", "# fmt: off\n", "PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", "# fmt: on\n", "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", "\n", "LOCATION = \"global\"\n", "\n", "from google import genai\n", "\n", "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "markdown", "metadata": { "id": "2a7225e4390a" }, "source": [ "## Multimodal Function Calling in Action" ] }, { "cell_type": "markdown", "metadata": { "id": "jXHfaVS66_01" }, "source": [ "### Import libraries\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "lslYAvw37JGQ" }, "outputs": [], "source": [ "import wikipedia\n", "from IPython.display import Markdown, display\n", "from google.genai.types import (\n", " Content,\n", " FunctionDeclaration,\n", " GenerateContentConfig,\n", " Part,\n", " Tool,\n", " UserContent,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "8cd167f70edf" }, "source": [ "### Initialize model\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "ed564d2b905c" }, "outputs": [], "source": [ "MODEL_ID = \"gemini-3.5-flash\"" ] }, { "cell_type": "markdown", "metadata": { "id": "aa432a6e021a" }, "source": [ "### Image-Based Function Calling: Finding Animal Habitats\n", "\n", "In this example, you'll send along an image of a bird and ask Gemini to identify its habitat. This involves defining a function that looks up regions where a given animal is found, creating a tool that uses this function, and then sending a request to Gemini.\n", "\n", "\n", "\n", "First, you define a `FunctionDeclaration` called `get_wildlife_region`. This function takes the name of an animal species as input and returns information about its typical region." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "ae36049d4512" }, "outputs": [], "source": [ "get_wildlife_region = FunctionDeclaration(\n", " name=\"get_wildlife_region\",\n", " description=\"Look up the region where an animal can be found\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"animal\": {\"type\": \"string\", \"description\": \"Species of animal\"}\n", " },\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "0933d807af15" }, "source": [ "Next, you create a `Tool` object that includes your `get_wildlife_region` function. Tools help group related functions that Gemini can use:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "c4218e572dc8" }, "outputs": [], "source": [ "image_tool = Tool(\n", " function_declarations=[\n", " get_wildlife_region,\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "28abd9546c48" }, "source": [ "Now you're ready to send a request to Gemini. Initialize the `GenerativeModel` and specify the image to analyze, along with a prompt. The `tools` argument tells Gemini to consider the functions in your `image_tool`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "id": "38b842d71bce" }, "outputs": [ { "data": { "text/plain": [ "FunctionCall(\n", " args={\n", " 'animal': 'Lilac-breasted Roller'\n", " },\n", " name='get_wildlife_region'\n", ")" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/multi-color-bird.jpg\",\n", " mime_type=\"image/jpeg\",\n", " ),\n", " \"What is the typical habitat or region where this animal lives?\",\n", " ],\n", " config=GenerateContentConfig(temperature=0, tools=[image_tool]),\n", ")\n", "\n", "model_response_content = response.candidates[0].content # Save for thought signature\n", "response.function_calls[0]" ] }, { "cell_type": "markdown", "metadata": { "id": "065787dbaa26" }, "source": [ "Let's examine the response from Gemini. You can extract the predicted function name:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "e2b92a75e5b9" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'get_wildlife_region'" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_name = response.function_calls[0].name\n", "function_name" ] }, { "cell_type": "markdown", "metadata": { "id": "4a6ba2cf6937" }, "source": [ "You can also get the arguments that Gemini predicted for the function call:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "c89f16d5082e" }, "outputs": [ { "data": { "text/plain": [ "{'animal': 'Lilac-breasted Roller'}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_args = dict(response.function_calls[0].args.items())\n", "function_args" ] }, { "cell_type": "markdown", "metadata": { "id": "180ef53579a0" }, "source": [ "Now, you'll call an external API (in this case, using the `wikipedia` Python package) using the animal name that Gemini extracted from the image:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "07eed3ae7aa3" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'The lilac-breasted roller (Coracias caudatus) is an African bird of the roller family, Coraciidae. It is widely distributed in Southern and Eastern Africa, and is a vagrant to the southern Arabian Peninsula. It prefers open woodland and savanna, and it is for the most part absent from treeless places. Usually found alone or in pairs, it perches at the tops of trees, poles or other high vantage points from where it can spot insects, amphibians and small birds moving about on the ground. Nesting t'" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "api_response = wikipedia.page(function_args[\"animal\"]).content\n", "api_response[:500]" ] }, { "cell_type": "markdown", "metadata": { "id": "f238cad25c36" }, "source": [ "Finally, you return the API response to Gemini so it can generate a final answer in natural language:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "02ee532ce187" }, "outputs": [ { "data": { "text/markdown": "The bird in the image is a **Lilac-breasted Roller** (*Coracias caudatus*), known for its stunning multi-colored plumage.\n\nThis species is widely distributed across **Southern and Eastern Africa**. Its range includes:\n\n* **Eastern Africa:** From Eritrea and Ethiopia through Kenya (where it is the unofficial national bird), Tanzania (including Zanzibar), and Somalia.\n* **Southern Africa:** Commonly found in Namibia (except for the Namib Desert), Botswana, Zimbabwe, Mozambique, and northeastern South Africa.\n* **Western Africa:** Its range extends to the Atlantic coasts of Angola and Namibia.\n* **Arabian Peninsula:** It is occasionally seen as a vagrant in Oman and Yemen.\n\n**Typical Habitat:**\nThey prefer **open woodlands and savannas** with scattered trees and shrubs. You will often see them perched alone on high vantage points like treetops, telephone poles, or even on the backs of large herbivores, where they wait to swoop down on insects, lizards, and other small prey. They generally avoid treeless plains and heavily forested areas.", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " UserContent(\n", " parts=[\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/multi-color-bird.jpg\",\n", " mime_type=\"image/jpeg\",\n", " ),\n", " Part.from_text(\n", " text=\"Inspect the image and get the regions where this animal can be found\",\n", " ),\n", " ]\n", " ),\n", " model_response_content, # Function call with thought signature\n", " Content(\n", " role=\"tool\",\n", " parts=[\n", " Part.from_function_response(\n", " name=function_name,\n", " response={\n", " \"content\": api_response,\n", " },\n", " ),\n", " ],\n", " ),\n", " ],\n", " config=GenerateContentConfig(temperature=0, tools=[image_tool]),\n", ")\n", "display(Markdown(response.text))" ] }, { "cell_type": "markdown", "metadata": { "id": "6e5f13d0e644" }, "source": [ "This example showcases how Gemini's multimodal function calling processes an image, predicts a relevant function and its parameters, and integrates with external APIs to provide comprehensive user information. This process opens up exciting possibilities for building intelligent applications that can \"see\" and understand the world around them via API calls to Gemini." ] }, { "cell_type": "markdown", "metadata": { "id": "6036faa1fb70" }, "source": [ "### Video-Based Function Calling: Identifying Product Features" ] }, { "cell_type": "markdown", "metadata": { "id": "4dd489a96132" }, "source": [ "Now let's explore how Gemini can extract information from videos for the purpose of invoking a function call. You'll use a video showcasing multiple products and ask Gemini to identify its key features.\n", "\n", "\n", "\n", "Start by defining a function called `get_feature_info` that takes a list of product features as input and could potentially be used to retrieve additional details about those features:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "41d1ed66b8b3" }, "outputs": [], "source": [ "get_feature_info = FunctionDeclaration(\n", " name=\"get_feature_info\",\n", " description=\"Get additional information about a product feature\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"features\": {\n", " \"type\": \"array\",\n", " \"description\": \"A list of product features\",\n", " \"items\": {\"type\": \"string\", \"description\": \"Product feature\"},\n", " }\n", " },\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "b972769f37e6" }, "source": [ "Next, create a tool that includes your `get_feature_info` function:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "id": "f134fc04e6bb" }, "outputs": [], "source": [ "video_tool = Tool(\n", " function_declarations=[\n", " get_feature_info,\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "c16c497b85d3" }, "source": [ "Send a video to Gemini, along with a prompt asking for information about the product features, making sure to include your `video_tool` in the `tools` kwarg:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "09fbe282c3d3" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'features': [\n", " 'Gemini',\n", " 'Gemini Live',\n", " 'Pixel 9',\n", " 'Pixel 9 Pro',\n", " 'Pixel 9 Pro XL',\n", " <... 10 more items ...>,\n", " ]\n", " },\n", " name='get_feature_info'\n", " )]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/made-by-google-24.mp4\",\n", " mime_type=\"video/mp4\",\n", " ),\n", " \"Inspect the video and get information about the product features shown\",\n", " ],\n", " config=GenerateContentConfig(temperature=0, tools=[video_tool]),\n", ")\n", "\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "4115fd61850b" }, "source": [ "Gemini correctly predicted the `get_feature_info` function:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "1ae3bb7a4847" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'get_feature_info'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_name = response.function_calls[0].name\n", "function_name" ] }, { "cell_type": "markdown", "metadata": { "id": "c17668290dd0" }, "source": [ "And you can see the list of product features that Gemini extracted from the video, which are available as structured function arguments that adhere to the JSON schema we defined in the `FunctionDeclaration`:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "id": "992c59809c7b" }, "outputs": [ { "data": { "text/plain": [ "{'features': ['Gemini',\n", " 'Gemini Live',\n", " 'Pixel 9',\n", " 'Pixel 9 Pro',\n", " 'Pixel 9 Pro XL',\n", " 'Pixel 9 Pro Fold',\n", " 'Made You Look',\n", " 'Add Me',\n", " 'Magic Editor',\n", " 'Video Boost',\n", " 'Call Notes',\n", " 'Pixel Screenshots',\n", " 'Pixel Studio',\n", " 'Pixel Watch 3',\n", " 'Pixel Buds Pro 2']}" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_args = dict(response.function_calls[0].args.items())\n", "function_args" ] }, { "cell_type": "markdown", "metadata": { "id": "bd8dd10f1219" }, "source": [ "This example demonstrates Gemini's ability to understand video content. By defining a relevant function, you can use Gemini to extract structured information from videos and perform further actions based on that information.\n", "\n", "Now that the multimodal function call response is complete, you could use the function name and function arguments to call an external API using any REST API or client library of your choice, similar to how we did in the previous example with the `wikipedia` Python package.\n", "\n", "Since this sample notebook is focused on the mechanics of multimodal function calling rather than the subsequent function calls and API calls, we'll move on to another example with different multimodal inputs. You can refer to other sample notebooks on Gemini Function Calling for more details on where to go from here." ] }, { "cell_type": "markdown", "metadata": { "id": "41a08eda3be8" }, "source": [ "### Audio-Based Function Calling: Generating Book Recommendations" ] }, { "cell_type": "markdown", "metadata": { "id": "a94cd078b43c" }, "source": [ "In this example, you'll explore using audio input with Gemini's multimodal function calling. You'll send a podcast episode to Gemini and ask for book recommendations related to the topics discussed.\n", "\n", ">>> \"SRE is just a production system specific manifestation of systems thinking ... and we kind of do it in an informal way.\"
\n", ">>> \"The book called 'Thinking in Systems' ... it's a really good primer on this topic.\"
\n", ">>> \"An example of ... systems structure behavior thinking ... is the idea of like the cascading failure, that kind of vicious cycle of load that causes retries that causes more load ... \"
\n", ">>> \"The worst pattern is the single embedded SRE that turns into the ops person ... you just end up doing all of the toil, all of the grunt work.\"
\n", ">>> \"Take that moment, take a breath, and really analyze the problem and understand how it's working as a system and understand how you can intervene to improve that.\"
\n", ">>> \"Avoid just doing what you've done before and kicking the can down the road, and really think deeply about your problems.\"
\n", "\n", "Define a function called `get_recommended_books` that takes a list of topics as input and (hypothetically) returns relevant book recommendations:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "9714025043bd" }, "outputs": [], "source": [ "get_recommended_books = FunctionDeclaration(\n", " name=\"get_recommended_books\",\n", " description=\"Get recommended books based on a list of topics\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"topics\": {\n", " \"type\": \"array\",\n", " \"description\": \"A list of topics\",\n", " \"items\": {\"type\": \"string\", \"description\": \"Topic\"},\n", " },\n", " },\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "f23465d0938f" }, "source": [ "Now create a tool that includes your newly defined function:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "id": "d61600788e03" }, "outputs": [], "source": [ "audio_tool = Tool(\n", " function_declarations=[\n", " get_recommended_books,\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "560afbd15a17" }, "source": [ "Provide Gemini with the audio file and a prompt to recommend books based on the podcast content:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "id": "47228e6631a3" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'topics': [\n", " 'Site Reliability Engineering',\n", " 'System Thinking',\n", " 'Scalability',\n", " 'Software Reliability',\n", " 'Entrepreneurship for Engineers',\n", " <... 1 more items ...>,\n", " ]\n", " },\n", " name='get_recommended_books'\n", " )]" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/google-cloud-sre-podcast-s2-e8.mp3\",\n", " mime_type=\"audio/mpeg\",\n", " ),\n", " \"Inspect the audio file and generate a list of recommended books based on the topics discussed.\",\n", " ],\n", " config=GenerateContentConfig(temperature=0, tools=[audio_tool]),\n", ")\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "db9b85de9752" }, "source": [ "You can see that Gemini has successfully predicted your `get_recommended_books` function:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "eabef4d9faf4" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'get_recommended_books'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_name = response.function_calls[0].name\n", "function_name" ] }, { "cell_type": "markdown", "metadata": { "id": "ea00f52eb487" }, "source": [ "And the function arguments contain the list of topics that Gemini identified and extracted from the input audio file:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "8c8f32e930c9" }, "outputs": [ { "data": { "text/plain": [ "{'topics': ['Site Reliability Engineering',\n", " 'System Thinking',\n", " 'Scalability',\n", " 'Software Reliability',\n", " 'Entrepreneurship for Engineers',\n", " 'Incident Response']}" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_args = dict(response.function_calls[0].args.items())\n", "function_args" ] }, { "cell_type": "markdown", "metadata": { "id": "8acd15dd7cec" }, "source": [ "This example highlights Gemini's capacity to understand and extract information from audio, enabling you to create applications that respond to spoken content or audio-based interactions." ] }, { "cell_type": "markdown", "metadata": { "id": "93577f2d2fe1" }, "source": [ "### PDF-Based Function Calling: Extracting Company Data from Invoices" ] }, { "cell_type": "markdown", "metadata": { "id": "924cf8d1c711" }, "source": [ "This example demonstrates how to use Gemini's multimodal function calling to process PDF documents. You'll work with a set of invoices and extract the names of the (fictitious) companies involved.\n", "\n", "\n", "\n", "Define a function called `get_company_information` that (in a real-world scenario) could be used to fetch details about a given list of companies:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "ba57e626e9d2" }, "outputs": [], "source": [ "get_company_information = FunctionDeclaration(\n", " name=\"get_company_information\",\n", " description=\"Get information about a list of companies\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"companies\": {\n", " \"type\": \"array\",\n", " \"description\": \"A list of companies\",\n", " \"items\": {\"type\": \"string\", \"description\": \"Company name\"},\n", " }\n", " },\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "fae1c7a7d8a9" }, "source": [ "Package your newly defined function into a tool:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "a62ed01019f0" }, "outputs": [], "source": [ "invoice_tool = Tool(\n", " function_declarations=[\n", " get_company_information,\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "91dcbcbb0f50" }, "source": [ "Now you can provide Gemini with multiple PDF invoices and ask it to get company information:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "e509abf4d73a" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'companies': [\n", " 'AMNOSH SUPPLIERS',\n", " 'BIKBEAR LAW FIRM',\n", " 'CUCUONAR Agency Pvt Ltd',\n", " 'CWLEY PLC',\n", " 'LUVBPTK Company',\n", " ]\n", " },\n", " name='get_company_information'\n", " )]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=[\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/invoice-synthetic-1.pdf\",\n", " mime_type=\"application/pdf\",\n", " ),\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/invoice-synthetic-2.pdf\",\n", " mime_type=\"application/pdf\",\n", " ),\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/invoice-synthetic-3.pdf\",\n", " mime_type=\"application/pdf\",\n", " ),\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/invoice-synthetic-4.pdf\",\n", " mime_type=\"application/pdf\",\n", " ),\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/invoice-synthetic-5.pdf\",\n", " mime_type=\"application/pdf\",\n", " ),\n", " \"Inspect the PDF files of invoices and retrieve information about each company\",\n", " ],\n", " config=GenerateContentConfig(temperature=0, tools=[invoice_tool]),\n", ")\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "974a138f3c6c" }, "source": [ "As expected, Gemini predicted the `get_company_information` function:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "766fdbafed76" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'get_company_information'" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_name = response.function_calls[0].name\n", "function_name" ] }, { "cell_type": "markdown", "metadata": { "id": "c80e9f280f5c" }, "source": [ "The function arguments contain the list of company names extracted from the PDF invoices:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "9fa7a22d85b2" }, "outputs": [ { "data": { "text/plain": [ "{'companies': ['AMNOSH SUPPLIERS',\n", " 'BIKBEAR LAW FIRM',\n", " 'CUCUONAR Agency Pvt Ltd',\n", " 'CWLEY PLC',\n", " 'LUVBPTK Company']}" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "function_args = dict(response.function_calls[0].args.items())\n", "function_args" ] }, { "cell_type": "markdown", "metadata": { "id": "03e710b9d195" }, "source": [ "This example shows the power of Gemini for processing and extracting structured data from documents, a common requirement in many real-world applications." ] }, { "cell_type": "markdown", "metadata": { "id": "77f53d886376" }, "source": [ "### Image-Based Chat: Building a Multimodal Chatbot" ] }, { "cell_type": "markdown", "metadata": { "id": "d145dc63a74a" }, "source": [ "Let's put it all together and build a simple multimodal chatbot. This chatbot will understand image inputs and respond to questions using the functions you define.\n", "\n", "\n", "\n", "First, define three functions: `get_animal_details`, `search_similar_images`, and `check_color_palette`. These functions represent the capabilities of your chatbot and could potentially be used to retrieve additional details using REST API calls:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "1618b25ad3e0" }, "outputs": [], "source": [ "get_animal_details = FunctionDeclaration(\n", " name=\"get_animal_details\",\n", " description=\"Look up information about a given animal species\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"animal\": {\"type\": \"string\", \"description\": \"Species of animal\"}\n", " },\n", " },\n", ")\n", "\n", "search_similar_images = FunctionDeclaration(\n", " name=\"search_similar_images\",\n", " description=\"Search for similar images in a stock photo database\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"query\": {\n", " \"type\": \"string\",\n", " \"description\": \"Search query describing the image\",\n", " }\n", " },\n", " },\n", ")\n", "\n", "check_color_palette = FunctionDeclaration(\n", " name=\"check_color_palette\",\n", " description=\"Check hex color codes for accessibility\",\n", " parameters={\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"colors\": {\n", " \"type\": \"array\",\n", " \"description\": \"A list of colors in hexadecimal format\",\n", " \"items\": {\n", " \"type\": \"string\",\n", " \"description\": \"Hexadecimal representation of color, as in #355E3B\",\n", " },\n", " }\n", " },\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "ca63d74adeba" }, "source": [ "Group your functions into a tool:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "id": "178ce7754626" }, "outputs": [], "source": [ "chat_tool = Tool(\n", " function_declarations=[\n", " get_animal_details,\n", " search_similar_images,\n", " check_color_palette,\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "0eb1e7629b9e" }, "source": [ "Start a chat session with Gemini, providing it with your `chat_tool`:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "id": "ac1ebff348c9" }, "outputs": [], "source": [ "chat = client.chats.create(\n", " model=MODEL_ID, config=GenerateContentConfig(temperature=0, tools=[chat_tool])\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "05bb7db4be62" }, "source": [ "Send an image of a fox, along with a simple prompt:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "id": "c3e47a96df7e" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'animal': 'red fox'\n", " },\n", " name='get_animal_details'\n", " )]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = chat.send_message(\n", " [\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/baby-fox.jpg\",\n", " mime_type=\"image/jpeg\",\n", " ),\n", " \"Tell me about the animal shown in this picture\",\n", " ]\n", ")\n", "\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "38c96a599b94" }, "source": [ "Now ask the model to find similar images:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "id": "153f7b93eb65" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'query': 'red fox in a grassy field with flowers'\n", " },\n", " name='search_similar_images'\n", " )]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = chat.send_message(\n", " [\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/baby-fox.jpg\",\n", " mime_type=\"image/jpeg\",\n", " ),\n", " \"Find similar images to this one\",\n", " ]\n", ")\n", "\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "db6363659da8" }, "source": [ "And finally, ask for a color palette based the image:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "id": "af519b9c7bc5" }, "outputs": [ { "data": { "text/plain": [ "[FunctionCall(\n", " args={\n", " 'colors': [\n", " '#C06C31',\n", " '#2A241F',\n", " '#6B8E23',\n", " '#B05E8C',\n", " '#2E4617',\n", " <... 1 more items ...>,\n", " ]\n", " },\n", " name='check_color_palette'\n", " )]" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = chat.send_message(\n", " [\n", " Part.from_uri(\n", " file_uri=\"gs://github-repo/generative-ai/gemini/function-calling/baby-fox.jpg\",\n", " mime_type=\"image/jpeg\",\n", " ),\n", " \"Get the color palette of this image and check it for accessibility\",\n", " ]\n", ")\n", "\n", "response.function_calls" ] }, { "cell_type": "markdown", "metadata": { "id": "4e38eb5eeb5b" }, "source": [ "While this chatbot doesn't actually execute the predicted functions, it demonstrates creating an interactive experience using multimodal inputs and function calling in a chat format. You can extend this example by implementing REST API calls or client library requests for each function to create a truly functional and engaging multimodal chatbot that's connected to the real world." ] }, { "cell_type": "markdown", "metadata": { "id": "ee5711d51ae0" }, "source": [ "## Conclusions\n", "\n", "In this notebook, you explored the powerful capabilities of Gemini's multimodal function calling. You learned how to:\n", "\n", "- Define functions and package them into tools.\n", "- Send multimodal inputs (images, video, audio, PDFs) and prompts to Gemini.\n", "- Extract predicted function calls and their parameters.\n", "- Use the predicted output to make (or potentially make) API calls.\n", "- Return API responses to Gemini for natural language generation.\n", "\n", "You've seen how Gemini can understand and act on a range of different multimodal inputs, which opens up a world of possibilities for building innovative and engaging multimodal applications. You can now use these powerful tools to create your own intelligent applications that seamlessly integrate media, natural language, and calls to external APIs and system.\n", "\n", "Experiment with different modalities, functions, and prompts to discover the full potential of Gemini's multimodal and function calling capabilities. And you can continue learning by exploring other sample notebooks in this repository and exploring the [documentation for Gemini Function Calling](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling)." ] } ], "metadata": { "colab": { "name": "multimodal_function_calling.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }