wehub-resource-sync
1643 lines
57 KiB
Plaintext
1643 lines
57 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "77nhZxV05gIY"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2026 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rJ43O7oc6xGP"
|
|
},
|
|
"source": [
|
|
"# Intro to Gemini 3.1 Pro\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgetting-started%2Fintro_gemini_3_1_pro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_1_pro.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8MqT58L6Rm_q"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |\n",
|
|
"| [Holt Skinner](https://github.com/holtskinner) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8sg1R6UQ7Q3t"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"**Gemini 3.1 Pro** is Google's latest flagship model. Key architectural and performance improvements include:\n",
|
|
"\n",
|
|
"* **Enhanced stability & grounding**: Improve factuality and reduce repetitive response patterns \n",
|
|
"* **Advanced coding & agentic capabilities:** Significant improvements in software engineering and agentic performance. \n",
|
|
"* **Efficiency & reasoning modes:** Improved token efficiency across reasoning tasks Introduces Medium in thinking level to balance depth with latency. \n",
|
|
"* **Core quality improvements:** Continued advancements in general reasoning, instruction following, and creative writing quality.\n",
|
|
"\n",
|
|
"This notebook serves as a quickstart guide for developers to begin interacting with the **Gemini 3.1 Pro** model via the Google Gen AI SDK on Vertex AI. It is designed to demonstrate key model capabilities and showcase new API features."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gPiTOAHURvTM"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-tn3uw268iw4"
|
|
},
|
|
"source": [
|
|
"### Install [Google Gen AI SDK for Python](https://github.com/googleapis/python-genai)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3CaXL22k8iw4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "EdvJRUWRNGHE"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "qgdSpVmDbdQ9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"from IPython.display import Markdown, display\n",
|
|
"from google import genai\n",
|
|
"from google.genai import types\n",
|
|
"from pydantic import BaseModel"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SY-GRP3m8iw5"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "06RAe75C8iw5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Ve4YBlDqzyj9"
|
|
},
|
|
"source": [
|
|
"### Autenticate your Google Cloud Project for Vertex AI\n",
|
|
"\n",
|
|
"You can use a Google Cloud Project or an API Key for authentication. This tutorial uses a Google Cloud Project.\n",
|
|
"\n",
|
|
"- [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "a3265ecb5f26"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = \"global\"\n",
|
|
"\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eXHJi5B6P5vd"
|
|
},
|
|
"source": [
|
|
"### Choose a Gemini 3.1 Pro model\n",
|
|
"\n",
|
|
"Use `gemini-3.1-pro-preview` in this tutorial. Learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"cellView": "form",
|
|
"id": "eGbUoVu-931G"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-3.1-pro-preview\" # @param [\"gemini-3.1-pro-preview\"] {type: \"string\"}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "37CH91ddY9kG"
|
|
},
|
|
"source": [
|
|
"## 🚀 Quickstart\n",
|
|
"\n",
|
|
"By default, Gemini 3.1 Pro uses dynamic thinking to reason through prompts. For faster, lower-latency responses when complex reasoning isn't required, you can constrain the model's thinking level by setting parameter `thinking_level` to `low`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "xRJuHj0KZ8xz"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"How does AI work?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.LOW # For faster and lower-latency responses\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "aRrVcsKdlI9N"
|
|
},
|
|
"source": [
|
|
"## 🚨 New API Features"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "95aeab702af3"
|
|
},
|
|
"source": [
|
|
"### 1️⃣ Thinking Level\n",
|
|
"\n",
|
|
"The `thinking_level` parameter allows you to specify a thinking budget for the model's response generation. By selecting a state, you can explicitly balance the trade-offs between response quality/reasoning complexity and latency/cost.\n",
|
|
"\n",
|
|
"- **Low**: Minimizes latency and cost. Best for simple instruction following or chat. Maps to a token budget between `1` and `1,000`.\n",
|
|
"- **Medium**: A balance between reasoning depth and latency. Maps to a token budget between `1,001` and `16,384`.\n",
|
|
"- **High** (default): Maximizes reasoning depth. The model may take significantly longer to reach a first token, but the output will be more thoroughly vetted. Maps to a token budget between `16,384` and `32,768`.\n",
|
|
"\n",
|
|
"#### ⚠️ Notes\n",
|
|
"\n",
|
|
"- If `thinking_level` is not specified, the model defaults to `high`, which uses a dynamic token budget based on prompt complexity.\n",
|
|
"- Setting a token budget of `0` will result in an error.\n",
|
|
"- As a mitigation for looping, the model will output a thought block, even if it's empty, for every step in a multi-step interaction.\n",
|
|
"- You cannot use both `thinking_level` and the legacy `thinking_budget` parameter in the same request. Doing so will return a 400 error.\n",
|
|
"- The OpenAI Chat Completions API `reasoning_effort` `medium` maps to `thinking_level` `high`.\n",
|
|
"- Thinking can't be turned off for Gemini 3.1 Pro.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "364133e30ba5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"You are tasked with implementing the classic Thread-Safe Double-Checked Locking (DCL) Singleton pattern in modern C++. This task is non-trivial and requires specialized concurrency knowledge to prevent memory reordering issues.\n",
|
|
"\n",
|
|
"Write a complete, runnable C++ program named `dcl_singleton.cpp` that defines a class `Singleton` with a private constructor and a static `getInstance()` method.\n",
|
|
"\n",
|
|
"Your solution MUST adhere to the following strict constraints:\n",
|
|
"1. The Singleton instance pointer (`static Singleton*`) must be wrapped in `std::atomic` to correctly manage memory visibility across threads.\n",
|
|
"2. The `getInstance()` method must use `std::memory_order_acquire` when reading the instance pointer in the outer check.\n",
|
|
"3. The instance creation and write-back must use `std::memory_order_release` when writing to the atomic pointer.\n",
|
|
"4. A standard `std::mutex` must be used only to protect the critical section (the actual instantiation).\n",
|
|
"5. The `main` function must demonstrate safe, concurrent access by launching at least three threads, each calling `Singleton::getInstance()`, and printing the address of the returned instance to prove all threads received the same object.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.HIGH # Dynamic thinking for high reasoning tasks\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3YVn52e6lkd9"
|
|
},
|
|
"source": [
|
|
"### 2️⃣ Media Resolution\n",
|
|
"\n",
|
|
"Gemini 3 introduces granular control over multimodal vision processing via the `media_resolution` parameter. Higher resolutions improve the model's ability to read fine text or identify small details, but increase token usage and latency. The `media_resolution` parameter determines the maximum number of tokens allocated per input image or video frame.\n",
|
|
"\n",
|
|
"**⚠️ Note**: Because media resolution directly impacts token count, you may need to lower the resolution (e.g., to `low`) to fit very long inputs, such as long videos or extensive documents."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "71DgyBXAKZtM"
|
|
},
|
|
"source": [
|
|
"You can set the resolution to `low`, `medium`, or `high` per individual media part, as example:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "dKOelAysBYdl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/jpeg\",\n",
|
|
" ),\n",
|
|
" media_resolution=types.PartMediaResolution(\n",
|
|
" level=types.PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/video/behind_the_scenes_pixel.mp4\",\n",
|
|
" mime_type=\"video/mp4\",\n",
|
|
" ),\n",
|
|
" media_resolution=types.PartMediaResolution(\n",
|
|
" level=types.PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" \"When does the image appear in the video? What is the context?\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MCBrvJiiKcyX"
|
|
},
|
|
"source": [
|
|
"Or set it globally via `GenerateContentConfig`. If unspecified, the model uses optimal defaults based on the media type."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "MGBBPXJCKLYX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/jpeg\",\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" \"What is in the image?\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "sx6md380ftML"
|
|
},
|
|
"source": [
|
|
"### 3️⃣ Thought Signature\n",
|
|
"\n",
|
|
"[Thought signatures](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thinking#signatures) are encrypted tokens that preserve the model's reasoning state during multi-turn conversations, specifically when using Function Calling.\n",
|
|
"\n",
|
|
"When a thinking model decides to call an external tool, it pauses its internal reasoning process. The thought signature acts as a \"save state\", allowing the model to resume its chain of thought seamlessly once you provide the function's result.\n",
|
|
"\n",
|
|
"Gemini 3 enforces stricter validation and updated handling on thought signatures which were originally introduced in Gemini 2.5. To ensure the model maintains context across multiple turns of a conversation, you must return the thought signatures in your subsequent requests.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Y7wuJBfkRnd8"
|
|
},
|
|
"source": [
|
|
"#### Automatic Handling of Thought Signatures (Recommended)\n",
|
|
"\n",
|
|
"If you are using the Google Gen AI SDKs (Python, Node.js, Go, Java) or OpenAI Chat Completions API, and utilizing the standard chat history features or appending the full model response, thought signatures are handled automatically. You do not need to make any changes to your code.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nEjLXZ-GTqb9"
|
|
},
|
|
"source": [
|
|
"#### **Example 1**: Automatic Function Calling\n",
|
|
"\n",
|
|
"When using the Gen AI SDK in automatic function calling, thought signatures are handled automatically.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cpcIVcrp7V4i"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_weather(city: str) -> str:\n",
|
|
" \"\"\"Gets the weather in a city.\"\"\"\n",
|
|
" if \"london\" in city.lower():\n",
|
|
" return \"Rainy\"\n",
|
|
" if \"new york\" in city.lower():\n",
|
|
" return \"Sunny\"\n",
|
|
" return \"Cloudy\"\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What's the weather in London and New York?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# The SDK handles the function calls and thought signatures, and returns the final text\n",
|
|
"print(\"Final response:\", response.text)\n",
|
|
"\n",
|
|
"# Print function calling history\n",
|
|
"hist_turn = response.automatic_function_calling_history[1]\n",
|
|
"print(\"\\nFunction Call 1:\", hist_turn.parts[1].function_call.name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XWJqKKVNTEHD"
|
|
},
|
|
"source": [
|
|
"#### **Example 2**: Manual Function Calling\n",
|
|
"\n",
|
|
"When using the Gen AI SDK in manual function calling, thought signatures are also handled automatically if you append the full model response in sequential model requests."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "EVXwCkQET4qQ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Define your tool\n",
|
|
"get_weather_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_weather\",\n",
|
|
" description=\"Gets the current weather temperature for a given location.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\"location\": {\"type\": \"string\"}},\n",
|
|
" \"required\": [\"location\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"get_weather_tool = types.Tool(function_declarations=[get_weather_declaration])\n",
|
|
"\n",
|
|
"# 2. Send a message that triggers the tool\n",
|
|
"prompt = \"What's the weather like in London?\"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" thinking_config=types.ThinkingConfig(include_thoughts=True),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 3. Handle the function call\n",
|
|
"function_call = response.function_calls[0]\n",
|
|
"location = function_call.args[\"location\"]\n",
|
|
"print(f\"Model wants to call: {function_call.name}\")\n",
|
|
"\n",
|
|
"# Execute your tool (e.g., call an API)\n",
|
|
"# (This is a mock response for the example)\n",
|
|
"print(f\"Calling external tool for: {location}\")\n",
|
|
"function_response_data = {\n",
|
|
" \"location\": location,\n",
|
|
" \"temperature\": \"30C\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"# 4. Send the tool's result back\n",
|
|
"# Append this turn's messages to history for a final response.\n",
|
|
"# The `content` object automatically attaches the required thought_signature behind the scenes.\n",
|
|
"history = [\n",
|
|
" types.Content(role=\"user\", parts=[types.Part(text=prompt)]),\n",
|
|
" response.candidates[0].content, # Signature preserved here\n",
|
|
" types.Content(\n",
|
|
" role=\"tool\",\n",
|
|
" parts=[\n",
|
|
" types.Part.from_function_response(\n",
|
|
" name=function_call.name,\n",
|
|
" response=function_response_data,\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response_2 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=history,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" thinking_config=types.ThinkingConfig(include_thoughts=True),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 5. Get the final, natural-language answer\n",
|
|
"print(f\"\\nFinal model response: {response_2.text}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "J6Bn9rRSUexu"
|
|
},
|
|
"source": [
|
|
"#### Manual Handling of Thought Signatures\n",
|
|
"\n",
|
|
"If you are interacting with the API directly or managing raw JSON payloads, you must correctly handle the `thought_signature` included in the model's turn.\n",
|
|
"You must return this signature in the exact part where it was received when sending the conversation history back.\n",
|
|
"\n",
|
|
"⚠️ If proper signatures are not returned, the model will return a 400 Error `\"Function Call in the content block is missing a thought_signature\"`.\n",
|
|
"\n",
|
|
"See [documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#thinking) for more details."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "h6sa6Sj1lse6"
|
|
},
|
|
"source": [
|
|
"### 4️⃣ Streaming Function Calling\n",
|
|
"\n",
|
|
"You can use streaming partial function call arguments to improve streaming experience on tool use. This feature can be enabled by explicitly setting `stream_function_call_arguments` to true."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "VY_Ir0OdID5v"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"get_weather_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_weather\",\n",
|
|
" description=\"Gets the current weather temperature for a given location.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\"location\": {\"type\": \"string\"}},\n",
|
|
" \"required\": [\"location\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"get_weather_tool = types.Tool(function_declarations=[get_weather_declaration])\n",
|
|
"\n",
|
|
"\n",
|
|
"for chunk in client.models.generate_content_stream(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What's the weather in London and New York?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" tool_config=types.ToolConfig(\n",
|
|
" function_calling_config=types.FunctionCallingConfig(\n",
|
|
" mode=types.FunctionCallingConfigMode.AUTO,\n",
|
|
" stream_function_call_arguments=True,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
"):\n",
|
|
" function_call = chunk.function_calls[0]\n",
|
|
" if function_call and function_call.name:\n",
|
|
" print(f\"{function_call.name}\")\n",
|
|
" print(f\"will_continue={function_call.will_continue}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "FODGuNi1XR-V"
|
|
},
|
|
"source": [
|
|
"### 5️⃣ Multimodal Function Responses\n",
|
|
"\n",
|
|
"Multimodal function calling allows users to have function responses containing multimodal objects allowing for improved utilization of function calling capabilities of the model. Currently function calling only supports text based function responses.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "nX7EDrhaPkVh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Define the function tool\n",
|
|
"get_image_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_image\",\n",
|
|
" description=\"Retrieves the image file reference for a specific order item.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"item_name\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The name or description of the item ordered (e.g., 'green shirt').\",\n",
|
|
" }\n",
|
|
" },\n",
|
|
" \"required\": [\"item_name\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"tool_config = types.Tool(function_declarations=[get_image_declaration])\n",
|
|
"\n",
|
|
"# 2. Send a message that triggers the tool\n",
|
|
"prompt = \"Show me the green shirt I ordered last month.\"\n",
|
|
"response_1 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[prompt],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[tool_config],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 3. Handle the function call\n",
|
|
"function_call = response_1.function_calls[0]\n",
|
|
"requested_item = function_call.args[\"item_name\"]\n",
|
|
"print(f\"Model wants to call: {function_call.name}\")\n",
|
|
"\n",
|
|
"# Execute your tool (e.g., call an API)\n",
|
|
"# (This is a mock response for the example)\n",
|
|
"print(f\"Calling external tool for: {requested_item}\")\n",
|
|
"\n",
|
|
"function_response_data = {\n",
|
|
" \"image_ref\": {\"$ref\": \"dress.jpg\"},\n",
|
|
"}\n",
|
|
"\n",
|
|
"function_response_multimodal_data = types.FunctionResponsePart(\n",
|
|
" file_data=types.FunctionResponseFileData(\n",
|
|
" mime_type=\"image/png\",\n",
|
|
" display_name=\"dress.jpg\",\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/dress.jpg\",\n",
|
|
" )\n",
|
|
")\n",
|
|
"\n",
|
|
"# 4. Send the tool's result back\n",
|
|
"# Append this turn's messages to history for a final response.\n",
|
|
"history = [\n",
|
|
" types.Content(role=\"user\", parts=[types.Part(text=prompt)]),\n",
|
|
" response_1.candidates[0].content,\n",
|
|
" types.Content(\n",
|
|
" role=\"tool\",\n",
|
|
" parts=[\n",
|
|
" types.Part.from_function_response(\n",
|
|
" name=function_call.name,\n",
|
|
" response=function_response_data,\n",
|
|
" parts=[function_response_multimodal_data],\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response_2 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=history,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[tool_config],\n",
|
|
" thinking_config=types.ThinkingConfig(include_thoughts=True),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"\\nFinal model response: {response_2.text}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7pFaibjllZwB"
|
|
},
|
|
"source": [
|
|
"## ⭐️ Supported Features"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "El1lx8P9ElDq"
|
|
},
|
|
"source": [
|
|
"### ✅ Set System Instructions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "7A-yANiyCLaO"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"system_instruction = \"\"\"\n",
|
|
" You are a helpful language translator.\n",
|
|
" Your mission is to translate text in English to Spanish.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"prompt = \"\"\"\n",
|
|
" User input: I like bagels.\n",
|
|
" Answer:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "hIJVEr0RQY8S"
|
|
},
|
|
"source": [
|
|
"### ✅ Configure Model Parameters\n",
|
|
"\n",
|
|
"⚠️ **Notes**: For Gemini 3, we strongly recommend keeping the `temperature` parameter at its default value of `1.0`.\n",
|
|
"\n",
|
|
"While previous models often benefitted from tuning `temperature` to control creativity versus determinism, Gemini 3's reasoning capabilities are optimized for the default setting.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d9NXP5N2Pmfo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Tell me how the internet works, but pretend I'm a puppy who only understands squeaky toys.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" temperature=1.0,\n",
|
|
" top_p=0.9,\n",
|
|
" max_output_tokens=8000,\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" include_thoughts=True,\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6lLIxqS6_-l8"
|
|
},
|
|
"source": [
|
|
"### ✅ Generate Content Stream"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ZiwWBhXsAMnv"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.\n",
|
|
"How much does the ball cost?\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"for chunk in client.models.generate_content_stream(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
"):\n",
|
|
" print(chunk.text, end=\"\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "c66712160c15"
|
|
},
|
|
"source": [
|
|
"### ✅ Thought Summaries\n",
|
|
"\n",
|
|
"You can include thought summaries in model response by setting `include_thoughts` to `true`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "60d74a351671"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"How many R's are in the word strawberry?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" include_thoughts=True,\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"for part in response.candidates[0].content.parts:\n",
|
|
" if part.thought:\n",
|
|
" display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"## Thoughts:\n",
|
|
" {part.text}\n",
|
|
" \"\"\"\n",
|
|
" )\n",
|
|
" )\n",
|
|
" else:\n",
|
|
" display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"## Answer:\n",
|
|
" {part.text}\n",
|
|
" \"\"\"\n",
|
|
" )\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "29jFnHZZWXd7"
|
|
},
|
|
"source": [
|
|
"### ✅ Multi-turn Chat\n",
|
|
"\n",
|
|
"Conversation: Starting and maintaining a Multi-turn Chat history."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "DbM12JaLWjiF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"chat = client.chats.create(\n",
|
|
" model=MODEL_ID,\n",
|
|
" config=types.GenerateContentConfig(),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "JQem1halYDBW"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = chat.send_message(\"Write a function that checks if a year is a leap year.\")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "vUJR4Pno-LGK"
|
|
},
|
|
"source": [
|
|
"This follow-up prompt shows how the model responds based on the previous prompt:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6Fn69TurZ9DB"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = chat.send_message(\"Write a unit test of the generated function.\")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IX9vzE6SNPVe"
|
|
},
|
|
"source": [
|
|
"### ✅ Safety Filters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "H62XABiaYx0S"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"system_instruction = \"Be as mean and hateful as possible.\"\n",
|
|
"\n",
|
|
"prompt = \"\"\"\n",
|
|
"Write a list of 5 disrespectful things that I might say to the universe after stubbing my toe in the dark.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"safety_settings = [\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_HARASSMENT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_HATE_SPEECH,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" safety_settings=safety_settings,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# Response will be `None` if it is blocked.\n",
|
|
"print(response.text)\n",
|
|
"# Finish Reason will be `SAFETY` if it is blocked.\n",
|
|
"print(response.candidates[0].finish_reason)\n",
|
|
"# Safety Ratings show the levels for each filter.\n",
|
|
"for safety_rating in response.candidates[0].safety_ratings:\n",
|
|
" print(safety_rating)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "arLJE4wOuhh6"
|
|
},
|
|
"source": [
|
|
"### ✅ Send Asynchronous Requests\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "gSReaLazs-dP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = await client.aio.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Compose a song about the adventures of a time-traveling squirrel.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rZV2TY5Pa3Dd"
|
|
},
|
|
"source": [
|
|
"### ✅ Multimodality\n",
|
|
"\n",
|
|
"- If your content is stored in [Google Cloud Storage](https://cloud.google.com/storage), you can use the `from_uri` method to create a `Part` object.\n",
|
|
"- If your content is stored in your local file system, you can read it in as bytes data and use the `from_bytes` method to create a `Part` object.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "w4npg1tNTYB9"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Image**\n",
|
|
"\n",
|
|
"In this example, we will use an image stored locally.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "umhZ61lrSyJh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Download and open an image locally.\n",
|
|
"! wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/meal.png\n",
|
|
"\n",
|
|
"with open(\"meal.png\", \"rb\") as f:\n",
|
|
" image = f.read()\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_bytes(data=image, mime_type=\"image/png\"),\n",
|
|
" \"Write a short and engaging blog post based on this picture.\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e7b6170c9255"
|
|
},
|
|
"source": [
|
|
"#### 💡 **PDF**\n",
|
|
"\n",
|
|
"In this example, we will use a PDF Document stored on Google Cloud Storage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "1d58b914d798"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/pdf/1706.03762v7.pdf\",\n",
|
|
" mime_type=\"application/pdf\",\n",
|
|
" ),\n",
|
|
" \"Summarize the document.\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "b247a2ee0e38"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Audio**\n",
|
|
"\n",
|
|
"This example uses an audio file stored at a general web URL.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cbe8c9c67ba7"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://traffic.libsyn.com/secure/e780d51f-f115-44a6-8252-aed9216bb521/KPOD242.mp3\",\n",
|
|
" mime_type=\"audio/mpeg\",\n",
|
|
" ),\n",
|
|
" \"Write a summary of this podcast episode.\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" audio_timestamp=True,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8D3_oNUTuW2q"
|
|
},
|
|
"source": [
|
|
"#### 💡 **YouTube Video**\n",
|
|
"\n",
|
|
"This example is the YouTube video [Google — 25 Years in Search: The Most Searched](https://www.youtube.com/watch?v=3KtWfp0UopM).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "l7-w8G_2wAOw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://www.youtube.com/watch?v=3KtWfp0UopM\",\n",
|
|
" mime_type=\"video/mp4\",\n",
|
|
" ),\n",
|
|
" \"At what point in the video is Harry Potter shown?\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "df8013cfa7f7"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Web Page (HTTP Support)**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "337793322c91"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://cloud.google.com/vertex-ai/generative-ai/docs\",\n",
|
|
" mime_type=\"text/html\",\n",
|
|
" ),\n",
|
|
" \"Write a summary of this documentation.\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rVlo0mWuZGkQ"
|
|
},
|
|
"source": [
|
|
"### ✅ Structured Output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "O9NhOqV-fzfN"
|
|
},
|
|
"source": [
|
|
"#### 💡 [Pydantic](https://docs.pydantic.dev/latest/) Model Schema support"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "OjSgf2cDN_bG"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"class CountryInfo(BaseModel):\n",
|
|
" name: str\n",
|
|
" population: int\n",
|
|
" capital: str\n",
|
|
" continent: str\n",
|
|
" gdp: int\n",
|
|
" official_language: str\n",
|
|
" total_area_sq_mi: int\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Give me information for the United States.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=CountryInfo,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"# Response as JSON\n",
|
|
"print(response.text)\n",
|
|
"# Response as Pydantic object\n",
|
|
"print(response.parsed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IrKldRiPgPr2"
|
|
},
|
|
"source": [
|
|
"#### 💡 [OpenAPI Schema](https://swagger.io/specification/) support"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bRgilziNgCyo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response_schema = {\n",
|
|
" \"required\": [\n",
|
|
" \"name\",\n",
|
|
" \"population\",\n",
|
|
" \"capital\",\n",
|
|
" \"continent\",\n",
|
|
" \"gdp\",\n",
|
|
" \"official_language\",\n",
|
|
" \"total_area_sq_mi\",\n",
|
|
" ],\n",
|
|
" \"properties\": {\n",
|
|
" \"name\": {\"type\": \"STRING\"},\n",
|
|
" \"population\": {\"type\": \"INTEGER\"},\n",
|
|
" \"capital\": {\"type\": \"STRING\"},\n",
|
|
" \"continent\": {\"type\": \"STRING\"},\n",
|
|
" \"gdp\": {\"type\": \"INTEGER\"},\n",
|
|
" \"official_language\": {\"type\": \"STRING\"},\n",
|
|
" \"total_area_sq_mi\": {\"type\": \"INTEGER\"},\n",
|
|
" },\n",
|
|
" \"type\": \"OBJECT\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Give me information for the United States.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=response_schema,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"# As JSON\n",
|
|
"print(response.text)\n",
|
|
"# As Dict\n",
|
|
"print(response.parsed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "_BsP0vXOY7hg"
|
|
},
|
|
"source": [
|
|
"### ✅ Search as a Tool\n",
|
|
"\n",
|
|
"Gemini 3.1 includes native support for formatting citations directly in the text (e.g., `Spain won Euro 2024... [1](https://...)`)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yeR09J3AZT4U"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Where will the next FIFA World Cup be held?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[types.Tool(google_search=types.GoogleSearch())],\n",
|
|
" ),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "96bc624e2b1a"
|
|
},
|
|
"source": [
|
|
"Use this helper function to print the full citations."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3786d7ab276d"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def print_grounding_data(response: types.GenerateContentResponse) -> None:\n",
|
|
" \"\"\"Print Gemini response with grounding citations in Markdown format.\"\"\"\n",
|
|
" candidate = response.candidates[0] if response.candidates else None\n",
|
|
" metadata = getattr(candidate, \"grounding_metadata\", None)\n",
|
|
"\n",
|
|
" if not metadata:\n",
|
|
" print(\"Response does not contain grounding metadata.\")\n",
|
|
" display(Markdown(response.text))\n",
|
|
" return\n",
|
|
"\n",
|
|
" ENCODING = \"utf-8\"\n",
|
|
" text_bytes = response.text.encode(ENCODING)\n",
|
|
" parts = []\n",
|
|
" last = 0\n",
|
|
"\n",
|
|
" # Insert citation markers into text\n",
|
|
" for support in metadata.grounding_supports or []:\n",
|
|
" end = support.segment.end_index\n",
|
|
" parts.append(text_bytes[last:end].decode(ENCODING))\n",
|
|
" parts.append(\n",
|
|
" \" \" + \"\".join(f\"[{i + 1}]\" for i in support.grounding_chunk_indices)\n",
|
|
" )\n",
|
|
" last = end\n",
|
|
"\n",
|
|
" parts.append(text_bytes[last:].decode(ENCODING))\n",
|
|
" parts.append(\"\\n\\n----\\n## Grounding Sources\\n\")\n",
|
|
"\n",
|
|
" # Grounding chunks\n",
|
|
" if chunks := metadata.grounding_chunks:\n",
|
|
" parts.append(\"### Grounding Chunks\\n\")\n",
|
|
" for i, chunk in enumerate(chunks, 1):\n",
|
|
" if not (ctx := chunk.web or chunk.retrieved_context or chunk.maps):\n",
|
|
" continue\n",
|
|
" uri = (\n",
|
|
" (ctx.uri or \"\")\n",
|
|
" .replace(\"gs://\", \"https://storage.googleapis.com/\", 1)\n",
|
|
" .replace(\" \", \"%20\")\n",
|
|
" )\n",
|
|
" parts.append(f\"{i}. [{ctx.title or 'Source'}]({uri})\\n\")\n",
|
|
" if getattr(ctx, \"place_id\", None):\n",
|
|
" parts.append(f\" - Place ID: `{ctx.place_id}`\\n\\n\")\n",
|
|
" if getattr(ctx, \"text\", None):\n",
|
|
" parts.append(f\"{ctx.text}\\n\\n\")\n",
|
|
"\n",
|
|
" # Queries\n",
|
|
" if metadata.web_search_queries:\n",
|
|
" parts.append(f\"\\n**Web Search Queries:** {metadata.web_search_queries}\\n\")\n",
|
|
" if metadata.search_entry_point:\n",
|
|
" parts.append(\n",
|
|
" f\"\\n**Search Entry Point:**\\n{metadata.search_entry_point.rendered_content}\\n\"\n",
|
|
" )\n",
|
|
" elif metadata.retrieval_queries:\n",
|
|
" parts.append(f\"\\n**Retrieval Queries:** {metadata.retrieval_queries}\\n\")\n",
|
|
"\n",
|
|
" display(Markdown(\"\".join(parts)))\n",
|
|
"\n",
|
|
"\n",
|
|
"print_grounding_data(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MhDs2X3o0neK"
|
|
},
|
|
"source": [
|
|
"### ✅ Code Execution"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "1W-3c7sy0nyz"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define code execution tool\n",
|
|
"code_execution_tool = types.Tool(code_execution=types.ToolCodeExecution())\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Calculate 20th fibonacci number. Then find the nearest palindrome to it.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[code_execution_tool],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"\n",
|
|
"## Code\n",
|
|
"\n",
|
|
"```py\n",
|
|
"{response.executable_code}\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Output\n",
|
|
"\n",
|
|
"```\n",
|
|
"{response.code_execution_result}\n",
|
|
"```\n",
|
|
"\"\"\"\n",
|
|
" )\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "cr566SQAOmUB"
|
|
},
|
|
"source": [
|
|
"### ✅ URL Context"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "l9R3XD6dNYjP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the Url context tool\n",
|
|
"url_context_tool = types.Tool(url_context=types.UrlContext)\n",
|
|
"\n",
|
|
"url = \"https://blog.google/technology/developers/introducing-gemini-cli-open-source-ai-agent/\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=f\"Summarize this document: {url}\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[url_context_tool],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))\n",
|
|
"print(response.candidates[0].grounding_metadata)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "T0pb-Kh1xEHU"
|
|
},
|
|
"source": [
|
|
"### ✅ Function Calling\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "aRR8HZhLlR-E"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_weather(location: str):\n",
|
|
" \"\"\"Get the current weather in a specific location.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" location: The city and state, e.g. San Francisco, CA or a zip code.\n",
|
|
" \"\"\"\n",
|
|
" # This is a placeholder for a real API call.\n",
|
|
" return {\"temperature\": \"32\", \"unit\": \"celsius\"}\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What is the weather like in Toronto?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yjoQryztjBx2"
|
|
},
|
|
"source": [
|
|
"### ✅ Count Tokens"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yOxaKemujDfQ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Count tokens\n",
|
|
"response = client.models.count_tokens(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"why is the sky blue?\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SrHw5Ip1lerP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compute tokens\n",
|
|
"response = client.models.compute_tokens(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"why is the sky blue?\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"collapsed_sections": [
|
|
"nEjLXZ-GTqb9",
|
|
"XWJqKKVNTEHD"
|
|
],
|
|
"name": "intro_gemini_3_1_pro.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|