wehub-resource-sync
1632 lines
57 KiB
Plaintext
1632 lines
57 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"cellView": "form",
|
|
"id": "2RexaI2BoZZy"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rJ43O7oc6xGP"
|
|
},
|
|
"source": [
|
|
"# Intro to Gemini 3 Flash (Preview)\n",
|
|
"\n",
|
|
"> **NOTE:** On June 15, 2026, we will remove access to Gemini 3 Flash Preview on Gemini Enterprise Agent Platform for new and inactive projects only. We will also turn off model tuning on this date. If you have a new project, please use Gemini 3.5 Flash (GA).\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgetting-started%2Fintro_gemini_3_flash.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/getting-started/intro_gemini_3_flash.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_3_flash.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "uYusEvgifp9U"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |\n",
|
|
"| [Holt Skinner](https://github.com/holtskinner) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8sg1R6UQ7Q3t"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook serves as a quickstart guide for developers to begin interacting with the **[Gemini 3 Flash](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-flash)** model via the Google Gen AI SDK on Vertex AI. It is designed to demonstrate key model capabilities and showcase new API features."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gPiTOAHURvTM"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-tn3uw268iw4"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK for Python\n",
|
|
"\n",
|
|
"Gemini 3 Flash API features require Gen AI SDK for Python version `1.56.0` or later."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3CaXL22k8iw4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xW5WwfAOfp9V"
|
|
},
|
|
"source": [
|
|
"### Import libraries\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "o0JUCkvVfp9V"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"from IPython.display import HTML, Markdown, display\n",
|
|
"from google import genai\n",
|
|
"from google.genai import types\n",
|
|
"from pydantic import BaseModel"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SY-GRP3m8iw5"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "06RAe75C8iw5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "wkInCpT9fp9V"
|
|
},
|
|
"source": [
|
|
"### Autenticate your Google Cloud Project for Vertex AI\n",
|
|
"\n",
|
|
"You can use a Google Cloud Project or an API Key for authentication. This tutorial uses a Google Cloud Project.\n",
|
|
"\n",
|
|
"- [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "z-jVOPQVfp9V"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = \"global\"\n",
|
|
"\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eXHJi5B6P5vd"
|
|
},
|
|
"source": [
|
|
"### Choose a Gemini 3 model\n",
|
|
"\n",
|
|
"Use `gemini-3-flash-preview` in this tutorial. Learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "eGbUoVu-931G"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-3-flash-preview\" # @param [\"gemini-3-flash-preview\"] {type: \"string\"}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "sHanNp-jfp9W"
|
|
},
|
|
"source": [
|
|
"## 🚀 Quickstart\n",
|
|
"\n",
|
|
"By default, Gemini 3 uses dynamic thinking to reason through prompts. For faster, lower-latency responses when complex reasoning isn't required, you can constrain the model's thinking level by setting parameter `thinking_level` to `minimal`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SzYgqNCNfp9W"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"How does AI work?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" # For faster and lower-latency responses\n",
|
|
" thinking_level=types.ThinkingLevel.MINIMAL\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ko-nGGunfp9W"
|
|
},
|
|
"source": [
|
|
"## 🚨 New API Features"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Mwpbk8wOfp9W"
|
|
},
|
|
"source": [
|
|
"### 1️⃣ Thinking Level\n",
|
|
"\n",
|
|
"The `thinking_level` parameter allows you to specify a thinking budget for the model's response generation. By selecting one of two states, you can explicitly balance the trade-offs between response quality/reasoning complexity and latency/cost.\n",
|
|
"\n",
|
|
"- **`MINIMAL`:** Constrains the model to use as few tokens as possible for thinking and is best used for low-complexity tasks that wouldn't benefit from extensive reasoning. `MINIMAL` is as close as possible to a zero budget for thinking but still requires thought signatures.\n",
|
|
"- **`LOW`:** Constrains the model to use fewer tokens for thinking and is suitable for simpler tasks where extensive reasoning is not required.\n",
|
|
"- **`MEDIUM`:** Offers a balanced approach suitable for tasks of moderate complexity that benefit from reasoning but don't require deep, multi-step planning. It provides more reasoning capability than `LOW` while maintaining lower latency than `HIGH`.\n",
|
|
"- **`HIGH`:** Maximizes reasoning depth. The model may take significantly longer to reach a first token, but the output will be more thoroughly vetted.\n",
|
|
"\n",
|
|
"#### ⚠️ Notes\n",
|
|
"\n",
|
|
"- If `thinking_level` is not specified, the model defaults to `HIGH`, which is a dynamic setting that adjusts based on prompt complexity.\n",
|
|
"- You cannot use both `thinking_level` and the legacy `thinking_budget` parameter in the same request. Doing so will return a 400 error.\n",
|
|
"- The OpenAI Chat Completions API `reasoning_effort` `medium` maps to `thinking_level` `high`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "0UMeFK5Tfp9W"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"You are tasked with implementing the classic Thread-Safe Double-Checked Locking (DCL) Singleton pattern in modern C++. This task is non-trivial and requires specialized concurrency knowledge to prevent memory reordering issues.\n",
|
|
"\n",
|
|
"Write a complete, runnable C++ program named `dcl_singleton.cpp` that defines a class `Singleton` with a private constructor and a static `getInstance()` method.\n",
|
|
"\n",
|
|
"Your solution MUST adhere to the following strict constraints:\n",
|
|
"1. The Singleton instance pointer (`static Singleton*`) must be wrapped in `std::atomic` to correctly manage memory visibility across threads.\n",
|
|
"2. The `getInstance()` method must use `std::memory_order_acquire` when reading the instance pointer in the outer check.\n",
|
|
"3. The instance creation and write-back must use `std::memory_order_release` when writing to the atomic pointer.\n",
|
|
"4. A standard `std::mutex` must be used only to protect the critical section (the actual instantiation).\n",
|
|
"5. The `main` function must demonstrate safe, concurrent access by launching at least three threads, each calling `Singleton::getInstance()`, and printing the address of the returned instance to prove all threads received the same object.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.HIGH # Dynamic thinking for high reasoning tasks\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6OFja2Qvfp9X"
|
|
},
|
|
"source": [
|
|
"### 2️⃣ Media Resolution\n",
|
|
"\n",
|
|
"Gemini 3 introduces granular control over multimodal vision processing via the `media_resolution` parameter. Higher resolutions improve the model's ability to read fine text or identify small details, but increase token usage and latency. The `media_resolution` parameter determines the maximum number of tokens allocated per input image or video frame.\n",
|
|
"\n",
|
|
"**⚠️ Note**: Because media resolution directly impacts token count, you may need to lower the resolution (e.g., to `low`) to fit very long inputs, such as long videos or extensive documents."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "71DgyBXAKZtM"
|
|
},
|
|
"source": [
|
|
"You can set the resolution to `LOW`, `MEDIUM`, `HIGH`, or `ULTRA_HIGH` per individual media part, as example:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Lc7i2xBEfp9X"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/jpeg\",\n",
|
|
" ),\n",
|
|
" media_resolution=types.PartMediaResolution(\n",
|
|
" level=types.PartMediaResolutionLevel.MEDIA_RESOLUTION_ULTRA_HIGH\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/video/behind_the_scenes_pixel.mp4\",\n",
|
|
" mime_type=\"video/mp4\",\n",
|
|
" ),\n",
|
|
" media_resolution=types.PartMediaResolution(\n",
|
|
" level=types.PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" \"When does the image appear in the video? What is the context?\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW)\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MCBrvJiiKcyX"
|
|
},
|
|
"source": [
|
|
"Or set it globally via `GenerateContentConfig`. If unspecified, the model uses optimal defaults based on the media type.\n",
|
|
"\n",
|
|
"**⚠️ NOTE:** `ULTRA_HIGH` is only supported for individual parts."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "MGBBPXJCKLYX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part(\n",
|
|
" file_data=types.FileData(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/jpeg\",\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
" \"What is in the image?\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0c00UyBkfp9X"
|
|
},
|
|
"source": [
|
|
"### 3️⃣ Thought Signature\n",
|
|
"\n",
|
|
"[Thought signatures](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thinking#signatures) are encrypted tokens that preserve the model's reasoning state during multi-turn conversations, specifically when using Function Calling.\n",
|
|
"\n",
|
|
"When a thinking model decides to call an external tool, it pauses its internal reasoning process. The thought signature acts as a \"save state\", allowing the model to resume its chain of thought seamlessly once you provide the function's result.\n",
|
|
"\n",
|
|
"Gemini 3 enforces stricter validation and updated handling on thought signatures which were originally introduced in Gemini 2.5. To ensure the model maintains context across multiple turns of a conversation, you must return the thought signatures in your subsequent requests.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Y7wuJBfkRnd8"
|
|
},
|
|
"source": [
|
|
"#### Automatic Handling of Thought Signatures (Recommended)\n",
|
|
"\n",
|
|
"If you are using the Google Gen AI SDKs (Python, Node.js, Go, Java) or OpenAI Chat Completions API, and utilizing the standard chat history features or appending the full model response, thought signatures are handled automatically. You do not need to make any changes to your code.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nEjLXZ-GTqb9"
|
|
},
|
|
"source": [
|
|
"####**Example 1**: Automatic Function Calling\n",
|
|
"\n",
|
|
"When using the Gen AI SDK in automatic function calling, thought signatures are handled automatically.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "riOjhr8Rfp9Y"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_weather(city: str):\n",
|
|
" \"\"\"Gets the weather in a city.\"\"\"\n",
|
|
" if \"london\" in city.lower():\n",
|
|
" return \"Rainy\"\n",
|
|
" if \"new york\" in city.lower():\n",
|
|
" return \"Sunny\"\n",
|
|
" return \"Cloudy\"\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What's the weather in London and New York?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# The SDK handles the function calls and thought signatures, and returns the final text\n",
|
|
"display(Markdown(\"## Final Response\"))\n",
|
|
"display(Markdown(response.text))\n",
|
|
"\n",
|
|
"# Print function calling history\n",
|
|
"hist_turn = response.automatic_function_calling_history[1]\n",
|
|
"print(\"\\nFunction Call 1:\", hist_turn.parts[1].function_call.name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XWJqKKVNTEHD"
|
|
},
|
|
"source": [
|
|
"#### **Example 2**: Manual Function Calling\n",
|
|
"\n",
|
|
"When using the Gen AI SDK in manual function calling, thought signatures are also handled automatically if you append the full model response in sequential model requests."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "EVXwCkQET4qQ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Define your tool\n",
|
|
"get_weather_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_weather\",\n",
|
|
" description=\"Gets the current weather temperature for a given location.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\"location\": {\"type\": \"string\"}},\n",
|
|
" \"required\": [\"location\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"get_weather_tool = types.Tool(function_declarations=[get_weather_declaration])\n",
|
|
"\n",
|
|
"# 2. Send a message that triggers the tool\n",
|
|
"prompt = \"What's the weather like in London?\"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" thinking_config=types.ThinkingConfig(include_thoughts=True),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 3. Handle the function call\n",
|
|
"function_call = response.function_calls[0]\n",
|
|
"location = function_call.args[\"location\"]\n",
|
|
"print(f\"Model wants to call: {function_call.name}\")\n",
|
|
"\n",
|
|
"# Execute your tool (e.g., call an API)\n",
|
|
"# (This is a mock response for the example)\n",
|
|
"print(f\"Calling external tool for: {location}\")\n",
|
|
"function_response_data = {\n",
|
|
" \"location\": location,\n",
|
|
" \"temperature\": \"30C\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"# 4. Send the tool's result back\n",
|
|
"# Append this turn's messages to history for a final response.\n",
|
|
"# The `content` object automatically attaches the required thought_signature behind the scenes.\n",
|
|
"history = [\n",
|
|
" types.Content(role=\"user\", parts=[types.Part(text=prompt)]),\n",
|
|
" response.candidates[0].content, # Signature preserved here\n",
|
|
" types.Content(\n",
|
|
" role=\"tool\",\n",
|
|
" parts=[\n",
|
|
" types.Part.from_function_response(\n",
|
|
" name=function_call.name,\n",
|
|
" response=function_response_data,\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response_2 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=history,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" thinking_config=types.ThinkingConfig(include_thoughts=True),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 5. Get the final, natural-language answer\n",
|
|
"print(f\"\\nFinal model response: {response_2.text}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "J6Bn9rRSUexu"
|
|
},
|
|
"source": [
|
|
"#### Manual Handling of Thought Signatures\n",
|
|
"\n",
|
|
"If you are interacting with the API directly or managing raw JSON payloads, you must correctly handle the `thought_signature` included in the model's turn.\n",
|
|
"You must return this signature in the exact part where it was received when sending the conversation history back.\n",
|
|
"\n",
|
|
"⚠️ If proper signatures are not returned, the model will return a 400 Error `\"Function Call in the content block is missing a thought_signature\"`.\n",
|
|
"\n",
|
|
"See [documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#thinking) for more details."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "eNKS8D9Ofp9Y"
|
|
},
|
|
"source": [
|
|
"### 4️⃣ Streaming Function Calling\n",
|
|
"\n",
|
|
"You can use streaming partial function call arguments to improve streaming experience on tool use. This feature can be enabled by explicitly setting `stream_function_call_arguments` to true."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "VY_Ir0OdID5v"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"get_weather_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_weather\",\n",
|
|
" description=\"Gets the current weather temperature for a given location.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\"location\": {\"type\": \"string\"}},\n",
|
|
" \"required\": [\"location\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"get_weather_tool = types.Tool(function_declarations=[get_weather_declaration])\n",
|
|
"\n",
|
|
"\n",
|
|
"for chunk in client.models.generate_content_stream(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What's the weather in London and New York?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather_tool],\n",
|
|
" tool_config=types.ToolConfig(\n",
|
|
" function_calling_config=types.FunctionCallingConfig(\n",
|
|
" mode=types.FunctionCallingConfigMode.AUTO,\n",
|
|
" stream_function_call_arguments=True,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
"):\n",
|
|
" function_call = chunk.function_calls[0]\n",
|
|
" if function_call and function_call.name:\n",
|
|
" print(f\"{function_call.name}\")\n",
|
|
" print(f\"will_continue={function_call.will_continue}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gyoTiy1Cfp9Y"
|
|
},
|
|
"source": [
|
|
"### 5️⃣ Multimodal Function Responses\n",
|
|
"\n",
|
|
"Multimodal function calling allows users to have function responses containing multimodal objects allowing for improved utilization of function calling capabilities of the model. Currently function calling only supports text based function responses.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "nX7EDrhaPkVh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# 1. Define the function tool\n",
|
|
"get_image_declaration = types.FunctionDeclaration(\n",
|
|
" name=\"get_image\",\n",
|
|
" description=\"Retrieves the image file reference for a specific order item.\",\n",
|
|
" parameters={\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"item_name\": {\n",
|
|
" \"type\": \"string\",\n",
|
|
" \"description\": \"The name or description of the item ordered (e.g., 'green shirt').\",\n",
|
|
" }\n",
|
|
" },\n",
|
|
" \"required\": [\"item_name\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"tool_config = types.Tool(function_declarations=[get_image_declaration])\n",
|
|
"\n",
|
|
"# 2. Send a message that triggers the tool\n",
|
|
"prompt = \"Show me the green shirt I ordered last month.\"\n",
|
|
"response_1 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[prompt],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[tool_config],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# 3. Handle the function call\n",
|
|
"function_call = response_1.function_calls[0]\n",
|
|
"requested_item = function_call.args[\"item_name\"]\n",
|
|
"print(f\"Model wants to call: {function_call.name}\")\n",
|
|
"\n",
|
|
"# Execute your tool (e.g., call an API)\n",
|
|
"# (This is a mock response for the example)\n",
|
|
"print(f\"Calling external tool for: {requested_item}\")\n",
|
|
"\n",
|
|
"function_response_data = {\n",
|
|
" \"image_ref\": {\"$ref\": \"dress.jpg\"},\n",
|
|
"}\n",
|
|
"\n",
|
|
"function_response_multimodal_data = types.FunctionResponsePart(\n",
|
|
" file_data=types.FunctionResponseFileData(\n",
|
|
" mime_type=\"image/png\",\n",
|
|
" display_name=\"dress.jpg\",\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/image/dress.jpg\",\n",
|
|
" )\n",
|
|
")\n",
|
|
"\n",
|
|
"# 4. Send the tool's result back\n",
|
|
"# Append this turn's messages to history for a final response.\n",
|
|
"history = [\n",
|
|
" types.Content(role=\"user\", parts=[types.Part(text=prompt)]),\n",
|
|
" response_1.candidates[0].content,\n",
|
|
" types.Content(\n",
|
|
" role=\"tool\",\n",
|
|
" parts=[\n",
|
|
" types.Part.from_function_response(\n",
|
|
" name=function_call.name,\n",
|
|
" response=function_response_data,\n",
|
|
" parts=[function_response_multimodal_data],\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response_2 = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=history,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[tool_config], thinking_config=types.ThinkingConfig(include_thoughts=True)\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"\\nFinal model response: {response_2.text}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xJH7TdIVfp9Z"
|
|
},
|
|
"source": [
|
|
"## ⭐️ Supported Features"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MELl7Nmpfp9Z"
|
|
},
|
|
"source": [
|
|
"### ✅ Set System Instructions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "GBCwUvdTfp9Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"system_instruction = \"\"\"\n",
|
|
" You are a helpful language translator.\n",
|
|
" Your mission is to translate text in English to Spanish.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"prompt = \"\"\"\n",
|
|
" User input: I like bagels.\n",
|
|
" Answer:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ddMgdnZ3fp9Z"
|
|
},
|
|
"source": [
|
|
"### ✅ Configure Model Parameters\n",
|
|
"\n",
|
|
"⚠️ **Notes**: For Gemini 3, we strongly recommend keeping the `temperature` parameter at its default value of `1.0`.\n",
|
|
"\n",
|
|
"While previous models often benefitted from tuning `temperature` to control creativity versus determinism, Gemini 3's reasoning capabilities are optimized for the default setting.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "MMiH7-ilfp9Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Tell me how the internet works, but pretend I'm a puppy who only understands squeaky toys.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" temperature=1.0,\n",
|
|
" max_output_tokens=500,\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.MINIMAL,\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "lxZ--hIvfp9Z"
|
|
},
|
|
"source": [
|
|
"### ✅ Generate Content Stream"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "xDDqwvWlfp9Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.\n",
|
|
"How much does the ball cost?\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"for chunk in client.models.generate_content_stream(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
"):\n",
|
|
" print(chunk.text, end=\"\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kbQAJXx6fp9Z"
|
|
},
|
|
"source": [
|
|
"### ✅ Thought Summaries\n",
|
|
"\n",
|
|
"You can include thought summaries in model response by setting `include_thoughts` to `true`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LyfDWcvufp9Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"How many R's are in the word strawberry?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" include_thoughts=True,\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"for part in response.candidates[0].content.parts:\n",
|
|
" if part.thought:\n",
|
|
" display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"## Thoughts:\n",
|
|
" {part.text}\n",
|
|
" \"\"\"\n",
|
|
" )\n",
|
|
" )\n",
|
|
" else:\n",
|
|
" display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"## Answer:\n",
|
|
" {part.text}\n",
|
|
" \"\"\"\n",
|
|
" )\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xyY3tHCGfp9Z"
|
|
},
|
|
"source": [
|
|
"### ✅ Multi-turn Chat\n",
|
|
"\n",
|
|
"Conversation: Starting and maintaining a Multi-turn Chat history."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ys-5lm0nfp9Z"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"chat = client.chats.create(\n",
|
|
" model=MODEL_ID,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.MINIMAL)\n",
|
|
" ),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "I2M0EaJOfp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = chat.send_message(\n",
|
|
" \"Write a Python function that checks if a year is a leap year.\"\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ePgqfpFXfp9a"
|
|
},
|
|
"source": [
|
|
"This follow-up prompt shows how the model responds based on the previous prompt:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AqA_5KiTfp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = chat.send_message(\"Write a unit test of the generated function.\")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dn_pfhEyfp9a"
|
|
},
|
|
"source": [
|
|
"### ✅ Safety Filters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yud7PFzafp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"system_instruction = \"Be as mean and hateful as possible.\"\n",
|
|
"\n",
|
|
"prompt = \"\"\"\n",
|
|
"Write a list of 5 disrespectful things that I might say to the universe after stubbing my toe in the dark.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"safety_settings = [\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_HARASSMENT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_HATE_SPEECH,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
" types.SafetySetting(\n",
|
|
" category=types.HarmCategory.HARM_CATEGORY_JAILBREAK,\n",
|
|
" threshold=types.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=prompt,\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" safety_settings=safety_settings,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"# Response will be `None` if it is blocked.\n",
|
|
"print(response.text)\n",
|
|
"# Finish Reason will be `SAFETY` if it is blocked.\n",
|
|
"print(response.candidates[0].finish_reason)\n",
|
|
"# Safety Ratings show the levels for each filter.\n",
|
|
"for safety_rating in response.candidates[0].safety_ratings:\n",
|
|
" if safety_rating.blocked:\n",
|
|
" print(safety_rating)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SONkqnLRfp9a"
|
|
},
|
|
"source": [
|
|
"### ✅ Send Asynchronous Requests\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "PhPZyeN0fp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = await client.aio.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Compose a song about the adventures of a time-traveling squirrel.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.MINIMAL # Dynamic thinking for high reasoning tasks\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kYSiUnZzfp9a"
|
|
},
|
|
"source": [
|
|
"### ✅ Multimodality\n",
|
|
"\n",
|
|
"- If your content is stored in [Google Cloud Storage](https://cloud.google.com/storage) or on a public website, you can use the `from_uri` method to create a `Part` object.\n",
|
|
"- If your content is stored in your local file system, you can read it in as bytes data and use the `from_bytes` method to create a `Part` object.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "wdSJ_Sunfp9a"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Image**\n",
|
|
"\n",
|
|
"In this example, we will use an image stored locally.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "fTMhSuxWfp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Download and open an image locally.\n",
|
|
"! wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/meal.png\n",
|
|
"\n",
|
|
"with open(\"meal.png\", \"rb\") as f:\n",
|
|
" image = f.read()\n",
|
|
" display(Image(image, width=500))\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_bytes(data=image, mime_type=\"image/png\"),\n",
|
|
" \"Write a short and engaging blog post based on this picture.\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.MINIMAL\n",
|
|
" )\n",
|
|
" media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "P__jJRLVfp9a"
|
|
},
|
|
"source": [
|
|
"#### 💡 **PDF**\n",
|
|
"\n",
|
|
"In this example, we will use a PDF Document stored on Google Cloud Storage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NCT96Py3fp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/pdf/1706.03762v7.pdf\",\n",
|
|
" mime_type=\"application/pdf\",\n",
|
|
" ),\n",
|
|
" \"Summarize the document.\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(\n",
|
|
" thinking_level=types.ThinkingLevel.LOW,\n",
|
|
" ),\n",
|
|
" media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yDYVS10kfp9a"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Audio**\n",
|
|
"\n",
|
|
"This example uses an audio file stored at a general web URL.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "KDosCL1Ofp9a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://traffic.libsyn.com/secure/e780d51f-f115-44a6-8252-aed9216bb521/KPOD242.mp3\",\n",
|
|
" mime_type=\"audio/mpeg\",\n",
|
|
" ),\n",
|
|
" \"Write a summary of this podcast episode.\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" audio_timestamp=True,\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "jeoh77yDfp9b"
|
|
},
|
|
"source": [
|
|
"#### 💡 **YouTube Video**\n",
|
|
"\n",
|
|
"This example is the YouTube video [Google — 25 Years in Search: The Most Searched](https://www.youtube.com/watch?v=3KtWfp0UopM).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mZ4wow7Rfp9b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://www.youtube.com/watch?v=3KtWfp0UopM\",\n",
|
|
" mime_type=\"video/mp4\",\n",
|
|
" ),\n",
|
|
" \"At what point in the video is Harry Potter shown?\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW)\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "AgnjHBphfp9b"
|
|
},
|
|
"source": [
|
|
"#### 💡 **Web Page (HTTP Support)**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "H7fj17B9fp9b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" types.Part.from_uri(\n",
|
|
" file_uri=\"https://cloud.google.com/vertex-ai/generative-ai/docs\",\n",
|
|
" mime_type=\"text/html\",\n",
|
|
" ),\n",
|
|
" \"Write a summary of this documentation.\",\n",
|
|
" ],\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW)\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NykdwhDHfp9b"
|
|
},
|
|
"source": [
|
|
"### ✅ Structured Output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "O9NhOqV-fzfN"
|
|
},
|
|
"source": [
|
|
"#### 💡 [Pydantic](https://docs.pydantic.dev/latest/) Model Schema support"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "IWx_KWGyfp9b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"class CountryInfo(BaseModel):\n",
|
|
" name: str\n",
|
|
" population: int\n",
|
|
" capital: str\n",
|
|
" continent: str\n",
|
|
" gdp: int\n",
|
|
" official_language: str\n",
|
|
" total_area_sq_mi: int\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Give me information for the United States.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=CountryInfo,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"# Response as JSON\n",
|
|
"print(response.text)\n",
|
|
"# Response as Pydantic object\n",
|
|
"print(response.parsed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "i7lkv2CsCZEq"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"class CountryInfo(BaseModel):\n",
|
|
" name: str\n",
|
|
" population: int\n",
|
|
" capital: str\n",
|
|
" continent: str\n",
|
|
" gdp: int\n",
|
|
" official_language: str\n",
|
|
" total_area_sq_mi: int\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Give me information for the United States.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=CountryInfo,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"# Response as JSON\n",
|
|
"print(response.text)\n",
|
|
"# Response as Pydantic object\n",
|
|
"print(response.parsed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IrKldRiPgPr2"
|
|
},
|
|
"source": [
|
|
"#### 💡 [OpenAPI Schema](https://swagger.io/specification/) support"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bRgilziNgCyo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response_schema = {\n",
|
|
" \"required\": [\n",
|
|
" \"name\",\n",
|
|
" \"population\",\n",
|
|
" \"capital\",\n",
|
|
" \"continent\",\n",
|
|
" \"gdp\",\n",
|
|
" \"official_language\",\n",
|
|
" \"total_area_sq_mi\",\n",
|
|
" ],\n",
|
|
" \"properties\": {\n",
|
|
" \"name\": {\"type\": \"STRING\"},\n",
|
|
" \"population\": {\"type\": \"INTEGER\"},\n",
|
|
" \"capital\": {\"type\": \"STRING\"},\n",
|
|
" \"continent\": {\"type\": \"STRING\"},\n",
|
|
" \"gdp\": {\"type\": \"INTEGER\"},\n",
|
|
" \"official_language\": {\"type\": \"STRING\"},\n",
|
|
" \"total_area_sq_mi\": {\"type\": \"INTEGER\"},\n",
|
|
" },\n",
|
|
" \"type\": \"OBJECT\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Give me information for the United States.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=response_schema,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"# As JSON\n",
|
|
"print(response.text)\n",
|
|
"# As Dict\n",
|
|
"print(response.parsed)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tkwGORmQfp9b"
|
|
},
|
|
"source": [
|
|
"### ✅ Search as a Tool\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SnTT3W6Ifp9b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Where will the next FIFA World Cup be held?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[types.Tool(google_search=types.GoogleSearch())],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))\n",
|
|
"print(response.candidates[0].grounding_metadata.grounding_chunks)\n",
|
|
"display(\n",
|
|
" HTML(response.candidates[0].grounding_metadata.search_entry_point.rendered_content)\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "CjVbPV9wfp9b"
|
|
},
|
|
"source": [
|
|
"### ✅ Code Execution"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "0rDZz6tdfp9b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define code execution tool\n",
|
|
"code_execution_tool = types.Tool(code_execution=types.ToolCodeExecution())\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"Calculate 20th fibonacci number. Then find the nearest palindrome to it.\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[code_execution_tool],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(\n",
|
|
" Markdown(\n",
|
|
" f\"\"\"\n",
|
|
"## Code\n",
|
|
"\n",
|
|
"```py\n",
|
|
"{response.executable_code}\n",
|
|
"```\n",
|
|
"\n",
|
|
"### Output\n",
|
|
"\n",
|
|
"```\n",
|
|
"{response.code_execution_result}\n",
|
|
"```\n",
|
|
"\"\"\"\n",
|
|
" )\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WSfbJod3fp9b"
|
|
},
|
|
"source": [
|
|
"### ✅ URL Context"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AbA6x42lfp9c"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define the Url context tool\n",
|
|
"url_context_tool = types.Tool(url_context=types.UrlContext)\n",
|
|
"\n",
|
|
"url = \"https://blog.google/technology/developers/introducing-gemini-cli-open-source-ai-agent/\"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=f\"Summarize this document: {url}\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[url_context_tool],\n",
|
|
" thinking_config=types.ThinkingConfig(thinking_level=types.ThinkingLevel.LOW),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))\n",
|
|
"print(response.candidates[0].grounding_metadata)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "x7Bgf4vVfp9c"
|
|
},
|
|
"source": [
|
|
"### ✅ Function Calling\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "B5Z5miUefp9c"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_weather(location: str):\n",
|
|
" \"\"\"Get the current weather in a specific location.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" location: The city and state, e.g. San Francisco, CA or a zip code.\n",
|
|
" \"\"\"\n",
|
|
" # This is a placeholder for a real API call.\n",
|
|
" return {\"temperature\": \"32\", \"unit\": \"celsius\"}\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What is the weather like in Toronto?\",\n",
|
|
" config=types.GenerateContentConfig(\n",
|
|
" tools=[get_weather],\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yjoQryztjBx2"
|
|
},
|
|
"source": [
|
|
"### ✅ Count Tokens"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "yOxaKemujDfQ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Count tokens\n",
|
|
"response = client.models.count_tokens(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"why is the sky blue?\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SrHw5Ip1lerP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Compute tokens\n",
|
|
"response = client.models.compute_tokens(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"why is the sky blue?\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(response)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "intro_gemini_3_flash.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|