wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vision/getting-started/gemini_omni_flash_video_gen.ipynb
+785
View File
@@ -0,0 +1,785 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jv_d8Y2XCyRm"
},
"outputs": [],
"source": [
"# Copyright 2026 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WIgrkCFIC3Fq"
},
"source": [
"# Gemini Omni Flash Video Generation\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fgemini_omni_flash_video_gen.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NbZi7r8HDLd_"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "COhyHClwDPFM"
},
"source": [
"## Overview\n",
"\n",
"Gemini Omni Flash is a model that understands the world around you, allowing you to animate photos or create videos from any input. Built on Gemini's world understanding and native multimodality, Gemini Omni Flash creates outputs that reflect the logic of the real world and lets you shape them step-by-step through natural conversation.\n",
"\n",
"In this tutorial, you'll learn how to use Gemini Omni Flash in Agent Platform with the Google Gen AI SDK to try out the following scenarios:\n",
"\n",
"- Video generation:\n",
" - Text-to-video generation\n",
" - Providing a reference image as a starting frame\n",
" - Supplying reference images to guide video generation\n",
" - Async video generation\n",
"- Video editing:\n",
" - Prompt-based video editing\n",
" - Multi-turn video editing (chat)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gIpYz_L8D9eK"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "axSoEhOzERDm"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mfJgJBMeLAWi"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cfk06hgxEiYS"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NW-We3ATemq6"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YF5OKjXxEpp4"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f1HU9IT_erEb"
},
"outputs": [],
"source": [
"import os\n",
"import base64\n",
"\n",
"import matplotlib.image as img\n",
"import matplotlib.pyplot as plt\n",
"from IPython.display import Video, display\n",
"from google import genai\n",
"from google.genai import interactions"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-yg0mBBaEtXn"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "80On1BQxetma"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P2OgDTYvEzZs"
},
"source": [
"### Define a helper function to display media"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZDN9qlAGe44s"
},
"outputs": [],
"source": [
"def show_video(video_bytes):\n",
" data = base64.b64decode(video_bytes)\n",
" with open(\"sample.mp4\", \"wb\") as out_file:\n",
" out_file.write(data)\n",
" display(Video(\"sample.mp4\", embed=True, width=600))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Cj-ZmZbME24B"
},
"source": [
"### Load the video generation model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vg2SdM1ue7IM"
},
"outputs": [],
"source": [
"omni_model = \"gemini-omni-flash-preview\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hLHGlUYNLIjB"
},
"source": [
"## Video generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WxaeXTZlWgrI"
},
"source": [
"### Generate videos from a text prompt\n",
"\n",
"With Gemini Omni Flash, you can generate videos directly from text prompts via the [Interactions API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/interactions-api). To generate a video in the sample below, specify the following information:\n",
"- **Prompt:** A detailed description of the video you would like to see. For best results, consider attributes such as shot framing, motion, style, lighting, location, and action.\n",
"- **Generation Config:** Within this configuration, you can specify a `VideoConfig` that contains a `task` parameter.\n",
" - **Task:** Valid options are `text_to_video`, `image_to_video`, `reference_to_video`, or `edit`. Make sure to set each task accordingly based on your inputs and desired behavior.\n",
"- **Video Response Format:** Configure `aspect_ratio`, `duration`, and `delivery` parameters. Alternatively, you can specify these configurations through the text prompt.\n",
" - **Aspect ratio:** 16:9 or 9:16\n",
" - **Duration:** 3s - 10s\n",
" - **Delivery:** If you'd like to save your generated videos to GCS, set `delivery=\"uri\"`. Additionally, set `gcs_uri=\"gs://<GCS_BUCKET>\"`, making sure to replace `<GCS_BUCKET>` with your desired bucket path. To find your video's location, look inside the content list and copy the value next to `uri=` in the `VideoContent` object.\n",
"\n",
"\n",
"Notes on output videos:\n",
"- **Audio generation:** Audio will be generated alongside the output video.\n",
"- **Resolution:** 720p\n",
"\n",
"All videos from `gemini-omni-flash-preview` include both [C2PA metadata](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/content-credentials) and a [SynthID watermark](https://deepmind.google/technologies/synthid/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mk2tOv2Rf7e8"
},
"outputs": [],
"source": [
"prompt = \"The words Gemini Omni Flash displayed boldly and centered. The words rapidly and instantly transform, cycling through different styles, fonts, and materials. The words remains perfectly positioned and constant while the colors, textures, and environments instantaneously change around it.\"\n",
"\n",
"interaction = client.interactions.create(\n",
" model=omni_model,\n",
" input=prompt,\n",
" generation_config=interactions.GenerationConfig(\n",
" video_config=interactions.VideoConfig(\n",
" task=\"text_to_video\"\n",
" )\n",
" ),\n",
" response_format=interactions.VideoResponseFormat(\n",
" aspect_ratio=\"16:9\",\n",
" duration=\"9s\",\n",
" # delivery=\"uri\",\n",
" # gcs_uri=\"gs://<GCS_BUCKET>\"\n",
" )\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-aV_ahDKWjPU"
},
"source": [
"### Video generation from a starting image"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v3xtidCsGiCH"
},
"source": [
"#### Download the starting image\n",
"\n",
"You can also generate a video by starting with an input image. In this example, you'll locally download an image that's stored in Google Cloud Storage. If you'd like, you can provide the URL of an image to display it below. If you have a local image you'd like to use, you can specify that in the following steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "p6TPPMW66XCl"
},
"outputs": [],
"source": [
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/suitcase.png"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oi7eAyOhGmoU"
},
"source": [
"If you'd like to use a different local image, modify the file name in `start`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qsneJ_lP6cKk"
},
"outputs": [],
"source": [
"start = \"suitcase.png\"\n",
"\n",
"fig, ax1 = plt.subplots(1, 1, figsize=(12, 6))\n",
"ax1.imshow(img.imread(start))\n",
"ax1.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2_g6-aP661EV"
},
"outputs": [],
"source": [
"prompt = \"A single continuous shot of a hard-shell suitcase rolling. It stops, stands vertically, unzips alongside and then opens in half. The wheels and shell stay solid. White bubbly retro text reading 'Omnicase' pops out of the center. Colorful travel stickers pop up around the text: a plane, palm tree, boat, city skyline, and a train.\"\n",
"\n",
"with open(start, \"rb\") as f:\n",
" img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n",
"\n",
"interaction = client.interactions.create(\n",
" model=omni_model,\n",
" input=[\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" {\"type\": \"image\", \"mime_type\": \"image/png\", \"data\": img_b64},\n",
" ],\n",
" generation_config=interactions.GenerationConfig(\n",
" video_config=interactions.VideoConfig(\n",
" task=\"image_to_video\"\n",
" )\n",
" ),\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YdBtdsaLWo0F"
},
"source": [
"### Reference inputs in video generation\n",
"\n",
"With Gemini Omni Flash, you can use existing media to generate and edit new videos. The model processes these inputs in one of two ways:\n",
"\n",
" - **Source Media:** This content acts as the direct foundation for the final output.\n",
" - *Examples:* In image-to-video, the provided image becomes the literal first frame. In video editing, the original video is modified via natural language prompts while keeping its core structure intact.\n",
" - **Reference Media:** This content acts as a guide or inspiration rather than the literal base. You are asking the model to extract a specific style, subject, or mood to creatively reimagine it.\n",
" - *Examples:* Storyboard frames, product images, style guides, audio jingles, or reference scenery. You can also use reference media for video editing—such as providing an image of a new character or art style to apply to a base video.\n",
"\n",
"\n",
"**NOTE:** Video and audio reference inputs are not currently supported."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jVjssXdqMrMo"
},
"source": [
"#### Download and display reference images\n",
"\n",
"In this example, you'll locally download two images that are stored in Google Cloud Storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5FvshnXiyNKb"
},
"outputs": [],
"source": [
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/woman.jpeg\n",
"\n",
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/arcade-game.png"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "j1jN_J7onPyY"
},
"outputs": [],
"source": [
"character = \"woman.jpeg\"\n",
"product = \"arcade-game.png\"\n",
"\n",
"# Display the images\n",
"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))\n",
"ax1.imshow(img.imread(character))\n",
"ax2.imshow(img.imread(product))\n",
"ax1.axis(\"off\")\n",
"ax2.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1ZLbIRXDJa7T"
},
"source": [
"**Tip:** You can supply storyboard reference images to better direct the video without supplying all details in a text prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LKofc-c5mH_H"
},
"outputs": [],
"source": [
"prompt = \"A woman walks up to the Omni-Sphere game on the side wall of a bowling alley. Bass music begins to play as she starts playing the game by pressing a button. 9:16 aspect ratio. 7 second video.\"\n",
"\n",
"image_files = [character, product]\n",
"\n",
"images_input = []\n",
"for img_path in image_files:\n",
" with open(img_path, \"rb\") as f:\n",
" img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n",
" images_input.append({\"type\": \"image\", \"mime_type\": \"image/jpeg\",\n",
" \"data\": img_b64})\n",
"\n",
"interaction = client.interactions.create(\n",
" model=omni_model,\n",
" input=[\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" *images_input,\n",
" ],\n",
" generation_config=interactions.GenerationConfig(\n",
" video_config=interactions.VideoConfig(\n",
" task=\"reference_to_video\"\n",
" )\n",
" ),\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qWVoa9xpi6l-"
},
"source": [
"### Async video generation\n",
"\n",
"To generate a video that can be checked on later, set the `background` parameter to True. You can then use `interactions.get` to check the status of the initial interaction ID."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "nVOn-pwrgcil"
},
"outputs": [],
"source": [
"import time\n",
"\n",
"prompt = \"Raw cookie ingredients beautifully laid out on a kitchen counter. The view glides smoothly from one ingredient to the next. As each individual ingredient fills the view, a sleek 3D text graphic pops up to label it. The sequence finishes by resting intimately on the rich textures and details of the final ingredient after the label disappears.\"\n",
"\n",
"initial_interaction = client.interactions.create(\n",
" model=omni_model,\n",
" input=prompt,\n",
" background=True,\n",
")\n",
"interaction = initial_interaction\n",
"\n",
"while interaction.status not in [\"completed\", \"failed\"]:\n",
" print(f\"Current Status: {interaction.status}\")\n",
" time.sleep(10)\n",
" interaction = client.interactions.get(id=initial_interaction.id)\n",
"\n",
"if interaction.status == \"completed\":\n",
" contents = []\n",
" steps = getattr(interaction, \"steps\", []) or []\n",
" for step in steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
" show_video(contents[0].data)\n",
"else:\n",
" print(f\"Status: {interaction.status}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VlXnMxwyWsfM"
},
"source": [
"## Video editing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SJs81D2SNf4e"
},
"source": [
"#### Download and display the video\n",
"\n",
"In this example, you'll locally download a video and an image that are stored in Google Cloud Storage.\n",
"\n",
"**Note:** You can supply reference images when editing source videos. The input source video must be under 10 seconds."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DOJhxrMKGfiK"
},
"outputs": [],
"source": [
"from PIL import Image as PIL_IMAGE\n",
"\n",
"video_uri = \"gs://cloud-samples-data/generative-ai/video/dog_day1.mp4\"\n",
"image_uri = \"gs://cloud-samples-data/generative-ai/image/chair-cat.png\"\n",
"\n",
"!gcloud storage cp {video_uri} {\"dog_day1.mp4\"}\n",
"!gcloud storage cp {image_uri} {\"chair-cat.png\"}\n",
"\n",
"display(Video(\"dog_day1.mp4\", embed=True, width=600))\n",
"\n",
"fig, ax1 = plt.subplots(1, 1, figsize=(6, 6))\n",
"ax1.imshow(PIL_IMAGE.open(\"chair-cat.png\"))\n",
"ax1.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5lHP-aNbJwLS"
},
"source": [
"With video editing, you can add, remove, or alter existing objects in the source video. You can also change the content in the video to a different style or reference provided images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lWvtlz6AIRwe"
},
"outputs": [],
"source": [
"prompt = \"Change the dog to the cat. Remove the backpack and add a propeller hat. Change the tennis balls to balls of yarn.\"\n",
"interaction = client.interactions.create(\n",
" model=omni_model,\n",
" input=[\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" {\"type\": \"image\", \"mime_type\": \"image/png\", \"uri\": image_uri},\n",
" {\"type\": \"video\", \"mime_type\": \"video/mp4\", \"uri\": video_uri},\n",
" ],\n",
" generation_config=interactions.GenerationConfig(\n",
" video_config=interactions.VideoConfig(\n",
" task=\"edit\"\n",
" )\n",
" ),\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k1yLTAvyWx0x"
},
"source": [
"### Multi-turn editing (chat)\n",
"\n",
"You can iteratively chat with the model to continuously change aspects of the video using the example below.\n",
"\n",
"To successfully accomplish this using the Interactions API, make sure to provide the previous chat history in the `input` parameter."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Z9qWfdgEKg_U"
},
"outputs": [],
"source": [
"prompt1 = \"A claymation explainer of Newton's First Law of Motion, everything is made out of clay, no hands, stop motion, sync character's words to mouth movements, show a cute ball character moving and then being stopped by a wall\"\n",
"\n",
"interaction1 = client.interactions.create(\n",
" model=omni_model,\n",
" input=prompt1,\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction1.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XcEvGMlQKqfg"
},
"outputs": [],
"source": [
"interaction1.steps"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4OudxC1_KtoK"
},
"outputs": [],
"source": [
"prompt2 = \"Now make the same video in a doodle style.\"\n",
"\n",
"turn2_input = interaction1.steps + [\n",
" {\n",
" \"type\": \"user_input\",\n",
" \"content\": [\n",
" {\n",
" \"type\": \"text\",\n",
" \"text\": prompt2\n",
" }\n",
" ]\n",
" }\n",
"]\n",
"\n",
"interaction2 = client.interactions.create(\n",
" model=omni_model,\n",
" input=turn2_input,\n",
")\n",
"\n",
"contents = []\n",
"for step in interaction2.steps:\n",
" if step.type == \"model_output\":\n",
" contents.extend(step.content)\n",
"\n",
"show_video(contents[0].data)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2ycGCBPq8pRI"
},
"source": [
"\n",
"**Note:** Support for audio references, video references, last frame, scene extension, and higher resolutions for the Gemini Omni Flash via Gemini Enterprise Agent Platform API will be available soon."
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/imagen3_editing.ipynb
+838
View File
@@ -0,0 +1,838 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9003470a8d3b"
},
"source": [
"# Imagen 3 Image Editing\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fimagen3_editing.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/imagen3_editing.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen3_editing.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "G1KDmM_PBAXz"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"### Imagen 3\n",
"\n",
"Imagen 3 on Agent Platform brings Google's state of the art generative AI capabilities to application developers. Imagen 3 is Google's highest quality text-to-image model to date. It's capable of creating images with astonishing detail. Thus, developers have more control when building next-generation AI products that transform their imagination into high quality visual assets.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DrkcqHrrwMAo"
},
"source": [
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Imagen 3 and modify existing images with mask-based editing and mask-free editing in the following modes:\n",
"\n",
"- Inpainting\n",
"- Product background editing\n",
"- Outpainting\n",
"- Mask-free"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Get started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ua6PDqB1iBSb"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google import genai\n",
"from google.genai.types import (\n",
" EditImageConfig,\n",
" GenerateImagesConfig,\n",
" Image,\n",
" MaskReferenceConfig,\n",
" MaskReferenceImage,\n",
" RawReferenceImage,\n",
")\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Sr2Y3lFwKW1M"
},
"source": [
"### Define helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "r_38e5rRKB6s"
},
"outputs": [],
"source": [
"import io\n",
"import urllib\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from PIL import Image as PIL_Image\n",
"\n",
"# Gets the image bytes from a PIL Image object.\n",
"\n",
"\n",
"def get_bytes_from_pil(image: PIL_Image) -> bytes:\n",
" byte_io_png = io.BytesIO()\n",
" image.save(byte_io_png, \"PNG\")\n",
" return byte_io_png.getvalue()\n",
"\n",
"\n",
"# Pads an image for outpainting.\n",
"def pad_to_target_size(\n",
" source_image,\n",
" target_size=(1536, 1536),\n",
" mode=\"RGB\",\n",
" vertical_offset_ratio=0,\n",
" horizontal_offset_ratio=0,\n",
" fill_val=255,\n",
"):\n",
" orig_image_size_w, orig_image_size_h = source_image.size\n",
" target_size_w, target_size_h = target_size\n",
"\n",
" insert_pt_x = (target_size_w - orig_image_size_w) // 2 + int(\n",
" horizontal_offset_ratio * target_size_w\n",
" )\n",
" insert_pt_y = (target_size_h - orig_image_size_h) // 2 + int(\n",
" vertical_offset_ratio * target_size_h\n",
" )\n",
" insert_pt_x = min(insert_pt_x, target_size_w - orig_image_size_w)\n",
" insert_pt_y = min(insert_pt_y, target_size_h - orig_image_size_h)\n",
"\n",
" if mode == \"RGB\":\n",
" source_image_padded = PIL_Image.new(\n",
" mode, target_size, color=(fill_val, fill_val, fill_val)\n",
" )\n",
" elif mode == \"L\":\n",
" source_image_padded = PIL_Image.new(mode, target_size, color=(fill_val))\n",
" else:\n",
" raise ValueError(\"source image mode must be RGB or L.\")\n",
"\n",
" source_image_padded.paste(source_image, (insert_pt_x, insert_pt_y))\n",
" return source_image_padded\n",
"\n",
"\n",
"# Pads and resizes image and mask to the same target size.\n",
"def pad_image_and_mask(\n",
" image: PIL_Image,\n",
" mask: PIL_Image,\n",
" target_size,\n",
" vertical_offset_ratio,\n",
" horizontal_offset_ratio,\n",
"):\n",
" image.thumbnail(target_size)\n",
" mask.thumbnail(target_size)\n",
"\n",
" image = pad_to_target_size(\n",
" image,\n",
" target_size=target_size,\n",
" mode=\"RGB\",\n",
" vertical_offset_ratio=vertical_offset_ratio,\n",
" horizontal_offset_ratio=horizontal_offset_ratio,\n",
" fill_val=0,\n",
" )\n",
" mask = pad_to_target_size(\n",
" mask,\n",
" target_size=target_size,\n",
" mode=\"L\",\n",
" vertical_offset_ratio=vertical_offset_ratio,\n",
" horizontal_offset_ratio=horizontal_offset_ratio,\n",
" fill_val=255,\n",
" )\n",
" return image, mask\n",
"\n",
"\n",
"def display_images(original_image, modified_image) -> None:\n",
" fig, axis = plt.subplots(1, 2, figsize=(12, 6))\n",
" axis[0].imshow(original_image)\n",
" axis[0].set_title(\"Original Image\")\n",
" axis[1].imshow(modified_image)\n",
" axis[1].set_title(\"Edited Image\")\n",
" for ax in axis:\n",
" ax.axis(\"off\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VLmwIj2RD0Fx"
},
"source": [
"### Load the image models\n",
"\n",
"Imagen 3 Generation: `imagen-3.0-generate-002`\n",
"\n",
"Imagen 3 Editing: `imagen-3.0-capability-001`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "F-gd2ypQhh7K"
},
"outputs": [],
"source": [
"generation_model = \"imagen-3.0-generate-002\"\n",
"\n",
"edit_model = \"imagen-3.0-capability-001\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f64d92aef6cb"
},
"source": [
"### Inpainting insert\n",
"\n",
"In these examples you will specify a targeted area to apply edits to. In the case of inpainting insert, you'll use a mask area to add image content to an existing image. Start by generating an image using Imagen 3. Then create two ```ReferenceImage``` objects, one for your reference image and one for your mask. For the ```MaskReferenceImage``` set ```reference_image=None```, this will allow for automatic mask detection based on the specified ```mask_mode```.\n",
"\n",
"When generating images you can also set the `safety_filter_level` and `person_generation` parameters accordingly:\n",
"* `person_generation`: DONT_ALLOW, ALLOW_ADULT, ALLOW_ALL\n",
"* `safety_filter_level`: BLOCK_LOW_AND_ABOVE, BLOCK_MEDIUM_AND_ABOVE, BLOCK_ONLY_HIGH, BLOCK_NONE"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wwZBW0UW-PiW"
},
"outputs": [],
"source": [
"image_prompt = \"\"\"\n",
"a small wooden bowl with grapes and apples on a marble kitchen counter, light brown cabinets blurred in the background\n",
"\"\"\"\n",
"generated_image = client.models.generate_images(\n",
" model=generation_model,\n",
" prompt=image_prompt,\n",
" config=GenerateImagesConfig(\n",
" number_of_images=1,\n",
" aspect_ratio=\"1:1\",\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"DONT_ALLOW\",\n",
" ),\n",
")\n",
"\n",
"edit_prompt = \"a small white ceramic bowl with lemons and limes\"\n",
"raw_ref_image = RawReferenceImage(\n",
" reference_image=generated_image.generated_images[0].image, reference_id=0\n",
")\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=None,\n",
" config=MaskReferenceConfig(\n",
" mask_mode=\"MASK_MODE_FOREGROUND\",\n",
" mask_dilation=0.1,\n",
" ),\n",
")\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=edit_prompt,\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_INPAINT_INSERTION\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"display_images(\n",
" generated_image.generated_images[0].image._pil_image,\n",
" edited_image.generated_images[0].image._pil_image,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ec7135f4de3d"
},
"source": [
"This next example demonstrates another instance of inpainting insert. However, you'll use the semantic mask mode. When using this mask mode, you'll need to specify the class ID of the object in the image that you wish to mask and replace. A list of possible instance types is shown at the end of this notebook. Once you've found the correct segmentation class ID, list it in ```segmentation_classes```.\n",
"\n",
"Within the ```MaskReferenceImage``` object you can also configure the dilation value. This float between 0 and 1 represents the percentage of the provided mask."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8pyAJlvQsocc"
},
"outputs": [],
"source": [
"image_prompt = \"\"\"\n",
"a french bulldog sitting in a living room on a couch with green throw pillows and a throw blanket,\n",
"a circular mirror is on the wall above the couch\n",
"\"\"\"\n",
"generated_image = client.models.generate_images(\n",
" model=generation_model,\n",
" prompt=image_prompt,\n",
" config=GenerateImagesConfig(\n",
" number_of_images=1,\n",
" aspect_ratio=\"1:1\",\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"DONT_ALLOW\",\n",
" ),\n",
")\n",
"\n",
"edit_prompt = \"a corgi sitting on a couch\"\n",
"raw_ref_image = RawReferenceImage(\n",
" reference_image=generated_image.generated_images[0].image, reference_id=0\n",
")\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=None,\n",
" config=MaskReferenceConfig(\n",
" mask_mode=\"MASK_MODE_SEMANTIC\",\n",
" segmentation_classes=[8],\n",
" mask_dilation=0.1,\n",
" ),\n",
")\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=edit_prompt,\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_INPAINT_INSERTION\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"display_images(\n",
" generated_image.generated_images[0].image._pil_image,\n",
" edited_image.generated_images[0].image._pil_image,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AnneP_O4vdL1"
},
"source": [
"Below you'll see another instance of inpainting insert. This time you'll use a local image and mask that have been downloaded from Google Cloud Storage. When using your own mask, you'll specify \"MASK_MODE_USER_PROVIDED\" as the ```mask_mode```."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "87yClMcjsM_O"
},
"outputs": [],
"source": [
"! gcloud storage cp \"gs://cloud-samples-data/generative-ai/image/image-dog.png\" .\n",
"! gcloud storage cp \"gs://cloud-samples-data/generative-ai/image/image-dog-mask.png\" .\n",
"initial_image = Image.from_file(location=\"image-dog.png\")\n",
"initial_image_mask = Image.from_file(location=\"image-dog-mask.png\")\n",
"\n",
"edit_prompt = \"a Persian cat sitting in a white cat bed\"\n",
"raw_ref_image = RawReferenceImage(reference_image=initial_image, reference_id=0)\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=initial_image_mask,\n",
" config=MaskReferenceConfig(\n",
" mask_mode=\"MASK_MODE_USER_PROVIDED\",\n",
" mask_dilation=0.1,\n",
" ),\n",
")\n",
"\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=edit_prompt,\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_INPAINT_INSERTION\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"display_images(\n",
" initial_image._pil_image, edited_image.generated_images[0].image._pil_image\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6ad62258e803"
},
"source": [
"### Inpainting remove\n",
"\n",
"Inpainting remove allows you to use a mask area to remove image content.\n",
"\n",
"In this next example, you'll take an image in Google Cloud Storage of a wall with a mirror and some photos and create a mask over detected mirror instances. You'll then remove this object by setting the edit mode to \"EDIT_MODE_INPAINT_REMOVAL.\" For these types of requests the prompt can be an empty string."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mOdKM3ZCB4g6"
},
"outputs": [],
"source": [
"starting_image = Image(gcs_uri=\"gs://cloud-samples-data/generative-ai/image/mirror.png\")\n",
"raw_ref_image = RawReferenceImage(reference_image=starting_image, reference_id=0)\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=None,\n",
" config=MaskReferenceConfig(\n",
" mask_mode=\"MASK_MODE_SEMANTIC\", segmentation_classes=[85]\n",
" ),\n",
")\n",
"\n",
"remove_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=\"\",\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_INPAINT_REMOVAL\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"starting_image_show = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/mirror.png\"\n",
" )\n",
")\n",
"\n",
"display_images(\n",
" starting_image_show,\n",
" remove_image.generated_images[0].image._pil_image,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "68909c926952"
},
"source": [
"### Product background editing via background swap mode\n",
"\n",
"\n",
"You can also use Imagen 3 for product image editing. By setting `edit_mode` to \"EDIT_MODE_BGSWAP\", you can maintain the product content while modifying the image background.\n",
"\n",
"For this example, start with an image stored in a Google Cloud Storage bucket, and provide a prompt describing the new background scene. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "S5zv8PqYweHW"
},
"outputs": [],
"source": [
"product_image = Image(\n",
" gcs_uri=\"gs://cloud-samples-data/generative-ai/image/suitcase.png\"\n",
")\n",
"raw_ref_image = RawReferenceImage(reference_image=product_image, reference_id=0)\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=None,\n",
" config=MaskReferenceConfig(mask_mode=\"MASK_MODE_BACKGROUND\"),\n",
")\n",
"\n",
"prompt = \"a light blue suitcase in front of a window in an airport, lots of bright, natural lighting coming in from the windows, planes taking off in the distance\"\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=prompt,\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_BGSWAP\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"product_image_show = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/suitcase.png\"\n",
" )\n",
")\n",
"display_images(product_image_show, edited_image.generated_images[0].image._pil_image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "76df73e7bbd2"
},
"source": [
"### Outpainting\n",
"\n",
"Imagen 3 editing can be used for image outpainting. Outpainting is used to expand the content of an image to a larger area or area with different dimensions. To use the outpainting feature, you must create an image mask and prepare the original image by padding some empty space around it. Once you've padded the image, you can use the ```outpainting``` editing mode to fill in the empty space."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yUevoHxxIsIN"
},
"outputs": [],
"source": [
"! gcloud storage cp \"gs://cloud-samples-data/generative-ai/image/living-room.png\" .\n",
"initial_image = Image.from_file(location=\"living-room.png\")\n",
"mask = PIL_Image.new(\"L\", initial_image._pil_image.size, 0)\n",
"\n",
"target_size_w = int(2500 * eval(\"3/4\"))\n",
"target_size = (target_size_w, 2500)\n",
"image_pil_outpaint, mask_pil_outpaint = pad_image_and_mask(\n",
" initial_image._pil_image,\n",
" mask,\n",
" target_size,\n",
" 0,\n",
" 0,\n",
")\n",
"image_pil_outpaint_image = Image(image_bytes=get_bytes_from_pil(image_pil_outpaint))\n",
"mask_pil_outpaint_image = Image(image_bytes=get_bytes_from_pil(mask_pil_outpaint))\n",
"\n",
"raw_ref_image = RawReferenceImage(\n",
" reference_image=image_pil_outpaint_image, reference_id=0\n",
")\n",
"mask_ref_image = MaskReferenceImage(\n",
" reference_id=1,\n",
" reference_image=mask_pil_outpaint_image,\n",
" config=MaskReferenceConfig(\n",
" mask_mode=\"MASK_MODE_USER_PROVIDED\",\n",
" mask_dilation=0.03,\n",
" ),\n",
")\n",
"\n",
"prompt = \"a chandelier hanging from the ceiling\"\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=prompt,\n",
" reference_images=[raw_ref_image, mask_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_OUTPAINT\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"display_images(\n",
" initial_image._pil_image, edited_image.generated_images[0].image._pil_image\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D3A7GIQBjSQX"
},
"source": [
"### Mask-free editing\n",
"\n",
"Imagen 3 editing also lets you edit images without a mask. Simply write the changes you wish to make to the image in the prompt and provide the original image as the sole reference image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E8df7clFJNRa"
},
"outputs": [],
"source": [
"original_image = Image(gcs_uri=\"gs://cloud-samples-data/generative-ai/image/latte.jpg\")\n",
"raw_ref_image = RawReferenceImage(reference_image=original_image, reference_id=0)\n",
"\n",
"\n",
"prompt = \"swan latte art in the coffee cup and an assortment of red velvet cupcakes in gold wrappers on the white plate\"\n",
"edited_image = client.models.edit_image(\n",
" model=edit_model,\n",
" prompt=prompt,\n",
" reference_images=[raw_ref_image],\n",
" config=EditImageConfig(\n",
" edit_mode=\"EDIT_MODE_DEFAULT\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" person_generation=\"ALLOW_ADULT\",\n",
" ),\n",
")\n",
"\n",
"original_image_show = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/latte.jpg\"\n",
" )\n",
")\n",
"display_images(original_image_show, edited_image.generated_images[0].image._pil_image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5c37c0c51d8f"
},
"source": [
"### Semantic segmentation classes\n",
"\n",
"| Class ID | Instance Type | Class ID | Instance Type | Class ID | Instance Type | Class ID | Instance Type |\n",
"| --- | --- | --- | --- | --- | --- | --- | --- |\n",
"| 0 | backpack | 50 | carrot | 100 | sidewalk_pavement | 150 | skis |\n",
"| 1 | umbrella | 51 | hot_dog | 101 | runway | 151 | snowboard |\n",
"| 2 | bag | 52 | pizza | 102 | terrain | 152 | sports_ball |\n",
"| 3 | tie | 53 | donut | 103 | book | 153 | kite |\n",
"| 4 | suitcase | 54 | cake | 104 | box | 154 | baseball_bat |\n",
"| 5 | case | 55 | fruit_other | 105 | clock | 155 | baseball_glove |\n",
"| 6 | bird | 56 | food_other | 106 | vase | 156 | skateboard |\n",
"| 7 | cat | 57 | chair_other | 107 | scissors | 157 | surfboard |\n",
"| 8 | dog | 58 | armchair | 108 | plaything_other | 158 | tennis_racket |\n",
"| 9 | horse | 59 | swivel_chair | 109 | teddy_bear | 159 | net |\n",
"| 10 | sheep | 60 | stool | 110 | hair_dryer | 160 | base |\n",
"| 11 | cow | 61 | seat | 111 | toothbrush | 161 | sculpture |\n",
"| 12 | elephant | 62 | couch | 112 | painting | 162 | column |\n",
"| 13 | bear | 63 | trash_can | 113 | poster | 163 | fountain |\n",
"| 14 | zebra | 64 | potted_plant | 114 | bulletin_board | 164 | awning |\n",
"| 15 | giraffe | 65 | nightstand | 115 | bottle | 165 | apparel |\n",
"| 16 | animal_other | 66 | bed | 116 | cup | 166 | banner |\n",
"| 17 | microwave | 67 | table | 117 | wine_glass | 167 | flag |\n",
"| 18 | radiator | 68 | pool_table | 118 | knife | 168 | blanket |\n",
"| 19 | oven | 69 | barrel | 119 | fork | 169 | curtain_other |\n",
"| 20 | toaster | 70 | desk | 120 | spoon | 170 | shower_curtain |\n",
"| 21 | storage_tank | 71 | ottoman | 121 | bowl | 171 | pillow |\n",
"| 22 | conveyor_belt | 72 | wardrobe | 122 | tray | 172 | towel |\n",
"| 23 | sink | 73 | crib | 123 | range_hood | 173 | rug_floormat |\n",
"| 24 | refrigerator | 74 | basket | 124 | plate | 174 | vegetation |\n",
"| 25 | washer_dryer | 75 | chest_of_drawers | 125 | person | 175 | bicycle |\n",
"| 26 | fan | 76 | bookshelf | 126 | rider_other | 176 | car |\n",
"| 27 | dishwasher | 77 | counter_other | 127 | bicyclist | 177 | autorickshaw |\n",
"| 28 | toilet | 78 | bathroom_counter | 128 | motorcyclist | 178 | motorcycle |\n",
"| 29 | bathtub | 79 | kitchen_island | 129 | paper | 179 | airplane |\n",
"| 30 | shower | 80 | door | 130 | streetlight | 180 | bus |\n",
"| 31 | tunnel | 81 | light_other | 131 | road_barrier | 181 | train |\n",
"| 32 | bridge | 82 | lamp | 132 | mailbox | 182 | truck |\n",
"| 33 | pier_wharf | 83 | sconce | 133 | cctv_camera | 183 | trailer |\n",
"| 34 | tent | 84 | chandelier | 134 | junction_box | 184 | boat_ship |\n",
"| 35 | building | 85 | mirror | 135 | traffic_sign | 185 | slow_wheeled_object |\n",
"| 36 | ceiling | 86 | whiteboard | 136 | traffic_light | 186 | river_lake |\n",
"| 37 | laptop | 87 | shelf | 137 | fire_hydrant | 187 | sea |\n",
"| 38 | keyboard | 88 | stairs | 138 | parking_meter | 188 | water_other |\n",
"| 39 | mouse | 89 | escalator | 139 | bench | 189 | swimming_pool |\n",
"| 40 | remote | 90 | cabinet | 140 | bike_rack | 190 | waterfall |\n",
"| 41 | cell phone | 91 | fireplace | 141 | billboard | 191 | wall |\n",
"| 42 | television | 92 | stove | 142 | sky | 192 | window |\n",
"| 43 | floor | 93 | arcade_machine | 143 | pole | 193 | window_blind |\n",
"| 44 | stage | 94 | gravel | 144 | fence | | |\n",
"| 45 | banana | 95 | platform | 145 | railing_banister | | |\n",
"| 46 | apple | 96 | playingfield | 146 | guard_rail | | |\n",
"| 47 | sandwich | 97 | railroad | 147 | mountain_hill | | |\n",
"| 48 | orange | 98 | road | 148 | rock | | |\n",
"| 49 | broccoli | 99 | snow | 149 | frisbee | | |\n"
]
}
],
"metadata": {
"colab": {
"name": "imagen3_editing.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/imagen4_upscale.ipynb
+372
View File
@@ -0,0 +1,372 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Imagen 4 Image Upscale\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fimagen4_upscale.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/imagen4_upscale.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/imagen4_upscale.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "G1KDmM_PBAXz"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"### Imagen 4\n",
"\n",
"Imagen 4 on Agent Platform brings Google's state-of-the-art generative AI capabilities to application developers. It's capable of creating images with astonishing detail and rendered text across a wide variety of styles. With Imagen 4, you can also upscale a generated, edited, or existing image to increase the size of an image without losing quality.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DrkcqHrrwMAo"
},
"source": [
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Imagen 4's upscaling model to increase images from:\n",
"\n",
"- Imagen outputs\n",
"- Local files\n",
"- Cloud Storage\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Get started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ua6PDqB1iBSb"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h_SlrGfc-d74"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "6-0EZL74_GCA"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google import genai\n",
"from google.genai import types\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VLmwIj2RD0Fx"
},
"source": [
"### Load the Imagen 4 models"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "Fx2O_9h5sNAI"
},
"outputs": [],
"source": [
"generation_model = \"imagen-4.0-generate-001\"\n",
"upscale_model = \"imagen-4.0-upscale-preview\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mvH2OTrK8IRS"
},
"source": [
"## Upscale images generated with Imagen\n",
"\n",
"With Imagen 4, you can generate high quality images with the following parameters:\n",
"- `aspect ratio`: `1:1`, `9:16`, `16:9`, `3:4`, `4:3`\n",
"- `number_of_images`: any value between 1 and 4\n",
"- `image_size`: `1K` or `2K`\n",
"\n",
"Once generated, you can then upscale it by a factor of `x2`, `x3`, or `x4`.\n",
"\n",
"By default, a digital watermark, or [SynthID](https://deepmind.google/technologies/synthid/), is added to Imagen 4 images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KJqHuGqi31rE"
},
"outputs": [],
"source": [
"prompt = \"A cartoon logo of a cat in a chef hat\"\n",
"\n",
"image = client.models.generate_images(\n",
" model=generation_model,\n",
" prompt=prompt,\n",
" config=types.GenerateImagesConfig(\n",
" aspect_ratio=\"1:1\",\n",
" number_of_images=1,\n",
" image_size=\"2K\",\n",
" ),\n",
")\n",
"\n",
"upscale = client.models.upscale_image(\n",
" model=upscale_model,\n",
" image=image.generated_images[0].image,\n",
" upscale_factor=\"x2\",\n",
")\n",
"upscale.generated_images[0].image.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nDCthO_aoX0x"
},
"source": [
"## Upscale local images\n",
"\n",
"You're also able to upscale local images with Imagen 4. In the next cell you'll download an image to use in the request. If you'd like to use a different image, modify the `image` variable below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pwdaN5x1Z3mc"
},
"outputs": [],
"source": [
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/boats.jpeg\n",
"\n",
"image = \"boats.jpeg\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Mi08aLXpaEX0"
},
"outputs": [],
"source": [
"upscale = client.models.upscale_image(\n",
" model=upscale_model,\n",
" image=types.Image.from_file(location=image),\n",
" upscale_factor=\"x3\",\n",
")\n",
"upscale.generated_images[0].image.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UND0wrdf3mGS"
},
"source": [
"## Upscale images from Cloud Storage\n",
"\n",
"Additionally, you can upscale images stored in Cloud Storage. If you'd like to change the initial image in this example modify the `image` variable below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e5VzcTI0bvon"
},
"outputs": [],
"source": [
"image = \"gs://cloud-samples-data/generative-ai/image/daisy.jpg\"\n",
"\n",
"upscale = client.models.upscale_image(\n",
" model=upscale_model,\n",
" image=types.Image(gcs_uri=image),\n",
" upscale_factor=\"x4\",\n",
")\n",
"upscale.generated_images[0].image.show()"
]
}
],
"metadata": {
"colab": {
"name": "imagen4_upscale.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/veo3_advanced_controls.ipynb
+540
View File
@@ -0,0 +1,540 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KjTHAV8FgEza"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UdL4uvQQs76x"
},
"source": [
"# Veo 3.1 Advanced Controls\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fveo3_advanced_controls.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lUCUMoTmN_lJ"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rDjAqcgigwdX"
},
"source": [
"## Overview\n",
"\n",
"### Veo 3.1\n",
"\n",
"Veo 3.1 on Agent Platform gives application developers access to Google's cutting-edge video generation. This model creates videos with stunning detail and realistic physics across a wide array of visual styles. Veo 3.1 enhances video quality from text and image prompts, and now includes dialogue and audio generation.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Veo 3.1 advanced control features, including:\n",
" - Frame interpolation\n",
" - Video extend\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v2_iOv5uhXVg"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K4uerc9Xhf1f"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rJyFNKoQhiwF"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GWYnCW0-h6HI"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bqz5LUG6h8fA"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LVrasKoriKZn"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c5XgNQ2HD_HH"
},
"outputs": [],
"source": [
"import os\n",
"import time\n",
"import urllib.request\n",
"\n",
"import matplotlib.image as img\n",
"import matplotlib.pyplot as plt\n",
"from IPython.display import Video, display\n",
"from PIL import Image as PIL_Image\n",
"from google import genai\n",
"from google.genai import types"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yxBkUEqdiB1g"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GtjPBmYHiEfx"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qD_bwA9hiMzL"
},
"source": [
"### Define a helper function to display media"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GUrEwbvFiPhJ"
},
"outputs": [],
"source": [
"def show_video(video: str | bytes) -> None:\n",
" if isinstance(video, str):\n",
" file_name = video.split(\"/\")[-1]\n",
" !gsutil cp {video} {file_name}\n",
" display(Video(file_name, embed=True, width=600))\n",
" else:\n",
" with open(\"sample.mp4\", \"wb\") as out_file:\n",
" out_file.write(video)\n",
" display(Video(\"sample.mp4\", embed=True, width=600))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2jaSOOadiUj6"
},
"source": [
"### Load the video generation model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "APRfTklCiYR2"
},
"outputs": [],
"source": [
"video_model = \"veo-3.1-generate-001\"\n",
"video_extension_model = \"veo-3.1-generate-preview\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_1-VPOexGWe8"
},
"source": [
"## Video generation with Veo 3.1 advanced controls"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P604bVwQeDPN"
},
"source": [
"### Frame interpolation\n",
"\n",
"With frame interpolation, you can specify the first and last frame of the video clip, and Veo will generate the rest.\n",
"\n",
"#### Load the images\n",
"\n",
"Download the first and last frames that you will use in the request. If you would like to use local images, simply comment out the `wget` command and edit the `first_frame` and `last_frame` variables.\n",
"\n",
"**Note:** This feature tends to work best when the starting and ending frames are similar in nature."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5-fpFTzwefcH"
},
"outputs": [],
"source": [
"# Download first frame\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies.png\n",
"# fmt: off\n",
"first_frame = \"cookies.png\" # @param {type: 'string'}\n",
"\n",
"# Download last frame\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies-milk.png\n",
"last_frame = \"cookies-milk.png\" # @param {type: 'string'}\n",
"# fmt: on\n",
"\n",
"# Display the images\n",
"fig, axis = plt.subplots(1, 2, figsize=(12, 6))\n",
"axis[0].imshow(img.imread(first_frame))\n",
"axis[0].set_title(\"First Frame\")\n",
"axis[1].imshow(img.imread(last_frame))\n",
"axis[1].set_title(\"Last Frame\")\n",
"for ax in axis:\n",
" ax.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yKR0riV_K5nT"
},
"source": [
"With Veo 3.1, you have the option to generate videos with audio from a text prompt and/or input image(s). In order to generate a video in the following sample, specify the following info:\n",
"- **Prompt:** A description of the video you would like to see. Since you're starting with images you can also leave the prompt field empty.\n",
"- **Aspect ratio:** Select either 16:9 or 9:16.\n",
"- **Resolution:** 720p, 1080p or 4k.\n",
"- **Number of videos:** Set this value to 1, 2, 3, or 4.\n",
"- **Video duration:** Can be 4, 6, or 8 seconds.\n",
"- **Person generation:** Set to `allow_adult` or `dont_allow`.\n",
"- **Audio generation:** Set `generate_audio` to True if you'd like audio included in your generated video.\n",
"\n",
"\n",
"**Safety:** All Veo videos include [SynthID](https://deepmind.google/technologies/synthid/), which embeds a digital watermark directly into the AI-generated video.\n",
"\n",
"**NOTE:** Generating a video in 4k will introduce increased latency up to several minutes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3YzkQI5JekEh"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"a hand reaches in and places a glass of milk next to the plate of cookies\" # @param {type: 'string'}\n",
"# fmt: on\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" image=types.Image.from_file(location=first_frame),\n",
" config=types.GenerateVideosConfig(\n",
" last_frame=types.Image.from_file(location=last_frame),\n",
" aspect_ratio=\"9:16\",\n",
" resolution=\"1080p\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(10)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Rln2_Oi3Piu-"
},
"source": [
"In this next example, you'll continue to use first and last frames stored in Cloud Storage. However, in this request, you'll reference the images in Cloud Storage directly, rather than downloading them locally first. If you'd like to use a different Cloud Storage image, replace the URL and `gcs_uri` variables below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aCzame3wGEpi"
},
"outputs": [],
"source": [
"first_frame = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/forest-1.png\"\n",
" )\n",
")\n",
"first_frame_gcs = \"gs://cloud-samples-data/generative-ai/image/forest-1.png\"\n",
"\n",
"last_frame = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/forest-2.png\"\n",
" )\n",
")\n",
"last_frame_gcs = \"gs://cloud-samples-data/generative-ai/image/forest-2.png\"\n",
"\n",
"# Display the images\n",
"fig, axis = plt.subplots(1, 2, figsize=(18, 6))\n",
"axis[0].imshow(first_frame)\n",
"axis[1].imshow(last_frame)\n",
"for ax in axis:\n",
" ax.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XPu4QrPAGanp"
},
"source": [
"Rather than output `video_bytes` in this section, you'll save your video to Cloud Storage. In order to accomplish this, set your Cloud Storage bucket location in `output_gcs`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ylUHNer1GleN"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"the chipmunk climbs down the tree to sit next to the rabbit\" # @param {type: 'string'}\n",
"# fmt: on\n",
"output_gcs = \"gs://[your-bucket-path]\" # @param {type: 'string'}\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" image=types.Image(gcs_uri=first_frame_gcs, mime_type=\"image/png\"),\n",
" config=types.GenerateVideosConfig(\n",
" last_frame=types.Image(gcs_uri=last_frame_gcs, mime_type=\"image/png\"),\n",
" output_gcs_uri=output_gcs,\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=6,\n",
" resolution=\"1080p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(10)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.uri)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gg6mf3iWbjRP"
},
"source": [
"### Video extend\n",
"\n",
"In this next example, you'll use Veo to extend an existing video. You'll use the same parameters as before. However, the following are configured differently in this instance:\n",
"- **Input video:** Since you're starting with a video, specify a Cloud Storage video location in the `video_gcs` variable.\n",
"- **Output video:** As these videos will be bigger, you'll save your output video in Cloud Storage. Include the Cloud Storage bucket in the `output_gcs` field.\n",
"- **Video duration:** You can extend your video by 7 seconds."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-dXqvIuAbzVQ"
},
"outputs": [],
"source": [
"prompt = \"a butterfly flies in and lands on the flower\" # @param {type: 'string'}\n",
"# fmt: off\n",
"video_gcs = \"gs://cloud-samples-data/generative-ai/video/red-flower.mp4\" # @param {type: 'string'}\n",
"# fmt: on\n",
"output_gcs = \"gs://[your-bucket-path]\" # @param {type: 'string'}\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_extension_model,\n",
" prompt=prompt,\n",
" video=types.Video(uri=video_gcs, mime_type=\"video/mp4\"),\n",
" config=types.GenerateVideosConfig(\n",
" output_gcs_uri=output_gcs,\n",
" number_of_videos=1,\n",
" duration_seconds=7,\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(5)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.uri)"
]
}
],
"metadata": {
"colab": {
"name": "veo3_advanced_controls.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/veo3_reference_to_video.ipynb
+732
View File
@@ -0,0 +1,732 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bZKaz0oSwAx-"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UdL4uvQQs76x"
},
"source": [
"# Veo 3.1 Reference to Video\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fveo3_reference_to_video.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/veo3_reference_to_video.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_reference_to_video.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NnMaDH8jwReT"
},
"source": [
"| | |\n",
"|-|-|\n",
"|Author(s) | [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vIDE4FhjwW67"
},
"source": [
"## Overview\n",
"\n",
"### Veo 3.1\n",
"\n",
"Veo 3.1 on Agent Platform gives application developers access to Google's cutting-edge video generation. Veo 3.1 enhances video quality from text and image prompts, and now includes dialogue and audio generation.\n",
"\n",
"Reference-to-Video functionality on Veo 3.1 is optimized for high-energy, short-form narratives. It excels at maintaining consistency across rapid intercutting and dynamic camera transitions for better visual storytelling.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Veo 3.1 to:\n",
"- Generate a video from asset images, including subjects, objects and scenes\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dEPqvne0w4qx"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s8p3AOlALGpj"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LWGj2AmpLJ2D"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aDvFfD83w7iL"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "iTfXlEVQw9xV"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PYCYpliKxFES"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ffYy0e81xAV6"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "PMz0sZASxCTU"
},
"outputs": [],
"source": [
"import os\n",
"import time\n",
"import urllib.request\n",
"\n",
"import matplotlib.image as img\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from IPython.display import Video, display\n",
"from PIL import Image as PIL_Image\n",
"from google import genai\n",
"from google.genai import types\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "br8QTmuyxL5R"
},
"source": [
"### Define helper functions"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "TgkK6Vr4xN5j"
},
"outputs": [],
"source": [
"def show_video(video):\n",
" if isinstance(video, str):\n",
" file_name = video.split(\"/\")[-1]\n",
" !gcloud storage cp {video} {file_name}\n",
" display(Video(file_name, embed=True, width=600))\n",
" else:\n",
" with open(\"sample.mp4\", \"wb\") as out_file:\n",
" out_file.write(video)\n",
" display(Video(\"sample.mp4\", embed=True, width=600))\n",
"\n",
"\n",
"def show_images(\n",
" images: list[str],\n",
"):\n",
" fig, axes = plt.subplots(1, len(images), figsize=(12, 6))\n",
" if len(images) == 1:\n",
" axes = np.array([axes])\n",
" for i, ax in enumerate(axes):\n",
" image = img.imread(images[i])\n",
" ax.imshow(image)\n",
" ax.axis(\"off\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5a6UxKZfxQoH"
},
"source": [
"### Load the video model"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "H6K66dOfxSmr"
},
"outputs": [],
"source": [
"video_model = \"veo-3.1-generate-preview\"\n",
"video_model_fast = \"veo-3.1-fast-generate-preview\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UMANSB1YN11I"
},
"source": [
"## Reference images to videos\n",
"\n",
"With Reference-to-Video in Veo 3.1, you can use reference images to generate videos. The reference images are `asset` images of subjects, objects, or scenes that will be included in the final video output.\n",
"\n",
"**NOTE:** You can include up to 3 `asset` images in a request."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UBPpj5BTZaY9"
},
"source": [
"### Asset references\n",
"\n",
"Download and display the asset images that you'll use in the following requests. To use your own local images, modify the URLs in the `wget` command and update the `first_image`, `second_image`, and/or `third_image` variables accordingly."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X81WV5lIpSEe"
},
"source": [
"#### Subject reference images\n",
"\n",
"In this example, you'll use two subject reference images of different people. You'll generate a new scene for them based on a text prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E2vrXKXdu39e"
},
"outputs": [],
"source": [
"# Download subject images from Cloud Storage\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/man-in-field.png\n",
"\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/woman.jpeg"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "twURgtG16oUu"
},
"source": [
"Set the `first_image` and `second_image` variables."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e_aPJ5dzvCnS"
},
"outputs": [],
"source": [
"first_image = \"man-in-field.png\" # @param {type: 'string'}\n",
"second_image = \"woman.jpeg\" # @param {type: 'string'}\n",
"\n",
"show_images([first_image, second_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LcrdbeRE9oW5"
},
"source": [
"Now, you'll send a request to generate a video. With Veo 3.1, you can generate videos with audio from a text prompt, input image(s), or both. In order to generate a video in the following sample, specify the following info:\n",
"\n",
" - **Prompt:** A description of the video you would like to see with the reference images.\n",
" - **Reference images:** Up to three `asset` images.\n",
" - **Aspect ratio:** 16:9 (Landscape), 9:16 (Portrait)\n",
" - **Number of videos:** Set this value to 1, 2, 3, or 4\n",
" - **Video duration:** 8 seconds\n",
" - **Resolution:** 720p, 1080p, 4k\n",
" - **Person generation:** Set to `allow_adult` or `dont_allow`.\n",
" - **Generate audio:** Set to `True` if you'd like audio in your generated video.\n",
"\n",
" **NOTE:** Generating a video in 4k will introduce increased latency up to several minutes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ym4PP2AvvPW2"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"a woman and a man drinking a cup of coffee in a cafe, chatting about the new restaurant around the block\n",
"\"\"\"\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model_fast,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" reference_images=[\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image.from_file(location=first_image),\n",
" reference_type=\"asset\",\n",
" ),\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image.from_file(location=second_image),\n",
" reference_type=\"asset\",\n",
" ),\n",
" ],\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"1080p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZSPqF01yySYl"
},
"source": [
"#### Setting reference image\n",
"\n",
"Now, you'll use a single scenery reference image and a text prompt to generate a video with different subjects and actions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h5Jz6bYKyby5"
},
"outputs": [],
"source": [
"# Download the image from Cloud Storage\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/room.png"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IotQjwiP7Og2"
},
"source": [
"Set the `first_image` variable."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ULTrObxdylip"
},
"outputs": [],
"source": [
"first_image = \"room.png\" # @param {type: 'string'}\n",
"\n",
"show_images([first_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tqEjvGNJ7TD4"
},
"source": [
"Run the request. Update the `prompt` if you'd like to see different content within the scene."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "w7KnIJMYyv66"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"a Corgi walks around in a living room, then jumps on the couch and starts reading a book on the coffee table\n",
"\"\"\"\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" reference_images=[\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image.from_file(location=first_image),\n",
" reference_type=\"asset\",\n",
" ),\n",
" ],\n",
" aspect_ratio=\"9:16\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"720p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ybB1oG05W4X"
},
"source": [
"#### Product reference image\n",
"\n",
"Next, you'll use a product reference image and a text prompt to generate a video. This will demonstrate how Veo maintains product consistency while in motion."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nnfezu0-6aAz"
},
"outputs": [],
"source": [
"# Download the image from Cloud Storage\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/vase.png"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zEqhYMQG8C5w"
},
"source": [
"Set the `first_image` variable."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fl9hn5oT78Wx"
},
"outputs": [],
"source": [
"first_image = \"vase.png\" # @param {type: 'string'}\n",
"\n",
"show_images([first_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4ivJgSH68RE3"
},
"source": [
"Run the request. Update the `prompt` if you'd like to visualize the product in a different manner."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hvFkg-vzfcY-"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"a person walks in carrying a vase full of flowers and places the vase on a kitchen table\n",
"\"\"\"\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" reference_images=[\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image.from_file(location=first_image),\n",
" reference_type=\"asset\",\n",
" ),\n",
" ],\n",
" aspect_ratio=\"9:16\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"720p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FOqUuJRGoQtO"
},
"source": [
"#### Three distinct reference images\n",
"\n",
"In this example, you'll use three different reference images (a product, a subject, and a scene) from Google Cloud Storage. Instead of downloading them, you'll reference their Cloud Storage URIs directly. To use your own images, replace the gcs_uri variables below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5m1xhMENZrYx"
},
"outputs": [],
"source": [
"first_image = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/flowers.png\"\n",
" )\n",
")\n",
"first_image_gcs = \"gs://cloud-samples-data/generative-ai/image/flowers.png\"\n",
"\n",
"second_image = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/suitcase.png\"\n",
" )\n",
")\n",
"second_image_gcs = \"gs://cloud-samples-data/generative-ai/image/suitcase.png\"\n",
"\n",
"third_image = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/woman.jpg\"\n",
" )\n",
")\n",
"third_image_gcs = \"gs://cloud-samples-data/generative-ai/image/woman.jpg\"\n",
"\n",
"# Display the images\n",
"fig, axis = plt.subplots(1, 3, figsize=(18, 6))\n",
"axis[0].imshow(first_image)\n",
"axis[1].imshow(second_image)\n",
"axis[2].imshow(third_image)\n",
"for ax in axis:\n",
" ax.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5PAnNvt79xNU"
},
"source": [
"Rather than output video_bytes in this section, you'll save your video to Cloud Storage. In order to accomplish this, set your Cloud Storage bucket location in `output_gcs`.\n",
"\n",
"**Safety:** All Veo videos include [SynthID](https://deepmind.google/science/synthid/), which embeds a digital watermark directly into the AI-generated video."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0Jq-yKddAFJw"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"a wide shot of a woman wheeling a blue suitcase through a flower field\" # @param {type: 'string'}\n",
"# fmt: on\n",
"output_gcs = \"gs://[your-bucket-path]\" # @param {type: 'string'}\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" reference_images=[\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image(gcs_uri=first_image_gcs, mime_type=\"image/png\"),\n",
" reference_type=\"asset\",\n",
" ),\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image(gcs_uri=second_image_gcs, mime_type=\"image/png\"),\n",
" reference_type=\"asset\",\n",
" ),\n",
" types.VideoGenerationReferenceImage(\n",
" image=types.Image(gcs_uri=third_image_gcs, mime_type=\"image/jpeg\"),\n",
" reference_type=\"asset\",\n",
" ),\n",
" ],\n",
" output_gcs_uri=output_gcs,\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"1080p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.uri)"
]
}
],
"metadata": {
"colab": {
"name": "veo3_reference_to_video.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/veo3_video_generation.ipynb
+739
View File
@@ -0,0 +1,739 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KjTHAV8FgEza"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UdL4uvQQs76x"
},
"source": [
"# Veo 3.1 Video Generation\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fveo3_video_generation.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/veo3_video_generation.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_video_generation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lUCUMoTmN_lJ"
},
"source": [
"| | |\n",
"|-|-|\n",
"|Author(s) | [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rDjAqcgigwdX"
},
"source": [
"## Overview\n",
"\n",
"### Veo 3.1\n",
"\n",
"Veo 3.1 on Agent Platform gives application developers access to Google's cutting-edge video generation. This model creates videos with stunning detail and realistic physics across a wide array of visual styles. Veo 3.1 enhances video quality from text and image prompts, and now includes dialogue and audio generation.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Veo 3.1 and generate new videos with audio from:\n",
"- Enhanced text prompts\n",
"- Starting input images"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v2_iOv5uhXVg"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K4uerc9Xhf1f"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rJyFNKoQhiwF"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GWYnCW0-h6HI"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "bqz5LUG6h8fA"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LVrasKoriKZn"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yxBkUEqdiB1g"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "GtjPBmYHiEfx"
},
"outputs": [],
"source": [
"import os\n",
"import time\n",
"\n",
"import matplotlib.image as img\n",
"import matplotlib.pyplot as plt\n",
"from IPython.display import Markdown, Video, display\n",
"from google import genai\n",
"from google.genai import types\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)\n",
"\n",
"gemini_client = genai.Client(enterprise=True, project=PROJECT_ID, location=\"global\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qD_bwA9hiMzL"
},
"source": [
"### Define a helper function to display media"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "GUrEwbvFiPhJ"
},
"outputs": [],
"source": [
"def show_video(video):\n",
" if isinstance(video, str):\n",
" file_name = video.split(\"/\")[-1]\n",
" !gcloud storage cp {video} {file_name}\n",
" display(Video(file_name, embed=True, width=600))\n",
" else:\n",
" with open(\"sample.mp4\", \"wb\") as out_file:\n",
" out_file.write(video)\n",
" display(Video(\"sample.mp4\", embed=True, width=600))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2jaSOOadiUj6"
},
"source": [
"### Load the video generation model"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "APRfTklCiYR2"
},
"outputs": [],
"source": [
"video_model = \"veo-3.1-generate-001\"\n",
"video_model_fast = \"veo-3.1-fast-generate-001\"\n",
"video_model_lite = \"veo-3.1-lite-generate-001\"\n",
"gemini_model = \"gemini-3.5-flash\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "H_R1_Y76i4QB"
},
"source": [
"## Generate videos\n",
"Now, you'll generate videos from text and/or image prompts. You can get started with your own prompts or complete the next section to optimize your prompts with some established best practices.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v9ERu5b3EEdp"
},
"source": [
"### [Optional] Optimize your prompt: Text-to-video\n",
"\n",
"By considering the following options in your prompt, you can use Veo to create higher quality videos that more closely resemble your desired outcome. Learn more about advanced [prompting techniques for Veo 3.1](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1). To get started specify the following, or leave them as `None` if they don't align with your specific goals.\n",
"- **Subject:** The \"who\" or \"what\" of your video\n",
"- **Action:** Describe movements, interactions, etc.\n",
"- **Scene:** The \"where\" and \"when\" of your video\n",
"- **Camera angles:** The shot's viewpoint\n",
"- **Camera movements:** For a more cinematic/dynamic experience\n",
"- **Lens effects:** How the camera \"sees\" the world\n",
"- **Style:** The video's artistic filter\n",
"- **Temporal elements:** To imply changes in time\n",
"- **Audio:** Various sound effects or dialogue that guides the visuals through sound"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "yx7DKfqri3Vf"
},
"outputs": [],
"source": [
"subject = \"a detective\" # @param {type: 'string'}\n",
"action = \"interrogating a rubber duck\" # @param {type: 'string'}\n",
"scene = \"in a dark interview room\" # @param {type: 'string'}\n",
"\n",
"# fmt: off\n",
"camera_angle = \"Over-the-Shoulder Shot\" # @param [\"None\", \"Eye-Level Shot\", \"Low-Angle Shot\", \"High-Angle Shot\", \"Bird's-Eye View\", \"Top-Down Shot\", \"Worm's-Eye View\", \"Dutch Angle\", \"Canted Angle\", \"Close-Up\", \"Extreme Close-Up\", \"Medium Shot\", \"Full Shot\", \"Long Shot\", \"Wide Shot\", \"Establishing Shot\", \"Over-the-Shoulder Shot\", \"Point-of-View (POV) Shot\"]\n",
"camera_movement = \"Zoom (In)\" # @param [\"None\", \"Static Shot (or fixed)\", \"Pan (left)\", \"Pan (right)\", \"Tilt (up)\", \"Tilt (down)\", \"Dolly (In)\", \"Dolly (Out)\", \"Zoom (In)\", \"Zoom (Out)\", \"Truck (Left)\", \"Truck (Right)\", \"Pedestal (Up)\", \"Pedestal (Down)\", \"Crane Shot\", \"Aerial Shot\", \"Drone Shot\", \"Handheld\", \"Shaky Cam\", \"Whip Pan\", \"Arc Shot\"]\n",
"lens_effects = \"None\" # @param [\"None\", \"Wide-Angle Lens (e.g., 24mm)\", \"Telephoto Lens (e.g., 85mm)\", \"Shallow Depth of Field\", \"Bokeh\", \"Deep Depth of Field\", \"Lens Flare\", \"Rack Focus\", \"Fisheye Lens Effect\", \"Vertigo Effect (Dolly Zoom)\"]\n",
"style = \"Cinematic\" # @param [\"None\", \"Photorealistic\", \"Cinematic\", \"Vintage\", \"Japanese anime style\", \"Claymation style\", \"Stop-motion animation\", \"In the style of Van Gogh\", \"Surrealist painting\", \"Monochromatic black and white\", \"Vibrant and saturated\", \"Film noir style\", \"High-key lighting\", \"Low-key lighting\", \"Golden hour glow\", \"Volumetric lighting\", \"Backlighting to create a silhouette\"]\n",
"temporal_elements = \"None\" # @param [\"None\", \"Slow-motion\", \"Fast-paced action\", \"Time-lapse\", \"Hyperlapse\", \"Pulsating light\", \"Rhythmic movement\"]\n",
"\n",
"sound_effects = \"Ticking clock\" # @param [\"None\", \"Sound of a phone ringing\", \"Water splashing\", \"Soft house sounds\", \"Ticking clock\", \"City traffic and sirens\", \"Waves crashing\", \"Quiet office hum\"]\n",
"# fmt: on\n",
"dialogue = \"Where were you last night?\" # @param {type: 'string'}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PMFj078dLG0D"
},
"source": [
"Now, you'll use Gemini to take all of these keywords and combine them into a detailed Veo prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f3jvksummytE"
},
"outputs": [],
"source": [
"prompt = \"\"\n",
"\n",
"keywords = [subject, action, scene]\n",
"optional_keywords = [\n",
" camera_angle,\n",
" camera_movement,\n",
" lens_effects,\n",
" style,\n",
" temporal_elements,\n",
" sound_effects,\n",
"]\n",
"for keyword in optional_keywords:\n",
" if keyword != \"None\":\n",
" keywords.append(keyword)\n",
"if dialogue != \"\":\n",
" keywords.append(dialogue)\n",
"\n",
"gemini_prompt = f\"\"\"\n",
"You are an expert video prompt engineer for Google's Veo model. Your task is to construct the most effective and optimal prompt string using the following keywords. Every single keyword MUST be included. Synthesize them into a single, cohesive, and cinematic instruction. Do not add any new core concepts. Output ONLY the final prompt string, without any introduction or explanation. Mandatory Keywords: {\",\".join(keywords)}\n",
"\"\"\"\n",
"response = gemini_client.models.generate_content(\n",
" model=gemini_model,\n",
" contents=gemini_prompt,\n",
")\n",
"\n",
"# Set Gemini's response in a prompt variable\n",
"prompt = response.text\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aDaTx8WCidRG"
},
"source": [
"### Generate videos from a text prompt\n",
"\n",
"With Veo 3.1, you have the option to generate videos from a text prompt. In order to generate a video in the following sample, specify the following info:\n",
"- **Prompt:** A detailed description of the video you would like to see. Only edit the prompt if you didn't generate a detailed prompt with Gemini in the previous section.\n",
"- **Audio generation:** Set `generate_audio` to True if you'd like audio to be included in the output video.\n",
"- **Aspect ratio:** 16:9 or 9:16.\n",
"- **Number of videos:** Set this value to 1 or 2.\n",
"- **Video duration:** 4, 6, or 8 seconds.\n",
"- **Resolution:** Can be 1080p, 720p or 4k.\n",
"\n",
"**NOTE:** Generating a video in 4k will introduce increased latency up to several minutes."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "W_iqdJNGl2Sa"
},
"outputs": [],
"source": [
"if prompt == \"\":\n",
" # fmt: off\n",
" prompt = \"a garden gnome singing a pop song in a whimsical outdoor garden\" # @param {type: 'string'}\n",
"# fmt: on"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tygfLLlWyTo_"
},
"outputs": [],
"source": [
"print(prompt)\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=6,\n",
" resolution=\"1080p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "688nb6GEwqR4"
},
"source": [
"- **Veo 3.1 Fast:** For this next example, you'll use Veo 3.1 Fast. This model is a great option for use cases where latency is a priority.\n",
"- **File location:** In this request, the video will be stored in Cloud Storage once video generation is complete. Specify the bucket path where you would like this video to be stored in the `output_gcs` field.\n",
"- **Person generation:** When generating videos of people you can also set the `person_generation` parameter accordingly: `allow_adult`, `dont_allow`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sp0K0WYUwxLJ"
},
"outputs": [],
"source": [
"# fmt: off\n",
"prompt = \"sculpting a bowl on a pottery wheel while the person sculpting remarks: wow, I love this one\" # @param {type: 'string'}\n",
"# fmt: on\n",
"output_gcs = \"gs://[your-bucket-path]\" # @param {type: 'string'}\n",
"\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model_fast,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" aspect_ratio=\"9:16\",\n",
" output_gcs_uri=output_gcs,\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"720p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.uri)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8494d440c7d4"
},
"source": [
"- **Veo 3.1 Lite:** In the next example, you'll use Veo 3.1 Lite, which is ideal for rapid prototyping and iteration."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dc95738712eb"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"A sleek, modern white swivel chair begins to rotate smoothly in a full 360-degree circle. As it spins, a vibrant purple glow gradually washes over the fabric, transitioning the chair from pure white to a deep royal purple by the time it completes its rotation.\n",
"\"\"\"\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model_lite,\n",
" prompt=prompt,\n",
" config=types.GenerateVideosConfig(\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=8,\n",
" resolution=\"720p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(10)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YysYLyiVj8Zd"
},
"source": [
"### Generate videos from an image\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X-NdoBONKpJD"
},
"source": [
"#### Download the starting image\n",
"\n",
"You can also generate a video by starting with an input image. In this example, you'll locally download an image that's stored in Google Cloud Storage. If you'd like, you can add the URL of the image you'd like to use to display it below. If you have a local image you'd like to use, you can specify that in the following steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h_X5dS1Rye2Q"
},
"outputs": [],
"source": [
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/flowers.png"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WGtJsg-MPSXi"
},
"source": [
"If you'd like to use a different local image, modify the file name in `starting_image`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cl_lTdvEIt8k"
},
"outputs": [],
"source": [
"starting_image = \"flowers.png\" # @param {type: 'string'}\n",
"\n",
"# Display the image\n",
"fig, axis = plt.subplots(1, 1, figsize=(12, 6))\n",
"axis.imshow(img.imread(starting_image))\n",
"axis.set_title(\"Starting Frame\")\n",
"axis.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FsHmRk-_NYya"
},
"source": [
"#### [Optional] Optimize your prompt: Image-to-video\n",
"\n",
"By considering the following options in your prompt, you can use Veo to create higher quality videos that more closely resemble your desired outcome. Learn more about advanced [prompting techniques for Veo 3.1](https://cloud.google.com/blog/products/ai-machine-learning/ultimate-prompting-guide-for-veo-3-1). To get started specify the following, or leave them as `None` if they don't align with your specific goals.\n",
"- **Camera motion:** The camera movement that occurs while the rest of the scene remains static\n",
"- **Subject animation:** Main character or object movement \n",
"- **Environmental animation:** Background or atmosphere movement\n",
"- **Audio:** Various sound effects or dialogue that guides the visuals through sound"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6UE7AfftqSM5"
},
"outputs": [],
"source": [
"# fmt: off\n",
"camera_motion = \"Zoom (In)\" # @param [\"None\", \"Eye-Level Shot\", \"Low-Angle Shot\", \"High-Angle Shot\", \"Bird's-Eye View\", \"Top-Down Shot\", \"Worm's-Eye View\", \"Dutch Angle\", \"Canted Angle\", \"Close-Up\", \"Extreme Close-Up\", \"Medium Shot\", \"Full Shot\", \"Long Shot\", \"Wide Shot\", \"Establishing Shot\", \"Over-the-Shoulder Shot\", \"Point-of-View (POV) Shot\", \"Static Shot (or fixed)\", \"Pan (left)\", \"Pan (right)\", \"Tilt (up)\", \"Tilt (down)\", \"Dolly (In)\", \"Dolly (Out)\", \"Zoom (In)\", \"Zoom (Out)\", \"Truck (Left)\", \"Truck (Right)\", \"Pedestal (Up)\", \"Pedestal (Down)\", \"Crane Shot\", \"Aerial Shot\", \"Drone Shot\", \"Handheld\", \"Shaky Cam\", \"Whip Pan\", \"Arc Shot\", \"Wide-Angle Lens (e.g., 24mm)\", \"Telephoto Lens (e.g., 85mm)\", \"Shallow Depth of Field\", \"Bokeh\", \"Deep Depth of Field\", \"Lens Flare\", \"Rack Focus\", \"Fisheye Lens Effect\", \"Vertigo Effect (Dolly Zoom)\"]\n",
"\n",
"subject_animation = \"None\" # @param [\"None\", \"The subject's head turns slowly\", \"The subject blinks slowly\", \"The subject's hair and clothes flutter gently in the wind\", \"A subtle smile appears on the subject's face\"]\n",
"environmental_animation = \"Light changes subtly\" # @param [\"None\", \"Fog rolls in slowly\", \"Rain starts to fall gently\", \"Leaves rustle in the wind\", \"Light changes subtly\", \"Reflections move on water\"]\n",
"\n",
"sound_effects = \"None\" # @param [\"None\", \"Sound of a phone ringing\", \"Water splashing\", \"Soft house sounds\", \"Ticking clock\", \"City traffic and sirens\", \"Waves crashing\", \"Quiet office hum\"]\n",
"# fmt: on\n",
"dialogue = \"\" # @param {type: 'string'}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nG4tknxOR70c"
},
"source": [
"Now, you'll use Gemini to take all of these keywords and combine them into a detailed Veo prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a1NvfHEHrx2m"
},
"outputs": [],
"source": [
"prompt = \"\"\n",
"\n",
"keywords = []\n",
"optional_keywords = [\n",
" camera_motion,\n",
" subject_animation,\n",
" environmental_animation,\n",
" sound_effects,\n",
"]\n",
"for keyword in optional_keywords:\n",
" if keyword != \"None\":\n",
" keywords.append(keyword)\n",
"if dialogue != \"\":\n",
" keywords.append(dialogue)\n",
"\n",
"gemini_prompt = f\"\"\"\n",
"You are an expert prompt engineer for Google's Veo model. Analyze the provided image and combine its content with the following motion and audio keywords to generate a single, cohesive, and cinematic prompt. Integrate the image's subject and scene with the requested motion and audio effects. The final output must be ONLY the prompt itself, with no preamble. Mandatory Keywords: {\",\".join(keywords)}\n",
"\"\"\"\n",
"with open(starting_image, \"rb\") as f:\n",
" image = f.read()\n",
"\n",
"response = gemini_client.models.generate_content(\n",
" model=gemini_model,\n",
" contents=[gemini_prompt, types.Part.from_bytes(data=image, mime_type=\"image/png\")],\n",
")\n",
"\n",
"# Set Gemini's response in a prompt variable\n",
"prompt = response.text\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mCc77j59KjGN"
},
"source": [
"#### Send the video request\n",
"\n",
"If you're generating a video from an image you don't need to provide a prompt. The model will simply add motion to your image; however, a detailed prompt will help with video quality.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "thPYb0eNtwxN"
},
"outputs": [],
"source": [
"if prompt == \"\":\n",
" # fmt: off\n",
" prompt = \"zoom out of the flower field, play whimsical music\" # @param {type: 'string'}\n",
"# fmt: on"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFgQYeXHnidx"
},
"outputs": [],
"source": [
"print(prompt)\n",
"\n",
"operation = client.models.generate_videos(\n",
" model=video_model,\n",
" prompt=prompt,\n",
" image=types.Image.from_file(location=starting_image),\n",
" config=types.GenerateVideosConfig(\n",
" aspect_ratio=\"16:9\",\n",
" number_of_videos=1,\n",
" duration_seconds=6,\n",
" resolution=\"1080p\",\n",
" person_generation=\"allow_adult\",\n",
" generate_audio=True,\n",
" ),\n",
")\n",
"\n",
"while not operation.done:\n",
" time.sleep(15)\n",
" operation = client.operations.get(operation)\n",
" print(operation)\n",
"\n",
"if operation.response:\n",
" show_video(operation.result.generated_videos[0].video.video_bytes)"
]
}
],
"metadata": {
"colab": {
"name": "veo3_video_generation.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
vision/getting-started/virtual_try_on.ipynb
+682
View File
@@ -0,0 +1,682 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uxCkB_DXTHzf"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hny4I-ODTIS6"
},
"source": [
"# Virtual Try-On: Image Generation\n",
"\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fvirtual_try_on.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/virtual_try_on.ipynb\">\n",
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<p>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/virtual_try_on.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>\n",
"</p>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EXljS1-Ib0ue"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Jorj Ismailyan](https://github.com/jismailyan-google) |\n",
"| [Katie Nguyen](https://github.com/katiemn) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-nLS57E2TO5y"
},
"source": [
"## Overview\n",
"\n",
"### Virtual Try-On\n",
"\n",
"[Virtual Try-On](https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-virtual-try-on-images) uses Google's cutting-edge image generation models to create high-quality images of people virtually trying on clothes. By providing an image of a model and a clothing item, you can generate a new image of the model wearing that product.\n",
"\n",
"In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with the Virtual Try-On model to:\n",
"- Try-on multiple clothing items from locally stored images\n",
"- Try-on a clothing item in Cloud Storage with an Imagen generated person\n",
"\n",
"Learn more about [quotas](https://cloud.google.com/vertex-ai/generative-ai/docs/models/imagen/virtual-try-on-preview-08-04) and [pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing#imagen-models) for Virtual Try-On in the product documentation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mvKl-BtQTRiQ"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B-UOCMvJdmlq"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "u5lOntr-doIT"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "opUxT_k5TdgP"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vbNgv4q1T2Mi"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Sh5eUOaLsJ46"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ybBXSukZkgjg"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "q7YvbXXdtzDT"
},
"outputs": [],
"source": [
"import os\n",
"import typing\n",
"import urllib.request\n",
"\n",
"import IPython.display\n",
"import matplotlib.image as img\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from PIL import Image as PIL_Image\n",
"from PIL import ImageOps as PIL_ImageOps\n",
"from google import genai\n",
"from google.genai.types import (\n",
" GenerateImagesConfig,\n",
" Image,\n",
" ProductImage,\n",
" RecontextImageConfig,\n",
" RecontextImageSource,\n",
")\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ic4fVqs8skdA"
},
"source": [
"### Define helper functions to display media"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "hk8bfm7Xsvg8"
},
"outputs": [],
"source": [
"def display_image(\n",
" image,\n",
" max_width: int = 700,\n",
" max_height: int = 400,\n",
") -> None:\n",
" pil_image = typing.cast(PIL_Image.Image, image._pil_image)\n",
" if pil_image.mode != \"RGB\":\n",
" # RGB is supported by all Jupyter environments (e.g. RGBA is not yet)\n",
" pil_image = pil_image.convert(\"RGB\")\n",
" image_width, image_height = pil_image.size\n",
" if max_width < image_width or max_height < image_height:\n",
" # Resize to display a smaller notebook image\n",
" pil_image = PIL_ImageOps.contain(pil_image, (max_width, max_height))\n",
" IPython.display.display(pil_image)\n",
"\n",
"\n",
"def display_local_image(\n",
" images: list[str],\n",
") -> None:\n",
" fig, axes = plt.subplots(1, len(images), figsize=(12, 6))\n",
" if len(images) == 1:\n",
" axes = np.array([axes])\n",
" for i, ax in enumerate(axes):\n",
" image = img.imread(images[i])\n",
" ax.imshow(image)\n",
" ax.axis(\"off\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qnEcP0UUsy-_"
},
"source": [
"### Load the image models"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "w_TfEzggs2fp"
},
"outputs": [],
"source": [
"virtual_try_on = \"virtual-try-on-001\"\n",
"image_generation = \"imagen-4.0-generate-001\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fU32286ooc8Q"
},
"source": [
"## Virtual Try-On with local files\n",
"\n",
"In this section, you'll download images of a person and clothing items to try them on from local files.\n",
"\n",
"Supported Clothing:\n",
" - **Tops:** shirts, hoodies, sweaters, tank tops, blouses\n",
" - **Bottoms:** pants, leggings, shorts, skirts\n",
" - **Other:** shoes, full body items\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "faTZezkUs7KP"
},
"source": [
"### Download an image of a person\n",
"\n",
"First, download an image of a person. The following example uses an image from Cloud Storage. If you prefer to use a different image, you can either change the URL in the `wget` command or, if you have a local file, update the `person_image` variable in the subsequent step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "19kR3OR7Fwgv"
},
"outputs": [],
"source": [
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/man-in-field.png"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p04U4s73tRe2"
},
"source": [
"If you'd like to use a different local image, modify the file name in `person_image`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Spzul-ObF00N"
},
"outputs": [],
"source": [
"person_image = \"man-in-field.png\" # @param {type: 'string'}\n",
"\n",
"display_local_image([person_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SRgtZr0YteN2"
},
"source": [
"### Download clothing images\n",
"\n",
"Next, download the clothing images. The examples below are stored in Cloud Storage, but you can use your own by modifying the URLs or by specifying the paths to local images in the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b5HxMgDpMQta"
},
"outputs": [],
"source": [
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/sweater.jpg\n",
"\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/trousers.jpg"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mm-jYPq5t7yN"
},
"source": [
"If you'd like to use a different top, modify the file name in `top_image`. The same goes for the file name in `bottom_image`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IY0LHYHfMjkK"
},
"outputs": [],
"source": [
"top_image = \"sweater.jpg\" # @param {type: 'string'}\n",
"bottom_image = \"trousers.jpg\" # @param {type: 'string'}\n",
"\n",
"display_local_image([top_image, bottom_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mTgQo5Hauk92"
},
"source": [
"### Send the request\n",
"\n",
"With the Virtual Try-On model, you can only specify one clothing item to try on at a time. Since this example has two clothing items, you'll need to make two separate requests. In each call, you can specify the following parameters in addition to the `person_image` and `product_images`:\n",
" - **Number of images:** 1 - 4\n",
"\n",
"You'll save the output image locally so that it can be referenced in the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RHnSHJl-t8ud"
},
"outputs": [],
"source": [
"response = client.models.recontext_image(\n",
" model=virtual_try_on,\n",
" source=RecontextImageSource(\n",
" person_image=Image.from_file(location=person_image),\n",
" product_images=[\n",
" ProductImage(product_image=Image.from_file(location=top_image))\n",
" ],\n",
" ),\n",
" config=RecontextImageConfig(\n",
" output_mime_type=\"image/jpeg\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_LOW_AND_ABOVE\",\n",
" ),\n",
")\n",
"\n",
"response.generated_images[0].image.save(\"try-on.jpeg\")\n",
"display_image(response.generated_images[0].image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cP_gkDUVx-3Z"
},
"source": [
"When generating images you can also set the `safety_filter_level` parameter accordingly:\n",
"- `safety_filter_level`\n",
" - `BLOCK_LOW_AND_ABOVE`\n",
" - `BLOCK_MEDIUM_AND_ABOVE`\n",
" - `BLOCK_ONLY_HIGH`\n",
" - `BLOCK_NONE`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JKtLpyrZ3wA5"
},
"outputs": [],
"source": [
"response = client.models.recontext_image(\n",
" model=virtual_try_on,\n",
" source=RecontextImageSource(\n",
" person_image=Image.from_file(location=\"try-on.jpeg\"),\n",
" product_images=[\n",
" ProductImage(product_image=Image.from_file(location=bottom_image))\n",
" ],\n",
" ),\n",
" config=RecontextImageConfig(\n",
" output_mime_type=\"image/jpeg\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_LOW_AND_ABOVE\",\n",
" ),\n",
")\n",
"display_image(response.generated_images[0].image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TmLGH9LAd371"
},
"source": [
"### Try-on multiple items at once\n",
"\n",
"By combining multiple clothing items into one photo, you can have the model try-on multiple clothing items in one request. You'll start by downloading and displaying a new model image and outfit image for this example. Like before, you can use your own images by modifying the URLs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "na3_ELDnefNI"
},
"outputs": [],
"source": [
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/woman.jpeg\n",
"\n",
"!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/outfit.png\n",
"\n",
"person_image = \"woman.jpeg\" # @param {type: 'string'}\n",
"outfit_image = \"outfit.png\" # @param {type: 'string'}\n",
"\n",
"display_local_image([person_image, outfit_image])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xMKIaRsWgxSw"
},
"source": [
"Now, you'll send the request."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bhJhmUraWngz"
},
"outputs": [],
"source": [
"response = client.models.recontext_image(\n",
" model=virtual_try_on,\n",
" source=RecontextImageSource(\n",
" person_image=Image.from_file(location=person_image),\n",
" product_images=[\n",
" ProductImage(product_image=Image.from_file(location=outfit_image))\n",
" ],\n",
" ),\n",
" config=RecontextImageConfig(\n",
" output_mime_type=\"image/png\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_LOW_AND_ABOVE\",\n",
" ),\n",
")\n",
"display_image(response.generated_images[0].image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mBLJtICO8iMQ"
},
"source": [
"## Virtual Try-On with files in Cloud Storage\n",
"\n",
"In this section, you'll use images of a clothing item stored in Cloud Storage and a person generated with Imagen for virtual try-on."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "08ZEbEDgyyCB"
},
"source": [
"### Generate an image of a person\n",
"\n",
"In this example, you'll generate a person to try on the clothing item with Imagen. Run the step below and change the prompt if you see fit."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I9caIrZ7Dek1"
},
"outputs": [],
"source": [
"prompt = \"\"\"\n",
"A high-resolution, full-body, head-on photograph of a woman standing in a photography studio wearing jeans and a white t-shirt. The backdrop is clean and neutral.\n",
"\"\"\"\n",
"\n",
"image = client.models.generate_images(\n",
" model=image_generation,\n",
" prompt=prompt,\n",
" config=GenerateImagesConfig(\n",
" output_mime_type=\"image/jpeg\",\n",
" number_of_images=1,\n",
" image_size=\"2K\",\n",
" safety_filter_level=\"BLOCK_MEDIUM_AND_ABOVE\",\n",
" ),\n",
")\n",
"display_image(image.generated_images[0].image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l2p7TJtXzhpM"
},
"source": [
"### View clothing image\n",
"\n",
"Here, you'll view the clothing image that is stored in Cloud Storage. If you'd like, you can modify the URL below to use a different image in Cloud Storage."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qj70GAQjfCPC"
},
"outputs": [],
"source": [
"clothing_image = PIL_Image.open(\n",
" urllib.request.urlopen(\n",
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/dress.jpg\"\n",
" )\n",
")\n",
"\n",
"# Display the image\n",
"fig, axis = plt.subplots(1, 2, figsize=(12, 6))\n",
"axis[0].imshow(clothing_image)\n",
"for ax in axis:\n",
" ax.axis(\"off\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "P-yAQ2rt_pll"
},
"source": [
"### Send the request\n",
"\n",
"By default, a digital watermark, or [SynthID](https://deepmind.google/technologies/synthid/), is added to images. If you would like to explicitly set the watermark to True, you can do so with the `add_watermark` parameter."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c9N8l0oo_cWs"
},
"outputs": [],
"source": [
"response = client.models.recontext_image(\n",
" model=virtual_try_on,\n",
" source=RecontextImageSource(\n",
" person_image=image.generated_images[0].image,\n",
" product_images=[\n",
" ProductImage(\n",
" product_image=Image(\n",
" gcs_uri=\"gs://cloud-samples-data/generative-ai/image/dress.jpg\"\n",
" )\n",
" )\n",
" ],\n",
" ),\n",
" config=RecontextImageConfig(\n",
" output_mime_type=\"image/jpeg\",\n",
" number_of_images=1,\n",
" safety_filter_level=\"BLOCK_LOW_AND_ABOVE\",\n",
" ),\n",
")\n",
"display_image(response.generated_images[0].image)"
]
}
],
"metadata": {
"colab": {
"name": "virtual_try_on.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}