wehub-resource-sync
786 lines
29 KiB
Plaintext
786 lines
29 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Jv_d8Y2XCyRm"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2026 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WIgrkCFIC3Fq"
|
|
},
|
|
"source": [
|
|
"# Gemini Omni Flash Video Generation\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fgemini_omni_flash_video_gen.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/gemini_omni_flash_video_gen.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NbZi7r8HDLd_"
|
|
},
|
|
"source": [
|
|
"| Author |\n",
|
|
"| --- |\n",
|
|
"| [Katie Nguyen](https://github.com/katiemn) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "COhyHClwDPFM"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"Gemini Omni Flash is a model that understands the world around you, allowing you to animate photos or create videos from any input. Built on Gemini's world understanding and native multimodality, Gemini Omni Flash creates outputs that reflect the logic of the real world and lets you shape them step-by-step through natural conversation.\n",
|
|
"\n",
|
|
"In this tutorial, you'll learn how to use Gemini Omni Flash in Agent Platform with the Google Gen AI SDK to try out the following scenarios:\n",
|
|
"\n",
|
|
"- Video generation:\n",
|
|
" - Text-to-video generation\n",
|
|
" - Providing a reference image as a starting frame\n",
|
|
" - Supplying reference images to guide video generation\n",
|
|
" - Async video generation\n",
|
|
"- Video editing:\n",
|
|
" - Prompt-based video editing\n",
|
|
" - Multi-turn video editing (chat)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "gIpYz_L8D9eK"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "axSoEhOzERDm"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK for Python"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mfJgJBMeLAWi"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "cfk06hgxEiYS"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NW-We3ATemq6"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YF5OKjXxEpp4"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "f1HU9IT_erEb"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import base64\n",
|
|
"\n",
|
|
"import matplotlib.image as img\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from IPython.display import Video, display\n",
|
|
"from google import genai\n",
|
|
"from google.genai import interactions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-yg0mBBaEtXn"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "80On1BQxetma"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
|
|
"\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "P2OgDTYvEzZs"
|
|
},
|
|
"source": [
|
|
"### Define a helper function to display media"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ZDN9qlAGe44s"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def show_video(video_bytes):\n",
|
|
" data = base64.b64decode(video_bytes)\n",
|
|
" with open(\"sample.mp4\", \"wb\") as out_file:\n",
|
|
" out_file.write(data)\n",
|
|
" display(Video(\"sample.mp4\", embed=True, width=600))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Cj-ZmZbME24B"
|
|
},
|
|
"source": [
|
|
"### Load the video generation model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "vg2SdM1ue7IM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"omni_model = \"gemini-omni-flash-preview\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "hLHGlUYNLIjB"
|
|
},
|
|
"source": [
|
|
"## Video generation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WxaeXTZlWgrI"
|
|
},
|
|
"source": [
|
|
"### Generate videos from a text prompt\n",
|
|
"\n",
|
|
"With Gemini Omni Flash, you can generate videos directly from text prompts via the [Interactions API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/interactions-api). To generate a video in the sample below, specify the following information:\n",
|
|
"- **Prompt:** A detailed description of the video you would like to see. For best results, consider attributes such as shot framing, motion, style, lighting, location, and action.\n",
|
|
"- **Generation Config:** Within this configuration, you can specify a `VideoConfig` that contains a `task` parameter.\n",
|
|
" - **Task:** Valid options are `text_to_video`, `image_to_video`, `reference_to_video`, or `edit`. Make sure to set each task accordingly based on your inputs and desired behavior.\n",
|
|
"- **Video Response Format:** Configure `aspect_ratio`, `duration`, and `delivery` parameters. Alternatively, you can specify these configurations through the text prompt.\n",
|
|
" - **Aspect ratio:** 16:9 or 9:16\n",
|
|
" - **Duration:** 3s - 10s\n",
|
|
" - **Delivery:** If you'd like to save your generated videos to GCS, set `delivery=\"uri\"`. Additionally, set `gcs_uri=\"gs://<GCS_BUCKET>\"`, making sure to replace `<GCS_BUCKET>` with your desired bucket path. To find your video's location, look inside the content list and copy the value next to `uri=` in the `VideoContent` object.\n",
|
|
"\n",
|
|
"\n",
|
|
"Notes on output videos:\n",
|
|
"- **Audio generation:** Audio will be generated alongside the output video.\n",
|
|
"- **Resolution:** 720p\n",
|
|
"\n",
|
|
"All videos from `gemini-omni-flash-preview` include both [C2PA metadata](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/content-credentials) and a [SynthID watermark](https://deepmind.google/technologies/synthid/)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mk2tOv2Rf7e8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"The words Gemini Omni Flash displayed boldly and centered. The words rapidly and instantly transform, cycling through different styles, fonts, and materials. The words remains perfectly positioned and constant while the colors, textures, and environments instantaneously change around it.\"\n",
|
|
"\n",
|
|
"interaction = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=prompt,\n",
|
|
" generation_config=interactions.GenerationConfig(\n",
|
|
" video_config=interactions.VideoConfig(\n",
|
|
" task=\"text_to_video\"\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" response_format=interactions.VideoResponseFormat(\n",
|
|
" aspect_ratio=\"16:9\",\n",
|
|
" duration=\"9s\",\n",
|
|
" # delivery=\"uri\",\n",
|
|
" # gcs_uri=\"gs://<GCS_BUCKET>\"\n",
|
|
" )\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-aV_ahDKWjPU"
|
|
},
|
|
"source": [
|
|
"### Video generation from a starting image"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "v3xtidCsGiCH"
|
|
},
|
|
"source": [
|
|
"#### Download the starting image\n",
|
|
"\n",
|
|
"You can also generate a video by starting with an input image. In this example, you'll locally download an image that's stored in Google Cloud Storage. If you'd like, you can provide the URL of an image to display it below. If you have a local image you'd like to use, you can specify that in the following steps."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "p6TPPMW66XCl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/suitcase.png"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "oi7eAyOhGmoU"
|
|
},
|
|
"source": [
|
|
"If you'd like to use a different local image, modify the file name in `start`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "qsneJ_lP6cKk"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"start = \"suitcase.png\"\n",
|
|
"\n",
|
|
"fig, ax1 = plt.subplots(1, 1, figsize=(12, 6))\n",
|
|
"ax1.imshow(img.imread(start))\n",
|
|
"ax1.axis(\"off\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "2_g6-aP661EV"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"A single continuous shot of a hard-shell suitcase rolling. It stops, stands vertically, unzips alongside and then opens in half. The wheels and shell stay solid. White bubbly retro text reading 'Omnicase' pops out of the center. Colorful travel stickers pop up around the text: a plane, palm tree, boat, city skyline, and a train.\"\n",
|
|
"\n",
|
|
"with open(start, \"rb\") as f:\n",
|
|
" img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n",
|
|
"\n",
|
|
"interaction = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=[\n",
|
|
" {\"type\": \"text\", \"text\": prompt},\n",
|
|
" {\"type\": \"image\", \"mime_type\": \"image/png\", \"data\": img_b64},\n",
|
|
" ],\n",
|
|
" generation_config=interactions.GenerationConfig(\n",
|
|
" video_config=interactions.VideoConfig(\n",
|
|
" task=\"image_to_video\"\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YdBtdsaLWo0F"
|
|
},
|
|
"source": [
|
|
"### Reference inputs in video generation\n",
|
|
"\n",
|
|
"With Gemini Omni Flash, you can use existing media to generate and edit new videos. The model processes these inputs in one of two ways:\n",
|
|
"\n",
|
|
" - **Source Media:** This content acts as the direct foundation for the final output.\n",
|
|
" - *Examples:* In image-to-video, the provided image becomes the literal first frame. In video editing, the original video is modified via natural language prompts while keeping its core structure intact.\n",
|
|
" - **Reference Media:** This content acts as a guide or inspiration rather than the literal base. You are asking the model to extract a specific style, subject, or mood to creatively reimagine it.\n",
|
|
" - *Examples:* Storyboard frames, product images, style guides, audio jingles, or reference scenery. You can also use reference media for video editing—such as providing an image of a new character or art style to apply to a base video.\n",
|
|
"\n",
|
|
"\n",
|
|
"**NOTE:** Video and audio reference inputs are not currently supported."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "jVjssXdqMrMo"
|
|
},
|
|
"source": [
|
|
"#### Download and display reference images\n",
|
|
"\n",
|
|
"In this example, you'll locally download two images that are stored in Google Cloud Storage."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "5FvshnXiyNKb"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/woman.jpeg\n",
|
|
"\n",
|
|
"!wget -q https://storage.googleapis.com/cloud-samples-data/generative-ai/image/arcade-game.png"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "j1jN_J7onPyY"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"character = \"woman.jpeg\"\n",
|
|
"product = \"arcade-game.png\"\n",
|
|
"\n",
|
|
"# Display the images\n",
|
|
"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))\n",
|
|
"ax1.imshow(img.imread(character))\n",
|
|
"ax2.imshow(img.imread(product))\n",
|
|
"ax1.axis(\"off\")\n",
|
|
"ax2.axis(\"off\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "1ZLbIRXDJa7T"
|
|
},
|
|
"source": [
|
|
"**Tip:** You can supply storyboard reference images to better direct the video without supplying all details in a text prompt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LKofc-c5mH_H"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"A woman walks up to the Omni-Sphere game on the side wall of a bowling alley. Bass music begins to play as she starts playing the game by pressing a button. 9:16 aspect ratio. 7 second video.\"\n",
|
|
"\n",
|
|
"image_files = [character, product]\n",
|
|
"\n",
|
|
"images_input = []\n",
|
|
"for img_path in image_files:\n",
|
|
" with open(img_path, \"rb\") as f:\n",
|
|
" img_b64 = base64.b64encode(f.read()).decode(\"utf-8\")\n",
|
|
" images_input.append({\"type\": \"image\", \"mime_type\": \"image/jpeg\",\n",
|
|
" \"data\": img_b64})\n",
|
|
"\n",
|
|
"interaction = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=[\n",
|
|
" {\"type\": \"text\", \"text\": prompt},\n",
|
|
" *images_input,\n",
|
|
" ],\n",
|
|
" generation_config=interactions.GenerationConfig(\n",
|
|
" video_config=interactions.VideoConfig(\n",
|
|
" task=\"reference_to_video\"\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "qWVoa9xpi6l-"
|
|
},
|
|
"source": [
|
|
"### Async video generation\n",
|
|
"\n",
|
|
"To generate a video that can be checked on later, set the `background` parameter to True. You can then use `interactions.get` to check the status of the initial interaction ID."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "nVOn-pwrgcil"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"prompt = \"Raw cookie ingredients beautifully laid out on a kitchen counter. The view glides smoothly from one ingredient to the next. As each individual ingredient fills the view, a sleek 3D text graphic pops up to label it. The sequence finishes by resting intimately on the rich textures and details of the final ingredient after the label disappears.\"\n",
|
|
"\n",
|
|
"initial_interaction = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=prompt,\n",
|
|
" background=True,\n",
|
|
")\n",
|
|
"interaction = initial_interaction\n",
|
|
"\n",
|
|
"while interaction.status not in [\"completed\", \"failed\"]:\n",
|
|
" print(f\"Current Status: {interaction.status}\")\n",
|
|
" time.sleep(10)\n",
|
|
" interaction = client.interactions.get(id=initial_interaction.id)\n",
|
|
"\n",
|
|
"if interaction.status == \"completed\":\n",
|
|
" contents = []\n",
|
|
" steps = getattr(interaction, \"steps\", []) or []\n",
|
|
" for step in steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
" show_video(contents[0].data)\n",
|
|
"else:\n",
|
|
" print(f\"Status: {interaction.status}\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "VlXnMxwyWsfM"
|
|
},
|
|
"source": [
|
|
"## Video editing"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SJs81D2SNf4e"
|
|
},
|
|
"source": [
|
|
"#### Download and display the video\n",
|
|
"\n",
|
|
"In this example, you'll locally download a video and an image that are stored in Google Cloud Storage.\n",
|
|
"\n",
|
|
"**Note:** You can supply reference images when editing source videos. The input source video must be under 10 seconds."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "DOJhxrMKGfiK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from PIL import Image as PIL_IMAGE\n",
|
|
"\n",
|
|
"video_uri = \"gs://cloud-samples-data/generative-ai/video/dog_day1.mp4\"\n",
|
|
"image_uri = \"gs://cloud-samples-data/generative-ai/image/chair-cat.png\"\n",
|
|
"\n",
|
|
"!gcloud storage cp {video_uri} {\"dog_day1.mp4\"}\n",
|
|
"!gcloud storage cp {image_uri} {\"chair-cat.png\"}\n",
|
|
"\n",
|
|
"display(Video(\"dog_day1.mp4\", embed=True, width=600))\n",
|
|
"\n",
|
|
"fig, ax1 = plt.subplots(1, 1, figsize=(6, 6))\n",
|
|
"ax1.imshow(PIL_IMAGE.open(\"chair-cat.png\"))\n",
|
|
"ax1.axis(\"off\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5lHP-aNbJwLS"
|
|
},
|
|
"source": [
|
|
"With video editing, you can add, remove, or alter existing objects in the source video. You can also change the content in the video to a different style or reference provided images."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "lWvtlz6AIRwe"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"Change the dog to the cat. Remove the backpack and add a propeller hat. Change the tennis balls to balls of yarn.\"\n",
|
|
"interaction = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=[\n",
|
|
" {\"type\": \"text\", \"text\": prompt},\n",
|
|
" {\"type\": \"image\", \"mime_type\": \"image/png\", \"uri\": image_uri},\n",
|
|
" {\"type\": \"video\", \"mime_type\": \"video/mp4\", \"uri\": video_uri},\n",
|
|
" ],\n",
|
|
" generation_config=interactions.GenerationConfig(\n",
|
|
" video_config=interactions.VideoConfig(\n",
|
|
" task=\"edit\"\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "k1yLTAvyWx0x"
|
|
},
|
|
"source": [
|
|
"### Multi-turn editing (chat)\n",
|
|
"\n",
|
|
"You can iteratively chat with the model to continuously change aspects of the video using the example below.\n",
|
|
"\n",
|
|
"To successfully accomplish this using the Interactions API, make sure to provide the previous chat history in the `input` parameter."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Z9qWfdgEKg_U"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt1 = \"A claymation explainer of Newton's First Law of Motion, everything is made out of clay, no hands, stop motion, sync character's words to mouth movements, show a cute ball character moving and then being stopped by a wall\"\n",
|
|
"\n",
|
|
"interaction1 = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=prompt1,\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction1.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "XcEvGMlQKqfg"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"interaction1.steps"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4OudxC1_KtoK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt2 = \"Now make the same video in a doodle style.\"\n",
|
|
"\n",
|
|
"turn2_input = interaction1.steps + [\n",
|
|
" {\n",
|
|
" \"type\": \"user_input\",\n",
|
|
" \"content\": [\n",
|
|
" {\n",
|
|
" \"type\": \"text\",\n",
|
|
" \"text\": prompt2\n",
|
|
" }\n",
|
|
" ]\n",
|
|
" }\n",
|
|
"]\n",
|
|
"\n",
|
|
"interaction2 = client.interactions.create(\n",
|
|
" model=omni_model,\n",
|
|
" input=turn2_input,\n",
|
|
")\n",
|
|
"\n",
|
|
"contents = []\n",
|
|
"for step in interaction2.steps:\n",
|
|
" if step.type == \"model_output\":\n",
|
|
" contents.extend(step.content)\n",
|
|
"\n",
|
|
"show_video(contents[0].data)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "2ycGCBPq8pRI"
|
|
},
|
|
"source": [
|
|
"\n",
|
|
"**Note:** Support for audio references, video references, last frame, scene extension, and higher resolutions for the Gemini Omni Flash via Gemini Enterprise Agent Platform API will be available soon."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"name": "python"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|