wehub-resource-sync
386 lines
14 KiB
Plaintext
386 lines
14 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "14fb06d20aa6"
|
|
},
|
|
"source": [
|
|
"# Identifcation of Scene Transitions in Movies Using Gemini\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fvideo-analysis%2Fscene_transition.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/video-analysis/scene_transition.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/scene_transition.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| Author |\n",
|
|
"| --- |\n",
|
|
"| [Vijaylaxmi Lendale](https://github.com/VJlaxmi) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"- This notebook demonstrates how to use Gemini to automatically detect **scene transitions** in videos using both **video content** and associated **subtitle (VTT) files**."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK and other required packages\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dmWOrTJ3gx13"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
|
"import os\n",
|
|
"\n",
|
|
"from google import genai\n",
|
|
"\n",
|
|
"PROJECT = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"LOCATION = \"us-central1\"\n",
|
|
"if not PROJECT or PROJECT == \"[your-project-id]\":\n",
|
|
" PROJECT = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"client = genai.Client(vertexai=True, project=PROJECT, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5303c05f7aa6"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "6fc324893334"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from google.genai.types import GenerateContentConfig, Part"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e43229f3ad4f"
|
|
},
|
|
"source": [
|
|
"### Output schema and model config"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3a66645e2210"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pydantic import BaseModel, Field\n",
|
|
"\n",
|
|
"\n",
|
|
"class Scene(BaseModel):\n",
|
|
" scene_number: int = Field(\n",
|
|
" ..., description=\"The scene number in sequential order, e.g., 1, 2, 3.\"\n",
|
|
" )\n",
|
|
" start_time: str = Field(\n",
|
|
" ...,\n",
|
|
" description=\"The start time of the scene in the format HRS:MIN:SEC (e.g., 01:15:30, 00:12:45).\",\n",
|
|
" pattern=\"^([0-9]{2}):([0-9]{2}):([0-9]{2})$\",\n",
|
|
" )\n",
|
|
" end_time: str = Field(\n",
|
|
" ...,\n",
|
|
" description=\"The end time of the scene in the format HRS:MIN:SEC (e.g., 01:20:45, 00:20:18).\",\n",
|
|
" pattern=\"^([0-9]{2}):([0-9]{2}):([0-9]{2})$\",\n",
|
|
" )\n",
|
|
" description: str = Field(..., description=\"A brief description of the scene.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d0ecc98d434a"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"MODEL_ID = \"gemini-2.0-flash\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3ba3674af6dd"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"config = GenerateContentConfig(\n",
|
|
" temperature=0,\n",
|
|
" top_p=1,\n",
|
|
" max_output_tokens=8000,\n",
|
|
" response_mime_type=\"application/json\",\n",
|
|
" response_schema=list[Scene],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "56e2a1cb0cf4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Input video paths\n",
|
|
"input_video_path = (\n",
|
|
" \"gs://github-repo/generative-ai/gemini/use-cases/scene-transition/video3.mp4\"\n",
|
|
")\n",
|
|
"input_vtt_path = (\n",
|
|
" \"gs://github-repo/generative-ai/gemini/use-cases/scene-transition/video3.vtt\"\n",
|
|
")\n",
|
|
"\n",
|
|
"# Print paths\n",
|
|
"print(f\"Input Video Path: {input_video_path}\")\n",
|
|
"print(f\"Input VTT Path: {input_vtt_path}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "da820b7b4458"
|
|
},
|
|
"source": [
|
|
"### Prompt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4d16d3abb0b1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"base_instructions = \"\"\"\n",
|
|
"You are a multimodal Scene Boundary Detector.\n",
|
|
"Your task is to analyze a video along with its VTT (subtitle) file to identify cohesive and meaningful scene transitions, \n",
|
|
"ensuring accurate segmentation into self-contained scenes with distinct narrative arcs.\n",
|
|
"** Key criteria for identifying scene transitions:** \n",
|
|
"- Narrative Changes: Transitions must reflect a significant shift in story elements such as location, time, characters, or topic of dialogue.\n",
|
|
"- Don't treat jump-cuts or insert shots as transitions unless they signify meaningful narrative shifts.\n",
|
|
"- Visual Cues: Changes in location, character appearance, or recognition of visual elements strongly indicate scene changes.\n",
|
|
"- Dialogue Topics: Continuous dialogues between a stable set of characters typically belong to a single scene; changes in dialogue themes can signal scene transitions.\n",
|
|
"- Audio Elements: Shifts in background music or sound effects often accompany scene transitions, reinforcing narrative changes.\n",
|
|
"- Cohesion: Ensure each identified scene is cohesive, with a contained beginning, middle, and end, contributing to the overarching narrative.\n",
|
|
"Ensure that the scene transition timestamps you identify strictly fall within the start and end time boundaries of the input video, and accurately reflect the exact position of the scene boundaries in the video.\n",
|
|
"\"\"\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "45fb1bddc112"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_text(text=base_instructions),\n",
|
|
" \"Apply the scene transition instructions above.\",\n",
|
|
" Part.from_uri(file_uri=input_video_path, mime_type=\"video/mp4\"),\n",
|
|
" Part.from_uri(file_uri=input_vtt_path, mime_type=\"text/vtt\"),\n",
|
|
" ],\n",
|
|
" config=config,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ee1eefb07e8c"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"model_outputs = []\n",
|
|
"try:\n",
|
|
" model_outputs.append(response.parsed)\n",
|
|
"except (AttributeError, TypeError) as e:\n",
|
|
" print(f\"Couldn't access parsed response: {e}\")\n",
|
|
"\n",
|
|
"# Flatten list of lists and print scene transitions\n",
|
|
"for scenes in model_outputs:\n",
|
|
" for scene in scenes:\n",
|
|
" print(scene)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "scene_transition.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|