GenMedia Live enables multimodal AI creation by extending the Gemini Live API capabilities to image and video generation using Gemini Pro Image and Veo through function calling.

You can interact with voice or text to:

Generate images from descriptions or edit existing ones
Create videos from text prompts or animate images
Create short movies by combining multiple video clips
Ask questions about what you show on camera or upload
Extract frames from videos for further editing

Users can reference previously generated images, or use images from their camera, screen share, or upload images from the UI as reference.

Features

Voice Interaction: Real-time voice conversations with the Gemini Live API
Text Input: Type messages as an alternative to voice
Camera: Share your camera for visual context
Screen Share: Share your screen for visual context
Image Upload: Upload images from your device as reference
Image Generation: Create and edit images using Gemini Pro Image
Video Generation: Generate videos with Veo, including image-to-video
Frame Extraction: Extract frames from videos using ffmpeg
Video Combining: Merge multiple videos into one using ffmpeg
Session Management: 30-minute conversation history with automatic reconnection

Prerequisites

Python 3.10+
Google Cloud project with Vertex AI API enabled
ffmpeg (for video operations)

Installing ffmpeg

Linux (Ubuntu/Debian):

sudo apt update && sudo apt install ffmpeg

macOS:

brew install ffmpeg

Windows: Download from ffmpeg.org and add to PATH.

Setup

Clone the repository:

git clone https://github.com/GoogleCloudPlatform/generative-ai.git
cd generative-ai/vision/sample-apps/genmedia-live

Create and activate virtual environment:

python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate  # Windows

Install dependencies:

pip install -r requirements.txt

Run the application:

python app.py

Open http://localhost:8080 in your browser.

Authentication

GenMedia Live requires users to authenticate with their own Google Cloud credentials.

Steps to Authenticate:

Enter your Project ID in the sidebar
Click "Open Cloud Shell" - this opens Google Cloud Shell in a new tab
Run the command in Cloud Shell:
```
gcloud auth print-access-token
```
Copy the token and paste it in the "Paste Token" field
Click "Validate & Connect"

If validation succeeds, you'll see "Connected!" and can start using the app.

Note

: Access tokens expire after ~1 hour. You'll need to generate a new token if your session expires.

Usage

Complete authentication (see above)
Click Voice to start a voice conversation, or type in the text box
Click Camera to share your camera as visual context
Click Screen to share your screen as visual context
Use Upload to add reference images from your device
Ask GenMedia Live to generate images or videos

Project Structure

genmedia-live/
├── app.py                 # Flask backend
├── index.html             # Main page
├── style.css              # Global styles
├── requirements.txt       # Python dependencies
├── outputs/               # Generated files
│   ├── images/
│   └── videos/
└── src/
    ├── main.js            # App initialization
    ├── ui.js              # UI utilities
    └── features/
        ├── genmedia-chat.js       # Voice chat functionality
        └── templates/
            └── genmedia-chat.html

Deployment

Cloud Run

gcloud run deploy genmedia-live \
  --source . \
  --region us-central1 \
  --allow-unauthenticated

License

Licensed under the Apache License, Version 2.0.