wehub-resource-sync
142 lines
4.2 KiB
Markdown
142 lines
4.2 KiB
Markdown
# GenMedia Live
|
|
|
|
Author: [Wafae Bakkali](https://github.com/WafaeBakkali)
|
|
|
|
**Real-time multimodal creation enabled by extending the Gemini Live API with image and video generation tools.**
|
|
|
|
GenMedia Live enables multimodal AI creation by extending the [Gemini Live API](https://cloud.google.com/vertex-ai/generative-ai/docs/live-api) capabilities to image and video generation using Gemini Pro Image and Veo through function calling.
|
|
|
|
You can interact with voice or text to:
|
|
- **Generate images** from descriptions or edit existing ones
|
|
- **Create videos** from text prompts or animate images
|
|
- **Create short movies** by combining multiple video clips
|
|
- **Ask questions** about what you show on camera or upload
|
|
- **Extract frames** from videos for further editing
|
|
|
|
Users can reference previously generated images, or use images from their camera, screen share, or upload images from the UI as reference.
|
|
|
|
## Features
|
|
|
|
- **Voice Interaction**: Real-time voice conversations with the Gemini Live API
|
|
- **Text Input**: Type messages as an alternative to voice
|
|
- **Camera**: Share your camera for visual context
|
|
- **Screen Share**: Share your screen for visual context
|
|
- **Image Upload**: Upload images from your device as reference
|
|
- **Image Generation**: Create and edit images using Gemini Pro Image
|
|
- **Video Generation**: Generate videos with Veo, including image-to-video
|
|
- **Frame Extraction**: Extract frames from videos using ffmpeg
|
|
- **Video Combining**: Merge multiple videos into one using ffmpeg
|
|
- **Session Management**: 30-minute conversation history with automatic reconnection
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.10+
|
|
- Google Cloud project with Vertex AI API enabled
|
|
- ffmpeg (for video operations)
|
|
|
|
### Installing ffmpeg
|
|
|
|
**Linux (Ubuntu/Debian):**
|
|
```bash
|
|
sudo apt update && sudo apt install ffmpeg
|
|
```
|
|
|
|
**macOS:**
|
|
```bash
|
|
brew install ffmpeg
|
|
```
|
|
|
|
**Windows:**
|
|
Download from [ffmpeg.org](https://ffmpeg.org/download.html) and add to PATH.
|
|
|
|
## Setup
|
|
|
|
1. Clone the repository:
|
|
```bash
|
|
git clone https://github.com/GoogleCloudPlatform/generative-ai.git
|
|
cd generative-ai/vision/sample-apps/genmedia-live
|
|
```
|
|
|
|
2. Create and activate virtual environment:
|
|
```bash
|
|
python -m venv venv
|
|
source venv/bin/activate # Linux/Mac
|
|
# or
|
|
venv\Scripts\activate # Windows
|
|
```
|
|
|
|
3. Install dependencies:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
4. Run the application:
|
|
```bash
|
|
python app.py
|
|
```
|
|
|
|
5. Open http://localhost:8080 in your browser.
|
|
|
|
## Authentication
|
|
|
|
GenMedia Live requires users to authenticate with their own Google Cloud credentials.
|
|
|
|
### Steps to Authenticate:
|
|
|
|
1. **Enter your Project ID** in the sidebar
|
|
2. **Click "Open Cloud Shell"** - this opens Google Cloud Shell in a new tab
|
|
3. **Run the command** in Cloud Shell:
|
|
```bash
|
|
gcloud auth print-access-token
|
|
```
|
|
4. **Copy the token** and paste it in the "Paste Token" field
|
|
5. **Click "Validate & Connect"**
|
|
|
|
If validation succeeds, you'll see "Connected!" and can start using the app.
|
|
|
|
> **Note**: Access tokens expire after ~1 hour. You'll need to generate a new token if your session expires.
|
|
|
|
## Usage
|
|
|
|
1. Complete authentication (see above)
|
|
2. Click **Voice** to start a voice conversation, or type in the text box
|
|
3. Click **Camera** to share your camera as visual context
|
|
4. Click **Screen** to share your screen as visual context
|
|
5. Use **Upload** to add reference images from your device
|
|
6. Ask GenMedia Live to generate images or videos
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
genmedia-live/
|
|
├── app.py # Flask backend
|
|
├── index.html # Main page
|
|
├── style.css # Global styles
|
|
├── requirements.txt # Python dependencies
|
|
├── outputs/ # Generated files
|
|
│ ├── images/
|
|
│ └── videos/
|
|
└── src/
|
|
├── main.js # App initialization
|
|
├── ui.js # UI utilities
|
|
└── features/
|
|
├── genmedia-chat.js # Voice chat functionality
|
|
└── templates/
|
|
└── genmedia-chat.html
|
|
```
|
|
|
|
## Deployment
|
|
|
|
### Cloud Run
|
|
|
|
```bash
|
|
gcloud run deploy genmedia-live \
|
|
--source . \
|
|
--region us-central1 \
|
|
--allow-unauthenticated
|
|
```
|
|
|
|
## License
|
|
|
|
Licensed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).
|