The Multimodal Live API is a low-latency bidirectional streaming API that supports text, audio and video input, with audio and text output. This capability allows for natural, human-like conversations, giving you the option to interrupt the model at any point in the interaction. The model's abilities are further enhanced through video understanding, enabling you to share live video input and screencasts so that Gemini can reason across this information and respond to questions. You can also provide system instructions to better control the model's output.

Demo

The following video demonstrating the Multimodal Live API shows an example use case of conversing with the model. This demo app can be run locally or via Cloud Shell so that you can test your own use cases. Instructions to get this application running are in the websocket-demo-app directory.

Getting Started

Intro Notebooks

Multimodal Live API: Directly access the Multimodal Live API. This notebook will demonstrate text-to-text generation, as well as text-to-audio generation.
Multimodal Live API via Gen AI SDK: Use this tutorial to access the Multimodal Live API using the Google Gen AI SDK in Vertex AI. You'll see examples of text-to-text generation and text-to-audio generation.

Use Cases

Interactive Loan Application Assistant: This notebook provides a comprehensive demonstration of how Gemini 2.0 can act as your personal file assistant across various storage platforms. It empowers users to seamlessly understand and interact with their loan documents.
Real-time RAG for Retail: Users will learn to develop a real-time Retrieval Augmented Generation (RAG) system leveraging the Multimodal Live API for a retail use-case. This system will generate audio and text responses grounded in provided documents.

Demo Applications

Getting Started Web App: A basic web application to get started with voice and camera interactions.
React Demo App: A comprehensive React client featuring real-time streaming, tool use, and media handling.
Plain JS Demo App: A simple, dependency-free JavaScript implementation for understanding core API mechanics.
Plain JS + Python SDK Demo App: A demo using the Google Gen AI Python SDK for the backend and vanilla JS for the frontend.
Real-time Advisor: A specialized advisor persona that can switch between silent and outspoken modes.
Customer Support Agent: An advanced agent with emotion detection, multimodal input, and tool execution.
Gaming Assistant: A gaming companion with persona switching and screen sharing capabilities.

Tools

PCM Audio Debugger: A standalone tool for testing and debugging raw PCM audio streams and WebSocket connections.