wehub-resource-sync
487 lines
20 KiB
Plaintext
487 lines
20 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "EWeQ3WOjAzDa"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "lvYXxb3XCf0W"
|
|
},
|
|
"source": [
|
|
"# Gemini Live API Quickstart\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fmultimodal-live-api%2Flive_api_quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
|
|
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "76b2vG9eCxLW"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "TfARntTuC2wn"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"This notebook demonstrates how to connect to the **Gemini Live API** for real-time, bidirectional audio streaming. You will learn to establish a session with a Gemini model, simulate an audio stream from a file, and play back the generated audio response.\n",
|
|
"\n",
|
|
"The guide provides two implementation examples:\n",
|
|
"\n",
|
|
"- [Gen AI SDK](https://github.com/googleapis/python-genai): A simplified approach using the Google **Gen AI SDK** to manage the session and handle interruptions.\n",
|
|
"\n",
|
|
"- [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API): A low-level approach using standard **WebSockets** to construct the handshake and manage raw JSON payloads."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kQX8-lbsC8JJ"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "_0OdbM-RDDw2"
|
|
},
|
|
"source": [
|
|
"### Install required libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "0JwX5N8fDhnc"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "EVo5SimcDWKF"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "BRtdID2HsInB"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import asyncio\n",
|
|
"import base64\n",
|
|
"import json\n",
|
|
"import os\n",
|
|
"import sys\n",
|
|
"import wave\n",
|
|
"\n",
|
|
"import numpy as np\n",
|
|
"import websockets\n",
|
|
"from IPython.display import Audio, display\n",
|
|
"from google import genai\n",
|
|
"from google.genai import types"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WBZPC10PDcxk"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "V-Ry6-JdDYKz"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "mdTSBOzZDhie"
|
|
},
|
|
"source": [
|
|
"### Authenticate your Google Cloud Project for Vertex AI\n",
|
|
"\n",
|
|
"You can use a Google Cloud Project or an API Key for authentication. This tutorial uses a Google Cloud Project.\n",
|
|
"\n",
|
|
"- [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AR6vYdRTsSfv"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = \"us-central1\" # @param {type: \"string\", placeholder: \"global\"}\n",
|
|
"\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MFozk2URDrTE"
|
|
},
|
|
"source": [
|
|
"### Choose a Gemini model\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"cellView": "form",
|
|
"id": "lBEljCMawnyN"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-live-2.5-flash-native-audio\" # @param {type: \"string\"}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "z5mKXikOEVjC"
|
|
},
|
|
"source": [
|
|
"### About audio streaming\n",
|
|
"\n",
|
|
"In these guides, you send an audio file to the model and receive audio in response. In production, the input audio would be a microphone stream. The Live API supports the following audio formats:\n",
|
|
"\n",
|
|
"- **Input audio**: Raw 16-bit PCM audio at 16kHz, little-endian\n",
|
|
"- **Output audio**: Raw 16-bit PCM audio at 24kHz, little-endian\n",
|
|
"\n",
|
|
"The client must maintain a playback buffer. The server streams audio in chunks (within server_content messages). The client's responsibility is to:\n",
|
|
"\n",
|
|
"- **Decode**: Base64 decode the `inline_data`.\n",
|
|
"- **Buffer**: Append the binary data to a queue.\n",
|
|
"- **Play**: Feed the data to the audio hardware.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bwUgw2r6u7Sh"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Download a sample audio input file\n",
|
|
"audio_file = \"input.wav\"\n",
|
|
"audio_file_url = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav\"\n",
|
|
"\n",
|
|
"!wget -q $audio_file_url -O $audio_file\n",
|
|
"\n",
|
|
"with wave.open(audio_file, \"rb\") as wf:\n",
|
|
" frames = wf.readframes(wf.getnframes())\n",
|
|
" print(f\"Read audio: {len(frames)} bytes\")\n",
|
|
" print(f\"Channels: {wf.getnchannels()}\")\n",
|
|
" print(f\"Rate: {wf.getframerate()}Hz\")\n",
|
|
" print(f\"Width: {wf.getsampwidth()} bytes\")\n",
|
|
"\n",
|
|
"display(Audio(filename=audio_file, autoplay=True))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5FLPsMtxFZcD"
|
|
},
|
|
"source": [
|
|
"### 🚀 Quickstart 1: Using Gen AI SDK\n",
|
|
"\n",
|
|
"In this quickstart, you learn how to connect to the Live API using the [Google Gen AI SDK](https://github.com/googleapis/python-genai), and establish a session, send an audio file to the model and receive audio in response. This is a simplified approach to manage the session and handle interruptions.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "hsE6yc94sn6i"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Configuration\n",
|
|
"config = {\n",
|
|
" \"response_modalities\": [\"audio\"],\n",
|
|
"}\n",
|
|
"\n",
|
|
"\n",
|
|
"async def main():\n",
|
|
" # Establish WebSocket session\n",
|
|
" async with client.aio.live.connect(model=MODEL_ID, config=config) as session:\n",
|
|
" print(\"Session established. Sending audio...\")\n",
|
|
"\n",
|
|
" # Send Input (Simulated from file)\n",
|
|
" # In production, this would be a microphone stream\n",
|
|
" with open(\"input.wav\", \"rb\") as f:\n",
|
|
" while chunk := f.read(1024):\n",
|
|
" await session.send_realtime_input(\n",
|
|
" audio=types.Blob(data=chunk, mime_type=\"audio/pcm;rate=16000\")\n",
|
|
" )\n",
|
|
" await asyncio.sleep(0.01) # Simulate real-time stream\n",
|
|
"\n",
|
|
" audio_data = []\n",
|
|
"\n",
|
|
" # Receive Output\n",
|
|
" async for message in session.receive():\n",
|
|
" if message.server_content:\n",
|
|
" # Check for interruptions (User barge-in)\n",
|
|
" if message.server_content.interrupted:\n",
|
|
" print(\"[Interrupted] Clear client audio buffer immediately.\")\n",
|
|
" continue\n",
|
|
"\n",
|
|
" # Process Audio Chunks\n",
|
|
" model_turn = message.server_content.model_turn\n",
|
|
" if model_turn and model_turn.parts:\n",
|
|
" for part in model_turn.parts:\n",
|
|
" if part.inline_data:\n",
|
|
" # Output is PCM, 24kHz, 16-bit, Mono\n",
|
|
" print(\n",
|
|
" f\"Received audio chunk: {len(part.inline_data.data)} bytes\"\n",
|
|
" )\n",
|
|
" audio_data.append(\n",
|
|
" np.frombuffer(part.inline_data.data, dtype=np.int16)\n",
|
|
" )\n",
|
|
"\n",
|
|
" if message.server_content.turn_complete:\n",
|
|
" print(\"Turn complete.\")\n",
|
|
" display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run directly in notebook\n",
|
|
"await main()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NjtDVNQNG6eD"
|
|
},
|
|
"source": [
|
|
"## 🚀 Quickstart 2: Using WebSocket\n",
|
|
"\n",
|
|
"In this quickstart, you learn how to connect to the Live API using [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API), and send an audio file to the model and receive audio in response. This is a low-level approach using standard WebSockets to construct the handshake and manage raw JSON payloads."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Id-qQxr2ytE3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Authentication\n",
|
|
"token_list = !gcloud auth application-default print-access-token\n",
|
|
"headers = {\"Authorization\": f\"Bearer {token_list[0]}\"}\n",
|
|
"\n",
|
|
"# Configuration\n",
|
|
"MODEL = (\n",
|
|
" f\"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}\"\n",
|
|
")\n",
|
|
"config = {\n",
|
|
" \"response_modalities\": [\"audio\"],\n",
|
|
"}\n",
|
|
"\n",
|
|
"# Construct the WSS URL\n",
|
|
"HOST = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
|
|
"URI = f\"wss://{HOST}/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent\"\n",
|
|
"\n",
|
|
"\n",
|
|
"async def main():\n",
|
|
" # Connect to the server\n",
|
|
" async with websockets.connect(URI, additional_headers=headers) as ws:\n",
|
|
" # Send Setup (Handshake)\n",
|
|
" await ws.send(\n",
|
|
" json.dumps({\"setup\": {\"model\": MODEL, \"generation_config\": config}})\n",
|
|
" )\n",
|
|
" print(\"Session established. Sending audio...\")\n",
|
|
"\n",
|
|
" # Receive setup response\n",
|
|
" raw_response = await ws.recv(decode=False)\n",
|
|
" setup_response = json.loads(raw_response.decode(\"ascii\"))\n",
|
|
"\n",
|
|
" # Send Input (Simulated from file)\n",
|
|
" # In production, this would be a microphone stream\n",
|
|
" with open(\"input.wav\", \"rb\") as f:\n",
|
|
" while chunk := f.read(1024):\n",
|
|
" msg = {\n",
|
|
" \"realtime_input\": {\n",
|
|
" \"media_chunks\": [\n",
|
|
" {\n",
|
|
" \"mime_type\": \"audio/pcm;rate=16000\",\n",
|
|
" \"data\": base64.b64encode(chunk).decode(\"utf-8\"),\n",
|
|
" }\n",
|
|
" ]\n",
|
|
" }\n",
|
|
" }\n",
|
|
" await ws.send(json.dumps(msg))\n",
|
|
" await asyncio.sleep(0.01) # Simulate real-time stream\n",
|
|
"\n",
|
|
" audio_data = []\n",
|
|
"\n",
|
|
" # Receive chunks of server response\n",
|
|
" async for raw_response in ws:\n",
|
|
" data = json.loads(raw_response.decode())\n",
|
|
" try:\n",
|
|
" parts = data[\"serverContent\"][\"modelTurn\"][\"parts\"]\n",
|
|
" for part in parts:\n",
|
|
" if \"inlineData\" in part:\n",
|
|
" b64_audio = part[\"inlineData\"][\"data\"]\n",
|
|
" print(f\"Received chunk: {len(b64_audio)} bytes\")\n",
|
|
" pcm_data = base64.b64decode(b64_audio)\n",
|
|
" audio_data.append(np.frombuffer(pcm_data, dtype=np.int16))\n",
|
|
" except KeyError:\n",
|
|
" pass\n",
|
|
"\n",
|
|
" if data.get(\"serverContent\", {}).get(\"turnComplete\"):\n",
|
|
" print(\"Turn complete.\")\n",
|
|
" display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))\n",
|
|
" break\n",
|
|
"\n",
|
|
"\n",
|
|
"# Run directly in notebook\n",
|
|
"await main()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "A3qNOSDcLBMq"
|
|
},
|
|
"source": [
|
|
"## What's next\n",
|
|
"\n",
|
|
"Now that you have established a basic connection to the Gemini Live API, try these advanced capabilities to build a production-ready application:\n",
|
|
"\n",
|
|
"- [Getting Started with the Live API in Vertex AI using WebSockets](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb)\n",
|
|
"- [Getting Started with the Live API using Gen AI SDK](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api_genai_sdk.ipynb)\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "live_api_quickstart.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|