wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
googlecloudplatform--genera…/gemini/sample-apps/gemini-live-telephony-app/design_doc.md
T
wehub-resource-sync 914fea506e chore: import upstream snapshot with attribution
2026-07-13 13:30:30 +08:00

14 KiB
Raw Permalink Blame History

Architecting a Low-Latency, High-Fidelity Conversational AI: A Definitive Guide to Integrating Twilio, FastAPI, and Gemini Live on Cloud Run

Executive Summary

This report provides a comprehensive architectural blueprint for a real-time, bidirectional voice-to-AI application. It addresses the three core technical challenges presented by the integration of Twilio, a FastAPI backend, and the Google Gemini Live API. These challenges are: (1) the system-level integration of Twilio's telephony with a custom backend via bidirectional WebSockets, (2) the design of a high-fidelity, low-latency audio transcoding pipeline to bridge the disparate audio formats (8kHz µ-law, 16kHz PCM, and 24kHz PCM), and (3) the deployment of this stateful, low-latency service on the stateless, serverless Google Cloud Run platform.

The analysis concludes that the most critical, non-obvious design decision is the selection of a streaming digital signal processing (DSP) library for audio resampling. Naive, chunk-by-chunk processing with standard libraries will introduce audible artifacts. The report identifies python-samplerate (a wrapper for libsamplerate) as the optimal choice, as its stateful "Full API" is specifically designed for high-quality, real-time chunked audio processing.

Furthermore, this document presents a production-ready Cloud Run architecture that addresses the platform's inherent challenges. It demonstrates that "excellent low latency" is achieved by mitigating cold starts using the min-instances=1 configuration. It also solves the critical state-management problem for a horizontally-scaling, stateless service by proposing a hybrid state model. This model utilizes Cloud Run's session affinity for in-memory DSP state.

special note: Having a Google Memorystore (Redis) for externalized conversational state, ensures a seamless, scalable, and low-latency conversational experience.

I. System Architecture: A High-Throughput Pipeline for Conversational AI

This section details the high-level architecture of the complete system, outlining the components, their responsibilities, and the flow of data and control from the user's mobile phone to the Google Gemini Live API and back.

A. The End-to-End Data and Control Flow

The entire system is orchestrated as a series of handoffs, transitioning from a standard HTTP webhook model to a persistent, bidirectional WebSocket stream. The lifecycle of a single interaction is as follows:

image
  1. Initiation (HTTP): A user dials the Twilio phone number provisioned for the service.
  2. Webhook Trigger: Twilio's infrastructure receives the inbound call. Per its configuration, Twilio sends a synchronous HTTP POST request (a webhook) to the application's pre-defined HTTP endpoint (/twiml).
  3. TwiML Response (HTTP): The FastAPI server, deployed on Cloud Run, receives this HTTP request. It dynamically generates and returns a TwiML (Twilio Markup Language) document.
  4. WebSocket Connection (WSS): The TwiML response contains the <Connect><Stream> verb. Upon receiving this, Twilio's media servers initiate a persistent, secure WebSocket (WSS) connection to the application's WebSocket endpoint (/ws/twilio).
  5. Bidirectional Streaming (WSS): The WebSocket connection is established. The FastAPI server, using asyncio, manages two concurrent audio streams:
    • Inbound Stream (User-to-AI): Receiving 8kHz µ-law audio from Twilio, transcoding it in real-time to 16kHz PCM, and forwarding it to the Google Gemini Live API.
    • Outbound Stream (AI-to-User): Receiving 24kHz PCM audio from the Google Gemini Live API, transcoding it in real-time to 8kHz µ-law, and streaming it back to Twilio.
  6. Termination: The call concludes when the user hangs up or the connection is otherwise closed. The application cleans up resources, including the temporary transcription file.

B. Component 1: Twilio Programmable Voice & TwiML

This component is the gateway between the public telephone network and the application.

Phone Number Configuration

The Twilio phone number is configured with a webhook that points to the /twiml endpoint of the deployed Cloud Run service.

The <Connect><Stream> TwiML

The application uses the <Connect><Stream> TwiML verb to establish a bidirectional WebSocket connection. The /twiml endpoint in main.py generates this TwiML dynamically, inserting the WebSocket URL of the service.

C. Component 2: FastAPI Backend on Cloud Run

The core of the application is a FastAPI server deployed on Google Cloud Run.

Endpoints

  • /twiml (POST): Receives the initial webhook from Twilio and responds with the TwiML to establish the WebSocket stream.
  • /ws/twilio (WebSocket): The main endpoint for the bidirectional audio stream. It orchestrates the flow of audio between Twilio and the Google Gemini Live API.

Asynchronous Audio Handling

The application leverages asyncio to handle the concurrent inbound and outbound audio streams. asyncio.Queue is used to pass audio chunks between the different processing tasks.

Audio Transcoding Pipeline

A critical part of the application is the real-time audio transcoding pipeline:

  1. Twilio to Gemini:
    • The incoming base64-encoded µ-law audio from Twilio is decoded.
    • audioop.ulaw2lin converts the µ-law audio to 16-bit PCM.
    • The PCM audio is converted to a NumPy array of floats.
    • samplerate.Resampler upsamples the audio from 8kHz to 16kHz.
    • The resampled audio is converted back to 16-bit PCM bytes and sent to the Gemini Live API.
  2. Gemini to Twilio:
    • The 24kHz PCM audio from the Gemini Live API is received as bytes.
    • The bytes are converted to a NumPy array of floats.
    • samplerate.Resampler downsamples the audio from 24kHz to 8kHz.
    • The resampled audio is converted to 16-bit PCM.
    • audioop.lin2ulaw converts the PCM audio to µ-law.
    • The µ-law audio is base64-encoded and sent to Twilio.

D. Component 3: Google Gemini Live API

The application uses the google-genai library to interact with the Google Gemini Live API via the Vertex AI platform.

Session Management

The run_gemini_session function in utils/live_api.py manages the persistent connection with the Gemini Live API. It:

  1. Utilizes session_handle for seamless session resumption, ensuring conversational context is maintained across potential connection drops.
  2. Constructs a LiveConnectConfig using utils/live_api_config.py, which includes system instructions, audio configuration, and real-time input settings (VAD).
  3. Establishes a single, persistent session for the duration of the call, with automatic reconnection logic if the stream is interrupted.
  4. Orchestrates two concurrent tasks (sender_loop and heartbeat_loop) and a primary message receiver to manage bidirectional audio and control signals.

E. Component 4: The utils/ Package

The logic is modularized into a utils/ package containing specialized modules:

  • live_api.py: Contains run_gemini_session, the core orchestrator for Gemini Live interactions, supporting sender/receiver loops and session resumption.
  • live_api_config.py: Defines the LiveConnectConfig, centralizing API settings, system instructions, and Voice Activity Detection (VAD) parameters.
  • audio_transcoding.py: Implements the real-time audio pipeline, handling conversions between Twilio's 8kHz µ-law and Gemini's 16kHz/24kHz PCM formats.
  • prompt.py: Stores the BASE_SYSTEM_INSTRUCTION and other persona-related configurations.

F. State Management

For this sample application, state management is handled using an in-memory and handle-based approach:

  • In-memory state: The call_state dictionary maintains the real-time status of the call, including activity flags and stream SIDs.
  • Session Resumption Handle: Instead of file-based history, the system leverages Gemini's session_handle property. This token is captured from session updates and used during reconnection to restore the full conversational context within the API itself.

For a production system, a more robust solution like Google Memorystore (Redis) would be recommended to externalize the conversational state, allowing for horizontal scaling and better resilience.

Implementation Plan:

A step-by-step process to create the Gemini Live Telephony application, a real-time, bidirectional voice-to-AI system integrating Twilio, FastAPI, and Google Gemini Live on Cloud Run.

1. Project Setup and Dependencies

  • Initialize Project: Set up a Python project with a virtual environment.
  • Install Dependencies: Install necessary Python libraries as listed in requirements.txt:
    • fastapi: For the web server.
    • uvicorn: As the ASGI server.
    • python-dotenv: To manage environment variables.
    • google-generativeai: The Google Gemini SDK.
    • numpy: For numerical operations on audio data.
    • samplerate: For high-quality audio resampling.
    • audioop: For µ-law and linear PCM audio conversions.
  • Environment Configuration: Create a .env file to store information like GOOGLE_CLOUD_PROJECT, SERVICE_URL and other gemini configurations.

2. FastAPI Server Implementation (main.py)

  • Create main.py: This will be the entry point for the FastAPI application.
  • Implement /twiml Endpoint:
    • Create an HTTP POST endpoint that responds to Twilio's webhook.
    • This endpoint will generate and return a TwiML response with the <Connect><Stream> verb, pointing to the WebSocket endpoint.
  • Implement /ws/twilio WebSocket Endpoint:
    • This will be the core of the application, handling the bidirectional audio stream.
    • It will manage the WebSocket lifecycle: connection, message handling, and disconnection.
    • It will use asyncio.create_task to run three concurrent tasks:
      1. handle_twilio_to_gemini: Processes audio from Twilio to Gemini.
      2. handle_gemini_to_twilio: Processes audio from Gemini to Twilio.
      3. run_gemini_session: Manages the high-level Gemini session logic from utils/live_api.py.

3. Audio Transcoding Pipeline (utils/audio_transcoding.py)

  • Inbound (Twilio to Gemini):
    • Base64 Decoding: Decode the Base64 payload from Twilio media messages.
    • µ-law to PCM: Convert the 8kHz µ-law audio to 16-bit linear PCM using audioop.ulaw2lin.
    • Upsampling: Upsample the 8kHz PCM to 16kHz PCM using the samplerate library.
  • Outbound (Gemini to Twilio):
    • Downsampling: Downsample the 24kHz PCM audio from Gemini to 8kHz PCM using samplerate.
    • PCM to µ-law: Convert the 8kHz linear PCM to µ-law using audioop.lin2ulaw.
    • Base64 Encoding: Encode the µ-law audio to Base64 to be sent back to Twilio.

4. Gemini Live Integration (utils/live_api.py)

  • Initialize Gemini Client: Set up the Gemini client with Vertex AI and project/location settings.
  • run_gemini_session Function:
    • LiveConnectConfig: Configure the session via utils/live_api_config.py.
    • Session Management: Establish a persistent connection that stays active throughout the call.
    • Sender/Heartbeat/Receiver Loops: Use concurrent tasks to stream audio and monitor for session updates.
    • Resumption Logic: Utilize session_handle to restore state during reconnection.

5. Utility Package (utils/)

  • Modular Organization: The code is separated into multiple files within the utils/ directory.
  • live_api_config.py: Manages all connection parameters, including VAD sensitivity and system instructions.
  • prompt.py: Centralizes the BASE_SYSTEM_INSTRUCTION for the AI persona.
  • audio_transcoding.py: Contains the resampling and transcoding logic previously in main.py.

6. State Management

  • In-memory State: Use a Python dictionary (call_state) to manage the active state of the call within a single Cloud Run instance.
  • Session Resumption Logic:
    • Instead of file-based history, captures session_resumption_update messages.
    • Saves the new_handle to be reused in subsequent connect calls.
  • Note on Production Systems: For a production-grade application, externalize state to a robust solution like Google Memorystore (Redis).

7. Containerization and Deployment

  • Create Dockerfile:
    • Use a slim Python base image.
    • Install system dependencies like libsamplerate0.
    • Install Python dependencies from requirements.txt.
    • Use uvicorn to run the application.
  • Create deploy.sh:
    • Write a shell script to automate the process of building the Docker image and deploying it to Google Cloud Run.
  • Deploy to Cloud Run:
    • Configure the Cloud Run service with the following settings, as seen in deploy.sh:
      • --min-instances=1: This is crucial for a low-latency application to avoid "cold starts," ensuring that an instance is always running and ready to accept calls.
      • --timeout=3600: Sets a long request timeout (1 hour) to accommodate long-running WebSocket connections for phone calls.
      • --memory=2Gi and --cpu=2: Allocates sufficient resources for the CPU-intensive audio resampling tasks.
      • --session-affinity: Ensures that requests from the same client (Twilio, in this case) are routed to the same Cloud Run instance, which is important for maintaining the WebSocket connection.
      • --concurrency=1: This is a critical setting for this application. Since a single instance handles a single, stateful phone call at a time, setting concurrency to 1 ensures that each instance is dedicated to a single call. This prevents issues with managing multiple concurrent calls on a single instance.
      • --no-cpu-throttling: The audio resampling process is CPU-intensive and sensitive to latency. Disabling CPU throttling ensures that the instance has access to the full allocated CPU, which is essential for real-time audio processing and maintaining a smooth, low-latency conversation.
Reference in New Issue View Git Blame Copy Permalink