wehub-resource-sync
191 lines
12 KiB
Markdown
191 lines
12 KiB
Markdown
# Travel Booking Multi-Agent Assistant (LiveKit ADK)
|
|
|
|
An enterprise-grade multi-agent voice orchestrator system built on **Google's Agent Development Kit (ADK)** and **LiveKit**. This application establishes a low-latency WebRTC-to-Gemini-Live bridge allowing users to naturally converse with specialized travel agents to search, book, and manage flights and hotels in a single session.
|
|
|
|
---
|
|
|
|
## 1. System Architecture & Flow
|
|
|
|
The system coordinates high-performance WebRTC audio streams, translates them into real-time bidirectional ADK session events, and communicates with Gemini Live API models.
|
|
|
|
```
|
|
┌───────────────────────────────────────┐
|
|
│ Client Browser │
|
|
└───────────────────────────────────────┘
|
|
▲ │
|
|
WebRTC │ (Render Audio / │ (User Audio /
|
|
DataChannel │ Data Transcription) │ WebRTC Media)
|
|
│ ▼
|
|
┌──────────────────────┼───────────────────────────────────────────────────────┐
|
|
│ │ LiveKit Server │
|
|
│ │ │
|
|
│ └──────────────┬───────────────▲────────────────────────┘
|
|
│ │ │
|
|
│ Audio Frames │ │ Audio Frames
|
|
│ (Gemini -> LK) │ │ (LK -> Gemini)
|
|
│ ▼ │
|
|
│ ┌────────────────────────────────────────┴────────────┐ │
|
|
│ │ FastAPI Application Server │ │
|
|
│ │ │ │
|
|
│ │ ┌─────────────────────────────────────────────┐ │ │
|
|
│ │ │ app/livekit_bridge.py │ │ │
|
|
│ │ │ - Converts 48kHz LK Audio -> 16kHz PCM │ │ │
|
|
│ │ │ - Feeds LiveRequestQueue │ │ │
|
|
│ │ └──────────────────────┬──────────────────────┘ │ │
|
|
│ │ │ │ │
|
|
│ │ ▼ │ │
|
|
│ │ ┌─────────────────────┐ │ │
|
|
│ │ │ LiveRequestQueue │ │ │
|
|
│ │ └──────────┬──────────┘ │ │
|
|
│ │ │ │ │
|
|
│ │ ▼ │ │
|
|
│ │ ┌─────────────────────┐ │ │
|
|
│ │ │ ADK Runner │ │ │
|
|
│ │ └──────────┬──────────┘ │ │
|
|
│ └──────────────────────────┼──────────────────────────┘ │
|
|
└───────────────────────────────────────┼──────────────────────────────────────┘
|
|
│
|
|
│ Bidirectional WebSockets (v1alpha)
|
|
▼
|
|
┌───────────────────────────────────────┐
|
|
│ Gemini Live API │
|
|
│ (gemini-live-2.5-flash) │
|
|
└───────────────────────────────────────┘
|
|
```
|
|
|
|
### Protocol Lifecycle:
|
|
1. **Session Establishment**: Client requests a token from `/token`, triggering the background instantiation of the `LiveKitGeminiBridge`.
|
|
2. **WebRTC Join**: The client browser joins the LiveKit room. The `LiveKitGeminiBridge` also connects a virtual audio participant into the room.
|
|
3. **Upstream Pipeline (User -> gemini)**:
|
|
- Client streams user audio to the LiveKit Room using **WebRTC**.
|
|
- `LiveKitGeminiBridge` subscribes to the track, extracts the 48kHz frames, downsamples to **16kHz mono PCM**, and pushes them to the ADK `LiveRequestQueue`.
|
|
- The ADK `Runner` streams the queue content to Gemini Live API via an underlying **secure WebSocket connection (`bidiGenerateContent` protocol)**.
|
|
4. **Downstream Pipeline (Gemini -> User)**:
|
|
- Gemini responds in real-time with audio buffers and textual transcriptions via the secure WebSocket.
|
|
- ADK `Runner` intercepts the events and passes them to the bridge.
|
|
- The bridge pushes the audio (resampled to 24kHz) back into LiveKit via the LocalAudioTrack, and publishes transcription text through WebRTC **DataChannels**.
|
|
|
|
---
|
|
|
|
## 2. The Booking Agents (Multi-Agent Orchestrator)
|
|
|
|
Inside the `app/travel_booking/` package, we implement a **three-tier hierarchical multi-agent orchestrator**:
|
|
|
|
```
|
|
┌──────────────────────┐
|
|
│ session_orchestrator │
|
|
│ (Central Router) │
|
|
└──────────┬───────────┘
|
|
│
|
|
┌──────────────────┴──────────────────┐
|
|
▼ ▼
|
|
┌─────────────────────┐ ┌─────────────────────┐
|
|
│ FlightBookingAgent │◀─────────────▶│ HotelBookingAgent │
|
|
│ (Flight search/book)│ A2A Transfer│ (Hotel search/book) │
|
|
└─────────────────────┘ └─────────────────────┘
|
|
```
|
|
|
|
1. **Session Orchestrator** (`app/travel_booking/agent.py`):
|
|
- The root agent. Listens to the user's intent and delegates control to the specialist sub-agents (`FlightBookingAgent` or `HotelBookingAgent`) via ADK's native agent-routing tool calls.
|
|
2. **FlightBookingAgent** (`app/travel_booking/agents/flight_booking.py`):
|
|
- Specialized in travel search and reservations.
|
|
- Equipped with 3 mock tools: `search_flights`, `book_flight`, `cancel_flight`.
|
|
- Configured with `HotelBookingAgent` as a sub-agent to enable a direct, silent handoff when the user switches context to hotel bookings.
|
|
3. **HotelBookingAgent** (`app/travel_booking/agents/hotel_booking.py`):
|
|
- Specialized in lodging discovery and booking.
|
|
- Equipped with 3 mock tools: `search_hotels`, `book_hotel`, `cancel_hotel`.
|
|
|
|
---
|
|
|
|
## 3. LiveKit Bridge & Configuration Overview
|
|
|
|
### `app/livekit_bridge.py` Overview
|
|
This class acts as the low-latency audio converter and routing engine.
|
|
- **Audio Buffering & Downsampling**: Standardizes LiveKit's high-fidelity audio stream (downsamples 48kHz stereo to 16kHz mono PCM) and sends it in small, optimized 20ms buffers (640 bytes) to reduce WebSocket latency.
|
|
- **Realtime Event Mapping**: Listens to `runner.run_live` event generators, translates model text outputs and prints them to the server console, and publishes data events onto WebRTC DataChannels.
|
|
|
|
### Runner Configuration (`app/main.py`)
|
|
- The `Runner` is configured with standard `InMemorySessionService` or `DatabaseSessionService` to persist dialogue histories.
|
|
- Integrated with a custom **`SessionResumptionIsolationPlugin`**: Clears the transparency resumption handles when transferring between sub-agents, preventing key-based Gemini API errors during sub-agent handoffs.
|
|
|
|
```python
|
|
runner = Runner(
|
|
app_name="livekit-adk",
|
|
agent=agent.root_agent,
|
|
session_service=session_service,
|
|
auto_create_session=True,
|
|
plugins=[SessionResumptionIsolationPlugin()]
|
|
)
|
|
```
|
|
|
|
### Run Configuration (`app/livekit_bridge.py`)
|
|
Configured specifically to handle high-performance native audio models over WebRTC:
|
|
```python
|
|
run_config = RunConfig(
|
|
streaming_mode=StreamingMode.BIDI,
|
|
response_modalities=["AUDIO"],
|
|
input_audio_transcription=types.AudioTranscriptionConfig(),
|
|
output_audio_transcription=types.AudioTranscriptionConfig(),
|
|
session_resumption=types.SessionResumptionConfig(),
|
|
enable_affective_dialog=True
|
|
)
|
|
```
|
|
- **`response_modalities=["AUDIO"]`**: Crucial for native audio Gemini models; enforces direct-to-audio responses to ensure natural voice cadence.
|
|
- **`output_audio_transcription`**: Tells Gemini to send text transcripts along with the audio stream so the bridge can display them on the UI.
|
|
|
|
---
|
|
|
|
## 4. Configuration & Run Instructions
|
|
|
|
### 1. Environment Setup (`app/.env`)
|
|
Create a file named `app/.env` inside the project. Configure the variables:
|
|
|
|
```bash
|
|
|
|
# Model selection
|
|
DEMO_AGENT_MODEL="gemini-live-2.5-flash-native-audio"
|
|
|
|
# LiveKit Settings (Local development values)
|
|
USE_LIVEKIT=true
|
|
LIVEKIT_URL=ws://localhost:7880
|
|
LIVEKIT_API_KEY=devkey
|
|
LIVEKIT_API_SECRET=secret
|
|
```
|
|
|
|
### 2. Installing & Running LiveKit (macOS)
|
|
|
|
For local development, you need a running LiveKit server instance. On macOS, install and run it using Homebrew:
|
|
|
|
```bash
|
|
# Install LiveKit Server
|
|
brew install livekit
|
|
|
|
# Start the server in development mode
|
|
livekit-server --dev
|
|
```
|
|
|
|
The `--dev` flag automatically starts the server on `localhost:7880` using the default credentials:
|
|
- **API Key**: `devkey`
|
|
- **API Secret**: `secret`
|
|
|
|
*(These credentials match the default `.env` template above.)*
|
|
|
|
### 3. Launching the Application
|
|
|
|
Ensure you are running commands using the project's virtual environment.
|
|
|
|
**Step 1: Sync dependencies**
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
**Step 2: Start the Dev Server**
|
|
Navigate into the `app` folder and run:
|
|
```bash
|
|
uv run --project .. python3 -m uvicorn main:app --reload
|
|
```
|
|
|
|
The application starts a FastAPI server listening at `http://127.0.0.1:8000/static/livekit`. Open this URL in your browser to interact with the voice assistant!
|
|
|
|
---
|