wehub-resource-sync
61 lines
2.5 KiB
Markdown
61 lines
2.5 KiB
Markdown
# Realtime Demo App
|
|
|
|
A web-based realtime voice assistant demo with a FastAPI backend and HTML/JS frontend.
|
|
|
|
## Installation
|
|
|
|
Install the required dependencies:
|
|
|
|
```bash
|
|
uv add fastapi uvicorn websockets
|
|
```
|
|
|
|
## Usage
|
|
|
|
Start the application with a single command:
|
|
|
|
```bash
|
|
cd examples/realtime/app && uv run python server.py
|
|
```
|
|
|
|
Then open your browser to: http://localhost:8000
|
|
|
|
### Debugging Realtime usage
|
|
|
|
Set `LOG_LEVEL=DEBUG` to log the raw `response.done` usage, the typed per-response usage with modality details, and the cumulative session usage:
|
|
|
|
```bash
|
|
cd examples/realtime/app && LOG_LEVEL=DEBUG uv run python server.py
|
|
```
|
|
|
|
The debug logs include concise summaries for server, model, session, history, tool, handoff, error, and usage events. Audio frames and high-volume delta events are omitted, and transcript content is not logged. Uvicorn and WebSocket protocol logging remain at INFO so `LOG_LEVEL=DEBUG` does not dump wire payloads.
|
|
|
|
## Customization
|
|
|
|
To use the same UI with your own agents, edit `agent.py` and ensure get_starting_agent() returns the right starting agent for your use case.
|
|
|
|
## How to Use
|
|
|
|
1. Click **Connect** to establish a realtime session
|
|
2. Audio capture starts automatically - just speak naturally
|
|
3. Click the **Mic On/Off** button to mute/unmute your microphone
|
|
4. To send an image, enter an optional prompt and click **🖼️ Send Image** (select a file)
|
|
5. Watch the conversation unfold in the left pane (image thumbnails are shown)
|
|
6. Monitor raw events in the right pane (click to expand/collapse)
|
|
7. Click **Disconnect** when done
|
|
|
|
### Human-in-the-loop approvals
|
|
|
|
- The seat update tool now requires approval. When the agent wants to run it, the browser shows a `window.confirm` dialog so you can allow or deny the tool call before it executes.
|
|
|
|
## Architecture
|
|
|
|
- **Backend**: FastAPI server with WebSocket connections for real-time communication
|
|
- **Session Management**: Each connection gets a unique session with the OpenAI Realtime API
|
|
- **Image Inputs**: The UI uploads images and the server forwards a `conversation.item.create` event with `input_image` (plus optional `input_text`), followed by `response.create` to start the model response. The messages pane renders image bubbles for `input_image` content.
|
|
- **Audio Processing**: 24kHz mono audio capture and playback
|
|
- **Event Handling**: Full event stream processing with transcript generation
|
|
- **Frontend**: Vanilla JavaScript with clean, responsive CSS
|
|
|
|
The demo showcases the core patterns for building realtime voice applications with the OpenAI Agents SDK.
|