wehub-resource-sync
738 lines
40 KiB
Plaintext
738 lines
40 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Context Summarization with Realtime API\n",
|
||
"## 1. Overview\n",
|
||
"Build an end‑to‑end **voice bot** that listens to your mic, speaks back in real time and **summarises long conversations** so quality never drops.\n",
|
||
"\n",
|
||
"### What You’ll Learn\n",
|
||
"1. **Live microphone streaming** → OpenAI *Realtime* (voice‑to‑voice) endpoint.\n",
|
||
"2. **Instant transcripts & speech playback** on every turn.\n",
|
||
"3. **Conversation state container** that stores **every** user/assistant message.\n",
|
||
"4. **Automatic “context trim”** – when the token window becomes very large (configurable), older turns are compressed into a summary.\n",
|
||
"5. **Extensible design** you can adapt to support customer‑support bots, kiosks, or multilingual assistants.\n",
|
||
"\n",
|
||
"\n",
|
||
"### Prerequisites\n",
|
||
"\n",
|
||
"| Requirement | Details |\n",
|
||
"|-------------|---------|\n",
|
||
"| **Python ≥ 3.10** | Will ensure that you don't hit any issues |\n",
|
||
"| **OpenAI API key** | Set `OPENAI_API_KEY` in your shell or paste inline (*not ideal for prod*) |\n",
|
||
"| Mic + speakers | Grant OS permission if prompted |\n",
|
||
"\n",
|
||
"\n",
|
||
"**Need help setting up the key?** \n",
|
||
"> Follow the [official quick‑start guide](https://platform.openai.com/docs/quickstart#step-2-set-your-api-key).\n",
|
||
"\n",
|
||
"\n",
|
||
"*Notes:*\n",
|
||
"> 1. gpt-realtime supports a 32k token context window, though in certain use cases, you may notice performance degrade as you stuff more tokens into the context window.\n",
|
||
"> 2. Token window = all tokens (words and audio tokens) the model currently keeps in memory for the session.x\n",
|
||
"\n",
|
||
"### One‑liner install (run in a fresh cell)\n",
|
||
"\n",
|
||
"*New API Parameters:*\n",
|
||
"> 1. The Realtime API GA has releases a new parameter [`truncation`](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update#realtime-client-events/session/update-session-realtime-session-configuration-truncation). This parameter automatically optimizes context truncation, preserving relevant information while maximizing cache hit rates."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Run once to install or upgrade dependencies (comment out if already installed)\n",
|
||
"# !pip install --upgrade openai websockets sounddevice simpleaudio"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Standard library imports\n",
|
||
"import os\n",
|
||
"import sys\n",
|
||
"import io\n",
|
||
"import json\n",
|
||
"import base64\n",
|
||
"import pathlib\n",
|
||
"import wave\n",
|
||
"from dataclasses import dataclass, field\n",
|
||
"from typing import List, Literal\n",
|
||
"\n",
|
||
"# Third-party imports\n",
|
||
"import asyncio\n",
|
||
"import numpy as np\n",
|
||
"import sounddevice as sd # microphone capture\n",
|
||
"import simpleaudio # speaker playback\n",
|
||
"import websockets # WebSocket client\n",
|
||
"import openai # OpenAI Python SDK >= 1.14.0"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Set your API key safely\n",
|
||
"openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"\")\n",
|
||
"if not openai.api_key:\n",
|
||
" raise ValueError(\"OPENAI_API_KEY not found – please set env var or edit this cell.\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Token Utilisation – Text vs Voice\n",
|
||
"\n",
|
||
"Large‑token windows are precious, every extra token you use costs latency + money. \n",
|
||
"For **audio** the input token window increases much faster than for plain text because amplitude, timing, and other acoustic details must be represented.\n",
|
||
"\n",
|
||
"In practice you’ll often see **≈ 10 ×** more tokens for the *same* sentence in audio versus text.\n",
|
||
"\n",
|
||
"\n",
|
||
"* gpt-realtime accepts up to **32k tokens** and as the token size increases, instruction adherence can drift.\n",
|
||
"* Every user/assistant turn consumes tokens → the window **only grows**.\n",
|
||
"* **Strategy**: Summarise older turns into a single assistant message, keep the last few verbatim turns, and continue.\n",
|
||
"\n",
|
||
"<img src=\"../images/text-vs-audio-tokens.png\" alt=\"drawing\" width=\"800\"/>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Helper Functions\n",
|
||
"The following helper functions will enable us to run the full script."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.1 Conversation State\n",
|
||
"Unlike HTTP-based Chat Completions, the Realtime API maintains an open, **stateful** session with two key components:\n",
|
||
"\n",
|
||
"| Component | Purpose |\n",
|
||
"|----------------|---------|\n",
|
||
"| **Session** | Controls global settings — model, voice, modalities, VAD, etc. |\n",
|
||
"| **Conversation** | Stores turn-by-turn messages between user and assistant — both audio and text. |\n",
|
||
"\n",
|
||
"This notebook wraps these components inside a simple `ConversationState` object to keep your logic clean, track history, and manage summarization when context windows fill up."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"@dataclass\n",
|
||
"class Turn:\n",
|
||
" \"\"\"One utterance in the dialogue (user **or** assistant).\"\"\"\n",
|
||
" role: Literal[\"user\", \"assistant\"]\n",
|
||
" item_id: str # Server‑assigned identifier\n",
|
||
" text: str | None = None # Filled once transcript is ready\n",
|
||
"\n",
|
||
"@dataclass\n",
|
||
"class ConversationState:\n",
|
||
" \"\"\"All mutable data the session needs — nothing more, nothing less.\"\"\"\n",
|
||
" history: List[Turn] = field(default_factory=list) # Ordered log\n",
|
||
" waiting: dict[str, asyncio.Future] = field(default_factory=dict) # Pending transcript fetches\n",
|
||
" summary_count: int = 0\n",
|
||
"\n",
|
||
" latest_tokens: int = 0 # Window size after last reply\n",
|
||
" summarising: bool = False # Guard so we don’t run two summaries at once"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"A quick helper to peek at the transcript:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def print_history(state) -> None:\n",
|
||
" \"\"\"Pretty-print the running transcript so far.\"\"\"\n",
|
||
" print(\"—— Conversation so far ———————————————\")\n",
|
||
" for turn in state.history:\n",
|
||
" text_preview = (turn.text or \"\").strip().replace(\"\\n\", \" \")\n",
|
||
" print(f\"[{turn.role:<9}] {text_preview} ({turn.item_id})\")\n",
|
||
" print(\"——————————————————————————————————————————\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.2 · Streaming Audio\n",
|
||
"We’ll stream raw PCM‑16 microphone data straight into the Realtime API.\n",
|
||
"\n",
|
||
"The pipeline is: mic ─► async.Queue ─► WebSocket ─► Realtime API"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 3.2.1 Capture Microphone Input\n",
|
||
"We’ll start with a coroutine that:\n",
|
||
"\n",
|
||
"* Opens the default mic at **24 kHz, mono, PCM‑16** (one of the [format](https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-input_audio_format) Realtime accepts). \n",
|
||
"* Slices the stream into **≈ 40 ms** blocks. \n",
|
||
"* Dumps each block into an `asyncio.Queue` so another task (next section) can forward it to OpenAI."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"async def mic_to_queue(pcm_queue: asyncio.Queue[bytes]) -> None:\n",
|
||
" \"\"\"\n",
|
||
" Capture raw PCM‑16 microphone audio and push ~CHUNK_DURATION_MS chunks\n",
|
||
" to *pcm_queue* until the surrounding task is cancelled.\n",
|
||
"\n",
|
||
" Parameters\n",
|
||
" ----------\n",
|
||
" pcm_queue : asyncio.Queue[bytes]\n",
|
||
" Destination queue for PCM‑16 frames (little‑endian int16).\n",
|
||
" \"\"\"\n",
|
||
" blocksize = int(SAMPLE_RATE_HZ * CHUNK_DURATION_MS / 1000)\n",
|
||
"\n",
|
||
" def _callback(indata, _frames, _time, status):\n",
|
||
" if status: # XRuns, device changes, etc.\n",
|
||
" print(\"⚠️\", status, file=sys.stderr)\n",
|
||
" try:\n",
|
||
" pcm_queue.put_nowait(bytes(indata)) # 1‑shot enqueue\n",
|
||
" except asyncio.QueueFull:\n",
|
||
" # Drop frame if upstream (WebSocket) can’t keep up.\n",
|
||
" pass\n",
|
||
"\n",
|
||
" # RawInputStream is synchronous; wrap in context manager to auto‑close.\n",
|
||
" with sd.RawInputStream(\n",
|
||
" samplerate=SAMPLE_RATE_HZ,\n",
|
||
" blocksize=blocksize,\n",
|
||
" dtype=\"int16\",\n",
|
||
" channels=1,\n",
|
||
" callback=_callback,\n",
|
||
" ):\n",
|
||
" try:\n",
|
||
" # Keep coroutine alive until cancelled by caller.\n",
|
||
" await asyncio.Event().wait()\n",
|
||
" finally:\n",
|
||
" print(\"⏹️ Mic stream closed.\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 3.2.2 Send Audio Chunks to the API\n",
|
||
"\n",
|
||
"Our mic task is now filling an `asyncio.Queue` with raw PCM‑16 blocks. \n",
|
||
"Next step: pull chunks off that queue, **base‑64 encode** them (the protocol requires JSON‑safe text), and ship each block to the Realtime WebSocket as an `input_audio_buffer.append` event."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Helper function to encode audio chunks in base64\n",
|
||
"b64 = lambda blob: base64.b64encode(blob).decode()\n",
|
||
"\n",
|
||
"async def queue_to_websocket(pcm_queue: asyncio.Queue[bytes], ws):\n",
|
||
" \"\"\"Read audio chunks from queue and send as JSON events.\"\"\"\n",
|
||
" try:\n",
|
||
" while (chunk := await pcm_queue.get()) is not None:\n",
|
||
" await ws.send(json.dumps({\n",
|
||
" \"type\": \"input_audio_buffer.append\",\n",
|
||
" \"audio\": b64(chunk),\n",
|
||
" }))\n",
|
||
" except websockets.ConnectionClosed:\n",
|
||
" print(\"WebSocket closed – stopping uploader\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 3.2.3 Handle Incoming Events \n",
|
||
"Once audio reaches the server, the Realtime API pushes a stream of JSON events back over the **same** WebSocket. \n",
|
||
"Understanding these events is critical for:\n",
|
||
"\n",
|
||
"* Printing live transcripts \n",
|
||
"* Playing incremental audio back to the user \n",
|
||
"* Keeping an accurate [`Conversation State`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/created) so context trimming works later \n",
|
||
"\n",
|
||
"\n",
|
||
"| Event type | When it arrives | Why it matters | Typical handler logic |\n",
|
||
"|------------|-----------------|---------------|-----------------------|\n",
|
||
"| **`session.created`** | Immediately after the WebSocket handshake | Confirms the session is open and provides the `session.id`. | Log the ID for traceability and verify the connection. |\n",
|
||
"| **`session.updated`** | After you send a `session.update` call | Acknowledges that the server applied new session settings. | Inspect the echoed settings and update any local cache. |\n",
|
||
"| **`conversation.item.created`** (user) | A few ms after the user stops speaking (client VAD fires) | Reserves a timeline slot; transcript may still be **`null`**. | Insert a *placeholder* user turn in `state.history` marked “pending transcript”. |\n",
|
||
"| **`conversation.item.retrieved`** | ~100 – 300 ms later, once audio transcription is complete | Supplies the final user transcript (with timing). | Replace the placeholder with the transcript and print it if desired. |\n",
|
||
"| **`response.audio.delta`** | Every 20 – 60 ms while the assistant is speaking | Streams PCM‑16 audio chunks (and optional incremental text). | Buffer each chunk and play it; optionally show partial text in the console. |\n",
|
||
"| **`response.done`** | After the assistant’s last token | Signals both audio & text are complete; includes usage stats. | Finalize the assistant turn, update `state.latest_tokens`, and log usage. |\n",
|
||
"| **`conversation.item.deleted`** | Whenever you prune with `conversation.item.delete` | Confirms a turn was removed, freeing tokens on the server. | Mirror the deletion locally so your context window matches the server’s. |\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3.3 Detect When to Summarise\n",
|
||
"The Realtime model keeps a **large 32 k‑token window**, but quality can drift long before that limit as you stuff more context into the model.\n",
|
||
"\n",
|
||
"Our goal: **auto‑summarise** once the running window nears a safe threshold (default **2 000 tokens** for the notebook), then prune the superseded turns both locally *and* server‑side.\n",
|
||
"\n",
|
||
"We monitor latest_tokens returned in `response.done`. When it exceeds SUMMARY_TRIGGER and we have more than KEEP_LAST_TURNS, we spin up a background summarization coroutine.\n",
|
||
"\n",
|
||
"We compress everything except the last 2 turns into a single French paragraph, then:\n",
|
||
"\n",
|
||
"1. Insert that paragraph as a new assistant message at the top of the conversation.\n",
|
||
"\n",
|
||
"2. Delete the message items that was used for the summary.\n",
|
||
"\n",
|
||
"We will later ask the Voice agent what language was the summary to test if the Summary insertion into Realtime API Conversation Context was successful."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"async def run_summary_llm(text: str) -> str:\n",
|
||
" \"\"\"Call a lightweight model to summarise `text`.\"\"\"\n",
|
||
" resp = await asyncio.to_thread(lambda: openai.chat.completions.create(\n",
|
||
" model=SUMMARY_MODEL,\n",
|
||
" temperature=0,\n",
|
||
" messages=[\n",
|
||
" {\"role\": \"system\", \"content\": \"Summarise in French the following conversation \"\n",
|
||
" \"in one concise paragraph so it can be used as \"\n",
|
||
" \"context for future dialogue.\"},\n",
|
||
" {\"role\": \"user\", \"content\": text},\n",
|
||
" ],\n",
|
||
" ))\n",
|
||
" return resp.choices[0].message.content.strip()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Important implementation detail:\n",
|
||
"- The summary is appended as a SYSTEM message rather than an ASSISTANT message. Testing revealed that, during extended conversations, using ASSISTANT messages for summaries can cause the model to mistakenly switch from audio responses to text responses. By using SYSTEM messages for summaries (which can also include additional custom instructions), we clearly signal to the model that these are context-setting instructions, preventing it from incorrectly adopting the modality of the ongoing user-assistant interaction."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"async def summarise_and_prune(ws, state):\n",
|
||
" \"\"\"Summarise old turns, delete them server‑side, and prepend a single summary\n",
|
||
" turn locally + remotely.\"\"\"\n",
|
||
" state.summarising = True\n",
|
||
" print(\n",
|
||
" f\"⚠️ Token window ≈{state.latest_tokens} ≥ {SUMMARY_TRIGGER}. Summarising…\",\n",
|
||
" )\n",
|
||
" old_turns, recent_turns = state.history[:-KEEP_LAST_TURNS], state.history[-KEEP_LAST_TURNS:]\n",
|
||
" convo_text = \"\\n\".join(f\"{t.role}: {t.text}\" for t in old_turns if t.text)\n",
|
||
" \n",
|
||
" if not convo_text:\n",
|
||
" print(\"Nothing to summarise (transcripts still pending).\")\n",
|
||
" state.summarising = False\n",
|
||
"\n",
|
||
" summary_text = await run_summary_llm(convo_text) if convo_text else \"\"\n",
|
||
" state.summary_count += 1\n",
|
||
" summary_id = f\"sum_{state.summary_count:03d}\"\n",
|
||
" state.history[:] = [Turn(\"assistant\", summary_id, summary_text)] + recent_turns\n",
|
||
" \n",
|
||
" print_history(state) \n",
|
||
"\n",
|
||
" # Create summary on server\n",
|
||
" await ws.send(json.dumps({\n",
|
||
" \"type\": \"conversation.item.create\",\n",
|
||
" \"previous_item_id\": \"root\",\n",
|
||
" \"item\": {\n",
|
||
" \"id\": summary_id,\n",
|
||
" \"type\": \"message\",\n",
|
||
" \"role\": \"system\",\n",
|
||
" \"content\": [{\"type\": \"input_text\", \"text\": summary_text}],\n",
|
||
" },\n",
|
||
" }))\n",
|
||
"\n",
|
||
" # Delete old items\n",
|
||
" for turn in old_turns:\n",
|
||
" await ws.send(json.dumps({\n",
|
||
" \"type\": \"conversation.item.delete\",\n",
|
||
" \"item_id\": turn.item_id,\n",
|
||
" }))\n",
|
||
"\n",
|
||
" print(f\"✅ Summary inserted ({summary_id})\")\n",
|
||
" \n",
|
||
" state.summarising = False"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The following function lets us poll for transcripts over time. This is useful in cases where the user's audio hasn't been transcribed immediately, so we can retrieve the final result later."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"async def fetch_full_item(\n",
|
||
" ws, item_id: str, state: ConversationState, attempts: int = 1\n",
|
||
"):\n",
|
||
" \"\"\"\n",
|
||
" Ask the server for a full conversation item; retry up to 5× if the\n",
|
||
" transcript field is still null. Resolve the waiting future when done.\n",
|
||
" \"\"\"\n",
|
||
" # If there is already a pending fetch, just await it\n",
|
||
" if item_id in state.waiting:\n",
|
||
" return await state.waiting[item_id]\n",
|
||
"\n",
|
||
" fut = asyncio.get_running_loop().create_future()\n",
|
||
" state.waiting[item_id] = fut\n",
|
||
"\n",
|
||
" await ws.send(json.dumps({\n",
|
||
" \"type\": \"conversation.item.retrieve\",\n",
|
||
" \"item_id\": item_id,\n",
|
||
" }))\n",
|
||
" item = await fut\n",
|
||
"\n",
|
||
" # If transcript still missing retry (max 5×)\n",
|
||
" if attempts < 5 and not item.get(\"content\", [{}])[0].get(\"transcript\"):\n",
|
||
" await asyncio.sleep(0.4 * attempts)\n",
|
||
" return await fetch_full_item(ws, item_id, state, attempts + 1)\n",
|
||
"\n",
|
||
" # Done – remove the marker\n",
|
||
" state.waiting.pop(item_id, None)\n",
|
||
" return item\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. End‑to‑End Workflow Demonstration\n",
|
||
"\n",
|
||
"Run the two cells below to launch an interactive session. Interrupt the cell stop recording.\n",
|
||
"\n",
|
||
"> **Note:** \n",
|
||
"> This notebook uses `SUMMARY_TRIGGER = 2000` and `KEEP_LAST_TURNS = 2` to make summarization easier to demo quickly. \n",
|
||
"> In production, you should tune these values based on your application's needs. \n",
|
||
"> - A typical `SUMMARY_TRIGGER` falls between **20,000–32,000 tokens**, depending on how performance degrades with larger context for your use case."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Audio/config knobs\n",
|
||
"SAMPLE_RATE_HZ = 24_000 # Required by pcm16\n",
|
||
"CHUNK_DURATION_MS = 40 # chunk size for audio capture\n",
|
||
"BYTES_PER_SAMPLE = 2 # pcm16 = 2 bytes/sample\n",
|
||
"SUMMARY_TRIGGER = 2_000 # Summarise when context ≥ this\n",
|
||
"KEEP_LAST_TURNS = 2 # Keep these turns verbatim\n",
|
||
"SUMMARY_MODEL = \"gpt-4o-mini\" # Cheaper, fast summariser"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# --------------------------------------------------------------------------- #\n",
|
||
"# Realtime session #\n",
|
||
"# --------------------------------------------------------------------------- #\n",
|
||
"async def realtime_session(model=\"gpt-realtime\", voice=\"shimmer\", enable_playback=True):\n",
|
||
" \"\"\"\n",
|
||
" Main coroutine: connects to the Realtime endpoint, spawns helper tasks,\n",
|
||
" and processes incoming events in a big async‑for loop.\n",
|
||
" \"\"\"\n",
|
||
" state = ConversationState() # Reset state for each run\n",
|
||
"\n",
|
||
" pcm_queue: asyncio.Queue[bytes] = asyncio.Queue()\n",
|
||
" assistant_audio: List[bytes] = []\n",
|
||
"\n",
|
||
" # ----------------------------------------------------------------------- #\n",
|
||
" # Open the WebSocket connection to the Realtime API #\n",
|
||
" # ----------------------------------------------------------------------- #\n",
|
||
" url = f\"wss://api.openai.com/v1/realtime?model={model}\"\n",
|
||
" headers = {\"Authorization\": f\"Bearer {openai.api_key}\"}\n",
|
||
"\n",
|
||
" async with websockets.connect(url, extra_headers=headers, max_size=1 << 24) as ws:\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" # Wait until server sends session.created #\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" while json.loads(await ws.recv())[\"type\"] != \"session.created\":\n",
|
||
" pass\n",
|
||
" print(\"session.created ✅\")\n",
|
||
"\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" # Configure session: voice, modalities, audio formats, transcription #\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" await ws.send(json.dumps({\n",
|
||
" \"type\": \"session.update\",\n",
|
||
" \"session\": {\n",
|
||
" \"type\": \"realtime\",\n",
|
||
" model: \"gpt-realtime\",\n",
|
||
" \"voice\": voice,\n",
|
||
" \"modalities\": [\"audio\", \"text\"],\n",
|
||
" \"input_audio_format\": \"pcm16\",\n",
|
||
" \"output_audio_format\": \"pcm16\",\n",
|
||
" \"input_audio_transcription\": {\"model\": \"gpt-4o-transcribe\"},\n",
|
||
" },\n",
|
||
" }))\n",
|
||
"\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" # Launch background tasks: mic capture → queue → websocket #\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" mic_task = asyncio.create_task(mic_to_queue(pcm_queue))\n",
|
||
" upl_task = asyncio.create_task(queue_to_websocket(pcm_queue, ws))\n",
|
||
"\n",
|
||
" print(\"🎙️ Speak now (Ctrl‑C to quit)…\")\n",
|
||
"\n",
|
||
" try:\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" # Main event loop: process incoming events from the websocket #\n",
|
||
" # ------------------------------------------------------------------- #\n",
|
||
" async for event_raw in ws:\n",
|
||
" event = json.loads(event_raw)\n",
|
||
" etype = event[\"type\"]\n",
|
||
"\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" # User just spoke ⇢ conversation.item.created (role = user) #\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" if etype == \"conversation.item.created\" and event[\"item\"][\"role\"] == \"user\":\n",
|
||
" item = event[\"item\"]\n",
|
||
" text = None\n",
|
||
" if item[\"content\"]:\n",
|
||
" text = item[\"content\"][0].get(\"transcript\")\n",
|
||
" \n",
|
||
" state.history.append(Turn(\"user\", event[\"item\"][\"id\"], text))\n",
|
||
" \n",
|
||
" # If transcript not yet available, fetch it later\n",
|
||
" if text is None:\n",
|
||
" asyncio.create_task(fetch_full_item(ws, item[\"id\"], state))\n",
|
||
"\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" # Transcript fetched ⇢ conversation.item.retrieved #\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" elif etype == \"conversation.item.retrieved\":\n",
|
||
" content = event[\"item\"][\"content\"][0]\n",
|
||
" # Fill missing transcript in history\n",
|
||
" for t in state.history:\n",
|
||
" if t.item_id == event[\"item\"][\"id\"]:\n",
|
||
" t.text = content.get(\"transcript\")\n",
|
||
" break\n",
|
||
"\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" # Assistant audio arrives in deltas #\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" elif etype == \"response.audio.delta\":\n",
|
||
" assistant_audio.append(base64.b64decode(event[\"delta\"]))\n",
|
||
"\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" # Assistant reply finished ⇢ response.done #\n",
|
||
" # --------------------------------------------------------------- #\n",
|
||
" elif etype == \"response.done\":\n",
|
||
" for item in event[\"response\"][\"output\"]:\n",
|
||
" if item[\"role\"] == \"assistant\":\n",
|
||
" txt = item[\"content\"][0][\"transcript\"]\n",
|
||
" state.history.append(Turn(\"assistant\", item[\"id\"], txt))\n",
|
||
" # print(f\"\\n🤖 {txt}\\n\")\n",
|
||
" state.latest_tokens = event[\"response\"][\"usage\"][\"total_tokens\"]\n",
|
||
" print(f\"—— response.done (window ≈{state.latest_tokens} tokens) ——\")\n",
|
||
" print_history(state)\n",
|
||
" \n",
|
||
" # Fetch any still‑missing user transcripts\n",
|
||
" for turn in state.history:\n",
|
||
" if (turn.role == \"user\"\n",
|
||
" and turn.text is None\n",
|
||
" and turn.item_id not in state.waiting):\n",
|
||
" asyncio.create_task(\n",
|
||
" fetch_full_item(ws, turn.item_id, state)\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Playback collected audio once reply completes\n",
|
||
" if enable_playback and assistant_audio:\n",
|
||
" simpleaudio.play_buffer(b\"\".join(assistant_audio), 1, BYTES_PER_SAMPLE, SAMPLE_RATE_HZ)\n",
|
||
" assistant_audio.clear()\n",
|
||
"\n",
|
||
" # Summarise if context too large – fire in background so we don't block dialogue\n",
|
||
" if state.latest_tokens >= SUMMARY_TRIGGER and len(state.history) > KEEP_LAST_TURNS and not state.summarising:\n",
|
||
" asyncio.create_task(summarise_and_prune(ws, state))\n",
|
||
"\n",
|
||
" except KeyboardInterrupt:\n",
|
||
" print(\"\\nStopping…\")\n",
|
||
" finally:\n",
|
||
" mic_task.cancel()\n",
|
||
" await pcm_queue.put(None)\n",
|
||
" await upl_task"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Run the realtime session (this cell blocks until you stop it)\n",
|
||
"await realtime_session()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"```raw\n",
|
||
"session.created ✅\n",
|
||
"🎙️ Speak now (Ctrl‑C to quit)…\n",
|
||
"—— response.done (window ≈979 tokens) ——\n",
|
||
"—— Conversation so far ———————————————\n",
|
||
"[user ] Can you tell me a quick story? (item_BTuMOcpUqp8qknKhLzlkA)\n",
|
||
"[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end. (item_BTuMPRWxqpv0ph6QM46DK)\n",
|
||
"——————————————————————————————————————————\n",
|
||
"—— response.done (window ≈2755 tokens) ——\n",
|
||
"—— Conversation so far ———————————————\n",
|
||
"[user ] Can you tell me a quick story? (item_BTuMOcpUqp8qknKhLzlkA)\n",
|
||
"[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end. (item_BTuMPRWxqpv0ph6QM46DK)\n",
|
||
"[user ] Can you tell me three extremely funny stories? (item_BTuNN64LdULM21OyC4vzN)\n",
|
||
"[assistant] Sure, let's dive into some giggle-worthy tales: **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast! **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks! **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days. (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
|
||
"——————————————————————————————————————————\n",
|
||
"⚠️ Token window ≈2755 ≥ 2000. Summarising…\n",
|
||
"—— Conversation so far ———————————————\n",
|
||
"[assistant] L'utilisateur a demandé une histoire rapide, et l'assistant a raconté celle d'un chat nommé Whiskers qui, après avoir trouvé une pierre mystérieuse dans son jardin, a obtenu le pouvoir de parler aux oiseaux. Avec ses nouveaux amis oiseaux, Whiskers a vécu de grandes aventures, résolvant des mystères et explorant le village, devenant ainsi le chat le plus aventurier du village. (sum_001)\n",
|
||
"[user ] Can you tell me three extremely funny stories? (item_BTuNN64LdULM21OyC4vzN)\n",
|
||
"[assistant] Sure, let's dive into some giggle-worthy tales: **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast! **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks! **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days. (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
|
||
"——————————————————————————————————————————\n",
|
||
"✅ Summary inserted (sum_001)\n",
|
||
"—— response.done (window ≈2147 tokens) ——\n",
|
||
"—— Conversation so far ———————————————\n",
|
||
"[assistant] L'utilisateur a demandé une histoire rapide, et l'assistant a raconté celle d'un chat nommé Whiskers qui, après avoir trouvé une pierre mystérieuse dans son jardin, a obtenu le pouvoir de parler aux oiseaux. Avec ses nouveaux amis oiseaux, Whiskers a vécu de grandes aventures, résolvant des mystères et explorant le village, devenant ainsi le chat le plus aventurier du village. (sum_001)\n",
|
||
"[user ] Can you tell me three extremely funny stories? (item_BTuNN64LdULM21OyC4vzN)\n",
|
||
"[assistant] Sure, let's dive into some giggle-worthy tales: **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast! **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks! **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days. (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
|
||
"[user ] (item_BTuPLaCv8ATdIwAQ2rLgO)\n",
|
||
"[assistant] Sure! The first summary I provided between us was in French. (item_BTuPLa7BaSQToGCVOmfBK)\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"---\n",
|
||
"We had a conversation with our Voice AI. After several turns, the total token count reached SUMMARY_MAX, which triggered the conversation summarization step. This generated a summary of the earlier messages.\n",
|
||
"\n",
|
||
"Since there were N = 4 total messages, we summarized the first N - 2 = 2 messages:\n",
|
||
"```txt\n",
|
||
"—— Conversation so far ———————————————\n",
|
||
"[user ] Can you tell me a quick story? (item_BTuMOcpUqp8qknKhLzlkA)\n",
|
||
"[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end. (item_BTuMPRWxqpv0ph6QM46DK)\n",
|
||
"```\n",
|
||
"\n",
|
||
"We then created a summary in French and inserted it into the conversation history using the root: true flag. This ensured the summary appeared as the first message in the conversation. After that, we deleted the original items, using `\"type\": \"conversation.item.delete\"`, that were summarized.\n",
|
||
"\n",
|
||
"To validate the summary insertion, we asked the Voice AI what language the summary was in. It correctly responded:\n",
|
||
"\n",
|
||
"```txt\n",
|
||
"[assistant] Sure! The first summary I provided between us was in French. (item_BTuPLa7BaSQToGCVOmfBK)\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5 · Real‑World Applications\n",
|
||
"\n",
|
||
"Context summarisation can be useful for **long‑running voice experiences**. \n",
|
||
"Here are a use case ideas:\n",
|
||
"\n",
|
||
"| Use‑case | Added Value | Why Useful |\n",
|
||
"|----------|-------------|------------|\n",
|
||
"| **Customer‑support voicebot** | 24/7 natural phone tree; auto‑generate ticket summaries | Summarizes long customer calls for efficient handoff and record-keeping, reducing agent workload and improving response quality. |\n",
|
||
"| **Language tutor** | Real‑time conversation practice with corrective feedback | Helps track learner progress and highlights recurring mistakes, enabling personalized feedback and more effective language acquisition. |\n",
|
||
"| **AI therapist / coach** | Safe, always‑available listener that remembers sessions | Maintains continuity across sessions by recalling key topics and emotional tone, supporting a more empathetic and effective experience. |\n",
|
||
"| **Meeting assistant** | Live transcripts + concise action‑item recap in Slack | Distills lengthy meetings into actionable summaries, saving team members time and ensuring important points are not missed. |\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6 · Next Steps & Further Reading\n",
|
||
"Try out the notebook and try integrating context summary into your application.\n",
|
||
"\n",
|
||
"Few things you can try:\n",
|
||
"| Try this… | What you’ll learn |\n",
|
||
"|-----------|------------------|\n",
|
||
"| **A/B test summarisation**<br/>Run your eval suite with summarisation *on* vs *off*. | Whether trimming actually improves quality for your domain—and how it affects latency & cost. |\n",
|
||
"| **Swap summary styles**<br/>Change the system prompt to bullet points, JSON, English vs French, etc. | Which format the downstream assistant absorbs best; how language choice influences follow‑up answers. |\n",
|
||
"| **Vary thresholds**<br/>Play with `SUMMARY_TRIGGER_TOKENS` (2 k → 8 k). | The sweet spot between model drift and summarisation overhead. |\n",
|
||
"| **Cost tracing**<br/>Log `usage.total_tokens` before/after summarisation. | Concrete ROI: token savings per hour of conversation. |\n",
|
||
"\n",
|
||
"\n",
|
||
"### Resources:\n",
|
||
"- [OpenAI Realtime Guide](https://platform.openai.com/docs/guides/realtime)\n",
|
||
"- [OpenAI Realtime Conversations](https://platform.openai.com/docs/guides/realtime-conversations)\n",
|
||
"- [OpenAI Realtime API Reference](https://platform.openai.com/docs/api-reference/realtime)\n",
|
||
"- [Voice AI and Voice Agents](https://voiceaiandvoiceagents.com/)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "openai",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.8"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|