wehub/openai--openai-agents-python
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:39:17 +08:00
commit 4ed4e9ff99
1368 changed files with 334957 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/realtime/guide.md
+344
View File
@@ -0,0 +1,344 @@
# Realtime agents guide
This guide explains how the OpenAI Agents SDK's realtime layer maps onto the OpenAI Realtime API, and what extra behavior the Python SDK adds on top.
!!! note "Start here"
If you want the default Python path, read the [quickstart](quickstart.md) first. If you are deciding whether your app should use server-side WebSocket or SIP, read [Realtime transport](transport.md). Browser WebRTC transport is not part of the Python SDK.
## Overview
Realtime agents keep a long-lived connection open to the Realtime API so the model can process text and audio incrementally, stream audio output, call tools, and handle interruptions without restarting a fresh request on every turn.
The main SDK components are:
- **RealtimeAgent**: Instructions, tools, output guardrails, and handoffs for one realtime specialist
- **RealtimeRunner**: Session factory that wires a starting agent to a realtime transport
- **RealtimeSession**: A live session that sends input, receives events, tracks history, and executes tools
- **RealtimeModel**: The transport abstraction. The default is OpenAI's server-side WebSocket implementation.
## Session lifecycle
A typical realtime session looks like this:
1. Create one or more `RealtimeAgent`s.
2. Create a `RealtimeRunner` with the starting agent.
3. Call `await runner.run()` to get a `RealtimeSession`.
4. Enter the session with `async with session:` or `await session.enter()`.
5. Send user input with `send_message()` or `send_audio()`.
6. Iterate over session events until the conversation ends.
Unlike text-only runs, `runner.run()` does not produce a final result immediately. It returns a live session object that keeps local history, background tool execution, guardrail state, and the active agent configuration in sync with the transport layer.
By default, `RealtimeRunner` uses `OpenAIRealtimeWebSocketModel`, so the default Python path is a server-side WebSocket connection to the Realtime API. If you pass a different `RealtimeModel`, the same session lifecycle and agent features still apply, while the connection mechanics can change.
## Agent and session configuration
`RealtimeAgent` is intentionally narrower than the regular `Agent` type:
- Model choice is configured at the session level, not per agent.
- Structured outputs are not supported.
- Voice can be configured, but it cannot change after the session has already produced spoken audio.
- Instructions, function tools, handoffs, hooks, and output guardrails all still work.
`RealtimeSessionModelSettings` supports both a newer nested `audio` config and older flat aliases. Prefer the nested shape for new code, and start with `gpt-realtime-2.1` for new realtime agents:
```python
runner = RealtimeRunner(
starting_agent=agent,
config={
"model_settings": {
"model_name": "gpt-realtime-2.1",
"audio": {
"input": {
"format": "pcm16",
"transcription": {"model": "gpt-4o-mini-transcribe"},
"turn_detection": {"type": "semantic_vad", "interrupt_response": True},
},
"output": {"format": "pcm16", "voice": "ash"},
},
"tool_choice": "auto",
}
},
)
```
Useful session-level settings include:
- `audio.input.format`, `audio.output.format`
- `audio.input.transcription`
- `audio.input.noise_reduction`
- `audio.input.turn_detection`
- `audio.output.voice`, `audio.output.speed`
- `output_modalities`
- `tool_choice`
- `prompt`
- `tracing`
Useful run-level settings on `RealtimeRunner(config=...)` include:
- `async_tool_calls`
- `output_guardrails`
- `guardrails_settings.debounce_text_length`
- `tool_error_formatter`
- `tracing_disabled`
See [`RealtimeRunConfig`][agents.realtime.config.RealtimeRunConfig] and [`RealtimeSessionModelSettings`][agents.realtime.config.RealtimeSessionModelSettings] for the full typed surface.
## Inputs and outputs
### Text and structured user messages
Use [`session.send_message()`][agents.realtime.session.RealtimeSession.send_message] for plain text or structured realtime messages.
```python
from agents.realtime import RealtimeUserInputMessage
await session.send_message("Summarize what we discussed so far.")
message: RealtimeUserInputMessage = {
"type": "message",
"role": "user",
"content": [
{"type": "input_text", "text": "Describe this image."},
{"type": "input_image", "image_url": image_data_url, "detail": "high"},
],
}
await session.send_message(message)
```
Structured messages are the main way to include image input in a realtime conversation. The example web demo in [`examples/realtime/app/server.py`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/app/server.py) forwards `input_image` messages this way.
### Audio input
Use [`session.send_audio()`][agents.realtime.session.RealtimeSession.send_audio] to stream raw audio bytes:
```python
await session.send_audio(audio_bytes)
```
If server-side turn detection is disabled, you are responsible for marking turn boundaries. The high-level convenience is:
```python
await session.send_audio(audio_bytes, commit=True)
```
If you need lower-level control, you can also send raw client events such as `input_audio_buffer.commit` through the underlying model transport.
### Manual response control
`session.send_message()` sends user input using the high-level path and starts a response for you. Raw audio buffering does **not** automatically do the same in every configuration.
At the Realtime API level, manual turn control means clearing `turn_detection` with a raw `session.update`, then sending `input_audio_buffer.commit` and `response.create` yourself.
If you are managing turns manually, you can send raw client events through the model transport:
```python
from agents.realtime.model_inputs import RealtimeModelSendRawMessage
await session.model.send_event(
RealtimeModelSendRawMessage(
message={
"type": "response.create",
}
)
)
```
This pattern is useful when:
- `turn_detection` is disabled and you want to decide when the model should respond
- you want to inspect or gate user input before triggering a response
- you need a custom prompt for an out-of-band response
The SIP example in [`examples/realtime/twilio_sip/server.py`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio_sip/server.py) uses a raw `response.create` to force an opening greeting.
## Events, history, and interruptions
`RealtimeSession` emits higher-level SDK events while still forwarding raw model events when you need them.
High-value session events include:
- `audio`, `audio_end`, `audio_interrupted`
- `agent_start`, `agent_end`
- `tool_start`, `tool_end`, `tool_approval_required`
- `handoff`
- `history_added`, `history_updated`
- `guardrail_tripped`
- `input_audio_timeout_triggered`
- `error`
- `raw_model_event`
The most useful events for UI state are usually `history_added` and `history_updated`. They expose the session's local history as `RealtimeItem` objects, including user messages, assistant messages, and tool calls.
### Interruptions and playback tracking
When the user interrupts the assistant, the session emits `audio_interrupted` and updates history so the server-side conversation stays aligned with what the user actually heard.
In low-latency local playback, the default playback tracker is often enough. In remote or delayed playback scenarios, especially telephony, use [`RealtimePlaybackTracker`][agents.realtime.model.RealtimePlaybackTracker] so interruption truncation is based on actual playback progress rather than assuming all generated audio has already been heard.
The Twilio example in [`examples/realtime/twilio/twilio_handler.py`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio/twilio_handler.py) shows this pattern.
## Tools, approvals, handoffs, and guardrails
### Function tools
Realtime agents support function tools during live conversations:
```python
from agents import function_tool
@function_tool
def get_weather(city: str) -> str:
"""Get current weather for a city."""
return f"The weather in {city} is sunny, 72F."
agent = RealtimeAgent(
name="Assistant",
instructions="You can answer weather questions.",
tools=[get_weather],
)
```
### Tool approvals
Function tools can require human approval before execution. When that happens, the session emits `tool_approval_required` and pauses the tool run until you call `approve_tool_call()` or `reject_tool_call()`.
If the tool also has input guardrails, those guardrails run immediately before execution after approval. To run them before the approval event is emitted, create the runner with `RealtimeRunner(..., config={"tool_execution": {"pre_approval_tool_input_guardrails": True}})`. Calls that pass this pre-approval check are still checked again after approval before execution.
```python
async for event in session:
if event.type == "tool_approval_required":
await session.approve_tool_call(event.call_id)
```
For a concrete server-side approval loop, see [`examples/realtime/app/server.py`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/app/server.py). The human-in-the-loop docs also point back to this flow in [Human in the loop](../human_in_the_loop.md).
### Handoffs
Realtime handoffs let one agent transfer the live conversation to another specialist:
```python
from agents.realtime import RealtimeAgent, realtime_handoff
billing_agent = RealtimeAgent(
name="Billing Support",
instructions="You specialize in billing issues.",
)
main_agent = RealtimeAgent(
name="Customer Service",
instructions="Triage the request and hand off when needed.",
handoffs=[
realtime_handoff(
billing_agent,
tool_description_override="Transfer to billing support",
)
],
)
```
Bare `RealtimeAgent` handoffs are auto-wrapped, and `realtime_handoff(...)` lets you customize names, descriptions, validation, callbacks, and availability. Realtime handoffs do **not** support the regular handoff `input_filter`.
### Guardrails
Realtime agents support output guardrails on agent responses and input guardrails on function-tool calls. Output guardrails run on debounced transcript accumulation rather than on every partial token, and they emit `guardrail_tripped` instead of raising an exception.
```python
from agents.guardrail import GuardrailFunctionOutput, OutputGuardrail
def sensitive_data_check(context, agent, output):
return GuardrailFunctionOutput(
tripwire_triggered="password" in output,
output_info=None,
)
agent = RealtimeAgent(
name="Assistant",
instructions="...",
output_guardrails=[OutputGuardrail(guardrail_function=sensitive_data_check)],
)
```
When a realtime output guardrail trips, the session interrupts the active response, forces `response.cancel`, emits `guardrail_tripped`, and sends a follow-up user message that names the triggered guardrail so the model can produce a replacement response. Your audio player should still listen for `audio_interrupted` and stop local playback immediately, because guardrails run on debounced transcript text and some audio may already be buffered when the tripwire fires.
## SIP and telephony
The Python SDK includes a first-class SIP attach flow via [`OpenAIRealtimeSIPModel`][agents.realtime.openai_realtime.OpenAIRealtimeSIPModel].
Use it when a call arrives through the Realtime Calls API and you want to attach an agent session to the resulting `call_id`:
```python
from agents.realtime import RealtimeRunner
from agents.realtime.openai_realtime import OpenAIRealtimeSIPModel
runner = RealtimeRunner(starting_agent=agent, model=OpenAIRealtimeSIPModel())
async with await runner.run(
model_config={
"call_id": call_id_from_webhook,
}
) as session:
async for event in session:
...
```
If you need to accept the call first and want the accept payload to match the agent-derived session configuration, use `OpenAIRealtimeSIPModel.build_initial_session_payload(...)`. The complete flow is shown in [`examples/realtime/twilio_sip/server.py`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio_sip/server.py).
## Low-level access and custom endpoints
You can access the underlying transport object through `session.model`.
Use this when you need:
- custom listeners via `session.model.add_listener(...)`
- raw client events such as `response.create` or `session.update`
- custom `url`, `headers`, or `api_key` handling through `model_config`
- `call_id` attach to an existing realtime call
`RealtimeModelConfig` supports:
- `api_key`
- `url`
- `headers`
- `initial_model_settings`
- `playback_tracker`
- `call_id`
This repository's shipped `call_id` example is SIP. The broader Realtime API also uses `call_id` for some server-side control flows, but those are not packaged as Python examples here.
When connecting to Azure OpenAI, pass a GA Realtime endpoint URL and explicit headers. For example:
```python
session = await runner.run(
model_config={
"url": "wss://<your-resource>.openai.azure.com/openai/v1/realtime?model=<deployment-name>",
"headers": {"api-key": "<your-azure-api-key>"},
}
)
```
For token-based authentication, use a bearer token in `headers`:
```python
session = await runner.run(
model_config={
"url": "wss://<your-resource>.openai.azure.com/openai/v1/realtime?model=<deployment-name>",
"headers": {"authorization": f"Bearer {token}"},
}
)
```
If you pass `headers`, the SDK does not add `Authorization` automatically. Avoid the legacy beta path (`/openai/realtime?api-version=...`) with realtime agents.
## Further reading
- [Realtime transport](transport.md)
- [Quickstart](quickstart.md)
- [OpenAI Realtime conversations](https://developers.openai.com/api/docs/guides/realtime-conversations/)
- [OpenAI Realtime server-side controls](https://developers.openai.com/api/docs/guides/realtime-server-controls/)
- [`examples/realtime`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime)
docs/realtime/quickstart.md
+154
View File
@@ -0,0 +1,154 @@
# Quickstart
Realtime agents in the Python SDK are server-side, low-latency agents built on the OpenAI Realtime API over WebSocket transport.
!!! note "Python SDK boundary"
The Python SDK does **not** provide a browser WebRTC transport. This page only covers Python-managed realtime sessions over server-side WebSockets. Use this SDK for server-side orchestration, tools, approvals, and telephony integrations. See also [Realtime transport](transport.md).
## Prerequisites
- Python 3.10 or higher
- OpenAI API key
- Basic familiarity with the OpenAI Agents SDK
## Installation
If you haven't already, install the OpenAI Agents SDK:
```bash
pip install openai-agents
```
## Create a server-side realtime session
### 1. Import the realtime components
```python
import asyncio
from agents.realtime import RealtimeAgent, RealtimeRunner
```
### 2. Define the starting agent
```python
agent = RealtimeAgent(
name="Assistant",
instructions="You are a helpful voice assistant. Keep responses short and conversational.",
)
```
### 3. Configure the runner
Prefer the nested `audio.input` / `audio.output` session settings shape for new code. For new realtime agents, start with `gpt-realtime-2.1`.
```python
runner = RealtimeRunner(
starting_agent=agent,
config={
"model_settings": {
"model_name": "gpt-realtime-2.1",
"audio": {
"input": {
"format": "pcm16",
"transcription": {"model": "gpt-4o-mini-transcribe"},
"turn_detection": {
"type": "semantic_vad",
"interrupt_response": True,
},
},
"output": {
"format": "pcm16",
"voice": "ash",
},
},
}
},
)
```
### 4. Start the session and send input
`runner.run()` returns a `RealtimeSession`. The connection is opened when you enter the session context.
```python
async def main() -> None:
session = await runner.run()
async with session:
await session.send_message("Say hello in one short sentence.")
async for event in session:
if event.type == "audio":
# Forward or play event.audio.data.
pass
elif event.type == "history_added":
print(event.item)
elif event.type == "agent_end":
# One assistant turn finished.
break
elif event.type == "error":
print(f"Error: {event.error}")
if __name__ == "__main__":
asyncio.run(main())
```
`session.send_message()` accepts either a plain string or a structured realtime message. For raw audio chunks, use [`session.send_audio()`][agents.realtime.session.RealtimeSession.send_audio].
## What this quickstart does not include
- Microphone capture and speaker playback code. See the realtime examples in [`examples/realtime`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime).
- SIP / telephony attach flows. See [Realtime transport](transport.md) and the [SIP section](guide.md#sip-and-telephony).
## Key settings
Once the basic session works, the settings most people reach for next are:
- `model_name`
- `audio.input.format`, `audio.output.format`
- `audio.input.transcription`
- `audio.input.noise_reduction`
- `audio.input.turn_detection` for automatic turn detection
- `audio.output.voice`
- `tool_choice`, `prompt`, `tracing`
- `async_tool_calls`, `tool_execution.pre_approval_tool_input_guardrails`, `guardrails_settings.debounce_text_length`, `tool_error_formatter`
The older flat aliases such as `input_audio_format`, `output_audio_format`, `input_audio_transcription`, and `turn_detection` still work, but nested `audio` settings are preferred for new code.
For manual turn control, use a raw `session.update` / `input_audio_buffer.commit` / `response.create` flow as described in the [Realtime agents guide](guide.md#manual-response-control).
For the full schema, see [`RealtimeRunConfig`][agents.realtime.config.RealtimeRunConfig] and [`RealtimeSessionModelSettings`][agents.realtime.config.RealtimeSessionModelSettings].
## Connection options
Set your API key in the environment:
```bash
export OPENAI_API_KEY="your-api-key-here"
```
Or pass it directly when starting the session:
```python
session = await runner.run(model_config={"api_key": "your-api-key"})
```
`model_config` also supports:
- `url`: Custom WebSocket endpoint
- `headers`: Custom request headers
- `call_id`: Attach to an existing realtime call. In this repo, the documented attach flow is SIP.
- `playback_tracker`: Report how much audio the user has actually heard
If you pass `headers` explicitly, the SDK will **not** inject an `Authorization` header for you.
When connecting to Azure OpenAI, pass a GA Realtime endpoint URL in `model_config["url"]` and explicit headers. Avoid the legacy beta path (`/openai/realtime?api-version=...`) with realtime agents. See the [Realtime agents guide](guide.md#low-level-access-and-custom-endpoints) for details.
## Next steps
- Read [Realtime transport](transport.md) to choose between server-side WebSocket and SIP.
- Read the [Realtime agents guide](guide.md) for lifecycle, structured input, approvals, handoffs, guardrails, and low-level control.
- Browse the examples in [`examples/realtime`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime).
docs/realtime/transport.md
+72
View File
@@ -0,0 +1,72 @@
# Realtime transport
Use this page to decide how realtime agents fit into your Python application.
!!! note "Python SDK boundary"
The Python SDK does **not** include a browser WebRTC transport. This page is only about Python SDK transport choices: server-side WebSockets and SIP attach flows. Browser WebRTC is a separate platform topic, documented in the official [Realtime API with WebRTC](https://developers.openai.com/api/docs/guides/realtime-webrtc/) guide.
## Decision guide
| Goal | Start with | Why |
| --- | --- | --- |
| Build a server-managed realtime app | [Quickstart](quickstart.md) | The default Python path is a server-side WebSocket session managed by `RealtimeRunner`. |
| Understand which transport and deployment shape to choose | This page | Use this before you commit to a transport or deployment shape. |
| Attach agents to phone or SIP calls | [Realtime guide](guide.md) and [`examples/realtime/twilio_sip`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio_sip) | The repo ships a SIP attach flow driven by `call_id`. |
## Server-side WebSocket is the default Python path
`RealtimeRunner` uses `OpenAIRealtimeWebSocketModel` unless you pass a custom `RealtimeModel`.
That means the standard Python topology looks like this:
1. Your Python service creates a `RealtimeRunner`.
2. `await runner.run()` returns a `RealtimeSession`.
3. Enter the session and send text, structured messages, or audio.
4. Consume `RealtimeSessionEvent` items and forward audio or transcripts to your application.
This is the topology used by the core demo app, the CLI example, and the Twilio Media Streams example:
- [`examples/realtime/app`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/app)
- [`examples/realtime/cli`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/cli)
- [`examples/realtime/twilio`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio)
Use this path when your server owns the audio pipeline, tool execution, approval flow, and history handling.
## SIP attach is the telephony path
For the telephony flow documented in this repository, the Python SDK attaches to an existing realtime call via `call_id`.
This topology looks like:
1. OpenAI sends your service a webhook such as `realtime.call.incoming`.
2. Your service accepts the call through the Realtime Calls API.
3. Your Python service starts a `RealtimeRunner(..., model=OpenAIRealtimeSIPModel())`.
4. The session connects with `model_config={"call_id": ...}` and then processes events like any other realtime session.
This is the topology shown in [`examples/realtime/twilio_sip`](https://github.com/openai/openai-agents-python/tree/main/examples/realtime/twilio_sip).
The broader Realtime API also uses `call_id` for some server-side control patterns, but this repository's shipped attach example is SIP.
## Browser WebRTC is outside this SDK
If your app's primary client is a browser using Realtime WebRTC:
- Treat it as outside the scope of the Python SDK docs in this repository.
- Use the official [Realtime API with WebRTC](https://developers.openai.com/api/docs/guides/realtime-webrtc/) and [Realtime conversations](https://developers.openai.com/api/docs/guides/realtime-conversations/) docs for the client-side flow and event model.
- Use the official [Realtime server-side controls](https://developers.openai.com/api/docs/guides/realtime-server-controls/) guide if you need a sideband server connection on top of a browser WebRTC client.
- Do not expect this repository to provide a browser-side `RTCPeerConnection` abstraction or a ready-made browser WebRTC sample.
This repository also does not currently ship a browser WebRTC plus Python sideband example.
## Custom endpoints and attach points
The transport configuration surface in [`RealtimeModelConfig`][agents.realtime.model.RealtimeModelConfig] lets you adapt the default paths:
- `url`: Override the WebSocket endpoint
- `headers`: Provide explicit headers such as Azure auth headers
- `api_key`: Pass an API key directly or via callback
- `call_id`: Attach to an existing realtime call. In this repository, the documented example is SIP.
- `playback_tracker`: Report actual playback progress for interruption handling
See the [Realtime agents guide](guide.md) for the detailed lifecycle and capability surface once you've chosen a topology.