chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,81 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
# 管道和工作流
|
||||
|
||||
[`VoicePipeline`][agents.voice.pipeline.VoicePipeline] 是一个类，可让你轻松将智能体式工作流转换为语音应用。你传入要运行的工作流，管道会负责转写输入音频、检测音频何时结束、在合适的时机调用你的工作流，并将工作流输出转换回音频。
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
%% Input
|
||||
A["🎤 Audio Input"]
|
||||
|
||||
%% Voice Pipeline
|
||||
subgraph Voice_Pipeline [Voice Pipeline]
|
||||
direction TB
|
||||
B["Transcribe (speech-to-text)"]
|
||||
C["Your Code"]:::highlight
|
||||
D["Text-to-speech"]
|
||||
B --> C --> D
|
||||
end
|
||||
|
||||
%% Output
|
||||
E["🎧 Audio Output"]
|
||||
|
||||
%% Flow
|
||||
A --> Voice_Pipeline
|
||||
Voice_Pipeline --> E
|
||||
|
||||
%% Custom styling
|
||||
classDef highlight fill:#ffcc66,stroke:#333,stroke-width:1px,font-weight:700;
|
||||
|
||||
```
|
||||
|
||||
## 管道配置
|
||||
|
||||
创建管道时，你可以设置以下几项：
|
||||
|
||||
1. [`workflow`][agents.voice.workflow.VoiceWorkflowBase]，即每当有新的音频被转写时运行的代码。
|
||||
2. 使用的 [`speech-to-text`][agents.voice.model.STTModel] 和 [`text-to-speech`][agents.voice.model.TTSModel] 模型
|
||||
3. [`config`][agents.voice.pipeline_config.VoicePipelineConfig]，可用于配置以下内容：
|
||||
- 模型提供方，可将模型名称映射到模型
|
||||
- 追踪，包括是否禁用追踪、是否上传音频文件、工作流名称、追踪 ID 等。
|
||||
- TTS 和 STT 模型的设置，例如提示词、语言以及使用的数据类型。
|
||||
|
||||
## 管道运行
|
||||
|
||||
你可以通过 [`run()`][agents.voice.pipeline.VoicePipeline.run] 方法运行管道，该方法允许你以两种形式传入音频输入：
|
||||
|
||||
1. 当你已有完整的音频输入，并且只想为其生成结果时，可以使用 [`AudioInput`][agents.voice.input.AudioInput]。这适用于不需要检测说话者何时说完的场景；例如，你有预录音频，或者在按键通话应用中，用户何时说完是明确的。
|
||||
2. 当你可能需要检测用户何时说完时，可以使用 [`StreamedAudioInput`][agents.voice.input.StreamedAudioInput]。它允许你在检测到音频片段时将其推送进去，语音管道会通过一个称为“活动检测”的过程，在合适的时机自动运行智能体工作流。
|
||||
|
||||
## 结果
|
||||
|
||||
语音管道运行的结果是 [`StreamedAudioResult`][agents.voice.result.StreamedAudioResult]。这是一个对象，可让你在事件发生时以流式方式传输这些事件。[`VoiceStreamEvent`][agents.voice.events.VoiceStreamEvent] 有几种类型，包括：
|
||||
|
||||
1. [`VoiceStreamEventAudio`][agents.voice.events.VoiceStreamEventAudio]，其中包含一个音频片段。
|
||||
2. [`VoiceStreamEventLifecycle`][agents.voice.events.VoiceStreamEventLifecycle]，用于告知你轮次开始或结束等生命周期事件。
|
||||
3. [`VoiceStreamEventError`][agents.voice.events.VoiceStreamEventError]，这是一种错误事件。
|
||||
|
||||
```python
|
||||
|
||||
result = await pipeline.run(input)
|
||||
|
||||
async for event in result.stream():
|
||||
if event.type == "voice_stream_event_audio":
|
||||
# play audio
|
||||
pass
|
||||
elif event.type == "voice_stream_event_lifecycle":
|
||||
# lifecycle
|
||||
pass
|
||||
elif event.type == "voice_stream_event_error":
|
||||
# error
|
||||
pass
|
||||
```
|
||||
|
||||
## 最佳实践
|
||||
|
||||
### 中断
|
||||
|
||||
Agents SDK 目前没有为 [`StreamedAudioInput`][agents.voice.input.StreamedAudioInput] 提供任何内置的中断处理。相反，每个检测到的轮次都会触发你的工作流单独运行一次。如果你想在应用内处理中断，可以监听 [`VoiceStreamEventLifecycle`][agents.voice.events.VoiceStreamEventLifecycle] 事件。`turn_started` 表示已转写出新的轮次，且处理即将开始。`turn_ended` 会在相应轮次的所有音频都分发完毕后触发。你可以使用这些事件，在模型开始一个轮次时将说话者的麦克风静音，并在该轮次的所有相关音频都刷新完后取消静音。
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
# 快速入门
|
||||
|
||||
## 前置条件
|
||||
|
||||
请确保已按照 Agents SDK 的基础[快速入门说明](../quickstart.md)完成操作，并设置好虚拟环境。然后，从 SDK 安装可选的语音依赖项：
|
||||
|
||||
```bash
|
||||
pip install 'openai-agents[voice]'
|
||||
```
|
||||
|
||||
## 概念
|
||||
|
||||
需要了解的核心概念是 [`VoicePipeline`][agents.voice.pipeline.VoicePipeline]，它包含以下三个步骤：
|
||||
|
||||
1. 运行语音转文本模型，将音频转换为文本。
|
||||
2. 运行你的代码（通常是智能体工作流）以生成结果。
|
||||
3. 运行文本转语音模型，将结果文本转换回音频。
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
%% Input
|
||||
A["🎤 Audio Input"]
|
||||
|
||||
%% Voice Pipeline
|
||||
subgraph Voice_Pipeline [Voice Pipeline]
|
||||
direction TB
|
||||
B["Transcribe (speech-to-text)"]
|
||||
C["Your Code"]:::highlight
|
||||
D["Text-to-speech"]
|
||||
B --> C --> D
|
||||
end
|
||||
|
||||
%% Output
|
||||
E["🎧 Audio Output"]
|
||||
|
||||
%% Flow
|
||||
A --> Voice_Pipeline
|
||||
Voice_Pipeline --> E
|
||||
|
||||
%% Custom styling
|
||||
classDef highlight fill:#ffcc66,stroke:#333,stroke-width:1px,font-weight:700;
|
||||
|
||||
```
|
||||
|
||||
## 智能体
|
||||
|
||||
首先，我们来设置一些智能体。如果你曾使用此 SDK 构建过智能体，这部分应该会很熟悉。我们将使用两个智能体、一次任务转移和一个工具。
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import random
|
||||
|
||||
from agents import (
|
||||
Agent,
|
||||
function_tool,
|
||||
)
|
||||
from agents.extensions.handoff_prompt import prompt_with_handoff_instructions
|
||||
|
||||
|
||||
|
||||
@function_tool
|
||||
def get_weather(city: str) -> str:
|
||||
"""Get the weather for a given city."""
|
||||
print(f"[debug] get_weather called with city: {city}")
|
||||
choices = ["sunny", "cloudy", "rainy", "snowy"]
|
||||
return f"The weather in {city} is {random.choice(choices)}."
|
||||
|
||||
|
||||
spanish_agent = Agent(
|
||||
name="Spanish",
|
||||
handoff_description="A Spanish-speaking agent.",
|
||||
instructions=prompt_with_handoff_instructions(
|
||||
"You're speaking to a human, so be polite and concise. Speak in Spanish.",
|
||||
),
|
||||
model="gpt-5.6-sol",
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions=prompt_with_handoff_instructions(
|
||||
"You're speaking to a human, so be polite and concise. If the user speaks in Spanish, hand off to the Spanish agent.",
|
||||
),
|
||||
model="gpt-5.6-sol",
|
||||
handoffs=[spanish_agent],
|
||||
tools=[get_weather],
|
||||
)
|
||||
```
|
||||
|
||||
## 语音管线
|
||||
|
||||
我们将设置一个简单的语音管线，并使用 [`SingleAgentVoiceWorkflow`][agents.voice.workflow.SingleAgentVoiceWorkflow] 作为工作流。
|
||||
|
||||
```python
|
||||
from agents.voice import SingleAgentVoiceWorkflow, VoicePipeline
|
||||
pipeline = VoicePipeline(workflow=SingleAgentVoiceWorkflow(agent))
|
||||
```
|
||||
|
||||
## 管线运行
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import sounddevice as sd
|
||||
from agents.voice import AudioInput
|
||||
|
||||
# For simplicity, we'll just create 3 seconds of silence
|
||||
# In reality, you'd get microphone data
|
||||
buffer = np.zeros(24000 * 3, dtype=np.int16)
|
||||
audio_input = AudioInput(buffer=buffer)
|
||||
|
||||
result = await pipeline.run(audio_input)
|
||||
|
||||
# Create an audio player using `sounddevice`
|
||||
player = sd.OutputStream(samplerate=24000, channels=1, dtype=np.int16)
|
||||
player.start()
|
||||
|
||||
# Play the audio stream as it comes in
|
||||
async for event in result.stream():
|
||||
if event.type == "voice_stream_event_audio":
|
||||
player.write(event.data)
|
||||
|
||||
```
|
||||
|
||||
## 完整整合
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import random
|
||||
|
||||
import numpy as np
|
||||
import sounddevice as sd
|
||||
|
||||
from agents import (
|
||||
Agent,
|
||||
function_tool,
|
||||
set_tracing_disabled,
|
||||
)
|
||||
from agents.voice import (
|
||||
AudioInput,
|
||||
SingleAgentVoiceWorkflow,
|
||||
VoicePipeline,
|
||||
)
|
||||
from agents.extensions.handoff_prompt import prompt_with_handoff_instructions
|
||||
|
||||
|
||||
@function_tool
|
||||
def get_weather(city: str) -> str:
|
||||
"""Get the weather for a given city."""
|
||||
print(f"[debug] get_weather called with city: {city}")
|
||||
choices = ["sunny", "cloudy", "rainy", "snowy"]
|
||||
return f"The weather in {city} is {random.choice(choices)}."
|
||||
|
||||
|
||||
spanish_agent = Agent(
|
||||
name="Spanish",
|
||||
handoff_description="A Spanish-speaking agent.",
|
||||
instructions=prompt_with_handoff_instructions(
|
||||
"You're speaking to a human, so be polite and concise. Speak in Spanish.",
|
||||
),
|
||||
model="gpt-5.6-sol",
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions=prompt_with_handoff_instructions(
|
||||
"You're speaking to a human, so be polite and concise. If the user speaks in Spanish, hand off to the Spanish agent.",
|
||||
),
|
||||
model="gpt-5.6-sol",
|
||||
handoffs=[spanish_agent],
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
|
||||
async def main():
|
||||
pipeline = VoicePipeline(workflow=SingleAgentVoiceWorkflow(agent))
|
||||
buffer = np.zeros(24000 * 3, dtype=np.int16)
|
||||
audio_input = AudioInput(buffer=buffer)
|
||||
|
||||
result = await pipeline.run(audio_input)
|
||||
|
||||
# Create an audio player using `sounddevice`
|
||||
player = sd.OutputStream(samplerate=24000, channels=1, dtype=np.int16)
|
||||
player.start()
|
||||
|
||||
# Play the audio stream as it comes in
|
||||
async for event in result.stream():
|
||||
if event.type == "voice_stream_event_audio":
|
||||
player.write(event.data)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
运行此示例后，智能体就会与你进行语音交流！请查看 [examples/voice/static](https://github.com/openai/openai-agents-python/tree/main/examples/voice/static) 中的示例，了解如何亲自与智能体进行语音交流。
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
# 追踪
|
||||
|
||||
就像[智能体会被追踪](../tracing.md)一样，语音管线也会被自动追踪。
|
||||
|
||||
你可以阅读上面的追踪文档以了解基本的追踪信息；此外，你还可以通过[`VoicePipelineConfig`][agents.voice.pipeline_config.VoicePipelineConfig]配置管线的追踪。
|
||||
|
||||
与追踪相关的关键字段包括：
|
||||
|
||||
- [`tracing_disabled`][agents.voice.pipeline_config.VoicePipelineConfig.tracing_disabled]：控制是否禁用追踪。默认情况下，追踪处于启用状态。
|
||||
- [`trace_include_sensitive_data`][agents.voice.pipeline_config.VoicePipelineConfig.trace_include_sensitive_data]：控制追踪是否包含可能敏感的数据，例如音频转录文本。这专门针对语音管线，不适用于你的工作流内部发生的任何事情。
|
||||
- [`trace_include_sensitive_audio_data`][agents.voice.pipeline_config.VoicePipelineConfig.trace_include_sensitive_audio_data]：控制追踪是否包含音频数据。
|
||||
- [`workflow_name`][agents.voice.pipeline_config.VoicePipelineConfig.workflow_name]：追踪工作流的名称。
|
||||
- [`group_id`][agents.voice.pipeline_config.VoicePipelineConfig.group_id]：追踪的`group_id`，可用于关联多个追踪。
|
||||
- [`trace_metadata`][agents.voice.pipeline_config.VoicePipelineConfig.trace_metadata]：要包含在追踪中的额外元数据。
|
||||
Reference in New Issue
Block a user