wehub-resource-sync
457 lines
18 KiB
Markdown
457 lines
18 KiB
Markdown
<!-- WEHUB_ZH_README -->
|
||
> [!NOTE]
|
||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||
> [English](./README.en.md) · [原始项目](https://github.com/livekit/agents) · [上游 README](https://github.com/livekit/agents/blob/HEAD/README.md)
|
||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||
|
||
<!--BEGIN_BANNER_IMAGE-->
|
||
|
||
<picture>
|
||
<source media="(prefers-color-scheme: dark)" srcset="/.github/banner_dark.png">
|
||
<source media="(prefers-color-scheme: light)" srcset="/.github/banner_light.png">
|
||
<img style="width:100%;" alt="LiveKit 图标、仓库名称以及背景中的示例代码。" src="https://raw.githubusercontent.com/livekit/agents/main/.github/banner_light.png">
|
||
</picture>
|
||
|
||
<!--END_BANNER_IMAGE-->
|
||
<br />
|
||
|
||
|
||
[](https://pepy.tech/projects/livekit-agents)
|
||
[](https://livekit.io/join-slack)
|
||
[](https://twitter.com/livekit)
|
||
[](https://deepwiki.com/livekit/agents)
|
||
[](https://github.com/livekit/livekit/blob/master/LICENSE)
|
||
|
||
<br />
|
||
|
||
在找 JS/TS 库？请查看 [AgentsJS](https://github.com/livekit/agents-js)
|
||
|
||
## Agents 是什么？
|
||
|
||
<!--BEGIN_DESCRIPTION-->
|
||
|
||
Agent Framework 专为构建在服务器上运行的实时、可编程参与者而设计。使用它可以创建能够看、听和理解的对话式、多模态语音智能体（voice agents）。
|
||
|
||
<!--END_DESCRIPTION-->
|
||
|
||
## 功能特性
|
||
|
||
- **灵活的集成**：全面的生态系统，可按需混搭合适的 STT、LLM、TTS 和 Realtime API。
|
||
- **集成的任务调度**：内置任务调度与分发，配合 [dispatch APIs](https://docs.livekit.io/agents/build/dispatch/) 将终端用户连接到智能体。
|
||
- **丰富的 WebRTC 客户端**：使用 LiveKit 的开源 SDK 生态系统构建客户端应用，支持所有主流平台。
|
||
- **电话集成**：与 LiveKit 的 [telephony stack](https://docs.livekit.io/sip/), 无缝配合，使智能体能够拨打或接听电话。
|
||
- **与客户端交换数据**：使用 [RPCs](https://docs.livekit.io/home/client/data/rpc/) 及其他 [Data APIs](https://docs.livekit.io/home/client/data/) 与客户端无缝交换数据。
|
||
- **语义化话轮检测（Semantic turn detection）**：使用 transformer 模型检测用户是否已完成当前话轮，有助于减少打断。
|
||
- **MCP 支持**：原生支持 MCP。一行代码即可集成 MCP 服务器提供的工具。
|
||
- **内置测试框架**：编写测试并使用评判器（judges）确保智能体按预期运行。
|
||
- **开源**：完全开源，允许你在自己的服务器上运行整个技术栈，包括使用最广泛的 WebRTC 媒体服务器之一 [LiveKit server](https://github.com/livekit/livekit),。
|
||
|
||
## 安装
|
||
|
||
安装核心 Agents 库以及常用模型提供商的插件：
|
||
|
||
```bash
|
||
pip install "livekit-agents[openai,deepgram,cartesia]"
|
||
```
|
||
|
||
## 文档与指南
|
||
|
||
有关该框架及其使用方法的文档可[在此](https://docs.livekit.io/agents/) 查阅。
|
||
|
||
### 使用 AI 编程智能体进行开发
|
||
|
||
如果你使用 AI 编程助手来基于 LiveKit Agents 进行开发，我们建议采用以下配置以获得最佳效果：
|
||
|
||
1. **安装 [LiveKit Docs MCP server](https://docs.livekit.io/mcp)**** — 让你的编程智能体访问最新的 LiveKit 文档、跨 LiveKit 仓库的代码搜索以及可用的示例。
|
||
|
||
2. **安装 [LiveKit Agent Skill](https://github.com/livekit/agent-skills)**** — 为你的编程智能体提供构建语音 AI 应用的架构指导与最佳实践，包括工作流设计、交接（handoffs）、任务和测试模式。
|
||
|
||
```shell
|
||
npx skills add livekit/agent-skills --skill livekit-agents
|
||
```
|
||
|
||
Agent Skill 与 MCP 服务器配合使用效果最佳：Skill 教会你的智能体*如何着手*基于 LiveKit 进行开发，而 MCP 服务器提供*最新的 API 细节*以确保正确实现。
|
||
|
||
## 核心概念
|
||
|
||
- Agent：具有明确定义指令的基于 LLM 的应用。
|
||
- AgentSession：管理智能体与终端用户交互的容器。
|
||
- entrypoint：交互式会话的入口，类似于 Web 服务器中的请求处理器。
|
||
- AgentServer：协调任务调度并为用户会话启动智能体的主进程。
|
||
|
||
## 用法
|
||
|
||
### 简单语音智能体
|
||
|
||
---
|
||
|
||
```python
|
||
from livekit.agents import (
|
||
Agent,
|
||
AgentServer,
|
||
AgentSession,
|
||
JobContext,
|
||
RunContext,
|
||
cli,
|
||
function_tool,
|
||
inference,
|
||
)
|
||
|
||
|
||
@function_tool
|
||
async def lookup_weather(
|
||
context: RunContext,
|
||
location: str,
|
||
):
|
||
"""Used to look up weather information."""
|
||
|
||
return {"weather": "sunny", "temperature": 70}
|
||
|
||
|
||
server = AgentServer()
|
||
|
||
|
||
@server.rtc_session()
|
||
async def entrypoint(ctx: JobContext):
|
||
session = AgentSession(
|
||
vad=inference.VAD(),
|
||
# any combination of STT, LLM, TTS, or realtime API can be used
|
||
# this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
|
||
# to use model provider keys directly, replace with the following:
|
||
# from livekit.plugins import deepgram, openai, cartesia
|
||
# stt=deepgram.STT(model="nova-3"),
|
||
# llm=openai.LLM(model="gpt-4.1-mini"),
|
||
# tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
|
||
stt=inference.STT("deepgram/nova-3", language="multi"),
|
||
llm=inference.LLM("openai/gpt-4.1-mini"),
|
||
tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
|
||
)
|
||
|
||
agent = Agent(
|
||
instructions="You are a friendly voice assistant built by LiveKit.",
|
||
tools=[lookup_weather],
|
||
)
|
||
|
||
await session.start(agent=agent, room=ctx.room)
|
||
await session.generate_reply(instructions="greet the user and ask about their day")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
cli.run_app(server)
|
||
```
|
||
|
||
运行此示例需要以下环境变量：
|
||
|
||
- LIVEKIT_URL
|
||
- LIVEKIT_API_KEY
|
||
- LIVEKIT_API_SECRET
|
||
|
||
### 多智能体交接
|
||
|
||
---
|
||
|
||
此代码片段为节选。完整示例请参阅 [multi_agent.py](examples/voice_agents/multi_agent.py)
|
||
|
||
```python
|
||
...
|
||
class IntroAgent(Agent):
|
||
def __init__(self) -> None:
|
||
super().__init__(
|
||
instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging."
|
||
"Ask the user for their name and where they are from"
|
||
)
|
||
|
||
async def on_enter(self):
|
||
self.session.generate_reply(instructions="greet the user and gather information")
|
||
|
||
@function_tool
|
||
async def information_gathered(
|
||
self,
|
||
context: RunContext,
|
||
name: str,
|
||
location: str,
|
||
):
|
||
"""Called when the user has provided the information needed to make the story personalized and engaging.
|
||
|
||
Args:
|
||
name: The name of the user
|
||
location: The location of the user
|
||
"""
|
||
|
||
context.userdata.name = name
|
||
context.userdata.location = location
|
||
|
||
story_agent = StoryAgent(name, location)
|
||
return story_agent, "Let's start the story!"
|
||
|
||
|
||
class StoryAgent(Agent):
|
||
def __init__(self, name: str, location: str) -> None:
|
||
super().__init__(
|
||
instructions=f"You are a storyteller. Use the user's information in order to make the story personalized."
|
||
f"The user's name is {name}, from {location}",
|
||
# override the default model, switching to Realtime API from standard LLMs
|
||
llm=openai.realtime.RealtimeModel(voice="echo"),
|
||
chat_ctx=chat_ctx,
|
||
)
|
||
|
||
async def on_enter(self):
|
||
self.session.generate_reply()
|
||
|
||
|
||
@server.rtc_session()
|
||
async def entrypoint(ctx: JobContext):
|
||
userdata = StoryData()
|
||
session = AgentSession[StoryData](
|
||
vad=inference.VAD(),
|
||
stt="deepgram/nova-3",
|
||
llm="openai/gpt-4.1-mini",
|
||
tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
|
||
userdata=userdata,
|
||
)
|
||
|
||
await session.start(
|
||
agent=IntroAgent(),
|
||
room=ctx.room,
|
||
)
|
||
...
|
||
```
|
||
|
||
### 测试
|
||
|
||
自动化测试对于构建可靠的 Agent 至关重要，尤其是在 LLM（大语言模型）具有非确定性行为的情况下。LiveKit Agents 内置了测试集成，可帮助你创建稳定可靠的 Agent。
|
||
|
||
```python
|
||
@pytest.mark.asyncio
|
||
async def test_no_availability() -> None:
|
||
llm = google.LLM()
|
||
async with AgentSession(llm=llm) as sess:
|
||
await sess.start(MyAgent())
|
||
result = await sess.run(
|
||
user_input="Hello, I need to place an order."
|
||
)
|
||
result.expect.skip_next_event_if(type="message", role="assistant")
|
||
result.expect.next_event().is_function_call(name="start_order")
|
||
result.expect.next_event().is_function_call_output()
|
||
await (
|
||
result.expect.next_event()
|
||
.is_message(role="assistant")
|
||
.judge(llm, intent="assistant should be asking the user what they would like")
|
||
)
|
||
|
||
```
|
||
|
||
## 示例
|
||
|
||
更多示例和详细设置说明，请参阅 [examples 目录](examples/)。如需更多示例，请参阅 [python-agents-examples](https://github.com/livekit-examples/python-agents-examples) 仓库。
|
||
|
||
<table>
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>🎙️ Starter Agent</h3>
|
||
<p>针对语音对话优化的入门 Agent。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/basic_agent.py">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>🔄 Multi-user push to talk</h3>
|
||
<p>通过按键通话（push-to-talk）响应房间内多位用户。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/push_to_talk.py">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>🎵 Background audio</h3>
|
||
<p>背景环境音和思考音效，提升真实感。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/background_audio.py">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>🛠️ Dynamic tool creation</h3>
|
||
<p>动态创建函数工具。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/dynamic_tool_creation.py">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>☎️ Outbound caller</h3>
|
||
<p>进行外呼电话的 Agent</p>
|
||
<p>
|
||
<a href="https://github.com/livekit-examples/outbound-caller-python">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>📋 Structured output</h3>
|
||
<p>使用 LLM 的结构化输出来指导 TTS 语调。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/structured_output.py">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>🔌 MCP support</h3>
|
||
<p>使用 MCP 服务器提供的工具</p>
|
||
<p>
|
||
<a href="examples/voice_agents/mcp">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>💬 Text-only agent</h3>
|
||
<p>完全跳过语音，使用相同代码进行纯文本集成</p>
|
||
<p>
|
||
<a href="examples/other/text_only.py">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>📝 Multi-user transcriber</h3>
|
||
<p>为房间内所有用户生成转录文本</p>
|
||
<p>
|
||
<a href="examples/other/transcription/multi-user-transcriber.py">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>🎥 Video avatars</h3>
|
||
<p>使用 Tavus、Bithuman、LemonSlice 等添加 AI 虚拟形象</p>
|
||
<p>
|
||
<a href="examples/avatar_agents/">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
<tr>
|
||
<td width="50%">
|
||
<h3>🍽️ Restaurant ordering and reservations</h3>
|
||
<p>处理餐厅来电的完整 Agent 示例。</p>
|
||
<p>
|
||
<a href="examples/voice_agents/restaurant_agent.py">代码</a>
|
||
</p>
|
||
</td>
|
||
<td width="50%">
|
||
<h3>👁️ Gemini Live vision</h3>
|
||
<p>具备视觉能力的 Gemini Live Agent 完整示例（含 iOS 应用）。</p>
|
||
<p>
|
||
<a href="https://github.com/livekit-examples/vision-demo">代码</a>
|
||
</p>
|
||
</td>
|
||
</tr>
|
||
|
||
</table>
|
||
|
||
## 运行你的 Agent
|
||
|
||
### 在终端中测试
|
||
|
||
```shell
|
||
python myagent.py console
|
||
```
|
||
|
||
在终端模式下运行 Agent，启用本地音频输入和输出以便测试。
|
||
此模式无需外部服务器或依赖，适合快速验证行为。
|
||
|
||
### 使用 LiveKit 客户端进行开发
|
||
|
||
```shell
|
||
python myagent.py dev
|
||
```
|
||
|
||
启动 Agent 服务器，并在文件变更时启用热重载。此模式允许每个进程高效托管多个并发 Agent。
|
||
|
||
Agent 会连接到 LiveKit Cloud 或你自托管的服务器。请设置以下环境变量：
|
||
- LIVEKIT_URL
|
||
- LIVEKIT_API_KEY
|
||
- LIVEKIT_API_SECRET
|
||
|
||
你可以使用任意 LiveKit 客户端 SDK 或电话集成进行连接。
|
||
要快速上手，请试用 [Agents Playground](https://agents-playground.livekit.io/).
|
||
|
||
### 生产环境运行
|
||
|
||
```shell
|
||
python myagent.py start
|
||
```
|
||
|
||
以面向生产环境的优化配置运行 Agent。
|
||
|
||
## 许可证
|
||
|
||
Agents 框架采用 [Apache-2.0](LICENSE) 许可证。LiveKit 话轮检测（turn detection）模型采用 [LiveKit Model License](MODEL_LICENSE) 许可证。
|
||
|
||
## 贡献
|
||
|
||
Agents 框架正处于快速发展领域中的积极开发阶段。我们欢迎并感谢各类贡献，无论是反馈、错误修复、功能、新插件和工具，还是更完善的文档。你可以在本仓库提交 issue、发起 PR，或在 [LiveKit 社区](https://docs.livekit.io/intro/community/).
|
||
|
||
### 开发环境设置
|
||
|
||
本项目使用 [uv](https://docs.astral.sh/uv/) 进行包管理。要安装开发依赖：
|
||
|
||
```shell
|
||
uv sync --all-extras --dev
|
||
```
|
||
|
||
### 示例
|
||
|
||
本项目在 [`examples`](examples/) 目录中包含许多示例。要运行它们，请创建文件 `examples/.env`，并填入 LiveKit Server 及所需模型提供商的凭据（参见 `examples/.env.example`），然后运行：
|
||
|
||
```shell
|
||
uv run examples/voice_agents/basic_agent.py dev
|
||
```
|
||
|
||
更多信息请参阅 [examples README](examples/README.md)。
|
||
|
||
### 测试
|
||
|
||
单元测试位于 `tests` 目录，可通过以下命令运行：
|
||
|
||
```shell
|
||
uv run pytest --unit
|
||
```
|
||
|
||
各插件的集成测试需要多种 API 凭据，并在项目维护者提交的 PR 中于 GitHub CI 自动运行。详情请参阅 [tests workflow](.github/workflows/tests.yml)。
|
||
|
||
### 格式化
|
||
|
||
本项目使用 [ruff](https://github.com/astral-sh/ruff) 进行格式化和 lint：
|
||
|
||
```shell
|
||
uv run ruff format
|
||
uv run ruff check --fix
|
||
```
|
||
|
||
### 文档
|
||
|
||
要使用 [pdoc](https://github.com/pdoc3/pdoc): 在本地生成文档：
|
||
|
||
```shell
|
||
uv sync --all-extras --group docs
|
||
uv run --active pdoc --skip-errors --html --output-dir=docs livekit
|
||
```
|
||
|
||
<!--BEGIN_REPO_NAV-->
|
||
<br/><table>
|
||
<thead><tr><th colspan="2">LiveKit 生态系统</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>Agents SDK</td><td><b>Python</b> · <a href="https://github.com/livekit/agents-js">Node.js</a></td></tr><tr></tr>
|
||
<tr><td>LiveKit SDK</td><td><a href="https://github.com/livekit/client-sdk-js">Browser</a> · <a href="https://github.com/livekit/client-sdk-swift">Swift</a> · <a href="https://github.com/livekit/client-sdk-android">Android</a> · <a href="https://github.com/livekit/client-sdk-flutter">Flutter</a> · <a href="https://github.com/livekit/client-sdk-react-native">React Native</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/client-sdk-unity">Unity</a> · <a href="https://github.com/livekit/client-sdk-unity-web">Unity (WebGL)</a> · <a href="https://github.com/livekit/client-sdk-esp32">ESP32</a> · <a href="https://github.com/livekit/client-sdk-cpp">C++</a></td></tr><tr></tr>
|
||
<tr><td>入门应用</td><td><a href="https://github.com/livekit-examples/agent-starter-python">Python Agent</a> · <a href="https://github.com/livekit-examples/agent-starter-node">TypeScript Agent</a> · <a href="https://github.com/livekit-examples/agent-starter-react">React App</a> · <a href="https://github.com/livekit-examples/agent-starter-swift">SwiftUI App</a> · <a href="https://github.com/livekit-examples/agent-starter-android">Android App</a> · <a href="https://github.com/livekit-examples/agent-starter-flutter">Flutter App</a> · <a href="https://github.com/livekit-examples/agent-starter-react-native">React Native App</a> · <a href="https://github.com/livekit-examples/agent-starter-embed">Web Embed</a></td></tr><tr></tr>
|
||
<tr><td>UI 组件</td><td><a href="https://github.com/livekit/components-js">React</a> · <a href="https://github.com/livekit/components-android">Android Compose</a> · <a href="https://github.com/livekit/components-swift">SwiftUI</a> · <a href="https://github.com/livekit/components-flutter">Flutter</a></td></tr><tr></tr>
|
||
<tr><td>服务器 API</td><td><a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/server-sdk-go">Golang</a> · <a href="https://github.com/livekit/server-sdk-ruby">Ruby</a> · <a href="https://github.com/livekit/server-sdk-kotlin">Java/Kotlin</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/agence104/livekit-server-sdk-php">PHP (community)</a> · <a href="https://github.com/pabloFuente/livekit-server-sdk-dotnet">.NET (community)</a></td></tr><tr></tr>
|
||
<tr><td>资源</td><td><a href="https://docs.livekit.io">Docs</a> · <a href="https://docs.livekit.io/mcp">Docs MCP Server</a> · <a href="https://github.com/livekit/livekit-cli">CLI</a> · <a href="https://cloud.livekit.io">LiveKit Cloud</a></td></tr><tr></tr>
|
||
<tr><td>LiveKit Server OSS</td><td><a href="https://github.com/livekit/livekit">LiveKit server</a> · <a href="https://github.com/livekit/egress">Egress</a> · <a href="https://github.com/livekit/ingress">Ingress</a> · <a href="https://github.com/livekit/sip">SIP</a></td></tr><tr></tr>
|
||
<tr><td>社区</td><td><a href="https://community.livekit.io">Developer Community</a> · <a href="https://livekit.io/join-slack">Slack</a> · <a href="https://x.com/livekit">X</a> · <a href="https://www.youtube.com/@livekit_io">YouTube</a></td></tr>
|
||
</tbody>
|
||
</table>
|
||
<!--END_REPO_NAV-->
|