diff --git a/README.md b/README.md index 248367f..53365b2 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,32 @@ + +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/simular-ai/Agent-S) · [上游 README](https://github.com/simular-ai/Agent-S/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 +

Logo Agent S: - Use Computer Like a Human + 像人类一样使用计算机

-

🏆 Agent S3: First to Surpass Human Performance on OSWorld (72.60%)

+

🏆 Agent S3:首个在 OSWorld 上超越人类表现(72.60%)

  - 🌐 [S3 blog]  - 📄 [S3 Paper]  - 🎥 [S3 Video] + 🌐 [S3 博客]  + 📄 [S3 论文]  + 🎥 [S3 视频]

  - 🌐 [S2 blog]  - 📄 [S2 Paper (COLM 2025)]  - 🎥 [S2 Video] + 🌐 [S2 博客]  + 📄 [S2 论文(COLM 2025)]  + 🎥 [S2 视频]

  - 🌐 [S1 blog]  - 📄 [S1 Paper (ICLR 2025)]  - 🎥 [S1 Video] + 🌐 [S1 博客]  + 📄 [S1 论文(ICLR 2025)]  + 🎥 [S1 视频]

  @@ -54,101 +60,101 @@

   -

Skip the setup? Try Agent S in Simular Cloud +

跳过配置?在 Simular Cloud 中试用 Agent S

-## 🥳 Updates -- [x] **2025/12/15**: Agent S3 is the **first** to surpass human-level performance on OSWorld with an impressive score of **72.60%**! -- [x] **2025/10/02**: Released Agent S3 and its [technical paper](https://arxiv.org/abs/2510.02250), setting a new SOTA of **69.9%** on OSWorld (approaching 72% human performance), with strong generalizability on WindowsAgentArena and AndroidWorld! It is also simpler, faster, and more flexible. -- [x] **2025/08/01**: Agent S2.5 is released (gui-agents v0.2.5): simpler, better, and faster! New SOTA on [OSWorld-Verified](https://os-world.github.io)! -- [x] **2025/07/07**: The [Agent S2 paper](https://arxiv.org/abs/2504.00906) is accepted to COLM 2025! See you in Montreal! -- [x] **2025/04/27**: The Agent S paper won the Best Paper Award 🏆 at ICLR 2025 Agentic AI for Science Workshop! -- [x] **2025/04/01**: Released the [Agent S2 paper](https://arxiv.org/abs/2504.00906) with new SOTA results on OSWorld, WindowsAgentArena, and AndroidWorld! -- [x] **2025/03/12**: Released Agent S2 along with v0.2.0 of [gui-agents](https://github.com/simular-ai/Agent-S), the new state-of-the-art for computer use agents (CUA), outperforming OpenAI's CUA/Operator and Anthropic's Claude 3.7 Sonnet Computer-Use! -- [x] **2025/01/22**: The [Agent S paper](https://arxiv.org/abs/2410.08164) is accepted to ICLR 2025! -- [x] **2025/01/21**: Released v0.1.2 of [gui-agents](https://github.com/simular-ai/Agent-S) library, with support for Linux and Windows! -- [x] **2024/12/05**: Released v0.1.0 of [gui-agents](https://github.com/simular-ai/Agent-S) library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease! -- [x] **2024/10/10**: Released the [Agent S paper](https://arxiv.org/abs/2410.08164) and codebase! +## 🥳 更新 +- [x] **2025/12/15**:Agent S3 **首次**在 OSWorld 上以 **72.60%** 的出色成绩超越人类水平表现! +- [x] **2025/10/02**:发布 Agent S3 及其[技术论文](https://arxiv.org/abs/2510.02250), setting a new SOTA of **69.9%** on OSWorld (approaching 72% human performance), with strong generalizability on WindowsAgentArena and AndroidWorld! It is also simpler, faster, and more flexible. +- [x] **2025/08/01**:发布 Agent S2.5(gui-agents v0.2.5):更简单、更好、更快![OSWorld-Verified](https://os-world.github.io)! 上创下新的 SOTA +- [x] **2025/07/07**:[Agent S2 论文](https://arxiv.org/abs/2504.00906) 被 COLM 2025 接收!蒙特利尔见! +- [x] **2025/04/27**:Agent S 论文在 ICLR 2025 Agentic AI for Science Workshop 荣获最佳论文奖 🏆! +- [x] **2025/04/01**:发布 [Agent S2 论文](https://arxiv.org/abs/2504.00906),在 OSWorld、WindowsAgentArena 和 AndroidWorld 上取得新的 SOTA 结果! +- [x] **2025/03/12**:发布 Agent S2 以及 [gui-agents](https://github.com/simular-ai/Agent-S), v0.2.0,成为计算机使用智能体(CUA)的新标杆,性能超越 OpenAI 的 CUA/Operator 和 Anthropic 的 Claude 3.7 Sonnet Computer-Use! +- [x] **2025/01/22**:[Agent S 论文](https://arxiv.org/abs/2410.08164) 被 ICLR 2025 接收! +- [x] **2025/01/21**:发布 [gui-agents](https://github.com/simular-ai/Agent-S) v0.1.2 库,支持 Linux 和 Windows! +- [x] **2024/12/05**:发布 [gui-agents](https://github.com/simular-ai/Agent-S) v0.1.0 库,让你轻松在 Mac、OSWorld 和 WindowsAgentArena 上使用 Agent-S! +- [x] **2024/10/10**:发布 [Agent S 论文](https://arxiv.org/abs/2410.08164) 及代码库! -## Table of Contents +## 目录 -1. [💡 Introduction](#-introduction) -2. [🎯 Current Results](#-current-results) -3. [🛠️ Installation & Setup](#%EF%B8%8F-installation--setup) -4. [🚀 Usage](#-usage) -5. [🤝 Acknowledgements](#-acknowledgements) -6. [💬 Citation](#-citation) +1. [💡 简介](#-introduction) +2. [🎯 当前成果](#-current-results) +3. [🛠️ 安装与配置](#%EF%B8%8F-installation--setup) +4. [🚀 用法](#-usage) +5. [🤝 致谢](#-acknowledgements) +6. [💬 引用](#-citation) -## 💡 Introduction +## 💡 简介 -Welcome to **Agent S**, an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer. +欢迎使用 **Agent S**,这是一个开源框架,旨在通过智能体-计算机接口(Agent-Computer Interface)实现与计算机的自主交互。我们的使命是构建智能 GUI 智能体,使其能够从过往经验中学习,并在你的计算机上自主完成复杂任务。 -Whether you're interested in AI, automation, or contributing to cutting-edge agent-based systems, we're excited to have you here! +无论你是对 AI、自动化感兴趣,还是希望为前沿的智能体系统做贡献,我们都很高兴你能来到这里! -## 🎯 Current Results +## 🎯 当前成果

Agent S3 Results

-On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, *surpassing* human-level performance on OSWorld (~72%)! +在 OSWorld 上,仅 Agent S3 在 100 步设置下即可达到 66%,已超过此前 63.4% 的最先进水平(GTA1 w/ GPT-5)。加入 Behavior Best-of-N 后,性能进一步提升至 72.6%,*超越* OSWorld 上的人类水平表现(约 72%)! -Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6% +Agent S3 还展现出强大的零样本泛化能力!在 WindowsAgentArena 上,仅使用 Agent S3 时准确率为 50.2%,通过从 3 次 rollout 中择优选择可提升至 56.6%。在 AndroidWorld 上,性能也从 68.1% 提升至 71.6% -## 🛠️ Installation & Setup +## 🛠️ 安装与配置 -### Prerequisites -- **Single Monitor**: Our agent is designed for single monitor screens -- **Security**: The agent runs Python code to control your computer - use with care -- **Supported Platforms**: Linux, Mac, and Windows +### 前置要求 +- **单显示器**:我们的智能体面向单显示器屏幕设计 +- **安全性**:该智能体会运行 Python 代码来控制你的计算机——请谨慎使用 +- **支持的平台**:Linux、Mac 和 Windows -### Installation -To install Agent S3 without cloning the repository, run +### 安装 +若无需克隆仓库即可安装 Agent S3,请运行 ```bash pip install gui-agents ``` -If you would like to test Agent S3 while making changes, clone the repository and install using +如果你想在修改代码的同时测试 Agent S3,请克隆仓库并按以下方式安装 ``` pip install -e . ``` -Don't forget to also `brew install tesseract`! Pytesseract requires this extra installation to work. +别忘了还要 `brew install tesseract`!Pytesseract 需要此额外安装才能正常工作。 -### API Configuration +### API 配置 -#### Option 1: Environment Variables -Add to your `.bashrc` (Linux) or `.zshrc` (MacOS): +#### 选项 1:环境变量 +添加到你的 `.bashrc`(Linux)或 `.zshrc`(MacOS): ```bash export OPENAI_API_KEY= export ANTHROPIC_API_KEY= export HF_TOKEN= ``` -#### Option 2: Python Script +#### 选项 2:Python 脚本 ```python import os os.environ["OPENAI_API_KEY"] = "" ``` -### Supported Models -We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See [models.md](models.md) for details. +### 支持的模型 +我们支持 Azure OpenAI、Anthropic、Gemini、Open Router 和 vLLM 推理。详见 [models.md](models.md)。 -### Grounding Models (Required) -For optimal performance, we recommend [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) hosted on Hugging Face Inference Endpoints or another provider. See [Hugging Face Inference Endpoints](https://huggingface.co/learn/cookbook/en/enterprise_dedicated_endpoints) for setup instructions. +### 定位模型(必需) +为获得最佳性能,我们推荐使用托管在 Hugging Face Inference Endpoints 或其他提供商上的 [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B)。设置说明请参阅 [Hugging Face Inference Endpoints](https://huggingface.co/learn/cookbook/en/enterprise_dedicated_endpoints)。 -## 🚀 Usage +## 🚀 用法 -> ⚡️ **Recommended Setup:** -> For the best configuration, we recommend using **OpenAI gpt-5-2025-08-07** as the main model, paired with **UI-TARS-1.5-7B** for grounding. +> ⚡️ **推荐配置:** +> 为获得最佳配置,我们建议使用 **OpenAI gpt-5-2025-08-07** 作为主模型,并搭配 **UI-TARS-1.5-7B** 进行 grounding。 ### CLI -Note, this is running Agent S3, our improved agent, without bBoN. +请注意,这里运行的是我们改进后的 Agent S3,未使用 bBoN。 -Run Agent S3 with the required parameters: +使用必需参数运行 Agent S3: ```bash agent_s \ @@ -161,8 +167,8 @@ agent_s \ --grounding_height 1080 ``` -#### Local Coding Environment (Optional) -For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment: +#### 本地编码环境(可选) +对于需要执行代码的任务(例如数据处理、文件操作、系统自动化),你可以启用本地编码环境: ```bash agent_s \ @@ -176,59 +182,59 @@ agent_s \ --enable_local_env ``` -⚠️ **WARNING**: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs. +⚠️ **警告**:本地编码环境会在你的计算机上本地执行任意 Python 和 Bash 代码。请仅在受信任的环境中、并使用受信任的输入时使用此功能。 -#### Required Parameters -- **`--provider`**: Main generation model provider (e.g., openai, anthropic, etc.) - Default: "openai" -- **`--model`**: Main generation model name (e.g., gpt-5-2025-08-07) - Default: "gpt-5-2025-08-07" -- **`--ground_provider`**: The provider for the grounding model - **Required** -- **`--ground_url`**: The URL of the grounding model - **Required** -- **`--ground_model`**: The model name for the grounding model - **Required** -- **`--grounding_width`**: Width of the output coordinate resolution from the grounding model - **Required** -- **`--grounding_height`**: Height of the output coordinate resolution from the grounding model - **Required** +#### 必需参数 +- **`--provider`**:主生成模型提供商(例如 openai、anthropic 等)- 默认值:"openai" +- **`--model`**:主生成模型名称(例如 gpt-5-2025-08-07)- 默认值:"gpt-5-2025-08-07" +- **`--ground_provider`**:接地(grounding)模型的提供商 - **必需** +- **`--ground_url`**:接地模型的 URL - **必需** +- **`--ground_model`**:接地模型的模型名称 - **必需** +- **`--grounding_width`**:接地模型输出坐标分辨率的宽度 - **必需** +- **`--grounding_height`**:接地模型输出坐标分辨率的高度 - **必需** -#### Optional Parameters -- **`--model_temperature`**: The temperature to fix all model calls to (necessary to set to 1.0 for models like o3 but can be left blank for other models) +#### 可选参数 +- **`--model_temperature`**:将所有模型调用固定为此温度(对于 o3 等模型需要设置为 1.0,但对于其他模型可以留空) -#### Grounding Model Dimensions -The grounding width and height should match the output coordinate resolution of your grounding model: -- **UI-TARS-1.5-7B**: Use `--grounding_width 1920 --grounding_height 1080` -- **UI-TARS-72B**: Use `--grounding_width 1000 --grounding_height 1000` +#### 接地模型尺寸 +接地宽度和高度应与你的接地模型的输出坐标分辨率相匹配: +- **UI-TARS-1.5-7B**:使用 `--grounding_width 1920 --grounding_height 1080` +- **UI-TARS-72B**:使用 `--grounding_width 1000 --grounding_height 1000` -#### Optional Parameters -- **`--model_url`**: Custom API URL for main generation model - Default: "" -- **`--model_api_key`**: API key for main generation model - Default: "" -- **`--ground_api_key`**: API key for grounding model endpoint - Default: "" -- **`--max_trajectory_length`**: Maximum number of image turns to keep in trajectory - Default: 8 -- **`--enable_reflection`**: Enable reflection agent to assist the worker agent - Default: True -- **`--enable_local_env`**: Enable local coding environment for code execution (WARNING: Executes arbitrary code locally) - Default: False +#### 可选参数 +- **`--model_url`**:主生成模型的自定义 API URL - 默认值:"" +- **`--model_api_key`**:主生成模型的 API 密钥 - 默认值:"" +- **`--ground_api_key`**:接地模型端点的 API 密钥 - 默认值:"" +- **`--max_trajectory_length`**:轨迹中保留的最大图像轮次数量 - 默认值:8 +- **`--enable_reflection`**:启用反思智能体(reflection agent)以辅助工作智能体 - 默认值:True +- **`--enable_local_env`**:启用用于代码执行的本地编码环境(警告:会在本地执行任意代码)- 默认值:False -#### Local Coding Environment Details -The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for: +#### 本地编码环境详情 +本地编码环境使 Agent S3 能够直接在你的计算机上执行 Python 和 Bash 代码。这对于以下场景特别有用: -- **Data Processing**: Manipulating spreadsheets, CSV files, or databases -- **File Operations**: Bulk file processing, content extraction, or file organization -- **System Automation**: Configuration changes, system setup, or automation scripts -- **Code Development**: Writing, editing, or executing code files -- **Text Processing**: Document manipulation, content editing, or formatting +- **数据处理**:操作电子表格、CSV 文件或数据库 +- **文件操作**:批量文件处理、内容提取或文件整理 +- **系统自动化**:配置更改、系统设置或自动化脚本 +- **代码开发**:编写、编辑或执行代码文件 +- **文本处理**:文档操作、内容编辑或格式化 -When enabled, the agent can use the `call_code_agent` action to execute code blocks for tasks that can be completed through programming rather than GUI interaction. +启用后,智能体可以使用 `call_code_agent` 操作来执行代码块,以完成可通过编程而非 GUI 交互完成的任务。 -**Requirements:** -- **Python**: The same Python interpreter used to run Agent S3 (automatically detected) -- **Bash**: Available at `/bin/bash` (standard on macOS and Linux) -- **System Permissions**: The agent runs with the same permissions as the user executing it +**要求:** +- **Python**:用于运行 Agent S3 的同一 Python 解释器(自动检测) +- **Bash**:位于 `/bin/bash`(在 macOS 和 Linux 上为标准配置) +- **系统权限**:智能体以执行它的用户相同的权限运行 -**Security Considerations:** -- The local environment executes arbitrary code with the same permissions as the user running the agent -- Only enable this feature in trusted environments -- Be cautious when the agent generates code for system-level operations -- Consider running in a sandboxed environment for untrusted tasks -- Bash scripts are executed with a 30-second timeout to prevent hanging processes +**安全注意事项:** +- 本地环境以运行智能体的用户相同的权限执行任意代码 +- 仅在受信任的环境中启用此功能 +- 当智能体生成用于系统级操作的代码时要谨慎 +- 对于不受信任的任务,考虑在沙盒环境中运行 +- Bash 脚本以 30 秒超时执行,以防止进程挂起 ### `gui_agents` SDK -First, we import the necessary modules. `AgentS3` is the main agent class for Agent S3. `OSWorldACI` is our grounding agent that translates agent actions into executable python code. +首先,我们导入必要的模块。`AgentS3` 是 Agent S3 的主智能体类。`OSWorldACI` 是我们的接地智能体,它将智能体动作翻译为可执行的 Python 代码。 ```python import pyautogui import io @@ -243,7 +249,7 @@ load_dotenv() current_platform = "linux" # "darwin", "windows" ``` -Next, we define our engine parameters. `engine_params` is used for the main agent, and `engine_params_for_grounding` is for grounding. For `engine_params_for_grounding`, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router. +接下来,我们定义引擎参数。`engine_params` 用于主智能体,`engine_params_for_grounding` 用于接地。对于 `engine_params_for_grounding`,我们支持 HuggingFace TGI、vLLM 和 Open Router 等自定义端点。 ```python engine_params = { @@ -276,7 +282,7 @@ engine_params_for_grounding = { } ``` -Then, we define our grounding agent and Agent S3. +然后,我们定义接地智能体和 Agent S3。 ```python # Optional: Enable local coding environment @@ -301,7 +307,7 @@ agent = AgentS3( ) ``` -Finally, let's query the agent! +最后,让我们查询智能体! ```python # Get screenshot. @@ -320,15 +326,15 @@ info, action = agent.predict(instruction=instruction, observation=obs) exec(action[0]) ``` -Refer to `gui_agents/s3/cli_app.py` for more details on how the inference loop works. +有关推理循环工作原理的更多详情,请参阅 `gui_agents/s3/cli_app.py`。 ### OSWorld -To deploy Agent S3 in OSWorld, follow the [OSWorld Deployment instructions](osworld_setup/s3/OSWorld.md). +要在 OSWorld 中部署 Agent S3,请遵循 [OSWorld 部署说明](osworld_setup/s3/OSWorld.md)。 -## 💬 Citations +## 💬 引用 -If you find this codebase useful, please cite: +如果你认为此代码库有用,请引用: ``` @misc{Agent-S3, @@ -360,6 +366,6 @@ If you find this codebase useful, please cite: } ``` -## Star History +## Star 历史 [![Star History Chart](https://api.star-history.com/svg?repos=simular-ai/Agent-S&type=Date)](https://star-history.com/#simular-ai/Agent-S&Date)