diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..248367f --- /dev/null +++ b/README.en.md @@ -0,0 +1,365 @@ +

+ Logo Agent S: + Use Computer Like a Human +

+ +

๐Ÿ† Agent S3: First to Surpass Human Performance on OSWorld (72.60%)

+ +

  + ๐ŸŒ [S3 blog]  + ๐Ÿ“„ [S3 Paper]  + ๐ŸŽฅ [S3 Video] +

+ +

  + ๐ŸŒ [S2 blog]  + ๐Ÿ“„ [S2 Paper (COLM 2025)]  + ๐ŸŽฅ [S2 Video] +

+ +

  + ๐ŸŒ [S1 blog]  + ๐Ÿ“„ [S1 Paper (ICLR 2025)]  + ๐ŸŽฅ [S1 Video] +

+ +

  +simular-ai%2FAgent-S | Trendshift +

+ +

+ Windows + macOS + Linux + + Discord + +    + + PyPI Downloads + +

+ +
+ + Deutsch | + Espaรฑol | + franรงais | + ๆ—ฅๆœฌ่ชž | + ํ•œ๊ตญ์–ด | + Portuguรชs | + ะ ัƒััะบะธะน | + ไธญๆ–‡ +
+ +
+    +

Skip the setup? Try Agent S in Simular Cloud +

+ +## ๐Ÿฅณ Updates +- [x] **2025/12/15**: Agent S3 is the **first** to surpass human-level performance on OSWorld with an impressive score of **72.60%**! +- [x] **2025/10/02**: Released Agent S3 and its [technical paper](https://arxiv.org/abs/2510.02250), setting a new SOTA of **69.9%** on OSWorld (approaching 72% human performance), with strong generalizability on WindowsAgentArena and AndroidWorld! It is also simpler, faster, and more flexible. +- [x] **2025/08/01**: Agent S2.5 is released (gui-agents v0.2.5): simpler, better, and faster! New SOTA on [OSWorld-Verified](https://os-world.github.io)! +- [x] **2025/07/07**: The [Agent S2 paper](https://arxiv.org/abs/2504.00906) is accepted to COLM 2025! See you in Montreal! +- [x] **2025/04/27**: The Agent S paper won the Best Paper Award ๐Ÿ† at ICLR 2025 Agentic AI for Science Workshop! +- [x] **2025/04/01**: Released the [Agent S2 paper](https://arxiv.org/abs/2504.00906) with new SOTA results on OSWorld, WindowsAgentArena, and AndroidWorld! +- [x] **2025/03/12**: Released Agent S2 along with v0.2.0 of [gui-agents](https://github.com/simular-ai/Agent-S), the new state-of-the-art for computer use agents (CUA), outperforming OpenAI's CUA/Operator and Anthropic's Claude 3.7 Sonnet Computer-Use! +- [x] **2025/01/22**: The [Agent S paper](https://arxiv.org/abs/2410.08164) is accepted to ICLR 2025! +- [x] **2025/01/21**: Released v0.1.2 of [gui-agents](https://github.com/simular-ai/Agent-S) library, with support for Linux and Windows! +- [x] **2024/12/05**: Released v0.1.0 of [gui-agents](https://github.com/simular-ai/Agent-S) library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease! +- [x] **2024/10/10**: Released the [Agent S paper](https://arxiv.org/abs/2410.08164) and codebase! + +## Table of Contents + +1. [๐Ÿ’ก Introduction](#-introduction) +2. [๐ŸŽฏ Current Results](#-current-results) +3. [๐Ÿ› ๏ธ Installation & Setup](#%EF%B8%8F-installation--setup) +4. [๐Ÿš€ Usage](#-usage) +5. [๐Ÿค Acknowledgements](#-acknowledgements) +6. [๐Ÿ’ฌ Citation](#-citation) + +## ๐Ÿ’ก Introduction + +Welcome to **Agent S**, an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer. + +Whether you're interested in AI, automation, or contributing to cutting-edge agent-based systems, we're excited to have you here! + +## ๐ŸŽฏ Current Results + +

+ Agent S3 Results +

+ +On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, *surpassing* human-level performance on OSWorld (~72%)! + +Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6% + +## ๐Ÿ› ๏ธ Installation & Setup + +### Prerequisites +- **Single Monitor**: Our agent is designed for single monitor screens +- **Security**: The agent runs Python code to control your computer - use with care +- **Supported Platforms**: Linux, Mac, and Windows + + +### Installation +To install Agent S3 without cloning the repository, run +```bash +pip install gui-agents +``` +If you would like to test Agent S3 while making changes, clone the repository and install using +``` +pip install -e . +``` + +Don't forget to also `brew install tesseract`! Pytesseract requires this extra installation to work. + +### API Configuration + +#### Option 1: Environment Variables +Add to your `.bashrc` (Linux) or `.zshrc` (MacOS): +```bash +export OPENAI_API_KEY= +export ANTHROPIC_API_KEY= +export HF_TOKEN= +``` + +#### Option 2: Python Script +```python +import os +os.environ["OPENAI_API_KEY"] = "" +``` + +### Supported Models +We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See [models.md](models.md) for details. + +### Grounding Models (Required) +For optimal performance, we recommend [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) hosted on Hugging Face Inference Endpoints or another provider. See [Hugging Face Inference Endpoints](https://huggingface.co/learn/cookbook/en/enterprise_dedicated_endpoints) for setup instructions. + +## ๐Ÿš€ Usage + + +> โšก๏ธ **Recommended Setup:** +> For the best configuration, we recommend using **OpenAI gpt-5-2025-08-07** as the main model, paired with **UI-TARS-1.5-7B** for grounding. + + +### CLI + +Note, this is running Agent S3, our improved agent, without bBoN. + +Run Agent S3 with the required parameters: + +```bash +agent_s \ + --provider openai \ + --model gpt-5-2025-08-07 \ + --ground_provider huggingface \ + --ground_url http://localhost:8080 \ + --ground_model ui-tars-1.5-7b \ + --grounding_width 1920 \ + --grounding_height 1080 +``` + +#### Local Coding Environment (Optional) +For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment: + +```bash +agent_s \ + --provider openai \ + --model gpt-5-2025-08-07 \ + --ground_provider huggingface \ + --ground_url http://localhost:8080 \ + --ground_model ui-tars-1.5-7b \ + --grounding_width 1920 \ + --grounding_height 1080 \ + --enable_local_env +``` + +โš ๏ธ **WARNING**: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs. + +#### Required Parameters +- **`--provider`**: Main generation model provider (e.g., openai, anthropic, etc.) - Default: "openai" +- **`--model`**: Main generation model name (e.g., gpt-5-2025-08-07) - Default: "gpt-5-2025-08-07" +- **`--ground_provider`**: The provider for the grounding model - **Required** +- **`--ground_url`**: The URL of the grounding model - **Required** +- **`--ground_model`**: The model name for the grounding model - **Required** +- **`--grounding_width`**: Width of the output coordinate resolution from the grounding model - **Required** +- **`--grounding_height`**: Height of the output coordinate resolution from the grounding model - **Required** + +#### Optional Parameters +- **`--model_temperature`**: The temperature to fix all model calls to (necessary to set to 1.0 for models like o3 but can be left blank for other models) + +#### Grounding Model Dimensions +The grounding width and height should match the output coordinate resolution of your grounding model: +- **UI-TARS-1.5-7B**: Use `--grounding_width 1920 --grounding_height 1080` +- **UI-TARS-72B**: Use `--grounding_width 1000 --grounding_height 1000` + +#### Optional Parameters +- **`--model_url`**: Custom API URL for main generation model - Default: "" +- **`--model_api_key`**: API key for main generation model - Default: "" +- **`--ground_api_key`**: API key for grounding model endpoint - Default: "" +- **`--max_trajectory_length`**: Maximum number of image turns to keep in trajectory - Default: 8 +- **`--enable_reflection`**: Enable reflection agent to assist the worker agent - Default: True +- **`--enable_local_env`**: Enable local coding environment for code execution (WARNING: Executes arbitrary code locally) - Default: False + +#### Local Coding Environment Details +The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for: + +- **Data Processing**: Manipulating spreadsheets, CSV files, or databases +- **File Operations**: Bulk file processing, content extraction, or file organization +- **System Automation**: Configuration changes, system setup, or automation scripts +- **Code Development**: Writing, editing, or executing code files +- **Text Processing**: Document manipulation, content editing, or formatting + +When enabled, the agent can use the `call_code_agent` action to execute code blocks for tasks that can be completed through programming rather than GUI interaction. + +**Requirements:** +- **Python**: The same Python interpreter used to run Agent S3 (automatically detected) +- **Bash**: Available at `/bin/bash` (standard on macOS and Linux) +- **System Permissions**: The agent runs with the same permissions as the user executing it + +**Security Considerations:** +- The local environment executes arbitrary code with the same permissions as the user running the agent +- Only enable this feature in trusted environments +- Be cautious when the agent generates code for system-level operations +- Consider running in a sandboxed environment for untrusted tasks +- Bash scripts are executed with a 30-second timeout to prevent hanging processes + +### `gui_agents` SDK + +First, we import the necessary modules. `AgentS3` is the main agent class for Agent S3. `OSWorldACI` is our grounding agent that translates agent actions into executable python code. +```python +import pyautogui +import io +from gui_agents.s3.agents.agent_s import AgentS3 +from gui_agents.s3.agents.grounding import OSWorldACI +from gui_agents.s3.utils.local_env import LocalEnv # Optional: for local coding environment + +# Load in your API keys. +from dotenv import load_dotenv +load_dotenv() + +current_platform = "linux" # "darwin", "windows" +``` + +Next, we define our engine parameters. `engine_params` is used for the main agent, and `engine_params_for_grounding` is for grounding. For `engine_params_for_grounding`, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router. + +```python +engine_params = { + "engine_type": provider, + "model": model, + "base_url": model_url, # Optional + "api_key": model_api_key, # Optional + "temperature": model_temperature # Optional +} + +# Load the grounding engine from a custom endpoint +ground_provider = "" +ground_url = "" +ground_model = "" +ground_api_key = "" + +# Set grounding dimensions based on your model's output coordinate resolution +# UI-TARS-1.5-7B: grounding_width=1920, grounding_height=1080 +# UI-TARS-72B: grounding_width=1000, grounding_height=1000 +grounding_width = 1920 # Width of output coordinate resolution +grounding_height = 1080 # Height of output coordinate resolution + +engine_params_for_grounding = { + "engine_type": ground_provider, + "model": ground_model, + "base_url": ground_url, + "api_key": ground_api_key, # Optional + "grounding_width": grounding_width, + "grounding_height": grounding_height, +} +``` + +Then, we define our grounding agent and Agent S3. + +```python +# Optional: Enable local coding environment +enable_local_env = False # Set to True to enable local code execution +local_env = LocalEnv() if enable_local_env else None + +grounding_agent = OSWorldACI( + env=local_env, # Pass local_env for code execution capability + platform=current_platform, + engine_params_for_generation=engine_params, + engine_params_for_grounding=engine_params_for_grounding, + width=1920, # Optional: screen width + height=1080 # Optional: screen height +) + +agent = AgentS3( + engine_params, + grounding_agent, + platform=current_platform, + max_trajectory_length=8, # Optional: maximum image turns to keep + enable_reflection=True # Optional: enable reflection agent +) +``` + +Finally, let's query the agent! + +```python +# Get screenshot. +screenshot = pyautogui.screenshot() +buffered = io.BytesIO() +screenshot.save(buffered, format="PNG") +screenshot_bytes = buffered.getvalue() + +obs = { + "screenshot": screenshot_bytes, +} + +instruction = "Close VS Code" +info, action = agent.predict(instruction=instruction, observation=obs) + +exec(action[0]) +``` + +Refer to `gui_agents/s3/cli_app.py` for more details on how the inference loop works. + +### OSWorld + +To deploy Agent S3 in OSWorld, follow the [OSWorld Deployment instructions](osworld_setup/s3/OSWorld.md). + +## ๐Ÿ’ฌ Citations + +If you find this codebase useful, please cite: + +``` +@misc{Agent-S3, + title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, + author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang}, + year={2025}, + eprint={2510.02250}, + archivePrefix={arXiv}, + primaryClass={cs.AI}, + url={https://arxiv.org/abs/2510.02250}, +} + +@misc{Agent-S2, + title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, + author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang}, + year={2025}, + eprint={2504.00906}, + archivePrefix={arXiv}, + primaryClass={cs.AI}, + url={https://arxiv.org/abs/2504.00906}, +} + +@inproceedings{Agent-S, + title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}}, + author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang}, + booktitle={International Conference on Learning Representations (ICLR)}, + year={2025}, + url={https://arxiv.org/abs/2410.08164} +} +``` + +## Star History + +[![Star History Chart](https://api.star-history.com/svg?repos=simular-ai/Agent-S&type=Date)](https://star-history.com/#simular-ai/Agent-S&Date)