chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,102 @@
|
||||
# Agent-S - Autonomous GUI Agent
|
||||
|
||||
Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. It combines vision and action understanding to interact with any GUI element.
|
||||
|
||||
## What It Does
|
||||
|
||||
Agent-S can:
|
||||
- Navigate and interact with desktop applications
|
||||
- Fill forms, click buttons, and manipulate GUI elements
|
||||
- Complete multi-step workflows across different applications
|
||||
- Take screenshots and understand visual interfaces
|
||||
- Execute complex GUI automation tasks autonomously
|
||||
|
||||
## When to Use
|
||||
|
||||
Use Agent-S when you need to:
|
||||
- Automate GUI-based tasks that don't have CLI alternatives
|
||||
- Interact with desktop applications programmatically
|
||||
- Complete workflows that require visual understanding
|
||||
- Perform actions across multiple applications
|
||||
- Test GUI interfaces
|
||||
|
||||
## How to Invoke
|
||||
|
||||
Call the Agent-S wrapper via bash from the OpenClaw skills directory:
|
||||
|
||||
```bash
|
||||
./agent_s_task "task description"
|
||||
```
|
||||
|
||||
Or if installed in the default OpenClaw skills location:
|
||||
|
||||
```bash
|
||||
~/.openclaw/workspace/skills/agent-s/agent_s_task "task description"
|
||||
```
|
||||
|
||||
**Note**: Agent-S tasks can take 2-5 minutes to complete (up to 15 steps by default). The wrapper will wait for completion.
|
||||
|
||||
## Parameters
|
||||
|
||||
- `task` (required): Natural language description of the GUI task to complete
|
||||
- `max_steps` (optional): Maximum steps the agent can take (default: 15)
|
||||
- `enable_reflection` (optional): Enable self-reflection for better performance (default: true)
|
||||
|
||||
## Examples
|
||||
|
||||
```python
|
||||
# Basic navigation
|
||||
agent_s_task(task="Open Finder and create a new folder called 'Reports'")
|
||||
|
||||
# Form filling
|
||||
agent_s_task(task="Open TextEdit, create a new document, and type 'Hello World'")
|
||||
|
||||
# Multi-step workflows
|
||||
agent_s_task(task="Open Chrome, search for 'Python tutorials', and bookmark the first result")
|
||||
|
||||
# Application interaction
|
||||
agent_s_task(task="Open System Preferences and check the current display resolution")
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
Agent-S uses:
|
||||
- **Main Model**: Claude Sonnet 4.5 for reasoning and planning
|
||||
- **Grounding Model**: UI-TARS-1.5-7B for visual grounding and coordinate extraction
|
||||
- **Screen Resolution**: Automatically scaled to 2400px max dimension
|
||||
- **Platform Support**: macOS, Linux, Windows
|
||||
|
||||
## Safety
|
||||
|
||||
- Agent-S has full GUI control - only use for trusted tasks
|
||||
- The agent will pause on Ctrl+C and can be resumed with Esc
|
||||
- Each action is logged to `~/workspace/Agent-S/logs/`
|
||||
- Tasks timeout after 15 steps by default
|
||||
|
||||
## Configuration
|
||||
|
||||
Agent-S requires configuration via environment variables:
|
||||
|
||||
**Required:**
|
||||
- `ANTHROPIC_API_KEY`: API key for Claude model
|
||||
- `AGENT_S_GROUND_URL`: Grounding model endpoint URL
|
||||
- `AGENT_S_GROUND_MODEL`: Grounding model name (default: ui-tars-1.5-7b)
|
||||
- `AGENT_S_GROUNDING_WIDTH`: Output width (default: 1920)
|
||||
- `AGENT_S_GROUNDING_HEIGHT`: Output height (default: 1080)
|
||||
|
||||
**Optional:**
|
||||
- `AGENT_S_GROUND_API_KEY`: API key for grounding endpoint
|
||||
|
||||
See the README.md in this directory for detailed setup instructions.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Cannot interact with system-level dialogs requiring admin approval
|
||||
- Performance depends on screen resolution and GUI complexity
|
||||
- Some applications may have accessibility restrictions
|
||||
- Voice/audio commands are not supported
|
||||
|
||||
## Source
|
||||
|
||||
Agent-S GitHub: https://github.com/simular-ai/Agent-S
|
||||
Installation: `pip install gui-agents`
|
||||
Reference in New Issue
Block a user