# Agent-S OpenClaw Integration

This integration enables [OpenClaw](https://github.com/openclaw/openclaw) to use [Agent-S](https://github.com/simular-ai/Agent-S) for autonomous GUI automation tasks.

## Overview

Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. This integration provides a simple wrapper that allows OpenClaw agents to invoke Agent-S for GUI automation.

## Prerequisites

### Required Software

1. **Agent-S**: Install the gui-agents package
   ```bash
   pip install gui-agents
   ```

2. **Tesseract**: Required for OCR functionality
   ```bash
   brew install tesseract  # macOS
   # or
   sudo apt install tesseract-ocr  # Linux
   ```

3. **OpenClaw**: This integration is designed to work with OpenClaw

### Required Environment Variables

You need at least one API key for your chosen provider:

- **`ANTHROPIC_API_KEY`**: For Claude models (Anthropic provider)
  ```bash
  export ANTHROPIC_API_KEY="your-api-key-here"
  ```

- **`OPENAI_API_KEY`**: For GPT models (OpenAI provider)
  ```bash
  export OPENAI_API_KEY="your-api-key-here"
  ```

- **`GEMINI_API_KEY`**: For Gemini models (Google provider)
  ```bash
  export GEMINI_API_KEY="your-api-key-here"
  ```

By default, the wrapper uses Anthropic's Claude Sonnet 4.5. You can modify `agent_s_wrapper.py` to use a different provider and model.

### Grounding Model Configuration (Required)

Agent-S requires a grounding model for visual element detection. We recommend [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B):

- **`AGENT_S_GROUND_URL`** (Required): Grounding model endpoint URL
- **`AGENT_S_GROUND_MODEL`** (Required): Model name (default: "ui-tars-1.5-7b")
- **`AGENT_S_GROUNDING_WIDTH`** (Required): Output coordinate width (default: "1920")
- **`AGENT_S_GROUNDING_HEIGHT`** (Required): Output coordinate height (default: "1080")
- **`AGENT_S_GROUND_API_KEY`** (Optional): API key for grounding endpoint

Example configuration:
```bash
export AGENT_S_GROUND_URL="http://localhost:8080"
export AGENT_S_GROUND_API_KEY="your-grounding-api-key"
export AGENT_S_GROUND_MODEL="ui-tars-1.5-7b"
export AGENT_S_GROUNDING_WIDTH="1920"
export AGENT_S_GROUNDING_HEIGHT="1080"
```

See the [Agent-S documentation](https://github.com/simular-ai/Agent-S#grounding-models-required) for details on setting up grounding models.

## Installation

1. **Clone or copy this directory** to your OpenClaw skills folder:
   ```bash
   cp -r integrations/openclaw ~/.openclaw/workspace/skills/agent-s
   ```

2. **Make scripts executable**:
   ```bash
   chmod +x ~/.openclaw/workspace/skills/agent-s/agent_s_task
   chmod +x ~/.openclaw/workspace/skills/agent-s/agent_s_wrapper.py
   ```

3. **Verify installation**:
   ```bash
   which agent_s
   # Should show the path to agent_s executable
   ```

## Usage

### From OpenClaw Agent

The OpenClaw agent can invoke Agent-S by reading the SKILL.md file and using the bash tool:

```bash
~/.openclaw/workspace/skills/agent-s/agent_s_task "Open Safari and go to google.com"
```

### From Command Line

You can test the integration directly:

```bash
# Basic usage
./agent_s_task "Open System Preferences"

# Using the Python wrapper with options
./agent_s_wrapper.py "Open TextEdit and type Hello World" --max-steps 10 --json
```

### Advanced Options

```bash
# Custom max steps
./agent_s_wrapper.py "complex task" --max-steps 30

# Disable reflection (faster but less accurate)
./agent_s_wrapper.py "simple task" --no-reflection

# Enable local code environment (WARNING: executes arbitrary code)
./agent_s_wrapper.py "task requiring code execution" --enable-local-env

# JSON output (for programmatic use)
./agent_s_wrapper.py "task" --json
```

## Testing

### Quick Test

Verify the integration works:

```bash
# Test 1: Check help
./agent_s_wrapper.py --help

# Test 2: Simple task (will actually execute)
./agent_s_task "Open Calculator"
```

### Testing with OpenClaw Agent

1. **Start OpenClaw**:
   ```bash
   openclaw
   ```

2. **Ask your agent** to use Agent-S:
   - "Can you use Agent-S to open the Calculator app?"
   - "I need you to use the Agent-S skill to open Safari and navigate to github.com"
   - "Read the Agent-S skill documentation and then use it to open System Preferences"

3. **Expected behavior**:
   - Agent reads `SKILL.md` in the skills directory
   - Agent executes `agent_s_task` command via bash tool
   - Agent-S launches and completes the GUI task
   - Results are returned to OpenClaw agent

### Verification Checklist

- [ ] `agent_s` executable is in PATH
- [ ] `ANTHROPIC_API_KEY` is set
- [ ] `AGENT_S_GROUND_URL` is set (grounding model endpoint)
- [ ] Scripts are executable
- [ ] OpenClaw agent can read skill files
- [ ] Test task executes successfully

## Configuration

All configuration is done via environment variables (see Prerequisites section above).

### Customizing the Provider and Model

By default, the wrapper uses Anthropic's Claude Sonnet 4.5. To use a different provider or model, modify the `agent_s_wrapper.py` file:

```python
# For OpenAI
cmd = [
    agent_s_path,
    "--provider", "openai",
    "--model", "gpt-5-2025-08-07",  # or other OpenAI models
    ...
]

# For Gemini
cmd = [
    agent_s_path,
    "--provider", "gemini",
    "--model", "gemini-2.0-flash-exp",  # or other Gemini models
    ...
]
```

See the [Agent-S models documentation](https://github.com/simular-ai/Agent-S/blob/main/models.md) for all supported providers and models.

### Logs

Agent-S logs are stored in: `~/workspace/Agent-S/logs/`

Check these logs if something goes wrong:
```bash
ls -lt ~/workspace/Agent-S/logs/ | head -5
tail -f ~/workspace/Agent-S/logs/debug-*.log
```

## Safety

- Agent-S has full GUI control access
- Only use for trusted automation tasks
- All actions are logged
- Can be paused with Ctrl+C and resumed with Esc
- Timeout: 10 minutes per task by default

## Troubleshooting

### Agent-S not found

Check that agent_s is in your PATH:
```bash
which agent_s
```

If not found, install gui-agents:
```bash
pip install gui-agents
```

### Permission errors

Ensure scripts are executable:
```bash
chmod +x ./agent_s_task
chmod +x ./agent_s_wrapper.py
```

### API errors

Check that your API key is set for your chosen provider:
```bash
# For Anthropic (default)
echo $ANTHROPIC_API_KEY

# For OpenAI
echo $OPENAI_API_KEY

# For Gemini
echo $GEMINI_API_KEY
```

If empty, add it to your shell profile (`~/.zshrc` or `~/.bashrc`):
```bash
export ANTHROPIC_API_KEY="your-key-here"
# or
export OPENAI_API_KEY="your-key-here"
# or
export GEMINI_API_KEY="your-key-here"

source ~/.zshrc  # or ~/.bashrc
```

### Task failures

1. Check the logs in `~/workspace/Agent-S/logs/` for detailed error messages
2. Verify grounding configuration if using custom endpoint
3. Ensure task description is clear and specific
4. Try with `--no-reflection` for simpler tasks

### Grounding model issues

If you see errors about grounding:
- Verify `AGENT_S_GROUND_URL` is accessible
- Check `AGENT_S_GROUND_API_KEY` is correct
- Ensure grounding dimensions match your model's output resolution

## Files

- **`README.md`** - This file
- **`SKILL.md`** - Skill documentation for the OpenClaw agent
- **`agent_s_wrapper.py`** - Python wrapper for invoking Agent-S
- **`agent_s_task`** - Simple bash entry point for task execution

## Support

- **Agent-S**: https://github.com/simular-ai/Agent-S
- **OpenClaw**: https://github.com/openclaw/openclaw
- **Report issues**: Use the Agent-S repository issue tracker for integration-specific issues

## License

This integration follows the same license as Agent-S. See the main repository for details.