chore: import upstream snapshot with attribution

2026-07-13 12:23:35 +08:00
commit c8c954c85d
127 changed files with 22519 additions and 0 deletions
@@ -0,0 +1,289 @@
+# Agent-S OpenClaw Integration
+
+This integration enables [OpenClaw](https://github.com/openclaw/openclaw) to use [Agent-S](https://github.com/simular-ai/Agent-S) for autonomous GUI automation tasks.
+
+## Overview
+
+Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. This integration provides a simple wrapper that allows OpenClaw agents to invoke Agent-S for GUI automation.
+
+## Prerequisites
+
+### Required Software
+
+1. **Agent-S**: Install the gui-agents package
+   ```bash
+   pip install gui-agents
+   ```
+
+2. **Tesseract**: Required for OCR functionality
+   ```bash
+   brew install tesseract  # macOS
+   # or
+   sudo apt install tesseract-ocr  # Linux
+   ```
+
+3. **OpenClaw**: This integration is designed to work with OpenClaw
+
+### Required Environment Variables
+
+You need at least one API key for your chosen provider:
+
+- **`ANTHROPIC_API_KEY`**: For Claude models (Anthropic provider)
+  ```bash
+  export ANTHROPIC_API_KEY="your-api-key-here"
+  ```
+
+- **`OPENAI_API_KEY`**: For GPT models (OpenAI provider)
+  ```bash
+  export OPENAI_API_KEY="your-api-key-here"
+  ```
+
+- **`GEMINI_API_KEY`**: For Gemini models (Google provider)
+  ```bash
+  export GEMINI_API_KEY="your-api-key-here"
+  ```
+
+By default, the wrapper uses Anthropic's Claude Sonnet 4.5. You can modify `agent_s_wrapper.py` to use a different provider and model.
+
+### Grounding Model Configuration (Required)
+
+Agent-S requires a grounding model for visual element detection. We recommend [UI-TARS-1.5-7B](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B):
+
+- **`AGENT_S_GROUND_URL`** (Required): Grounding model endpoint URL
+- **`AGENT_S_GROUND_MODEL`** (Required): Model name (default: "ui-tars-1.5-7b")
+- **`AGENT_S_GROUNDING_WIDTH`** (Required): Output coordinate width (default: "1920")
+- **`AGENT_S_GROUNDING_HEIGHT`** (Required): Output coordinate height (default: "1080")
+- **`AGENT_S_GROUND_API_KEY`** (Optional): API key for grounding endpoint
+
+Example configuration:
+```bash
+export AGENT_S_GROUND_URL="http://localhost:8080"
+export AGENT_S_GROUND_API_KEY="your-grounding-api-key"
+export AGENT_S_GROUND_MODEL="ui-tars-1.5-7b"
+export AGENT_S_GROUNDING_WIDTH="1920"
+export AGENT_S_GROUNDING_HEIGHT="1080"
+```
+
+See the [Agent-S documentation](https://github.com/simular-ai/Agent-S#grounding-models-required) for details on setting up grounding models.
+
+## Installation
+
+1. **Clone or copy this directory** to your OpenClaw skills folder:
+   ```bash
+   cp -r integrations/openclaw ~/.openclaw/workspace/skills/agent-s
+   ```
+
+2. **Make scripts executable**:
+   ```bash
+   chmod +x ~/.openclaw/workspace/skills/agent-s/agent_s_task
+   chmod +x ~/.openclaw/workspace/skills/agent-s/agent_s_wrapper.py
+   ```
+
+3. **Verify installation**:
+   ```bash
+   which agent_s
+   # Should show the path to agent_s executable
+   ```
+
+## Usage
+
+### From OpenClaw Agent
+
+The OpenClaw agent can invoke Agent-S by reading the SKILL.md file and using the bash tool:
+
+```bash
+~/.openclaw/workspace/skills/agent-s/agent_s_task "Open Safari and go to google.com"
+```
+
+### From Command Line
+
+You can test the integration directly:
+
+```bash
+# Basic usage
+./agent_s_task "Open System Preferences"
+
+# Using the Python wrapper with options
+./agent_s_wrapper.py "Open TextEdit and type Hello World" --max-steps 10 --json
+```
+
+### Advanced Options
+
+```bash
+# Custom max steps
+./agent_s_wrapper.py "complex task" --max-steps 30
+
+# Disable reflection (faster but less accurate)
+./agent_s_wrapper.py "simple task" --no-reflection
+
+# Enable local code environment (WARNING: executes arbitrary code)
+./agent_s_wrapper.py "task requiring code execution" --enable-local-env
+
+# JSON output (for programmatic use)
+./agent_s_wrapper.py "task" --json
+```
+
+## Testing
+
+### Quick Test
+
+Verify the integration works:
+
+```bash
+# Test 1: Check help
+./agent_s_wrapper.py --help
+
+# Test 2: Simple task (will actually execute)
+./agent_s_task "Open Calculator"
+```
+
+### Testing with OpenClaw Agent
+
+1. **Start OpenClaw**:
+   ```bash
+   openclaw
+   ```
+
+2. **Ask your agent** to use Agent-S:
+   - "Can you use Agent-S to open the Calculator app?"
+   - "I need you to use the Agent-S skill to open Safari and navigate to github.com"
+   - "Read the Agent-S skill documentation and then use it to open System Preferences"
+
+3. **Expected behavior**:
+   - Agent reads `SKILL.md` in the skills directory
+   - Agent executes `agent_s_task` command via bash tool
+   - Agent-S launches and completes the GUI task
+   - Results are returned to OpenClaw agent
+
+### Verification Checklist
+
+- [ ] `agent_s` executable is in PATH
+- [ ] `ANTHROPIC_API_KEY` is set
+- [ ] `AGENT_S_GROUND_URL` is set (grounding model endpoint)
+- [ ] Scripts are executable
+- [ ] OpenClaw agent can read skill files
+- [ ] Test task executes successfully
+
+## Configuration
+
+All configuration is done via environment variables (see Prerequisites section above).
+
+### Customizing the Provider and Model
+
+By default, the wrapper uses Anthropic's Claude Sonnet 4.5. To use a different provider or model, modify the `agent_s_wrapper.py` file:
+
+```python
+# For OpenAI
+cmd = [
+    agent_s_path,
+    "--provider", "openai",
+    "--model", "gpt-5-2025-08-07",  # or other OpenAI models
+    ...
+]
+
+# For Gemini
+cmd = [
+    agent_s_path,
+    "--provider", "gemini",
+    "--model", "gemini-2.0-flash-exp",  # or other Gemini models
+    ...
+]
+```
+
+See the [Agent-S models documentation](https://github.com/simular-ai/Agent-S/blob/main/models.md) for all supported providers and models.
+
+### Logs
+
+Agent-S logs are stored in: `~/workspace/Agent-S/logs/`
+
+Check these logs if something goes wrong:
+```bash
+ls -lt ~/workspace/Agent-S/logs/ | head -5
+tail -f ~/workspace/Agent-S/logs/debug-*.log
+```
+
+## Safety
+
+- Agent-S has full GUI control access
+- Only use for trusted automation tasks
+- All actions are logged
+- Can be paused with Ctrl+C and resumed with Esc
+- Timeout: 10 minutes per task by default
+
+## Troubleshooting
+
+### Agent-S not found
+
+Check that agent_s is in your PATH:
+```bash
+which agent_s
+```
+
+If not found, install gui-agents:
+```bash
+pip install gui-agents
+```
+
+### Permission errors
+
+Ensure scripts are executable:
+```bash
+chmod +x ./agent_s_task
+chmod +x ./agent_s_wrapper.py
+```
+
+### API errors
+
+Check that your API key is set for your chosen provider:
+```bash
+# For Anthropic (default)
+echo $ANTHROPIC_API_KEY
+
+# For OpenAI
+echo $OPENAI_API_KEY
+
+# For Gemini
+echo $GEMINI_API_KEY
+```
+
+If empty, add it to your shell profile (`~/.zshrc` or `~/.bashrc`):
+```bash
+export ANTHROPIC_API_KEY="your-key-here"
+# or
+export OPENAI_API_KEY="your-key-here"
+# or
+export GEMINI_API_KEY="your-key-here"
+
+source ~/.zshrc  # or ~/.bashrc
+```
+
+### Task failures
+
+1. Check the logs in `~/workspace/Agent-S/logs/` for detailed error messages
+2. Verify grounding configuration if using custom endpoint
+3. Ensure task description is clear and specific
+4. Try with `--no-reflection` for simpler tasks
+
+### Grounding model issues
+
+If you see errors about grounding:
+- Verify `AGENT_S_GROUND_URL` is accessible
+- Check `AGENT_S_GROUND_API_KEY` is correct
+- Ensure grounding dimensions match your model's output resolution
+
+## Files
+
+- **`README.md`** - This file
+- **`SKILL.md`** - Skill documentation for the OpenClaw agent
+- **`agent_s_wrapper.py`** - Python wrapper for invoking Agent-S
+- **`agent_s_task`** - Simple bash entry point for task execution
+
+## Support
+
+- **Agent-S**: https://github.com/simular-ai/Agent-S
+- **OpenClaw**: https://github.com/openclaw/openclaw
+- **Report issues**: Use the Agent-S repository issue tracker for integration-specific issues
+
+## License
+
+This integration follows the same license as Agent-S. See the main repository for details.
@@ -0,0 +1,102 @@
+# Agent-S - Autonomous GUI Agent
+
+Agent-S is a powerful autonomous agent that can control your computer's graphical interface to complete complex tasks. It combines vision and action understanding to interact with any GUI element.
+
+## What It Does
+
+Agent-S can:
+- Navigate and interact with desktop applications
+- Fill forms, click buttons, and manipulate GUI elements
+- Complete multi-step workflows across different applications
+- Take screenshots and understand visual interfaces
+- Execute complex GUI automation tasks autonomously
+
+## When to Use
+
+Use Agent-S when you need to:
+- Automate GUI-based tasks that don't have CLI alternatives
+- Interact with desktop applications programmatically
+- Complete workflows that require visual understanding
+- Perform actions across multiple applications
+- Test GUI interfaces
+
+## How to Invoke
+
+Call the Agent-S wrapper via bash from the OpenClaw skills directory:
+
+```bash
+./agent_s_task "task description"
+```
+
+Or if installed in the default OpenClaw skills location:
+
+```bash
+~/.openclaw/workspace/skills/agent-s/agent_s_task "task description"
+```
+
+**Note**: Agent-S tasks can take 2-5 minutes to complete (up to 15 steps by default). The wrapper will wait for completion.
+
+## Parameters
+
+- `task` (required): Natural language description of the GUI task to complete
+- `max_steps` (optional): Maximum steps the agent can take (default: 15)
+- `enable_reflection` (optional): Enable self-reflection for better performance (default: true)
+
+## Examples
+
+```python
+# Basic navigation
+agent_s_task(task="Open Finder and create a new folder called 'Reports'")
+
+# Form filling
+agent_s_task(task="Open TextEdit, create a new document, and type 'Hello World'")
+
+# Multi-step workflows
+agent_s_task(task="Open Chrome, search for 'Python tutorials', and bookmark the first result")
+
+# Application interaction
+agent_s_task(task="Open System Preferences and check the current display resolution")
+```
+
+## Technical Details
+
+Agent-S uses:
+- **Main Model**: Claude Sonnet 4.5 for reasoning and planning
+- **Grounding Model**: UI-TARS-1.5-7B for visual grounding and coordinate extraction
+- **Screen Resolution**: Automatically scaled to 2400px max dimension
+- **Platform Support**: macOS, Linux, Windows
+
+## Safety
+
+- Agent-S has full GUI control - only use for trusted tasks
+- The agent will pause on Ctrl+C and can be resumed with Esc
+- Each action is logged to `~/workspace/Agent-S/logs/`
+- Tasks timeout after 15 steps by default
+
+## Configuration
+
+Agent-S requires configuration via environment variables:
+
+**Required:**
+- `ANTHROPIC_API_KEY`: API key for Claude model
+- `AGENT_S_GROUND_URL`: Grounding model endpoint URL
+- `AGENT_S_GROUND_MODEL`: Grounding model name (default: ui-tars-1.5-7b)
+- `AGENT_S_GROUNDING_WIDTH`: Output width (default: 1920)
+- `AGENT_S_GROUNDING_HEIGHT`: Output height (default: 1080)
+
+**Optional:**
+- `AGENT_S_GROUND_API_KEY`: API key for grounding endpoint
+
+See the README.md in this directory for detailed setup instructions.
+
+## Limitations
+
+- Cannot interact with system-level dialogs requiring admin approval
+- Performance depends on screen resolution and GUI complexity
+- Some applications may have accessibility restrictions
+- Voice/audio commands are not supported
+
+## Source
+
+Agent-S GitHub: https://github.com/simular-ai/Agent-S
+Installation: `pip install gui-agents`
@@ -0,0 +1,15 @@
+#!/bin/bash
+# Agent-S Task Executor for OpenClaw
+# Usage: agent_s_task "task description"
+
+TASK="$1"
+
+if [ -z "$TASK" ]; then
+    echo "Error: Task description required"
+    echo "Usage: agent_s_task \"task description\""
+    exit 1
+fi
+
+# Execute the Python wrapper using relative path
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec "$SCRIPT_DIR/agent_s_wrapper.py" "$TASK"
@@ -0,0 +1,181 @@
+#!/usr/bin/env python3
+"""
+Agent-S Wrapper for OpenClaw Integration
+
+This script provides a simple interface for OpenClaw to invoke Agent-S
+for GUI automation tasks.
+"""
+
+import argparse
+import json
+import subprocess
+import sys
+import os
+import shutil
+
+
+def run_agent_s(task, max_steps=15, enable_reflection=True, enable_local_env=False):
+    """
+    Execute an Agent-S task and return the result.
+
+    Args:
+        task: Natural language task description
+        max_steps: Maximum number of steps (default: 15)
+        enable_reflection: Enable reflection agent (default: True)
+        enable_local_env: Enable local code execution (default: False, WARNING: executes arbitrary code)
+
+    Returns:
+        Dictionary with status and message
+    """
+
+    # Path to agent_s executable - auto-detect or use environment variable
+    agent_s_path = os.environ.get("AGENT_S_PATH") or shutil.which("agent_s")
+    if not agent_s_path:
+        return {
+            "status": "error",
+            "message": "agent_s not found in PATH. Install with: pip install gui-agents",
+            "error": "agent_s executable not found"
+        }
+
+    # Build base command
+    cmd = [
+        agent_s_path,
+        "--provider", "anthropic",
+        "--model", "claude-sonnet-4-5",
+        "--model_temperature", "1.0",
+        "--max_trajectory_length", str(max_steps),
+        "--task", task,
+    ]
+    
+    # Add optional grounding configuration from environment variables
+    ground_url = os.environ.get("AGENT_S_GROUND_URL")
+    ground_api_key = os.environ.get("AGENT_S_GROUND_API_KEY")
+    ground_model = os.environ.get("AGENT_S_GROUND_MODEL", "ui-tars-1.5-7b")
+    grounding_width = os.environ.get("AGENT_S_GROUNDING_WIDTH", "1920")
+    grounding_height = os.environ.get("AGENT_S_GROUNDING_HEIGHT", "1080")
+    
+    if ground_url:
+        cmd.extend(["--ground_provider", "huggingface"])
+        cmd.extend(["--ground_url", ground_url])
+        cmd.extend(["--ground_model", ground_model])
+        cmd.extend(["--grounding_width", grounding_width])
+        cmd.extend(["--grounding_height", grounding_height])
+        if ground_api_key:
+            cmd.extend(["--ground_api_key", ground_api_key])
+
+    if enable_reflection:
+        cmd.append("--enable_reflection")
+
+    if enable_local_env:
+        cmd.append("--enable_local_env")
+
+    try:
+        # Run Agent-S
+        print(f"Starting Agent-S with task: {task}", file=sys.stderr)
+        print(f"Command: {' '.join(cmd)}", file=sys.stderr)
+
+        # Agent-S can take 2-5 minutes for complex tasks (15 steps max)
+        # Don't capture output - let it stream to allow real-time GUI interaction
+        result = subprocess.run(
+            cmd,
+            capture_output=False,  # Changed: let output stream
+            text=True,
+            timeout=600  # 10 minute timeout
+        )
+
+        if result.returncode == 0:
+            return {
+                "status": "success",
+                "message": f"Agent-S completed the task: {task}",
+                "logs_directory": os.path.expanduser("~/workspace/Agent-S/logs/"),
+                "note": "Output was streamed to terminal. Check logs for details."
+            }
+        else:
+            return {
+                "status": "error",
+                "message": f"Agent-S failed with return code {result.returncode}",
+                "logs_directory": os.path.expanduser("~/workspace/Agent-S/logs/"),
+                "note": "Check logs for error details."
+            }
+
+    except subprocess.TimeoutExpired:
+        return {
+            "status": "error",
+            "message": f"Agent-S timed out after 10 minutes for task: {task}",
+            "error": "Timeout expired"
+        }
+
+    except Exception as e:
+        return {
+            "status": "error",
+            "message": f"Failed to execute Agent-S: {str(e)}",
+            "error": str(e)
+        }
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="OpenClaw wrapper for Agent-S GUI automation"
+    )
+    parser.add_argument(
+        "task",
+        type=str,
+        help="Natural language description of the GUI task to perform"
+    )
+    parser.add_argument(
+        "--max-steps",
+        type=int,
+        default=15,
+        help="Maximum number of agent steps (default: 15)"
+    )
+    parser.add_argument(
+        "--enable-reflection",
+        action="store_true",
+        default=True,
+        help="Enable reflection agent for better performance"
+    )
+    parser.add_argument(
+        "--no-reflection",
+        action="store_false",
+        dest="enable_reflection",
+        help="Disable reflection agent"
+    )
+    parser.add_argument(
+        "--enable-local-env",
+        action="store_true",
+        default=False,
+        help="Enable local code execution (WARNING: executes arbitrary code)"
+    )
+    parser.add_argument(
+        "--json",
+        action="store_true",
+        help="Output result as JSON"
+    )
+
+    args = parser.parse_args()
+
+    # Execute Agent-S task
+    result = run_agent_s(
+        task=args.task,
+        max_steps=args.max_steps,
+        enable_reflection=args.enable_reflection,
+        enable_local_env=args.enable_local_env
+    )
+
+    # Output result
+    if args.json:
+        print(json.dumps(result, indent=2))
+    else:
+        if result["status"] == "success":
+            print(f"✓ {result['message']}")
+            if result.get("output"):
+                print(f"\nOutput:\n{result['output']}")
+        else:
+            print(f"✗ {result['message']}")
+            if result.get("error"):
+                print(f"\nError:\n{result['error']}", file=sys.stderr)
+            sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()