chore: import upstream snapshot with attribution

2026-07-13 13:03:19 +08:00
commit 91e75e620b
3227 changed files with 1307078 additions and 0 deletions
@@ -0,0 +1,76 @@
+# Computer Agent Benchmarks
+
+This directory contains benchmarks designed to test agent providers in the Computer Agent SDK against reference agent implementations.
+
+## Overview
+
+The benchmark system evaluates models on GUI grounding tasks, specifically click prediction accuracy. It supports both:
+
+- **Computer Agent SDK providers** (using model strings like `"huggingface-local/HelloKKMe/GTA1-7B"`)
+- **Reference agent implementations** (custom model classes implementing the `ModelProtocol`)
+
+## Available Benchmarks
+
+### 1. ScreenSpot-v2 (`ss-v2.py`)
+
+- **Dataset**: ScreenSpot-v2 (click-only GUI grounding)
+- **Format**: Standard resolution screenshots
+- **Task**: Predict click coordinates given an instruction and image
+- **Metrics**: Accuracy, Error Rate, Timing, VRAM usage
+
+### 2. ScreenSpot-Pro (`ss-pro.py`)
+
+- **Dataset**: ScreenSpot-Pro (high-resolution click-only GUI grounding)
+- **Format**: High-resolution screenshots
+- **Task**: Predict click coordinates given an instruction and image
+- **Metrics**: Accuracy, Error Rate, Timing, VRAM usage
+
+### 3. Interactive Testing (`interactive.py`)
+
+- **Real-time testing**: Take screenshots and visualize model predictions
+- **Commands**:
+  - Type instruction → test all models on last screenshot
+  - `screenshot` → take screenshot
+  - `models` → list available models
+  - `quit`/`exit` → exit tool
+- **Output**: Visual predictions with crosshairs for each model
+
+## Running Benchmarks
+
+### 1. Configure Models
+
+Edit `utils.py` to specify which models you want to test in `get_available_models()`.
+
+### 2. Run Benchmark
+
+```bash
+# ScreenSpot-v2 benchmark
+python ss-v2.py --samples 50
+
+# ScreenSpot-Pro benchmark
+python ss-pro.py --samples 50
+
+# Interactive testing
+python interactive.py
+```
+
+## Output
+
+### Console Output
+
+```
+Model Results:
+  Accuracy: 85.50% (171/200)
+  Avg Time: 1.23s (0.89s - 2.45s)
+  VRAM Usage: 4.5GB (max) / 3.4GB (avg)
+```
+
+### Generated Files
+
+- **Markdown Report**: `*_results.md` with detailed results tables
+- **Visualizations**: `output/` directory with prediction visualizations
+- **Interactive Output**: `interactive_output/` for interactive session results
+
+## Contributing
+
+To add a new reference model, follow the instructions in [contrib.md](contrib.md).