chore: import upstream snapshot with attribution

libs/python/agent/benchmarks/.gitignore

@@ -0,0 +1,3 @@
+output/
+interactive_output/
+*_results.md

libs/python/agent/benchmarks/README.md

@@ -0,0 +1,76 @@
+# Computer Agent Benchmarks
+
+This directory contains benchmarks designed to test agent providers in the Computer Agent SDK against reference agent implementations.
+
+## Overview
+
+The benchmark system evaluates models on GUI grounding tasks, specifically click prediction accuracy. It supports both:
+
+- **Computer Agent SDK providers** (using model strings like `"huggingface-local/HelloKKMe/GTA1-7B"`)
+- **Reference agent implementations** (custom model classes implementing the `ModelProtocol`)
+
+## Available Benchmarks
+
+### 1. ScreenSpot-v2 (`ss-v2.py`)
+
+- **Dataset**: ScreenSpot-v2 (click-only GUI grounding)
+- **Format**: Standard resolution screenshots
+- **Task**: Predict click coordinates given an instruction and image
+- **Metrics**: Accuracy, Error Rate, Timing, VRAM usage
+
+### 2. ScreenSpot-Pro (`ss-pro.py`)
+
+- **Dataset**: ScreenSpot-Pro (high-resolution click-only GUI grounding)
+- **Format**: High-resolution screenshots
+- **Task**: Predict click coordinates given an instruction and image
+- **Metrics**: Accuracy, Error Rate, Timing, VRAM usage
+
+### 3. Interactive Testing (`interactive.py`)
+
+- **Real-time testing**: Take screenshots and visualize model predictions
+- **Commands**:
+  - Type instruction → test all models on last screenshot
+  - `screenshot` → take screenshot
+  - `models` → list available models
+  - `quit`/`exit` → exit tool
+- **Output**: Visual predictions with crosshairs for each model
+
+## Running Benchmarks
+
+### 1. Configure Models
+
+Edit `utils.py` to specify which models you want to test in `get_available_models()`.
+
+### 2. Run Benchmark
+
+```bash
+# ScreenSpot-v2 benchmark
+python ss-v2.py --samples 50
+
+# ScreenSpot-Pro benchmark
+python ss-pro.py --samples 50
+
+# Interactive testing
+python interactive.py
+```
+
+## Output
+
+### Console Output
+
+```
+Model Results:
+  Accuracy: 85.50% (171/200)
+  Avg Time: 1.23s (0.89s - 2.45s)
+  VRAM Usage: 4.5GB (max) / 3.4GB (avg)
+```
+
+### Generated Files
+
+- **Markdown Report**: `*_results.md` with detailed results tables
+- **Visualizations**: `output/` directory with prediction visualizations
+- **Interactive Output**: `interactive_output/` for interactive session results
+
+## Contributing
+
+To add a new reference model, follow the instructions in [contrib.md](contrib.md).

libs/python/agent/benchmarks/contrib.md

@@ -0,0 +1,166 @@
+# Contributing Reference Agent Implementations
+
+This guide explains how to add your own reference agent implementations to the benchmark system.
+
+## Adding Reference Agent Implementations
+
+### 1. Implement the ModelProtocol
+
+Create a new file in `models/` directory implementing the `ModelProtocol`:
+
+```python
+from models.base import ModelProtocol
+from typing import Optional, Tuple
+from PIL import Image
+
+class YourModelName(ModelProtocol):
+    def __init__(self, model_path: str):
+        self.model_path = model_path
+        self._model = None
+
+    @property
+    def model_name(self) -> str:
+        return self.model_path
+
+    async def load_model(self) -> None:
+        """Load the model into memory."""
+        # Your model loading logic here
+        pass
+
+    async def unload_model(self) -> None:
+        """Unload the model from memory."""
+        # Your model cleanup logic here
+        pass
+
+    async def predict_click(self, image: Image.Image, instruction: str) -> Optional[Tuple[int, int]]:
+        """
+        Predict click coordinates for the given image and instruction.
+
+        Args:
+            image: PIL Image to analyze
+            instruction: Text instruction describing what to click
+
+        Returns:
+            Tuple of (x, y) coordinates or None if prediction fails
+        """
+        # Your prediction logic here
+        return (x, y)  # Return predicted coordinates
+```
+
+### 2. Register Your Model
+
+Add your model to the `get_available_models()` function in `utils.py`:
+
+```python
+def get_available_models() -> List[Union[str, ModelProtocol]]:
+    models = [
+        # Computer Agent SDK providers
+        "huggingface-local/HelloKKMe/GTA1-7B",
+
+        # Reference implementations
+        GTA1Model("HelloKKMe/GTA1-7B"),
+        YourModelName("path/to/your/model"),  # Add your model here
+    ]
+    return models
+```
+
+### 3. Test Your Implementation
+
+Before submitting, test your model with the interactive tool:
+
+```bash
+python interactive.py
+```
+
+This will help you verify that your model loads correctly and produces reasonable predictions.
+
+## Example: Adding a New Model
+
+Here's a complete example of adding a hypothetical "MyVisionModel":
+
+1. **Create `models/my_vision_model.py`:**
+
+```python
+import torch
+from transformers import AutoModel, AutoProcessor
+from models.base import ModelProtocol
+from typing import Optional, Tuple
+from PIL import Image
+
+class MyVisionModel(ModelProtocol):
+    def __init__(self, model_path: str):
+        self.model_path = model_path
+        self.model = None
+        self.processor = None
+
+    @property
+    def model_name(self) -> str:
+        return f"MyVisionModel({self.model_path})"
+
+    async def load_model(self) -> None:
+        """Load the model and processor."""
+        self.processor = AutoProcessor.from_pretrained(self.model_path)
+        self.model = AutoModel.from_pretrained(
+            self.model_path,
+            torch_dtype=torch.float16,
+            device_map="auto"
+        )
+
+    async def unload_model(self) -> None:
+        """Clean up model resources."""
+        del self.model
+        del self.processor
+        self.model = None
+        self.processor = None
+        torch.cuda.empty_cache()
+
+    async def predict_click(self, image: Image.Image, instruction: str) -> Optional[Tuple[int, int]]:
+        """Predict click coordinates."""
+        try:
+            # Preprocess inputs
+            inputs = self.processor(
+                text=instruction,
+                images=image,
+                return_tensors="pt"
+            )
+
+            # Run inference
+            with torch.no_grad():
+                outputs = self.model(**inputs)
+
+            # Extract coordinates (model-specific logic)
+            x, y = self._extract_coordinates(outputs)
+            return (int(x), int(y))
+
+        except Exception as e:
+            print(f"Prediction failed: {e}")
+            return None
+
+    def _extract_coordinates(self, outputs):
+        """Extract x, y coordinates from model outputs."""
+        # Your model-specific coordinate extraction logic
+        pass
+```
+
+2. **Update `models/__init__.py`:**
+
+```python
+from .gta1 import GTA1Model
+from .my_vision_model import MyVisionModel
+
+__all__ = ["GTA1Model", "MyVisionModel"]
+```
+
+3. **Update `utils.py`:**
+
+```python
+from models import GTA1Model, MyVisionModel
+
+def get_available_models() -> List[Union[str, ModelProtocol]]:
+    models = [
+        "huggingface-local/HelloKKMe/GTA1-7B",
+        GTA1Model("HelloKKMe/GTA1-7B"),
+        MyVisionModel("my-org/my-vision-model"),  # Add here
+    ]
+    return models
+```

libs/python/agent/benchmarks/interactive.py

@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+"""
+Interactive Click Prediction Tool
+
+Takes screenshots and allows testing multiple models interactively.
+Models are loaded/unloaded one at a time to avoid memory issues.
+"""
+
+import asyncio
+import os
+from datetime import datetime
+from typing import Any, Dict, List
+
+from utils import (
+    ModelWrapper,
+    get_available_models,
+    save_prediction_visualization,
+    take_screenshot,
+)
+
+
+async def predict_with_all_models(image, instruction: str, models) -> List[Dict[str, Any]]:
+    """
+    Predict click coordinates with all models sequentially.
+
+    Args:
+        image: PIL Image to analyze
+        instruction: Instruction text
+        models: List of model instances
+
+    Returns:
+        List of prediction results
+    """
+    predictions = []
+
+    for model in models:
+        model_wrapper = ModelWrapper(model)
+        print(f"\n🔄 Loading {model_wrapper.model_name}...")
+
+        try:
+            # Load model
+            await model_wrapper.load_model()
+
+            # Predict
+            coords = await model_wrapper.predict_click(image, instruction)
+
+            predictions.append(
+                {"model_name": model_wrapper.model_name, "coords": coords, "error": None}
+            )
+
+            if coords:
+                print(f"✅ {model_wrapper.model_name}: ({coords[0]}, {coords[1]})")
+            else:
+                print(f"❌ {model_wrapper.model_name}: No prediction")
+
+        except Exception as e:
+            print(f"❌ {model_wrapper.model_name}: ERROR - {str(e)}")
+            predictions.append(
+                {"model_name": model_wrapper.model_name, "coords": None, "error": str(e)}
+            )
+
+        finally:
+            # Always unload model to free memory
+            try:
+                await model_wrapper.unload_model()
+                print(f"🗑️  Unloaded {model_wrapper.model_name}")
+            except Exception as e:
+                print(f"⚠️  Error unloading {model_wrapper.model_name}: {e}")
+
+    return predictions
+
+
+def print_header():
+    """Print the interactive tool header."""
+    print("=" * 60)
+    print("🖱️  Interactive Click Prediction Tool")
+    print("=" * 60)
+    print("Commands:")
+    print("  • Type an instruction to test models on last screenshot")
+    print("  • 'screenshot' - Take a new screenshot")
+    print("  • 'models' - List available models")
+    print("  • 'quit' or 'exit' - Exit the tool")
+    print("=" * 60)
+    print("💡 Tip: Take a screenshot first, then send instructions to test models!")
+
+
+def print_models(models):
+    """Print available models."""
+    print("\n📋 Available Models:")
+    for i, model in enumerate(models, 1):
+        if isinstance(model, str):
+            print(f"  {i}. {model}")
+        else:
+            print(f"  {i}. models.{model.__class__.__name__}")
+
+
+async def main():
+    """
+    Main interactive loop.
+    """
+    print_header()
+
+    # Get available models
+    models = get_available_models()
+    print_models(models)
+
+    # Create output directory for visualizations
+    output_dir = "interactive_output"
+    os.makedirs(output_dir, exist_ok=True)
+
+    session_count = 0
+    last_screenshot = None
+    screenshot_timestamp = None
+
+    while True:
+        try:
+            # Get user input
+            print(f"\n{'='*40}")
+            user_input = input("🎯 Enter instruction (or command): ").strip()
+
+            if not user_input:
+                continue
+
+            # Handle commands
+            if user_input.lower() in ["quit", "exit", "q"]:
+                print("👋 Goodbye!")
+                break
+
+            elif user_input.lower() == "models":
+                print_models(models)
+                continue
+
+            elif user_input.lower() == "screenshot":
+                print("📸 Taking screenshot...")
+                try:
+                    last_screenshot = take_screenshot()
+                    screenshot_timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+                    screenshot_path = os.path.join(
+                        output_dir, f"screenshot_{screenshot_timestamp}.png"
+                    )
+                    last_screenshot.save(screenshot_path)
+                    print(f"✅ Screenshot captured and saved to: {screenshot_path}")
+                    print(f"📝 Ready for instructions! Screenshot size: {last_screenshot.size}")
+                except Exception as e:
+                    print(f"❌ Error taking screenshot: {e}")
+                continue
+
+            # Handle instruction input
+            if last_screenshot is None:
+                print(
+                    "⚠️  No screenshot available! Please take a screenshot first using 'screenshot' command."
+                )
+                continue
+
+            session_count += 1
+            print(f"\n🎯 Session {session_count}: '{user_input}'")
+            print(f"📷 Using screenshot from: {screenshot_timestamp}")
+
+            # Predict with all models using last screenshot
+            print(f"\n🤖 Testing {len(models)} models on screenshot...")
+            predictions = await predict_with_all_models(last_screenshot, user_input, models)
+
+            # Display results summary
+            print("\n📊 Results Summary:")
+            print("-" * 50)
+            for pred in predictions:
+                if pred["coords"]:
+                    print(f"✅ {pred['model_name']}: ({pred['coords'][0]}, {pred['coords'][1]})")
+                elif pred["error"]:
+                    print(f"❌ {pred['model_name']}: ERROR - {pred['error']}")
+                else:
+                    print(f"❌ {pred['model_name']}: No prediction")
+
+            # Save visualization
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            vis_filename = f"session_{session_count:03d}_{timestamp}.png"
+            vis_path = os.path.join(output_dir, vis_filename)
+
+            try:
+                save_prediction_visualization(last_screenshot, user_input, predictions, vis_path)
+                print(f"\n💾 Visualization saved to: {vis_path}")
+            except Exception as e:
+                print(f"⚠️  Error saving visualization: {e}")
+
+            print(f"\n✨ Session {session_count} completed!")
+
+        except KeyboardInterrupt:
+            print("\n\n👋 Interrupted by user. Goodbye!")
+            break
+        except Exception as e:
+            print(f"\n❌ Unexpected error: {e}")
+            print("Continuing...")
+
+
+if __name__ == "__main__":
+    try:
+        asyncio.run(main())
+    except KeyboardInterrupt:
+        print("\n👋 Goodbye!")
+    except Exception as e:
+        print(f"❌ Fatal error: {e}")

libs/python/agent/benchmarks/models/__init__.py

@@ -0,0 +1,3 @@
+from .base import ModelProtocol
+
+__all__ = ["ModelProtocol"]

libs/python/agent/benchmarks/models/base.py

@@ -0,0 +1,39 @@
+"""
+Base protocol for benchmark models.
+"""
+
+from typing import Optional, Protocol, Tuple
+
+from PIL import Image
+
+
+class ModelProtocol(Protocol):
+    """Protocol for benchmark models that can predict click coordinates."""
+
+    @property
+    def model_name(self) -> str:
+        """Return the name of the model."""
+        ...
+
+    async def load_model(self) -> None:
+        """Load the model into memory."""
+        ...
+
+    async def unload_model(self) -> None:
+        """Unload the model from memory."""
+        ...
+
+    async def predict_click(
+        self, image: Image.Image, instruction: str
+    ) -> Optional[Tuple[int, int]]:
+        """
+        Predict click coordinates for the given image and instruction.
+
+        Args:
+            image: PIL Image to analyze
+            instruction: Text instruction describing what to click
+
+        Returns:
+            Tuple of (x, y) coordinates or None if prediction fails
+        """
+        ...

libs/python/agent/benchmarks/models/gta1.py

@@ -0,0 +1,158 @@
+"""
+GTA1 model implementation for benchmarking.
+"""
+
+import gc
+import re
+from typing import Optional, Tuple
+
+import torch
+from PIL import Image
+from qwen_vl_utils import process_vision_info, smart_resize
+from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
+
+from .base import ModelProtocol
+
+
+class GTA1Model:
+    """Ground truth GTA1 model implementation."""
+
+    def __init__(self, model_path: str = "HelloKKMe/GTA1-7B"):
+        self.model_path = model_path
+        self.model = None
+        self.processor = None
+        self.max_new_tokens = 32
+
+        self.system_prompt = """
+You are an expert UI element locator. Given a GUI image and a user's element description, provide the coordinates of the specified element as a single (x,y) point. The image resolution is height {height} and width {width}. For elements with area, return the center point.
+
+Output the coordinate pair exactly:
+(x,y)
+""".strip()
+
+    @property
+    def model_name(self) -> str:
+        """Return the name of the model."""
+        return f"GTA1-{self.model_path.split('/')[-1]}"
+
+    async def load_model(self) -> None:
+        """Load the model into memory."""
+        if self.model is None:
+            print(f"Loading GTA1 model: {self.model_path}")
+            self.model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+                self.model_path, torch_dtype=torch.bfloat16, device_map="auto"
+            )
+            self.processor = AutoProcessor.from_pretrained(
+                self.model_path, min_pixels=3136, max_pixels=4096 * 2160
+            )
+            print("GTA1 model loaded successfully")
+
+    async def unload_model(self) -> None:
+        """Unload the model from memory."""
+        if self.model is not None:
+            print("Unloading GTA1 model from GPU...")
+            del self.model
+            del self.processor
+            self.model = None
+            self.processor = None
+            gc.collect()
+            if torch.cuda.is_available():
+                torch.cuda.empty_cache()
+            print("GTA1 model unloaded")
+
+    def _extract_coordinates(self, raw_string: str) -> Tuple[int, int]:
+        """Extract coordinates from model output."""
+        try:
+            matches = re.findall(r"\((-?\d*\.?\d+),\s*(-?\d*\.?\d+)\)", raw_string)
+            return tuple(map(int, map(float, matches[0])))  # type: ignore
+        except:
+            return (0, 0)
+
+    async def predict_click(
+        self, image: Image.Image, instruction: str
+    ) -> Optional[Tuple[int, int]]:
+        """
+        Predict click coordinates for the given image and instruction.
+
+        Args:
+            image: PIL Image to analyze
+            instruction: Text instruction describing what to click
+
+        Returns:
+            Tuple of (x, y) coordinates or None if prediction fails
+        """
+        if self.model is None or self.processor is None:
+            await self.load_model()
+
+        assert self.processor is not None
+        assert self.model is not None
+
+        try:
+            width, height = image.width, image.height
+
+            # Resize image according to processor requirements
+            resized_height, resized_width = smart_resize(
+                image.height,
+                image.width,
+                factor=self.processor.image_processor.patch_size
+                * self.processor.image_processor.merge_size,
+                min_pixels=self.processor.image_processor.min_pixels,
+                max_pixels=self.processor.image_processor.max_pixels,
+            )
+            resized_image = image.resize((resized_width, resized_height))
+            scale_x, scale_y = width / resized_width, height / resized_height
+
+            # Prepare messages
+            system_message = {
+                "role": "system",
+                "content": self.system_prompt.format(height=resized_height, width=resized_width),
+            }
+
+            user_message = {
+                "role": "user",
+                "content": [
+                    {"type": "image", "image": resized_image},
+                    {"type": "text", "text": instruction},
+                ],
+            }
+
+            # Process inputs
+            image_inputs, video_inputs = process_vision_info([system_message, user_message])  # type: ignore
+            text = self.processor.apply_chat_template(
+                [system_message, user_message], tokenize=False, add_generation_prompt=True
+            )
+            inputs = self.processor(
+                text=[text],
+                images=image_inputs,
+                videos=video_inputs,
+                padding=True,
+                return_tensors="pt",
+            )
+            inputs = inputs.to(self.model.device)
+
+            # Generate prediction
+            output_ids = self.model.generate(
+                **inputs,
+                max_new_tokens=self.max_new_tokens,
+                do_sample=False,
+                temperature=1.0,
+                use_cache=True,
+            )
+            generated_ids = [
+                output_ids[len(input_ids) :]
+                for input_ids, output_ids in zip(inputs.input_ids, output_ids)
+            ]
+            output_text = self.processor.batch_decode(
+                generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
+            )[0]
+
+            # Extract and rescale coordinates
+            pred_x, pred_y = self._extract_coordinates(output_text)
+            pred_x = int(pred_x * scale_x)
+            pred_y = int(pred_y * scale_y)
+
+            return (pred_x, pred_y)
+
+        except Exception as e:
+            print(f"Error in GTA1 prediction: {e}")
+            return None

libs/python/agent/benchmarks/ss-pro.py

@@ -0,0 +1,193 @@
+#!/usr/bin/env python3
+"""
+ScreenSpot-Pro Benchmark Script
+
+Evaluates models on the ScreenSpot-Pro dataset for click prediction accuracy.
+Supports both ComputerAgent model strings and custom model classes.
+"""
+
+import argparse
+import asyncio
+import random
+import statistics
+import time
+from typing import Optional
+
+from datasets import load_dataset
+from tqdm import tqdm
+from utils import (
+    ModelWrapper,
+    get_available_models,
+    get_gpu_memory,
+    is_click_in_bbox,
+    save_results_to_markdown,
+    save_visualizations,
+)
+
+
+async def evaluate_model(
+    model_wrapper: ModelWrapper, dataset, max_samples: Optional[int] = None
+) -> dict:
+    """
+    Evaluate a model on the ScreenSpot-Pro dataset.
+
+    Args:
+        model_wrapper: ModelWrapper instance
+        dataset: ScreenSpot-Pro dataset (list of samples)
+        max_samples: Maximum number of samples to evaluate (None for all)
+
+    Returns:
+        Dictionary with evaluation results
+    """
+    print(f"\nEvaluating model: {model_wrapper.model_name}")
+
+    # Load model
+    await model_wrapper.load_model()
+
+    total_samples = len(dataset)
+    if max_samples is not None:
+        total_samples = min(max_samples, total_samples)
+
+    correct_predictions = 0
+    error_predictions = 0
+    results = []
+
+    for i in tqdm(range(total_samples), desc=f"Evaluating {model_wrapper.model_name}"):
+        sample = dataset[i]
+
+        # Extract sample data
+        image = sample["image"]
+        instruction = sample["instruction"]
+        bbox = sample["bbox"]  # [x1, y1, x2, y2]
+        sample_id = sample["img_filename"]
+
+        # Predict click coordinates with timing
+        start_time = time.time()
+        click_coords = await model_wrapper.predict_click(image, instruction)
+        prediction_time = time.time() - start_time
+
+        # Check if prediction is correct
+        is_correct = is_click_in_bbox(click_coords, bbox)
+
+        if is_correct:
+            correct_predictions += 1
+
+        results.append(
+            {
+                "id": sample_id,
+                "instruction": instruction,
+                "bbox": bbox,
+                "predicted_coords": click_coords,
+                "is_correct": is_correct,
+                "failed": False,
+                "prediction_time": prediction_time,
+            }
+        )
+
+    # Unload model
+    await model_wrapper.unload_model()
+
+    # Calculate metrics
+    accuracy = correct_predictions / total_samples if total_samples > 0 else 0.0
+    error_rate = error_predictions / total_samples if total_samples > 0 else 0.0
+
+    # Calculate timing statistics
+    successful_times = [r["prediction_time"] for r in results if not r["failed"]]
+    avg_prediction_time = sum(successful_times) / len(successful_times) if successful_times else 0.0
+    median_prediction_time = statistics.median(successful_times) if successful_times else 0.0
+    min_prediction_time = min(successful_times) if successful_times else 0.0
+    max_prediction_time = max(successful_times) if successful_times else 0.0
+
+    # Get VRAM statistics
+    vram_stats = model_wrapper.get_vram_stats()
+
+    return {
+        "model_name": model_wrapper.model_name,
+        "total_samples": total_samples,
+        "correct_predictions": correct_predictions,
+        "failed_predictions": error_predictions,
+        "accuracy": accuracy,
+        "failure_rate": error_rate,
+        "avg_prediction_time": avg_prediction_time,
+        "median_prediction_time": median_prediction_time,
+        "min_prediction_time": min_prediction_time,
+        "max_prediction_time": max_prediction_time,
+        "vram_max_mb": vram_stats["max_mb"],
+        "vram_avg_mb": vram_stats["avg_mb"],
+        "results": results,
+    }
+
+
+async def main():
+    """
+    Main function to run the benchmark.
+    """
+    # Parse command line arguments
+    parser = argparse.ArgumentParser(description="ScreenSpot-Pro Benchmark Script")
+    parser.add_argument(
+        "--samples", type=int, default=300, help="Number of samples to evaluate (default: 300)"
+    )
+    parser.add_argument(
+        "--seed", type=int, default=42, help="Random seed for shuffling (default: 42)"
+    )
+    args = parser.parse_args()
+
+    # Set random seed
+    random.seed(args.seed)
+
+    # Load dataset
+    print("Loading ScreenSpot-Pro dataset...")
+    ds = load_dataset("lmms-lab/ScreenSpot-Pro")
+    dataset = ds["train"]  # type: ignore
+    # Convert to list to support indexing
+    dataset_list = list(dataset)
+    print(f"Dataset loaded: {len(dataset_list)} samples")
+
+    # Shuffle dataset with seed
+    random.shuffle(dataset_list)
+    print(f"Dataset shuffled with seed {args.seed}")
+
+    # Get available models
+    models = get_available_models()
+
+    # Evaluation settings
+    max_samples = args.samples  # Use command line argument
+
+    # Run evaluations
+    all_results = []
+
+    for model in models:
+        model_wrapper = ModelWrapper(model)
+        result = await evaluate_model(model_wrapper, dataset_list, max_samples)
+        all_results.append(result)
+
+        # Print summary
+        print(f"\n{result['model_name']} Results:")
+        print(f"  Accuracy: {result['accuracy']*100:.2f}%")
+        print(f"  Correct: {result['correct_predictions']}/{result['total_samples']}")
+        print(f"  Errors: {result['failed_predictions']}")
+        print(f"  Error Rate: {result['failure_rate']*100:.2f}%")
+        print(f"  Avg Time: {result['avg_prediction_time']:.2f}s")
+        print(f"  Median Time: {result['median_prediction_time']:.2f}s")
+        print(
+            f"  Time Range: {result['min_prediction_time']:.2f}s - {result['max_prediction_time']:.2f}s"
+        )
+        print(f"  VRAM Max: {result['vram_max_mb']:.1f}MB")
+        print(f"  VRAM Avg: {result['vram_avg_mb']:.1f}MB")
+
+        # Print GPU memory info
+        gpu_memory = get_gpu_memory()
+        if gpu_memory and gpu_memory[0] > 0:
+            print(f"  GPU Free Memory: {gpu_memory[0]:.1f}MB")
+
+    # Save results
+    if all_results:
+        save_results_to_markdown(all_results)
+        save_visualizations(all_results, dataset_list)
+        print("\nBenchmark completed successfully!")
+    else:
+        print("\nNo successful evaluations completed.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

libs/python/agent/benchmarks/ss-v2.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""
+ScreenSpot-v2 Benchmark Script
+
+Evaluates models on the ScreenSpot-v2 dataset for click prediction accuracy.
+Supports both ComputerAgent model strings and custom model classes.
+"""
+
+import argparse
+import asyncio
+import random
+import statistics
+import time
+from typing import Optional
+
+from datasets import load_dataset
+from tqdm import tqdm
+from utils import (
+    ModelWrapper,
+    get_available_models,
+    get_gpu_memory,
+    is_click_in_bbox,
+    save_results_to_markdown,
+    save_visualizations,
+)
+
+
+async def evaluate_model(
+    model_wrapper: ModelWrapper, samples, max_samples: Optional[int] = None
+) -> dict:
+    """
+    Evaluate a model on any iterable of samples.
+
+    Args:
+        model_wrapper: ModelWrapper instance
+        samples: Iterable of dicts with keys: image, bbox, instruction
+        max_samples: Maximum number of samples to evaluate (None for all)
+
+    Returns:
+        Dictionary with evaluation results
+    """
+    print(f"\nEvaluating model: {model_wrapper.model_name}")
+
+    # Load model
+    await model_wrapper.load_model()
+
+    # Convert to list if needed and limit samples
+    if hasattr(samples, "__len__"):
+        total_samples = len(samples)
+        if max_samples is not None:
+            total_samples = min(max_samples, total_samples)
+        sample_list = list(samples)[:total_samples]
+    else:
+        # For iterators, take max_samples or all
+        sample_list = list(samples)
+        if max_samples is not None:
+            sample_list = sample_list[:max_samples]
+        total_samples = len(sample_list)
+
+    correct_predictions = 0
+    error_predictions = 0
+    results = []
+
+    for i, sample in enumerate(tqdm(sample_list, desc=f"Evaluating {model_wrapper.model_name}")):
+        # Extract required data (only these 3 keys matter)
+        image = sample["image"]
+        instruction = sample["instruction"]
+        bbox = sample["bbox"]  # [x1, y1, x2, y2]
+
+        # Predict click coordinates with timing
+        start_time = time.time()
+        click_coords = await model_wrapper.predict_click(image, instruction)
+        prediction_time = time.time() - start_time
+
+        # Check if prediction is correct
+        is_correct = is_click_in_bbox(click_coords, bbox)
+
+        if is_correct:
+            correct_predictions += 1
+
+        results.append(
+            {
+                "sample_idx": i,
+                "instruction": instruction,
+                "bbox": bbox,
+                "predicted_coords": click_coords,
+                "is_correct": is_correct,
+                "failed": False,
+                "prediction_time": prediction_time,
+            }
+        )
+
+    # Unload model
+    await model_wrapper.unload_model()
+
+    # Calculate metrics
+    accuracy = correct_predictions / total_samples if total_samples > 0 else 0.0
+    error_rate = error_predictions / total_samples if total_samples > 0 else 0.0
+
+    # Calculate timing statistics
+    successful_times = [r["prediction_time"] for r in results if not r["failed"]]
+    avg_prediction_time = sum(successful_times) / len(successful_times) if successful_times else 0.0
+    median_prediction_time = statistics.median(successful_times) if successful_times else 0.0
+    min_prediction_time = min(successful_times) if successful_times else 0.0
+    max_prediction_time = max(successful_times) if successful_times else 0.0
+
+    # Get VRAM statistics
+    vram_stats = model_wrapper.get_vram_stats()
+
+    return {
+        "model_name": model_wrapper.model_name,
+        "total_samples": total_samples,
+        "correct_predictions": correct_predictions,
+        "failed_predictions": error_predictions,
+        "accuracy": accuracy,
+        "failure_rate": error_rate,
+        "avg_prediction_time": avg_prediction_time,
+        "median_prediction_time": median_prediction_time,
+        "min_prediction_time": min_prediction_time,
+        "max_prediction_time": max_prediction_time,
+        "vram_max_mb": vram_stats["max_mb"],
+        "vram_avg_mb": vram_stats["avg_mb"],
+        "results": results,
+    }
+
+
+async def main():
+    """
+    Main function to run the benchmark.
+    """
+    # Parse command line arguments
+    parser = argparse.ArgumentParser(description="ScreenSpot-v2 Benchmark Script")
+    parser.add_argument(
+        "--samples", type=int, default=500, help="Number of samples to evaluate (default: 500)"
+    )
+    parser.add_argument(
+        "--seed", type=int, default=42, help="Random seed for shuffling (default: 42)"
+    )
+    args = parser.parse_args()
+
+    # Set random seed
+    random.seed(args.seed)
+
+    # Load dataset
+    print("Loading ScreenSpot-v2 dataset...")
+    ds = load_dataset("lmms-lab/ScreenSpot-v2")
+    dataset = ds["train"]  # type: ignore
+    # Convert to simple list of dicts with only required keys
+    samples = []
+    for item in dataset:
+        # Convert dataset item to dict if needed
+        item_dict = dict(item) if hasattr(item, "keys") else item
+
+        # Convert ScreenSpot-v2 bbox format [x, y, w, h] to [x1, y1, x2, y2]
+        bbox_xywh = item_dict["bbox"]  # type: ignore
+        x, y, w, h = bbox_xywh
+        bbox_xyxy = [x, y, x + w, y + h]
+
+        samples.append(
+            {
+                "image": item_dict["image"],  # type: ignore
+                "instruction": item_dict["instruction"],  # type: ignore
+                "bbox": bbox_xyxy,
+            }
+        )
+    print(f"Dataset loaded: {len(samples)} samples")
+
+    # Shuffle samples with seed
+    random.shuffle(samples)
+    print(f"Samples shuffled with seed {args.seed}")
+
+    # Get available models
+    models = get_available_models()
+
+    # Evaluation settings
+    max_samples = args.samples  # Use command line argument
+
+    # Run evaluations
+    all_results = []
+
+    for model in models:
+        model_wrapper = ModelWrapper(model)
+        result = await evaluate_model(model_wrapper, samples, max_samples)
+        all_results.append(result)
+
+        # Print summary
+        print(f"\n{result['model_name']} Results:")
+        print(f"  Accuracy: {result['accuracy']*100:.2f}%")
+        print(f"  Correct: {result['correct_predictions']}/{result['total_samples']}")
+        print(f"  Errors: {result['failed_predictions']}")
+        print(f"  Error Rate: {result['failure_rate']*100:.2f}%")
+        print(f"  Avg Time: {result['avg_prediction_time']:.2f}s")
+        print(f"  Median Time: {result['median_prediction_time']:.2f}s")
+        print(
+            f"  Time Range: {result['min_prediction_time']:.2f}s - {result['max_prediction_time']:.2f}s"
+        )
+        print(f"  VRAM Max: {result['vram_max_mb']:.1f}MB")
+        print(f"  VRAM Avg: {result['vram_avg_mb']:.1f}MB")
+
+        # Print GPU memory info
+        gpu_memory = get_gpu_memory()
+        if gpu_memory and gpu_memory[0] > 0:
+            print(f"  GPU Free Memory: {gpu_memory[0]:.1f}MB")
+
+    # Save results
+    if all_results:
+        save_results_to_markdown(
+            all_results, "screenspot_v2_results.md", title="ScreenSpot-v2 Benchmark Results"
+        )
+        save_visualizations(all_results, samples)
+        print("\nBenchmark completed successfully!")
+    else:
+        print("\nNo successful evaluations completed.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

libs/python/agent/benchmarks/utils.py

@@ -0,0 +1,444 @@
+#!/usr/bin/env python3
+"""
+Shared utilities for ScreenSpot-Pro benchmarking and interactive testing.
+"""
+
+import dotenv
+
+dotenv.load_dotenv()
+
+import asyncio
+import base64
+import gc
+import os
+import statistics
+import subprocess as sp
+import sys
+from datetime import datetime
+from io import BytesIO
+from typing import List, Optional, Tuple, Union
+
+import torch
+from PIL import Image, ImageDraw
+from tqdm import tqdm
+
+# Add parent directory to path for imports
+sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
+from cua_agent.agent import ComputerAgent
+from models.base import ModelProtocol
+
+
+def get_gpu_memory() -> List[int]:
+    """
+    Get GPU memory usage using nvidia-smi.
+
+    Returns:
+        List of free memory values in MB for each GPU
+    """
+    try:
+        command = "nvidia-smi --query-gpu=memory.free --format=csv"
+        memory_free_info = sp.check_output(command.split()).decode("ascii").split("\n")[:-1][1:]
+        memory_free_values = [int(x.split()[0]) for i, x in enumerate(memory_free_info)]
+        return memory_free_values
+    except (sp.CalledProcessError, FileNotFoundError, IndexError):
+        # Fallback to torch if nvidia-smi is not available
+        if torch.cuda.is_available():
+            device = torch.cuda.current_device()
+            total = torch.cuda.get_device_properties(device).total_memory / 1024 / 1024
+            reserved = torch.cuda.memory_reserved(device) / 1024 / 1024
+            return [int(total - reserved)]
+        return [0]
+
+
+def get_vram_usage() -> dict:
+    """
+    Get current VRAM usage statistics.
+
+    Returns:
+        Dictionary with VRAM usage info (in MB)
+    """
+    if torch.cuda.is_available():
+        device = torch.cuda.current_device()
+        allocated = torch.cuda.memory_allocated(device) / 1024 / 1024  # Convert to MB
+        reserved = torch.cuda.memory_reserved(device) / 1024 / 1024  # Convert to MB
+        total = torch.cuda.get_device_properties(device).total_memory / 1024 / 1024
+        return {
+            "allocated_mb": allocated,
+            "reserved_mb": reserved,
+            "total_mb": total,
+            "free_mb": total - reserved,
+        }
+    else:
+        return {"allocated_mb": 0.0, "reserved_mb": 0.0, "total_mb": 0.0, "free_mb": 0.0}
+
+
+def get_available_models() -> List[Union[str, ModelProtocol]]:
+    """
+    Get list of available models for testing.
+
+    Returns:
+        List of model strings and model classes
+    """
+    local_provider = "huggingface-local/"  # Options: huggingface-local/ or mlx/
+
+    # from models.gta1 import GTA1Model
+
+    models = [
+        # === ComputerAgent model strings ===
+        "openai/computer-use-preview",
+        "anthropic/claude-opus-4-20250514",
+        # f"{local_provider}HelloKKMe/GTA1-7B",
+        # f"{local_provider}HelloKKMe/GTA1-32B",
+        "openai/computer-use-preview+openai/gpt-4o-mini",
+        "anthropic/claude-opus-4-20250514+openai/gpt-4o-mini",
+        # === Reference model classes ===
+        # GTA1Model("HelloKKMe/GTA1-7B"),
+        # GTA1Model("HelloKKMe/GTA1-32B"),
+    ]
+
+    return models
+
+
+def is_click_in_bbox(click_coords: Optional[Tuple[int, int]], bbox: List[int]) -> bool:
+    """
+    Check if click coordinates are within the bounding box.
+
+    Args:
+        click_coords: (x, y) coordinates or None
+        bbox: [x1, y1, x2, y2] bounding box
+
+    Returns:
+        True if click is within bbox, False otherwise
+    """
+    if click_coords is None:
+        return False
+
+    x, y = click_coords
+    x1, y1, x2, y2 = bbox
+
+    return x1 <= x <= x2 and y1 <= y <= y2
+
+
+def image_to_base64(image: Image.Image) -> str:
+    """
+    Convert PIL Image to base64 string.
+
+    Args:
+        image: PIL Image
+
+    Returns:
+        Base64 encoded image string
+    """
+    buffered = BytesIO()
+    image.save(buffered, format="PNG")
+    return base64.b64encode(buffered.getvalue()).decode()
+
+
+class ModelWrapper:
+    """
+    Wrapper to provide unified interface for both ComputerAgent and custom models.
+    """
+
+    def __init__(self, model: Union[str, ModelProtocol]):
+        self.model = model
+        self.is_computer_agent = isinstance(model, str)
+        self.agent: Optional[ComputerAgent] = None
+        self.vram_usage_history: List[float] = []  # Track VRAM usage over time
+
+        if self.is_computer_agent:
+            self.model_name = str(model)
+        else:
+            self.model_name = (
+                f"{model.__class__.__name__}('{getattr(model, 'model_name', 'unknown')}')"
+            )
+
+    async def load_model(self) -> None:
+        """Load the model."""
+        if self.is_computer_agent:
+            self.agent = ComputerAgent(model=str(self.model))
+        else:
+            await self.model.load_model()  # type: ignore
+
+        # Record initial VRAM usage after loading
+        vram_info = get_vram_usage()
+        self.vram_usage_history.append(vram_info["allocated_mb"])
+
+    async def unload_model(self) -> None:
+        """Unload the model."""
+        if not self.is_computer_agent:
+            await self.model.unload_model()  # type: ignore
+        else:
+            del self.agent
+            self.agent = None
+            gc.collect()
+            if torch.cuda.is_available():
+                torch.cuda.empty_cache()
+
+        # Record VRAM usage after unloading
+        vram_info = get_vram_usage()
+        self.vram_usage_history.append(vram_info["allocated_mb"])
+
+    def get_vram_stats(self) -> dict:
+        """Get VRAM usage statistics for this model."""
+        if not self.vram_usage_history:
+            return {"max_mb": 0.0, "avg_mb": 0.0}
+
+        return {
+            "max_mb": max(self.vram_usage_history),
+            "avg_mb": sum(self.vram_usage_history) / len(self.vram_usage_history),
+        }
+
+    async def predict_click(
+        self, image: Image.Image, instruction: str
+    ) -> Optional[Tuple[int, int]]:
+        """Predict click coordinates."""
+        # Record VRAM usage before prediction
+        vram_info = get_vram_usage()
+        self.vram_usage_history.append(vram_info["allocated_mb"])
+
+        if self.is_computer_agent:
+            if self.agent is None:
+                await self.load_model()
+
+            if self.agent is not None:
+                image_b64 = image_to_base64(image)
+                result = await self.agent.predict_click(
+                    instruction=instruction, image_b64=image_b64
+                )
+
+                # Record VRAM usage after prediction
+                vram_info = get_vram_usage()
+                self.vram_usage_history.append(vram_info["allocated_mb"])
+
+                return result
+            return None
+        else:
+            result = await self.model.predict_click(image, instruction)  # type: ignore
+
+            # Record VRAM usage after prediction
+            vram_info = get_vram_usage()
+            self.vram_usage_history.append(vram_info["allocated_mb"])
+
+            return result
+
+
+def save_results_to_markdown(
+    all_results: List[dict],
+    output_file: str = "screenspot_pro_results.md",
+    title: str = "ScreenSpot-Pro Benchmark Results",
+) -> None:
+    """
+    Save evaluation results to a markdown table.
+
+    Args:
+        all_results: List of evaluation results for each model
+        output_file: Output markdown file path
+    """
+    with open(output_file, "w", encoding="utf-8") as f:
+        f.write(f"# {title}\n\n")
+        f.write(f"**Evaluation Date:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n")
+
+        # Summary table
+        f.write("## Summary\n\n")
+        f.write(
+            "| Model | Total Samples | Correct | Errors | Accuracy | Error Rate | Avg Time (s) | Median Time (s) | Time Range (s) | VRAM Max (GB) | VRAM Avg (GB) |\n"
+        )
+        f.write(
+            "|-------|---------------|---------|--------|----------|------------|--------------|-----------------|----------------|---------------|---------------|\n"
+        )
+
+        for result in all_results:
+            model_name = result["model_name"]
+            total = result["total_samples"]
+            correct = result["correct_predictions"]
+            errors = result["failed_predictions"]
+            accuracy = result["accuracy"] * 100
+            error_rate = result["failure_rate"] * 100
+            avg_time = result.get("avg_prediction_time", 0.0)
+            median_time = result.get("median_prediction_time", 0.0)
+            min_time = result.get("min_prediction_time", 0.0)
+            max_time = result.get("max_prediction_time", 0.0)
+            time_range = f"{min_time:.2f} - {max_time:.2f}"
+            vram_max = result.get("vram_max_mb", 0.0) / 1024
+            vram_avg = result.get("vram_avg_mb", 0.0) / 1024
+
+            f.write(
+                f"| {model_name} | {total} | {correct} | {errors} | {accuracy:.2f}% | {error_rate:.2f}% | {avg_time:.2f} | {median_time:.2f} | {time_range} | {vram_max:.1f} | {vram_avg:.1f} |\n"
+            )
+
+        # Detailed results for each model
+        for result in all_results:
+            f.write(f"\n## {result['model_name']} - Detailed Results\n\n")
+            f.write(
+                "| Sample Index | Instruction | BBox | Predicted | Correct | Error | Time (s) |\n"
+            )
+            f.write("|-----------|-------------|------|-----------|---------|-------|----------|\n")
+
+            for sample_result in result["results"][:10]:  # Show first 10 samples
+                sample_idx = sample_result["sample_idx"]
+                instruction = (
+                    sample_result["instruction"][:50] + "..."
+                    if len(sample_result["instruction"]) > 50
+                    else sample_result["instruction"]
+                )
+                bbox = str(sample_result["bbox"])
+                predicted = (
+                    str(sample_result["predicted_coords"])
+                    if sample_result["predicted_coords"]
+                    else "None"
+                )
+                correct = "PASS" if sample_result["is_correct"] else "FAIL"
+                error = "YES" if sample_result["failed"] else "NO"
+                pred_time = sample_result.get("prediction_time", 0.0)
+
+                f.write(
+                    f"| {sample_idx} | {instruction} | {bbox} | {predicted} | {correct} | {error} | {pred_time:.2f} |\n"
+                )
+
+            if len(result["results"]) > 10:
+                f.write(f"\n*Showing first 10 of {len(result['results'])} samples*\n")
+
+    print(f"\nResults saved to: {output_file}")
+
+
+def save_visualizations(all_results: List[dict], samples, output_dir: str = "output") -> None:
+    """
+    Save visualizations of predicted coordinates vs bboxes to an output folder.
+
+    Args:
+        all_results: List of evaluation results for each model
+        samples: List of sample dicts with image, bbox, instruction keys
+        output_dir: Output directory path
+    """
+    os.makedirs(output_dir, exist_ok=True)
+
+    for result in all_results:
+        model_name = result["model_name"].replace("/", "_").replace("\\", "_")
+        model_dir = os.path.join(output_dir, model_name)
+        os.makedirs(model_dir, exist_ok=True)
+
+        print(f"Saving visualizations for {result['model_name']}...")
+
+        # Save first 10 samples for visualization
+        for i, sample_result in enumerate(
+            tqdm(result["results"][:10], desc=f"Saving {model_name} visualizations")
+        ):
+            # Get sample data using index
+            sample_idx = sample_result["sample_idx"]
+
+            if sample_idx < len(samples):
+                sample = samples[sample_idx]
+                image = sample["image"].copy()  # Make a copy to avoid modifying original
+            else:
+                print(f"Warning: Could not find sample at index {sample_idx}")
+                continue
+
+            bbox = sample_result["bbox"]
+            predicted_coords = sample_result["predicted_coords"]
+            is_correct = sample_result["is_correct"]
+
+            # Draw on image
+            draw = ImageDraw.Draw(image)
+
+            # Draw bounding box (ground truth) in green
+            x1, y1, x2, y2 = bbox
+            draw.rectangle([x1, y1, x2, y2], outline="green", width=3)
+            draw.text((x1, y1 - 20), "Ground Truth", fill="green")
+
+            # Draw predicted click in red or blue
+            if predicted_coords is not None:
+                px, py = predicted_coords
+                color = "blue" if is_correct else "red"
+                # Draw crosshair
+                crosshair_size = 15
+                draw.line(
+                    [(px - crosshair_size, py), (px + crosshair_size, py)], fill=color, width=3
+                )
+                draw.line(
+                    [(px, py - crosshair_size), (px, py + crosshair_size)], fill=color, width=3
+                )
+                draw.text((px + 10, py - 20), f"Predicted ({px},{py})", fill=color)
+
+            # Add status text
+            status = "CORRECT" if is_correct else "INCORRECT"
+            status_color = "blue" if is_correct else "red"
+            draw.text((10, 10), f"Status: {status}", fill=status_color)
+            draw.text(
+                (10, 30), f"Instruction: {sample_result['instruction'][:50]}...", fill="black"
+            )
+
+            # Save image
+            filename = f"sample_{i+1:02d}_idx{sample_idx}_{status.lower()}.png"
+            filepath = os.path.join(model_dir, filename)
+            image.save(filepath)
+
+        print(f"Visualizations saved to: {model_dir}")
+
+
+def save_prediction_visualization(
+    image: Image.Image,
+    instruction: str,
+    predictions: List[dict],
+    output_file: str = "interactive_prediction.png",
+) -> None:
+    """
+    Save visualization of multiple model predictions on a single image.
+
+    Args:
+        image: PIL Image to visualize
+        instruction: Instruction text
+        predictions: List of prediction dicts with keys: model_name, coords, error
+        output_file: Output file path
+    """
+    # Create a copy of the image
+    vis_image = image.copy()
+    draw = ImageDraw.Draw(vis_image)
+
+    # Colors for different models
+    colors = ["red", "blue", "orange", "purple", "brown", "pink", "gray", "olive"]
+
+    # Draw predictions
+    for i, pred in enumerate(predictions):
+        color = colors[i % len(colors)]
+        model_name = pred["model_name"]
+        coords = pred.get("coords")
+        error = pred.get("error")
+
+        if coords is not None:
+            px, py = coords
+            # Draw crosshair
+            crosshair_size = 20
+            draw.line([(px - crosshair_size, py), (px + crosshair_size, py)], fill=color, width=4)
+            draw.line([(px, py - crosshair_size), (px, py + crosshair_size)], fill=color, width=4)
+            # Draw model name
+            draw.text((px + 15, py + 15), f"{model_name}: ({px},{py})", fill=color)
+        else:
+            # Draw error text
+            draw.text((10, 50 + i * 20), f"{model_name}: ERROR - {error}", fill=color)
+
+    # Add instruction at the top
+    draw.text((10, 10), f"Instruction: {instruction}", fill="black")
+
+    # Save image
+    vis_image.save(output_file)
+    print(f"Prediction visualization saved to: {output_file}")
+
+
+def take_screenshot() -> Image.Image:
+    """
+    Take a screenshot of the current screen.
+
+    Returns:
+        PIL Image of the screenshot
+    """
+    try:
+        from PIL import ImageGrab
+
+        screenshot = ImageGrab.grab()
+        return screenshot
+    except ImportError:
+        print("PIL/Pillow not installed. Please install it with: pip install pillow")
+        raise
+    except Exception as e:
+        print(f"Error taking screenshot: {e}")
+        raise