trycua--cua/libs/cua-bench/scripts/GUIDE.md

Ohm-Bench: Packaging & Evaluation Guide

Prerequisites

cd libs/cua-bench
uv sync
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
echo "OPENAI_API_KEY=sk-..." >> .env

Build the environment image (KiCad 9 pre-installed):

cd libs/xfce
docker build -t trycua/cua-xfce:latest .

---

1. Packaging a Submission

Converts a DaaS submission JSON into a self-contained scripts/tasks/<id>/ directory.

uv run python scripts/package_kicad_submission.py <submission.json> --output scripts/tasks/

Input JSON fields required:

• submission_id — unique ID (becomes the task directory name)
• circuit_prompt — task description shown to the agent
• netlist.s3Uri — S3 URI of the reference netlist zip
• circuit_pcb_file.s3Uri — S3 URI of the initial KiCad project zip

Output structure:

scripts/tasks/<submission_id>/
  main.py          # cb task harness
  reference.net    # ground-truth KiCad netlist
  initial/         # initial KiCad project files shown to agent
    <project>/
      *.kicad_sch
      *.kicad_pcb
      ...

---

2. Running Evaluations

Single task (quick test)

uv run cb run dataset scripts/tasks --task-filter "<submission_id>" \
  --agent cua-agent --model anthropic/claude-opus-4-6 --max-steps 150

Multiple specific tasks

uv run cb run dataset scripts/tasks \
  --task-filter "154d0750,1625e97a,2c11fe94" \
  --agent cua-agent --model anthropic/claude-opus-4-6 \
  --max-parallel 5 --max-steps 150

Full dataset

uv run cb run dataset scripts/tasks \
  --agent cua-agent --model anthropic/claude-opus-4-6 \
  --max-parallel 5 --max-steps 150

Monitor a run

uv run cb run watch <run_id>
uv run cb run info <run_id>

---

3. Calibration (N attempts per task per model)

Runs each task N times with Claude and N times with OpenAI, then computes pass rates and difficulty tiers.

# Specific tasks, 5 attempts each, 1000 max steps
uv run python scripts/run_calibration.py \
  --tasks "154d0750,1625e97a,2c11fe94,34877be3,458f5f5c" \
  --attempts 5 \
  --max-steps 1000

# All tasks (runs sequentially: all claude attempts, then all openai attempts)
uv run python scripts/run_calibration.py \
  --tasks "$(ls scripts/tasks | grep -v '.json' | tr '\n' ',' | sed 's/,$//')" \
  --attempts 5 \
  --max-steps 1000

What it does per task:

1. Launches --attempts simultaneous cb run dataset calls with anthropic/claude-opus-4-6
2. Waits for all to finish
3. Launches --attempts simultaneous calls with openai/computer-use-preview
4. Waits for all to finish
5. Moves to next task

---

4. Cleaning Up

Kill all running containers between runs:

docker ps -a --format "{{.Names}}" | grep "cua-" | xargs -r docker rm -f

Stop all active cb runs:

uv run cb run list | grep "^[a-f0-9]" | awk '{print $1}' | xargs -I{} uv run cb run stop {}