wehub-resource-sync
91e75e620b
CI: cua-driver distro-compat matrix / debian:12 (glibc 2.36) (push) Has been cancelled
CI: SPDX Headers / Check SPDX headers (warn-only) (push) Has been cancelled
CD: Docs MCP Server / build (linux/amd64) (push) Has been cancelled
CD: Docs MCP Server / build (linux/arm64) (push) Has been cancelled
CD: Docs MCP Server / merge (push) Has been cancelled
CI: cua-driver distro-compat matrix / Resolve release version (push) Has been cancelled
CI: cua-driver distro-compat matrix / fedora:41 (glibc 2.40) (push) Has been cancelled
CI: cua-driver distro-compat matrix / rockylinux:9 (glibc 2.34) (push) Has been cancelled
CI: cua-driver distro-compat matrix / ubuntu:22.04 (glibc 2.35) (push) Has been cancelled
CI: cua-driver distro-compat matrix / ubuntu:24.04 (glibc 2.39) (push) Has been cancelled
CI: cua-driver distro-compat matrix / Distro compat summary (push) Has been cancelled
CI: Rust Linux unit / Rust Linux unit and compile (push) Has been cancelled
CI: Rust Windows unit / Rust Windows unit and compile (push) Has been cancelled
CI: Nix Linux Rust source / Nix / compositor build (push) Has been cancelled
CI: Nix Linux Rust source / Nix / driver package (push) Has been cancelled
CI: Nix Linux Rust source / Nix / Rust unit tests (push) Has been cancelled
123 lines
2.9 KiB
Markdown
123 lines
2.9 KiB
Markdown
# Ohm-Bench: Packaging & Evaluation Guide
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
cd libs/cua-bench
|
|
uv sync
|
|
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
|
|
echo "OPENAI_API_KEY=sk-..." >> .env
|
|
```
|
|
|
|
Build the environment image (KiCad 9 pre-installed):
|
|
```bash
|
|
cd libs/xfce
|
|
docker build -t trycua/cua-xfce:latest .
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Packaging a Submission
|
|
|
|
Converts a DaaS submission JSON into a self-contained `scripts/tasks/<id>/` directory.
|
|
|
|
```bash
|
|
uv run python scripts/package_kicad_submission.py <submission.json> --output scripts/tasks/
|
|
```
|
|
|
|
**Input JSON fields required:**
|
|
- `submission_id` — unique ID (becomes the task directory name)
|
|
- `circuit_prompt` — task description shown to the agent
|
|
- `netlist.s3Uri` — S3 URI of the reference netlist zip
|
|
- `circuit_pcb_file.s3Uri` — S3 URI of the initial KiCad project zip
|
|
|
|
**Output structure:**
|
|
```
|
|
scripts/tasks/<submission_id>/
|
|
main.py # cb task harness
|
|
reference.net # ground-truth KiCad netlist
|
|
initial/ # initial KiCad project files shown to agent
|
|
<project>/
|
|
*.kicad_sch
|
|
*.kicad_pcb
|
|
...
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Running Evaluations
|
|
|
|
### Single task (quick test)
|
|
|
|
```bash
|
|
uv run cb run dataset scripts/tasks --task-filter "<submission_id>" \
|
|
--agent cua-agent --model anthropic/claude-opus-4-6 --max-steps 150
|
|
```
|
|
|
|
### Multiple specific tasks
|
|
|
|
```bash
|
|
uv run cb run dataset scripts/tasks \
|
|
--task-filter "154d0750,1625e97a,2c11fe94" \
|
|
--agent cua-agent --model anthropic/claude-opus-4-6 \
|
|
--max-parallel 5 --max-steps 150
|
|
```
|
|
|
|
### Full dataset
|
|
|
|
```bash
|
|
uv run cb run dataset scripts/tasks \
|
|
--agent cua-agent --model anthropic/claude-opus-4-6 \
|
|
--max-parallel 5 --max-steps 150
|
|
```
|
|
|
|
### Monitor a run
|
|
|
|
```bash
|
|
uv run cb run watch <run_id>
|
|
uv run cb run info <run_id>
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Calibration (N attempts per task per model)
|
|
|
|
Runs each task N times with Claude and N times with OpenAI, then computes pass rates and difficulty tiers.
|
|
|
|
```bash
|
|
# Specific tasks, 5 attempts each, 1000 max steps
|
|
uv run python scripts/run_calibration.py \
|
|
--tasks "154d0750,1625e97a,2c11fe94,34877be3,458f5f5c" \
|
|
--attempts 5 \
|
|
--max-steps 1000
|
|
```
|
|
|
|
```bash
|
|
# All tasks (runs sequentially: all claude attempts, then all openai attempts)
|
|
uv run python scripts/run_calibration.py \
|
|
--tasks "$(ls scripts/tasks | grep -v '.json' | tr '\n' ',' | sed 's/,$//')" \
|
|
--attempts 5 \
|
|
--max-steps 1000
|
|
```
|
|
|
|
**What it does per task:**
|
|
1. Launches `--attempts` simultaneous `cb run dataset` calls with `anthropic/claude-opus-4-6`
|
|
2. Waits for all to finish
|
|
3. Launches `--attempts` simultaneous calls with `openai/computer-use-preview`
|
|
4. Waits for all to finish
|
|
5. Moves to next task
|
|
|
|
---
|
|
|
|
## 4. Cleaning Up
|
|
|
|
Kill all running containers between runs:
|
|
```bash
|
|
docker ps -a --format "{{.Names}}" | grep "cua-" | xargs -r docker rm -f
|
|
```
|
|
|
|
Stop all active cb runs:
|
|
```bash
|
|
uv run cb run list | grep "^[a-f0-9]" | awk '{print $1}' | xargs -I{} uv run cb run stop {}
|
|
```
|