chore: import upstream snapshot with attribution

2026-07-13 12:43:05 +08:00
commit 426e9eeabd
41828 changed files with 9656266 additions and 0 deletions
@@ -0,0 +1,121 @@
+# === Environment & Secrets ===
+.env
+.env.*
+
+# === Python ===
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.egg
+*.egg-info/
+dist/
+build/
+.eggs/
+*.whl
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+htmlcov/
+.coverage
+.coverage.*
+coverage.xml
+
+# === Node / TypeScript ===
+node_modules/
+*.tsbuildinfo
+package-lock.json
+
+# === Rust ===
+target/
+Cargo.lock
+
+# === OS / IDE ===
+.DS_Store
+Thumbs.db
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# =============================================================
+# Generated Benchmark Run Outputs
+# =============================================================
+
+# Any directory starting with "benchmark_results" (everywhere).
+# Benchmark snapshots are generated run output and should not be committed.
+**/benchmark_results*/
+
+# Trajectory exports
+**/trajectories/
+**/*.art.jsonl
+**/*.grpo.groups.json
+
+# Orchestrator run groups and DB files are regenerated locally.
+/benchmark_results/rg_*/
+/benchmark_results/orchestrator.sqlite*
+/benchmark_results/viewer_data.json
+
+# Test output
+**/test_output/
+
+# Benchmark caches
+**/.benchmark_cache/
+
+# Local source checkouts for external harnesses
+**/openclaw-src/
+**/hermes-agent-src/
+.agents/
+
+# --- Ad-hoc run output directories (all benchmarks) ---
+**/canonical_*/
+**/smoke_*/
+**/traj_*/
+**/verified_*/
+**/post_b_*/
+**/post_delete_*/
+**/real_db/
+**/real_eliza_*/
+**/full_eliza_*/
+**/eliza_run_*/
+**/ws_test*/
+**/lateral_test*/
+**/smart_smoke/
+**/final_benchmark_results/
+
+# --- Social-alpha ---
+social-alpha/results/
+social-alpha/trust_marketplace_benchmark.egg-info/
+
+# Large dataset files (should be fetched separately or use LFS)
+social-alpha/trenches-chat-dataset/data/
+social-alpha/trenches-chat-dataset/compressed/
+social-alpha/trenches-chat-dataset/price_fetch_progress.json
+
+# --- Framework generated results (keep .gitkeep) ---
+framework/results/*.json
+
+# --- Stale refs ---
+**/refs/
+
+# SWE-bench cloned task workspaces (nested git repos; local checkout only)
+swe-bench-workspace/
+
+# --- Per-benchmark committed run output (regenerated locally; never commit) ---
+# Underscore-prefixed smoke output dirs (the smoke_*/ rule above misses these).
+**/_smoke_*/
+**/_cli_smoke/
+# CompactBench --output captures
+compactbench/results*.jsonl
+compactbench/results*.json
+compactbench/external/
+# Deleted EVM benchmark local run output
+evm/metrics/
+evm/skill_runner/_test_skill.ts
+# Deleted LoCA benchmark local lockfile
+loca-bench/uv.lock
+# Trust handler result captures
+trust/results*.json
+# VoiceBench synthesized dataset (rebuilt by run.sh)
+voicebench/shared/generated/
@@ -0,0 +1,123 @@
+# Benchmarks — Agent Guide
+
+The elizaOS evaluation suite. The **registry** declares every benchmark; the
+**orchestrator** runs them; each benchmark lives in its own directory with its
+own `README.md` / `AGENTS.md` / `CLAUDE.md`.
+
+## Layout
+
+```
+registry/        Canonical benchmark definitions (id, command, requirements, scorer)
+orchestrator/    Runner: executes registry benchmarks, normalizes results, viewer, gates
+framework/ lib/  Shared harness framework + helpers
+standard/        MMLU / HumanEval / GSM8K / MT-Bench adapters (dispatched by run.py)
+viewer/          Static results UI
+tests/           Suite-level tests (registry, scoring, normalization, acceptance gate)
+*-adapter/       Agent harness bridges: eliza / hermes / openclaw / smithers
+agentbench_matrix/  Code-agent comparison adapter (driven by orchestrator/code_agent_matrix.py); the duplicate *_matrix/ + app_eval/ import-shim variants were removed in #9475
+loadperf/ memperf/ mobile-resource/ view-bundle-size/
+                 Resource/device/bundle KPI harnesses (infra/CPU/memory/battery/bundle size),
+                 NOT agent benchmarks — own CI lanes, not orchestrator adapters
+<benchmark>/     One self-contained benchmark per directory
+benchmark_results/   Generated run output — GITIGNORED, never commit
+```
+
+`orchestrator/ci_coverage.py` classifies every registered benchmark's CI lane
+(scheduled / smoke / manual), and `tests/test_ci_coverage.py` keeps that mapping
+1:1 with the registry.
+
+## Run a benchmark
+
+```bash
+# List integrated benchmarks + adapter coverage
+python -m benchmarks.orchestrator list-benchmarks
+
+# Run one (idempotent: skips already-successful signatures)
+python -m benchmarks.orchestrator run --benchmarks <id> --provider <p> --model <m>
+
+# Run all
+python -m benchmarks.orchestrator run --all --provider cerebras --model gemma-4-31b
+```
+
+`--rerun-failed` reruns only failed signatures; `--force` always makes a fresh
+run; `--extra '<json>'` passes benchmark-specific options. Each benchmark's own
+`AGENTS.md` documents the direct (non-orchestrator) command and a no-key
+smoke/mock path.
+
+## Test the harnesses
+
+```bash
+pytest tests/ -v                                   # suite-level
+pytest <benchmark>/.../tests/ -v                   # one benchmark (see its AGENTS.md)
+```
+
+TypeScript/Bun benchmarks (`eliza-1`, `vision-language`, `configbench`,
+`interrupt-bench`, `personality-bench`, `three-agent-dialogue`) test with
+`bun test`; Rust components (HyperliquidBench runner) with `cargo test`.
+
+## Conventions
+
+- **One directory per benchmark.** All of a benchmark's code, data, tests, and
+  docs live under its directory. Don't scatter benchmark code into shared dirs.
+- **The registry is the source of truth.** A benchmark is "integrated" only when
+  it has an entry in `registry/commands.py` and a scorer in `registry/scores.py`.
+  Some directories are run-only / experimental and not yet registered — their
+  `AGENTS.md` says so.
+- **Results are generated, not committed.** Anything under `benchmark_results/`
+  (and per-benchmark run output) is gitignored. Never commit result JSON, SQLite
+  DBs, trajectories, logs, or coverage.
+- **Every benchmark carries all three docs.** `README.md` (overview),
+  `AGENTS.md` (how to run + smoke + test), `CLAUDE.md` (pointer to AGENTS.md).
+
+## Add a benchmark
+
+1. Create `<your-benchmark>/` (harness + tests + three docs).
+2. Add a `BenchmarkDefinition` in `registry/commands.py` and a `_score_from_*`
+   in `registry/scores.py`.
+3. Verify with `python -m benchmarks.orchestrator list-benchmarks`.
+
+Operator runbook (remote GPU, calibration/readiness gates, code-agent matrix):
+[`ORCHESTRATOR_SUBAGENT_BENCHMARK_RUNBOOK.md`](ORCHESTRATOR_SUBAGENT_BENCHMARK_RUNBOOK.md),
+[`orchestrator/README.md`](orchestrator/README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,123 @@
+# Benchmarks — Agent Guide
+
+The elizaOS evaluation suite. The **registry** declares every benchmark; the
+**orchestrator** runs them; each benchmark lives in its own directory with its
+own `README.md` / `AGENTS.md` / `CLAUDE.md`.
+
+## Layout
+
+```
+registry/        Canonical benchmark definitions (id, command, requirements, scorer)
+orchestrator/    Runner: executes registry benchmarks, normalizes results, viewer, gates
+framework/ lib/  Shared harness framework + helpers
+standard/        MMLU / HumanEval / GSM8K / MT-Bench adapters (dispatched by run.py)
+viewer/          Static results UI
+tests/           Suite-level tests (registry, scoring, normalization, acceptance gate)
+*-adapter/       Agent harness bridges: eliza / hermes / openclaw / smithers
+agentbench_matrix/  Code-agent comparison adapter (driven by orchestrator/code_agent_matrix.py); the duplicate *_matrix/ + app_eval/ import-shim variants were removed in #9475
+loadperf/ memperf/ mobile-resource/ view-bundle-size/
+                 Resource/device/bundle KPI harnesses (infra/CPU/memory/battery/bundle size),
+                 NOT agent benchmarks — own CI lanes, not orchestrator adapters
+<benchmark>/     One self-contained benchmark per directory
+benchmark_results/   Generated run output — GITIGNORED, never commit
+```
+
+`orchestrator/ci_coverage.py` classifies every registered benchmark's CI lane
+(scheduled / smoke / manual), and `tests/test_ci_coverage.py` keeps that mapping
+1:1 with the registry.
+
+## Run a benchmark
+
+```bash
+# List integrated benchmarks + adapter coverage
+python -m benchmarks.orchestrator list-benchmarks
+
+# Run one (idempotent: skips already-successful signatures)
+python -m benchmarks.orchestrator run --benchmarks <id> --provider <p> --model <m>
+
+# Run all
+python -m benchmarks.orchestrator run --all --provider cerebras --model gemma-4-31b
+```
+
+`--rerun-failed` reruns only failed signatures; `--force` always makes a fresh
+run; `--extra '<json>'` passes benchmark-specific options. Each benchmark's own
+`AGENTS.md` documents the direct (non-orchestrator) command and a no-key
+smoke/mock path.
+
+## Test the harnesses
+
+```bash
+pytest tests/ -v                                   # suite-level
+pytest <benchmark>/.../tests/ -v                   # one benchmark (see its AGENTS.md)
+```
+
+TypeScript/Bun benchmarks (`eliza-1`, `vision-language`, `configbench`,
+`interrupt-bench`, `personality-bench`, `three-agent-dialogue`) test with
+`bun test`; Rust components (HyperliquidBench runner) with `cargo test`.
+
+## Conventions
+
+- **One directory per benchmark.** All of a benchmark's code, data, tests, and
+  docs live under its directory. Don't scatter benchmark code into shared dirs.
+- **The registry is the source of truth.** A benchmark is "integrated" only when
+  it has an entry in `registry/commands.py` and a scorer in `registry/scores.py`.
+  Some directories are run-only / experimental and not yet registered — their
+  `AGENTS.md` says so.
+- **Results are generated, not committed.** Anything under `benchmark_results/`
+  (and per-benchmark run output) is gitignored. Never commit result JSON, SQLite
+  DBs, trajectories, logs, or coverage.
+- **Every benchmark carries all three docs.** `README.md` (overview),
+  `AGENTS.md` (how to run + smoke + test), `CLAUDE.md` (pointer to AGENTS.md).
+
+## Add a benchmark
+
+1. Create `<your-benchmark>/` (harness + tests + three docs).
+2. Add a `BenchmarkDefinition` in `registry/commands.py` and a `_score_from_*`
+   in `registry/scores.py`.
+3. Verify with `python -m benchmarks.orchestrator list-benchmarks`.
+
+Operator runbook (remote GPU, calibration/readiness gates, code-agent matrix):
+[`ORCHESTRATOR_SUBAGENT_BENCHMARK_RUNBOOK.md`](ORCHESTRATOR_SUBAGENT_BENCHMARK_RUNBOOK.md),
+[`orchestrator/README.md`](orchestrator/README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,54 @@
+name: CI
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  build-test-demo:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Install Rust
+        uses: actions-rust-lang/setup-rust@v1
+        with:
+          rust-version: stable
+
+      - name: Cache cargo artifacts
+        uses: swatinem/rust-cache@v2
+
+      - name: Format (cargo fmt --check)
+        run: cargo fmt --all -- --check
+
+      - name: Clippy (deny warnings)
+        run: cargo clippy --no-deps -- -D warnings
+
+      - name: Run tests
+        run: cargo test
+
+      - name: Demo benchmark run
+        run: |
+          cargo run -p hl-runner --release -- \
+            --demo \
+            --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+            --out runs/demo
+          cargo run -p hl-evaluator --release -- \
+            --input runs/demo/per_action.jsonl \
+            --domains dataset/domains-hl.yaml \
+            --out-dir runs/demo
+
+      - name: Inspect demo score
+        run: jq '.finalScore' runs/demo/eval_score.json
+
+      - name: Upload demo artifacts
+        uses: actions/upload-artifact@v7
+        with:
+          name: demo-artifacts
+          path: runs/demo
@@ -0,0 +1,3 @@
+.idea/
+/target
+.env
@@ -0,0 +1,145 @@
+# HyperliquidBench — Agent Guide
+
+Measures **operational competence** of Hyperliquid perp trading agents: correct
+order routing, cancels, transfers, and leverage changes across two tracks —
+Coverage (breadth of action signatures across perp/account/risk domains) and
+HiaN (Haystack-in-a-Needle long-context precision). Registered in the suite
+registry as `hyperliquid_bench`.
+
+Scoring: `FINAL_SCORE = Base + Bonus − Penalty` computed by `hl-evaluator` (Rust).
+The Python `__main__.py` routes plan generation through the Eliza TS bridge (`--mode eliza`)
+and delegates execution/evaluation to the Rust crates.
+
+## Run
+
+```bash
+# Direct — demo mode (no funds at risk, no key required), eliza TS bridge
+python -m benchmarks.HyperliquidBench --demo
+
+# Direct — coverage scenario, demo mode
+python -m benchmarks.HyperliquidBench \
+  --coverage \
+  --demo
+
+# Through the suite orchestrator (stores results, resolves provider/model)
+python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --provider cerebras \
+  --model gpt-oss-120b
+
+# Live orchestrated run (all harnesses, Cerebras, no demo)
+HL_PRIVATE_KEY=0x... \
+CEREBRAS_API_KEY=csk-... \
+python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --all-harnesses \
+  --provider cerebras \
+  --model gpt-oss-120b \
+  --force \
+  --show-incompatible
+```
+
+## Smoke test (no API keys, no network)
+
+```bash
+# Deterministic local agent — no TS bridge, no Rust required for plan generation
+python -m benchmarks.HyperliquidBench --mode deterministic --demo
+
+# Rust runner demo mode (validates full pipeline without touching live endpoints)
+cargo run -p hl-runner --release -- \
+  --demo \
+  --out runs/demo
+
+cargo run -p hl-evaluator --release -- \
+  --input runs/demo/per_action.jsonl \
+  --domains dataset/domains-hl.yaml \
+  --out-dir runs/demo
+```
+
+The convenience wrapper `scripts/run_cov.sh` handles the two-step runner + evaluator
+call; omit `NETWORK` or set it to `demo` and pass `-- --demo` for offline runs.
+
+## Test the harness
+
+```bash
+# Rust unit tests (no API keys required)
+cargo test
+
+# Or via make
+make test
+```
+
+No Python pytest suite exists in this directory; harness logic is tested through
+the Rust `cargo test` target and the Makefile shortcuts (`make format`, `make check`, `make build`).
+
+## Layout
+
+| Path | Role |
+| --- | --- |
+| `__main__.py` | Python CLI entrypoint (`python -m benchmarks.HyperliquidBench`) |
+| `eliza_agent.py` | Local deterministic agent + scenario helpers |
+| `types.py` | `HLBenchConfig`, `TradingScenario` shared types |
+| `crates/hl-runner/` | Rust CLI: loads plans, signs + submits actions, writes artifacts |
+| `crates/hl-evaluator/` | Rust CLI: normalizes signatures, applies scoring, emits score reports |
+| `crates/hl-common/` | Shared plan schema, action types, time utils, artifact helpers |
+| `dataset/domains-hl.yaml` | Domain weights + signature allowlists (scoring config) |
+| `dataset/tasks/` | Authoritative coverage task JSONL files |
+| `dataset/hian/` | HiaN case bundles (prompt, ground truth, metadata) |
+| `scripts/run_cov.sh` | Convenience wrapper: runner + evaluator in one call |
+| `scripts/run_hian.sh` | HiaN demo runner + validator wrapper |
+| `frontend/` | Static leaderboard + trajectory explorer |
+
+## Notes
+
+- Results write to `HyperliquidBench/runs/<timestamp>/` (gitignored). The Python
+  entrypoint also writes an aggregated `hyperliquid_bench-<mode>-<timestamp>.json`
+  to `--output` (default: `runs/`).
+- Scored by `_score_from_hyperliquid_bench_json` in `registry/scores.py`.
+  Demo-mode results are intentionally rejected by the publishability gate.
+- Rust crates must be built before live runs:
+  `cargo build --release -p hl-runner -p hl-evaluator`
+- Live network runs require `HL_PRIVATE_KEY` and `--no-demo`.
+  Default model provider is Cerebras (`gpt-oss-120b`); OpenRouter is also supported.
+- Full background: [README.md](README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,145 @@
+# HyperliquidBench — Agent Guide
+
+Measures **operational competence** of Hyperliquid perp trading agents: correct
+order routing, cancels, transfers, and leverage changes across two tracks —
+Coverage (breadth of action signatures across perp/account/risk domains) and
+HiaN (Haystack-in-a-Needle long-context precision). Registered in the suite
+registry as `hyperliquid_bench`.
+
+Scoring: `FINAL_SCORE = Base + Bonus − Penalty` computed by `hl-evaluator` (Rust).
+The Python `__main__.py` routes plan generation through the Eliza TS bridge (`--mode eliza`)
+and delegates execution/evaluation to the Rust crates.
+
+## Run
+
+```bash
+# Direct — demo mode (no funds at risk, no key required), eliza TS bridge
+python -m benchmarks.HyperliquidBench --demo
+
+# Direct — coverage scenario, demo mode
+python -m benchmarks.HyperliquidBench \
+  --coverage \
+  --demo
+
+# Through the suite orchestrator (stores results, resolves provider/model)
+python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --provider cerebras \
+  --model gpt-oss-120b
+
+# Live orchestrated run (all harnesses, Cerebras, no demo)
+HL_PRIVATE_KEY=0x... \
+CEREBRAS_API_KEY=csk-... \
+python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --all-harnesses \
+  --provider cerebras \
+  --model gpt-oss-120b \
+  --force \
+  --show-incompatible
+```
+
+## Smoke test (no API keys, no network)
+
+```bash
+# Deterministic local agent — no TS bridge, no Rust required for plan generation
+python -m benchmarks.HyperliquidBench --mode deterministic --demo
+
+# Rust runner demo mode (validates full pipeline without touching live endpoints)
+cargo run -p hl-runner --release -- \
+  --demo \
+  --out runs/demo
+
+cargo run -p hl-evaluator --release -- \
+  --input runs/demo/per_action.jsonl \
+  --domains dataset/domains-hl.yaml \
+  --out-dir runs/demo
+```
+
+The convenience wrapper `scripts/run_cov.sh` handles the two-step runner + evaluator
+call; omit `NETWORK` or set it to `demo` and pass `-- --demo` for offline runs.
+
+## Test the harness
+
+```bash
+# Rust unit tests (no API keys required)
+cargo test
+
+# Or via make
+make test
+```
+
+No Python pytest suite exists in this directory; harness logic is tested through
+the Rust `cargo test` target and the Makefile shortcuts (`make format`, `make check`, `make build`).
+
+## Layout
+
+| Path | Role |
+| --- | --- |
+| `__main__.py` | Python CLI entrypoint (`python -m benchmarks.HyperliquidBench`) |
+| `eliza_agent.py` | Local deterministic agent + scenario helpers |
+| `types.py` | `HLBenchConfig`, `TradingScenario` shared types |
+| `crates/hl-runner/` | Rust CLI: loads plans, signs + submits actions, writes artifacts |
+| `crates/hl-evaluator/` | Rust CLI: normalizes signatures, applies scoring, emits score reports |
+| `crates/hl-common/` | Shared plan schema, action types, time utils, artifact helpers |
+| `dataset/domains-hl.yaml` | Domain weights + signature allowlists (scoring config) |
+| `dataset/tasks/` | Authoritative coverage task JSONL files |
+| `dataset/hian/` | HiaN case bundles (prompt, ground truth, metadata) |
+| `scripts/run_cov.sh` | Convenience wrapper: runner + evaluator in one call |
+| `scripts/run_hian.sh` | HiaN demo runner + validator wrapper |
+| `frontend/` | Static leaderboard + trajectory explorer |
+
+## Notes
+
+- Results write to `HyperliquidBench/runs/<timestamp>/` (gitignored). The Python
+  entrypoint also writes an aggregated `hyperliquid_bench-<mode>-<timestamp>.json`
+  to `--output` (default: `runs/`).
+- Scored by `_score_from_hyperliquid_bench_json` in `registry/scores.py`.
+  Demo-mode results are intentionally rejected by the publishability gate.
+- Rust crates must be built before live runs:
+  `cargo build --release -p hl-runner -p hl-evaluator`
+- Live network runs require `HL_PRIVATE_KEY` and `--no-demo`.
+  Default model provider is Cerebras (`gpt-oss-120b`); OpenRouter is also supported.
+- Full background: [README.md](README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,34 @@
+[workspace]
+members = [
+    "crates/hl-common",
+    "crates/hl-runner",
+    "crates/hl-evaluator",
+]
+resolver = "2"
+
+[workspace.package]
+edition = "2021"
+authors = ["HyperLiquidBench"]
+version = "0.1.0"
+
+[workspace.dependencies]
+anyhow = "1.0"
+chrono = { version = "0.4", features = ["clock"] }
+clap = { version = "4.5", features = ["derive", "env"] }
+csv = "1.3"
+dotenvy = "0.15"
+futures = "0.3"
+hyperliquid_rust_sdk = "0.6.0"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+serde_with = { version = "3.8", features = ["macros"] }
+serde_yaml = "0.9"
+thiserror = "2.0"
+tokio = { version = "1.37", features = ["rt-multi-thread", "macros", "signal", "time"] }
+tokio-stream = "0.1"
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+uuid = { version = "1.7", features = ["serde", "v4"] }
+ethers = "2.0"
+reqwest = { version = "0.13", default-features = false, features = ["json", "rustls"] }
+sha2 = "0.11"
@@ -0,0 +1,13 @@
+.PHONY: format check build test
+
+format:
+	cargo fmt
+
+check:
+	cargo clippy --no-deps -- --deny warnings
+
+build:
+	cargo build
+
+test:
+	cargo test
@@ -0,0 +1,475 @@
+# HyperliquidBench
+
+![logo.png](assets/logo.png)
+![frontend.png](assets/frontend.png)
+![leaderboard.png](assets/leaderboard.png)
+
+HyperLiquidBench is a reproducible benchmark for measuring the **operational
+competence** of Hyperliquid trading agents. It leans on the same “declare
+metrics → execute → evaluate” philosophy used in [SolanaBench](https://solana.com/news/solana-bench): prove an agent can
+route venue actions correctly, not just print PnL. The toolchain is Rust-first,
+ships deterministic datasets, and records every effect so results are auditable.
+
+* Product Deck: https://drive.google.com/file/d/1tnQpgina5jGGIV9QXQrfPQ96FWUwsiz6/view?usp=sharing
+* Demo Frontend: https://hyperliquid-bench.vercel.app/
+* YouTube Link: https://youtu.be/m8pSPQIDdxQ
+
+---
+
+## Philosophy & Goals
+
+- **Don’t trust, verify.** Agents must show evidence of correct order routing,
+cancels, transfers, and leverage changes. Each step is logged with the
+submission payload, HTTP acknowledgement, and websocket confirmation.
+- **Scoring favours breadth and discipline.** The evaluator converts effects into
+normalized signatures. A weighted **Base** score rewards unique signatures across
+domains (perp, account, risk). A **Bonus** rewards composing multiple distinct
+actions within a short nonce window (default 200 ms). A **Penalty** highlights
+spam beyond the configured per‑signature cap. Final score:
+  
+  `FINAL_SCORE = Base + Bonus − Penalty`
+- **Coverage + accuracy tracks.** The coverage track judges overall operational
+surface area. The HiaN (Haystack‑in‑a‑Needle) track verifies long-context
+precision: from a noisy prompt, execute the exact instructed actions and nothing
+else.
+- **Builder Codes are first‑class.** Every run can tag orders with a builder code
+so teams can credit routed flow. Orders are also written to `orders_routed.csv`
+for auditing.
+
+For the detailed hackathon rationale, see `docs/TECHSPEC_MASTER.md`. Coverage,
+evaluator, and HiaN implementation plans live under `docs/PLAN_*`.
+
+---
+
+## Repository Layout
+
+| Path | Description |
+| --- | --- |
+| `crates/hl-common` | Shared plan schema, action/price types, time utilities, and artifact helpers used by both CLIs. |
+| `crates/hl-runner` | Tokio CLI that loads plans, signs requests with the Hyperliquid Rust SDK, submits actions, listens to websocket channels, and writes run artifacts (`per_action.jsonl`, `ws_stream.jsonl`, `orders_routed.csv`, etc.). |
+| `crates/hl-evaluator` | CLI scorer: normalizes actions into signatures, applies domain weights and windowed bonus, emits score reports (coverage) and will host the HiaN validator. |
+| `dataset/` | Authoritative scoring config (`domains-hl.yaml`), curated coverage tasks (`tasks/*.jsonl`), and HiaN case bundles (`hian/*`). |
+| `frontend/` | Static leaderboard + trajectory explorer that consumes evaluator outputs for public sharing. |
+| `scripts/` | Convenience wrappers (`run_cov.sh`, `run_hian.sh`, `ws_dump.sh`). |
+| `docs/` | Detailed plans, TODOs, and technical specification backing every subsystem. |
+
+---
+
+## Prerequisites
+
+- Rust toolchain (1.74 or newer recommended).
+- Access to a Hyperliquid key (testnet, mainnet, or local). The runner signs
+requests via an Ethereum private key.
+- System dependencies required by `hyperliquid_rust_sdk` (OpenSSL/clang on macOS,
+`libssl-dev` on Linux).
+- Optional: Tailwind-compatible static host if you plan to serve the frontend.
+
+---
+
+## Environment Setup
+
+1. **Clone and fetch dependencies**
+   ```bash
+   git clone https://github.com/sigridjineth/hyperliquidbench.git
+   cd hyperliquidbench
+   cargo fetch
+   ```
+2. **Export required secrets** (or place them in a `.env` file; both CLIs load
+environment variables via `dotenvy`).
+   ```bash
+   export HL_PRIVATE_KEY=0xabc...                # required for signing
+   export CEREBRAS_API_KEY=csk-...               # required for gpt-oss-120b plan generation
+   export HL_BUILDER_CODE=mybuilder              # optional revenue share tag
+   export HL_API_URL=https://api.hyperliquid.com # optional override
+   export HL_WS_URL=wss://api.hyperliquid.com/ws # optional override
+   ```
+3. **Configure build tooling**
+   ```bash
+   make format  # runs cargo fmt
+   make check   # runs cargo clippy --deny warnings
+   cargo build
+   cargo test   # optional
+   ```
+   *Note:* `hl-evaluator` defaults to coverage mode; pass `hian` as the first
+   argument to run the long-context validator.
+
+---
+
+## Coverage Workflow (Step-by-Step)
+
+### 1. Choose or author a plan
+
+- Plans live in `dataset/tasks/*.jsonl`. Each line is a full plan matching
+  `hl_common::plan::Plan`. For example, the first scenario in
+  `dataset/tasks/hl_perp_basic_01.jsonl` places two ETH perp orders (ALO and
+  GTC) and cancels the last.
+- Authoring tips:
+  - Use the JSON schema documented in `docs/PLAN_3_1.md`.
+  - Prices can be absolute or mid±X% (`"mid-0.5%"`). The runner resolves `mid`
+    per coin using the info client.
+  - Insert `sleep_ms` steps to control composition windows if you need to avoid
+    coalescing actions into a single 200 ms bucket.
+
+### 2. Execute the plan with the runner
+
+#### Option A – Demo mode (no network, no key)
+
+Use this to validate the full pipeline or populate the frontend without touching live endpoints. The runner synthesizes acknowledgements and websocket effects while preserving the artifact schema expected by the evaluator and UI.
+
+```bash
+cargo run -p hl-runner --release -- \
+  --demo \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --out runs/demo
+
+cargo run -p hl-evaluator --release -- \
+  --input runs/demo/per_action.jsonl \
+  --domains dataset/domains-hl.yaml \
+  --out-dir runs/demo
+```
+
+Artifacts match the live format but `run_meta.json` includes `"demoMode": true` and synthetic websocket frames include `"demo": true` so you can flag mock data downstream.
+
+##### (Optional) Demo + LLM plan generation
+
+You can still exercise the LLM pipeline in demo mode. That lets you validate prompts, caching, and plan decoding without hitting the exchange.
+
+1. **Set credentials** (Cerebras is the benchmark default for gpt-oss-120b; OpenRouter remains usable if you explicitly select it):
+   ```bash
+   export CEREBRAS_API_KEY=csk-...
+   export BENCHMARK_MODEL_PROVIDER=cerebras
+   export BENCHMARK_MODEL_NAME=gpt-oss-120b
+   export HL_LLM_DRYRUN=1                     # skip on-venue execution entirely
+   ```
+2. **Generate a coverage plan via LLM**:
+   ```bash
+   cargo run -p hl-runner -- \
+     --plan llm:coverage \
+     --demo \
+     --llm-max-steps 3 \
+     --llm-allowed-coins BTC,ETH \
+     --llm-builder-code demo-builder
+   ```
+   The runner writes `plan.json` and `plan_raw.txt` using the LLM response, but no network calls are made because both `--demo` and `HL_LLM_DRYRUN=1` are active.
+3. **Inspect and (optionally) score**:
+   ```bash
+   RUN_DIR=$(ls -dt runs/* | head -n1)
+   cargo run -p hl-evaluator -- \
+     --input "$RUN_DIR/per_action.jsonl" \
+     --domains dataset/domains-hl.yaml \
+     --out-dir "$RUN_DIR"
+   ```
+
+`run_meta.json` records the LLM metadata (`model`, `temperature`, `prompt_hash`, cache hits) alongside the standard demo flags, giving you a reproducible artifact for each prompt test.
+
+##### Orchestrated Cerebras live run
+
+For release-comparable Eliza/Hermes/OpenClaw rows, use the orchestrator so all
+three harnesses share the same `gpt-oss-120b` request shape and latest snapshot
+contract:
+
+```bash
+HL_PRIVATE_KEY=0x... \
+CEREBRAS_API_KEY=csk-... \
+VISION_LANGUAGE_PROVIDER=local-eliza \
+VISION_LANGUAGE_MODEL=eliza-1-9b \
+VISION_LANGUAGE_TIER=eliza-1-9b \
+PYTHONPATH=. \
+python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --all-harnesses \
+  --provider cerebras \
+  --model gpt-oss-120b \
+  --force \
+  --show-incompatible
+```
+
+The normalized orchestrator path passes `--no-demo`; demo-mode Hyperliquid
+artifacts are intentionally rejected by the publishability gate.
+
+### Publishable live harness runs (testnet key + CI)
+
+Only live (`--no-demo`) rows are publishable. Demo-mode output is intentionally
+**unpublishable** — `validate-latest-publishability` rejects any row carrying
+demo/sample markers — so a real Hyperliquid signing key is the one hard
+requirement for a release-comparable run. The orchestrator enforces this with a
+runtime gate: `_has_hyperliquid_live_backend()` returns true iff `HL_PRIVATE_KEY`
+is set, and `validate-runtime-gates` fails (non-zero) for `hyperliquid_bench`
+when the key is absent.
+
+#### Get a Hyperliquid TESTNET key (low-risk)
+
+Run live execution on **testnet** first. Testnet keys are ephemeral and trade
+play funds, so a leaked or burned key costs nothing real; mainnet keys sign
+trades that move real funds.
+
+1. Open the Hyperliquid testnet UI: <https://app.hyperliquid-testnet.xyz>.
+2. Connect a fresh wallet (use a dedicated throwaway, never a personal mainnet
+   wallet) and fund it from the testnet faucet.
+3. Export the wallet's Ethereum private key (`0x…`). The runner signs requests
+   with this key via the Hyperliquid Rust SDK.
+
+Keep mainnet and testnet keys separate. `--network testnet` (the default for
+live runs here) maps to the Hyperliquid testnet base URLs; `--network mainnet`
+trades real funds.
+
+#### Local env path
+
+```bash
+# 1. Build the Rust execution crates once (required for live runs).
+cd packages/benchmarks/HyperliquidBench
+cargo build --release -p hl-runner -p hl-evaluator
+cd -
+
+# 2. Export the signing key (testnet) and the plan-generation provider key.
+export HL_PRIVATE_KEY=0x...        # testnet wallet private key
+export CEREBRAS_API_KEY=csk-...    # gpt-oss-120b plan generation
+
+# 3. Run the live harnesses (eliza, hermes, openclaw) with --no-demo,
+#    invoked from the repo root with PYTHONPATH=packages.
+PYTHONPATH=packages python -m benchmarks.orchestrator run \
+  --benchmarks hyperliquid_bench \
+  --all-harnesses \
+  --provider cerebras \
+  --model gpt-oss-120b \
+  --extra '{"no_demo": true, "network": "testnet"}' \
+  --force \
+  --show-incompatible
+
+# 4. Confirm the stored rows are publishable (rejects any demo row).
+PYTHONPATH=packages python -m benchmarks.orchestrator validate-latest-publishability \
+  --include-benchmarks hyperliquid_bench
+```
+
+`--extra '{"no_demo": true}'` is what flips the run to live execution; the
+`--demo` default (used whenever `HL_PRIVATE_KEY` is unset) produces rows the
+publishability gate discards.
+
+#### CI: `hyperliquid-bench-live.yml`
+
+The [`.github/workflows/hyperliquid-bench-live.yml`](../../../.github/workflows/hyperliquid-bench-live.yml)
+workflow runs exactly this pipeline (validate gates → live `--no-demo` run →
+publishability check) on manual dispatch.
+
+1. **Operator:** add `HL_PRIVATE_KEY` (and `CEREBRAS_API_KEY` for plan
+   generation) as GitHub **repo secrets**.
+2. Dispatch the **HyperliquidBench (live)** workflow (Actions tab → Run
+   workflow), choosing `network` (default `testnet`).
+
+With no `HL_PRIVATE_KEY` secret configured the workflow **skips gracefully**
+(emits a notice and exits 0) — it never fails CI when the key is unset. It is
+manual-dispatch only by design: live trading must be an explicit operator
+decision, never a scheduled run.
+
+#### Option B – Live network execution
+
+```bash
+cargo run -p hl-runner -- \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --network testnet \
+  --builder-code "$HL_BUILDER_CODE"
+```
+
+- The `:<N>` suffix selects a specific JSONL line (1-based). You can point to a plain JSON plan without the suffix.
+- `--network` accepts `testnet`, `mainnet`, or `local` and maps to Hyperliquid base URLs.
+- Useful environment overrides:
+  - `OUT_DIR` – force the output directory (defaults to `runs/<timestamp>`).
+  - `HL_EFFECT_TIMEOUT_MS` – overrides `--effect-timeout-ms` used when waiting for websocket confirmations.
+- Artifacts written (see `docs/PLAN_3_1.md`):
+  - `per_action.jsonl` – per step: request, ack, observed events, notes, window key.
+  - `ws_stream.jsonl` – raw websocket frames (order updates, fills, ledger updates).
+  - `orders_routed.csv` – timestamped orders with builder code attribution.
+  - `run_meta.json` – metadata (network, wallet, builder code, effect timeout).
+  - `plan.json` / `plan_raw.txt` – executed plan (pretty + raw).
+
+`scripts/run_cov.sh` wraps the two-step process (runner + evaluator) and accepts the same options. For demo runs omit `NETWORK` (or set it to `demo`) and forward `--demo` after the `--` separator:
+
+```bash
+OUT_DIR=runs/demo scripts/run_cov.sh dataset/tasks/hl_cancel_sweep_01.jsonl:1 -- \
+  --demo --builder-code "demo-builder"
+
+OUT_DIR=runs/testnet NETWORK=testnet \
+  scripts/run_cov.sh dataset/tasks/hl_cancel_sweep_01.jsonl:1 -- \
+  --builder-code "$HL_BUILDER_CODE"
+```
+
+### 3. Score the run with the evaluator
+
+```bash
+RUN_DIR=$(ls -dt runs/* | head -n1)
+cargo run -p hl-evaluator -- \
+  --input "$RUN_DIR/per_action.jsonl" \
+  --domains dataset/domains-hl.yaml \
+  --out-dir "$RUN_DIR"
+```
+
+Outputs:
+- `eval_per_action.jsonl` – normalized per-step summaries listing signatures,
+  ignored/no-op flags, reasons, and window keys.
+- `eval_score.json` – detailed score report: base/bonus/penalty, final score,
+  per-domain contributions, signature counts, unmapped signatures.
+- `unique_signatures.json` – sorted list of unique signatures observed (useful
+  for knowledge distillation and frontend displays).
+- Additional diagnostics (if enabled) appear on stdout (warnings for overlapping
+domains, missing matches, etc.).
+
+For convenience, the evaluator CLI also supports positional arguments identical
+to `scripts/run_cov.sh`. See `docs/PLAN_3_2.md` for the full argument list.
+
+---
+
+## Signature & Domain Semantics
+
+### Normalization
+
+Each confirmed action produces one or more signatures:
+- `perp.order.{TIF}:{reduceOnly}:{trigger}` (e.g., `perp.order.GTC:false:none`).
+- `perp.cancel.{scope}` (`last`, `oids`, `all`).
+- `account.usdClassTransfer.{direction}` (`toPerp`, `fromPerp`).
+- `risk.setLeverage.{coin}` (e.g., `risk.setLeverage.ETH`).
+
+Only steps with `ack.status == "ok"` and non-error statuses generate signatures.
+Rejected steps become `ignored: true`. Multiple orders in a single step produce
+a signature per accepted order.
+
+### Domain configuration (`dataset/domains-hl.yaml`)
+
+```yaml
+version: "0.1"
+per_action_window_ms: 200
+per_signature_cap: 3
+
+domains:
+  perp:
+    weight: 1.0
+    allow:
+      - "perp.order.*"
+      - "perp.cancel.*"
+  account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.*"
+  risk:
+    weight: 1.0
+    allow:
+      - "risk.setLeverage.*"
+```
+
+- Patterns use dot-separated segments with `*` as a single-segment wildcard.
+- `per_action_window_ms` controls the window size for composition bonus.
+- `per_signature_cap` defines how many times a single signature can contribute to
+  Base before penalties apply.
+- Treat updates to this file as scoring-version changes.
+
+### Scoring mechanics
+
+- **Base:** For each domain, count unique signatures that match the domain’s
+  allowlist. Multiply by the domain weight. Sum across domains.
+- **Bonus:** Group signatures by `window_key_ms` (provided by the runner as
+  floor(submit_ts_ms/window). For each bucket, add `0.25 × max(0, distinct-1)`.
+  This rewards distinct effects within the same nonce window (composition).
+- **Penalty:** For every occurrence of a signature beyond the cap, add `0.1` to
+  penalty. No-op steps (failed acknowledgements) neither add to Base nor incur
+  penalties.
+
+Refer to `docs/PLAN_3_2.md` §2–§4 for worked examples and the scoring rationale.
+
+---
+
+## Dataset & Tasks
+
+### Coverage tasks (`dataset/tasks/*.jsonl`)
+
+- `hl_perp_basic_01.jsonl` – place two ETH orders (ALO + GTC) and cancel last.
+- `hl_cancel_sweep_01.jsonl` – rest an ETH order, wait 150 ms, cancel all.
+- `hl_risk_and_account_01.jsonl` – transfer USDC to perp, set leverage, place IOC reduce-only order.
+
+Each file is documented in `dataset/tasks/README.md`. Append `:<line>` when
+referencing them from the runner. Extend the directory with your own deterministic
+scenarios to grow coverage.
+
+### HiaN cases (`dataset/hian/*`)
+
+- `case_128k/prompt.txt` – compact noisy-context seed containing a single needle
+  instruction: transfer 7.5 USDC to perps, place an ALO bid mid−1% on ETH.
+- `case_128k/ground_truth.json` – required ordered effects for PASS.
+- `case_128k/meta.json` – metadata (case ID, token estimate, prompt hash).
+
+The HiaN CLI parses these files, matches effects, and emits `eval_hian.json`
+plus diffs. You can scale prompts to the desired token count and update
+metadata accordingly.
+
+### Versioning & reproducibility
+
+- Keep datasets under git for deterministic scoring. When prompts exceed version
+control limits, store the SHA256 in `meta.json` and host the blob externally.
+- Bumping signature grammar or domain rules requires a new `version` field in
+  `domains-hl.yaml` and communication to users (see `docs/PLAN_4.md` §4.7).
+
+---
+
+## Frontend Snapshot
+
+The static frontend under `frontend/` reads two data sources:
+- `frontend/data/models.json` – leaderboard entries (agent name, rank, scores,
+  domain breakdown, artifact paths).
+- `frontend/data/samples/<run>/` – evaluator outputs (`per_action.jsonl`,
+  `eval_score.json`, `unique_signatures.json`, etc.).
+
+Entry points:
+- `index.html` – overview with leaderboard and charts (Chart.js via CDN).
+- `trajectories.html` – action log explorer; upload a run directory or pick a
+  hosted sample to inspect per-step signatures and raw details.
+- `lc.html` – domain breakdown table with links to policies and signature manifests.
+
+Serve locally:
+```bash
+cd frontend
+python -m http.server 8000
+```
+
+Deploy to any static host (Vercel, Cloudflare Pages, GitHub Pages). Update the
+JSON under `frontend/data/` whenever you publish new runs.
+
+---
+
+## Builder Code & Monetization
+
+- Specify `--builder-code <CODE>` (or `HL_BUILDER_CODE`) when running coverage.
+- When supported by the Hyperliquid SDK, orders posted through `hl-runner` will
+  attribute flow to this code. The evaluator doesn’t read the builder code but
+  it is stored in `orders_routed.csv` for downstream analytics.
+- The CSV schema (`ts, oid, coin, side, px, sz, tif, reduceOnly, builderCode`) is
+  designed for ingestion into dashboards or revenue sharing reports.
+
+---
+
+## Troubleshooting & Tips
+
+- **No websocket confirmation:** The runner records missing confirmations in the
+  `notes` field of `per_action.jsonl`. Increase `--effect-timeout-ms` if your
+  environment has higher latency.
+- **Signature unmatched:** The evaluator warns if a signature doesn’t match any
+  domain (it lands in `unmapped_signatures.json`). Update `domains-hl.yaml`
+  carefully and bump `version` for scoring changes.
+- **Penalty spikes:** Check `per_signature_counts` inside `eval_score.json` to see
+  which signatures exceeded the cap. Adjust plans or caps as needed.
+- **HiaN development:** `scripts/run_hian.sh` runs a local demo plan and invokes
+  `hl-evaluator hian`, writing `eval_hian.json` and a diff on failure.
+- **CI integration:** Use `scripts/run_cov.sh` in GitHub Actions to guard lagoon
+  runs. Persist the `runs/<ts>/` directory as an artifact for inspection.
+
+---
+
+## Further Reading
+
+- `docs/TECHSPEC_MASTER.md` – track alignment, prize narrative, and architecture.
+- `docs/PLAN_3_1.md` – runner action requirements and artifact contracts.
+- `docs/PLAN_3_2.md` – evaluator normalization and scoring rules.
+- `docs/PLAN_3_3.md` – HiaN validator design and acceptance notes.
+- `docs/PLAN_4.md` – domains, dataset layout, acceptance tests, and versioning.
+- `docs/TODO_PLAN_MASTER.md` – high-level success checklist for the hackathon.
+
+HyperLiquidBench is designed so other teams can adopt the same dataset, scoring
+config, and tooling. Contributions should keep the reproducibility guarantees
+intact and document any scoring changes clearly.
@@ -0,0 +1,6 @@
+"""
+HyperliquidBench – ElizaOS agent wrapper for the Hyperliquid trading benchmark.
+
+Run with:
+    python -m benchmarks.HyperliquidBench.eliza_agent
+"""
@@ -0,0 +1,500 @@
+"""
+CLI entry point for HyperliquidBench.
+
+Two modes are supported via ``--mode``:
+
+* ``eliza`` (default) — routes plan generation through the eliza TypeScript benchmark
+  server via ``eliza_adapter.hyperliquid.ElizaHyperliquidAgent``. The Rust
+  execution path (``hl-runner`` + ``hl-evaluator``) is reused unchanged.
+* ``deterministic`` / ``python`` — local deterministic demo plan generation,
+  retained for smoke tests and offline harness validation.
+
+Examples:
+    # Eliza TS bridge, demo (starts the benchmark server automatically)
+    python -m benchmarks.HyperliquidBench --demo
+
+    # Free-form coverage scenario with specific coins
+    python -m benchmarks.HyperliquidBench --coins ETH,BTC,SOL --max-steps 7
+
+    # Run scenarios from task files
+    python -m benchmarks.HyperliquidBench --tasks hl_perp_basic_01.jsonl
+
+    # Live testnet (requires HL_PRIVATE_KEY)
+    python -m benchmarks.HyperliquidBench --network testnet --no-demo
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import logging
+import os
+import sys
+from dataclasses import replace
+from datetime import datetime
+from pathlib import Path
+
+# Ensure the eliza-adapter package is importable for the eliza TS bridge mode.
+_ELIZA_ADAPTER_PKG = Path(__file__).resolve().parents[1] / "eliza-adapter"
+if _ELIZA_ADAPTER_PKG.exists() and str(_ELIZA_ADAPTER_PKG) not in sys.path:
+    sys.path.insert(0, str(_ELIZA_ADAPTER_PKG))
+
+EDGE_VARIANTS = (
+    "Use a conservative order size and avoid unnecessary leverage changes.",
+    "Prefer actions that are reversible in demo mode and cancel residual orders.",
+    "Exercise at least one cancellation path when the requested plan allows it.",
+    "Preserve the requested coin universe and do not introduce unrelated markets.",
+    "Route through builder-code handling if configured by the scenario.",
+    "Confirm transfer direction semantics before adding any USD class transfer step.",
+    "Use reduce-only only when the plan has an offsetting or risk-reducing intent.",
+    "Avoid market-impacting assumptions; use bounded prices or demo-safe placeholders.",
+    "Keep the plan under the scenario step budget even when adding validation actions.",
+    "Favor explicit time-in-force choices so evaluator coverage can attribute intent.",
+)
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        prog="benchmarks.HyperliquidBench",
+        description="Run HyperliquidBench scenarios through an Eliza agent",
+    )
+
+    # Mode selection: bridge-backed Eliza by default; deterministic local path
+    # remains for offline smoke tests.
+    parser.add_argument(
+        "--mode",
+        type=str,
+        default="eliza",
+        choices=["eliza", "deterministic", "python"],
+        help=(
+            "Agent backend. 'eliza' routes plan generation through the eliza "
+            "TypeScript benchmark server via eliza_adapter.hyperliquid (default). "
+            "'deterministic'/'python' use the local deterministic smoke agent."
+        ),
+    )
+
+    # Scenario selection
+    parser.add_argument(
+        "--tasks",
+        nargs="*",
+        default=None,
+        help=(
+            "Task JSONL filenames (relative to dataset/tasks/).  "
+            "If omitted, loads all task files or uses a free-form coverage scenario."
+        ),
+    )
+    parser.add_argument(
+        "--coverage",
+        action="store_true",
+        default=False,
+        help="Run a single free-form coverage scenario (agent decides the plan)",
+    )
+    parser.add_argument(
+        "--coins",
+        type=str,
+        default="ETH,BTC",
+        help="Comma-separated allowed coins (default: ETH,BTC)",
+    )
+    parser.add_argument(
+        "--max-steps",
+        type=int,
+        default=5,
+        help="Maximum steps the agent can include in a plan (default: 5)",
+    )
+    parser.add_argument(
+        "--builder-code",
+        type=str,
+        default=None,
+        help="Builder code to attach to orders",
+    )
+
+    # Execution settings
+    parser.add_argument(
+        "--demo",
+        action="store_true",
+        default=True,
+        help="Run in demo mode – no real trading (default)",
+    )
+    parser.add_argument(
+        "--no-demo",
+        action="store_true",
+        default=False,
+        help="Disable demo mode – execute on real network",
+    )
+    parser.add_argument(
+        "--network",
+        type=str,
+        default="testnet",
+        choices=["testnet", "mainnet", "local"],
+        help="Network to target (default: testnet)",
+    )
+
+    # Model settings
+    parser.add_argument(
+        "--model",
+        type=str,
+        default=None,
+        help=(
+            "Model name for plan generation. Defaults to gemma-4-31b for "
+            "Cerebras and openai/gpt-oss-120b otherwise."
+        ),
+    )
+    parser.add_argument(
+        "--temperature",
+        type=float,
+        default=0.2,
+        help="Sampling temperature (default: 0.2)",
+    )
+    parser.add_argument(
+        "--max-iterations",
+        type=int,
+        default=3,
+        help="Max iterations per scenario (default: 3)",
+    )
+
+    # Output
+    parser.add_argument(
+        "--output",
+        "--output-dir",
+        dest="output",
+        type=str,
+        default=None,
+        help=(
+            "Directory to write the aggregated result JSON file. "
+            "Defaults to <bench_root>/runs."
+        ),
+    )
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        default=False,
+        help="Enable verbose/debug logging",
+    )
+    parser.add_argument(
+        "--expand-scenarios",
+        action="store_true",
+        help="Run ten deterministic trading edge variants per selected scenario.",
+    )
+    parser.add_argument(
+        "--count-scenarios",
+        action="store_true",
+        help="Print base/edge/total scenario counts before running.",
+    )
+    parser.add_argument(
+        "--validate-scenarios",
+        action="store_true",
+        help="Validate selected scenarios and optional expansion before running.",
+    )
+
+    return parser.parse_args()
+
+
+def expand_scenarios(scenarios: list[object]) -> list[object]:
+    """Return base scenarios plus ten deterministic edge variants each."""
+
+    expanded = list(scenarios)
+    for scenario in scenarios:
+        scenario_id = str(getattr(scenario, "scenario_id", "scenario"))
+        description = str(getattr(scenario, "description", ""))
+        allowed_coins = list(getattr(scenario, "allowed_coins", []) or [])
+        for index, variant in enumerate(EDGE_VARIANTS, start=1):
+            coins = list(allowed_coins)
+            if coins and index % 2 == 0:
+                coins = [*coins[1:], coins[0]]
+            expanded.append(
+                replace(
+                    scenario,
+                    scenario_id=f"{scenario_id}__edge_{index:02d}",
+                    description=f"{description}\n\nEdge condition: {variant}",
+                    allowed_coins=coins,
+                )
+            )
+    return expanded
+
+
+def count_scenarios(scenarios: list[object], include_edge_scenarios: bool = False) -> dict[str, int]:
+    base = len(scenarios)
+    edge = base * len(EDGE_VARIANTS) if include_edge_scenarios else 0
+    return {
+        "base": base,
+        "edge": edge,
+        "edge_multiplier": len(EDGE_VARIANTS) if include_edge_scenarios else 0,
+        "total": base + edge,
+    }
+
+
+def validate_scenarios(scenarios: list[object], include_edge_scenarios: bool = False) -> None:
+    if not scenarios:
+        raise ValueError("HyperliquidBench selected scenario set is empty")
+    for index, scenario in enumerate(scenarios):
+        if not str(getattr(scenario, "scenario_id", "")).strip():
+            raise ValueError(f"scenario {index} missing scenario_id")
+        if not str(getattr(scenario, "description", "")).strip():
+            raise ValueError(f"scenario {index} missing description")
+        if int(getattr(scenario, "max_steps", 0)) <= 0:
+            raise ValueError(f"scenario {index} has non-positive max_steps")
+    if include_edge_scenarios:
+        expanded = expand_scenarios(scenarios)
+        expected = len(scenarios) * (len(EDGE_VARIANTS) + 1)
+        if len(expanded) != expected:
+            raise ValueError(f"expanded scenario count {len(expanded)} != {expected}")
+        ids = [str(getattr(scenario, "scenario_id", "")) for scenario in expanded]
+        if len(ids) != len(set(ids)):
+            raise ValueError("expanded HyperliquidBench scenarios have duplicate ids")
+
+
+def _build_results_summary(results: list[object]) -> dict[str, object]:
+    """Aggregate per-scenario results into the JSON the registry will read."""
+    scenarios_out: list[dict[str, object]] = []
+    total_score = 0.0
+    total_base = 0.0
+    total_bonus = 0.0
+    total_penalty = 0.0
+    passed = 0
+
+    for result in results:
+        evaluator = getattr(result, "evaluator", None)
+        runner = getattr(result, "runner", None)
+        scenario_id = getattr(result, "scenario_id", "")
+        error_message = getattr(result, "error_message", None)
+
+        success = bool(evaluator and getattr(evaluator, "success", False))
+        score = float(getattr(evaluator, "final_score", 0.0)) if evaluator else 0.0
+        base = float(getattr(evaluator, "base", 0.0)) if evaluator else 0.0
+        bonus = float(getattr(evaluator, "bonus", 0.0)) if evaluator else 0.0
+        penalty = float(getattr(evaluator, "penalty", 0.0)) if evaluator else 0.0
+        sigs = list(getattr(evaluator, "unique_signatures", [])) if evaluator else []
+        out_dir = getattr(runner, "out_dir", "") if runner else ""
+
+        if success:
+            passed += 1
+
+        total_score += score
+        total_base += base
+        total_bonus += bonus
+        total_penalty += penalty
+
+        scenarios_out.append({
+            "scenario_id": scenario_id,
+            "success": success,
+            "final_score": score,
+            "base": base,
+            "bonus": bonus,
+            "penalty": penalty,
+            "unique_signatures": sigs,
+            "out_dir": out_dir,
+            "error": error_message,
+        })
+
+    n = max(len(results), 1)
+    return {
+        "benchmark": "hyperliquid_bench",
+        "scenarios": scenarios_out,
+        "total_scenarios": len(results),
+        "passed_scenarios": passed,
+        "final_score": total_score / n,  # average per scenario
+        "total_score": total_score,
+        "base": total_base,
+        "bonus": total_bonus,
+        "penalty": total_penalty,
+    }
+
+
+async def _main() -> int:
+    args = _parse_args()
+
+    log_level = logging.DEBUG if args.verbose else logging.INFO
+    logging.basicConfig(
+        level=log_level,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%H:%M:%S",
+    )
+
+    # Lazy imports so --help is fast
+    from .eliza_agent import (
+        load_scenarios_from_tasks,
+        make_coverage_scenario,
+    )
+    from .types import HLBenchConfig, TradingScenario
+
+    bench_root = Path(__file__).resolve().parent
+
+    demo_mode = args.demo and not args.no_demo
+
+    provider = _detect_model_provider()
+
+    model_name = (args.model or os.environ.get("BENCHMARK_MODEL_NAME", "")).strip()
+    if not model_name:
+        model_name = _default_model_for_provider(provider)
+
+    if args.mode == "eliza":
+        _apply_model_environment(provider, model_name)
+
+    config = HLBenchConfig(
+        bench_root=bench_root,
+        demo_mode=demo_mode,
+        network=args.network,
+        builder_code=args.builder_code,
+        model_name=model_name,
+        temperature=args.temperature,
+        max_iterations=args.max_iterations,
+        verbose=args.verbose,
+    )
+
+    coins = [c.strip().upper() for c in args.coins.split(",") if c.strip()]
+    scenarios: list[TradingScenario] = []
+
+    if args.coverage:
+        scenarios.append(
+            make_coverage_scenario(
+                allowed_coins=coins,
+                max_steps=args.max_steps,
+                builder_code=args.builder_code,
+            )
+        )
+    elif args.tasks:
+        scenarios = load_scenarios_from_tasks(bench_root, task_files=args.tasks)
+    else:
+        scenarios.append(
+            make_coverage_scenario(
+                allowed_coins=coins,
+                max_steps=args.max_steps,
+                builder_code=args.builder_code,
+            )
+        )
+
+    if not scenarios:
+        logging.error("No scenarios to run")
+        return 1
+    if args.validate_scenarios:
+        validate_scenarios(scenarios, include_edge_scenarios=args.expand_scenarios)
+    scenario_counts = count_scenarios(scenarios, include_edge_scenarios=args.expand_scenarios)
+    if args.count_scenarios:
+        print(json.dumps(scenario_counts, sort_keys=True))
+    if args.expand_scenarios:
+        scenarios = expand_scenarios(scenarios)
+
+    # Pick the agent backend.
+    bridge_manager = None
+    if args.mode == "eliza":
+        from eliza_adapter.hyperliquid import ElizaHyperliquidAgent as _BridgeAgent
+        from eliza_adapter.server_manager import ElizaServerManager
+
+        bridge_manager = ElizaServerManager()
+        bridge_manager.start()
+        agent: object = _BridgeAgent(
+            config=config,
+            client=bridge_manager.client,
+            verbose=args.verbose,
+        )
+        logging.info("Using eliza TS bridge agent (eliza_adapter.hyperliquid)")
+    else:
+        from .eliza_agent import ElizaHyperliquidAgent as _PythonAgent
+
+        agent = _PythonAgent(config=config, verbose=args.verbose)
+        logging.info("Using local deterministic HyperliquidBench smoke agent")
+
+    try:
+        results = await agent.run_benchmark(scenarios=scenarios)  # type: ignore[attr-defined]
+    finally:
+        try:
+            await agent.cleanup()  # type: ignore[attr-defined]
+        finally:
+            if bridge_manager is not None:
+                bridge_manager.stop()
+
+    summary = _build_results_summary(results)
+    summary["mode"] = args.mode
+    summary["model"] = model_name
+    summary["network"] = args.network
+    summary["demo_mode"] = demo_mode
+    summary["include_edge_scenarios"] = bool(args.expand_scenarios)
+    summary["scenario_counts"] = scenario_counts
+
+    # Write the aggregated result JSON in a location the registry can locate.
+    output_dir = Path(args.output).resolve() if args.output else (bench_root / config.runs_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+    timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
+    out_file = output_dir / f"hyperliquid_bench-{args.mode}-{timestamp}.json"
+    out_file.write_text(json.dumps(summary, indent=2))
+    logging.info("Wrote aggregated results to %s", out_file)
+
+    # Print the human summary on stdout.
+    print("\n" + "=" * 60)
+    print(f"HyperliquidBench — {args.mode} mode results")
+    print("=" * 60)
+    for scenario in summary["scenarios"]:  # type: ignore[union-attr]
+        status = "PASS" if scenario["success"] else "FAIL"  # type: ignore[index]
+        print(f"\n  [{status}] {scenario['scenario_id']}")  # type: ignore[index]
+        print(
+            f"    Score: {scenario['final_score']:.3f}  "  # type: ignore[index]
+            f"(base={scenario['base']:.1f}, bonus={scenario['bonus']:.1f}, "
+            f"penalty={scenario['penalty']:.1f})"
+        )
+        if scenario["unique_signatures"]:  # type: ignore[index]
+            print(f"    Signatures: {', '.join(scenario['unique_signatures'])}")  # type: ignore[index]
+        if scenario["error"]:  # type: ignore[index]
+            print(f"    Error: {scenario['error']}")  # type: ignore[index]
+    print(f"\n  Average final_score: {summary['final_score']:.3f}")
+    print(f"  Scenarios: {summary['total_scenarios']}, Passed: {summary['passed_scenarios']}")
+    print(f"  Result file: {out_file}")
+    print("=" * 60)
+
+    if summary["passed_scenarios"] != summary["total_scenarios"]:
+        return 1
+    if not demo_mode:
+        live_signatures = [
+            sig
+            for scenario in summary["scenarios"]  # type: ignore[union-attr]
+            for sig in scenario.get("unique_signatures", [])  # type: ignore[union-attr]
+        ]
+        if not live_signatures:
+            logging.error("No confirmed live action signatures were recorded")
+            return 1
+
+    return 0
+
+
+def _detect_model_provider() -> str:
+    provider = os.environ.get("BENCHMARK_MODEL_PROVIDER", "").strip().lower()
+    if provider:
+        return provider
+    if os.environ.get("CEREBRAS_API_KEY"):
+        return "cerebras"
+    if os.environ.get("GROQ_API_KEY"):
+        return "groq"
+    if os.environ.get("OPENROUTER_API_KEY"):
+        return "openrouter"
+    if os.environ.get("OPENAI_API_KEY"):
+        return "openai"
+    return ""
+
+
+def _default_model_for_provider(provider: str) -> str:
+    if provider.strip().lower() == "cerebras":
+        return "gemma-4-31b"
+    return "openai/gpt-oss-120b"
+
+
+def _apply_model_environment(provider: str, model_name: str) -> None:
+    if provider:
+        os.environ["BENCHMARK_MODEL_PROVIDER"] = provider
+    os.environ["BENCHMARK_MODEL_NAME"] = model_name
+    os.environ["OPENAI_LARGE_MODEL"] = model_name
+    os.environ["OPENAI_SMALL_MODEL"] = model_name
+    os.environ["GROQ_LARGE_MODEL"] = model_name
+    os.environ["GROQ_SMALL_MODEL"] = model_name
+    os.environ["OPENROUTER_LARGE_MODEL"] = model_name
+    os.environ["OPENROUTER_SMALL_MODEL"] = model_name
+    os.environ["CEREBRAS_LARGE_MODEL"] = model_name
+    os.environ["CEREBRAS_SMALL_MODEL"] = model_name
+
+
+def main() -> None:
+    """Synchronous entry point."""
+    sys.exit(asyncio.run(_main()))
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,14 @@
+[package]
+name = "hl-common"
+version.workspace = true
+edition.workspace = true
+
+[dependencies]
+anyhow = { workspace = true }
+chrono = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true }
+serde_with = { workspace = true }
+thiserror = { workspace = true }
+uuid = { workspace = true }
+csv = { workspace = true }
@@ -0,0 +1,195 @@
+use std::{
+    fs::{self, File},
+    io::{BufWriter, Write},
+    path::{Path, PathBuf},
+};
+
+use anyhow::{Context, Result};
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+use crate::time::window_start_ms;
+
+const DEFAULT_WINDOW_MS: i64 = 200;
+
+#[derive(Debug, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ActionLogRecord {
+    pub step_idx: usize,
+    pub action: String,
+    pub submit_ts_ms: i64,
+    pub window_key_ms: i64,
+    pub request: Value,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub ack: Option<Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub observed: Option<Value>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub notes: Option<String>,
+}
+
+#[derive(Debug, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct RoutedOrderRecord {
+    pub ts_ms: i64,
+    pub oid: Option<u64>,
+    pub coin: String,
+    pub side: String,
+    pub px: f64,
+    pub sz: f64,
+    pub tif: String,
+    pub reduce_only: bool,
+    pub builder_code: Option<String>,
+}
+
+pub struct RunArtifacts {
+    per_action: BufWriter<File>,
+    ws_stream: BufWriter<File>,
+    routed_csv: csv::Writer<File>,
+    window_ms: i64,
+    per_action_path: PathBuf,
+    ws_stream_path: PathBuf,
+    meta_path: PathBuf,
+}
+
+impl RunArtifacts {
+    pub fn create<P: AsRef<Path>>(
+        out_dir: P,
+        plan: &Value,
+        plan_raw: Option<&str>,
+        window_ms: Option<i64>,
+    ) -> Result<Self> {
+        let out_dir = out_dir.as_ref();
+        fs::create_dir_all(out_dir)
+            .with_context(|| format!("failed to create run directory {}", out_dir.display()))?;
+
+        let per_action_path = out_dir.join("per_action.jsonl");
+        let ws_stream_path = out_dir.join("ws_stream.jsonl");
+        let routed_path = out_dir.join("orders_routed.csv");
+        let meta_path = out_dir.join("run_meta.json");
+        let plan_path = out_dir.join("plan.json");
+        let plan_raw_path = plan_raw.map(|_| out_dir.join("plan_raw.txt"));
+
+        let per_action = BufWriter::new(
+            File::create(&per_action_path)
+                .with_context(|| format!("failed to create {}", per_action_path.display()))?,
+        );
+        let ws_stream = BufWriter::new(
+            File::create(&ws_stream_path)
+                .with_context(|| format!("failed to create {}", ws_stream_path.display()))?,
+        );
+        let routed_file = File::create(&routed_path)
+            .with_context(|| format!("failed to create {}", routed_path.display()))?;
+        let mut routed_csv = csv::Writer::from_writer(routed_file);
+        routed_csv.write_record([
+            "ts",
+            "oid",
+            "coin",
+            "side",
+            "px",
+            "sz",
+            "tif",
+            "reduceOnly",
+            "builderCode",
+        ])?;
+
+        let plan_writer = File::create(&plan_path)
+            .with_context(|| format!("failed to create {}", plan_path.display()))?;
+        serde_json::to_writer_pretty(plan_writer, plan)
+            .with_context(|| format!("failed to write plan json {}", plan_path.display()))?;
+
+        if let (Some(raw), Some(raw_path)) = (plan_raw, plan_raw_path.as_ref()) {
+            let mut writer = BufWriter::new(
+                File::create(raw_path)
+                    .with_context(|| format!("failed to create {}", raw_path.display()))?,
+            );
+            writer.write_all(raw.as_bytes())?;
+        }
+
+        Ok(Self {
+            per_action,
+            ws_stream,
+            routed_csv,
+            window_ms: window_ms.unwrap_or(DEFAULT_WINDOW_MS),
+            per_action_path,
+            ws_stream_path,
+            meta_path,
+        })
+    }
+
+    pub fn log_action(&mut self, record: &ActionLogRecord) -> Result<()> {
+        serde_json::to_writer(&mut self.per_action, record).with_context(|| {
+            format!(
+                "failed to write action log to {}",
+                self.per_action_path.display()
+            )
+        })?;
+        self.per_action.write_all(b"\n")?;
+        self.per_action.flush()?;
+        Ok(())
+    }
+
+    pub fn log_ws_event(&mut self, raw: &Value) -> Result<()> {
+        serde_json::to_writer(&mut self.ws_stream, raw).with_context(|| {
+            format!(
+                "failed to write ws event to {}",
+                self.ws_stream_path.display()
+            )
+        })?;
+        self.ws_stream.write_all(b"\n")?;
+        Ok(())
+    }
+
+    pub fn log_routed_order(&mut self, record: &RoutedOrderRecord) -> Result<()> {
+        self.routed_csv.serialize(record)?;
+        self.routed_csv.flush()?;
+        Ok(())
+    }
+
+    pub fn write_meta(&self, meta: &Value) -> Result<()> {
+        let meta_file = File::create(&self.meta_path)
+            .with_context(|| format!("failed to create {}", self.meta_path.display()))?;
+        let mut writer = BufWriter::new(meta_file);
+        serde_json::to_writer_pretty(&mut writer, meta)
+            .with_context(|| format!("failed to write meta to {}", self.meta_path.display()))?;
+        writer.write_all(b"\n")?;
+        writer.flush()?;
+        Ok(())
+    }
+
+    pub fn window_ms(&self) -> i64 {
+        self.window_ms
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub fn make_action_record(
+        &self,
+        step_idx: usize,
+        action: impl Into<String>,
+        submit_ts_ms: i64,
+        request: Value,
+        ack: Option<Value>,
+        observed: Option<Value>,
+        notes: Option<String>,
+    ) -> ActionLogRecord {
+        let window_key_ms = window_start_ms(submit_ts_ms, self.window_ms);
+        ActionLogRecord {
+            step_idx,
+            action: action.into(),
+            submit_ts_ms,
+            window_key_ms,
+            request,
+            ack,
+            observed,
+            notes,
+        }
+    }
+}
+
+impl Drop for RunArtifacts {
+    fn drop(&mut self) {
+        let _ = self.per_action.flush();
+        let _ = self.ws_stream.flush();
+        let _ = self.routed_csv.flush();
+    }
+}
@@ -0,0 +1,11 @@
+pub mod artifacts;
+pub mod plan;
+pub mod sig;
+pub mod time;
+
+pub use artifacts::{ActionLogRecord, RoutedOrderRecord, RunArtifacts};
+pub use plan::{
+    load_plan_from_spec, ActionStep, CancelScope, OrderPrice, OrderSide, PerpOrder, Plan,
+};
+pub use sig::{normalize_tif, normalize_trigger, Signature};
+pub use time::{timestamp_ms, window_start_ms};
@@ -0,0 +1,374 @@
+use std::{
+    fs::File,
+    io::{BufRead, BufReader},
+    path::{Path, PathBuf},
+    str::FromStr,
+};
+
+use anyhow::{anyhow, Context, Result};
+use serde::{de, Deserialize, Deserializer, Serialize, Serializer};
+use serde_json::Value;
+
+/// Parsed representation of a runner plan.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Plan {
+    pub steps: Vec<ActionStep>,
+}
+
+impl Plan {
+    pub fn as_json(&self) -> Value {
+        serde_json::to_value(self).expect("plan must serialize")
+    }
+}
+
+/// Step variants supported by the runner.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(untagged)]
+pub enum ActionStep {
+    PerpOrders {
+        perp_orders: PerpOrdersStep,
+    },
+    CancelLast {
+        cancel_last: CancelLastStep,
+    },
+    CancelOids {
+        cancel_oids: CancelOidsStep,
+    },
+    CancelAll {
+        cancel_all: CancelAllStep,
+    },
+    UsdClassTransfer {
+        usd_class_transfer: UsdClassTransferStep,
+    },
+    SetLeverage {
+        set_leverage: SetLeverageStep,
+    },
+    Sleep {
+        sleep_ms: SleepMsStep,
+    },
+}
+
+impl ActionStep {
+    pub fn kind(&self) -> &'static str {
+        match self {
+            ActionStep::PerpOrders { .. } => "perp_orders",
+            ActionStep::CancelLast { .. } => "cancel_last",
+            ActionStep::CancelOids { .. } => "cancel_oids",
+            ActionStep::CancelAll { .. } => "cancel_all",
+            ActionStep::UsdClassTransfer { .. } => "usd_class_transfer",
+            ActionStep::SetLeverage { .. } => "set_leverage",
+            ActionStep::Sleep { .. } => "sleep_ms",
+        }
+    }
+
+    pub fn as_perp_orders(&self) -> Option<&PerpOrdersStep> {
+        match self {
+            ActionStep::PerpOrders { perp_orders } => Some(perp_orders),
+            _ => None,
+        }
+    }
+
+    pub fn as_cancel_scope(&self) -> Option<CancelScope<'_>> {
+        match self {
+            ActionStep::CancelLast { cancel_last } => Some(CancelScope::Last { cancel_last }),
+            ActionStep::CancelOids { cancel_oids } => Some(CancelScope::Oids { cancel_oids }),
+            ActionStep::CancelAll { cancel_all } => Some(CancelScope::All { cancel_all }),
+            _ => None,
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub enum CancelScope<'a> {
+    Last { cancel_last: &'a CancelLastStep },
+    Oids { cancel_oids: &'a CancelOidsStep },
+    All { cancel_all: &'a CancelAllStep },
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct PerpOrdersStep {
+    pub orders: Vec<PerpOrder>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub builder_code: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct CancelLastStep {
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub coin: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct CancelOidsStep {
+    pub coin: String,
+    pub oids: Vec<u64>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct CancelAllStep {
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub coin: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct UsdClassTransferStep {
+    pub to_perp: bool,
+    pub usdc: f64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct SetLeverageStep {
+    pub coin: String,
+    pub leverage: u32,
+    #[serde(default)]
+    pub cross: bool,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct SleepMsStep {
+    #[serde(alias = "ms")]
+    #[serde(alias = "duration_ms")]
+    pub duration_ms: u64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct PerpOrder {
+    pub coin: String,
+    #[serde(default)]
+    pub tif: PerpTif,
+    pub side: OrderSide,
+    pub sz: f64,
+    #[serde(default)]
+    pub reduce_only: bool,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub builder_code: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub cloid: Option<String>,
+    #[serde(default)]
+    pub trigger: Option<OrderTrigger>,
+    #[serde(deserialize_with = "deserialize_order_price")]
+    pub px: OrderPrice,
+}
+
+impl PerpOrder {
+    pub fn is_buy(&self) -> bool {
+        matches!(self.side, OrderSide::Buy)
+    }
+}
+
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, Default)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum PerpTif {
+    #[serde(alias = "Alo", alias = "alo")]
+    Alo,
+    #[default]
+    #[serde(alias = "Gtc", alias = "gtc")]
+    Gtc,
+    #[serde(alias = "Ioc", alias = "ioc")]
+    Ioc,
+}
+
+impl PerpTif {
+    pub fn as_sdk_str(&self) -> &'static str {
+        match self {
+            PerpTif::Alo => "Alo",
+            PerpTif::Gtc => "Gtc",
+            PerpTif::Ioc => "Ioc",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub enum OrderSide {
+    Buy,
+    Sell,
+}
+
+impl OrderSide {
+    pub fn as_bool(&self) -> bool {
+        matches!(self, OrderSide::Buy)
+    }
+}
+
+impl Serialize for OrderSide {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        match self {
+            OrderSide::Buy => serializer.serialize_str("buy"),
+            OrderSide::Sell => serializer.serialize_str("sell"),
+        }
+    }
+}
+
+impl<'de> Deserialize<'de> for OrderSide {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let value = String::deserialize(deserializer)?;
+        match value.to_ascii_lowercase().as_str() {
+            "buy" => Ok(OrderSide::Buy),
+            "sell" => Ok(OrderSide::Sell),
+            other => Err(de::Error::custom(format!("invalid side '{other}'"))),
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(tag = "kind", rename_all = "camelCase")]
+pub enum OrderTrigger {
+    None,
+    Tp { px: OrderPrice },
+    Sl { px: OrderPrice },
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub enum OrderPrice {
+    Absolute(f64),
+    MidPercent { offset_pct: f64 },
+}
+
+impl OrderPrice {
+    pub fn resolve_with_mid(&self, mid: f64) -> f64 {
+        match self {
+            OrderPrice::Absolute(px) => *px,
+            OrderPrice::MidPercent { offset_pct } => {
+                let factor = 1.0 + offset_pct / 100.0;
+                mid * factor
+            }
+        }
+    }
+}
+
+fn deserialize_order_price<'de, D>(deserializer: D) -> Result<OrderPrice, D::Error>
+where
+    D: Deserializer<'de>,
+{
+    struct PriceVisitor;
+    impl<'de> de::Visitor<'de> for PriceVisitor {
+        type Value = OrderPrice;
+
+        fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
+            formatter.write_str("a number or a string of the form 'mid±X%'")
+        }
+
+        fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>
+        where
+            E: de::Error,
+        {
+            Ok(OrderPrice::Absolute(value))
+        }
+
+        fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E>
+        where
+            E: de::Error,
+        {
+            self.visit_f64(value as f64)
+        }
+
+        fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>
+        where
+            E: de::Error,
+        {
+            self.visit_f64(value as f64)
+        }
+
+        fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
+        where
+            E: de::Error,
+        {
+            let trimmed = v.trim();
+            if let Some(rest) = trimmed.strip_prefix("mid") {
+                let rest = rest.trim();
+                let (sign, magnitude) = if let Some(v) = rest.strip_prefix('+') {
+                    (1.0_f64, v)
+                } else if let Some(v) = rest.strip_prefix('-') {
+                    (-1.0_f64, v)
+                } else {
+                    return Err(E::custom("expected '+' or '-' after 'mid'"));
+                };
+                let magnitude = magnitude.trim_end_matches('%').trim();
+                let pct = magnitude
+                    .parse::<f64>()
+                    .map_err(|_| E::custom("invalid mid% offset"))?;
+                Ok(OrderPrice::MidPercent {
+                    offset_pct: sign * pct,
+                })
+            } else {
+                let value = trimmed
+                    .parse::<f64>()
+                    .map_err(|_| E::custom("invalid absolute price"))?;
+                Ok(OrderPrice::Absolute(value))
+            }
+        }
+    }
+
+    deserializer.deserialize_any(PriceVisitor)
+}
+
+impl<'de> Deserialize<'de> for OrderPrice {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        deserialize_order_price(deserializer)
+    }
+}
+
+/// Loads a plan from a JSON file or JSONL specification.
+pub fn load_plan_from_spec(spec: &str) -> Result<Plan> {
+    let (path, selector) = split_spec(spec)?;
+    let plan_source = if let Some(index) = selector {
+        read_jsonl_entry(&path, index)?
+    } else {
+        std::fs::read_to_string(&path)
+            .with_context(|| format!("failed to read plan file {}", path.display()))?
+    };
+
+    let plan: Plan = serde_json::from_str(&plan_source)
+        .with_context(|| format!("failed to deserialize plan from {}", path.display()))?;
+    Ok(plan)
+}
+
+fn read_jsonl_entry(path: &Path, index: usize) -> Result<String> {
+    let file = File::open(path)
+        .with_context(|| format!("failed to open plan jsonl {}", path.display()))?;
+    let reader = BufReader::new(file);
+    let mut line = String::new();
+    for (idx, result) in reader.lines().enumerate() {
+        let idx = idx + 1; // 1-based for humans
+        let content =
+            result.with_context(|| format!("failed to read line {idx} from {}", path.display()))?;
+        if idx == index {
+            line = content;
+            break;
+        }
+    }
+
+    if line.is_empty() {
+        return Err(anyhow!("line {} not found in {}", index, path.display()));
+    }
+    Ok(line)
+}
+
+fn split_spec(spec: &str) -> Result<(PathBuf, Option<usize>)> {
+    let mut parts = spec.rsplitn(2, ':');
+    let trailing = parts.next().unwrap_or(spec);
+    if let Some(prefix) = parts.next() {
+        if let Ok(index) = usize::from_str(trailing) {
+            return Ok((PathBuf::from(prefix), Some(index)));
+        }
+    }
+    Ok((PathBuf::from(spec), None))
+}
@@ -0,0 +1,60 @@
+use serde::Serialize;
+
+/// Normalized coverage signature wrapper used by the evaluator.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)]
+pub struct Signature(pub String);
+
+impl Signature {
+    pub fn perp_order(tif: &str, reduce_only: bool, trigger: &str) -> Self {
+        Self(format!(
+            "perp.order.{}:{}:{}",
+            tif.to_ascii_uppercase(),
+            reduce_only,
+            trigger
+        ))
+    }
+
+    pub fn perp_cancel(scope: &str) -> Self {
+        Self(format!("perp.cancel.{}", scope))
+    }
+
+    pub fn account_usd_class_transfer(direction: &str) -> Self {
+        Self(format!("account.usdClassTransfer.{}", direction))
+    }
+
+    pub fn risk_set_leverage(coin: &str) -> Self {
+        Self(format!("risk.setLeverage.{}", coin.to_ascii_uppercase()))
+    }
+
+    pub fn into_inner(self) -> String {
+        self.0
+    }
+
+    pub fn as_str(&self) -> &str {
+        &self.0
+    }
+}
+
+pub fn normalize_tif(raw: &str) -> &'static str {
+    match raw.to_ascii_uppercase().as_str() {
+        "ALO" => "ALO",
+        "IOC" => "IOC",
+        _ => "GTC",
+    }
+}
+
+pub fn normalize_trigger(value: &serde_json::Value) -> String {
+    if let Some(trigger) = value.get("trigger") {
+        match trigger {
+            serde_json::Value::Object(map) => map
+                .get("kind")
+                .and_then(|v| v.as_str())
+                .map(|kind| kind.to_ascii_lowercase())
+                .unwrap_or_else(|| "none".to_string()),
+            serde_json::Value::String(label) => label.to_ascii_lowercase(),
+            _ => "none".to_string(),
+        }
+    } else {
+        "none".to_string()
+    }
+}
@@ -0,0 +1,14 @@
+use chrono::Utc;
+
+/// Returns the current unix timestamp in milliseconds.
+pub fn timestamp_ms() -> i64 {
+    Utc::now().timestamp_millis()
+}
+
+/// Returns the floor of the timestamp to the given window size in milliseconds.
+pub fn window_start_ms(ts_ms: i64, window_ms: i64) -> i64 {
+    if window_ms <= 0 {
+        return ts_ms;
+    }
+    (ts_ms / window_ms) * window_ms
+}
@@ -0,0 +1,18 @@
+[package]
+name = "hl-evaluator"
+version.workspace = true
+edition.workspace = true
+
+[dependencies]
+anyhow = { workspace = true }
+clap = { workspace = true }
+hl-common = { path = "../hl-common" }
+indexmap = { version = "2.5", features = ["serde"] }
+serde = { workspace = true }
+serde_json = { workspace = true }
+serde_yaml = { workspace = true }
+thiserror = { workspace = true }
+dotenvy = { workspace = true }
+
+[dev-dependencies]
+uuid = { workspace = true }
@@ -0,0 +1,795 @@
+use std::{
+    collections::{BTreeMap, BTreeSet, HashMap, HashSet},
+    fs::File,
+    io::{BufRead, BufReader, BufWriter, Write},
+    path::{Path, PathBuf},
+};
+
+use anyhow::{anyhow, Context, Result};
+use clap::Parser;
+use hl_common::{normalize_tif, normalize_trigger, ActionLogRecord, Signature};
+use indexmap::IndexMap;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use serde_yaml::Value as YamlValue;
+use thiserror::Error;
+
+const PENALTY_PER_EXTRA: f64 = 0.1;
+const BONUS_PER_EXTRA_SIGNATURE: f64 = 0.25;
+
+#[derive(Parser, Debug, Clone)]
+#[command(
+    author,
+    version,
+    about = "Evaluate HyperLiquidBench coverage runs",
+    disable_help_subcommand = true
+)]
+pub struct CoverageArgs {
+    /// Path to per_action.jsonl produced by hl-runner
+    #[arg(long)]
+    input: PathBuf,
+    /// Path to domains-hl.yaml configuration
+    #[arg(long)]
+    domains: PathBuf,
+    /// Output directory (defaults to parent directory of file input)
+    #[arg(long)]
+    out_dir: Option<PathBuf>,
+    /// Override window size in milliseconds for composition bonus
+    #[arg(long)]
+    window_ms: Option<i64>,
+    /// Override per-signature cap (defaults to value inside YAML)
+    #[arg(long)]
+    cap_per_sig: Option<usize>,
+}
+
+#[derive(Debug, Deserialize)]
+struct RawConfig {
+    #[serde(default)]
+    _version: Option<String>,
+    #[serde(default)]
+    per_action_window_ms: Option<i64>,
+    #[serde(default)]
+    per_signature_cap: Option<usize>,
+    domains: IndexMap<String, RawDomain>,
+}
+
+#[derive(Debug, Deserialize)]
+struct RawDomain {
+    weight: f64,
+    allow: Vec<String>,
+}
+
+#[derive(Debug, Clone)]
+struct DomainEntry {
+    name: String,
+    weight: f64,
+    patterns: Vec<Pattern>,
+}
+
+#[derive(Debug, Clone)]
+struct Pattern {
+    segments: Vec<PatternSegment>,
+    tail_wildcard: bool,
+}
+
+#[derive(Debug, Clone)]
+enum PatternSegment {
+    Literal(String),
+    Wildcard,
+}
+
+impl Pattern {
+    fn matches(&self, signature: &str) -> bool {
+        let sig_parts: Vec<&str> = signature.split('.').collect();
+
+        if !self.tail_wildcard && sig_parts.len() != self.segments.len() {
+            return false;
+        }
+        if self.tail_wildcard && sig_parts.len() < self.segments.len() {
+            return false;
+        }
+
+        for (idx, segment) in self.segments.iter().enumerate() {
+            if idx >= sig_parts.len() {
+                return false;
+            }
+            let value = sig_parts[idx];
+            match segment {
+                PatternSegment::Literal(lit) => {
+                    if !lit.eq_ignore_ascii_case(value) {
+                        return false;
+                    }
+                }
+                PatternSegment::Wildcard => {}
+            }
+        }
+
+        true
+    }
+}
+
+#[derive(Debug)]
+struct DomainMatcher {
+    entries: Vec<DomainEntry>,
+}
+
+impl DomainMatcher {
+    fn from_config(raw: RawConfig) -> Result<(Self, ConfigOptions)> {
+        let mut entries = Vec::new();
+        for (name, domain) in raw.domains.into_iter() {
+            if domain.allow.is_empty() {
+                return Err(anyhow!(
+                    "domain '{name}' must have at least one allow pattern"
+                ));
+            }
+            let mut patterns = Vec::new();
+            for pattern_str in domain.allow {
+                patterns.push(parse_pattern(&pattern_str).with_context(|| {
+                    format!("invalid allow pattern '{pattern_str}' in domain '{name}'")
+                })?);
+            }
+            entries.push(DomainEntry {
+                name,
+                weight: domain.weight,
+                patterns,
+            });
+        }
+
+        let opts = ConfigOptions {
+            window_ms: raw.per_action_window_ms.unwrap_or(200),
+            per_signature_cap: raw.per_signature_cap.unwrap_or(3),
+        };
+
+        Ok((DomainMatcher { entries }, opts))
+    }
+
+    fn domain_matches(&self, signature: &str) -> Vec<&DomainEntry> {
+        self.entries
+            .iter()
+            .filter(|entry| entry.patterns.iter().any(|pat| pat.matches(signature)))
+            .collect()
+    }
+
+    fn domain_for(&self, signature: &str) -> Option<&DomainEntry> {
+        let matches = self.domain_matches(signature);
+        if matches.len() > 1 {
+            let names: Vec<&str> = matches.iter().map(|d| d.name.as_str()).collect();
+            eprintln!(
+                "warning: signature '{}' matched multiple domains: {}",
+                signature,
+                names.join(", ")
+            );
+        }
+        if let Some(domain) = matches.iter().find(|domain| domain.name != "_other") {
+            return Some(*domain);
+        }
+        matches.into_iter().next()
+    }
+}
+
+fn parse_pattern(pattern: &str) -> Result<Pattern> {
+    let mut parts: Vec<&str> = pattern.split('.').collect();
+    let tail_wildcard = parts.last().map(|p| *p == "*").unwrap_or(false);
+    if tail_wildcard && parts.len() > 1 {
+        parts.pop();
+    }
+
+    let segments = parts
+        .into_iter()
+        .map(|part| {
+            if part == "*" {
+                Ok(PatternSegment::Wildcard)
+            } else if part.is_empty() {
+                Err(anyhow!("pattern segment cannot be empty"))
+            } else {
+                Ok(PatternSegment::Literal(part.to_string()))
+            }
+        })
+        .collect::<Result<Vec<_>>>()?;
+
+    Ok(Pattern {
+        segments,
+        tail_wildcard,
+    })
+}
+
+#[derive(Debug, Clone)]
+struct ConfigOptions {
+    window_ms: i64,
+    per_signature_cap: usize,
+}
+
+#[derive(Error, Debug)]
+enum NormalizeError {
+    #[error("missing acknowledgement")]
+    MissingAck,
+    #[error("ack status not ok")]
+    AckNotOk,
+    #[error("missing request payload")]
+    MissingRequest,
+    #[error("no effectful actions detected")]
+    NoEffect,
+    #[error("ack missing status entries for some orders")]
+    IncompleteAck,
+    #[error("unsupported action '{0}'")]
+    UnsupportedAction(String),
+}
+
+#[derive(Debug, Serialize)]
+#[serde(rename_all = "camelCase")]
+struct EvalActionRecord {
+    step_idx: usize,
+    action: String,
+    submit_ts_ms: i64,
+    window_key_ms: i64,
+    signatures: Vec<String>,
+    ignored: bool,
+    reason: Option<String>,
+}
+
+pub struct ScoreState<'a> {
+    matcher: &'a DomainMatcher,
+    cap_per_signature: usize,
+    window_ms: i64,
+    signature_counts: HashMap<String, usize>,
+    domain_uniques: HashMap<&'a str, HashSet<String>>,
+    window_signatures: BTreeMap<i64, HashSet<String>>,
+    all_signatures: BTreeSet<String>,
+    penalty: f64,
+    unmapped_signatures: HashSet<String>,
+}
+
+impl<'a> ScoreState<'a> {
+    fn new(matcher: &'a DomainMatcher, cap_per_signature: usize, window_ms: i64) -> Self {
+        let mut domain_uniques = HashMap::new();
+        for domain in &matcher.entries {
+            domain_uniques.insert(domain.name.as_str(), HashSet::new());
+        }
+        Self {
+            matcher,
+            cap_per_signature,
+            window_ms,
+            signature_counts: HashMap::new(),
+            domain_uniques,
+            window_signatures: BTreeMap::new(),
+            all_signatures: BTreeSet::new(),
+            penalty: 0.0,
+            unmapped_signatures: HashSet::new(),
+        }
+    }
+
+    fn incorporate(&mut self, action: &EvalActionRecord) {
+        if action.signatures.is_empty() {
+            return;
+        }
+        let window_entry = self
+            .window_signatures
+            .entry(action.window_key_ms)
+            .or_default();
+
+        for signature in &action.signatures {
+            window_entry.insert(signature.clone());
+            self.all_signatures.insert(signature.clone());
+
+            let counter = self.signature_counts.entry(signature.clone()).or_insert(0);
+            *counter += 1;
+            if *counter <= self.cap_per_signature {
+                if let Some(domain) = self.matcher.domain_for(signature) {
+                    if domain.name == "_other" {
+                        self.unmapped_signatures.insert(signature.clone());
+                    } else if let Some(set) = self.domain_uniques.get_mut(domain.name.as_str()) {
+                        set.insert(signature.clone());
+                    }
+                } else {
+                    self.unmapped_signatures.insert(signature.clone());
+                }
+            } else {
+                self.penalty += PENALTY_PER_EXTRA;
+            }
+        }
+    }
+
+    fn finalize(&self) -> ScoreReport {
+        let mut per_domain = Vec::new();
+        let mut base_total = 0.0;
+        for domain in &self.matcher.entries {
+            let uniques = self
+                .domain_uniques
+                .get(domain.name.as_str())
+                .cloned()
+                .unwrap_or_default();
+            let unique_count = uniques.len() as f64;
+            let contribution = domain.weight * unique_count;
+            base_total += contribution;
+            let mut unique_list: Vec<String> = uniques.into_iter().collect();
+            unique_list.sort();
+            per_domain.push(DomainBreakdown {
+                name: domain.name.clone(),
+                weight: domain.weight,
+                unique_signatures: unique_list,
+                unique_count: unique_count as usize,
+                contribution,
+            });
+        }
+
+        let mut bonus_total = 0.0;
+        for signatures in self.window_signatures.values() {
+            let distinct = signatures.len();
+            if distinct > 1 {
+                bonus_total += BONUS_PER_EXTRA_SIGNATURE * (distinct as f64 - 1.0);
+            }
+        }
+
+        let unique_signatures: Vec<String> = self.all_signatures.iter().cloned().collect();
+        let mut unmapped: Vec<String> = self.unmapped_signatures.iter().cloned().collect();
+        unmapped.sort();
+        let final_score = base_total + bonus_total - self.penalty;
+
+        ScoreReport {
+            final_score,
+            base: base_total,
+            bonus: bonus_total,
+            penalty: self.penalty,
+            per_domain,
+            unique_signatures,
+            cap_per_signature: self.cap_per_signature,
+            window_ms: self.window_ms,
+            unmapped_signatures: unmapped,
+        }
+    }
+}
+
+#[derive(Debug, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct DomainBreakdown {
+    name: String,
+    weight: f64,
+    unique_signatures: Vec<String>,
+    unique_count: usize,
+    contribution: f64,
+}
+
+#[derive(Debug, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ScoreReport {
+    pub final_score: f64,
+    pub base: f64,
+    pub bonus: f64,
+    pub penalty: f64,
+    pub per_domain: Vec<DomainBreakdown>,
+    pub unique_signatures: Vec<String>,
+    pub cap_per_signature: usize,
+    pub window_ms: i64,
+    pub unmapped_signatures: Vec<String>,
+}
+
+pub fn run(args: &CoverageArgs) -> Result<ScoreReport> {
+    let domains_raw: RawConfig = load_domains(&args.domains)?;
+    let (matcher, defaults) = DomainMatcher::from_config(domains_raw)?;
+
+    let window_ms = args.window_ms.unwrap_or(defaults.window_ms);
+    let cap_per_signature = args.cap_per_sig.unwrap_or(defaults.per_signature_cap);
+    if window_ms <= 0 {
+        return Err(anyhow!("window_ms must be positive"));
+    }
+    if cap_per_signature == 0 {
+        return Err(anyhow!("cap_per_sig must be positive"));
+    }
+
+    let out_dir = args
+        .out_dir
+        .clone()
+        .or_else(|| args.input.parent().map(|p| p.to_path_buf()))
+        .ok_or_else(|| anyhow!("could not determine output directory"))?;
+
+    std::fs::create_dir_all(&out_dir)
+        .with_context(|| format!("failed to create output directory {}", out_dir.display()))?;
+
+    let input = File::open(&args.input)
+        .with_context(|| format!("failed to open {}", args.input.display()))?;
+    let reader = BufReader::new(input);
+
+    let eval_path = out_dir.join("eval_per_action.jsonl");
+    let eval_file = File::create(&eval_path)
+        .with_context(|| format!("failed to create {}", eval_path.display()))?;
+    let mut eval_writer = BufWriter::new(eval_file);
+
+    let mut state = ScoreState::new(&matcher, cap_per_signature, window_ms);
+
+    for (line_no, line) in reader.lines().enumerate() {
+        let line = line.with_context(|| format!("failed to read line {}", line_no + 1))?;
+        if line.trim().is_empty() {
+            continue;
+        }
+        let record: ActionLogRecord = serde_json::from_str(&line)
+            .with_context(|| format!("failed to parse ActionLogRecord on line {}", line_no + 1))?;
+        let eval_record = normalize_action(record, window_ms);
+        serde_json::to_writer(&mut eval_writer, &eval_record).with_context(|| {
+            format!(
+                "failed to write eval_per_action.jsonl record for step {}",
+                eval_record.step_idx
+            )
+        })?;
+        eval_writer.write_all(b"\n")?;
+        if !eval_record.ignored {
+            state.incorporate(&eval_record);
+        }
+    }
+
+    eval_writer.flush()?;
+
+    let report = state.finalize();
+    let score_path = out_dir.join("eval_score.json");
+    serde_json::to_writer_pretty(
+        File::create(&score_path)
+            .with_context(|| format!("failed to create {}", score_path.display()))?,
+        &report,
+    )?;
+
+    let unique_path = out_dir.join("unique_signatures.json");
+    serde_json::to_writer_pretty(
+        File::create(&unique_path)
+            .with_context(|| format!("failed to create {}", unique_path.display()))?,
+        &report.unique_signatures,
+    )?;
+
+    let unmapped_path = out_dir.join("unmapped_signatures.json");
+    serde_json::to_writer_pretty(
+        File::create(&unmapped_path)
+            .with_context(|| format!("failed to create {}", unmapped_path.display()))?,
+        &report.unmapped_signatures,
+    )?;
+
+    Ok(report)
+}
+
+fn load_domains(path: &Path) -> Result<RawConfig> {
+    let file = File::open(path).with_context(|| format!("failed to open {}", path.display()))?;
+    let yaml: YamlValue = serde_yaml::from_reader(file)
+        .with_context(|| format!("failed to parse YAML {}", path.display()))?;
+    let config: RawConfig = serde_yaml::from_value(yaml)?;
+    Ok(config)
+}
+
+fn normalize_action(mut record: ActionLogRecord, window_ms: i64) -> EvalActionRecord {
+    let window_key_ms = (record.submit_ts_ms / window_ms) * window_ms;
+    record.window_key_ms = window_key_ms;
+
+    let (signatures, reason) = match record.action.as_str() {
+        "perp_orders" => normalize_perp_orders(&record),
+        "cancel_last" => normalize_cancel(&record, "last"),
+        "cancel_oids" => normalize_cancel(&record, "oids"),
+        "cancel_all" => normalize_cancel(&record, "all"),
+        "usd_class_transfer" => normalize_transfer(&record),
+        "set_leverage" => normalize_leverage(&record),
+        other => (
+            Vec::new(),
+            Some(NormalizeError::UnsupportedAction(other.to_string())),
+        ),
+    };
+
+    let (ignored, reason_str) = match reason {
+        Some(err) if signatures.is_empty() => (true, Some(err.to_string())),
+        Some(err) => (false, Some(err.to_string())),
+        None => (signatures.is_empty(), None),
+    };
+
+    EvalActionRecord {
+        step_idx: record.step_idx,
+        action: record.action,
+        submit_ts_ms: record.submit_ts_ms,
+        window_key_ms,
+        signatures,
+        ignored,
+        reason: reason_str,
+    }
+}
+
+fn normalize_perp_orders(record: &ActionLogRecord) -> (Vec<String>, Option<NormalizeError>) {
+    let ack = match record.ack.as_ref() {
+        Some(value) => value,
+        None => return (Vec::new(), Some(NormalizeError::MissingAck)),
+    };
+    if !ack_status_ok(ack) {
+        return (Vec::new(), Some(NormalizeError::AckNotOk));
+    }
+
+    let orders = record
+        .request
+        .get("perp_orders")
+        .and_then(|v| v.get("orders"))
+        .and_then(|v| v.as_array())
+        .cloned()
+        .unwrap_or_default();
+
+    if orders.is_empty() {
+        return (Vec::new(), Some(NormalizeError::MissingRequest));
+    }
+
+    let order_statuses = ack
+        .get("data")
+        .and_then(|d| d.get("statuses"))
+        .and_then(|s| s.as_array())
+        .cloned()
+        .unwrap_or_default();
+
+    let mut signatures = Vec::new();
+    let mut incomplete = false;
+    for (idx, order) in orders.iter().enumerate() {
+        let status_kind = order_statuses
+            .get(idx)
+            .and_then(|v| v.get("kind"))
+            .and_then(|v| v.as_str());
+
+        match status_kind {
+            Some(kind) if kind.eq_ignore_ascii_case("error") => continue,
+            Some(_) => {}
+            None => {
+                incomplete = true;
+                continue;
+            }
+        }
+
+        let tif_raw = order.get("tif").and_then(|v| v.as_str()).unwrap_or("GTC");
+        let tif = normalize_tif(tif_raw);
+        let reduce_only = order
+            .get("reduceOnly")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(false);
+        let trigger = normalize_trigger(order);
+        let signature = Signature::perp_order(tif, reduce_only, trigger.as_str()).into_inner();
+        signatures.push(signature);
+    }
+
+    if signatures.is_empty() {
+        if incomplete {
+            (signatures, Some(NormalizeError::IncompleteAck))
+        } else {
+            (signatures, Some(NormalizeError::NoEffect))
+        }
+    } else if incomplete {
+        (signatures, Some(NormalizeError::IncompleteAck))
+    } else {
+        (signatures, None)
+    }
+}
+
+fn normalize_cancel(
+    record: &ActionLogRecord,
+    scope: &str,
+) -> (Vec<String>, Option<NormalizeError>) {
+    let ack = match record.ack.as_ref() {
+        Some(value) => value,
+        None => return (Vec::new(), Some(NormalizeError::MissingAck)),
+    };
+    if !ack_status_ok(ack) {
+        return (Vec::new(), Some(NormalizeError::AckNotOk));
+    }
+    let signature = Signature::perp_cancel(scope).into_inner();
+    (vec![signature], None)
+}
+
+fn normalize_transfer(record: &ActionLogRecord) -> (Vec<String>, Option<NormalizeError>) {
+    let ack = match record.ack.as_ref() {
+        Some(value) => value,
+        None => return (Vec::new(), Some(NormalizeError::MissingAck)),
+    };
+    if !ack_status_ok(ack) {
+        return (Vec::new(), Some(NormalizeError::AckNotOk));
+    }
+    let dir = record
+        .request
+        .get("usd_class_transfer")
+        .and_then(|v| v.get("toPerp"))
+        .and_then(|v| v.as_bool())
+        .map(|to_perp| if to_perp { "toPerp" } else { "fromPerp" })
+        .unwrap_or("toPerp");
+    let signature = Signature::account_usd_class_transfer(dir).into_inner();
+    (vec![signature], None)
+}
+
+fn normalize_leverage(record: &ActionLogRecord) -> (Vec<String>, Option<NormalizeError>) {
+    let ack = match record.ack.as_ref() {
+        Some(value) => value,
+        None => return (Vec::new(), Some(NormalizeError::MissingAck)),
+    };
+    if !ack_status_ok(ack) {
+        return (Vec::new(), Some(NormalizeError::AckNotOk));
+    }
+    let coin = record
+        .request
+        .get("set_leverage")
+        .and_then(|v| v.get("coin"))
+        .and_then(|v| v.as_str())
+        .unwrap_or("UNKNOWN");
+    let signature = Signature::risk_set_leverage(coin).into_inner();
+    (vec![signature], None)
+}
+
+fn ack_status_ok(ack: &Value) -> bool {
+    ack.get("status")
+        .and_then(|v| v.as_str())
+        .map(|status| status.eq_ignore_ascii_case("ok"))
+        .unwrap_or(false)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn make_ack_ok(kind: &str) -> Value {
+        serde_json::json!({
+            "status": "ok",
+            "data": {
+                "statuses": [{"kind": kind}]
+            }
+        })
+    }
+
+    #[test]
+    fn pattern_matching() {
+        let pat = parse_pattern("perp.order.*").unwrap();
+        assert!(pat.matches("perp.order.GTC:false:none"));
+        assert!(pat.matches("PERP.ORDER.gtc:false:none"));
+        assert!(!pat.matches("perp.cancel.last"));
+    }
+
+    #[test]
+    fn pattern_tail_wildcard() {
+        let pat = parse_pattern("account.*").unwrap();
+        assert!(pat.matches("account.usdClassTransfer.toPerp"));
+        assert!(pat.matches("account.usdClassTransfer.toPerp.extra"));
+    }
+
+    #[test]
+    fn normalize_perp_order_success() {
+        let record = ActionLogRecord {
+            step_idx: 1,
+            action: "perp_orders".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            request: serde_json::json!({
+                "perp_orders": {
+                    "orders": [{
+                        "tif": "Gtc",
+                        "reduceOnly": false,
+                        "coin": "ETH"
+                    }]
+                }
+            }),
+            ack: Some(make_ack_ok("resting")),
+            observed: None,
+            notes: None,
+        };
+        let (signatures, reason) = super::normalize_perp_orders(&record);
+        assert!(reason.is_none());
+        assert_eq!(signatures, vec!["perp.order.GTC:false:none".to_string()]);
+    }
+
+    #[test]
+    fn normalize_perp_order_error_filtered() {
+        let record = ActionLogRecord {
+            step_idx: 1,
+            action: "perp_orders".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            request: serde_json::json!({
+                "perp_orders": {
+                    "orders": [{"tif": "Gtc", "reduceOnly": false }]
+                }
+            }),
+            ack: Some(make_ack_ok("error")),
+            observed: None,
+            notes: None,
+        };
+        let (signatures, reason) = super::normalize_perp_orders(&record);
+        assert!(signatures.is_empty());
+        assert!(matches!(reason, Some(NormalizeError::NoEffect)));
+    }
+
+    #[test]
+    fn normalize_perp_order_missing_status() {
+        let record = ActionLogRecord {
+            step_idx: 1,
+            action: "perp_orders".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            request: serde_json::json!({
+                "perp_orders": {
+                    "orders": [{"tif": "Gtc", "reduceOnly": false }]
+                }
+            }),
+            ack: Some(serde_json::json!({
+                "status": "ok",
+                "data": { "statuses": [] }
+            })),
+            observed: None,
+            notes: None,
+        };
+        let (signatures, reason) = super::normalize_perp_orders(&record);
+        assert!(signatures.is_empty());
+        assert!(matches!(reason, Some(NormalizeError::IncompleteAck)));
+    }
+
+    #[test]
+    fn normalize_perp_order_partial_ack() {
+        let record = ActionLogRecord {
+            step_idx: 1,
+            action: "perp_orders".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            request: serde_json::json!({
+                "perp_orders": {
+                    "orders": [
+                        {"tif": "Gtc", "reduceOnly": false},
+                        {"tif": "Ioc", "reduceOnly": false}
+                    ]
+                }
+            }),
+            ack: Some(serde_json::json!({
+                "status": "ok",
+                "data": { "statuses": [{"kind": "resting"}] }
+            })),
+            observed: None,
+            notes: None,
+        };
+        let (signatures, reason) = super::normalize_perp_orders(&record);
+        assert_eq!(signatures, vec!["perp.order.GTC:false:none".to_string()]);
+        assert!(matches!(reason, Some(NormalizeError::IncompleteAck)));
+    }
+
+    #[test]
+    fn score_state_bonus() {
+        let matcher = DomainMatcher {
+            entries: vec![DomainEntry {
+                name: "perp".to_string(),
+                weight: 1.0,
+                patterns: vec![parse_pattern("perp.order.*").unwrap()],
+            }],
+        };
+        let mut state = ScoreState::new(&matcher, 3, 200);
+        let action = EvalActionRecord {
+            step_idx: 0,
+            action: "perp_orders".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            signatures: vec![
+                "perp.order.GTC:false:none".to_string(),
+                "perp.order.ALO:false:none".to_string(),
+            ],
+            ignored: false,
+            reason: None,
+        };
+        state.incorporate(&action);
+        let report = state.finalize();
+        assert_eq!(report.bonus, BONUS_PER_EXTRA_SIGNATURE);
+        assert!((report.final_score - (2.0 + BONUS_PER_EXTRA_SIGNATURE)).abs() < 1e-6);
+    }
+
+    #[test]
+    fn unmapped_signatures_recorded() {
+        let matcher = DomainMatcher {
+            entries: vec![DomainEntry {
+                name: "perp".to_string(),
+                weight: 1.0,
+                patterns: vec![parse_pattern("perp.order.*").unwrap()],
+            }],
+        };
+        let mut state = ScoreState::new(&matcher, 3, 200);
+        let action = EvalActionRecord {
+            step_idx: 0,
+            action: "unknown".to_string(),
+            submit_ts_ms: 0,
+            window_key_ms: 0,
+            signatures: vec!["account.someNewAction".to_string()],
+            ignored: false,
+            reason: None,
+        };
+        state.incorporate(&action);
+        let report = state.finalize();
+        assert_eq!(
+            report.unmapped_signatures,
+            vec!["account.someNewAction".to_string()]
+        );
+    }
+}
@@ -0,0 +1,28 @@
+mod coverage;
+mod hian;
+
+use anyhow::Result;
+use clap::Parser;
+use std::ffi::OsString;
+
+fn main() -> Result<()> {
+    dotenvy::dotenv().ok();
+
+    let mut args: Vec<OsString> = std::env::args_os().collect();
+    if args.get(1).is_some_and(|arg| arg == "hian") {
+        args.remove(1);
+        let hian_args = hian::HianArgs::parse_from(args);
+        let report = hian::run(&hian_args)?;
+        println!("{}", if report.result.pass { "PASS" } else { "FAIL" });
+        println!(
+            "eval_hian={}",
+            report.out_dir.join("eval_hian.json").display()
+        );
+        return Ok(());
+    }
+
+    let coverage_args = coverage::CoverageArgs::parse();
+    let report = coverage::run(&coverage_args)?;
+    println!("FINAL_SCORE={:.3}", report.final_score);
+    Ok(())
+}
@@ -0,0 +1,24 @@
+[package]
+name = "hl-runner"
+version.workspace = true
+edition.workspace = true
+
+[dependencies]
+anyhow = { workspace = true }
+chrono = { workspace = true }
+clap = { workspace = true }
+csv = { workspace = true }
+dotenvy = { workspace = true }
+futures = { workspace = true }
+hl-common = { path = "../hl-common" }
+hyperliquid_rust_sdk = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }
+tokio = { workspace = true }
+tokio-stream = { workspace = true }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+uuid = { workspace = true }
+ethers = { workspace = true }
+reqwest = { workspace = true }
+sha2 = { workspace = true }
@@ -0,0 +1,255 @@
+mod openrouter;
+mod plan_decode;
+mod prompts;
+
+use std::{
+    env,
+    fs::{self, File},
+    path::PathBuf,
+};
+
+use anyhow::{anyhow, Context, Result};
+use hl_common::plan::{ActionStep, Plan};
+use openrouter::{
+    build_cached_payload, cache_filename, hash_prompt, parse_cached_payload, OpenRouter,
+    OpenRouterConfig,
+};
+use prompts::{coverage_prompts, hian_prompts, CoveragePrompt, HianPrompt};
+use serde::Serialize;
+use serde_json::Value;
+
+const OPENROUTER_ENDPOINT: &str = "https://openrouter.ai/api/v1/chat/completions";
+const USER_AGENT: &str = "HyperLiquidBenchRunner/0.1";
+const LLM_TITLE: &str = "HyperLiquidBench";
+const MAX_ORDER_SIZE: f64 = 1.0;
+const MIN_ORDER_SIZE: f64 = 0.0001;
+const MAX_LEVERAGE: u32 = 20;
+
+#[derive(Debug)]
+pub enum LlmPlanSpec {
+    Coverage,
+    Hian(PathBuf),
+}
+
+impl LlmPlanSpec {
+    pub fn parse(spec: &str) -> Option<Self> {
+        if let Some(rest) = spec.strip_prefix("llm:") {
+            if rest.eq_ignore_ascii_case("coverage") {
+                Some(LlmPlanSpec::Coverage)
+            } else {
+                rest.strip_prefix("hian:")
+                    .map(|path| LlmPlanSpec::Hian(PathBuf::from(path)))
+            }
+        } else {
+            None
+        }
+    }
+}
+
+#[derive(Clone)]
+pub struct LlmOptions {
+    pub api_key: String,
+    pub model: String,
+    pub temperature: f32,
+    pub top_p: f32,
+    pub max_output_tokens: u32,
+    pub max_steps: u32,
+    pub allowed_coins: Vec<String>,
+    pub default_builder_code: Option<String>,
+    pub cache_dir: Option<PathBuf>,
+    pub dry_run: bool,
+}
+
+#[derive(Debug, Serialize, Clone)]
+pub struct LlmMeta {
+    pub model: String,
+    pub temperature: f32,
+    pub top_p: f32,
+    pub max_output_tokens: u32,
+    pub max_steps: u32,
+    pub allowed_coins: Vec<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub default_builder_code: Option<String>,
+    pub prompt_hash: String,
+    pub cached: bool,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub usage: Option<openrouter::Usage>,
+}
+
+pub struct PlanResult {
+    pub plan: Plan,
+    pub raw: String,
+    pub meta: LlmMeta,
+}
+
+pub async fn generate_plan(spec: LlmPlanSpec, opts: &LlmOptions) -> Result<PlanResult> {
+    let system_user = match spec {
+        LlmPlanSpec::Coverage => {
+            let ctx = CoveragePrompt {
+                max_steps: opts.max_steps,
+                allowed_coins: &opts.allowed_coins,
+                builder_code: opts.default_builder_code.as_deref(),
+                network: "testnet",
+            };
+            coverage_prompts(&ctx)
+        }
+        LlmPlanSpec::Hian(ref path) => {
+            let context_text = fs::read_to_string(path)
+                .with_context(|| format!("failed to read HiaN context file {}", path.display()))?;
+            let ctx = HianPrompt {
+                max_steps: opts.max_steps,
+                allowed_coins: &opts.allowed_coins,
+                builder_code: opts.default_builder_code.as_deref(),
+                context: &context_text,
+            };
+            hian_prompts(&ctx)
+        }
+    };
+
+    let (system, user) = system_user;
+    let prompt_hash = hash_prompt(&opts.model, &system, &user, opts.temperature, opts.top_p);
+
+    let mut was_cached = false;
+    let completion = if let Some(ref cache_dir) = opts.cache_dir {
+        fs::create_dir_all(cache_dir)
+            .with_context(|| format!("failed to create cache directory {}", cache_dir.display()))?;
+        let cache_path = cache_dir.join(cache_filename(&prompt_hash));
+        if cache_path.exists() {
+            was_cached = true;
+            let cached_value: Value =
+                serde_json::from_reader(File::open(&cache_path).with_context(|| {
+                    format!("failed to open cache file {}", cache_path.display())
+                })?)
+                .context("failed to read cached completion")?;
+            parse_cached_payload(cached_value).context("failed to parse cached completion")?
+        } else {
+            let completion = request_completion(&system, &user, opts).await?;
+            let payload = build_cached_payload(&completion.content, completion.usage.as_ref());
+            serde_json::to_writer_pretty(
+                File::create(&cache_path).with_context(|| {
+                    format!("failed to create cache file {}", cache_path.display())
+                })?,
+                &payload,
+            )
+            .context("failed to persist cached completion")?;
+            completion
+        }
+    } else {
+        request_completion(&system, &user, opts).await?
+    };
+
+    let plan = plan_decode::decode_plan(&completion.content, opts.max_steps)?;
+    let mut plan = plan;
+    sanitize_plan(&mut plan, opts)?;
+
+    let meta = LlmMeta {
+        model: opts.model.clone(),
+        temperature: opts.temperature,
+        top_p: opts.top_p,
+        max_output_tokens: opts.max_output_tokens,
+        max_steps: opts.max_steps,
+        allowed_coins: opts.allowed_coins.clone(),
+        default_builder_code: opts.default_builder_code.clone(),
+        prompt_hash,
+        cached: was_cached,
+        usage: completion.usage.clone(),
+    };
+
+    Ok(PlanResult {
+        plan,
+        raw: completion.content.trim().to_string(),
+        meta,
+    })
+}
+
+async fn request_completion(
+    system: &str,
+    user: &str,
+    opts: &LlmOptions,
+) -> Result<openrouter::Completion> {
+    let config = OpenRouterConfig {
+        endpoint: OPENROUTER_ENDPOINT.to_string(),
+        api_key: opts.api_key.clone(),
+        model: opts.model.clone(),
+        temperature: opts.temperature,
+        top_p: opts.top_p,
+        max_tokens: opts.max_output_tokens,
+        title: LLM_TITLE.to_string(),
+        user_agent: USER_AGENT.to_string(),
+    };
+    let client = OpenRouter::new(config)?;
+    client.complete(system, user).await
+}
+
+fn sanitize_plan(plan: &mut Plan, opts: &LlmOptions) -> Result<()> {
+    for step in &mut plan.steps {
+        match step {
+            ActionStep::PerpOrders { perp_orders } => {
+                if perp_orders.builder_code.is_none() {
+                    if let Some(default) = opts.default_builder_code.as_ref() {
+                        perp_orders.builder_code = Some(default.clone());
+                    }
+                }
+                for order in &mut perp_orders.orders {
+                    if order.sz <= 0.0 {
+                        return Err(anyhow!("order size must be positive"));
+                    }
+                    if order.sz > MAX_ORDER_SIZE || order.sz < MIN_ORDER_SIZE {
+                        return Err(anyhow!(
+                            "order size {} must be between {} and {}",
+                            order.sz,
+                            MIN_ORDER_SIZE,
+                            MAX_ORDER_SIZE
+                        ));
+                    }
+                    if let Some(default) = opts.default_builder_code.as_ref() {
+                        if order.builder_code.is_none() {
+                            order.builder_code = Some(default.clone());
+                        }
+                    }
+                    order.trigger = None;
+                    if !opts
+                        .allowed_coins
+                        .iter()
+                        .any(|coin| coin.eq_ignore_ascii_case(&order.coin))
+                    {
+                        return Err(anyhow!("coin {} not allowed", order.coin));
+                    }
+                    order.coin = order.coin.to_uppercase();
+                }
+            }
+            ActionStep::SetLeverage { set_leverage } => {
+                if set_leverage.leverage == 0 || set_leverage.leverage > MAX_LEVERAGE {
+                    return Err(anyhow!(
+                        "leverage {} must be between 1 and {}",
+                        set_leverage.leverage,
+                        MAX_LEVERAGE
+                    ));
+                }
+            }
+            _ => {}
+        }
+    }
+    Ok(())
+}
+
+pub fn parse_allowed_coins(raw: &str) -> Vec<String> {
+    raw.split(',')
+        .filter_map(|part| {
+            let trimmed = part.trim();
+            if trimmed.is_empty() {
+                None
+            } else {
+                Some(trimmed.to_ascii_uppercase())
+            }
+        })
+        .collect()
+}
+
+pub fn discover_cache_dir() -> Option<PathBuf> {
+    env::var_os("HL_LLM_CACHE_DIR").map(PathBuf::from)
+}
+
+pub fn dry_run_enabled() -> bool {
+    env::var("HL_LLM_DRYRUN").is_ok()
+}
@@ -0,0 +1,158 @@
+use std::time::Duration;
+
+use anyhow::{anyhow, Context, Result};
+use reqwest::{Client, StatusCode};
+use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+
+pub struct OpenRouterConfig {
+    pub endpoint: String,
+    pub api_key: String,
+    pub model: String,
+    pub temperature: f32,
+    pub top_p: f32,
+    pub max_tokens: u32,
+    pub title: String,
+    pub user_agent: String,
+}
+
+pub struct OpenRouter {
+    client: Client,
+    config: OpenRouterConfig,
+}
+
+impl OpenRouter {
+    pub fn new(config: OpenRouterConfig) -> Result<Self> {
+        let client = Client::builder()
+            .timeout(Duration::from_secs(30))
+            .build()
+            .context("failed to build reqwest client")?;
+        Ok(Self { client, config })
+    }
+
+    pub async fn complete(&self, system: &str, user: &str) -> Result<Completion> {
+        let body = json!({
+            "model": self.config.model,
+            "temperature": self.config.temperature,
+            "top_p": self.config.top_p,
+            "max_tokens": self.config.max_tokens,
+            "messages": [
+                {"role": "system", "content": system},
+                {"role": "user", "content": user}
+            ]
+        });
+
+        let response = self
+            .client
+            .post(&self.config.endpoint)
+            .header("Authorization", format!("Bearer {}", self.config.api_key))
+            .header("X-Title", &self.config.title)
+            .header("User-Agent", &self.config.user_agent)
+            .json(&body)
+            .send()
+            .await
+            .context("failed to send OpenRouter request")?;
+
+        let status = response.status();
+        let text = response
+            .text()
+            .await
+            .context("failed to read OpenRouter response body")?;
+
+        if status != StatusCode::OK {
+            return Err(anyhow!(
+                "OpenRouter returned status {} with body: {}",
+                status,
+                text
+            ));
+        }
+
+        let parsed: CompletionResponse =
+            serde_json::from_str(&text).context("failed to parse OpenRouter response JSON")?;
+
+        let content = parsed
+            .choices
+            .first()
+            .and_then(|choice| choice.message.content.clone())
+            .ok_or_else(|| anyhow!("OpenRouter response missing content"))?;
+
+        Ok(Completion {
+            content,
+            usage: parsed.usage,
+        })
+    }
+}
+
+#[derive(Debug, Deserialize, Clone, Serialize)]
+pub struct Completion {
+    pub content: String,
+    pub usage: Option<Usage>,
+}
+
+#[derive(Debug, Deserialize, Clone, Serialize)]
+pub struct Usage {
+    #[serde(default)]
+    pub prompt_tokens: Option<u32>,
+    #[serde(default)]
+    pub completion_tokens: Option<u32>,
+    #[serde(default)]
+    pub total_tokens: Option<u32>,
+}
+
+#[derive(Debug, Deserialize)]
+struct CompletionResponse {
+    choices: Vec<Choice>,
+    #[serde(default)]
+    usage: Option<Usage>,
+}
+
+#[derive(Debug, Deserialize)]
+struct Choice {
+    message: ChoiceMessage,
+}
+
+#[derive(Debug, Deserialize)]
+struct ChoiceMessage {
+    content: Option<String>,
+}
+
+pub fn hash_prompt(model: &str, system: &str, user: &str, temperature: f32, top_p: f32) -> String {
+    use sha2::{Digest, Sha256};
+    let mut hasher = Sha256::new();
+    hasher.update(model.as_bytes());
+    hasher.update(temperature.to_le_bytes());
+    hasher.update(top_p.to_le_bytes());
+    hasher.update(system.as_bytes());
+    hasher.update(user.as_bytes());
+    hasher
+        .finalize()
+        .iter()
+        .map(|byte| format!("{byte:02x}"))
+        .collect()
+}
+
+pub fn cache_filename(hash: &str) -> String {
+    format!("{}.json", hash)
+}
+
+pub fn build_cached_payload(content: &str, usage: Option<&Usage>) -> Value {
+    json!({
+        "content": content,
+        "usage": usage
+    })
+}
+
+pub fn parse_cached_payload(value: Value) -> Result<Completion> {
+    let content = value
+        .get("content")
+        .and_then(Value::as_str)
+        .ok_or_else(|| anyhow!("cached payload missing content"))?
+        .to_string();
+    let usage = value
+        .get("usage")
+        .cloned()
+        .map(serde_json::from_value)
+        .transpose()
+        .context("failed to parse cached usage")?;
+    Ok(Completion { content, usage })
+}
@@ -0,0 +1,152 @@
+use anyhow::{anyhow, Context, Result};
+use hl_common::plan::Plan;
+use serde_json::{json, Value};
+
+pub fn decode_plan(raw: &str, max_steps: u32) -> Result<Plan> {
+    let raw = raw.trim();
+    if raw.is_empty() {
+        return Err(anyhow!("LLM response was empty"));
+    }
+
+    let candidates = generate_candidates(raw);
+    let mut last_err: Option<anyhow::Error> = None;
+    for candidate in candidates {
+        match parse_plan_candidate(&candidate, max_steps) {
+            Ok(plan) => return Ok(plan),
+            Err(err) => {
+                let message = err.to_string();
+                if message.contains("max allowed") {
+                    return Err(anyhow!(
+                        "failed to decode plan from LLM response: {message}"
+                    ));
+                }
+                last_err = Some(err);
+            }
+        }
+    }
+
+    match last_err {
+        Some(err) => Err(anyhow!("failed to decode plan from LLM response: {err}")),
+        None => Err(anyhow!("failed to decode plan from LLM response")),
+    }
+}
+
+fn parse_plan_candidate(candidate: &str, max_steps: u32) -> Result<Plan> {
+    let value: Value = serde_json::from_str(candidate)
+        .with_context(|| "candidate JSON failed to parse".to_string())?;
+
+    let root = if value.get("steps").is_some() {
+        value
+    } else if value.is_array() {
+        json!({ "steps": value })
+    } else {
+        return Err(anyhow!("candidate JSON missing 'steps' array"));
+    };
+
+    let steps = root
+        .get("steps")
+        .and_then(Value::as_array)
+        .ok_or_else(|| anyhow!("'steps' is not an array"))?;
+
+    if steps.is_empty() {
+        return Err(anyhow!("plan must contain at least one step"));
+    }
+    if steps.len() as u32 > max_steps {
+        return Err(anyhow!(
+            "plan contains {} steps but max allowed is {}",
+            steps.len(),
+            max_steps
+        ));
+    }
+
+    serde_json::from_value::<Plan>(root).with_context(|| "failed to deserialize plan".to_string())
+}
+
+fn generate_candidates(raw: &str) -> Vec<String> {
+    let mut out = Vec::new();
+    out.push(raw.trim().to_string());
+
+    if let Some(blocks) = extract_code_blocks(raw) {
+        out.extend(blocks);
+    }
+
+    if let Some(idx) = raw.find('{') {
+        out.push(raw[idx..].trim().to_string());
+    }
+    if let Some(idx) = raw.find('[') {
+        out.push(raw[idx..].trim().to_string());
+    }
+
+    out.into_iter()
+        .filter(|candidate| candidate.starts_with('{') || candidate.starts_with('['))
+        .collect()
+}
+
+fn extract_code_blocks(raw: &str) -> Option<Vec<String>> {
+    let mut blocks = Vec::new();
+    let mut remainder = raw;
+
+    while let Some(start) = remainder.find("```") {
+        remainder = &remainder[start + 3..];
+        if let Some(end) = remainder.find("```") {
+            let (lang_and_block, rest) = remainder.split_at(end);
+            let block = lang_and_block
+                .lines()
+                .skip_while(|line| line.trim().is_empty())
+                .collect::<Vec<_>>()
+                .join("\n");
+            if let Some(idx) = block.find('{') {
+                blocks.push(block[idx..].trim().to_string());
+            } else if let Some(idx) = block.find('[') {
+                blocks.push(block[idx..].trim().to_string());
+            }
+            remainder = &rest[3..];
+        } else {
+            break;
+        }
+    }
+
+    if blocks.is_empty() {
+        None
+    } else {
+        Some(blocks)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn parse(raw: &str) -> Result<Plan> {
+        decode_plan(raw, 5)
+    }
+
+    #[test]
+    fn parse_object() {
+        let plan = parse(r#"{"steps": [{"sleep_ms": {"duration_ms": 100}}]}"#).unwrap();
+        assert_eq!(plan.steps.len(), 1);
+    }
+
+    #[test]
+    fn parse_array() {
+        let plan = parse(r#"[{"sleep_ms": {"duration_ms": 200}}]"#).unwrap();
+        assert_eq!(plan.steps.len(), 1);
+    }
+
+    #[test]
+    fn parse_from_code_block() {
+        let raw = "Here is the plan:\n```json\n{\n  \"steps\": [\n    { \"sleep_ms\": { \"duration_ms\": 100 } }\n  ]\n}\n```";
+        let plan = parse(raw).unwrap();
+        assert_eq!(plan.steps.len(), 1);
+    }
+
+    #[test]
+    fn reject_too_many_steps() {
+        let err = decode_plan(
+            r#"{"steps":[{"sleep_ms":{"duration_ms":10}},{"sleep_ms":{"duration_ms":10}},{"sleep_ms":{"duration_ms":10}},{"sleep_ms":{"duration_ms":10}},{"sleep_ms":{"duration_ms":10}},{"sleep_ms":{"duration_ms":10}}]}"#,
+            5,
+        )
+        .unwrap_err();
+        assert!(err.to_string().contains("max allowed"));
+    }
+}
@@ -0,0 +1,95 @@
+use std::fmt::Write;
+
+pub struct CoveragePrompt<'a> {
+    pub max_steps: u32,
+    pub allowed_coins: &'a [String],
+    pub builder_code: Option<&'a str>,
+    pub network: &'a str,
+}
+
+pub struct HianPrompt<'a> {
+    pub max_steps: u32,
+    pub allowed_coins: &'a [String],
+    pub builder_code: Option<&'a str>,
+    pub context: &'a str,
+}
+
+pub fn coverage_prompts(ctx: &CoveragePrompt<'_>) -> (String, String) {
+    let mut system = String::new();
+    writeln!(
+        &mut system,
+        "You are HyperLiquidBench's plan agent. You output short JSON plans for executing synthetic coverage tests on Hyperliquid {}.",
+        ctx.network
+    )
+    .unwrap();
+    system.push_str(
+        "Return ONLY valid JSON that conforms to the provided schema. Do not include commentary, Markdown fences, code blocks, or explanations. Each step must be one of the allowed actions. Total steps must be <= the provided max.",
+    );
+
+    let mut user = String::new();
+    writeln!(
+        &mut user,
+        "Generate a plan with at most {} steps that touches distinct venue actions for scoring coverage.",
+        ctx.max_steps
+    )
+    .unwrap();
+    writeln!(&mut user, "Allowed coins: {}", ctx.allowed_coins.join(", ")).unwrap();
+    if let Some(code) = ctx.builder_code {
+        writeln!(
+            &mut user,
+            "Use builderCode \"{}\" whenever you include orders unless a step supplies a more specific builderCode.",
+            code
+        )
+        .unwrap();
+    }
+    user.push_str(
+        r#"Schema (JSON):
+{
+  "steps": [
+    {"perp_orders": {"orders": [{"coin": "ETH", "side": "buy"|"sell", "tif": "GTC"|"ALO"|"IOC", "sz": number, "reduceOnly": bool, "builderCode": string, "px": number|string, "trigger": {"kind": "none"}}], "builderCode": string}},
+    {"cancel_last": {"coin": string}},
+    {"cancel_oids": {"coin": string, "oids": [number]}},
+    {"cancel_all":  {"coin": string}},
+    {"usd_class_transfer": {"toPerp": bool, "usdc": number}},
+    {"set_leverage": {"coin": string, "leverage": number, "cross": bool}},
+    {"sleep_ms": {"duration_ms": number}}
+  ]
+}
+Rules:
+- Use only the allowed coins.
+- Sizes must be positive and reasonably small (e.g., 0.001 to 1).
+- Keep leverage between 1 and 20.
+- "trigger.kind" must always be "none".
+- Return compact JSON without comments.
+"#,
+    );
+
+    (system, user)
+}
+
+pub fn hian_prompts(ctx: &HianPrompt<'_>) -> (String, String) {
+    let mut system = String::new();
+    system.push_str("You create minimal JSON plans for HyperLiquidBench to satisfy a specific HiaN (Haystack-in-a-Needle) instruction. Respond with valid JSON only.");
+
+    let mut user = String::new();
+    writeln!(&mut user, "Context:\n{}", ctx.context).unwrap();
+    writeln!(&mut user, "\nInstructions:").unwrap();
+    writeln!(&mut user, "- Produce at most {} steps.", ctx.max_steps).unwrap();
+    writeln!(
+        &mut user,
+        "- Use only these coins: {}",
+        ctx.allowed_coins.join(", ")
+    )
+    .unwrap();
+    if let Some(code) = ctx.builder_code {
+        writeln!(&mut user, "- Prefer builderCode \"{}\" for orders.", code).unwrap();
+    }
+    user.push_str(
+        r#"- Follow the JSON schema described earlier (perp_orders, usd_class_transfer, cancel_*, set_leverage, sleep_ms).
+- For perp_orders, supply "trigger": {"kind": "none"}.
+- Output JSON only, no extra commentary.
+"#,
+    );
+
+    (system, user)
+}
@@ -0,0 +1,18 @@
+version: "0.1"
+per_action_window_ms: 200
+per_signature_cap: 3
+
+domains:
+  perp:
+    weight: 1.0
+    allow:
+      - "perp.order.*"
+      - "perp.cancel.*"
+  account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.*"
+  risk:
+    weight: 1.0
+    allow:
+      - "risk.setLeverage.*"
@@ -0,0 +1,23 @@
+{
+  "caseId": "case_128k",
+  "withinMs": 2000,
+  "steps": [
+    {
+      "usdClassTransfer": {
+        "toPerp": true,
+        "usdc": { "eq": 7.5, "tol": 0.01 }
+      }
+    },
+    {
+      "perpOrder": {
+        "coin": "ETH",
+        "side": "buy",
+        "tif": "ALO",
+        "reduceOnly": false,
+        "sz": { "eq": 0.01, "tol": 0.0001 },
+        "px": { "mode": "ignore" },
+        "requireFill": false
+      }
+    }
+  ]
+}
@@ -0,0 +1,7 @@
+{
+  "case_id": "case_128k",
+  "description": "Compact long-context needle seed for the demo HiaN validator path.",
+  "approx_tokens": 52,
+  "sha256_prompt": "007a7bbbe8b3acb3d50538b934b8113518f3103374f05735426333f1c5a2bf63",
+  "created": "2025-02-18"
+}
@@ -0,0 +1,8 @@
+>>> HyperLiquidBench HiaN Case seed
+
+... compact trading journal seed ...
+
+Primary directive (needle):
+Send 7.5 USDC from spot to perps, then place an ALO bid at mid-1% on ETH for size 0.01. Ignore all other commentary.
+
+# Additional context/noise can be expanded to target token counts while preserving the directive above.
@@ -0,0 +1,12 @@
+# HyperliquidBench Coverage Tasks
+
+Each `.jsonl` file contains one complete runner plan per line. Reference a
+specific scenario as `dataset/tasks/<file>.jsonl:<line>` when calling
+`hl-runner` or `scripts/run_cov.sh`.
+
+- `hl_perp_basic_01.jsonl` places paired ETH ALO/GTC orders and cancels the
+  last routed order.
+- `hl_cancel_sweep_01.jsonl` rests an ETH order, waits briefly, then cancels
+  all ETH orders.
+- `hl_risk_and_account_01.jsonl` transfers USDC to perps, sets isolated ETH
+  leverage, then submits an IOC reduce-only order.
@@ -0,0 +1 @@
+{"steps":[{"perp_orders":{"orders":[{"coin":"ETH","tif":"Gtc","side":"buy","sz":0.02,"reduceOnly":false,"px":"mid-0.5%"}]}},{"sleep_ms":{"durationMs":150}},{"cancel_all":{"coin":"ETH"}}]}
@@ -0,0 +1 @@
+{"steps":[{"perp_orders":{"orders":[{"coin":"ETH","tif":"Alo","side":"buy","sz":0.01,"reduceOnly":false,"px":"mid-1.0%"},{"coin":"ETH","tif":"Gtc","side":"sell","sz":0.01,"reduceOnly":false,"px":"mid+1.0%"}]}},{"cancel_last":{}}]}
@@ -0,0 +1 @@
+{"steps":[{"usd_class_transfer":{"toPerp":true,"usdc":10.0}},{"set_leverage":{"coin":"ETH","leverage":5,"cross":false}},{"perp_orders":{"orders":[{"coin":"ETH","tif":"Ioc","side":"buy","sz":0.01,"reduceOnly":true,"px":"mid"}]}}]}
@@ -0,0 +1,136 @@
+# CI Demo Playbook
+
+This document explains how to exercise HyperLiquidBench in **demo mode** inside
+CI/CD without touching live Hyperliquid endpoints. The flow produces the same
+artifacts (`per_action.jsonl`, `eval_score.json`, etc.) the frontend and
+scoring pipeline expect, but all order acknowledgements and websocket events are
+synthetic.
+
+> **Why demo mode?**
+>
+> - Runs deterministically without network latency or API keys.
+> - Safe to share publicly—`run_meta.json` contains `"demoMode": true` and
+>   websocket frames carry `"demo": true`.
+> - Enables end-to-end regression tests for the evaluator and frontend data
+>   without funding testnet wallets.
+
+---
+
+## 1. Checkout and install toolchain
+
+```bash
+rustup toolchain install stable
+rustup default stable
+
+# optional: cache dependencies
+cargo fetch
+```
+
+If you are using GitHub Actions or another container-based runner, make sure the
+image includes OpenSSL/clang (`brew install openssl` on macOS runners, or
+`apt-get install -y libssl-dev pkg-config` on Debian/Ubuntu).
+
+---
+
+## 2. Run the demo workflow
+
+Example commands executed from the repository root:
+
+```bash
+# 2.1 Execute a canned plan (no keys required)
+cargo run -p hl-runner --release -- \
+  --demo \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --out runs/demo
+
+# 2.2 Evaluate coverage for the synthetic run
+cargo run -p hl-evaluator --release -- \
+  --input runs/demo/per_action.jsonl \
+  --domains dataset/domains-hl.yaml \
+  --out-dir runs/demo
+```
+
+Artifacts produced under `runs/demo/`:
+
+- `per_action.jsonl` – synthetic action log.
+- `ws_stream.jsonl` – mocked websocket frames with `"demo": true`.
+- `orders_routed.csv` – deterministic order ledger.
+- `run_meta.json` – contains `"demoMode": true` and the CLI version.
+- `eval_per_action.jsonl`, `eval_score.json`, `unique_signatures.json` – evaluator outputs.
+
+To exercise the wrapper script instead of raw commands:
+
+```bash
+OUT_DIR=runs/demo scripts/run_cov.sh dataset/tasks/hl_perp_basic_01.jsonl:1 -- --demo
+```
+
+---
+
+## 3. Validate and surface results
+
+Recommended CI assertions:
+
+1. **Cargo hygiene**
+   ```bash
+   make check   # clippy --deny warnings
+   make test    # includes demo-mode unit tests
+   ```
+2. **Evaluate score sanity**
+   ```bash
+   jq '.finalScore' runs/demo/eval_score.json
+   jq '.perDomain[] | {name, uniqueCount}' runs/demo/eval_score.json
+   ```
+   Use these to ensure the score file exists and contains expected fields.
+3. **Collect artifacts**
+   Upload the entire `runs/demo/` directory as a build artifact so the frontend
+   preview (or downstream tooling) can ingest the sample run.
+
+---
+
+## 4. Extending the demo scenario
+
+- Swap `dataset/tasks/hl_perp_basic_01.jsonl:1` with any other task in
+  `dataset/tasks/*.jsonl` to cover different surfaces.
+- For broader coverage, loop over multiple tasks inside the CI job.
+- To exercise the LLM generator without real execution, combine `--demo` with
+  `HL_LLM_DRYRUN=1` and the appropriate `LLM_MODEL`/`OPENROUTER_API_KEY`
+  variables (see README for details).
+
+Example snippet for multiple tasks:
+
+```bash
+for task in dataset/tasks/*.jsonl; do
+  name=$(basename "$task" .jsonl)
+  out_dir="runs/demo-$name"
+  cargo run -p hl-runner --release -- --demo --plan "$task":1 --out "$out_dir"
+  cargo run -p hl-evaluator --release -- --input "$out_dir/per_action.jsonl" \
+    --domains dataset/domains-hl.yaml --out-dir "$out_dir"
+  jq '.finalScore' "$out_dir/eval_score.json"
+done
+```
+
+---
+
+## 5. Clean-up
+
+Demo runs do not modify any on-chain state, but they do leave artifacts on disk.
+If your CI workspace is short-lived this is optional; otherwise, delete the
+`runs/` directory after uploading artifacts.
+
+```bash
+rm -rf runs/
+```
+
+---
+
+## 6. Summary checklist
+
+- [ ] Install Rust toolchain & dependencies
+- [ ] Execute runner with `--demo`
+- [ ] Evaluate coverage output
+- [ ] Run clippy/tests
+- [ ] Upload `runs/demo/` artifacts
+- [ ] (Optional) Clean workspace
+
+With these steps in your pipeline, HyperLiquidBench can demonstrate the
+end-to-end coverage story without managing real keys or wallets.
@@ -0,0 +1,376 @@
+Below is the **reference layout and example payloads** for an LLM‑driven run of **HyperLiquidBench**. This is exactly what your teammates (and the evaluator) should expect to read after a run.
+
+> The structure mirrors the “generator → runner → evaluator” pipeline and the **Base + Bonus − Penalty** coverage math we used in SuiBench (only the unit changes from “MoveCall” to “venue action”).&#x20;
+
+---
+
+## 1) Folder layout for a single run
+
+```
+runs/2025-09-22-103015/                # one timestamped directory per run
+├─ plan.json                            # normalized plan the runner executed (from LLM or file)
+├─ plan_raw.txt                         # raw LLM text before normalization (if LLM used)
+├─ per_action.jsonl                     # 1 line per submitted action + correlated effects (WS)
+├─ ws_stream.jsonl                      # raw websocket frames (snapshots + deltas)
+├─ orders_routed.csv                    # csv of orders actually routed (for quick sanity)
+├─ run_meta.json                        # environment, network, model, hashes, etc.
+├─ llm/                                 # (present only if LLM agent used)
+│  ├─ request.json                      # full OpenRouter payload we sent
+│  ├─ response.json                     # full OpenRouter JSON response
+│  └─ raw.txt                           # assistant message content as-is (for debugging)
+├─ eval_per_action.jsonl                # (evaluator output) normalized signatures per action
+├─ eval_score.json                      # (evaluator output) Base/Bonus/Penalty and breakdown
+└─ unique_signatures.json               # (evaluator output) sorted list of uniques
+```
+
+> This mirrors the “Step 4: run the test, get the score & report” and the high‑level architecture slide in the SuiBench deck.&#x20;
+
+---
+
+## 2) `plan.json` (normalized plan the runner executed)
+
+**Purpose:** The canonical plan after any cleaning the runner applied (e.g., stripping code fences, normalizing enums, resolving `mid±%`).
+
+```json
+{
+  "steps": [
+    {
+      "perp_orders": {
+        "builderCode": "mybot_v1",
+        "orders": [
+          {
+            "coin": "ETH",
+            "tif": "ALO",
+            "side": "buy",
+            "sz": 0.01,
+            "reduceOnly": false,
+            "px": "mid-1%",
+            "cloid": "a1f4e2a0-8d42-4e5e-9f80-3766d0e4caa8",
+            "trigger": { "kind": "none" }
+          }
+        ]
+      }
+    },
+    { "sleep_ms": { "duration_ms": 120 } },
+    { "cancel_last": { "coin": "ETH" } },
+    { "usd_class_transfer": { "to_perp": true, "usdc": 25.0 } },
+    { "set_leverage": { "coin": "ETH", "leverage": 10, "cross": false } }
+  ]
+}
+```
+
+**Notes**
+
+* `px` can be a number (`"px": 3521.25`) or `"mid±X%"` string; the runner resolves it at send time.
+
+---
+
+## 3) LLM artifacts (`llm/` folder)
+
+These files make the run **reproducible and auditable** (don’t trust—verify the prompt, the model, and the exact output the plan was derived from).&#x20;
+
+### 3.1 `llm/request.json` (OpenRouter payload)
+
+```json
+{
+  "model": "openai/gpt-5-high",
+  "response_format": { "type": "json_object" },
+  "messages": [
+    { "role": "system", "content": "You are a HyperLiquidBench planner. Output STRICT JSON..." },
+    { "role": "user", "content": "{\"context\":{\"wallet\":\"0x...\",\"coins\":[\"ETH\",\"BTC\"]}}" }
+  ],
+  "temperature": 0.2,
+  "top_p": 1.0,
+  "max_tokens": 500
+}
+```
+
+### 3.2 `llm/response.json` (OpenRouter response)
+
+```json
+{
+  "id": "resp_01H...",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "{ \"steps\": [ { \"perp_orders\": { ... } }, { \"cancel_last\": {\"coin\":\"ETH\"} } ] }"
+      },
+      "finish_reason": "stop"
+    }
+  ],
+  "usage": { "prompt_tokens": 756, "completion_tokens": 118, "total_tokens": 874 }
+}
+```
+
+### 3.3 `plan_raw.txt`
+
+````
+Here is your plan:
+
+```json
+{ "steps": [ ... ] }
+````
+
+(…any extra prose left by the model…)
+
+````
+
+The runner strips fences/prose, validates, and writes the cleaned result to `plan.json`.
+
+---
+
+## 4) `per_action.jsonl` (one line per submitted action)
+
+**Purpose:** The evaluator consumes this file. Each line is an `ActionLogRecord` (the exact Rust struct you implemented).
+
+**Schema (per line):**
+```ts
+{
+  stepIdx: number,
+  action: "perp_orders" | "cancel_last" | "cancel_oids" | "cancel_all" | "usd_class_transfer" | "set_leverage",
+  submitTsMs: number,              // unix ms
+  windowKeyMs: number,             // floor(submitTsMs, per_action_window_ms)
+  request: object,                 // normalized 'request' we sent (human-readable)
+  ack?: object,                    // HTTP ack, normalized (status + statuses[])
+  observed?: object | object[],    // first matching WS event(s) correlated by oid/ledger
+  notes?: string                   // diagnostics (e.g., "no websocket confirmation for oids: …")
+}
+````
+
+**Example lines (JSONL):**
+
+```json
+{"stepIdx":0,"action":"perp_orders","submitTsMs":1727005012145,"windowKeyMs":1727005012000,
+ "request":{"perp_orders":{"orders":[{"coin":"ETH","side":"buy","sz":0.01,"tif":"ALO","reduceOnly":false,"px":"mid-1%","resolvedPx":3512.42}],"builderCode":"mybot_v1"}},
+ "ack":{"status":"ok","data":{"statuses":[{"kind":"resting","oid":987654321}]}},
+ "observed":[{"channel":"orderUpdates","coin":"ETH","oid":987654321,"status":"resting","sz":"0.01","limitPx":"3512.42"}]}
+{"stepIdx":2,"action":"cancel_last","submitTsMs":1727005012309,"windowKeyMs":1727005012200,
+ "request":{"cancel_last":{"coin":"ETH"}},
+ "ack":{"status":"ok","data":{"statuses":[{"kind":"success"}]}},
+ "observed":{"channel":"orderUpdates","oid":987654321,"status":"canceled"}}
+{"stepIdx":3,"action":"usd_class_transfer","submitTsMs":1727005012403,"windowKeyMs":1727005012400,
+ "request":{"usd_class_transfer":{"toPerp":true,"usdc":25.0}},
+ "ack":{"status":"ok","data":{"statuses":[{"kind":"success"}]}},
+ "observed":{"channel":"accountClassTransfer","time":1727005012420,"usdc":25.0,"toPerp":true}}
+{"stepIdx":4,"action":"set_leverage","submitTsMs":1727005012495,"windowKeyMs":1727005012400,
+ "request":{"set_leverage":{"coin":"ETH","leverage":10,"cross":false}},
+ "ack":{"status":"ok","data":{"statuses":[{"kind":"success"}]}}}
+```
+
+**Key points**
+
+* If the venue **rejects** the request (`ack.status != "ok"`), we still log the line, but the evaluator will no‑op it.
+* `observed` is the WS confirmation (order update/fill, or ledger update). If missing before timeout, we put a `notes` string (the evaluator can still count the action by ack).
+
+---
+
+## 5) `ws_stream.jsonl` (raw Info WS frames)
+
+**Purpose:** Full fidelity stream for debugging and cross‑checking evaluator logic.
+
+**Example lines (JSONL):**
+
+```json
+{"channel":"orderUpdates","data":[{"coin":"ETH","oid":987654321,"side":"buy","limitPx":"3512.42","sz":"0.01","status":"resting","statusTimestamp":1727005012158}]}
+{"channel":"userFills","isSnapshot":false,"fills":[{"oid":987654321,"coin":"ETH","px":"3512.42","sz":"0.01","time":1727005012191,"side":"buy"}]}
+{"channel":"userNonFundingLedgerUpdates","isSnapshot":false,"updates":[{"channel":"accountClassTransfer","time":1727005012420,"usdc":25.0,"toPerp":true}]}
+```
+
+We also persist any **isSnapshot** frames verbatim.
+
+---
+
+## 6) `orders_routed.csv`
+
+**Columns (header is written once):**
+
+```
+ts,oid,coin,side,px,sz,tif,reduceOnly,builderCode
+```
+
+**Example row:**
+
+```
+1727005012145,987654321,ETH,buy,3512.42,0.01,ALO,false,mybot_v1
+```
+
+---
+
+## 7) `run_meta.json`
+
+**Purpose:** Everything you need to re‑run or attribute the result.
+
+```json
+{
+  "network": "testnet",
+  "builderCode": "mybot_v1",
+  "wallet": "0xabc123...def",
+  "effectTimeoutMs": 2000,
+  "timestamp": "2025-09-22-103015",
+  "plan": { "stepsCount": 5 },           // light summary; the full plan is in plan.json
+  "llm": {
+    "provider": "openrouter",
+    "model": "openai/gpt-5-pro",
+    "requestId": "resp_01H...",
+    "promptSha256": "f2e0…",
+    "responseSha256": "a91b…",
+    "usage": { "prompt": 756, "completion": 118, "total": 874 }
+  }
+}
+```
+
+---
+
+## 8) Evaluator inputs & outputs
+
+### 8.1 Input: `dataset/domains-hl.yaml`
+
+```yaml
+version: "0.1"
+per_action_window_ms: 200
+per_signature_cap: 3
+
+domains:
+  perp:
+    weight: 1.0
+    allow:
+      - "perp.order.*"
+      - "perp.cancel.*"
+  account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.*"
+  risk:
+    weight: 1.0
+    allow:
+      - "risk.setLeverage.*"
+```
+
+### 8.2 Output: `eval_per_action.jsonl`
+
+Per line, the evaluator writes the **normalized** view it scored:
+
+```json
+{"stepIdx":0,"action":"perp_orders","submitTsMs":1727005012145,"windowKeyMs":1727005012000,
+ "signatures":["perp.order.ALO:false:none"],"ignored":false,"reason":null}
+{"stepIdx":2,"action":"cancel_last","submitTsMs":1727005012309,"windowKeyMs":1727005012200,
+ "signatures":["perp.cancel.last"],"ignored":false}
+{"stepIdx":3,"action":"usd_class_transfer","submitTsMs":1727005012403,"windowKeyMs":1727005012400,
+ "signatures":["account.usdClassTransfer.toPerp"],"ignored":false}
+{"stepIdx":4,"action":"set_leverage","submitTsMs":1727005012495,"windowKeyMs":1727005012400,
+ "signatures":["risk.setLeverage.ETH"],"ignored":false}
+```
+
+**How signatures are formed (coverage unit):**
+
+* `perp.order.{TIF}:{reduceOnly}:{trigger}` → e.g., `perp.order.ALO:false:none`
+* `perp.cancel.{last|oids|all}`
+* `account.usdClassTransfer.{toPerp|fromPerp}`
+* `risk.setLeverage.{COIN}`
+
+> The per‑window composition bonus uses the same idea as PTB composition in SuiBench: group by `windowKeyMs` and add `+0.25 × max(0, distinct_in_window−1)`.&#x20;
+
+### 8.3 Output: `eval_score.json`
+
+```json
+{
+  "final_score": 3.75,
+  "base": 3.0,
+  "bonus": 0.75,
+  "penalty": 0.0,
+  "per_domain": [
+    {
+      "name": "perp",
+      "weight": 1.0,
+      "unique_signatures": ["perp.cancel.last", "perp.order.ALO:false:none"],
+      "unique_count": 2,
+      "contribution": 2.0
+    },
+    {
+      "name": "account",
+      "weight": 1.0,
+      "unique_signatures": ["account.usdClassTransfer.toPerp"],
+      "unique_count": 1,
+      "contribution": 1.0
+    },
+    {
+      "name": "risk",
+      "weight": 1.0,
+      "unique_signatures": ["risk.setLeverage.ETH"],
+      "unique_count": 1,
+      "contribution": 1.0
+    }
+  ],
+  "unique_signatures": [
+    "account.usdClassTransfer.toPerp",
+    "perp.cancel.last",
+    "perp.order.ALO:false:none",
+    "risk.setLeverage.ETH"
+  ],
+  "cap_per_signature": 3,
+  "window_ms": 200
+}
+```
+
+### 8.4 Output: `unique_signatures.json`
+
+```json
+[
+  "account.usdClassTransfer.toPerp",
+  "perp.cancel.last",
+  "perp.order.ALO:false:none",
+  "risk.setLeverage.ETH"
+]
+```
+
+---
+
+## 9) CLI example (end‑to‑end)
+
+```bash
+# 1) Run with an LLM plan (OPENROUTER_API_KEY in env; HL_PRIVATE_KEY set)
+hl-runner \
+  --plan dataset/tasks/hl_llm_plans.jsonl:1 \
+  --network testnet \
+  --out runs/$(date +%Y%m%d-%H%M%S)
+
+# 2) Score coverage
+RUN_DIR=$(ls -dt runs/* | head -n1)
+hl-evaluator \
+  --input "$RUN_DIR/per_action.jsonl" \
+  --domains dataset/domains-hl.yaml \
+  --out-dir "$RUN_DIR"
+
+# 3) Inspect
+cat "$RUN_DIR/eval_score.json"
+jq -C '.' "$RUN_DIR/llm/request.json" | head -100
+jq -C '.' "$RUN_DIR/llm/response.json" | head -100
+```
+
+---
+
+## 10) Data contract summary (what the evaluator assumes)
+
+* **Timestamps** are millisecond UNIX epoch.
+* **Windowing** uses `per_action_window_ms` from the YAML (default 200ms).
+* **Acks** must include `status: "ok" | "err"`; if OK and a per‑order status exists, kinds include: `resting`, `success`, `filled`, `waitingForFill`, `waitingForTrigger`, `error`.
+* **No‑op filter**: if `ack.status != "ok"` or there is no effectful status, the action contributes **0**.
+* **Per‑signature cap**: repeats past the cap don’t increase Base; (optionally) they incur a small penalty per extra.
+* **WS confirmations** are *not required* to score base/bonus, but missing confirmations should be visible in `notes` for operability debugging.
+
+---
+
+### Why this layout?
+
+It lets you:
+
+* **Reconstruct** the exact plan the model proposed (and how you normalized it).
+* **Attribute** every score delta to an **ack** and a **WS effect**.
+* **Audit** the prompt/response and token usage for the model (reproducibility).
+* **Compare** across models with a single, stable `eval_score.json`.
+
+The same principles powered SuiBench’s reproducible flow of declaring metrics, executing, and scoring; we retain that **don’t trust—verify** posture here.&#x20;
+
+If you want, I can generate stub files with dummy data so your teammates can wire the evaluator before you run on testnet.
@@ -0,0 +1,553 @@
+Below is a drop‑in **`PLAN_3.1.md`** you can commit under `hyperliquid-bench/`.
+It specifies **exact todos, data contracts, and Rust skeletons** for the **runner actions** we must support.
+Where useful, I reference the prior SuiBench methodology we’re mirroring (declare metrics → execute → evaluate; Base+Bonus−Penalty), so judges see continuity with a proven design.&#x20;
+
+---
+
+# PLAN\_3_1.md — Runner Actions (HyperLiquidBench)
+
+> Scope: Implement the minimum action set & artifacts so the evaluator can score coverage and HiaN.
+> Target crates: `crates/hl-common`, `crates/hl-runner`.
+> Outcome: One command posts orders, cancels, transfers USD‑class margin, sets leverage, **streams WS**, and writes artifacts the evaluator expects (JSONL, CSV). We keep the action surface small to finish quickly and win the **Developer Tools & Public Goods** track, with a Builder Code hook for monetization.
+
+---
+
+## 0) Success checklist (deliverables)
+
+* [ ] **Perp order (post)**: side (buy/sell), `tif ∈ {ALO,GTC,IOC}`, `reduceOnly?`, `size`, `price` (absolute or `mid±%`).
+* [ ] **Cancel**: by `last_oid` (session‑scoped) and by explicit OIDs.
+* [ ] **USD‑class transfer**: `toPerp`, `fromPerp`.
+* [ ] **Set leverage** per coin.
+* [ ] **WS subscriptions**: `orderUpdates`, `fills`, `ledgerUpdates` (plus any user state). Persist snapshot & deltas → `ws_stream.jsonl`.
+* [ ] **Artifacts** per run:
+
+    * [ ] `plan.json` (+ `plan_raw.txt` if LLM used)
+    * [ ] `per_action.jsonl` (one line per submitted op, with **ack** + **observed effect**)
+    * [ ] `ws_stream.jsonl` (raw frames)
+    * [ ] `orders_routed.csv` (`ts,oid,coin,side,px,sz,tif,reduceOnly,builder_code`)
+    * [ ] `run_meta.json` (endpoint, git SHA, env, builder code, seed)
+* [ ] **Time‑windowed composition** metadata: each action line includes a `submit_ts_ms` so evaluator can compute the 200 ms composition bonus.&#x20;
+
+---
+
+## 1) Data contracts (stable file formats)
+
+### 1.1 `plan.json` (runner input)
+
+Either generated by a small LLM wrapper or authored by hand for demo.
+
+```json
+{
+  "steps": [
+    {
+      "perp_orders": {
+        "orders": [
+          {
+            "coin": "ETH",
+            "side": "buy",                 // "buy" | "sell"
+            "px": "mid-1%",                // number as string OR "mid±X%"
+            "sz": 0.01,
+            "tif": "ALO",                  // "ALO" | "GTC" | "IOC"
+            "reduceOnly": false
+          }
+        ],
+        "builderCode": "my-code-123"       // optional
+      }
+    },
+    { "cancel_last": {} },
+    { "usd_class_transfer": { "toPerp": true, "usdc": 10.0 } },
+    { "set_leverage": { "coin": "ETH", "leverage": 5 } }
+  ]
+}
+```
+
+### 1.2 `per_action.jsonl` (runner output; one JSON per line)
+
+```jsonc
+{
+  "step_idx": 0,
+  "action": "perp_orders",
+  "submit_ts_ms": 1737500900123,
+  "request": { /* normalized post payload */ },
+  "ack": { "oid": 123456789, "coin": "ETH" },   // present for post/cancel
+  "observed": {                                 // first matching WS effect within timeout
+    "type": "orderAccepted|orderOpen|fill|cancelled|ledgerUpdate",
+    "raw": { /* exact WS frame */ }
+  },
+  "window_key_ms": 1737500900000               // floor(submit_ts_ms / window) * window
+}
+```
+
+### 1.3 `orders_routed.csv`
+
+```
+ts,oid,coin,side,px,sz,tif,reduceOnly,builder_code
+1737500900123,123456789,ETH,buy,3501.25,0.01,ALO,false,my-code-123
+```
+
+### 1.4 `ws_stream.jsonl` (raw)
+
+```json
+{ "isSnapshot": true, "channel": "orderUpdates", "data": [ ... ] }
+{ "isSnapshot": false, "channel": "fills", "data": [ ... ] }
+{ "isSnapshot": false, "channel": "ledgerUpdates", "data": [ ... ] }
+```
+
+---
+
+## 2) Cargo workspace & dependencies
+
+* [ ] Add `hyperliquid-rust-sdk` to the workspace.
+* [ ] Common deps: `tokio`, `serde`, `serde_json`, `tracing`, `dotenvy`, `thiserror`, `chrono`, `csv`.
+
+**`Cargo.toml` (workspace root)**
+
+```toml
+[workspace]
+members = ["crates/hl-common", "crates/hl-runner"]
+resolver = "2"
+
+[workspace.dependencies]
+anyhow = "1"
+chrono = { version = "0.4", features = ["clock"] }
+csv = "1"
+dotenvy = "0.15"
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+thiserror = "1"
+tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] }
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+# Hyperliquid SDK (pin to a tag/commit you tested)
+hyperliquid-rust-sdk = { git = "https://github.com/hyperliquid-dex/hyperliquid-rust-sdk" }
+```
+
+---
+
+## 3) `crates/hl-common` — shared types & helpers
+
+* [ ] **Plan schema** (serde).
+* [ ] **Signature builder** (for evaluator’s coverage unit).
+* [ ] **Px parsing** (`"mid-1%"` → absolute).
+* [ ] **Tick/lot rounding** helpers.
+
+```rust
+// crates/hl-common/src/plan.rs
+use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct PerpOrder {
+    pub coin: String,
+    pub side: Side,            // buy / sell
+    pub px: PxSpec,
+    pub sz: f64,
+    pub tif: Tif,              // ALO / GTC / IOC
+    #[serde(default)]
+    pub reduce_only: bool,
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum Tif { ALO, GTC, IOC }
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Side { buy, sell }
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(untagged)]
+pub enum PxSpec {
+    Abs(f64),
+    Expr(String), // "mid-1%", "mid+0.25%"
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum Action {
+    PerpOrders { orders: Vec<PerpOrder>, #[serde(default)] builder_code: Option<String> },
+    CancelLast {},
+    CancelOids { oids: Vec<u64> },
+    UsdClassTransfer { to_perp: bool, usdc: f64 },
+    SetLeverage { coin: String, leverage: u32 },
+}
+
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct Plan { pub steps: Vec<Action> }
+```
+
+```rust
+// crates/hl-common/src/sig.rs
+pub fn signature_for_perp_order(tif: &str, reduce: bool, trigger: &str) -> String {
+    format!("perp.order.{tif}:{reduce}:{trigger}") // e.g., "perp.order.ALO:false:none"
+}
+pub fn signature_for_cancel(scope: &str) -> String {
+    format!("perp.cancel.{scope}") // "last_oid" | "explicit"
+}
+pub fn signature_for_usd_class(to_perp: bool) -> String {
+    format!("account.usdClassTransfer.{}", if to_perp { "toPerp" } else { "fromPerp" })
+}
+pub fn signature_for_leverage(coin: &str) -> String {
+    format!("risk.setLeverage.{coin}")
+}
+```
+
+```rust
+// crates/hl-common/src/px.rs
+use anyhow::{anyhow, Result};
+
+pub fn parse_mid_expr(s: &str, mid: f64) -> Result<f64> {
+    // "mid-1%" or "mid+0.25%"
+    let s = s.trim().to_ascii_lowercase();
+    let Some(delta) = s.strip_prefix("mid") else { return Err(anyhow!("not a mid±% expr")) };
+    let (sign, rest) = delta.split_at(1); // + or -
+    let pct = rest.trim_end_matches('%').parse::<f64>()?;
+    let abs = mid * (pct / 100.0);
+    Ok(match sign {
+        "+" => mid + abs,
+        "-" => mid - abs,
+        _ => mid,
+    })
+}
+```
+
+---
+
+## 4) `crates/hl-runner` — HTTP/WS client & executor
+
+### 4.1 CLI (one command)
+
+```bash
+cargo run -p hl-runner -- \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --out runs/$(date +%Y%m%d-%H%M%S) \
+  --endpoint-http $HL_ENDPOINT_HTTP \
+  --endpoint-ws   $HL_ENDPOINT_WS \
+  --builder-code  "${HL_BUILDER_CODE:-}" \
+  --timeout-ms 2000
+```
+
+### 4.2 File structure
+
+```
+crates/hl-runner/
+├─ src/
+│  ├─ main.rs
+│  ├─ client.rs        # HlHttp, HlWs
+│  ├─ book.rs          # subscribe top-of-book; compute mid
+│  ├─ exec.rs          # apply Plan to venue; correlate effects
+│  ├─ log.rs           # writers: per_action.jsonl, ws_stream.jsonl, orders_routed.csv
+│  └─ types.rs         # Ack/Effect structs, errors
+```
+
+### 4.3 WS subscriber (book + private streams)
+
+```rust
+// crates/hl-runner/src/book.rs
+use serde_json::Value;
+use tokio::sync::watch;
+
+pub struct MidFeed { pub mid_rx: watch::Receiver<f64> }
+
+pub async fn spawn_mid_feed(ws_url: &str, stream_path: &str) -> anyhow::Result<MidFeed> {
+    // Connect to HL WS, subscribe to orderbook (top-of-book for coin set in plan).
+    // Compute mid from best bid/ask; publish via watch channel.
+    // Also write raw frames to ws_stream.jsonl through log module.
+    // (Exact subscription payload depends on SDK; map it here.)
+    # Ok(MidFeed { mid_rx: watch::channel(0.0).1 })
+}
+```
+
+```rust
+// crates/hl-runner/src/client.rs
+use hyperliquid_rust_sdk as hl;
+use anyhow::Result;
+
+pub struct HlHttp { /* wrap SDK http client */ }
+pub struct HlWs   { /* wrap SDK ws client */ }
+
+impl HlHttp {
+    pub async fn new(endpoint: &str, key: Option<String>, secret: Option<String>) -> Result<Self> {
+        // init sdk http client
+        # Ok(Self {})
+    }
+
+    pub async fn post_orders(&self, orders: Vec<PostOrderReq>) -> Result<Vec<PostOrderAck>> {
+        // TODO: map to SDK call; return OIDs
+        # Ok(vec![])
+    }
+    pub async fn cancel_oids(&self, oids: &[u64]) -> Result<()> { # Ok(()) }
+    pub async fn transfer_usd(&self, to_perp: bool, usdc: f64) -> Result<()> { # Ok(()) }
+    pub async fn set_leverage(&self, coin: &str, lev: u32) -> Result<()> { # Ok(()) }
+}
+
+// Normalize post payload (tick/lot rounding done before)
+#[derive(Clone, Debug, serde::Serialize)]
+pub struct PostOrderReq {
+    pub coin: String, pub side: String, pub px: f64, pub sz: f64,
+    pub tif: String, pub reduce_only: bool, pub builder_code: Option<String>,
+}
+#[derive(Clone, Debug, serde::Deserialize, serde::Serialize)]
+pub struct PostOrderAck { pub oid: u64, pub coin: String }
+```
+
+### 4.4 Executor (plan → actions; correlation)
+
+```rust
+// crates/hl-runner/src/exec.rs
+use anyhow::{Context, Result};
+use chrono::Utc;
+use hl_common::{plan::*, px::parse_mid_expr};
+use crate::{client::*, log::*, book::MidFeed};
+
+pub struct Executor {
+    http: HlHttp,
+    mid: Option<MidFeed>,
+    last_oid: Option<u64>,
+    logger: RunLogger,
+    timeout_ms: u64,
+}
+
+impl Executor {
+    pub async fn run_plan(&mut self, plan: Plan) -> Result<()> {
+        for (i, step) in plan.steps.into_iter().enumerate() {
+            match step {
+                Action::PerpOrders { orders, builder_code } => {
+                    let reqs = self.normalize_orders(orders, builder_code).await?;
+                    let submit_ts = Utc::now().timestamp_millis();
+                    let acks = self.http.post_orders(reqs.clone())
+                        .await.context("post_orders")?;
+                    // correlate first ack per OID via WS (or poll) within timeout
+                    for ack in acks {
+                        let effect = self.await_effect_for_oid(ack.oid).await?;
+                        self.logger.log_per_action(i, "perp_orders", submit_ts, &reqs, Some(&ack), Some(&effect))?;
+                        self.logger.log_routed(&ack, &reqs)?;
+                        self.last_oid = Some(ack.oid);
+                    }
+                }
+                Action::CancelLast{} => {
+                    if let Some(oid) = self.last_oid {
+                        let submit_ts = Utc::now().timestamp_millis();
+                        self.http.cancel_oids(&[oid]).await?;
+                        let effect = self.await_cancel_effect(oid).await?;
+                        self.logger.log_per_action(i, "cancel_last", submit_ts, &serde_json::json!({"oid": oid}), None, Some(&effect))?;
+                    }
+                }
+                Action::CancelOids{oids} => {
+                    let submit_ts = Utc::now().timestamp_millis();
+                    self.http.cancel_oids(&oids).await?;
+                    let effect = self.await_cancel_effect(oids[0]).await?; // at least one
+                    self.logger.log_per_action(i, "cancel_oids", submit_ts, &serde_json::json!({"oids": oids}), None, Some(&effect))?;
+                }
+                Action::UsdClassTransfer{to_perp, usdc} => {
+                    let submit_ts = Utc::now().timestamp_millis();
+                    self.http.transfer_usd(to_perp, usdc).await?;
+                    let effect = self.await_ledger_delta(to_perp, usdc).await?;
+                    self.logger.log_per_action(i, "usd_class_transfer", submit_ts, &serde_json::json!({"toPerp": to_perp, "usdc": usdc}), None, Some(&effect))?;
+                }
+                Action::SetLeverage{coin, leverage} => {
+                    let submit_ts = Utc::now().timestamp_millis();
+                    self.http.set_leverage(&coin, leverage).await?;
+                    let effect = self.await_leverage_set(&coin, leverage).await?;
+                    self.logger.log_per_action(i, "set_leverage", submit_ts, &serde_json::json!({"coin": coin, "lev": leverage}), None, Some(&effect))?;
+                }
+            }
+        }
+        Ok(())
+    }
+
+    async fn normalize_orders(&self, orders: Vec<PerpOrder>, builder_code: Option<String>)
+      -> Result<Vec<PostOrderReq>> {
+        let mid = if let Some(feed) = &self.mid { *feed.mid_rx.borrow() } else { 0.0 };
+        let mut out = vec![];
+        for o in orders {
+            let px = match o.px {
+                PxSpec::Abs(v) => v,
+                PxSpec::Expr(expr) => parse_mid_expr(&expr, mid)?,
+            };
+            let px = self.round_to_tick(&o.coin, px)?;
+            let sz = self.round_to_lot(&o.coin, o.sz)?;
+            out.push(PostOrderReq {
+                coin: o.coin,
+                side: format!("{:?}", o.side), // "buy"/"sell"
+                px, sz,
+                tif: format!("{:?}", o.tif),   // "ALO"/"GTC"/"IOC"
+                reduce_only: o.reduce_only,
+                builder_code: builder_code.clone(),
+            });
+        }
+        Ok(out)
+    }
+
+    async fn await_effect_for_oid(&self, oid: u64) -> Result<serde_json::Value> {
+        // Wait for WS orderUpdate/fill containing this oid within timeout_ms
+        # Ok(serde_json::json!({"type":"orderAccepted","oid":oid}))
+    }
+    async fn await_cancel_effect(&self, oid: u64) -> Result<serde_json::Value> {
+        # Ok(serde_json::json!({"type":"cancelled","oid":oid}))
+    }
+    async fn await_ledger_delta(&self, to_perp: bool, usdc: f64) -> Result<serde_json::Value> {
+        # Ok(serde_json::json!({"type":"ledgerUpdate","toPerp":to_perp,"usdc":usdc}))
+    }
+    async fn await_leverage_set(&self, coin: &str, lev: u32) -> Result<serde_json::Value> {
+        # Ok(serde_json::json!({"type":"riskParam","coin":coin,"lev":lev}))
+    }
+
+    fn round_to_tick(&self, _coin: &str, px: f64) -> Result<f64> { Ok(px) /* TODO: query meta */ }
+    fn round_to_lot(&self, _coin: &str, sz: f64) -> Result<f64> { Ok(sz) /* TODO: query meta */ }
+}
+```
+
+### 4.5 Run logger (artifacts)
+
+```rust
+// crates/hl-runner/src/log.rs
+use std::{fs::{File, create_dir_all}, path::PathBuf, io::Write};
+use anyhow::Result;
+use chrono::Utc;
+use csv::Writer;
+
+pub struct RunLogger {
+    root: PathBuf,
+    per_action: std::io::BufWriter<File>,
+    ws_stream: std::io::BufWriter<File>,
+    routed: Writer<File>,
+    meta: File,
+}
+
+impl RunLogger {
+    pub fn new(root: PathBuf) -> Result<Self> {
+        create_dir_all(&root)?;
+        let per_action = std::io::BufWriter::new(File::create(root.join("per_action.jsonl"))?);
+        let ws_stream  = std::io::BufWriter::new(File::create(root.join("ws_stream.jsonl"))?);
+        let routed_csv = Writer::from_path(root.join("orders_routed.csv"))?;
+        let meta = File::create(root.join("run_meta.json"))?;
+        Ok(Self { root, per_action, ws_stream, routed: routed_csv, meta })
+    }
+    pub fn log_per_action<T: serde::Serialize, A: serde::Serialize, O: serde::Serialize>(
+        &mut self, idx: usize, action: &str, submit_ts_ms: i64, req: &T, ack: Option<&A>, obs: Option<&O>
+    ) -> Result<()> {
+        let window = (submit_ts_ms / 200) * 200; // default 200ms window (mirrors composition bonus window). :contentReference[oaicite:2]{index=2}
+        let line = serde_json::json!({
+            "step_idx": idx, "action": action, "submit_ts_ms": submit_ts_ms,
+            "request": req, "ack": ack, "observed": obs, "window_key_ms": window
+        });
+        serde_json::to_writer(&mut self.per_action, &line)?;
+        self.per_action.get_mut().write_all(b"\n")?;
+        Ok(())
+    }
+    pub fn log_ws_raw(&mut self, v: &serde_json::Value) -> Result<()> {
+        serde_json::to_writer(&mut self.ws_stream, v)?;
+        self.ws_stream.get_mut().write_all(b"\n")?;
+        Ok(())
+    }
+    pub fn log_routed(&mut self, ack: &crate::client::PostOrderAck, reqs: &[crate::client::PostOrderReq]) -> Result<()> {
+        // If multiple orders submitted, write one line per ack (use matched req if needed).
+        for r in reqs {
+            self.routed.write_record(&[
+                Utc::now().timestamp_millis().to_string(),
+                ack.oid.to_string(), &ack.coin, &r.side, &format!("{}", r.px), &format!("{}", r.sz),
+                &r.tif, &r.reduce_only.to_string(), r.builder_code.as_deref().unwrap_or("")
+            ])?;
+        }
+        self.routed.flush()?;
+        Ok(())
+    }
+    pub fn write_meta(&mut self, meta: &serde_json::Value) -> Result<()> {
+        serde_json::to_writer_pretty(&mut self.meta, meta)?;
+        Ok(())
+    }
+}
+```
+
+---
+
+## 5) WS correlation rules (deterministic, evaluator‑friendly)
+
+* [ ] **Order post**: treat **first** of `{orderAccepted, orderOpen, fill}` with **matching `oid`** as the observed effect.
+* [ ] **Cancel**: accept `{cancelled}` with `oid` OR disappearance event for that `oid`.
+* [ ] **USD transfer**: accept `ledgerUpdates` with matching delta (direction & magnitude).
+* [ ] **Set leverage**: accept a risk parameter / config update event (or HTTP readback if the SDK provides it).
+* [ ] **Timeout**: default **2000 ms**; if no effect arrives, record `"observed": null` and the evaluator will **no‑op filter** it to zero score.&#x20;
+
+---
+
+## 6) Edge‑case handling (must do)
+
+* [ ] **Tick / lot rounding**: round passive orders toward passivity (ALO → make price strictly passive vs best quote).
+* [ ] **Mid‑price dependency**: if `px="mid±%"` but mid feed not ready, **delay** up to 200 ms (retry 5× at 40 ms).
+* [ ] **Builder Code**: pass through where API allows; always log it in `orders_routed.csv`.
+* [ ] **Idempotency**: if HTTP returns duplicate ack for same request, dedupe by `(coin, side, px, sz, tif)` hash.
+* [ ] **Threading**: single threaded `Executor` (sequential steps) to keep correlation simple for MVP.
+
+---
+
+## 7) Quick smoke scripts
+
+**`scripts/run_cov.sh`**
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+OUT="runs/$(date +%Y%m%d-%H%M%S)"
+cargo run -p hl-runner -- \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --out "$OUT" \
+  --endpoint-http "${HL_ENDPOINT_HTTP}" \
+  --endpoint-ws   "${HL_ENDPOINT_WS}" \
+  ${HL_BUILDER_CODE:+--builder-code "$HL_BUILDER_CODE"}
+echo "wrote $OUT"
+```
+
+---
+
+## 8) Validation checklist (before evaluator)
+
+* [ ] `per_action.jsonl` contains at least one **perp\_orders** line with non‑null `ack` and an `observed` WS effect.
+* [ ] For the cancel step, `observed.type == "cancelled"` (or equivalent).
+* [ ] `ws_stream.jsonl` has at least one snapshot and one delta.
+* [ ] `orders_routed.csv` first row shows expected columns; `builder_code` is present if provided.
+* [ ] All action lines carry `submit_ts_ms` and `window_key_ms`.
+
+---
+
+## 9) Why this planning mirrors the proven design (for judges)
+
+* **Step 1→4 flow** (declare metrics → deploy/plan → execute → evaluate) is the same pattern you used in SuiBench, giving reproducibility and easy CI integration.&#x20;
+* **Coverage math** (Base + Bonus − Penalty with a **composition bonus** per time window) is identical in spirit—only the unit changes from “MoveCall” to **venue action signature**.&#x20;
+* **No‑op filter**: any action with no observable effect scores zero, preventing gaming.&#x20;
+
+---
+
+## 10) Minimal test stubs (optional but fast)
+
+```rust
+// crates/hl-runner/tests/smoke.rs
+use std::fs;
+#[test]
+fn writes_artifacts_shape() {
+    // Pretend to log a couple of lines and assert schema keys exist.
+    let sample = r#"{"step_idx":0,"action":"perp_orders","submit_ts_ms":1,"request":{},"ack":{"oid":1},"observed":{"type":"orderAccepted"},"window_key_ms":0}"#;
+    let v: serde_json::Value = serde_json::from_str(sample).unwrap();
+    assert!(v.get("submit_ts_ms").is_some());
+    assert!(v.get("window_key_ms").is_some());
+}
+```
+
+---
+
+## 11) Next (time‑boxed)
+
+* [ ] Wire the **actual** SDK calls inside `HlHttp` and WS subscription payloads in `HlWs`/`book`; test with a test key.
+* [ ] Prove one run of `hl_perp_basic_01.jsonl:1` writes all artifacts.
+* [ ] Hand to evaluator (coverage + HiaN).
+* [ ] (If time) Add `cancel_oids` multi‑OID and `IOC reduceOnly` example for a slightly higher score spread.
+
+---
+
+**Footnotes**
+The “Step 1–4” workflow slide, architecture diagram, and “Coverage Scoring Details (Base + Bonus − Penalty)” are reused ideas from your SuiBench deck to keep the learning curve for judges minimal and emphasize reproducibility & CI friendliness.&#x20;
+
+---
+
+If you want, I can also draft `domains-hl.yaml` and the first `hl_perp_basic_01.jsonl` in your repo style next.
@@ -0,0 +1,600 @@
+Below is a ready‑to‑drop‑in **`PLAN_3_2.md`** for HyperLiquidBench — the **coverage evaluator** (Step 3.2). It specifies *what to score*, *how to normalize* runner artifacts into **signatures**, **Base + Bonus − Penalty** math, file formats, CLI, and tests, along with Rust code skeletons you can paste into a new crate (`crates/hl-evaluator`).
+The scoring math follows the same blueprint you used for SuiBench (Base + Bonus − Penalty, windowed composition bonus, no‑op filter).&#x20;
+
+---
+
+# PLAN\_3\_2.md — HyperLiquidBench Evaluator (Coverage)
+
+> **Goal:** Convert confirmed runner effects (from `per_action.jsonl` and optional `orders_routed.csv`) into **normalized action signatures**, then compute a deterministic **FINAL\_SCORE = Base + Bonus − Penalty** per run.&#x20;
+
+## 0) Inputs & Outputs
+
+**Inputs (from `hl-runner`):**
+
+* `runs/<ts>/per_action.jsonl` — one JSON object per executed step (already logged by your runner).
+* `runs/<ts>/orders_routed.csv` — optional; used for cross‑checks (OIDs, coins, etc.).
+* `dataset/domains-hl.yaml` — domain weights + allowlists (see §2.3).
+
+**Outputs (written next to inputs):**
+
+* `runs/<ts>/eval_per_action.jsonl` — per‑step summarized effects + signatures.
+* `runs/<ts>/eval_score.json` — final score, domain breakdown, bonus/penalties, unique signatures.
+* `runs/<ts>/unique_signatures.json` — flat list of unique signatures seen (debug/inspection).
+
+---
+
+## 1) Effect → Signature normalization
+
+We **do not** score raw API payloads. We first **normalize** each confirmed effect to a compact *signature* string. These signatures are the unit of coverage.
+
+### 1.1 Signature vocabulary
+
+| Action family    | Signature pattern                         | Examples                                                               |
+| ---------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
+| Perp order       | `perp.order.{tif}:{reduceOnly}:{trigger}` | `perp.order.GTC:false:none`, `perp.order.ALO:true:none`                |
+| Perp cancel      | `perp.cancel.{scope}`                     | `perp.cancel.last`, `perp.cancel.oids`, `perp.cancel.all`              |
+| Account transfer | `account.usdClassTransfer.{direction}`    | `account.usdClassTransfer.toPerp`, `account.usdClassTransfer.fromPerp` |
+| Risk             | `risk.setLeverage.{coin}`                 | `risk.setLeverage.BTC`, `risk.setLeverage.ETH`                         |
+
+Notes:
+
+* **TIF** comes from order request (`Gtc/Alo/Ioc`) — preserve case as Hyperliquid expects.
+* **reduceOnly** is lowercased `true/false`.
+* **trigger** is currently `none` (we’ll add `tp/sl` later when runner supports triggers).
+* **scope** values: `last`, `oids`, `all`.
+* **coin** is the user‑visible symbol (e.g., `BTC`, `ETH`) as provided to the order API.
+
+### 1.2 What counts as a **confirmed effect**?
+
+A step contributes signatures **only if**:
+
+* `ack.status == "ok"` **and**
+* for `perp_orders`: **at least one** per‑order status is **not** `"error"` (e.g., `resting`, `filled`, `success`, `waitingForFill`, `waitingForTrigger`)
+  *(Your runner already produces a compact `ack` structure via `exchange_status_json()`.)*
+* for cancels/transfers/setLeverage: `ack.status == "ok"` is sufficient.
+
+**No‑op filter:** if `ack.status != "ok"` **and** there is **no** helpful `observed` WS evidence, the step is ignored (0 effect). This mirrors the *no‑op filter* used in SuiBench.&#x20;
+
+### 1.3 Multiple orders in one step
+
+For `perp_orders`, a single step may contain *N* orders. We produce up to *N* **signatures** (one per *accepted* order), all sharing the same `window_key_ms` (see §3.2).
+
+---
+
+## 2) Scoring model
+
+**FINAL\_SCORE = Base + Bonus − Penalty**. Details mirror the deck you used before, adapted from MoveCall→venue‑action.&#x20;
+
+### 2.1 Base (domain‑weighted uniques)
+
+* Partition signatures into **domains** using `domains-hl.yaml` (see §2.3).
+* For each domain *d*: **Base\_d = weight\[d] × |UniqueSignatures(d)|**.
+* **Base = Σ\_d Base\_d**.
+
+### 2.2 Windowed composition bonus
+
+* Group normalized signatures by their **window key**. We **reuse** `window_key_ms` that `hl-runner` already writes on each step (floor of `submit_ts_ms` to `window_ms`, default **200 ms**).
+* For each window: **`+0.25 × max(0, distinct_in_window − 1)`**.
+  (Encourages composing multiple distinct actions in a tightly batched intent; exact same formula you used in SuiBench.)&#x20;
+
+### 2.3 Domain configuration (`dataset/domains-hl.yaml`)
+
+```yaml
+version: "0.1"
+per_action_window_ms: 200          # default; can be overridden by CLI
+per_signature_cap: 3               # beyond this, repeats don’t add Base (see §2.4)
+
+domains:
+  perp:
+    weight: 1.0
+    allow:
+      - "perp.order.*"
+      - "perp.cancel.*"
+  account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.*"
+  risk:
+    weight: 1.0
+    allow:
+      - "risk.setLeverage.*"
+```
+
+> **Matching rule:** `*` is a single‑segment wildcard (glob on the `.`‑separated parts). We match **literal strings** otherwise.
+
+### 2.4 Penalties & caps
+
+* **Per‑signature cap**: after **3** occurrences of the *same* signature in Base, further repeats do not increase Base. (Optional: `--repeat-penalty -0.1` per excess repeat.)
+* **No‑op**: ignored (not a penalty; just not counted).
+* **Future hooks:** spam penalty, per‑window duplicate suppression, model‑wide cooldown.
+
+---
+
+## 3) Evaluator crate layout (`crates/hl-evaluator`)
+
+```
+crates/hl-evaluator/
+├── Cargo.toml
+└── src/
+    ├── main.rs            # CLI entry
+    ├── cli.rs             # args, subcommands
+    ├── config.rs          # domains-hl.yaml loader + glob matcher
+    ├── model.rs           # data structs (ActionRecord, Signature, Summary, Report)
+    ├── parse.rs           # read per_action.jsonl -> Effect(s)
+    ├── score.rs           # Base/Bonus/Penalty engine
+    └── util.rs            # io, hashing, window helpers
+```
+
+### 3.1 Cargo.toml (minimal)
+
+```toml
+[package]
+name = "hl-evaluator"
+version = "0.1.0"
+edition = "2021"
+
+[dependencies]
+anyhow = "1"
+clap = { version = "4.5", features = ["derive"] }
+globset = "0.4"
+indexmap = "2"
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+serde_yaml = "0.9"
+chrono = { version = "0.4", default-features = false, features = ["clock","std"] }
+```
+
+### 3.2 Data model (`model.rs`)
+
+```rust
+use serde::{Deserialize, Serialize};
+use indexmap::IndexMap;
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ActionLogRecord {
+    pub step_idx: usize,
+    pub action: String,
+    pub submit_ts_ms: i64,
+    pub window_key_ms: i64,
+    pub request: serde_json::Value,
+    pub ack: Option<serde_json::Value>,
+    pub observed: Option<serde_json::Value>,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)]
+pub struct Signature(pub String);
+
+#[derive(Debug, Serialize)]
+pub struct PerActionSummary {
+    pub step_idx: usize,
+    pub window_key_ms: i64,
+    pub signatures: Vec<Signature>,
+    pub ignored_noop: bool,
+}
+
+#[derive(Debug, Serialize)]
+pub struct ScoreReport {
+    pub final_score: f64,
+    pub by_domain: IndexMap<String, f64>,
+    pub bonus: f64,
+    pub penalty: f64,
+    pub unique_sigs: IndexMap<String, Vec<Signature>>,
+    pub per_signature_counts: IndexMap<String, usize>,
+}
+```
+
+### 3.3 Domain config & matching (`config.rs`)
+
+```rust
+use anyhow::{Context, Result};
+use globset::{Glob, GlobSet, GlobSetBuilder};
+use serde::Deserialize;
+use std::{collections::HashMap, fs::File, path::Path};
+
+#[derive(Debug, Deserialize)]
+pub struct DomainsConfig {
+    #[serde(default)]
+    pub per_action_window_ms: Option<i64>,
+    #[serde(default)]
+    pub per_signature_cap: Option<usize>,
+    pub domains: HashMap<String, DomainRule>,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct DomainRule {
+    pub weight: f64,
+    pub allow: Vec<String>,
+}
+
+pub struct DomainMatcher {
+    rules: Vec<(String, f64, GlobSet)>,
+}
+
+impl DomainMatcher {
+    pub fn new(cfg: &DomainsConfig) -> Result<Self> {
+        let mut rules = Vec::new();
+        for (name, rule) in &cfg.domains {
+            let mut b = GlobSetBuilder::new();
+            for pat in &rule.allow {
+                // interpret dot-separated signature segments with '*' wildcards
+                b.add(Glob::new(pat).with_context(|| format!("bad glob: {pat}"))?);
+            }
+            rules.push((name.clone(), rule.weight, b.build()?));
+        }
+        Ok(Self { rules })
+    }
+
+    pub fn classify(&self, sig: &str) -> Option<(&str, f64)> {
+        for (name, weight, set) in &self.rules {
+            if set.is_match(sig) {
+                return Some((name.as_str(), *weight));
+            }
+        }
+        None
+    }
+}
+
+pub fn load_domains(path: &Path) -> Result<DomainsConfig> {
+    let file = File::open(path).with_context(|| format!("open {:?}", path))?;
+    let cfg: DomainsConfig = serde_yaml::from_reader(file).context("parse domains-hl.yaml")?;
+    Ok(cfg)
+}
+```
+
+### 3.4 Parsing (`parse.rs`)
+
+```rust
+use anyhow::Result;
+use serde_json::Value;
+use crate::model::{ActionLogRecord, PerActionSummary, Signature};
+
+fn ack_ok(ack: &Value) -> bool {
+    ack.get("status").and_then(|s| s.as_str()) == Some("ok")
+}
+
+fn per_order_statuses(ack: &Value) -> Vec<String> {
+    ack.pointer("/data/statuses")
+        .and_then(|v| v.as_array())
+        .unwrap_or(&vec![])
+        .iter()
+        .filter_map(|s| s.get("kind").and_then(|k| k.as_str()).map(|s| s.to_string()))
+        .collect()
+}
+
+fn is_effectful_status(kind: &str) -> bool {
+    matches!(kind, "success" | "resting" | "filled" | "waitingForFill" | "waitingForTrigger")
+}
+
+pub fn summarize(record: ActionLogRecord) -> PerActionSummary {
+    let mut signatures = Vec::new();
+    let mut noop = false;
+
+    match record.action.as_str() {
+        "perp_orders" => {
+            let req_orders = record.request.pointer("/perp_orders/orders")
+                .and_then(|v| v.as_array()).cloned().unwrap_or_default();
+            let statuses = record.ack.as_ref().map(per_order_statuses).unwrap_or_default();
+
+            // align per-order: zip req_orders with statuses; if statuses shorter, assume ok
+            for (idx, req) in req_orders.into_iter().enumerate() {
+                let tif = req.get("tif").and_then(|v| v.as_str()).unwrap_or("Gtc");
+                let reduce = req.get("reduceOnly").and_then(|v| v.as_bool()).unwrap_or(false);
+                let trig = "none"; // future: read req["trigger"]
+                let sig = format!("perp.order.{tif}:{reduce}:{trig}");
+
+                let status_ok = statuses.get(idx)
+                    .map(|k| is_effectful_status(k))
+                    .unwrap_or_else(|| record.ack.as_ref().map(ack_ok).unwrap_or(false));
+
+                if status_ok {
+                    signatures.push(Signature(sig));
+                }
+            }
+            if signatures.is_empty() && record.ack.as_ref().map(ack_ok) != Some(true) && record.observed.is_none() {
+                noop = true;
+            }
+        }
+        "cancel_last" => {
+            if record.ack.as_ref().map(ack_ok) == Some(true) {
+                signatures.push(Signature("perp.cancel.last".to_string()));
+            } else { noop = true; }
+        }
+        "cancel_oids" => {
+            if record.ack.as_ref().map(ack_ok) == Some(true) {
+                signatures.push(Signature("perp.cancel.oids".to_string()));
+            } else { noop = true; }
+        }
+        "cancel_all" => {
+            if record.ack.as_ref().map(ack_ok) == Some(true) {
+                signatures.push(Signature("perp.cancel.all".to_string()));
+            } else { noop = true; }
+        }
+        "usd_class_transfer" => {
+            if record.ack.as_ref().map(ack_ok) == Some(true) {
+                let dir = record.request.pointer("/usd_class_transfer/toPerp")
+                    .and_then(|v| v.as_bool()).unwrap_or(true);
+                signatures.push(Signature(format!(
+                    "account.usdClassTransfer.{}",
+                    if dir { "toPerp" } else { "fromPerp" }
+                )));
+            } else { noop = true; }
+        }
+        "set_leverage" => {
+            if record.ack.as_ref().map(ack_ok) == Some(true) {
+                let coin = record.request.pointer("/set_leverage/coin")
+                    .and_then(|v| v.as_str()).unwrap_or("UNKNOWN");
+                signatures.push(Signature(format!("risk.setLeverage.{coin}")));
+            } else { noop = true; }
+        }
+        _ => {}
+    }
+
+    PerActionSummary {
+        step_idx: record.step_idx,
+        window_key_ms: record.window_key_ms,
+        signatures,
+        ignored_noop: noop,
+    }
+}
+```
+
+### 3.5 Scoring (`score.rs`)
+
+```rust
+use crate::config::DomainMatcher;
+use crate::model::{PerActionSummary, ScoreReport, Signature};
+use indexmap::IndexMap;
+use std::collections::{HashMap, HashSet};
+
+pub struct ScoreState<'a> {
+    domains: &'a DomainMatcher,
+    per_sig_cap: usize,
+    base_by_domain: IndexMap<String, HashSet<String>>,
+    counts_per_sig: HashMap<String, usize>,
+    bonus_total: f64,
+    penalty_total: f64,
+    // for bonus:
+    windows: HashMap<i64, HashSet<String>>,
+}
+
+impl<'a> ScoreState<'a> {
+    pub fn new(domains: &'a DomainMatcher, per_sig_cap: usize) -> Self {
+        Self {
+            domains,
+            per_sig_cap,
+            base_by_domain: IndexMap::new(),
+            counts_per_sig: HashMap::new(),
+            bonus_total: 0.0,
+            penalty_total: 0.0,
+            windows: HashMap::new(),
+        }
+    }
+
+    pub fn incorporate(&mut self, s: PerActionSummary) {
+        if s.ignored_noop { return; }
+
+        // composition window
+        let w = self.windows.entry(s.window_key_ms).or_default();
+
+        for Signature(sig) in s.signatures {
+            // bonus window collects *distinct signature strings*
+            w.insert(sig.clone());
+
+            // cap counts for Base
+            let c = self.counts_per_sig.entry(sig.clone()).or_insert(0);
+            if *c < self.per_sig_cap {
+                *c += 1;
+                if let Some((d, _w)) = self.domains.classify(&sig) {
+                    self.base_by_domain.entry(d.to_string()).or_default().insert(sig.clone());
+                }
+            } else {
+                // optional: accumulate penalties for spam beyond cap
+                // self.penalty_total += 0.0;
+            }
+        }
+    }
+
+    pub fn finalize(mut self, weights: &HashMap<String, f64>) -> ScoreReport {
+        // compute bonus
+        for (_win, set) in self.windows.drain() {
+            let k = set.len() as i64;
+            if k > 1 {
+                self.bonus_total += 0.25 * (k as f64 - 1.0);
+            }
+        }
+
+        // Base
+        let mut by_domain = IndexMap::new();
+        let mut unique = IndexMap::new();
+        for (d, set) in &self.base_by_domain {
+            let w = *weights.get(d).unwrap_or(&1.0);
+            by_domain.insert(d.clone(), w * set.len() as f64);
+            unique.insert(d.clone(), set.iter().cloned().map(Signature).collect());
+        }
+
+        let base_sum: f64 = by_domain.values().sum();
+        ScoreReport {
+            final_score: base_sum + self.bonus_total - self.penalty_total,
+            by_domain,
+            bonus: self.bonus_total,
+            penalty: self.penalty_total,
+            unique_sigs: unique,
+            per_signature_counts: self.counts_per_sig,
+        }
+    }
+}
+```
+
+### 3.6 CLI (`cli.rs` + `main.rs`)
+
+```rust
+// main.rs
+use anyhow::Result;
+#[tokio::main]
+async fn main() -> Result<()> { hl_evaluator::cli::run().await }
+
+// cli.rs
+use crate::{config::{load_domains, DomainMatcher}, model::{ActionLogRecord}, parse::summarize, score::ScoreState};
+use anyhow::{Context, Result};
+use chrono::Utc;
+use clap::Parser;
+use indexmap::IndexMap;
+use serde_json::Value;
+use std::{fs::File, io::{BufRead, BufReader, BufWriter, Write}, path::PathBuf};
+use std::collections::HashMap;
+
+#[derive(Parser, Debug)]
+#[command(about="HyperLiquidBench Evaluator (coverage)")]
+pub struct Cli {
+  #[arg(long)] input: PathBuf,                 // per_action.jsonl
+  #[arg(long)] domains: PathBuf,               // dataset/domains-hl.yaml
+  #[arg(long)] out_dir: Option<PathBuf>,
+  #[arg(long)] window_ms: Option<i64>,
+  #[arg(long, default_value_t=3)] cap_per_sig: usize,
+}
+
+pub async fn run() -> Result<()> {
+  let cli = Cli::parse();
+  let out_dir = cli.out_dir.unwrap_or_else(|| cli.input.parent().unwrap().to_path_buf());
+  std::fs::create_dir_all(&out_dir)?;
+
+  // load domains
+  let cfg = load_domains(&cli.domains)?;
+  let matcher = DomainMatcher::new(&cfg)?;
+  let window_ms = cli.window_ms.or(cfg.per_action_window_ms).unwrap_or(200);
+
+  let mut weights = HashMap::new();
+  for (name, rule) in cfg.domains.iter() { weights.insert(name.clone(), rule.weight); }
+
+  let eval_path = out_dir.join("eval_per_action.jsonl");
+  let mut eval_writer = BufWriter::new(File::create(&eval_path)?);
+
+  let mut state = ScoreState::new(&matcher, cli.cap_per_sig);
+
+  // read JSONL
+  let file = File::open(&cli.input).with_context(|| format!("open {:?}", cli.input))?;
+  for line in BufReader::new(file).lines() {
+      let line = line?;
+      if line.trim().is_empty() { continue; }
+      let rec: ActionLogRecord = serde_json::from_str(&line)?;
+      // override window if caller demanded different window size:
+      let mut rec = rec;
+      if window_ms > 0 && rec.window_key_ms % window_ms != 0 {
+          rec.window_key_ms = (rec.submit_ts_ms / window_ms) * window_ms;
+      }
+      let sum = summarize(rec);
+      serde_json::to_writer(&mut eval_writer, &sum)?; eval_writer.write_all(b"\n")?;
+      state.incorporate(sum);
+  }
+  eval_writer.flush()?;
+
+  let report = state.finalize(&weights);
+  let out = out_dir.join("eval_score.json");
+  serde_json::to_writer_pretty(File::create(out)?, &report)?;
+  Ok(())
+}
+```
+
+---
+
+## 4) Test plan
+
+**Unit tests** (table‑driven):
+
+* ✅ `perp_orders` GTC/ALO/IOC with mixed statuses (`resting`, `filled`, `error`) → only accept effectful ones; signatures mapped correctly.
+* ✅ `cancel_*` with `ack.status=ok` → one signature per step.
+* ✅ `usd_class_transfer` to/from perp → correct direction suffix.
+* ✅ `set_leverage` emits `risk.setLeverage.{coin}`.
+* ✅ **No‑op**: ack `err` and no `observed` → ignored.
+* ✅ **Window bonus**: two distinct signatures in same `window_key_ms` → `+0.25`.
+* ✅ **Per‑signature cap**: 4 repeats of identical `perp.order.GTC:false:none` → Base counts max 3.
+
+**Golden run**:
+
+* Place 2 orders (GTC false none) + cancel\_last within 200 ms window:
+  Unique: `{perp.order.GTC:false:none, perp.cancel.last}` → Base `2.0`.
+  Bonus: `0.25 × (2−1) = 0.25`.
+  **FINAL = 2.25** (mirrors your current plateau, good sanity check).
+* Add a *third* distinct signature in same window (e.g., `account.usdClassTransfer.toPerp`) → **3.5**.
+
+---
+
+## 5) How to run
+
+```bash
+# Build
+cargo build -p hl-evaluator
+
+# Score a run directory
+RUN_DIR=$(ls -dt runs/* | head -n1)
+cargo run -p hl-evaluator -- \
+  --input "$RUN_DIR/per_action.jsonl" \
+  --domains dataset/domains-hl.yaml \
+  --out-dir "$RUN_DIR" \
+  --window-ms 200 \
+  --cap-per-sig 3
+
+cat "$RUN_DIR/eval_score.json"
+```
+
+---
+
+## 6) Edge cases & guardrails
+
+* **Length mismatch** (`perp_orders`: N requests but M statuses): zip up to `min(N,M)`; remaining requests inherit step‑level `ack.status`.
+* **Case sensitivity**: keep `TIF` title‑case (`Gtc/Alo/Ioc`) because that’s what the SDK emits; stringify exactly in signatures.
+* **Coins**: treat as opaque symbols (`BTC`, `ETH`) from the request; do not lowercase unless you normalize in runner and domains.
+* **WS noise**: evaluator relies primarily on `ack.status`; `observed` is only a safety valve for the **no‑op filter**.
+* **Determinism**: with the same `per_action.jsonl`, same `domains-hl.yaml`, and same `window_ms`, the score is fully reproducible.
+
+---
+
+## 7) Why this design (rationale)
+
+* **Direct artifact scoring**: We score **what actually happened** (ack’d effects), not intentions.
+* **Windowed bonus** encourages tight, composed trading intents (equivalent to PTB composition).&#x20;
+* **Domain weights** let hackathon tracks (Programmable Trading / Builder Codes / Tooling) tune what “good coverage” means.
+* **Caps** prevent gaming via spam and keep coverage interpretable.
+
+---
+
+## 8) Next steps (follow‑ups for 3.3/3.4)
+
+* Extend signature vocabulary: `perp.order.{...:trigger}` with `tp/sl` once runner emits triggers.
+* Add **repeat penalties** and **cooldowns** (per signature / per domain).
+* Publish **example `domains-hl.yaml` variants** (Bronze/Silver/Gold) to change difficulty without code changes.
+* Produce a small HTML report (sparkline of windows, signatures timeline) from `eval_per_action.jsonl`.
+
+---
+
+**Appendix A — Example `per_action.jsonl` line (runner output)**
+
+```json
+{
+  "stepIdx": 1,
+  "action": "perp_orders",
+  "submitTsMs": 1737465405123,
+  "windowKeyMs": 1737465405000,
+  "request": {
+    "perp_orders": {
+      "orders": [
+        {"coin":"BTC","side":"buy","sz":0.01,"tif":"Gtc","reduceOnly":false,"px":"mid+0%","resolvedPx":98765.0}
+      ],
+      "builderCode":"myapp"
+    }
+  },
+  "ack": { "status":"ok", "responseType":"BulkOrder", "data":{"statuses":[{"kind":"resting","oid":12345678}] } },
+  "observed": [{"channel":"orderUpdates","oid":12345678,"status":"resting"}]
+}
+```
+
+**Normalized signatures (evaluator):**
+
+```
+["perp.order.Gtc:false:none"]
+```
+
+---
+
+This plan is **implementation‑ready** and consistent with your existing runner artifacts and prior SuiBench scoring philosophy. If you follow the file skeletons above, you’ll have a working `hl-evaluator` capable of producing a stable **FINAL\_SCORE** and domain breakdown for each run.
@@ -0,0 +1,529 @@
+Below is a *developer‑facing* spec you can drop into the repo as:
+
+```
+docs/DETAILED_PLAN_3_3.md
+```
+
+It is precise enough for another engineer to implement the **HiaN validator** end‑to‑end without further context.
+
+---
+
+# 3.3 HiaN (Haystack‑in‑a‑Needle) Validator — Detailed Plan
+
+**Goal:** Given a runner output directory (with `per_action.jsonl` and `ws_stream.jsonl`) and a `ground_truth.json`, deterministically decide **PASS/FAIL** for a long‑context task by verifying that the **exact required on‑venue effects** occurred (in order), and emit a compact diff when they did not.
+
+This validator is **orthogonal** to the coverage scorer (§3.2). It does not count diversity; it verifies *specific intent*.
+
+---
+
+## 0) Inputs & Outputs
+
+### Inputs
+
+* **`per_action.jsonl`** — one line per submitted step from the runner (already implemented in 3.1). Each line is an `ActionLogRecord`:
+
+  ```json
+  {
+    "stepIdx": 0,
+    "action": "usd_class_transfer",
+    "submitTsMs": 1737440123456,
+    "windowKeyMs": 1737440123400,
+    "request": { "usd_class_transfer": { "toPerp": true, "usdc": 25.0 } },
+    "ack":     { "status": "ok", "responseType": "OrderResponse", "data": ... },
+    "observed": { "channel": "accountClassTransfer", "toPerp": true, "usdc": 25.0, "time": 1737440123490 },
+    "notes": null
+  }
+  ```
+
+  *Produced by `hl-runner` via `RunArtifacts::log_action`.*
+
+* **`ws_stream.jsonl`** — raw websocket frames persisted by the runner (already implemented). Used as a fallback for correlating effects if a record’s `observed` field is missing.
+
+* **`ground_truth.json`** — the answer key for this HiaN case (schema below).
+
+### Outputs
+
+* **Exit code**: `0` on PASS, `2` on FAIL, `1` on internal error.
+* **`eval_hian.json`** — machine‑readable result:
+
+  ```json
+  {
+    "pass": true,
+    "matched": [
+      { "expectIdx": 0, "kind": "usd_class_transfer", "matchedAt": 2, "tsMs": 1737440123456 },
+      { "expectIdx": 1, "kind": "perp_order", "matchedAt": 3, "oid": 1234567890, "fill": {"px": "3875.1", "sz": "0.01"} }
+    ],
+    "missing": [],
+    "extra": [],
+    "metrics": {
+      "latencyMs": { "0": 34, "1": 211 },
+      "windowMs": 200
+    },
+    "settings": {
+      "amountTolerance": 0.01,
+      "pxTolerancePct": 0.2,
+      "szTolerancePct": 0.5,
+      "withinMs": 2000
+    }
+  }
+  ```
+* **`eval_hian_diff.txt`** — compact human diff on FAIL (see §6).
+
+---
+
+## 1) `ground_truth.json` schema
+
+HiaN cases describe a **sequence** of expected effects. Each step can specify **exact** values or **matchers** with tolerances.
+
+```jsonc
+{
+  "caseId": "auditor-transfer-then-sell",
+  "withinMs": 2000,              // optional: max allowed gap between consecutive steps
+  "windowMs": 200,               // optional: windowing to align with runner's window_key_ms
+  "steps": [
+    {
+      "usdClassTransfer": {
+        "toPerp": true,
+        "usdc": { "eq": 25.0, "tol": 0.01 }  // equals within ±0.01 USDC
+      }
+    },
+    {
+      "perpOrder": {
+        "coin": "ETH",
+        "side": "sell",                          // exact
+        "tif": "IOC",                            // exact
+        "reduceOnly": true,                      // exact
+        "sz": { "ge": 0.005, "le": 0.2 },        // size range (units per venue)
+        // price may be absolute or relative-to-mid; we verify the *executed* price or resting px
+        "px": { "mode": "abs", "val": 0 },       // mode ∈ {"abs","ignore"} — set "ignore" to skip px check
+        "requireFill": true                      // if true, must see fill; else resting accepted
+      }
+    }
+  ]
+}
+```
+
+**Notes**
+
+* For price we start conservative: **`"px": { "mode": "ignore" }`** for MVP (venues often change mid). If needed we can implement more expressive matchers later (e.g., `"mode": "midPct", "le": 0.5`).
+* Add optional global defaults via flags in the CLI (`--amount-tol`, `--px-tol-pct`, …) to override per‑step fields.
+
+---
+
+## 2) Crate & File Layout
+
+Add a new binary crate:
+
+```
+crates/hl-evaluator/
+  Cargo.toml
+  src/
+    main.rs            // CLI
+    cli.rs             // arg parsing
+    io.rs              // file readers (jsonl streaming)
+    types.rs           // GroundTruth structs, matchers
+    hian.rs            // core validator logic
+    diff.rs            // textual diff
+    util.rs            // float tolerance helpers, parsing enums
+```
+
+The crate depends on `hl-common` for `ActionLogRecord` and time/window helpers.
+
+---
+
+## 3) Types (Rust)
+
+```rust
+// crates/hl-evaluator/src/types.rs
+use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Deserialize)]
+pub struct GroundTruth {
+    pub case_id: String,
+    #[serde(default)]
+    pub within_ms: Option<u64>,
+    #[serde(default)]
+    pub window_ms: Option<i64>,
+    pub steps: Vec<ExpectStep>,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(untagged)]
+pub enum ExpectStep {
+    UsdClassTransfer { usd_class_transfer: ExpectTransfer },
+    PerpOrder       { perp_order: ExpectPerpOrder },
+    CancelLast      { cancel_last: ExpectCancelLast },
+    CancelOids      { cancel_oids: ExpectCancelOids },
+    CancelAll       { cancel_all: ExpectCancelAll },
+    SetLeverage     { set_leverage: ExpectSetLeverage },
+}
+
+#[derive(Debug, Deserialize)]
+pub struct ExpectTransfer {
+    pub to_perp: bool,
+    pub usdc: NumMatcher, // eq/ tol OR range
+}
+
+#[derive(Debug, Deserialize)]
+pub struct ExpectPerpOrder {
+    pub coin: String,
+    pub side: Side,           // "buy" | "sell"
+    pub tif: Tif,             // "ALO" | "GTC" | "IOC"
+    pub reduce_only: bool,
+    #[serde(default)] pub sz: NumMatcher,
+    #[serde(default)] pub px: PxMatcher,
+    #[serde(default)] pub require_fill: bool,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct ExpectCancelLast { pub coin: Option<String> }
+#[derive(Debug, Deserialize)]
+pub struct ExpectCancelOids { pub coin: String, pub oids: Vec<u64> }
+#[derive(Debug, Deserialize)]
+pub struct ExpectCancelAll  { pub coin: Option<String> }
+#[derive(Debug, Deserialize)]
+pub struct ExpectSetLeverage { pub coin: String, pub leverage: u32, #[serde(default)] pub cross: bool }
+
+#[derive(Debug, Deserialize, Clone, Copy, PartialEq, Eq)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum Tif { ALO, GTC, IOC }
+
+#[derive(Debug, Deserialize, Clone, Copy, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum Side { buy, sell }
+
+// Numeric matchers for tolerant comparisons:
+#[derive(Debug, Deserialize, Default)]
+#[serde(untagged)]
+pub enum NumMatcher {
+    // {"eq": 25.0, "tol": 0.01}
+    Eq { eq: f64, #[serde(default)] tol: Option<f64> },
+    // {"ge": 0.005, "le": 0.2}
+    Range { #[serde(default)] ge: Option<f64>, #[serde(default)] le: Option<f64> },
+    #[default]
+    Any,
+}
+
+#[derive(Debug, Deserialize, Default)]
+#[serde(tag = "mode", rename_all = "lowercase")]
+pub enum PxMatcher {
+    #[default]
+    Ignore,                  // do not check price
+    Abs { val: f64, #[serde(default)] tol: Option<f64> }, // |px - val| <= tol
+    // extend later: MidPct { le: f64, ge: Option<f64> }
+}
+
+// Results
+#[derive(Debug, Serialize)]
+pub struct HianResult {
+    pub pass: bool,
+    pub matched: Vec<MatchEntry>,
+    pub missing: Vec<MissingEntry>,
+    pub extra: Vec<usize>, // unmatched action indices if we ever enforce "exactly K"
+    pub metrics: serde_json::Value,
+    pub settings: serde_json::Value,
+}
+
+#[derive(Debug, Serialize)]
+pub struct MatchEntry {
+    pub expect_idx: usize,
+    pub kind: String,
+    pub matched_at: usize, // per_action line number (0-based)
+    pub ts_ms: i64,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub oid: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub fill: Option<serde_json::Value>,
+}
+
+#[derive(Debug, Serialize)]
+pub struct MissingEntry {
+    pub expect_idx: usize,
+    pub kind: String,
+    pub reason: String,
+}
+```
+
+---
+
+## 4) CLI
+
+```rust
+// crates/hl-evaluator/src/cli.rs
+use clap::Parser;
+
+#[derive(Parser, Debug)]
+#[command(about = "HyperLiquidBench evaluator (HiaN)")]
+pub struct Cli {
+    /// Path to ground_truth.json
+    #[arg(long)]
+    pub ground: std::path::PathBuf,
+    /// Path to per_action.jsonl (runner output)
+    #[arg(long)]
+    pub per_action: std::path::PathBuf,
+    /// Optional path to ws_stream.jsonl (for fallback observation)
+    #[arg(long)]
+    pub ws_stream: Option<std::path::PathBuf>,
+    /// Output directory (defaults to per_action's parent)
+    #[arg(long)]
+    pub out_dir: Option<std::path::PathBuf>,
+
+    // Global tolerances (override file if set)
+    #[arg(long)] pub amount_tol: Option<f64>,      // USDC absolute tol
+    #[arg(long)] pub px_tol: Option<f64>,          // absolute price tol
+    #[arg(long)] pub sz_ge: Option<f64>,           // default sz lower bound
+    #[arg(long)] pub within_ms: Option<u64>,       // inter-step time bound
+    #[arg(long, default_value_t = 200)] pub window_ms: i64,
+}
+```
+
+`main.rs` wires `Cli` → `hian::evaluate()` and writes `eval_hian.json` + prints `PASS`/`FAIL`.
+
+---
+
+## 5) Core Logic
+
+### 5.1 Read & normalize artifacts
+
+* Implement `io::read_per_action(path) -> Vec<ActionLogRecord>` by reading the JSONL file line by line (avoid loading `ws_stream.jsonl` unless needed).
+* Validate each `ActionLogRecord` has `action` ∈ the allowed set:
+
+    * `perp_orders`, `cancel_last`, `cancel_oids`, `cancel_all`, `usd_class_transfer`, `set_leverage`.
+* For matching, we **prefer `observed`**. If absent, try to reconstruct from `ack` or by scanning the fallback `ws_stream.jsonl` for a matching event near `submitTsMs` (±1s) using `window_key_ms` to narrow.
+
+### 5.2 Matching strategy (ordered sequence)
+
+We match **in order** across `ground_truth.steps`.
+
+Algorithm:
+
+```rust
+let mut cursor = 0usize; // index into per_action vector
+for (i, expect) in truth.steps.iter().enumerate() {
+    // search forward from cursor for the first action that matches `expect`
+    match find_match(expect, &actions[cursor..], config) {
+        Some((rel_idx, match_info)) => {
+            let idx = cursor + rel_idx;
+            // enforce inter-step temporal constraint if within_ms set
+            if i > 0 && config.within_ms.is_some() {
+                let prev_ts = matched.last().unwrap().ts_ms;
+                let now_ts  = actions[idx].submit_ts_ms;
+                if now_ts as u64 - prev_ts as u64 > config.within_ms.unwrap() {
+                    return FAIL("exceeded withinMs between steps i-1 and i")
+                }
+            }
+            record_match(i, idx, match_info);
+            cursor = idx + 1; // continue after the matched action
+        }
+        None => record_missing(i, expect, "no matching action observed in tail"),
+    }
+}
+```
+
+We do **not** require that there are no extra actions; we only care that *all expected* effects occur in sequence. (We can add a strict mode later.)
+
+### 5.3 Effect extractors
+
+Implement in `hian.rs`:
+
+```rust
+fn match_transfer(expect: &ExpectTransfer, act: &ActionLogRecord, cfg: &Cfg) -> Option<MatchEntry>;
+fn match_perp_order(expect: &ExpectPerpOrder, act: &ActionLogRecord, cfg: &Cfg) -> Option<MatchEntry>;
+fn match_cancel_last(expect: &ExpectCancelLast, act: &ActionLogRecord) -> Option<MatchEntry>;
+fn match_cancel_oids(expect: &ExpectCancelOids, act: &ActionLogRecord) -> Option<MatchEntry>;
+fn match_cancel_all(expect: &ExpectCancelAll, act: &ActionLogRecord) -> Option<MatchEntry>;
+fn match_set_leverage(expect: &ExpectSetLeverage, act: &ActionLogRecord) -> Option<MatchEntry>;
+```
+
+**Rules per kind:**
+
+* **`usd_class_transfer`**
+
+    * `act.action == "usd_class_transfer"` is required.
+    * Use `act.observed` if present, with shape:
+
+      ```json
+      { "channel": "accountClassTransfer", "toPerp": true, "usdc": 25.0, "time": ... }
+      ```
+    * Check `toPerp` equals, and `NumMatcher` against `usdc` (respect `amount_tol` override).
+    * Return `MatchEntry { ts_ms: act.submit_ts_ms }`.
+
+* **`perp_order`**
+
+    * `act.action == "perp_orders"` is required.
+    * Read `act.request.perp_orders.orders` vector and the corresponding `observed` (array of `orderUpdates`/`userFills`) and/or `ack`.
+    * We consider a match if **any single order** in this batch satisfies:
+
+        * `coin`, `side`, `tif`, `reduceOnly` equal (case‑insensitive for coin).
+        * `sz` satisfied by `NumMatcher` (if `Any`, skip).
+        * If `requireFill == true`, we must see at least one `userFills` event for this order ID (`oid`) in `observed`. Otherwise:
+
+            * Accept `Resting` or `Filled` in `ack.data.statuses`.
+    * Extract `oid` from `ack` (when status is `Resting`/`Filled`) using current helper `extract_oids` + per‑order index; fallback: parse from `observed` entry.
+    * Price (`px`) — for MVP with `"mode": "ignore"` do nothing. If `"Abs"` supplied: accept if `|executed_px - val| <= tol` (use `fill.px` when filled; else, use `request.resolvedPx`).
+    * Return `MatchEntry { oid, ts_ms, fill: Some({px, sz}) if filled }`.
+
+* **`cancel_last` / `cancel_oids` / `cancel_all`**
+
+    * `act.action` must match the expected cancel kind.
+    * For `cancel_last` with `coin: Some("ETH")`, the *request* must include the same coin; for `None`, accept any.
+    * For `cancel_oids`, compare the set of oids in `request` with expected `oids`.
+    * For `cancel_all`, if `coin: Some`, require the same coin in `request`.
+    * If `ack.status == "ok"`, treat as success (do not require WS confirm for MVP).
+
+* **`set_leverage`**
+
+    * `act.action == "set_leverage"`.
+    * Compare `coin`, `leverage`, `cross` exactly (from `request`).
+    * If `ack.status == "ok"`, accept; else fail.
+
+### 5.4 Utility matchers
+
+```rust
+fn num_match(m: &NumMatcher, val: f64, defaults: &Defaults) -> bool {
+    match m {
+        NumMatcher::Eq { eq, tol } => (val - *eq).abs() <= tol.unwrap_or(defaults.amount_tol),
+        NumMatcher::Range { ge, le } => {
+            let ok_ge = ge.map(|g| val >= g).unwrap_or(true);
+            let ok_le = le.map(|l| val <= l).unwrap_or(true);
+            ok_ge && ok_le
+        }
+        NumMatcher::Any => true,
+    }
+}
+```
+
+---
+
+## 6) Diff on FAIL (compact)
+
+`diff.rs` builds a short text file `eval_hian_diff.txt`:
+
+```
+HiaN FAIL (case auditor-transfer-then-sell)
+
+Step 0 expected: usd_class_transfer { toPerp: true, usdc ~= 25.00±0.01 }
+  ✗ Not found after action #1
+  Nearby events (±3):
+    #1 cancel_all { coin: "ETH" } @1737440123000
+    #2 usd_class_transfer { toPerp: true, usdc: 5.00 } @1737440123400  <-- amount mismatch
+    #3 perp_orders { coin: "ETH", side: "sell", tif: "IOC", sz: 0.01 } @1737440123456
+
+Step 1 expected: perp_order { coin: "ETH", side: sell, tif: IOC, reduceOnly: true, requireFill: true }
+  ✓ Matched at action #3, oid=1234567890, fill px=3875.1 sz=0.01
+```
+
+Implementation hints:
+
+* For a missing step, scan up to ±3 actions around `cursor` and print a one‑line summary derived from `request`.
+* Use emoji ticks/crosses for readability in terminal.
+
+---
+
+## 7) Tuning & Config
+
+* CLI overrides:
+
+    * `--within-ms`: enforce inter‑step maximum gap (default: from `ground_truth.json` if present).
+    * `--amount-tol`, `--px-tol`, etc. override defaults used in `NumMatcher::Eq`/`PxMatcher::Abs`.
+    * `--window-ms`: provide default if not present in truth file; also used to compute *latency per match* (`observed.time - submitTsMs`, rounded to window buckets).
+
+* Result metrics:
+
+    * For each matched step: record `latencyMs[i] = max(0, observed.time - submitTsMs)` if available; else `null`.
+    * Persist settings used so that runs are reproducible.
+
+---
+
+## 8) Tests
+
+Add `crates/hl-evaluator/tests/hian.rs` with fixtures.
+
+* **Fixture 1 — PASS minimal**
+
+    * `per_action.jsonl`:
+
+        1. `usd_class_transfer` observed `{toPerp:true, usdc:25.0}`
+        2. `perp_orders` observed `userFills` `{coin:"ETH", side:"sell", tif:"IOC", sz:"0.01", px:"3875.1", oid: 1}`
+    * `ground_truth.json` as in §1.
+    * Assert `pass == true`, `matched.len()==2`.
+
+* **Fixture 2 — FAIL amount off**
+
+    * Same as above but transfer `usdc: 24.9` with `tol: 0.01`.
+    * Assert `pass == false`, `missing[0].reason` contains `amount`.
+
+* **Fixture 3 — FAIL no fill required**
+
+    * Expect `requireFill: true` but only `Resting` ack and no `userFills`.
+    * Assert `pass == false`.
+
+* **Fixture 4 — Range matchers**
+
+    * `sz: {"ge":0.005,"le":0.02}` with `sz=0.01` → PASS.
+
+---
+
+## 9) Example wiring (CLI main)
+
+```rust
+// crates/hl-evaluator/src/main.rs
+use anyhow::Result;
+
+mod cli;  mod io;  mod types;  mod hian;  mod diff;  mod util;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let args = cli::Cli::parse();
+    let (result, out_dir) = hian::evaluate(args).await?;
+    let out = out_dir.join("eval_hian.json");
+    std::fs::write(&out, serde_json::to_string_pretty(&result)?)?;
+    println!("{}", if result.pass { "PASS" } else { "FAIL" });
+    std::process::exit(if result.pass {0} else {2});
+}
+```
+
+---
+
+## 10) Acceptance Criteria (MVP)
+
+* [ ] `hl-evaluator` builds as a standalone binary.
+* [ ] `hl-evaluator hian --ground runs/<ts>/ground_truth.json --per-action runs/<ts>/per_action.jsonl` produces `eval_hian.json` and **prints PASS/FAIL**.
+* [ ] **Ordered matching** is enforced, with optional `withinMs` constraint.
+* [ ] Uses `observed` first; gracefully falls back to `ack`.
+* [ ] Numeric comparisons respect matchers & tolerances.
+* [ ] On FAIL, `eval_hian_diff.txt` is created with a concise explanation.
+* [ ] Unit tests cover PASS/FAIL paths and tolerance edge cases.
+
+---
+
+## 11) Stretch (post‑MVP)
+
+* **Price match vs. mid**: introduce `PxMatcher::MidPct { le, ge }` and compute against a cached `resolvedPx` or a `mid` snapshot captured by the runner.
+* **Strict mode**: `--exact-k` to require no extraneous actions; generate `extra` entries.
+* **Multi‑order step**: add `"count": "atLeastOne" | "exactN"`.
+* **Latencies**: include p50/p95 of WS confirmation times.
+
+---
+
+## 12) Developer Notes
+
+* Keep the validator **pure read‑only**: it must not hit the network.
+* Treat all floats as `f64`, and **never** compare floats for exact equality; always use given tolerances or sensible defaults.
+* Defensive JSON parsing: values in `ack`/`observed` may be strings; parse to numbers with fallbacks.
+
+---
+
+## 13) Quick Run
+
+```bash
+# After running hl-runner and producing runs/<ts>/...
+cargo run -p hl-evaluator -- hian \
+  --ground runs/<ts>/ground_truth.json \
+  --per-action runs/<ts>/per_action.jsonl
+# -> prints PASS/FAIL and writes runs/<ts>/eval_hian.json
+```
+
+---
+
+This plan mirrors the artifacts emitted by the runner (3.1), adds a deterministic sequence‑matcher over venue effects, and produces actionable diagnostics that make HiaN tasks verifiable and repeatable.
@@ -0,0 +1,364 @@
+Below is a **developer‑ready** spec for **PLAN\_4.md – “Domains & Dataset”** for **HyperLiquidBench**. It locks down formats, file names, schemas, examples, and acceptance tests so another engineer can implement exactly what’s written.
+
+> **Design rationale.** Scoring follows the same principle as SuiBench — **FINAL\_SCORE = Base + Bonus − Penalty** — and uses a **no‑op/effect filter** so only meaningful actions count. See the “Coverage Scoring Details” slide (page 12) and “don’t trust, verify” motivation (pages 1–2) in the SuiBench deck.&#x20;
+
+---
+
+# PLAN\_4.md — Domains & Dataset (HyperLiquidBench)
+
+## Scope
+
+1. Define the **domains config** (`dataset/domains-hl.yaml`) used by the evaluator.
+2. Define the **coverage task set** (`dataset/tasks/*.jsonl`) that the runner can execute deterministically.
+3. Define the **HiaN (long‑context) case bundle** (`dataset/hian/**`) for pass/fail accuracy testing.
+4. Provide **CLI recipes + acceptance tests** to validate the pipeline end‑to‑end.
+
+---
+
+## 4.1 `dataset/domains-hl.yaml` (authoritative scoring config)
+
+### 4.1.1 File path
+
+```
+dataset/
+└── domains-hl.yaml
+```
+
+### 4.1.2 YAML schema (normative)
+
+```yaml
+version: "0.1"
+
+# Window (ms) for composition bonus: all distinct signatures whose
+# ActionLogRecord.windowKeyMs are equal are considered "composed".
+per_action_window_ms: 200
+
+# Max times a single signature may contribute to base score across the run.
+# Repeats beyond this cap incur a penalty.
+per_signature_cap: 3
+
+domains:
+  <domain-name>:
+    weight: <float>            # multiplier applied to the number of unique signatures in this domain
+    allow:                     # list of dot‑separated patterns with '*' wildcards (segment level)
+      - "<pattern>"
+      - "<pattern>"
+  ...
+```
+
+**Pattern grammar (dot‑segments)**
+
+* `Literal` segment: exact, case‑sensitive match (e.g., `perp`, `order`, `GTC:false:none`).
+* `*` segment: matches **any single** segment.
+* The number of segments in a pattern **must equal** the number in the signature.
+
+**Scoring defaults**
+
+* `per_action_window_ms` defaults to 200 if omitted.
+* `per_signature_cap` defaults to 3 if omitted.
+
+### 4.1.3 Signature grammar (produced by the evaluator’s normalizer)
+
+These are the only signatures v0.1 emits; every new action we add must define its signature mapping here.
+
+| Action family       | Signature format (dot separated)          | Notes                                                                                                                   |                                       |                                                          |
+| ------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | -------------------------------------------------------- |
+| Perp limit orders   | `perp.order.{TIF}:{reduceOnly}:{trigger}` | `TIF ∈ {ALO,GTC,IOC}` (upper‑case). `reduceOnly ∈ {true,false}`. `trigger ∈ {none,tp,sl,...}` (v0.1 emits `none` only). |                                       |                                                          |
+| Cancels             | \`perp.cancel.{last                       | oids                                                                                                                    | all}\`                                | Derived from `cancel_last`, `cancel_oids`, `cancel_all`. |
+| USDC class transfer | \`account.usdClassTransfer.{toPerp        | fromPerp}\`                                                                                                             | Direction taken from request payload. |                                                          |
+| Leverage            | `risk.setLeverage.{COIN}`                 | `COIN` is literal (e.g., `ETH`).                                                                                        |                                       |                                                          |
+
+> Effect filter: only actions with **ack.status == "ok"** and **non‑error** statuses are counted; otherwise the record is **ignored**. This implements the “no‑op filter” idea from the SuiBench method.&#x20;
+
+### 4.1.4 Reference config (drop‑in)
+
+```yaml
+version: "0.1"
+per_action_window_ms: 200
+per_signature_cap: 3
+
+domains:
+  perp:
+    weight: 1.0
+    allow:
+      - "perp.order.*"
+      - "perp.cancel.*"
+
+  account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.*"
+
+  risk:
+    weight: 1.0
+    allow:
+      - "risk.setLeverage.*"
+```
+
+> This matches the evaluator you’ve written (pattern semantics, windowing, cap). Keep this file under git; changes to it should be considered a **scoring version bump**.
+
+---
+
+## 4.2 Coverage tasks (runner‑ready plans)
+
+Coverage tasks are **JSONL** files where **each line is a complete plan** object the runner can execute. The evaluator **does not** read these; the runner does. The evaluator consumes only the run artifacts (e.g., `per_action.jsonl`) the runner produces.
+
+### 4.2.1 File paths
+
+```
+dataset/
+└── tasks/
+    ├── hl_perp_basic_01.jsonl
+    ├── hl_cancel_sweep_01.jsonl
+    ├── hl_risk_and_account_01.jsonl
+    └── README.md
+```
+
+### 4.2.2 Plan JSON schema (must match `hl-common::plan`)
+
+Top level:
+
+```json
+{
+  "steps": [ ActionStep, ... ]
+}
+```
+
+`ActionStep` is **untagged** (exactly like your `ActionStep` enum):
+
+* `{"perp_orders": { "orders": [PerpOrder, ...], "builderCode": "optional-override" }}`
+* `{"cancel_last": { "coin": "optional-coin" }}`
+* `{"cancel_oids": { "coin": "ETH", "oids": [123,456] }}`
+* `{"cancel_all": { "coin": "optional-coin" }}`
+* `{"usd_class_transfer": { "toPerp": true, "usdc": 10.5 }}`
+* `{"set_leverage": { "coin": "ETH", "leverage": 5, "cross": false }}`
+* `{"sleep_ms": { "durationMs": 250 }}`
+
+`PerpOrder`:
+
+```json
+{
+  "coin": "ETH",
+  "tif": "Gtc",             // Alo | Gtc | Ioc  (case‑insensitive; runner uppercases)
+  "side": "buy",            // buy | sell
+  "sz": 0.01,
+  "reduceOnly": false,
+  "builderCode": "optional",
+  "cloid": "optional-uuid",
+  "trigger": { "kind": "none" },
+  "px":  "mid-1.0%"         // or number (absolute); "mid±X%"
+}
+```
+
+> The runner converts `"px": "mid-1.0%"` using live mid from `InfoClient`; your implementation already handles this via `OrderPrice::MidPercent`.
+
+### 4.2.3 Starter tasks (copy‑paste)
+
+**`dataset/tasks/hl_perp_basic_01.jsonl`**
+
+```json
+{"steps":[
+  {"perp_orders":{"orders":[
+    {"coin":"ETH","tif":"Alo","side":"buy","sz":0.01,"reduceOnly":false,"px":"mid-1.0%"},
+    {"coin":"ETH","tif":"Gtc","side":"sell","sz":0.01,"reduceOnly":false,"px":"mid+1.0%"}
+  ]}},
+  {"cancel_last":{}}
+]}
+```
+
+**`dataset/tasks/hl_cancel_sweep_01.jsonl`**
+
+```json
+{"steps":[
+  {"perp_orders":{"orders":[
+    {"coin":"ETH","tif":"Gtc","side":"buy","sz":0.02,"reduceOnly":false,"px":"mid-0.5%"}
+  ]}},
+  {"sleep_ms":{"durationMs":150}},
+  {"cancel_all":{"coin":"ETH"}}
+]}
+```
+
+**`dataset/tasks/hl_risk_and_account_01.jsonl`**
+
+```json
+{"steps":[
+  {"usd_class_transfer":{"toPerp":true,"usdc":10.0}},
+  {"set_leverage":{"coin":"ETH","leverage":5,"cross":false}},
+  {"perp_orders":{"orders":[
+    {"coin":"ETH","tif":"Ioc","side":"buy","sz":0.01,"reduceOnly":true,"px":"mid"}
+  ]}}
+]}
+```
+
+> These three lines already produce multiple distinct signatures and exercise **window bonus** (200ms). To reliably get bonus for step combinations, keep consecutive actions close in time, or insert short `sleep_ms` values if you want to **avoid** coalescing.
+
+### 4.2.4 Runner commands (deterministic runs)
+
+```bash
+# 1) Pick a task line, run it, produce artifacts under runs/<ts>/
+export HL_PRIVATE_KEY=0x<your_test_key>
+cargo run -p hl-runner -- \
+  --plan dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --network testnet
+
+# 2) Evaluate using the domains config (writes eval_*.json next to per_action.jsonl)
+RUN_DIR=$(ls -dt runs/* | head -n1)
+cargo run -p hl-evaluator -- \
+  --input "$RUN_DIR/per_action.jsonl" \
+  --domains dataset/domains-hl.yaml
+cat "$RUN_DIR/eval_score.json"
+```
+
+**Acceptance (expected)**
+
+* `eval_per_action.jsonl` contains normalized `signatures` for each step.
+* `eval_score.json` exposes `{ finalScore, base, bonus, penalty, perDomain[], unique_signatures[] }`.
+
+---
+
+## 4.3 HiaN (Long‑Context “Needle”) cases
+
+HiaN cases test **accuracy under noise** (Pass/Fail) rather than breadth. The idea mirrors SuiBench’s LC track (page 10).&#x20;
+
+### 4.3.1 File layout
+
+```
+dataset/hian/case_128k/
+├── prompt.txt            # giant noisy context containing the needle + keys
+├── ground_truth.json     # exact effects we require
+└── meta.json             # reproducibility metadata
+```
+
+You can create multiple cases: `case_128k/`, `case_512k/`, `case_1m/` and variants `pos_05/`, `pos_50/`, `pos_95/` to vary **needle position**.
+
+### 4.3.2 `prompt.txt` content contract
+
+* The **needle instruction** is a single **highest‑priority** directive such as:
+  “**Send 7.5 USDC from spot to perps, then place an ALO bid mid-1% on ETH for 0.01**”
+* The **keys** (exact parameters) **must also be present** in the context, potentially far away (e.g., target asset, size, direction).
+* Everything else is **noise**: orderbook logs, unrelated docs, chats, etc.
+
+### 4.3.3 `ground_truth.json` schema (minimal v0.1)
+
+```json
+{
+  "require": [
+    {"signature":"account.usdClassTransfer.toPerp"},
+    {"signature":"perp.order.ALO:false:none"}
+  ],
+  "optional": [
+    {"signature":"perp.cancel.*"}          // if the instruction asks to cancel, put it in require
+  ]
+}
+```
+
+* The evaluator’s HiaN checker (added in step 5) will **scan `per_action.jsonl`** for ack‑OK actions and assert that every `require[*].signature` (wildcards allowed) **appears at least once**.
+* If any `require` is missing → **Fail**; otherwise **Pass**.
+* Store the final result in `runs/<ts>/eval_hian.json`:
+
+  ```json
+  { "passed": true, "missing": [] }
+  ```
+
+---
+
+## 4.4 Scoring rules (binding)
+
+> Mirrors the SuiBench scoring idea: **Base** (weighted unique signatures) + **Bonus** (composition inside a time window) − **Penalty** (spam beyond cap). See the “Coverage Scoring Details” page.&#x20;
+
+* **Base**
+  For each domain `d`:
+  `base += domain.weight * unique_signatures(d)`
+  Uniqueness is computed **after** mapping each action to its signature string.
+
+* **Bonus (composition)**
+  For each `windowKeyMs` bucket (equal window start), let `k = distinct_signatures_in_bucket`.
+  `bonus += 0.25 * max(0, k - 1)`.
+
+* **Penalty (spam)**
+  Maintain a count per signature across the entire run. If a signature occurs `> per_signature_cap`, for each extra occurrence add:
+  `penalty += 0.1`.
+
+* **Final score**
+  `FINAL_SCORE = base + bonus − penalty`.
+
+* **Effect/no‑op filter**
+  Any action with `ack.status != "ok"` or with an acknowledged **error** is ignored for both base and bonus. (Already implemented in your normalizer.)
+
+---
+
+## 4.5 Developer checklist
+
+* [x] Create `dataset/domains-hl.yaml` with the **reference config** above.
+* [x] Create `dataset/tasks/` and add **three starter JSONL** files from §4.2.3.
+* [x] Add `dataset/hian/case_128k/` and stub **prompt.txt**, **ground\_truth.json**, **meta.json** (fill with placeholders now).
+* [x] Add `dataset/tasks/README.md` that explains how to select a JSONL line with `:N`.
+* [x] Ensure CI caches the dataset directory, and publishes `runs/<ts>/eval_*` as artifacts.
+
+---
+
+## 4.6 QA / Acceptance tests
+
+1. **Smoke (coverage):**
+
+   ```bash
+   export HL_PRIVATE_KEY=0x<key>
+   cargo run -p hl-runner -- --plan dataset/tasks/hl_perp_basic_01.jsonl:1 --network testnet
+   RUN_DIR=$(ls -dt runs/* | head -n1)
+   cargo run -p hl-evaluator -- --input "$RUN_DIR/per_action.jsonl" --domains dataset/domains-hl.yaml
+   cat "$RUN_DIR/eval_per_action.jsonl" | head
+   cat "$RUN_DIR/eval_score.json"
+   ```
+
+   **Expect:**
+
+    * At least these signatures appear once:
+      `perp.order.ALO:false:none`, `perp.order.GTC:false:none`, `perp.cancel.last`.
+    * `bonus >= 0.25` (two distinct orders land in the same 200ms window if they were acked close enough; if not, insert a small `sleep_ms` to control it).
+
+2. **Spam/penalty:** Run a plan that repeats `perp.order.GTC:false:none` 5–6 times in quick succession; verify `penalty` increases by `0.1` per extra beyond the cap (3).
+
+3. **Cap/uniques:** Ensure repeating the **same** signature more than `cap` **does not** increase `unique_signatures` count, only the penalty.
+
+4. **HiaN (placeholder):**
+
+    * Put in `ground_truth.json` the two `require` signatures shown in §4.3.3.
+    * Run an `llm_hian` agent (step 5) that reads `prompt.txt` and generates a plan.
+    * Manually verify pass/fail by inspecting `eval_per_action.jsonl`.
+    * In step 5, add a small HiaN checker to produce `eval_hian.json`.
+
+---
+
+## 4.7 Maintenance & versioning
+
+* Any change to signature grammar or pattern semantics requires a **domains version bump** (`version: "0.2"`), and a corresponding tag in the repo.
+* Keep **dataset/** deterministic and under git. When prompts are too large, store a SHA256 in `meta.json` and host the blob externally — still track the hash for reproducibility (as recommended in the SuiBench doc’s workflow pages 11–13).&#x20;
+
+---
+
+## 4.8 Appendix — Worked example of normalization
+
+**Input line (`per_action.jsonl`)**
+
+```json
+{
+  "stepIdx": 0,
+  "action": "perp_orders",
+  "submitTsMs": 1711111111111,
+  "windowKeyMs": 1711111111000,
+  "request": {"perp_orders":{"orders":[
+    {"coin":"ETH","tif":"Gtc","side":"buy","sz":0.01,"reduceOnly":false,"px":"mid-1%"}
+  ]}},
+  "ack": {"status":"ok","data":{"statuses":[{"kind":"resting","oid":12345}]}},
+  "observed": [{"channel":"orderUpdates","oid":12345,"status":"resting"}]
+}
+```
+
+**Evaluator emits**
+`signatures = ["perp.order.GTC:false:none"]`, `ignored=false`.
+
+That signature matches `domains.perp.allow = ["perp.order.*"]` → contributes `+1.0` to `base`.
+
+---
+
+**That’s it.** With this document, another developer can: create the YAML, seed three tasks, run the runner, and compute consistent scores with the evaluator you’ve built, while keeping room for the HiaN track in step 5 (pass/fail).
@@ -0,0 +1,683 @@
+Below is a **developer‑facing** checklist you can hand to a teammate and have them implement 1:1. It assumes the code you already shared for `hl-common` and `hl-runner` is in place. Where your current code already matches, I mark ✅; where we should add/adjust, I mark 🟨 (add) or 🟥 (change).
+
+---
+
+# DETAILED\_PLAN\_5.md — Implementation checklist (by file)
+
+> Goal: finish the end‑to‑end **coverage** evaluation for HyperLiquidBench (runner → artifacts → evaluator), with deterministic scoring and scripts.
+
+```
+workspace/
+├─ crates/
+│  ├─ hl-common/                  # shared types, signatures, time, artifacts
+│  │  └─ src/{plan.rs, artifacts.rs, sig.rs, time.rs, lib.rs}
+│  ├─ hl-runner/                  # executes Plan against HL SDK, emits artifacts
+│  │  └─ src/{main.rs, ...}
+│  └─ hl-evaluator/               # parses artifacts, builds signatures, scores
+│     └─ src/{cli.rs, model.rs, domains.rs, parse.rs, scoring.rs, util.rs, main.rs}
+├─ dataset/
+│  └─ domains-hl.yaml            # domain weights & prefix rules (coverage)
+├─ scripts/
+│  ├─ run_cov.sh
+│  └─ run_cov_matrix.sh
+└─ runs/                          # per-run artifacts (created at runtime)
+```
+
+---
+
+## 5.1 `crates/hl-common` (shared)
+
+### 5.1.1 `plan.rs` (✅ done, keep)
+
+* Your current `Plan`, `ActionStep`, `PerpOrder`, `PerpTif`, `OrderSide`, `OrderTrigger`, `OrderPrice` are correct for the MVP.
+* **Acceptance**: plan JSON must deserialize from:
+
+  ```json
+  {
+    "steps": [
+      { "perpOrders": { "orders": [
+        { "coin": "BTC", "tif": "ALO", "side": "buy", "sz": 0.001, "reduceOnly": false, "px": "mid-0.2%" }
+      ]}},
+      { "cancelLast": { "coin": "BTC" } },
+      { "usdClassTransfer": { "toPerp": true, "usdc": 2.0 } },
+      { "setLeverage": { "coin":"BTC", "leverage":10, "cross": true } }
+    ]
+  }
+  ```
+
+### 5.1.2 `artifacts.rs` (✅ with 2 small tweaks)
+
+* You already produce:
+
+    * `per_action.jsonl` one line per submitted action with `{request, ack, observed, step_idx, submit_ts_ms, window_key_ms, notes?}`.
+    * `ws_stream.jsonl` raw WS messages.
+    * `orders_routed.csv` (header ok).
+    * `run_meta.json` via `write_meta`.
+
+* 🟨 **Add** `windowMs` to `run_meta.json`.
+
+  ```rust
+  // when writing meta in hl-runner (after artifacts created):
+  let meta = json!({
+    // ...
+    "windowMs": artifacts.lock().await.window_ms(),   // <— include
+    // ...
+  });
+  ```
+
+* 🟨 **Guarantee flushing** on drop for `ws_stream` (optional):
+
+  ```rust
+  impl Drop for RunArtifacts {
+      fn drop(&mut self) {
+          let _ = self.ws_stream.flush();
+          let _ = self.per_action.flush();
+          let _ = self.routed_csv.flush();
+      }
+  }
+  ```
+
+### 5.1.3 `time.rs` (✅)
+
+* `timestamp_ms()` + `window_start_ms()` already match the bonus windowing spec (default 200ms).
+
+### 5.1.4 `sig.rs` (🟨 add)
+
+Create a new module to **normalize effects into signatures** used by the evaluator.
+
+```rust
+// crates/hl-common/src/sig.rs
+use serde::Serialize;
+
+/// The normalized "coverage unit" string we count for uniqueness.
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize)]
+pub struct Signature(pub String);
+
+impl Signature {
+    pub fn perp_order(tif: &str, reduce_only: bool, trigger: &str) -> Self {
+        Self(format!("perp.order.{tif}:{reduce_only}:{trigger}"))
+    }
+    pub fn perp_cancel(scope: &str) -> Self {
+        Self(format!("perp.cancel.{scope}"))
+    }
+    pub fn account_usd_class_transfer(direction: &str) -> Self {
+        // direction ∈ {"toPerp","fromPerp"}
+        Self(format!("account.usdClassTransfer.{direction}"))
+    }
+    pub fn risk_set_leverage(coin: &str) -> Self {
+        Self(format!("risk.setLeverage.{}", coin.to_ascii_uppercase()))
+    }
+}
+
+/// Utility: normalize TIF casing from request/HL SDK ("ALO"/"GTC"/"IOC").
+pub fn normalize_tif(label: &str) -> &'static str {
+    match label.to_ascii_uppercase().as_str() {
+        "ALO" => "ALO",
+        "GTC" => "GTC",
+        "IOC" => "IOC",
+        _ => "GTC", // safe default
+    }
+}
+
+/// Utility: trigger normalization for MVP (only "none" supported by runner).
+pub fn normalize_trigger(raw: &serde_json::Value) -> &'static str {
+    // if later we log trigger in request: { "trigger": { "kind": "Tp", ... } }
+    // For now:
+    "none"
+}
+```
+
+* **Acceptance**: these exact string forms are what we count and match in domains.
+* Export in `hl-common/src/lib.rs`:
+
+  ```rust
+  pub mod sig;
+  pub use sig::{Signature, normalize_tif, normalize_trigger};
+  ```
+
+---
+
+## 5.2 `crates/hl-runner` (execute plan via SDK)
+
+Your `main.rs` already:
+
+* builds HL clients, spawns WS subscriber, executes each `ActionStep`, correlates effects, writes artifacts. ✅
+
+**Keep the following behaviors (already present):**
+
+* Price resolution for `"mid±X%"` with HTTP `all_mids()` cache.
+* Perp post uses `bulk_order` (and `bulk_order_with_builder` if `builder_code` present).
+* For each order you: compute resolved limit, build `ClientOrderRequest`, post, collect **ack OIDs**, and then **wait** for a WS confirmation (order update/fill) up to `effect_timeout_ms` (default 2000ms).
+* Cancels (`cancel_last`, `cancel_oids`, `cancel_all`) match effect correlation rules (wait per OID).
+* Class transfer waits for `UserNonFundingLedgerUpdates` with `AccountClassTransfer` and `to_perp` matching.
+* `orders_routed.csv` row per logical order with builder code.
+
+**Runner acceptance criteria**
+
+* For each submitted step, one JSON line in `per_action.jsonl` with:
+
+  ```json
+  {
+    "stepIdx": 0,
+    "action": "perp_orders",                      // or cancel_last / cancel_oids / cancel_all / usd_class_transfer / set_leverage
+    "submitTsMs": 1737459211000,
+    "windowKeyMs": 1737459211000,
+    "request": { ... },                           // human-readable request echo
+    "ack": { "status": "ok", "data": { ... } },   // or { "status": "err", ... }
+    "observed": [ { "channel": "orderUpdates", ... }, { "channel": "userFills", ... } ],
+    "notes": "missing cancel confirmations for [12345]" // optional
+  }
+  ```
+* `ws_stream.jsonl` contains every raw WS message we observed, including `isSnapshot` frames.
+
+**Minor runner tweaks to help the evaluator**
+
+* 🟨 In `request_value` for each `perp_orders` record, include **canonical TIF and reduceOnly** at per‑order level (you already do), and for trigger put:
+
+  ```json
+  "trigger": { "kind": "none" } // until triggers are supported
+  ```
+* ✅ You already add `"resolvedPx"`; keep it.
+* 🟨 In `set_leverage` request echo, ensure `"coin"` is present (you already do), we will uppercase on the evaluator side.
+
+---
+
+## 5.3 `crates/hl-evaluator` (new)
+
+> This crate reads a **run folder** and produces `eval_score.json` (coverage). It uses `domains-hl.yaml` to map signatures → domains and weights.
+
+### 5.3.1 `Cargo.toml`
+
+```toml
+[package]
+name = "hl-evaluator"
+version = "0.1.0"
+edition = "2021"
+
+[dependencies]
+anyhow = "1"
+indexmap = "2"
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+serde_yaml = "0.9"
+regex = "1"
+once_cell = "1"
+chrono = { version = "0.4", default-features = false, features=["clock","std"] }
+hl-common = { path = "../hl-common" }
+```
+
+### 5.3.2 `src/model.rs` (🟨)
+
+```rust
+use indexmap::IndexMap;
+use serde::Serialize;
+use hl_common::Signature;
+
+#[derive(Debug, Serialize)]
+pub struct ScoreMetadata {
+    pub bench_version: String,           // "hlbench-v0.1"
+    pub run_dir: String,                 // ./runs/<ts>
+    pub domains_hash: String,            // sha256(domains-hl.yaml)
+    pub window_ms: i64,
+    pub per_sig_cap: usize,
+}
+
+#[derive(Debug, Serialize)]
+pub struct ScoreReport {
+    pub final_score: f64,
+    pub by_domain: IndexMap<String, f64>,
+    pub bonus: f64,
+    pub penalty: f64,
+    pub unique_sigs: IndexMap<String, Vec<Signature>>,  // by domain
+    pub metadata: ScoreMetadata,
+}
+
+#[derive(Debug, Clone)]
+pub struct EffectSample {
+    pub window_key_ms: i64,
+    pub signatures: Vec<Signature>,      // 1..N signatures derived from one action record
+    pub is_noop: bool,                   // ack=err and no observed => no-op
+}
+```
+
+### 5.3.3 `src/domains.rs` (🟨)
+
+> Simple **prefix matcher** to map signature strings to domains.
+
+`dataset/domains-hl.yaml`
+
+```yaml
+domains:
+  perp:
+    weight: 1.00
+    allow:
+      - "perp.order."
+      - "perp.cancel."
+  account:
+    weight: 0.50
+    allow:
+      - "account.usdClassTransfer."
+  risk:
+    weight: 0.75
+    allow:
+      - "risk.setLeverage."
+per_sig_cap: 3
+```
+
+Implementation:
+
+```rust
+use anyhow::{Context, Result};
+use indexmap::IndexMap;
+use serde::Deserialize;
+
+#[derive(Debug, Deserialize)]
+pub struct DomainsConfig {
+    pub domains: IndexMap<String, DomainRule>,
+    #[serde(default)]
+    pub per_sig_cap: Option<usize>,
+}
+
+#[derive(Debug, Deserialize)]
+pub struct DomainRule {
+    pub weight: f64,
+    pub allow: Vec<String>, // treated as prefixes
+}
+
+impl DomainsConfig {
+    pub fn load(path: &std::path::Path) -> Result<Self> {
+        let bytes = std::fs::read(path)
+            .with_context(|| format!("failed to read {:?}", path))?;
+        let cfg: Self = serde_yaml::from_slice(&bytes)
+            .with_context(|| "failed to parse domains-hl.yaml")?;
+        Ok(cfg)
+    }
+
+    pub fn domain_for(&self, sig: &str) -> Option<(&str, f64)> {
+        for (name, rule) in &self.domains {
+            if rule.allow.iter().any(|p| sig.starts_with(p)) {
+                return Some((name.as_str(), rule.weight));
+            }
+        }
+        None
+    }
+}
+```
+
+### 5.3.4 `src/parse.rs` (🟨)
+
+> Convert `per_action.jsonl` lines to `EffectSample`.
+
+Rules:
+
+* **No‑op filter**: if `ack.status=="err"` **and** `observed` is absent/empty → `is_noop=true`.
+* Otherwise, **derive signatures** based on `action` and `request` echo (canonical TIF, reduceOnly, trigger, coin, direction).
+* For `perp_orders`, generate **one signature per order** in the step:
+  `perp.order.{TIF}:{reduceOnly}:{trigger}`
+* For `cancel_last`, `cancel_oids`, `cancel_all` → `perp.cancel.{scope}` with scope ∈ `last|oids|all`.
+* For `usd_class_transfer` → `account.usdClassTransfer.{toPerp|fromPerp}`.
+* For `set_leverage` → `risk.setLeverage.{COIN}` (uppercase).
+
+Implementation sketch:
+
+```rust
+use anyhow::{Context, Result};
+use hl_common::{Signature, normalize_tif, normalize_trigger};
+use serde_json::Value;
+
+pub fn parse_per_action_line(line: &str) -> Result<EffectSample> {
+    let v: Value = serde_json::from_str(line).context("bad JSON in per_action.jsonl")?;
+    let action = v["action"].as_str().unwrap_or_default();
+    let window_key = v["windowKeyMs"].as_i64().unwrap_or(0);
+    let ack_ok = v["ack"]["status"].as_str() == Some("ok");
+    let observed_nonempty = match &v["observed"] {
+        Value::Null => false,
+        Value::Array(a) if a.is_empty() => false,
+        _ => true,
+    };
+
+    let is_noop = !ack_ok && !observed_nonempty;
+
+    let mut sigs = Vec::<Signature>::new();
+    match action {
+        "perp_orders" => {
+            if let Some(orders) = v["request"]["perp_orders"]["orders"].as_array() {
+                for o in orders {
+                    let tif = o["tif"].as_str().map(normalize_tif).unwrap_or("GTC");
+                    let reduce_only = o["reduceOnly"].as_bool().unwrap_or(false);
+                    let trig = normalize_trigger(&o["trigger"]);
+                    sigs.push(Signature::perp_order(tif, reduce_only, trig));
+                }
+            }
+        }
+        "cancel_last" => sigs.push(Signature::perp_cancel("last")),
+        "cancel_oids" => sigs.push(Signature::perp_cancel("oids")),
+        "cancel_all"  => sigs.push(Signature::perp_cancel("all")),
+        "usd_class_transfer" => {
+            let dir = if v["request"]["usd_class_transfer"]["toPerp"].as_bool().unwrap_or(false) {
+                "toPerp"
+            } else { "fromPerp" };
+            sigs.push(Signature::account_usd_class_transfer(dir));
+        }
+        "set_leverage" => {
+            let coin = v["request"]["set_leverage"]["coin"].as_str().unwrap_or("UNKNOWN");
+            sigs.push(Signature::risk_set_leverage(coin));
+        }
+        _ => {}
+    }
+
+    Ok(EffectSample {
+        window_key_ms: window_key,
+        signatures: sigs,
+        is_noop,
+    })
+}
+```
+
+### 5.3.5 `src/scoring.rs` (🟨)
+
+**Parameters**
+
+* `window_ms` = from `run_meta.json` (`windowMs`) or default **200** if missing.
+* `per_sig_cap` = from `domains-hl.yaml` (default **3**).
+* `cap_penalty` = 0.0 (MVP; wire argument later if needed).
+
+**Algorithm**
+
+1. **Load** `per_action.jsonl` → `Vec<EffectSample>`.
+2. **Base score**: deduplicate signatures **by domain** globally:
+
+    * let `uniq[domain]` be a `HashSet<Signature>`.
+    * For every `EffectSample` that is **not** `is_noop`, add its signatures to the sets.
+    * `base = Σ weight(domain) × |uniq[domain]|`.
+3. **Bonus** (windowed composition):
+
+    * Group samples by `window_key_ms`.
+    * For each window, build a `HashSet<String>` of **signature strings** across all samples in that window.
+    * `bonus += 0.25 × max(0, distinct_in_window − 1)`.
+4. **Per‑signature cap**:
+
+    * Count frequency `freq[sig]` across **non‑noop** samples.
+    * If `freq[sig] > cap`, ignore additional repeats for any **future** bonus (already handled implicitly because bonus is per window unique) and (optionally) apply `cap_penalty` per extra. MVP: `cap_penalty = 0.0`.
+5. **Final**: `final_score = base + bonus − penalty`.
+
+**Output**
+
+* `eval_score.json` with:
+
+  ```json
+  {
+    "final_score": 3.75,
+    "by_domain": { "perp": 3.0, "account": 0.25, "risk": 0.5 },
+    "bonus": 0.25,
+    "penalty": 0.0,
+    "unique_sigs": {
+      "perp": ["perp.order.ALO:false:none","perp.cancel.last"],
+      "account": ["account.usdClassTransfer.toPerp"],
+      "risk": ["risk.setLeverage.BTC"]
+    },
+    "metadata": { "bench_version":"hlbench-v0.1", "run_dir":"runs/...", "domains_hash":"...", "window_ms":200, "per_sig_cap":3 }
+  }
+  ```
+
+Implementation sketch:
+
+```rust
+use anyhow::Result;
+use std::collections::{HashMap, HashSet};
+use indexmap::IndexMap;
+use crate::{model::*, domains::DomainsConfig};
+use serde_json::Value;
+
+pub fn score_run(run_dir: &std::path::Path, domains: &DomainsConfig) -> Result<ScoreReport> {
+    let per_action_path = run_dir.join("per_action.jsonl");
+    let meta_path = run_dir.join("run_meta.json");
+
+    let meta_v: Value = serde_json::from_slice(&std::fs::read(&meta_path)?)?;
+    let window_ms = meta_v.get("windowMs").and_then(|v| v.as_i64()).unwrap_or(200);
+    let per_sig_cap = domains.per_sig_cap.unwrap_or(3);
+
+    // Parse samples
+    let mut samples = Vec::<EffectSample>::new();
+    for line in std::io::BufRead::lines(std::io::BufReader::new(std::fs::File::open(per_action_path)?)) {
+        let line = line?;
+        let s = crate::parse::parse_per_action_line(&line)?;
+        samples.push(s);
+    }
+
+    // Base and uniques by domain
+    let mut uniq: IndexMap<String, HashSet<String>> = IndexMap::new();
+    let mut freq: HashMap<String, usize> = HashMap::new();
+
+    for s in samples.iter().filter(|s| !s.is_noop) {
+        for sig in &s.signatures {
+            let sig_str = &sig.0;
+            *freq.entry(sig_str.clone()).or_default() += 1;
+            if let Some((dom, _w)) = domains.domain_for(sig_str) {
+                uniq.entry(dom.to_string()).or_default().insert(sig_str.clone());
+            }
+        }
+    }
+
+    // Base
+    let mut by_domain = IndexMap::<String, f64>::new();
+    let mut base = 0.0;
+    for (dom, rule) in &domains.domains {
+        let count = uniq.get(dom).map(|s| s.len()).unwrap_or(0) as f64;
+        let v = rule.weight * count;
+        by_domain.insert(dom.clone(), v);
+        base += v;
+    }
+
+    // Bonus (windowed unique signatures)
+    use std::collections::BTreeMap;
+    let mut windows: BTreeMap<i64, HashSet<String>> = BTreeMap::new();
+    for s in samples.iter().filter(|s| !s.is_noop) {
+        let set = windows.entry(s.window_key_ms).or_default();
+        for sig in &s.signatures {
+            set.insert(sig.0.clone());
+        }
+    }
+    let mut bonus = 0.0;
+    for (_k, set) in windows {
+        let n = set.len() as i64;
+        if n > 1 {
+            bonus += 0.25 * (n as f64 - 1.0);
+        }
+    }
+
+    // Penalty (cap only if we decide to apply)
+    let mut penalty = 0.0;
+    let cap = per_sig_cap;
+    // MVP: penalty remains 0.0; if needed later:
+    // for (sig, c) in freq {
+    //     if c > cap { penalty += 0.1 * ((c - cap) as f64); }
+    // }
+
+    // unique_sigs by domain (sorted)
+    let mut unique_sigs = IndexMap::<String, Vec<hl_common::Signature>>::new();
+    for (dom, set) in uniq.iter() {
+        let mut v: Vec<_> = set.iter().cloned().map(hl_common::Signature).collect();
+        v.sort_by(|a,b| a.0.cmp(&b.0));
+        unique_sigs.insert(dom.clone(), v);
+    }
+
+    let report = ScoreReport {
+        final_score: base + bonus - penalty,
+        by_domain,
+        bonus,
+        penalty,
+        unique_sigs,
+        metadata: ScoreMetadata {
+            bench_version: "hlbench-v0.1".to_string(),
+            run_dir: run_dir.display().to_string(),
+            domains_hash: crate::util::sha256_file(&std::path::PathBuf::from("dataset/domains-hl.yaml"))?,
+            window_ms,
+            per_sig_cap: cap,
+        },
+    };
+
+    Ok(report)
+}
+```
+
+### 5.3.6 `src/cli.rs` and `src/main.rs` (🟨)
+
+```rust
+use anyhow::Result;
+use clap::Parser;
+
+#[derive(Parser, Debug)]
+#[command(about = "HyperLiquidBench evaluator (coverage)")]
+struct Cli {
+    /// Path to run directory (contains per_action.jsonl, ws_stream.jsonl, run_meta.json)
+    #[arg(long)]
+    input: std::path::PathBuf,
+
+    /// Path to domains-hl.yaml
+    #[arg(long, default_value = "dataset/domains-hl.yaml")]
+    domains: std::path::PathBuf,
+}
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    let cli = Cli::parse();
+    let cfg = crate::domains::DomainsConfig::load(&cli.domains)?;
+    let report = crate::scoring::score_run(&cli.input, &cfg)?;
+    let out = cli.input.join("eval_score.json");
+    std::fs::write(&out, serde_json::to_string_pretty(&report)?)?;
+    eprintln!("FINAL_SCORE: {}", report.final_score);
+    Ok(())
+}
+```
+
+### 5.3.7 `src/util.rs` (🟨 tiny helper)
+
+```rust
+use anyhow::Result;
+use sha2::{Digest, Sha256};
+
+pub fn sha256_file(path: &std::path::Path) -> Result<String> {
+    let bytes = std::fs::read(path)?;
+    let mut h = Sha256::new();
+    h.update(bytes);
+    Ok(format!("{:x}", h.finalize()))
+}
+```
+
+**Unit tests** (add a couple in `hl-evaluator/tests/`):
+
+* `signatures_from_perp_orders.rs`: feed one `per_action.jsonl` line with two orders ALO/GTC; expect 2 signatures, no‑op=false.
+* `bonus_windowing.rs`: two lines with same `windowKeyMs` and distinct signatures → bonus 0.25.
+* `noop_filter.rs`: ack.status="err" and observed missing → is\_noop=true (no base/bonus).
+
+---
+
+## 5.4 Scripts (🟨)
+
+### 5.4.1 `scripts/run_cov.sh`
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+OUT="runs/$(date +%Y%m%d-%H%M%S)"
+PLAN="${1:-dataset/tasks/hl_perp_basic_01.jsonl:1}"
+NETWORK="${NETWORK:-testnet}"
+
+echo "[*] Running hl-runner → $OUT"
+cargo run -p hl-runner -- \
+  --plan "$PLAN" \
+  --network "$NETWORK" \
+  --out "$OUT"
+
+echo "[*] Evaluating coverage"
+cargo run -p hl-evaluator -- \
+  --input "$OUT" \
+  --domains dataset/domains-hl.yaml
+
+jq . "$OUT/eval_score.json"
+```
+
+### 5.4.2 `scripts/run_cov_matrix.sh` (optional matrix across plans)
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+PLANS=(
+  "dataset/tasks/hl_perp_basic_01.jsonl:1"
+  "dataset/tasks/hl_perp_cancel_01.jsonl:2"
+)
+
+for P in "${PLANS[@]}"; do
+  ./scripts/run_cov.sh "$P"
+done
+```
+
+---
+
+## 5.5 Acceptance checklist (copy/paste)
+
+* [ ] Runner writes **one** action line per step in `per_action.jsonl` with `windowKeyMs`.
+* [ ] Runner writes **all** WS frames to `ws_stream.jsonl` (including `isSnapshot` frames).
+* [ ] Runner writes `orders_routed.csv` with header and one row per posted order.
+* [ ] Runner writes `run_meta.json` including `"windowMs"`.
+* [ ] Evaluator loads `domains-hl.yaml`, parses `per_action.jsonl`, and produces `eval_score.json`.
+* [ ] **Base** counts are consistent: changing TIF or reduceOnly changes `perp.order.*` signature.
+* [ ] **Bonus** increments by `+0.25` when ≥2 **distinct** signatures fall into the same window bucket.
+* [ ] **No‑op** lines (ack err & no observed) do **not** contribute to base or bonus.
+* [ ] Running `scripts/run_cov.sh` prints `FINAL_SCORE:` and pretty JSON.
+
+---
+
+## 5.6 “Gotchas” & invariants (read before coding)
+
+* A **perp\_orders** step with N orders must produce **N signatures** in that sample. We do **not** fan out into multiple lines in `per_action.jsonl`; we extract inside the evaluator.
+* Do **not** include `coin` in the **order** signature; uniqueness is about **behavioral variety**, not market variety. We do include `coin` in **setLeverage** signatures.
+* **Windowing**: use the **precomputed** `windowKeyMs` from the runner (do not recompute).
+* If **ack is ok** but **observed** is empty, still treat as **effect present** (not a no‑op). The no‑op rule only triggers when both fail (ack err & observed empty). This matches exchange behavior (e.g., accepted but low‑traffic WS).
+* Domains matching is **prefix‑based**. Keep signature strings stable; future changes require updating `domains-hl.yaml`.
+
+---
+
+## 5.7 Example end‑to‑end (sanity)
+
+1. Plan (JSONL line):
+
+```json
+{"steps":[
+  {"perpOrders":{"orders":[
+    {"coin":"BTC","tif":"ALO","side":"buy","sz":0.001,"reduceOnly":false,"px":"mid-0.3%"}
+  ]}},
+  {"usdClassTransfer":{"toPerp":true,"usdc":1.5}},
+  {"setLeverage":{"coin":"BTC","leverage":10,"cross":true}}
+]}
+```
+
+2. Run:
+
+```bash
+HL_PRIVATE_KEY=0x... ./scripts/run_cov.sh dataset/tasks/hl_perp_basic_01.jsonl:1
+```
+
+3. Expect `eval_score.json` roughly like:
+
+```json
+{
+  "final_score": 2.0,                    // e.g., perp.order.ALO:false:none (1) + account.usdClassTransfer.toPerp (1)
+  "by_domain": { "perp": 1.0, "account": 0.5, "risk": 0.5 },
+  "bonus": 0.25,                         // if they landed in same 200ms window and were distinct
+  "penalty": 0.0,
+  "unique_sigs": {
+    "perp": ["perp.order.ALO:false:none"],
+    "account": ["account.usdClassTransfer.toPerp"],
+    "risk": ["risk.setLeverage.BTC"]
+  },
+  "metadata": { "bench_version":"hlbench-v0.1", "window_ms":200, "per_sig_cap":3, ... }
+}
+```
+
+---
+
+With these files and exact behaviors implemented, your teammate can complete the evaluator quickly and your bench will produce **deterministic, explainable** coverage scores for HyperLiquid‑native actions with **minimal coding** beyond what you already have.
@@ -0,0 +1,607 @@
+Below is a drop‑in **DETAILED\_PLAN\_LLM.md** for HyperLiquidBench.
+It specifies *what to build*, *how to wire it*, *exact prompts*, *payload schemas*, and *robust parsing* so another developer can implement it 1:1 in Rust.
+
+---
+
+# DETAILED\_PLAN\_LLM.md — LLM Plan Generation & Execution for **HyperLiquidBench**
+
+> Goal: let the runner **ask an LLM** (via OpenRouter) for a short, machine‑readable trading plan and **execute** it against Hyperliquid (testnet/local), logging artifacts that the evaluator can score.
+> Non‑goals: strategy alpha, real‑money trading, unsupported order types (triggers can be stubbed for now).
+
+---
+
+## 0) High‑level architecture
+
+```
+hl-runner
+ ├─ detect plan spec:
+ │    ├─ file.json / file.jsonl:N  → load_plan_from_spec (already implemented)
+ │    └─ "llm:coverage"            → call LLM (coverage prompt) → Plan JSON
+ │       "llm:hian:<path>"         → load long context from <path> → call LLM (HiaN prompt) → Plan JSON
+ │
+ ├─ persist artifacts:
+ │    runs/<ts>/{plan.json, plan_raw.txt, per_action.jsonl, ws_stream.jsonl, orders_routed.csv, run_meta.json}
+ │
+ └─ execute_plan(Plan) → (already implemented: posts orders, waits for WS acks, logs)
+```
+
+---
+
+## 1) CLI & config surface
+
+**New CLI flags (hl-runner):**
+
+* `--plan llm:coverage`
+  Generate a short coverage plan via LLM.
+* `--plan llm:hian:<FILE>`
+  Generate a plan from a long‑context **HiaN** file.
+* `--llm-model <id>` (env: `LLM_MODEL`)
+  e.g., `openai/gpt-5`, `google/gemini-2.5-pro` (use any OpenRouter model your key can access).
+* `--llm-max-steps <n>` (default `5`)
+  Upper bound we instruct the model to respect.
+* `--llm-allowed-coins <CSV>` (optional)
+  Comma‑separated allowlist, e.g., `BTC,ETH,SOL`. If omitted, pass the top‑N from InfoClient `all_mids()`.
+* `--llm-builder-code <code>` (optional)
+  Default builder code to recommend in the prompt; step‑level `builderCode` still overrides.
+* `--llm-temperature <f64>` (default `0.2`)
+* `--llm-top-p <f64>` (default `1.0`)
+* `--llm-max-output-tokens <u32>` (default `800`)
+
+**Environment variables:**
+
+* `OPENROUTER_API_KEY` (required)
+* `LLM_MODEL` (if not provided via CLI)
+* `HL_LLM_CACHE_DIR` (optional) — directory to cache prompt → response (for reproducibility)
+* `HL_LLM_DRYRUN=1` (optional) — generate plan but **don’t** send to exchange; still write artifacts.
+
+---
+
+## 2) Plan JSON schema (must match `hl-common`)
+
+We reuse the existing types you implemented:
+
+```jsonc
+{
+  "steps": [
+    { "perp_orders": {
+        "orders": [
+          {
+            "coin": "ETH",
+            "tif": "GTC" | "ALO" | "IOC",
+            "side": "buy" | "sell",
+            "sz": 0.01,
+            "reduceOnly": false,
+            "builderCode": "myapp",       // optional
+            "cloid": "UUID-v4",           // optional
+            "trigger": { "kind": "none" },// currently only "none" is supported
+            "px":  3500.5                 // or string: "mid+0.25%" / "mid-0.5%"
+          }
+        ],
+        "builderCode": "default-code"     // optional; used when order-level is absent
+    }},
+
+    { "cancel_last": { "coin": "ETH" } },          // coin optional (last overall if omitted)
+    { "cancel_oids": { "coin": "BTC", "oids": [1] }},
+    { "cancel_all":  { "coin": "ETH" } },          // coin optional (all coins if omitted)
+    { "usd_class_transfer": { "toPerp": true, "usdc": 12.5 } },
+    { "set_leverage": { "coin": "ETH", "leverage": 5, "cross": false } },
+    { "sleep_ms": { "durationMs": 200 } }
+  ]
+}
+```
+
+> This schema is **exactly** what your `hl-common` parser understands today.
+
+---
+
+## 3) Implementation plan (code‑level)
+
+### 3.1 Add an LLM module to the runner
+
+**Files to add:**
+
+```
+crates/hl-runner/src/llm/mod.rs
+crates/hl-runner/src/llm/openrouter.rs
+crates/hl-runner/src/llm/plan_decode.rs
+crates/hl-runner/src/llm/prompts.rs
+```
+
+#### `openrouter.rs` — HTTP client
+
+```rust
+use anyhow::{anyhow, Context, Result};
+use reqwest::Client;
+use serde_json::{json, Value};
+
+pub struct OpenRouter {
+    client: Client,
+    endpoint: String,
+    api_key: String,
+    model: String,
+    temperature: f32,
+    top_p: f32,
+    max_tokens: u32,
+    title: String,
+}
+
+impl OpenRouter {
+    pub fn new(
+        api_key: String,
+        model: String,
+        temperature: f32,
+        top_p: f32,
+        max_tokens: u32,
+        title: String,
+    ) -> Result<Self> {
+        Ok(Self {
+            client: Client::builder().build()?,
+            endpoint: "https://openrouter.ai/api/v1/chat/completions".to_string(),
+            api_key,
+            model,
+            temperature,
+            top_p,
+            max_tokens,
+            title,
+        })
+    }
+
+    pub async fn chat_json(&self, system: &str, user: &str) -> Result<String> {
+        let body = json!({
+          "model": self.model,
+          "temperature": self.temperature,
+          "top_p": self.top_p,
+          "max_tokens": self.max_tokens,
+          "response_format": { "type": "json_object" },
+          "messages": [
+            { "role": "system", "content": system },
+            { "role": "user", "content": user }
+          ]
+        });
+
+        let resp = self.client
+          .post(&self.endpoint)
+          .header("Authorization", format!("Bearer {}", self.api_key))
+          .header("X-Title", &self.title)      // OpenRouter requires Referer or Title
+          .json(&body)
+          .send()
+          .await
+          .context("openrouter: HTTP error")?;
+
+        let status = resp.status();
+        let text = resp.text().await?;
+        if !status.is_success() {
+            return Err(anyhow!("openrouter {}: {}", status, text));
+        }
+
+        Ok(text) // raw JSON string; let decoder inspect it
+    }
+}
+```
+
+#### `plan_decode.rs` — tolerant response → `Plan`
+
+Robustly handle the typical failure modes: code fences, arrays of steps, extra prose.
+
+````rust
+use anyhow::{anyhow, Context, Result};
+use hl_common::plan::{Plan, ActionStep};
+use serde_json::Value;
+
+pub fn strip_fences(s: &str) -> &str {
+    let s = s.trim();
+    if s.starts_with("```") {
+        let s = s.trim_start_matches("```json")
+                 .trim_start_matches("```JSON")
+                 .trim_start_matches("```");
+        return s.trim_end_matches("```").trim();
+    }
+    s
+}
+
+pub fn decode_llm_body(raw_body: &str) -> Result<(Plan, String)> {
+    // The OpenRouter body itself is JSON: { choices: [ { message: { content: "..." } } ] }
+    let v: Value = serde_json::from_str(raw_body)
+        .context("openrouter: body is not JSON")?;
+    let content = v["choices"][0]["message"]["content"]
+        .as_str()
+        .ok_or_else(|| anyhow!("openrouter: missing content"))?;
+    let raw_content = content.to_string();
+
+    // Try strict first
+    decode_plan_from_content(content).or_else(|_| {
+        // Try after stripping fences
+        decode_plan_from_content(strip_fences(content))
+    }).or_else(|_| {
+        // If it's an array of steps, wrap it
+        let s = strip_fences(content);
+        if let Ok(steps) = serde_json::from_str::<Vec<ActionStep>>(s) {
+            Ok( (Plan { steps }, raw_content) )
+        } else {
+            Err(anyhow!("failed to parse LLM plan"))
+        }
+    })
+}
+
+fn decode_plan_from_content(s: &str) -> Result<(Plan, String)> {
+    let plan: Plan = serde_json::from_str(s)
+        .context("content not a Plan object")?;
+    Ok((plan, s.to_string()))
+}
+````
+
+#### `prompts.rs` — canonical prompts
+
+**Coverage prompt** (short planning):
+
+```rust
+pub fn coverage_system(max_steps: usize) -> String {
+    format!(r#"
+You are HyperLiquidBench Planner. Produce a STRICT JSON object (no prose) that
+describes a short sequence of trading actions for Hyperliquid testnet/local.
+
+HARD RULES:
+- Use at most {max_steps} steps.
+- Use ONLY these step shapes (camelCase keys):
+  1) {{ "perp_orders": {{ "orders": [ {{ "coin": <SYMBOL>, "side": "buy"|"sell",
+        "sz": <f64>, "tif": "GTC"|"ALO"|"IOC", "reduceOnly": <bool>,
+        "px": <number| "mid+X%" | "mid-X%">,
+        "builderCode": <string, optional>,
+        "cloid": <uuid, optional>,
+        "trigger": {{ "kind": "none" }} }} ], "builderCode": <string, optional> }} }}
+  2) {{ "cancel_last": {{ "coin": <SYMBOL, optional> }} }}
+  3) {{ "cancel_oids": {{ "coin": <SYMBOL>, "oids": [<u64>...] }} }}
+  4) {{ "cancel_all":  {{ "coin": <SYMBOL, optional> }} }}
+  5) {{ "usd_class_transfer": {{ "toPerp": <bool>, "usdc": <f64> }} }}
+  6) {{ "set_leverage": {{ "coin": <SYMBOL>, "leverage": <u32>, "cross": <bool> }} }}
+  7) {{ "sleep_ms": {{ "durationMs": <u64> }} }}
+
+CONSTRAINTS:
+- Only use coins from `allowedCoins` the user provides.
+- `trigger.kind` must be "none" (trigger orders unsupported).
+- Prefer maker behavior (GTC/ALO) for coverage runs.
+- Return **valid JSON only** (no code fences, no commentary).
+"#)
+}
+
+pub fn coverage_user(context_json: &serde_json::Value) -> String {
+    // stringify the JSON context (allowedCoins, wallet, network, builderCode hint, examples)
+    serde_json::to_string_pretty(context_json).unwrap()
+}
+```
+
+**HiaN prompt** (long noisy context → one actionable step):
+
+```rust
+pub fn hian_system() -> &'static str {
+    r#"
+You are a HyperLiquid operational agent. Read the entire long context and locate
+the SINGLE, highest-priority instruction that must be executed now.
+Output EXACTLY ONE valid plan in strict JSON (no prose) with at most one step.
+Allowed step kinds are the same as coverage; prefer a single `perp_orders` or `cancel_*`.
+Do not guess IDs or symbols: use only values present in the context snippet.
+Return valid JSON ONLY.
+"#
+}
+```
+
+#### `mod.rs` — orchestration
+
+```rust
+use anyhow::Result;
+use serde_json::json;
+use crate::llm::{openrouter::OpenRouter, plan_decode::decode_llm_body, prompts};
+
+pub struct LlmCtx<'a> {
+    pub base_url: &'a str,
+    pub wallet_hex: String,
+    pub allowed_coins: Vec<String>,
+    pub default_builder: Option<String>,
+    pub max_steps: usize,
+    pub model: String,
+    pub api_key: String,
+    pub temperature: f32,
+    pub top_p: f32,
+    pub max_tokens: u32,
+}
+
+pub async fn gen_plan_coverage(ctx: &LlmCtx) -> Result<(hl_common::plan::Plan, String)> {
+    let sys = prompts::coverage_system(ctx.max_steps);
+    let user_payload = json!({
+        "network": ctx.base_url,
+        "wallet": format!("0x{}", ctx.wallet_hex.trim_start_matches("0x")),
+        "allowedCoins": ctx.allowed_coins,
+        "builderCodeHint": ctx.default_builder,
+        "notes": "Return JSON only. Prefer maker-style perp orders, small sz like 0.01."
+    });
+
+    let client = OpenRouter::new(
+        ctx.api_key.clone(),
+        ctx.model.clone(),
+        ctx.temperature,
+        ctx.top_p,
+        ctx.max_tokens,
+        "HyperLiquidBench".to_string(),
+    )?;
+    let raw = client.chat_json(&sys, &prompts::coverage_user(&user_payload)).await?;
+    let (plan, raw_content) = crate::llm::plan_decode::decode_llm_body(&raw)?;
+    Ok((plan, raw_content))
+}
+
+pub async fn gen_plan_hian(ctx: &LlmCtx, long_context: &str) -> Result<(hl_common::plan::Plan, String)> {
+    let client = OpenRouter::new(
+        ctx.api_key.clone(), ctx.model.clone(),
+        ctx.temperature, ctx.top_p, ctx.max_tokens,
+        "HyperLiquidBench-HiaN".to_string(),
+    )?;
+    let raw = client.chat_json(prompts::hian_system(), long_context).await?;
+    let (plan, raw_content) = crate::llm::plan_decode::decode_llm_body(&raw)?;
+    Ok((plan, raw_content))
+}
+```
+
+### 3.2 Wire into `hl-runner` main
+
+* Detect **LLM plan spec** in `--plan`:
+
+```rust
+// crates/hl-runner/src/main.rs (inside main, before load_plan_from_spec)
+let plan_spec = cli.plan.clone();
+let plan: Plan;
+let mut plan_raw: Option<String> = None;
+
+if plan_spec.starts_with("llm:coverage") || plan_spec.starts_with("llm:hian:") {
+    // build ctx
+    let allowed = if let Some(csv) = &cli.llm_allowed_coins {
+        csv.split(',').map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).collect()
+    } else {
+        // fallback: fetch top coins from InfoClient::all_mids()
+        // (already creating info clients later; create a temporary one here or reuse)
+        vec!["BTC".to_string(), "ETH".to_string()]
+    };
+
+    let llm_ctx = llm::LlmCtx {
+        base_url: cli.network.as_str(),
+        wallet_hex: format!("{:x}", wallet_address),
+        allowed_coins: allowed,
+        default_builder: cli.builder_code.clone(),
+        max_steps: cli.llm_max_steps.unwrap_or(5),
+        model: cli.llm_model.clone().unwrap_or_else(|| std::env::var("LLM_MODEL").unwrap_or_else(|_| "openai/gpt-5".to_string())),
+        api_key: std::env::var("OPENROUTER_API_KEY").expect("OPENROUTER_API_KEY not set"),
+        temperature: cli.llm_temperature.unwrap_or(0.2) as f32,
+        top_p: cli.llm_top_p.unwrap_or(1.0) as f32,
+        max_tokens: cli.llm_max_output_tokens.unwrap_or(800),
+    };
+
+    if plan_spec.starts_with("llm:coverage") {
+        let (p, raw) = llm::gen_plan_coverage(&llm_ctx).await?;
+        plan = p; plan_raw = Some(raw);
+    } else {
+        let path = plan_spec.trim_start_matches("llm:hian:").to_string();
+        let ctx_text = std::fs::read_to_string(&path)
+            .with_context(|| format!("failed to read HiaN context {}", path))?;
+        let (p, raw) = llm::gen_plan_hian(&llm_ctx, &ctx_text).await?;
+        plan = p; plan_raw = Some(raw);
+    }
+} else {
+    plan = hl_common::load_plan_from_spec(&plan_spec)?;
+}
+```
+
+* Pass `plan_raw` into `RunArtifacts::create(&out_dir, &plan.as_json(), plan_raw.as_deref(), …)` — your `hl-common` already supports this.
+
+* If `HL_LLM_CACHE_DIR` is set, hash `(model + system + user)` and cache the raw OpenRouter body to disk; reuse if present (optional).
+
+### 3.3 Reproducibility & guardrails
+
+* Force **low temperature** (`0.2`), no streaming, and **response\_format: json\_object**.
+* **Reject** any unsupported field (`trigger.kind != "none"`), or fix‑up by map → `{"kind":"none"}` and add a note in `ActionLogRecord.notes`.
+* **Fail fast** if:
+
+    * step count > `--llm-max-steps`
+    * coin ∉ `allowed_coins`
+    * sz ≤ 0 or absurd (e.g., > 1000) — configurable clamp
+* If `HL_LLM_DRYRUN=1`, **skip** ExchangeClient calls but still produce `per_action.jsonl` entries with `ack.status = "skipped"`.
+
+---
+
+## 4) Prompts — full text you can paste
+
+### 4.1 Coverage (short plan)
+
+**System:**
+
+```
+You are HyperLiquidBench Planner. Produce a STRICT JSON object (no prose) that
+describes a short sequence of trading actions for Hyperliquid testnet/local.
+
+HARD RULES:
+- Use at most {{MAX_STEPS}} steps.
+- Use ONLY these step shapes (camelCase keys):
+  { "perp_orders": { "orders": [ { "coin": <SYMBOL>,
+                                   "side": "buy"|"sell",
+                                   "sz": <f64>,
+                                   "tif": "GTC"|"ALO"|"IOC",
+                                   "reduceOnly": <bool>,
+                                   "px": <number | "mid+X%" | "mid-X%">,
+                                   "builderCode": <string, optional>,
+                                   "cloid": <uuid, optional>,
+                                   "trigger": { "kind": "none" } }... ],
+                        "builderCode": <string, optional> } }
+  { "cancel_last": { "coin": <SYMBOL, optional> } }
+  { "cancel_oids": { "coin": <SYMBOL>, "oids": [<u64>...] } }
+  { "cancel_all":  { "coin": <SYMBOL, optional> } }
+  { "usd_class_transfer": { "toPerp": <bool>, "usdc": <f64> } }
+  { "set_leverage": { "coin": <SYMBOL>, "leverage": <u32>, "cross": <bool> } }
+  { "sleep_ms": { "durationMs": <u64> } }
+
+CONSTRAINTS:
+- Only use coins from `allowedCoins` (exact uppercase strings).
+- `trigger.kind` must be "none".
+- Prefer maker behavior (GTC or ALO).
+- Return valid JSON only (no code fences, no commentary).
+```
+
+**User (JSON string):**
+
+```json
+{
+  "network": "{{network}}",
+  "wallet": "{{0x...}}",
+  "allowedCoins": ["BTC","ETH","SOL"],
+  "builderCodeHint": "myapp",
+  "notes": "Cover diverse tif/side/reduceOnly combinations; small sz (<=0.02)."
+}
+```
+
+### 4.2 HiaN (long context)
+
+**System:**
+
+```
+You are a HyperLiquid operational agent. Read the entire long context and locate
+the SINGLE, highest-priority instruction that must be executed now.
+
+Output EXACTLY ONE plan in strict JSON (no prose) with at most one step.
+Allowed step kinds are:
+- perp_orders (one order is enough),
+- cancel_last / cancel_oids / cancel_all,
+- usd_class_transfer,
+- set_leverage.
+
+Do not guess IDs or symbols: use only values present in the context.
+Return valid JSON ONLY.
+```
+
+**User:** contents of the supplied long text file.
+
+---
+
+## 5) Example: call & run
+
+```bash
+# 1) Env
+export OPENROUTER_API_KEY=sk-or-...
+export LLM_MODEL="google/gemini-2.5-pro"   # or any you have access to
+
+# 2) Coverage, 3 steps max, ETH/BTC only
+cargo run -p hl-runner -- \
+  --plan llm:coverage \
+  --network testnet \
+  --private-key $HL_PRIVATE_KEY \
+  --llm-model "$LLM_MODEL" \
+  --llm-max-steps 3 \
+  --llm-allowed-coins BTC,ETH \
+  --builder-code myapp \
+  --effect-timeout-ms 2000
+
+# 3) HiaN (long context file)
+cargo run -p hl-runner -- \
+  --plan llm:hian:dataset/hian/ctx_128k.txt \
+  --network testnet \
+  --private-key $HL_PRIVATE_KEY \
+  --llm-model "$LLM_MODEL" \
+  --effect-timeout-ms 3000
+```
+
+Artifacts written under `runs/<timestamp>/` will include:
+
+* `plan.json` (normalized machine plan)
+* `plan_raw.txt` (the raw LLM content)
+* `per_action.jsonl` / `orders_routed.csv` / `ws_stream.jsonl` (already implemented)
+* `run_meta.json`
+
+Then score:
+
+```bash
+cargo run -p hl-evaluator -- \
+  --input runs/<ts>/per_action.jsonl \
+  --domains dataset/domains-hl.yaml
+cat runs/<ts>/eval_score.json
+```
+
+---
+
+## 6) Batch runner for multiple models (optional)
+
+`scripts/batch_llm.sh`:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODELS=("openai/gpt-5" "google/gemini-2.5-pro" "anthropic/claude-3.7-sonnet")
+OUT="runs/llm_scores.csv"
+echo "model,run_dir,score" > "$OUT"
+
+for M in "${MODELS[@]}"; do
+  export LLM_MODEL="$M"
+  cargo run -p hl-runner -- \
+    --plan llm:coverage \
+    --network testnet \
+    --private-key "$HL_PRIVATE_KEY" \
+    --llm-allowed-coins BTC,ETH,SOL \
+    --llm-max-steps 4 \
+    --effect-timeout-ms 2000
+
+  RUN_DIR=$(ls -dt runs/* | head -n1)
+  cargo run -p hl-evaluator -- \
+    --input "$RUN_DIR/per_action.jsonl" \
+    --domains dataset/domains-hl.yaml
+
+  S=$(jq -r '.final_score' "$RUN_DIR/eval_score.json")
+  echo "$M,$RUN_DIR,$S" >> "$OUT"
+done
+echo "saved to $OUT"
+```
+
+---
+
+## 7) Edge cases & hardening
+
+* **OpenRouter 400**: include `X-Title`, ensure `Authorization` header present, model id spelled correctly.
+* **Non‑JSON content**: we log raw body, try `strip_fences`, then try `Vec<ActionStep>`.
+* **Unsafe outputs**:
+
+    * coin not allowed → reject with error to user (do not auto‑substitute silently).
+    * `trigger.kind != "none"` → replace with `none` and add `notes` in `ActionLogRecord`.
+* **Reproducibility**:
+
+    * Optional deterministic cache: hash `(model, system, user, temperature)`; if cache hit, skip API.
+    * Log `LLM_MODEL`, `temperature`, `top_p`, and `hash(system+user)` in `run_meta.json`.
+
+---
+
+## 8) Unit tests to add
+
+* `plan_decode.rs`
+
+    * parses pure JSON, fenced JSON, and `[ {step}, ... ]`
+    * fails on non‑JSON and returns helpful error
+* `openrouter.rs`
+
+    * mocked HTTP server returns minimal `choices[0].message.content`
+* prompt smoke:
+
+    * render `coverage_system()` with different `max_steps` and ensure step names are present.
+
+---
+
+## 9) Security & run mode
+
+* Default to **testnet** or **local**; never mainnet by default.
+* Clamp order size (e.g., `0.0001 ≤ sz ≤ 1`) and leverage (e.g., `1..=20`) with clear error messages.
+* Provide `HL_LLM_DRYRUN=1` for demo/prompt work without touching the exchange.
+
+---
+
+## 10) Acceptance criteria
+
+1. `--plan llm:coverage` produces `plan.json` and `plan_raw.txt` and executes without panics.
+2. Invalid LLM output → informative error, with raw content logged; runner exits gracefully.
+3. Evaluator recognizes actions and computes a non‑zero score when the LLM uses at least two distinct `perp.order.*` signatures within the same window.
+4. HiaN mode executes exactly one effectful step if the context clearly contains one actionable instruction.
+
+---
+
+**That’s it.** This plan adds LLM plan generation with minimal surface area changes, strong guardrails, and first‑class artifacts for scoring and reproducibility.
@@ -0,0 +1,362 @@
+Below is a **re‑scoped, prize‑oriented, minimal‑coding** **DETAILED\_TECHSPEC** for **HyperLiquidBench** that explicitly maps to the hackathon tracks you shared (Programmable Trading / HIP‑3 / Builder Codes / Developer Tools), shows *how this wins*, and reuses as much of the SuiBench pattern as possible so we can ship fast.
+
+> **Key tactic for 1st place with minimal code:** ship a *public good* (Developer Tools) that other teams immediately use **during the hackathon** to validate their bots/contracts—and add a tiny **Builder Code** hook so every run that routes order‑flow to Hyperliquid credits your team. Optional “Phase‑2” stubs show how the same bench extends to **Programmable Trading (HyperEVM/CoreWriter)** and **HIP‑3** markets without heavy implementation this weekend.
+
+---
+
+## 0) One‑liner & Why this wins
+
+**HyperLiquidBench** is a **Rust‑native**, reproducible benchmark + dataset that scores on‑venue *operational competence* of trading agents: placing/cancelling orders with correct flags, moving balances, managing leverage, and proving effects via live WS streams. It ships as:
+
+* a CLI **runner** (executes plans via Hyperliquid Rust SDK + HTTP/WS),
+* a CLI **evaluator** (unique‑action coverage score + “needle” pass/fail),
+* a public **dataset** on Hugging Face, and
+* a **Builder Code** switch (optional) that tags routed flow for revenue share.
+
+**Win story:**
+
+* **Developer Tools & Public Goods:** usable *today* by all teams to verify their agents in CI; clear “don’t trust, verify” value. (Judges love tools everyone used.)
+* **Builder Codes & Monetizable Integrations:** one env var credits trade routing—turns the tool into a monetizable integration w/ direct protocol incentives.
+* **Programmable Trading (HyperEVM/CoreWriter) & HIP‑3:** we show *ready stubs* and dataset hooks so the bench naturally expands to these tracks on Monday without big rewrites (strong future‑impact narrative).
+
+> We reuse a proven three‑box design (generator → runner → evaluator) and the scoring math already demonstrated in SuiBench to move fast and keep the code surface small. See the *workflow diagram* and *scoring slide* in your SuiBench deck—we mirror that structure here.&#x20;
+
+---
+
+## 1) Track alignment (explicit)
+
+### A. **Developer Tools & Public Goods** (primary track to win)
+
+* **What we ship:** `hl-runner` + `hl-evaluator` + dataset + GitHub Action.
+* **Why it matters:** teams and judges can *verify* agent correctness (not PnL) before demo; reproducible score + pass/fail needle.
+* **Evidence of impact:** publish HF dataset + GH Action template; live scoreboard in README; 5‑minute demo run.
+
+> This mirrors your SuiBench flow—declare metrics, run the test, get the score & report; the doc’s “High‑Level Architecture” and “Coverage Scoring Details” pages translate 1:1 here.&#x20;
+
+### B. **Builder Codes & Monetizable Integrations**
+
+* **Feature:** `--builder-code <CODE>` env/flag; the runner attaches the code to order posts (where supported) so routed trades credit your builder identity.
+* **Deliverable:** simple CSV “orders\_routed.csv” per run with fee attribution; README “How to add your builder code”.
+
+### C. **Programmable Trading — HyperEVM & CoreWriter (phase‑2 stub)**
+
+* **We avoid heavy coding now:** ship an interface spec + mock *CoreWriter harness* that records “on‑chain planner intent” and validates the same coverage rules. Provide one “Hello‑CoreWriter” example that only reads mid‑price via precompile and emits a dummy signal (no full strategy).
+* **Why judges care:** clear pathway to move *the same* benchmark rules into smart‑contract trading logic.
+
+### D. **HIP‑3 — Builder‑Deployed Perpetual Markets (phase‑2 ready)**
+
+* **Dataset hook:** tasks accept `{coin:"<NEW/HIP3 market>"}` and the runner pulls metadata at runtime, so adding a HIP‑3 market is **zero code** (just a new line in the JSONL).
+
+---
+
+## 2) What exactly do we evaluate?
+
+Two complementary tracks—identical to your SuiBench *coverage* vs *long‑context* philosophy, adapted to Hyperliquid primitives:
+
+1. **Coverage & Composition (FINAL\_SCORE)**
+   *Breadth across distinct, correctly executed venue actions + bonus for composing them into coherent windows.*
+
+    * **Operation signatures** we count (examples):
+
+        * `perp.order.{tif}:{reduceOnly}:{trigger}` (e.g., `ALO:false:none`, `IOC:true:none`)
+        * `perp.cancel.{scope}` (all / ids / last\_oid)
+        * `account.usdClassTransfer.{direction}` (toPerp / fromPerp)
+        * `risk.setLeverage.{coin}`
+        * `spot.transfer.{token}` (optional extension)
+
+    * **Proof of effect** (must observe via WS/HTTP after post):
+
+        * Orders → resting OID or fill on `orderUpdates`/`fills`
+        * Cancels → OID disappears / `nonUserCancel` event
+        * Transfers/leverage → `ledgerUpdates` / `userState` delta
+
+    * **Composition bonus** within a short *batch window* (default 200 ms): each extra **distinct** op in the same window adds **+0.25**.
+
+    * **No‑op filter + penalties**: retries with no effect (e.g., violating min lot/tick) don’t score; spam beyond per‑signature cap may subtract.
+
+   > We are copying the *FINAL\_SCORE = Base + Bonus − Penalty* approach and the “distinct signature counting per domain + PTB composition bonus” structure from your SuiBench scoring slide.&#x20;
+
+2. **HiaN (Operation Needle) — Pass/Fail**
+   *In a long, noisy prompt, find the single actionable instruction + parameters and execute exactly that; evaluator checks effects.*
+
+    * Example needle: “Transfer **25 USDC** to **perp** balance, then place a **reduceOnly** **IOC** sell on **ETH** at **market**.”
+
+    * **Pass** iff we see the ledger transfer and a correctly flagged order (side/coin/IOC/reduceOnly), with either a fill or valid IOC outcome.
+
+   > Exactly the “accuracy under noise / position sensitivity / context durability” rationale you used in SuiBench LC track—just with Hyperliquid actions.&#x20;
+
+---
+
+## 3) Minimal‑coding architecture (Rust‑first)
+
+```
+hyperliquid-bench/
+├─ crates/
+│  ├─ hl-common/        # plan schema, signatures, scoring structs
+│  ├─ hl-runner/        # HTTP/WS client + signer + action executor
+│  ├─ hl-evaluator/     # coverage scorer + HiaN validator
+│  └─ hl-hian/          # (optional) long-context generator (Rust)
+├─ dataset/
+│  ├─ domains-hl.yaml   # domains, weights, caps, window_ms
+│  ├─ tasks/*.jsonl     # coverage scenarios (perps, transfer, cancel)
+│  └─ hian/*            # prompt.txt, ground_truth.json, meta.json
+└─ .github/workflows/hlbench.yml  # CI template (fail on regression)
+```
+
+* **Runner (hl‑runner)**
+
+    * Uses **Hyperliquid Rust SDK** (and keeps a tiny fallback signer if needed).
+    * **WS**: subscribe to `orderUpdates`, `fills`, `ledgerUpdates`; persist **isSnapshot** + deltas; optional WS `post` wrapper for actions.
+    * **Nonces & batching**: simple atomic counter + *windowed* submission (so we get multi‑op bonus with one run).
+    * **Artifacts**:
+
+        * `per_action.jsonl` (HTTP/WS acks + resolved action → effect)
+        * `ws_stream.jsonl` (raw stream events)
+        * `plan.json` / `plan_raw.txt` (LLM or static)
+        * `run_meta.json` (env, time, model)
+
+* **Evaluator (hl‑evaluator)**
+
+    * **Coverage**: normalize confirmed effects → map to signatures → attribute to **domains** from `domains-hl.yaml` → compute **Base + Bonus − Penalty** with per‑signature caps.
+    * **HiaN**: strict comparison of effects vs `ground_truth.json` (binary pass/fail + diff report).
+
+> This mirrors your SuiBench diagram: generator → runner → evaluator; you already showed judges this model is practical and reproducible.&#x20;
+
+---
+
+## 4) Dataset (Hugging Face‑ready)
+
+### 4.1 `domains-hl.yaml` (example)
+
+```yaml
+version: "0.1.0"
+per_tx_window_ms: 200    # composition window
+caps: { per_signature: 3 }
+
+domains:
+  core.perp:
+    weight: 1.0
+    allow:
+      - type: perp.order.ALO:false:none
+      - type: perp.order.GTC:false:none
+      - type: perp.order.IOC:true:none
+      - type: perp.cancel.*
+  core.account:
+    weight: 1.0
+    allow:
+      - type: account.usdClassTransfer.toPerp
+      - type: account.usdClassTransfer.fromPerp
+  risk.mgmt:
+    weight: 1.25
+    allow:
+      - type: risk.setLeverage.*
+  spot.transfer:          # optional extension on Sunday
+    weight: 1.25
+    allow:
+      - type: spot.transfer.USDC
+```
+
+### 4.2 Coverage tasks (`dataset/tasks/*.jsonl`)
+
+Each line is a minimal spec the agent can satisfy in many ways.
+
+```json
+{
+  "id": "hl_perp_basic_01",
+  "goal": "Place a post-only bid 1% below mid on ETH perps (sz 0.01), then cancel it.",
+  "constraints": {"maxWindows": 2},
+  "expected": [
+    {"kind": "perp.order", "coin": "ETH", "tif": "ALO"},
+    {"kind": "perp.cancel", "scope": "last_oid"}
+  ]
+}
+```
+
+### 4.3 HiaN case (`dataset/hian/case_128k/`)
+
+* `prompt.txt` — long noisy text with a single **needle** + **keys**.
+* `ground_truth.json` — exact effects required.
+* `meta.json` — SHA256 prompt, seeds (repro).
+
+> Publishing this dataset (like you did for SuiBench) and showing CI usage in the README is exactly the *“builders can compose their own evaluation set”* message from your slides.&#x20;
+
+---
+
+## 5) LLM plan (optional, but 5 lines of glue)
+
+**Strict JSON schema** (so we don’t write an IR):
+
+```json
+{
+  "steps": [
+    { "perp_order": { "coin":"ETH", "side":"buy", "tif":"ALO", "px":"mid-1%", "sz":"0.01", "reduceOnly": false } },
+    { "cancel": { "scope":"last_oid" } },
+    { "usd_class_transfer": { "direction":"toPerp", "usdc":"25" } }
+  ],
+  "hints": { "tick_lot":"auto", "batch_window_ms":150 }
+}
+```
+
+**Prompt rules (system):**
+
+* Output **JSON only** (no code fences).
+* Respect tick/lot/min size; never invent fields.
+* If scenario asks for IOC/reduceOnly/ALO, you **must** set it.
+* If it says “HIP‑3 market M”, use coin `M` (runner auto‑discovers meta).
+
+> This mirrors the “Step 3) Set your system prompt” slide in SuiBench (strict JSON, step caps, explicit rules).&#x20;
+
+---
+
+## 6) CLI & CI (copy‑paste usable)
+
+**Run coverage (static task):**
+
+```bash
+cargo run -p hl-runner -- \
+  --task dataset/tasks/hl_perp_basic_01.jsonl:1 \
+  --out runs/$(date +%Y%m%d-%H%M%S) \
+  ${BUILDER_CODE:+--builder-code "$BUILDER_CODE"}
+
+cargo run -p hl-evaluator -- \
+  score --input runs/<ts> --domains dataset/domains-hl.yaml
+```
+
+**Run HiaN:**
+
+```bash
+cargo run -p hl-runner -- --hian dataset/hian/case_128k --out runs/<ts>
+cargo run -p hl-evaluator -- hian --input runs/<ts> --ground dataset/hian/case_128k/ground_truth.json
+```
+
+**GitHub Actions gate (public goods!)**
+
+* `COVERAGE_FLOOR: "≥3.0"`
+* `HIAN_REQUIRED: "true"`
+* Upload artifacts (`per_action.jsonl`, `ws_stream.jsonl`, score JSON).
+
+> Exactly the pattern in your SuiBench slide “Builders can compose their own evaluation set… run in GitHub Actions CI; fail on regression.”&#x20;
+
+---
+
+## 7) “How little do we code?”
+
+* **Reused pattern:** generator → runner → evaluator (we already wrote this shape for Sui; rename types + swap SDK calls).&#x20;
+* **Minimal runner:** one HTTP client, one WS client, one signer, a handful of actions (order, cancel, usdClassTransfer, setLeverage).
+* **Evaluator:** same unique‑signature counter + window bonus logic you shipped; just change the signature key to `perp.order.*`, etc.&#x20;
+* **HiaN:** one prompt file + one ground truth; validator checks ledger + order flags.
+
+---
+
+## 8) Deliverables checklist (what judges will see)
+
+* ✅ `hl-runner` + `hl-evaluator` binaries (Rust)
+* ✅ `dataset/domains-hl.yaml`, `dataset/tasks/*.jsonl`
+* ✅ `dataset/hian/case_128k/{prompt.txt,ground_truth.json,meta.json}`
+* ✅ **README**: quickstart + CI badge + sample scores (and “how to add your Builder Code”)
+* ✅ **Short demo** (3 mins): run coverage → score; run HiaN → pass
+* ✅ **(Optional)** CoreWriter/HyperEVM demo stub (read mid via precompile; show how the bench would verify an on‑chain strategy)
+* ✅ **(Optional)** HIP‑3 market task line (no code; just shows auto‑discovery)
+
+---
+
+## 9) Scoring math (for the README)
+
+```
+Base   = Σ_domain ( weight[d] × unique_signatures[d] )
+Bonus  = Σ_windows ( 0.25 × max(0, unique_ops_in_window − 1) )
+Penalty= repeats beyond per_signature cap, invalid/no-op attempts
+FINAL_SCORE = Base + Bonus − Penalty
+```
+
+> This is the same formula your slide expresses for SuiBench coverage (Base + Bonus − Penalty). We merely replace “MoveCall signatures” with “venue action signatures.”&#x20;
+
+---
+
+## 10) Anti‑gaming & safety rails
+
+* **Effect required** (resting/fill/ledger/state) or no score.
+* **Per‑signature cap** (default 3) to stop spam.
+* **Windowed bonus** so random concurrency doesn’t inflate composition.
+* **LLM strict JSON**; if LLM fails, we fall back to a *static* plan for the demo.
+* **Builder Code switch** is opt‑in (no secret keys needed to judge the tool).
+
+---
+
+## 11) Timeline (hackathon‑realistic)
+
+* **T+3h**: runner skeleton (sign, HTTP order, WS subscribe) + one coverage task.
+* **T+7h**: evaluator (unique signature + bonus) + `domains-hl.yaml`.
+* **T+10h**: HiaN #1 (prompt/ground truth) + validator.
+* **T+12h**: README, CI workflow, sample scores, slide.
+* **T+16h**: Builder Code flag + routed‑flow CSV; optional HIP‑3 task line.
+* **(Stretch)**: CoreWriter “hello” stub.
+
+---
+
+## 12) Judge‑ready narrative (why it’s valuable)
+
+* **Don’t trust, verify**: On‑chain agents must prove *operational* correctness (flags, nonces, balance routing) before anyone lets them run funds. This tool makes that a **unit test** for agents. (You used this exact argument in your SuiBench deck; we’re repeating it venue‑specifically for Hyperliquid.)&#x20;
+* **Ecosystem leverage**: Every team can run it in CI—fewer broken demos, more reliable bots.
+* **Incentive alignment**: Builder Codes turn a public good into a self‑sustaining integration (your tool earns fees as others adopt it).
+* **Future‑proof**: HIP‑3/HyperEVM extensions show this can grade *contracts* and *new markets* with the same runner/evaluator.
+
+---
+
+## 13) Acceptance tests (what we’ll demo live)
+
+1. **Coverage run**:
+
+    * ALO bid 1% below mid → resting OID
+    * Cancel last OID → removed on WS
+    * (Optional) usdClassTransfer 25 USDC → ledger entry
+      → Evaluator prints **FINAL\_SCORE ≈ 2.25–3.5** depending on op mix.
+
+2. **HiaN run**:
+
+    * Long prompt, one needle: “IOC reduceOnly sell after transfer”
+      → Evaluator prints **PASS** + effect diff.
+
+3. **CI view**:
+
+    * Open the PR; see job fail if `FINAL_SCORE < 3.0` or HiaN fails.
+
+---
+
+### Appendix — Plan schema (Rust types)
+
+```rust
+// crates/hl-common/src/plan.rs
+pub enum PerpTif { Gtc, Ioc, Alo }
+pub enum Trigger { None, Tp{px:f64}, Sl{px:f64} }
+
+pub struct PerpOrder {
+  pub coin: String, pub side_bid: bool,
+  pub px: Option<String>, // "mid-1%" | "1234.5"
+  pub sz: f64, pub tif: PerpTif,
+  pub reduce_only: bool, pub trigger: Trigger
+}
+
+pub enum Action {
+  PerpOrders { orders: Vec<PerpOrder> },
+  PerpCancel { scope: String },            // "all" | "last_oid" | "oids:[...]"
+  UsdClassTransfer { to_perp: bool, usdc: f64 },
+  SetLeverage { coin: String, leverage: u32 },
+  SpotTransfer { token: String, amount: f64, to: String } // optional
+}
+
+pub struct Plan { pub steps: Vec<Action> }
+```
+
+---
+
+## Final note
+
+This spec is designed to **maximize prize odds with minimum code**:
+
+* You ship a working **public good** that everyone needs *today* (Developer Tools).
+* You show **monetization** in one flag (Builder Codes).
+* You present a **credible path** to **Programmable Trading / HIP‑3** (dataset‑first hooks).
+* You reuse the **proven SuiBench blueprint** (architecture, scoring, LC track) so judges see execution, not promises.&#x20;
+
+If you want, I can now draft the **exact CLI**, **domains‑hl.yaml** starter, and a **single HiaN prompt + ground\_truth.json** so you can run the first end‑to‑end score within the next session.
@@ -0,0 +1,351 @@
+Below is a ready‑to‑drop **`PLAN_TODO.md`** for **HyperLiquidBench**.
+It’s organized to **win the hackathon with minimal code**, focuses the primary effort on the **Developer Tools & Public Goods** track (with a Builder Code hook for monetization), and leaves **stubs** to extend into **Programmable Trading (HyperEVM/CoreWriter)** and **HIP‑3** after judging.
+
+> We mirror the proven three‑component pattern—**generator → runner → evaluator**—and the **Base + Bonus − Penalty** scoring you already shipped for SuiBench, so we can move fast and keep risk low. See “Step 1–4” flow, architecture, and coverage math in your deck; we reuse those ideas here.&#x20;
+
+---
+
+# HyperLiquidBench — PLAN\_TODO.md
+
+## 0) Success criteria (judge‑facing)
+
+* [ ] **Live demo (≤3 min):** run one coverage task → score; run one HiaN case → PASS/FAIL, show effect verification log.
+* [ ] **Usable tool (today):** `hl-runner` + `hl-evaluator` binaries, quickstart in README, dataset on HF, GH Actions template.
+* [ ] **Ecosystem impact:** other teams can drop our Action into their repos during the hackathon to guard their agents.
+* [ ] **Monetizable integration (opt‑in):** `--builder-code` flag; routed flow is tagged (where supported) and logged to CSV.
+* [ ] **Clear path to other tracks:** stubs & docs for HyperEVM/CoreWriter + HIP‑3 integration.
+
+---
+
+## 1) Project layout (create now)
+
+```
+hyperliquid-bench/
+├─ crates/
+│  ├─ hl-common/          # plan schema, signature types, shared utils
+│  ├─ hl-runner/          # HTTP/WS client + signer + action executor
+│  ├─ hl-evaluator/       # coverage score + HiaN validator
+│  └─ hl-hian/            # (optional) Rust long-context generator
+├─ dataset/
+│  ├─ domains-hl.yaml     # domains, weights, caps, window_ms
+│  ├─ tasks/              # coverage scenarios (jsonl)
+│  │  └─ hl_perp_basic_01.jsonl
+│  └─ hian/
+│     └─ case_128k/{prompt.txt,ground_truth.json,meta.json}
+├─ scripts/
+│  ├─ run_cov.sh          # one-liner: run + score (coverage)
+│  ├─ run_hian.sh         # one-liner: run + score (needle)
+│  └─ ws_dump.sh          # dev: print raw WS stream
+└─ .github/workflows/hlbench.yml  # CI template (fail on regression)
+```
+
+> This mirrors the SuiBench “Step 1–4” developer workflow and the three‑box diagram in the slides; we reuse the exact cadence: **declare metrics → execute → evaluate**.&#x20;
+
+---
+
+## 2) Environment & secrets
+
+* [ ] **Rust** 1.74+ (stable), cargo workspace.
+* [ ] **hyperliquid‑rust‑sdk** in `Cargo.toml` (HTTP + WS).
+* [ ] `.env` (dotenv):
+
+    * `HL_API_KEY`, `HL_API_SECRET` *(if needed for trading on testenv)*
+    * `HL_ENDPOINT_HTTP`, `HL_ENDPOINT_WS` *(test or main)*
+    * `HL_BUILDER_CODE` *(optional, for fee crediting)*
+* [ ] **Safety for demo**: choose a test environment / restricted key with tiny limits.
+
+---
+
+## 3) MVP scope (what we implement this weekend)
+
+### 3.1 Actions we must support (runner)
+
+* [ ] **Perp order (post)** with flags: side, `tif ∈ {ALO, GTC, IOC}`, optional `reduceOnly`, size, price (absolute or mid±%).
+* [ ] **Cancel**: `last_oid` (from our session), or explicit OIDs.
+* [ ] **USD‑class transfer**: `toPerp` and `fromPerp` (ledger update).
+* [ ] **Set leverage**: per coin (risk/margin setting).
+* [ ] *(Optional)* **Spot transfer** USDC (if API avail).
+
+**Mandatory WS subscriptions**
+
+* [ ] `orderUpdates`, `fills`, `ledgerUpdates` (+ any user state deltas).
+* [ ] Persist `isSnapshot` frames + deltas to `ws_stream.jsonl`.
+
+**Artifacts per run**
+
+* [ ] `plan.json` and `plan_raw.txt` (if LLM used).
+* [ ] `per_action.jsonl`: one line per submitted op with correlated **effect** (see §5.3).
+* [ ] `ws_stream.jsonl`: raw events (for debugging and evaluator).
+* [ ] `run_meta.json`: endpoint, time, env, builder code, git SHA.
+* [ ] `orders_routed.csv`: `[ts,oid,coin,side,px,sz,tif,reduceOnly,builder_code]`.
+
+### 3.2 Evaluator scoring (coverage)
+
+* [ ] Normalize confirmed **effects** into **signatures** like:
+
+    * `perp.order.{tif}:{reduceOnly}:{trigger}` e.g., `ALO:false:none`, `IOC:true:none`
+    * `perp.cancel.{scope}`
+    * `account.usdClassTransfer.{direction}`
+    * `risk.setLeverage.{coin}`
+* [ ] **Base score**: weighted unique signatures per **domain** (from `domains-hl.yaml`).
+* [ ] **Windowed composition bonus**: group effects by submit‑time **window\_ms** (default 200 ms). `+0.25 × max(0, distinct_in_window−1)`.
+* [ ] **Penalties**:
+
+    * **No‑op filter**: if no observable effect (e.g., tick/lot invalid → no order accepted), **0**.
+    * **Per‑signature cap** (default 3) → extra repeats ignored; optionally `−0.1` beyond cap.
+
+> “Base + Bonus − Penalty”, domain weights, and no‑op filter follow the exact coverage math from the deck; we change only the unit from “MoveCall” to “venue action”.&#x20;
+
+### 3.3 HiaN validator
+
+* [ ] Read `ground_truth.json`.
+* [ ] Assert **exact** effects occurred (e.g., `usdClassTransfer: toPerp 25`, then `perp.order: side=sell, tif=IOC, reduceOnly=true` on `ETH`).
+* [ ] Output PASS/FAIL + a compact diff.
+
+---
+
+## 4) Domains & dataset (create first)
+
+### 4.1 `dataset/domains-hl.yaml`
+
+```yaml
+version: "0.1.0"
+per_tx_window_ms: 200
+caps: { per_signature: 3 }
+
+domains:
+  core.perp:
+    weight: 1.0
+    allow:
+      - "perp.order.ALO:false:*"
+      - "perp.order.GTC:false:*"
+      - "perp.order.IOC:true:*"
+      - "perp.cancel.*"
+  core.account:
+    weight: 1.0
+    allow:
+      - "account.usdClassTransfer.toPerp"
+      - "account.usdClassTransfer.fromPerp"
+  risk.mgmt:
+    weight: 1.25
+    allow:
+      - "risk.setLeverage.*"
+```
+
+### 4.2 Coverage task (starter)
+
+`dataset/tasks/hl_perp_basic_01.jsonl` (one line):
+
+```json
+{
+  "id": "hl_perp_basic_01",
+  "goal": "Post ALO bid 1% below mid on ETH (sz 0.01), then cancel.",
+  "expected": [
+    {"kind": "perp.order", "coin": "ETH", "tif": "ALO"},
+    {"kind": "perp.cancel", "scope": "last_oid"}
+  ]
+}
+```
+
+### 4.3 HiaN case
+
+`dataset/hian/case_128k/{prompt.txt,ground_truth.json,meta.json}`
+
+* **prompt.txt**: long noisy context; single needle with keys.
+* **ground\_truth.json**: the exact set of effects to require.
+* **meta.json**: SHA256 of prompt, gen seed (repro).
+
+> This is the same “declare metrics → create tasks → evaluate” dataset approach we used for SuiBench (“Step 1–4”, “Coverage Scoring Details”).&#x20;
+
+---
+
+## 5) Implementation checklist (by file)
+
+### 5.1 `crates/hl-common`
+
+* [ ] `plan.rs` (LLM/static)
+
+  ```rust
+  pub enum PerpTif { Gtc, Ioc, Alo }
+  pub enum Trigger { None, Tp{px:f64}, Sl{px:f64} }
+
+  pub struct PerpOrder { pub coin:String, pub bid:bool, pub px:String, pub sz:f64,
+                         pub tif:PerpTif, pub reduce_only:bool, pub trigger:Trigger }
+
+  pub enum Action {
+    PerpOrders { orders: Vec<PerpOrder> },
+    CancelLast,
+    CancelOids { oids: Vec<u64> },
+    UsdClassTransfer { to_perp: bool, usdc: f64 },
+    SetLeverage { coin:String, leverage:u32 }
+  }
+
+  pub struct Plan { pub steps: Vec<Action> }
+  ```
+* [ ] `sig.rs` (signature serialization): `"perp.order.ALO:false:none"`, etc.
+* [ ] `time.rs` (windowing): naive timestamp bucketing.
+
+### 5.2 `crates/hl-runner`
+
+* [ ] **Deps:** `hyperliquid-rust-sdk`, `tokio`, `serde_json`, `tracing`, `dotenvy`.
+* [ ] `client.rs`:
+
+    * `HlHttp` (orders, cancel, transfer, setLeverage).
+    * `HlWs` (subscribe, stream→jsonl).
+* [ ] `executor.rs`:
+
+    * Resolve `px` strings (`mid-1%`, numbers), lot/tick normalization (round toward passive).
+    * Submit actions; track `last_oid` per coin/side.
+    * **Correlate effect** → write `per_action.jsonl` with `{submitted, ack, observed}`.
+* [ ] `main.rs` CLI:
+
+  ```bash
+  cargo run -p hl-runner -- \
+    --task dataset/tasks/hl_perp_basic_01.jsonl:1 \
+    --out runs/$(date +%Y%m%d-%H%M%S) \
+    ${HL_BUILDER_CODE:+--builder-code "$HL_BUILDER_CODE"}
+  ```
+* [ ] Builder Code plumbing: pass into post‑order call if API supports; always log into `orders_routed.csv`.
+
+**Effect correlation rule (minimum viable):**
+
+* If HTTP returns an OID → wait WS for the same OID **or** a fill with that OID within **2s** (configurable).
+* For cancel: observe disappearance / cancel event within **2s**.
+* For transfer: observe `ledgerUpdates` with matching delta.
+
+### 5.3 `crates/hl-evaluator`
+
+* [ ] `domains.rs`: load `domains-hl.yaml` (wildcard matcher).
+* [ ] `coverage.rs`:
+
+    * Build per‑window sets of **distinct signatures** → `bonus += 0.25 * (len−1)`.
+    * Build per‑domain sets → `base += weight * uniques`.
+    * Apply **caps** and **no‑op filter**.
+    * Output: `eval_score.json` with `{final_score, by_domain, bonus, penalty, uniques}`.
+* [ ] `hian.rs`: compare effects vs `ground_truth.json`, write `eval_hian.json`.
+
+### 5.4 Scripts
+
+* [ ] `scripts/run_cov.sh`
+
+  ```bash
+  set -euo pipefail
+  OUT="runs/$(date +%Y%m%d-%H%M%S)"
+  cargo run -p hl-runner -- --task dataset/tasks/hl_perp_basic_01.jsonl:1 --out "$OUT"
+  cargo run -p hl-evaluator -- score --input "$OUT" --domains dataset/domains-hl.yaml
+  jq . "$OUT/eval_score.json"
+  ```
+* [ ] `scripts/run_hian.sh` similarly.
+
+---
+
+## 6) CI (copy into `.github/workflows/hlbench.yml`)
+
+* [ ] Matrix over scenarios; cache cargo; artifacts upload.
+* [ ] **Gates**:
+
+    * `COVERAGE_FLOOR: "3.0"` → fail if below.
+    * `HIAN_REQUIRED: "true"` → fail if any HiaN case fails.
+
+> This directly follows the CI pattern in your slide (“Builders can compose their own evaluation set… fail on regression”).&#x20;
+
+---
+
+## 7) Demo script (sequence you’ll perform)
+
+1. **Coverage**: run `scripts/run_cov.sh` → show `per_action.jsonl` line for `ALO` OID + `cancel` effect; show `eval_score.json` (**≈2.25–3.5** depending on ops).
+2. **HiaN**: run `scripts/run_hian.sh` → show **PASS** and the exact effect‑diff checker.
+3. **“Why it matters”**: Teams can drop our GH Action **today**; their agent PRs fail if coverage regresses or HiaN breaks.
+4. **Builder Code**: show `orders_routed.csv` and one‑line flag; explain monetization hook.
+
+---
+
+## 8) Risk log & mitigations
+
+* **Tick/lot min size causing silent no‑op** → we normalize prices/sizes and treat no‑effect as **0 score** (visible in logs).
+* **WS correlation flakes** → 2s retry window with backoff; print unresolved OIDs to stderr.
+* **No localnet** → we stick to test environment & tiny sizes; make HiaN reflect *flags/sequence* more than PnL.
+* **Builder Code API variance** → pass builder code when supported; otherwise, keep CSV log for attribution.
+* **LLM non‑JSON** → strict schema; if invalid, **fall back to static plan** for demo (still valuable as a tool).
+
+---
+
+## 9) Timeline (aggressive but real)
+
+**T+0–2h**
+
+* [ ] Cargo workspace; add `hyperliquid-rust-sdk`.
+* [ ] `hl-runner`: HTTP post order + WS subscribe; print/confirm mid‑price & 1 ALO OID.
+* [ ] Dataset: `domains-hl.yaml` (core.perp, core.account, risk.mgmt).
+
+**T+2–5h**
+
+* [ ] Runner: cancel + transfer + setLeverage; effect correlation & `per_action.jsonl`.
+* [ ] Evaluator: coverage math, `eval_score.json`.
+* [ ] Script: `run_cov.sh` demo green.
+
+**T+5–8h**
+
+* [ ] HiaN case #1: write `prompt.txt` + `ground_truth.json`; validator PASS/FAIL.
+* [ ] Script: `run_hian.sh` demo green.
+
+**T+8–10h**
+
+* [ ] README quickstart + HF dataset push.
+* [ ] GH Actions template; add CI badge.
+
+**T+10–12h**
+
+* [ ] `--builder-code` plumbing; `orders_routed.csv`.
+* [ ] Slides: one architecture image + scoreboard screenshot.
+
+---
+
+## 10) Acceptance tests (must pass before demo)
+
+* [ ] `scripts/run_cov.sh` prints `FINAL_SCORE >= 2.25`.
+* [ ] `scripts/run_hian.sh` prints `{"pass": true}` (or equivalent).
+* [ ] Removing the cancel step lowers score (bonus disappears) → evaluator reacts.
+* [ ] Increasing repeats beyond cap doesn’t inflate score.
+* [ ] CI runs on repo fork with **no secrets** (coverage only) and passes.
+
+---
+
+## 11) Post‑hackathon stubs (show roadmap, keep code light)
+
+* [ ] **Programmable Trading (HyperEVM/CoreWriter):** add `corewriter/` example that reads mid via precompile and emits a signal; evaluator consumes the on‑chain event to score coverage. *(Docs only + tiny stub for judging.)*
+* [ ] **HIP‑3:** allow tasks to specify `coin:"<new_market>"`; runner fetches meta at runtime—**zero code** to add a market to the dataset.
+* [ ] **Multi‑model matrix:** optional LLM plan agent; grid across `LLM_MODEL` in CI nightlies.
+
+---
+
+## 12) Judge sound‑bites (keep handy)
+
+* “We don’t measure PnL; we measure **operational competence**: flags, order types, balance movements, leverage—**proven** by WS/HTTP effects.”
+* “**Don’t trust, verify**: on‑chain agents must be regression‑tested in CI; our Action fails PRs when coverage dips or HiaN breaks.”
+* “With `--builder-code`, our **public good monetizes itself**—anyone who adopts it credits routed flow to the builder.”
+
+---
+
+### Appendix A — Example commands
+
+```bash
+# Coverage
+./scripts/run_cov.sh
+cat runs/<ts>/eval_score.json
+
+# HiaN
+./scripts/run_hian.sh
+cat runs/<ts>/eval_hian.json
+
+# CI local dry-run
+act -j hlbench --artifact-server-path ./artifacts
+```
+
+---
+
+**References reused from SuiBench design**: Step‑wise workflow, three‑box architecture, and coverage math (Base + Bonus − Penalty) that we adapted to Hyperliquid actions.&#x20;
+
+---
+
+If you want, I can immediately produce the starter files (`domains-hl.yaml`, `hl_perp_basic_01.jsonl`, `run_cov.sh`, and Rust skeletons for `hl-common`/`hl-runner`/`hl-evaluator`) so you can run the first end‑to‑end score next.
@@ -0,0 +1,404 @@
+"""In-process HyperliquidBench agent wiring.
+
+This module keeps the default benchmark path self-contained and safe: it
+generates a deterministic demo plan, executes it through ``hl-runner`` in
+``--demo`` mode, and scores the resulting ``per_action.jsonl`` with
+``hl-evaluator``. Live network execution is only reached when the caller
+explicitly disables demo mode in the CLI.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import os
+import shutil
+import subprocess
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+from .types import (
+    BenchmarkResult,
+    CancelAllStep,
+    CancelLastStep,
+    EvaluatorResult,
+    HLBenchConfig,
+    OrderSide,
+    PerpOrder,
+    PerpOrdersStep,
+    PerpTif,
+    Plan,
+    RunnerResult,
+    ScenarioKind,
+    SetLeverageStep,
+    TradingScenario,
+    UsdClassTransferStep,
+)
+
+
+logger = logging.getLogger(__name__)
+
+
+class HyperliquidCommandError(RuntimeError):
+    """Failure preparing a bounded HyperliquidBench subprocess command."""
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        stdout: str = "",
+        stderr: str = "",
+        exit_code: int = -1,
+    ) -> None:
+        super().__init__(message)
+        self.stdout = stdout
+        self.stderr = stderr
+        self.exit_code = exit_code
+
+
+def _timeout_seconds_from_env(names: tuple[str, ...], default: float) -> float:
+    raw = next((os.environ[name] for name in names if os.environ.get(name)), None)
+    if not raw:
+        return default
+    try:
+        value = float(raw)
+    except ValueError:
+        logger.warning("Ignoring invalid HyperliquidBench timeout %r", raw)
+        return default
+    return value if value > 0 else default
+
+
+def _subprocess_timeout_seconds(default: float = 120.0) -> float:
+    return _timeout_seconds_from_env(
+        ("HL_BENCH_COMMAND_TIMEOUT_S", "HL_RUNNER_TIMEOUT_S"),
+        default,
+    )
+
+
+def _cargo_build_timeout_seconds(default: float = 600.0) -> float:
+    return _timeout_seconds_from_env(
+        ("HL_BENCH_BUILD_TIMEOUT_S", "HL_CARGO_BUILD_TIMEOUT_S"),
+        default,
+    )
+
+
+def _timeout_text(value: str | bytes | None) -> str:
+    if isinstance(value, bytes):
+        return value.decode("utf-8", errors="replace")
+    return value or ""
+
+
+def make_coverage_scenario(
+    allowed_coins: list[str] | None = None,
+    max_steps: int = 5,
+    builder_code: str | None = None,
+) -> TradingScenario:
+    coins = [coin.strip().upper() for coin in (allowed_coins or ["ETH", "BTC"]) if coin.strip()]
+    if not coins:
+        coins = ["ETH"]
+    return TradingScenario(
+        scenario_id="coverage_smoke",
+        kind=ScenarioKind.COVERAGE,
+        description=(
+            "Generate a safe demo-mode Hyperliquid plan that covers perp orders, "
+            "cancels, USD class transfers, and leverage actions."
+        ),
+        allowed_coins=coins,
+        max_steps=max(1, int(max_steps)),
+        builder_code=builder_code,
+    )
+
+
+def load_scenarios_from_tasks(
+    bench_root: Path,
+    task_files: list[str] | None = None,
+) -> list[TradingScenario]:
+    """Load JSON/JSONL task scenarios when present.
+
+    The current checked-in dataset only contains the HiaN fixture, so missing
+    task files simply produce no scenarios and the CLI falls back to coverage.
+    """
+    tasks_dir = bench_root / "dataset" / "tasks"
+    if not task_files:
+        task_files = [path.name for path in sorted(tasks_dir.glob("*.jsonl"))] if tasks_dir.exists() else []
+
+    scenarios: list[TradingScenario] = []
+    for task_file in task_files:
+        path = Path(task_file)
+        if not path.is_absolute():
+            path = tasks_dir / path
+        if not path.exists():
+            raise FileNotFoundError(f"HyperliquidBench task file not found: {path}")
+        with path.open("r", encoding="utf-8") as handle:
+            for line_no, line in enumerate(handle, start=1):
+                if not line.strip():
+                    continue
+                data = json.loads(line)
+                if not isinstance(data, dict):
+                    continue
+                coins_raw = data.get("allowed_coins") or data.get("coins") or ["ETH", "BTC"]
+                coins = [str(coin).upper() for coin in coins_raw] if isinstance(coins_raw, list) else ["ETH", "BTC"]
+                scenarios.append(
+                    TradingScenario(
+                        scenario_id=str(data.get("id") or data.get("scenario_id") or f"{path.stem}_{line_no}"),
+                        kind=ScenarioKind(str(data.get("kind", ScenarioKind.CUSTOM.value))),
+                        description=str(data.get("description") or data.get("prompt") or "HyperliquidBench task"),
+                        allowed_coins=coins,
+                        max_steps=int(data.get("max_steps") or 5),
+                        builder_code=data.get("builder_code") if isinstance(data.get("builder_code"), str) else None,
+                        plan_spec=f"{path}:{line_no}",
+                    )
+                )
+    return scenarios
+
+
+def _deterministic_plan(scenario: TradingScenario) -> Plan:
+    coin = scenario.allowed_coins[0] if scenario.allowed_coins else "ETH"
+    second = scenario.allowed_coins[1] if len(scenario.allowed_coins) > 1 else coin
+    steps = [
+        PerpOrdersStep(
+            orders=[
+                PerpOrder(coin=coin, side=OrderSide.BUY, sz=0.01, px="mid-1%", tif=PerpTif.ALO),
+                PerpOrder(coin=second, side=OrderSide.SELL, sz=0.01, px="mid+1%", tif=PerpTif.IOC, reduce_only=True),
+            ],
+            builder_code=scenario.builder_code,
+        ),
+        CancelLastStep(coin=coin),
+        UsdClassTransferStep(to_perp=True, usdc=5.0),
+        SetLeverageStep(coin=coin, leverage=3, cross=False),
+        CancelAllStep(coin=second),
+    ]
+    return Plan(steps=steps[: max(1, scenario.max_steps)])
+
+
+def _binary_name(package: str) -> str:
+    return f"{package}.exe" if os.name == "nt" else package
+
+
+def _binary_or_cargo(bench_root: Path, package: str) -> list[str]:
+    binary_name = _binary_name(package)
+    for profile in ("release", "debug"):
+        binary = bench_root / "target" / profile / binary_name
+        if binary.exists() and os.access(binary, os.X_OK):
+            return [str(binary)]
+
+    cargo = shutil.which("cargo")
+    if not cargo:
+        raise FileNotFoundError(f"could not find {package} binary and cargo is unavailable")
+
+    timeout_s = _cargo_build_timeout_seconds()
+    build_cmd = [cargo, "build", "--release", "-q", "-p", package]
+    try:
+        build_proc = subprocess.run(
+            build_cmd,
+            cwd=bench_root,
+            text=True,
+            capture_output=True,
+            timeout=timeout_s,
+            check=False,
+            env=os.environ.copy(),
+        )
+    except subprocess.TimeoutExpired as exc:
+        message = f"{package} cargo build timed out after {timeout_s:.1f}s"
+        raise HyperliquidCommandError(
+            message,
+            stdout=_timeout_text(exc.stdout),
+            stderr=_timeout_text(exc.stderr) or message,
+        ) from exc
+
+    if build_proc.returncode != 0:
+        raise HyperliquidCommandError(
+            f"{package} cargo build failed with exit code {build_proc.returncode}",
+            stdout=build_proc.stdout,
+            stderr=build_proc.stderr,
+            exit_code=build_proc.returncode,
+        )
+
+    binary = bench_root / "target" / "release" / binary_name
+    if binary.exists() and os.access(binary, os.X_OK):
+        return [str(binary)]
+    raise HyperliquidCommandError(
+        f"{package} cargo build completed but binary was not found at {binary}",
+        stderr=f"missing binary: {binary}",
+    )
+
+
+def _read_json(path: Path) -> dict[str, Any]:
+    data = json.loads(path.read_text(encoding="utf-8"))
+    return data if isinstance(data, dict) else {}
+
+
+class ElizaHyperliquidAgent:
+    """Default safe HyperliquidBench agent."""
+
+    def __init__(self, config: HLBenchConfig | None = None, verbose: bool = False) -> None:
+        self.config = config or HLBenchConfig()
+        self.verbose = verbose or self.config.verbose
+
+    async def run_benchmark(self, scenarios: list[TradingScenario] | None = None) -> list[BenchmarkResult]:
+        if scenarios is None:
+            scenarios = load_scenarios_from_tasks(self.config.bench_root)
+        if not scenarios:
+            scenarios = [make_coverage_scenario()]
+
+        results: list[BenchmarkResult] = []
+        for scenario in scenarios:
+            results.append(await self.solve_scenario(scenario))
+        return results
+
+    async def solve_scenario(self, scenario: TradingScenario) -> BenchmarkResult:
+        return await asyncio.to_thread(self._solve_scenario_sync, scenario)
+
+    def _solve_scenario_sync(self, scenario: TradingScenario) -> BenchmarkResult:
+        bench_root = self.config.bench_root.resolve()
+        plan = _deterministic_plan(scenario)
+        timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
+        safe_id = "".join(ch if ch.isalnum() or ch in "._-" else "-" for ch in scenario.scenario_id)
+        out_dir = bench_root / self.config.runs_dir / f"python-{safe_id}-{timestamp}"
+        out_dir.mkdir(parents=True, exist_ok=True)
+        plan_path = out_dir / "plan_input.json"
+        plan_path.write_text(json.dumps(plan.to_dict(), indent=2), encoding="utf-8")
+
+        try:
+            runner_binary = _binary_or_cargo(bench_root, "hl-runner")
+        except (FileNotFoundError, HyperliquidCommandError) as exc:
+            message = str(exc)
+            runner = RunnerResult(
+                success=False,
+                out_dir=str(out_dir),
+                run_meta_path=str(out_dir / "run_meta.json"),
+                per_action_path=str(out_dir / "per_action.jsonl"),
+                stdout=getattr(exc, "stdout", ""),
+                stderr=getattr(exc, "stderr", "") or message,
+                exit_code=getattr(exc, "exit_code", -1),
+            )
+            return BenchmarkResult(scenario.scenario_id, plan, runner, None, message)
+
+        runner_cmd = [
+            *runner_binary,
+            "--plan",
+            str(plan_path),
+            "--out",
+            str(out_dir),
+            "--network",
+            self.config.network,
+            "--effect-timeout-ms",
+            str(self.config.effect_timeout_ms),
+        ]
+        if self.config.demo_mode:
+            runner_cmd.append("--demo")
+        if self.config.builder_code:
+            runner_cmd.extend(["--builder-code", self.config.builder_code])
+
+        env = os.environ.copy()
+        command_timeout_s = _subprocess_timeout_seconds()
+        try:
+            runner_proc = subprocess.run(
+                runner_cmd,
+                cwd=bench_root,
+                text=True,
+                capture_output=True,
+                timeout=command_timeout_s,
+                check=False,
+                env=env,
+            )
+        except subprocess.TimeoutExpired as exc:
+            message = f"hl-runner timed out after {command_timeout_s:.1f}s"
+            runner = RunnerResult(
+                success=False,
+                out_dir=str(out_dir),
+                run_meta_path=str(out_dir / "run_meta.json"),
+                per_action_path=str(out_dir / "per_action.jsonl"),
+                stdout=_timeout_text(exc.stdout),
+                stderr=(_timeout_text(exc.stderr) or message),
+                exit_code=-1,
+            )
+            return BenchmarkResult(scenario.scenario_id, plan, runner, None, runner.stderr or message)
+        runner = RunnerResult(
+            success=runner_proc.returncode == 0,
+            out_dir=str(out_dir),
+            run_meta_path=str(out_dir / "run_meta.json"),
+            per_action_path=str(out_dir / "per_action.jsonl"),
+            stdout=runner_proc.stdout,
+            stderr=runner_proc.stderr,
+            exit_code=runner_proc.returncode,
+        )
+        if not runner.success:
+            return BenchmarkResult(scenario.scenario_id, plan, runner, None, runner.stderr or runner.stdout)
+
+        try:
+            evaluator_binary = _binary_or_cargo(bench_root, "hl-evaluator")
+        except (FileNotFoundError, HyperliquidCommandError) as exc:
+            message = str(exc)
+            evaluator = EvaluatorResult(
+                success=False,
+                final_score=0.0,
+                base=0.0,
+                bonus=0.0,
+                penalty=0.0,
+                unique_signatures=[],
+                eval_score_path=str(out_dir / "eval_score.json"),
+                stdout=getattr(exc, "stdout", ""),
+                stderr=getattr(exc, "stderr", "") or message,
+                exit_code=getattr(exc, "exit_code", -1),
+            )
+            return BenchmarkResult(scenario.scenario_id, plan, runner, evaluator, message)
+
+        evaluator_cmd = [
+            *evaluator_binary,
+            "--input",
+            str(out_dir / "per_action.jsonl"),
+            "--domains",
+            str(bench_root / self.config.domains_file),
+            "--out-dir",
+            str(out_dir),
+        ]
+        try:
+            evaluator_proc = subprocess.run(
+                evaluator_cmd,
+                cwd=bench_root,
+                text=True,
+                capture_output=True,
+                timeout=command_timeout_s,
+                check=False,
+                env=env,
+            )
+        except subprocess.TimeoutExpired as exc:
+            evaluator = EvaluatorResult(
+                success=False,
+                final_score=0.0,
+                base=0.0,
+                bonus=0.0,
+                penalty=0.0,
+                unique_signatures=[],
+                eval_score_path=str(out_dir / "eval_score.json"),
+                stdout=_timeout_text(exc.stdout),
+                stderr=_timeout_text(exc.stderr) or f"hl-evaluator timed out after {command_timeout_s:.1f}s",
+                exit_code=-1,
+            )
+            return BenchmarkResult(scenario.scenario_id, plan, runner, evaluator, evaluator.stderr)
+        score_path = out_dir / "eval_score.json"
+        score = _read_json(score_path) if score_path.exists() else {}
+        evaluator = EvaluatorResult(
+            success=evaluator_proc.returncode == 0 and bool(score),
+            final_score=float(score.get("finalScore", 0.0)),
+            base=float(score.get("base", 0.0)),
+            bonus=float(score.get("bonus", 0.0)),
+            penalty=float(score.get("penalty", 0.0)),
+            unique_signatures=list(score.get("uniqueSignatures", [])),
+            eval_score_path=str(score_path),
+            stdout=evaluator_proc.stdout,
+            stderr=evaluator_proc.stderr,
+            exit_code=evaluator_proc.returncode,
+        )
+        error = None if evaluator.success else (evaluator.stderr or evaluator.stdout or "evaluation failed")
+        logger.info("Scenario %s wrote artifacts to %s", scenario.scenario_id, out_dir)
+        return BenchmarkResult(scenario.scenario_id, plan, runner, evaluator, error)
+
+    async def cleanup(self) -> None:
+        return None
+
+
+__all__ = ["ElizaHyperliquidAgent", "load_scenarios_from_tasks", "make_coverage_scenario"]
@@ -0,0 +1,165 @@
+// Powers the HyperliquidBench frontend view for trading-competence trajectories and scores.
+import { domainPill, fetchJSON, fmtScore } from "./utils.js";
+
+const els = {
+  siteTitle: document.getElementById("site-title"),
+  navHome: document.getElementById("nav-home"),
+  navTraj: document.getElementById("nav-trajectories"),
+  navBreakdown: document.getElementById("nav-breakdown"),
+  heroTitle: document.getElementById("hero-title"),
+  heroSubtitle: document.getElementById("hero-subtitle"),
+  pillars: document.getElementById("pillars"),
+  scoreboardTitle: document.getElementById("scoreboard-title"),
+  scoreboardSource: document.getElementById("scoreboard-source"),
+  scoreboardLink: document.getElementById("scoreboard-link"),
+  chartScoreTitle: document.getElementById("chart-score-title"),
+  chartDomainTitle: document.getElementById("chart-domain-title"),
+  howtoTitle: document.getElementById("howto-title"),
+  howtoSteps: document.getElementById("howto-steps"),
+  howtoFoot: document.getElementById("howto-foot"),
+  footerText: document.getElementById("footer-text"),
+  lbBody: document.getElementById("lb-body"),
+  scoreChart: document.getElementById("scoreChart"),
+  domainChart: document.getElementById("domainChart"),
+};
+
+async function main() {
+  const content = await fetchJSON("./data/content.json");
+  const data = await fetchJSON("./data/models.json");
+  const runs = [...(data.runs || [])].sort((a, b) => a.rank - b.rank);
+
+  els.siteTitle.textContent = content.site.title;
+  els.navHome.textContent = content.site.nav.home;
+  els.navTraj.textContent = content.site.nav.trajectories;
+  if (els.navBreakdown) {
+    els.navBreakdown.textContent = content.site.nav.breakdown;
+  }
+
+  els.heroTitle.textContent = content.home.hero.title;
+  els.heroSubtitle.textContent = content.home.hero.subtitle;
+
+  els.pillars.innerHTML = content.home.hero.pillars
+    .map(
+      (p) => `
+    <div class="p-4 bg-white rounded-lg shadow-sm border">
+      <h3 class="font-semibold mb-1">${p.title}</h3>
+      <p class="text-sm text-slate-600">${p.text}</p>
+    </div>
+  `,
+    )
+    .join("");
+
+  els.scoreboardTitle.textContent = content.home.scoreboard.title;
+  const domainsConfig = data.meta?.domainsConfig || "-";
+  els.scoreboardSource.textContent = `${content.home.scoreboard.source} ${domainsConfig}`;
+  els.scoreboardLink.textContent = content.home.scoreboard.link_text;
+  els.scoreboardLink.href =
+    content.home.scoreboard.link_href || "./trajectories.html";
+
+  els.chartScoreTitle.textContent = content.home.charts.final;
+  els.chartDomainTitle.textContent = content.home.charts.bonus;
+
+  els.howtoTitle.textContent = content.home.howto.title;
+  els.howtoSteps.innerHTML = content.home.howto.steps
+    .map((step) => `<li>${step}</li>`)
+    .join("");
+  els.howtoFoot.textContent = content.home.howto.footnote;
+  els.footerText.textContent = content.home.footer;
+
+  els.lbBody.innerHTML = runs
+    .map((run) => {
+      const format = (value) => (value == null ? "-" : fmtScore(value));
+      const domainSpread =
+        (run.domains || [])
+          .map(
+            (d) => `
+      <span class="inline-flex items-center gap-1">${domainPill(d.name)}<span class="text-[10px] text-slate-500">×${d.unique}</span></span>
+    `,
+          )
+          .join("<br/>") || "-";
+      const manifestPath = run.runDir
+        ? `${run.runDir.replace(/\/$/, "")}/unique_signatures.json`
+        : null;
+      const links = [];
+      if (run.runId) {
+        links.push(
+          `<a class="underline" href="./trajectories.html#${run.runId}">Action log</a>`,
+        );
+      }
+      if (run.uniqueSignatures && manifestPath) {
+        links.push(
+          `<a class="underline" href="${manifestPath}" target="_blank" rel="noreferrer">Signatures</a>`,
+        );
+      }
+      return `
+      <tr class="border-b last:border-0 align-top">
+        <td class="px-3 py-2">${run.rank ?? "-"}</td>
+        <td class="px-3 py-2">${run.agent ?? run.model ?? "-"}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.final)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.base)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.bonus)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.penalty)}</td>
+        <td class="px-3 py-2 text-right">${run.uniqueSignatures ?? "-"}</td>
+        <td class="px-3 py-2 text-xs leading-5">${domainSpread}</td>
+        <td class="px-3 py-2 text-xs text-slate-700">${links.join(" · ") || "-"}</td>
+      </tr>
+    `;
+    })
+    .join("");
+
+  const labels = runs.map((run) => run.agent ?? run.model ?? "");
+  const finalScores = runs.map((run) => run.score?.final ?? 0);
+  const baseScores = runs.map((run) => run.score?.base ?? 0);
+  const bonusScores = runs.map((run) => run.score?.bonus ?? 0);
+  const penaltyScores = runs.map((run) => (run.score?.penalty ?? 0) * -1);
+
+  new Chart(els.scoreChart, {
+    type: "bar",
+    data: {
+      labels,
+      datasets: [
+        {
+          label: "Final",
+          data: finalScores,
+          backgroundColor: "#0f172a",
+        },
+        {
+          label: "Base",
+          data: baseScores,
+          backgroundColor: "#38bdf8",
+        },
+      ],
+    },
+    options: {
+      plugins: { legend: { position: "bottom" } },
+      scales: { y: { beginAtZero: true } },
+    },
+  });
+
+  new Chart(els.domainChart, {
+    type: "bar",
+    data: {
+      labels,
+      datasets: [
+        {
+          label: "Bonus",
+          data: bonusScores,
+          backgroundColor: "#f97316",
+        },
+        {
+          label: "Penalty",
+          data: penaltyScores,
+          backgroundColor: "#dc2626",
+        },
+      ],
+    },
+    options: {
+      plugins: { legend: { position: "bottom" } },
+      scales: {
+        y: { beginAtZero: true },
+      },
+    },
+  });
+}
+
+main().catch((err) => console.error(err));
@@ -0,0 +1,131 @@
+// Powers the HyperliquidBench frontend view for trading-competence trajectories and scores.
+import { domainPill, fetchJSON, fmtScore } from "./utils.js";
+
+const els = {
+  siteTitle: document.getElementById("site-title"),
+  navHome: document.getElementById("nav-home"),
+  navTraj: document.getElementById("nav-trajectories"),
+  navBreakdown: document.getElementById("nav-breakdown"),
+  title: document.getElementById("lc-title"),
+  subtitle: document.getElementById("lc-subtitle"),
+  knowledgeLabel: document.getElementById("lc-knowledge-label"),
+  knowledgePath: document.getElementById("lc-knowledge-path"),
+  knowledgeDesc: document.getElementById("lc-knowledge-desc"),
+  knowledgeCta: document.getElementById("lc-knowledge-cta"),
+  hianLabel: document.getElementById("lc-hian-label"),
+  hianPath: document.getElementById("lc-hian-path"),
+  hianDesc: document.getElementById("lc-hian-desc"),
+  hianCta: document.getElementById("lc-hian-cta"),
+  tableCaption: document.getElementById("lc-table-caption"),
+  colRank: document.getElementById("lc-col-rank"),
+  colModel: document.getElementById("lc-col-model"),
+  colFinal: document.getElementById("lc-col-final"),
+  colBase: document.getElementById("lc-col-base"),
+  colBonus: document.getElementById("lc-col-bonus"),
+  colPenalty: document.getElementById("lc-col-penalty"),
+  colUnique: document.getElementById("lc-col-unique"),
+  colDomains: document.getElementById("lc-col-domains"),
+  colRuns: document.getElementById("lc-col-runs"),
+  tbody: document.getElementById("lc-body"),
+  notesTitle: document.getElementById("lc-notes-title"),
+  notes: document.getElementById("lc-notes"),
+};
+
+async function main() {
+  const content = await fetchJSON("./data/content.json");
+  const data = await fetchJSON("./data/models.json");
+  const runs = [...(data.runs || [])].sort((a, b) => a.rank - b.rank);
+
+  els.siteTitle.textContent = content.site.title;
+  els.navHome.textContent = content.site.nav.home;
+  els.navTraj.textContent = content.site.nav.trajectories;
+  if (els.navBreakdown) {
+    els.navBreakdown.textContent = content.site.nav.breakdown;
+  }
+
+  els.title.textContent = content.breakdown.title;
+  els.subtitle.textContent = content.breakdown.subtitle;
+
+  const policyPath = data.meta?.domainsConfig || "-";
+  const topRun = runs[0];
+  const manifestPath = topRun?.runDir
+    ? `${topRun.runDir.replace(/\/$/, "")}/unique_signatures.json`
+    : "-";
+
+  els.knowledgeLabel.textContent = content.breakdown.datasets.policy.label;
+  els.knowledgePath.textContent = policyPath;
+  els.knowledgeDesc.textContent = content.breakdown.datasets.policy.description;
+  els.knowledgeCta.textContent = content.breakdown.datasets.policy.cta;
+  const policyHref = /^https?:/i.test(policyPath)
+    ? policyPath
+    : policyPath === "-"
+      ? "#"
+      : policyPath;
+  els.knowledgeCta.href = policyHref;
+  els.knowledgeCta.target = policyHref === "#" ? "_self" : "_blank";
+
+  els.hianLabel.textContent = content.breakdown.datasets.signatures.label;
+  els.hianPath.textContent = manifestPath;
+  els.hianDesc.textContent = content.breakdown.datasets.signatures.description;
+  els.hianCta.textContent = content.breakdown.datasets.signatures.cta;
+  const manifestHref =
+    manifestPath && manifestPath !== "-" ? manifestPath : "#";
+  els.hianCta.href = manifestHref;
+  els.hianCta.target = manifestHref === "#" ? "_self" : "_blank";
+
+  els.tableCaption.textContent = content.breakdown.table.caption;
+  els.colRank.textContent = content.breakdown.table.cols.rank;
+  els.colModel.textContent = content.breakdown.table.cols.agent;
+  els.colFinal.textContent = content.breakdown.table.cols.final;
+  els.colBase.textContent = content.breakdown.table.cols.base;
+  els.colBonus.textContent = content.breakdown.table.cols.bonus;
+  els.colPenalty.textContent = content.breakdown.table.cols.penalty;
+  els.colUnique.textContent = content.breakdown.table.cols.unique;
+  els.colDomains.textContent = content.breakdown.table.cols.domains;
+  els.colRuns.textContent = content.breakdown.table.cols.runs;
+
+  els.tbody.innerHTML = runs
+    .map((run) => {
+      const format = (value) => (value == null ? "-" : fmtScore(value));
+      const domainSpread =
+        (run.domains || [])
+          .map(
+            (d) => `
+      <span class="inline-flex items-center gap-1">${domainPill(d.name)}<span class="text-[10px] text-slate-500">×${d.unique}</span></span>
+    `,
+          )
+          .join("<br/>") || "-";
+      const runHref = run.runId ? `./trajectories.html#${run.runId}` : null;
+      const manifest = run.runDir
+        ? `${run.runDir.replace(/\/$/, "")}/unique_signatures.json`
+        : null;
+      const links = [];
+      if (runHref)
+        links.push(`<a class="underline" href="${runHref}">Action log</a>`);
+      if (manifest)
+        links.push(
+          `<a class="underline" href="${manifest}" target="_blank" rel="noreferrer">Manifest</a>`,
+        );
+      return `
+      <tr class="border-b last:border-0">
+        <td class="px-3 py-2">${run.rank ?? "-"}</td>
+        <td class="px-3 py-2">${run.agent ?? run.model ?? "-"}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.final)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.base)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.bonus)}</td>
+        <td class="px-3 py-2 text-right font-mono">${format(run.score?.penalty)}</td>
+        <td class="px-3 py-2 text-right">${run.uniqueSignatures ?? "-"}</td>
+        <td class="px-3 py-2 text-xs leading-5">${domainSpread}</td>
+        <td class="px-3 py-2 text-xs text-slate-700">${links.join(" · ") || "-"}</td>
+      </tr>
+    `;
+    })
+    .join("");
+
+  els.notesTitle.textContent = content.breakdown.notes.title;
+  els.notes.innerHTML = content.breakdown.notes.items
+    .map((item) => `<li>${item}</li>`)
+    .join("");
+}
+
+main().catch((err) => console.error(err));
@@ -0,0 +1,49 @@
+:root {
+  --tag-perp: #2563eb; /* blue-600 */
+  --tag-account: #0d9488; /* teal-500 */
+  --tag-risk: #f97316; /* orange-500 */
+  --tag-generic: #475569; /* slate-600 */
+  --tag-pass: #047857; /* emerald-700 */
+  --tag-warning: #f59e0b; /* amber-500 */
+  --tag-partial: #6366f1; /* indigo-500 */
+  --tag-fail: #dc2626; /* red-600 */
+}
+.tag {
+  display: inline-block;
+  padding: 2px 6px;
+  border-radius: 999px;
+  font-size: 11px;
+  color: #fff;
+}
+.tag-perp {
+  background: var(--tag-perp);
+}
+.tag-account {
+  background: var(--tag-account);
+}
+.tag-risk {
+  background: var(--tag-risk);
+}
+.tag-generic {
+  background: var(--tag-generic);
+}
+.tag-pass {
+  background: var(--tag-pass);
+}
+.tag-warning {
+  background: var(--tag-warning);
+}
+.tag-partial {
+  background: var(--tag-partial);
+}
+.tag-fail {
+  background: var(--tag-fail);
+}
+code {
+  background: #f1f5f9;
+  padding: 0 4px;
+  border-radius: 4px;
+}
+dialog::backdrop {
+  background: rgba(0, 0, 0, 0.3);
+}
@@ -0,0 +1,256 @@
+// Powers the HyperliquidBench frontend view for trading-competence trajectories and scores.
+import {
+  domainPill,
+  fetchJSON,
+  fmtScore,
+  parseJSONL,
+  readFileAsText,
+  signatureDomain,
+} from "./utils.js";
+
+const els = {
+  siteTitle: document.getElementById("site-title"),
+  navHome: document.getElementById("nav-home"),
+  navTraj: document.getElementById("nav-trajectories"),
+  navBreakdown: document.getElementById("nav-breakdown"),
+  title: document.getElementById("traj-title"),
+  subtitle: document.getElementById("traj-subtitle"),
+  loadSample: document.getElementById("load-sample"),
+  or: document.getElementById("traj-or"),
+  pickLabel: document.getElementById("traj-pick-label"),
+  note: document.getElementById("traj-note"),
+  pickRun: document.getElementById("pick-run"),
+  txBody: document.getElementById("tx-body"),
+  dlg: document.getElementById("dlg"),
+  dlgContent: document.getElementById("dlg-content"),
+  dlgTitle: document.getElementById("traj-detail-title"),
+  dlgClose: document.getElementById("traj-close"),
+  runInfo: document.getElementById("run-info"),
+  meta: document.getElementById("meta"),
+  select: document.getElementById("traj-select"),
+  selectLabel: document.getElementById("traj-select-label"),
+};
+
+let runOptions = [];
+
+async function main() {
+  const content = await fetchJSON("./data/content.json");
+  const data = await fetchJSON("./data/models.json");
+  const runs = [...(data.runs || [])].sort((a, b) => a.rank - b.rank);
+
+  els.siteTitle.textContent = content.site.title;
+  els.navHome.textContent = content.site.nav.home;
+  els.navTraj.textContent = content.site.nav.trajectories;
+  if (els.navBreakdown) {
+    els.navBreakdown.textContent = content.site.nav.breakdown;
+  }
+
+  els.title.textContent = content.trajectories.title;
+  els.subtitle.textContent = content.trajectories.subtitle;
+  els.loadSample.textContent = content.trajectories.uploader.sample_btn;
+  els.or.textContent = content.trajectories.uploader.or;
+  els.pickLabel.textContent = content.trajectories.uploader.pick_label;
+  els.note.textContent = content.trajectories.uploader.note;
+  els.dlgTitle.textContent = content.trajectories.detail_title;
+  els.dlgClose.textContent = content.trajectories.close;
+  els.selectLabel.textContent = content.trajectories.uploader.select_label;
+
+  runOptions = runs
+    .map((run) => ({
+      id: run.runId,
+      path: run.runDir,
+      rank: run.rank,
+      model: run.agent ?? run.model ?? "Unknown",
+      score: run.score,
+      capPerSignature: run.capPerSignature,
+      windowMs: run.windowMs,
+      notes: run.notes ?? null,
+    }))
+    .filter((opt) => opt.id && opt.path);
+
+  if (!runOptions.length) {
+    els.select.innerHTML =
+      '<option value="">No leaderboard runs found</option>';
+  } else {
+    els.select.innerHTML = runOptions
+      .map(
+        (opt) =>
+          `<option value="${opt.id}">#${opt.rank ?? "?"} · ${opt.model}</option>`,
+      )
+      .join("");
+  }
+
+  els.loadSample.addEventListener("click", async () => {
+    const opt = runOptions.find((o) => o.id === els.select.value);
+    if (opt) {
+      await loadFromHosted(opt);
+      window.location.hash = opt.id;
+    }
+  });
+
+  els.pickRun.addEventListener("change", async (e) => {
+    const files = Array.from(e.target.files || []);
+    const map = Object.fromEntries(
+      files.map((f) => [f.webkitRelativePath.split("/").pop(), f]),
+    );
+    if (!map["eval_per_action.jsonl"] || !map["per_action.jsonl"]) {
+      alert(
+        content.trajectories.errors?.missing_files || "Missing required files.",
+      );
+      return;
+    }
+    const evalText = await readFileAsText(map["eval_per_action.jsonl"]);
+    const actionText = await readFileAsText(map["per_action.jsonl"]);
+    const score = map["eval_score.json"]
+      ? JSON.parse(await readFileAsText(map["eval_score.json"]))
+      : null;
+    const meta = map["run_meta.json"]
+      ? JSON.parse(await readFileAsText(map["run_meta.json"]))
+      : null;
+    render(null, evalText, actionText, { score, meta });
+    window.location.hash = "";
+  });
+
+  const initialHash = window.location.hash.replace("#", "");
+  const initialOption =
+    runOptions.find((opt) => opt.id === initialHash) || runOptions[0] || null;
+  if (initialOption) {
+    els.select.value = initialOption.id;
+    await loadFromHosted(initialOption);
+    window.location.hash = initialOption.id;
+  }
+}
+
+async function loadFromHosted(option) {
+  if (!option) return;
+  const base = option.path.replace(/\/$/, "");
+  const [evalText, actionText] = await Promise.all([
+    fetch(`${base}/eval_per_action.jsonl`).then((r) => r.text()),
+    fetch(`${base}/per_action.jsonl`).then((r) => r.text()),
+  ]);
+  const [score, meta] = await Promise.all([
+    fetchJSON(`${base}/eval_score.json`).catch(() =>
+      option.score
+        ? {
+            finalScore: option.score.final,
+            base: option.score.base,
+            bonus: option.score.bonus,
+            penalty: option.score.penalty,
+            capPerSignature: option.capPerSignature,
+            windowMs: option.windowMs,
+          }
+        : null,
+    ),
+    fetchJSON(`${base}/run_meta.json`).catch(() => null),
+  ]);
+  render(option, evalText, actionText, { score, meta });
+}
+
+function render(option, evalText, actionText, extra = {}) {
+  const summaries = parseJSONL(evalText);
+  const actions = parseJSONL(actionText);
+  const rawByStep = new Map(
+    actions.map((entry) => [entry.stepIdx ?? entry.step_idx, entry]),
+  );
+  const { score, meta } = extra;
+
+  if (option || score || meta) {
+    const parts = [];
+    if (option) {
+      parts.push(`#${option.rank ?? "?"} · ${option.model}`);
+    } else {
+      parts.push("Uploaded run");
+    }
+    if (score) {
+      if (score.final != null || score.finalScore != null) {
+        const final = score.final != null ? score.final : score.finalScore;
+        parts.push(`Final ${fmtScore(final)}`);
+      }
+      if (score.base != null) parts.push(`Base ${fmtScore(score.base)}`);
+      if (score.bonus != null) parts.push(`Bonus ${fmtScore(score.bonus)}`);
+      if (score.penalty != null)
+        parts.push(`Penalty ${fmtScore(score.penalty)}`);
+      if (score.capPerSignature != null || score.cap_per_signature != null) {
+        const cap =
+          score.capPerSignature != null
+            ? score.capPerSignature
+            : score.cap_per_signature;
+        parts.push(`Cap per signature ${cap}`);
+      }
+      if (score.windowMs != null || score.window_ms != null) {
+        const windowMs =
+          score.windowMs != null ? score.windowMs : score.window_ms;
+        parts.push(`Window ${windowMs} ms`);
+      }
+    } else {
+      if (option?.capPerSignature)
+        parts.push(`Cap per signature ${option.capPerSignature}`);
+      if (option?.windowMs) parts.push(`Window ${option.windowMs} ms`);
+    }
+    if (meta?.benchVersion) parts.push(`Bench ${meta.benchVersion}`);
+    if (meta?.builderCode) parts.push(`Builder ${meta.builderCode}`);
+    if (option?.notes) parts.push(option.notes);
+    els.meta.innerHTML = parts.join(" · ");
+    els.runInfo.classList.remove("hidden");
+  } else {
+    els.runInfo.classList.add("hidden");
+  }
+
+  els.txBody.innerHTML = summaries
+    .map((summary) => {
+      const step = summary.stepIdx ?? summary.step_idx;
+      const signatures = summary.signatures || [];
+      const domainNames = Array.from(
+        new Set(signatures.map(signatureDomain).filter(Boolean)),
+      );
+      const domainsHtml = domainNames.length
+        ? domainNames.map((name) => domainPill(name)).join(" ")
+        : "-";
+      const sigHtml = signatures.length
+        ? signatures.map((sig) => `<code>${sig}</code>`).join("<br/>")
+        : "-";
+      const windowKey = summary.windowKeyMs ?? summary.window_key_ms;
+      const ignored = summary.ignored ? "Yes" : "No";
+      const reason = summary.reason || "-";
+      return `
+      <tr class="border-b last:border-0 align-top">
+        <td class="px-3 py-2 font-mono text-xs">${step}</td>
+        <td class="px-3 py-2">${summary.action || "-"}</td>
+        <td class="px-3 py-2 text-xs leading-5">${domainsHtml}</td>
+        <td class="px-3 py-2 text-xs leading-5">${sigHtml}</td>
+        <td class="px-3 py-2 text-right">${windowKey ?? "-"}</td>
+        <td class="px-3 py-2">${ignored}</td>
+        <td class="px-3 py-2 text-xs">${reason}</td>
+        <td class="px-3 py-2 text-right">
+          <button data-step="${step}" class="text-xs px-2 py-1 rounded border">Detail</button>
+        </td>
+      </tr>
+    `;
+    })
+    .join("");
+
+  els.txBody.querySelectorAll("button[data-step]").forEach((btn) => {
+    btn.addEventListener("click", () => {
+      const step = Number(btn.getAttribute("data-step"));
+      const summary = summaries.find((s) => (s.stepIdx ?? s.step_idx) === step);
+      const raw = rawByStep.get(step);
+      const detail = {
+        summary,
+        raw,
+        score,
+        meta,
+        leaderboard: option
+          ? {
+              id: option.id,
+              rank: option.rank,
+              model: option.model,
+            }
+          : null,
+      };
+      els.dlgContent.textContent = JSON.stringify(detail, null, 2);
+      els.dlg.showModal();
+    });
+  });
+}
+
+main().catch((err) => console.error(err));
@@ -0,0 +1,68 @@
+// Powers the HyperliquidBench frontend view for trading-competence trajectories and scores.
+export async function fetchJSON(url) {
+  const resp = await fetch(url);
+  if (!resp.ok) throw new Error(`Failed to fetch ${url}: ${resp.status}`);
+  return resp.json();
+}
+
+export function readFileAsText(file) {
+  return new Promise((resolve, reject) => {
+    const fr = new FileReader();
+    fr.onerror = () => reject(fr.error);
+    fr.onload = () => resolve(String(fr.result || ""));
+    fr.readAsText(file);
+  });
+}
+
+export function parseJSONL(text) {
+  return text
+    .split(/\r?\n/)
+    .filter(Boolean)
+    .map((line, i) => {
+      try {
+        return JSON.parse(line);
+      } catch (e) {
+        console.warn("Bad JSONL line", i + 1, e);
+        return null;
+      }
+    })
+    .filter(Boolean);
+}
+
+export function fmtScore(n) {
+  const x = Number(n || 0);
+  return (Math.round(x * 100) / 100).toFixed(2);
+}
+
+export function domainPill(name) {
+  const key = String(name || "-").toLowerCase();
+  const cls =
+    key === "perp"
+      ? "tag-perp"
+      : key === "account"
+        ? "tag-account"
+        : key === "risk"
+          ? "tag-risk"
+          : "tag-generic";
+  return `<span class="tag ${cls}">${name}</span>`;
+}
+
+export function signatureDomain(signature) {
+  if (!signature) return null;
+  return String(signature).split(".")[0] || null;
+}
+
+const HIAN_TAGS = {
+  PASS: "tag-pass",
+  PASS_WITH_WARNINGS: "tag-warning",
+  PARTIAL: "tag-partial",
+  FAIL: "tag-fail",
+};
+
+export function hianBadge(result) {
+  if (!result) return "-";
+  const key = String(result).toUpperCase();
+  const cls = HIAN_TAGS[key] || "tag-warning";
+  const label = key.replace(/_/g, " ");
+  return `<span class="tag ${cls}">${label}</span>`;
+}
@@ -0,0 +1,86 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width,initial-scale=1" />
+  <title>HyperLiquidBench — Operational Coverage</title>
+  <link rel="stylesheet" href="assets/style.css" />
+  <script src="https://cdn.tailwindcss.com"></script>
+  <script defer src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+  <script type="module" defer src="assets/app.js"></script>
+</head>
+<body class="bg-slate-50 text-slate-900 selection:bg-cyan-200">
+  <header class="border-b bg-white">
+    <div class="max-w-6xl mx-auto px-4 py-3 flex items-center justify-between">
+      <a id="site-title" href="./" class="font-semibold tracking-tight">HyperLiquidBench</a>
+      <nav class="text-sm">
+        <a id="nav-home" class="hover:underline mr-4" href="./">Overview</a>
+        <a id="nav-trajectories" class="hover:underline mr-4" href="./trajectories.html">Action Logs</a>
+        <a id="nav-breakdown" class="hover:underline" href="./lc.html">Domain Breakdown</a>
+      </nav>
+    </div>
+  </header>
+
+  <section class="bg-gradient-to-b from-white to-slate-50">
+    <div class="max-w-6xl mx-auto px-4 py-12">
+      <h1 id="hero-title" class="text-3xl md:text-4xl font-bold">Loading…</h1>
+      <p id="hero-subtitle" class="mt-3 max-w-2xl text-slate-600"></p>
+
+      <div id="pillars" class="mt-6 grid md:grid-cols-3 gap-4"></div>
+    </div>
+  </section>
+
+  <section class="max-w-6xl mx-auto px-4 py-8">
+    <div class="flex items-center justify-between flex-wrap gap-3">
+      <h2 id="scoreboard-title" class="text-xl font-semibold">Leaderboard</h2>
+      <div class="text-sm text-slate-500">
+        <span id="scoreboard-source">Datasets:</span>
+        <a id="scoreboard-link" class="underline" href="./trajectories.html">Inspect trajectories</a>
+      </div>
+    </div>
+    <div class="mt-4 overflow-x-auto">
+      <table class="w-full text-sm bg-white border rounded-lg shadow-sm min-w-[720px]">
+        <thead class="bg-slate-100 text-slate-700">
+          <tr>
+            <th class="px-3 py-2 text-left">Rank</th>
+            <th class="px-3 py-2 text-left">Agent</th>
+            <th class="px-3 py-2 text-right">Final</th>
+            <th class="px-3 py-2 text-right">Base</th>
+            <th class="px-3 py-2 text-right">Bonus</th>
+            <th class="px-3 py-2 text-right">Penalty</th>
+            <th class="px-3 py-2 text-right">Unique sigs</th>
+            <th class="px-3 py-2 text-left">Domains</th>
+            <th class="px-3 py-2 text-left">Run</th>
+          </tr>
+        </thead>
+        <tbody id="lb-body"></tbody>
+      </table>
+    </div>
+
+    <div class="mt-8 grid md:grid-cols-2 gap-6">
+      <div class="p-4 bg-white border rounded-lg shadow-sm">
+        <h3 id="chart-score-title" class="font-semibold">Final vs Base Score</h3>
+        <canvas id="scoreChart" class="mt-3"></canvas>
+      </div>
+      <div class="p-4 bg-white border rounded-lg shadow-sm">
+        <h3 id="chart-domain-title" class="font-semibold">Composition Bonus</h3>
+        <canvas id="domainChart" class="mt-3"></canvas>
+      </div>
+    </div>
+  </section>
+
+  <section class="bg-white border-t">
+    <div class="max-w-6xl mx-auto px-4 py-10">
+      <h2 id="howto-title" class="text-xl font-semibold">Keep the snapshot fresh</h2>
+      <ol id="howto-steps" class="list-decimal ml-6 mt-3 text-sm text-slate-700 space-y-2"></ol>
+      <p id="howto-foot" class="text-xs text-slate-500 mt-4"></p>
+    </div>
+  </section>
+
+  <footer class="border-t bg-slate-50">
+    <div id="footer-text" class="max-w-6xl mx-auto px-4 py-6 text-xs text-slate-500">
+      HyperLiquidBench
+    </div>
+  </footer>
+</body>
+</html>
@@ -0,0 +1,74 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width,initial-scale=1" />
+  <title>HyperLiquidBench — Domain Breakdown</title>
+  <link rel="stylesheet" href="assets/style.css" />
+  <script src="https://cdn.tailwindcss.com"></script>
+  <script type="module" defer src="assets/lc.js"></script>
+</head>
+<body class="bg-slate-50 text-slate-900">
+  <header class="border-b bg-white">
+    <div class="max-w-6xl mx-auto px-4 py-3 flex items-center justify-between">
+      <a id="site-title" href="./" class="font-semibold tracking-tight">HyperLiquidBench</a>
+      <nav class="text-sm">
+        <a id="nav-home" class="hover:underline mr-4" href="./">Overview</a>
+        <a id="nav-trajectories" class="hover:underline mr-4" href="./trajectories.html">Action Logs</a>
+        <a id="nav-breakdown" class="hover:underline" href="./lc.html">Domain Breakdown</a>
+      </nav>
+    </div>
+  </header>
+
+  <main class="max-w-6xl mx-auto px-4 py-8 space-y-8">
+    <header>
+      <h1 id="lc-title" class="text-2xl font-bold">Domain & signature breakdown</h1>
+      <p id="lc-subtitle" class="mt-2 text-sm text-slate-600"></p>
+    </header>
+
+    <section class="grid gap-4 md:grid-cols-2">
+      <article class="bg-white border rounded-lg shadow-sm p-4 flex flex-col gap-3">
+        <div class="text-xs uppercase tracking-wide text-slate-500" id="lc-knowledge-label">Domain policy</div>
+        <div class="text-sm font-semibold text-slate-800" id="lc-knowledge-path"></div>
+        <p id="lc-knowledge-desc" class="text-xs text-slate-500"></p>
+        <a id="lc-knowledge-cta" href="#" class="px-3 py-2 rounded bg-slate-900 text-white text-sm w-fit" target="_blank" rel="noreferrer">Open policy</a>
+      </article>
+      <article class="bg-white border rounded-lg shadow-sm p-4 flex flex-col gap-3">
+        <div class="text-xs uppercase tracking-wide text-slate-500" id="lc-hian-label">Unique signatures</div>
+        <div class="text-sm font-semibold text-slate-800" id="lc-hian-path"></div>
+        <p id="lc-hian-desc" class="text-xs text-slate-500"></p>
+        <a id="lc-hian-cta" href="#" class="px-3 py-2 rounded bg-slate-900 text-white text-sm w-fit" target="_blank" rel="noreferrer">Open manifest</a>
+      </article>
+    </section>
+
+    <section class="bg-white border rounded-lg shadow-sm">
+      <div class="p-4 border-b">
+        <h2 class="font-semibold text-lg" id="lc-table-caption">Knowledge & HiaN ranking</h2>
+      </div>
+      <div class="overflow-x-auto">
+        <table class="w-full text-sm min-w-[780px]">
+          <thead class="bg-slate-100 text-slate-700">
+            <tr>
+              <th class="px-3 py-2 text-left" id="lc-col-rank">Rank</th>
+              <th class="px-3 py-2 text-left" id="lc-col-model">Agent</th>
+              <th class="px-3 py-2 text-right" id="lc-col-final">Final</th>
+              <th class="px-3 py-2 text-right" id="lc-col-base">Base</th>
+              <th class="px-3 py-2 text-right" id="lc-col-bonus">Bonus</th>
+              <th class="px-3 py-2 text-right" id="lc-col-penalty">Penalty</th>
+              <th class="px-3 py-2 text-right" id="lc-col-unique">Unique sigs</th>
+              <th class="px-3 py-2 text-left" id="lc-col-domains">Domain spread</th>
+              <th class="px-3 py-2 text-left" id="lc-col-runs">Artifacts</th>
+            </tr>
+          </thead>
+          <tbody id="lc-body"></tbody>
+        </table>
+      </div>
+    </section>
+
+    <section class="bg-white border rounded-lg shadow-sm p-4">
+      <h2 id="lc-notes-title" class="font-semibold text-lg mb-2">Observations</h2>
+      <ul id="lc-notes" class="list-disc ml-5 text-sm text-slate-700 space-y-1"></ul>
+    </section>
+  </main>
+</body>
+</html>
@@ -0,0 +1,80 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width,initial-scale=1" />
+  <title>HyperLiquidBench — Action Logs</title>
+  <link rel="stylesheet" href="assets/style.css" />
+  <script src="https://cdn.tailwindcss.com"></script>
+  <script type="module" defer src="assets/trajectories.js"></script>
+</head>
+<body class="bg-slate-50 text-slate-900">
+  <header class="border-b bg-white">
+    <div class="max-w-6xl mx-auto px-4 py-3 flex items-center justify-between">
+      <a id="site-title" href="./" class="font-semibold tracking-tight">HyperLiquidBench</a>
+      <nav class="text-sm">
+        <a id="nav-home" class="hover:underline mr-4" href="./">Overview</a>
+        <a id="nav-trajectories" class="hover:underline mr-4" href="./trajectories.html">Action Logs</a>
+        <a id="nav-breakdown" class="hover:underline" href="./lc.html">Domain Breakdown</a>
+      </nav>
+    </div>
+  </header>
+
+  <main class="max-w-6xl mx-auto px-4 py-8">
+    <h1 id="traj-title" class="text-2xl font-bold">Action logs</h1>
+    <p id="traj-subtitle" class="mt-2 text-sm text-slate-600"></p>
+
+    <div class="mt-4 p-4 bg-white border rounded-lg shadow-sm">
+      <div class="flex flex-col lg:flex-row gap-3 lg:items-center">
+        <label class="text-sm flex items-center gap-2">
+          <span id="traj-select-label" class="text-slate-700"></span>
+          <select id="traj-select" class="border rounded px-2 py-1 text-sm bg-white"></select>
+        </label>
+        <button id="load-sample" class="px-3 py-2 rounded bg-slate-900 text-white text-sm">Load selected run</button>
+        <div class="text-xs text-slate-500" id="traj-or">or</div>
+        <label class="text-sm">
+          <input id="pick-run" type="file" webkitdirectory directory multiple class="hidden" />
+          <span id="traj-pick-label" class="px-3 py-2 rounded border bg-white inline-block cursor-pointer">Upload run folder</span>
+        </label>
+        <div class="text-xs text-slate-500" id="traj-note">Required: eval_per_tx.jsonl, per_tx.jsonl (optional: eval_score.json)</div>
+      </div>
+    </div>
+
+    <div id="run-info" class="mt-6 hidden">
+      <h2 class="font-semibold text-lg">Run</h2>
+      <div class="text-sm text-slate-600" id="meta"></div>
+    </div>
+
+    <div class="mt-4 overflow-x-auto">
+      <table class="w-full text-sm bg-white border rounded-lg shadow-sm">
+        <thead class="bg-slate-100 text-slate-700">
+          <tr>
+            <th class="px-3 py-2 text-left">Step</th>
+            <th class="px-3 py-2 text-left">Action</th>
+            <th class="px-3 py-2 text-left">Domains</th>
+            <th class="px-3 py-2 text-left">Signatures</th>
+            <th class="px-3 py-2 text-right">Window (ms)</th>
+            <th class="px-3 py-2 text-left">Ignored</th>
+            <th class="px-3 py-2 text-left">Reason</th>
+            <th class="px-3 py-2"></th>
+          </tr>
+        </thead>
+        <tbody id="tx-body"></tbody>
+      </table>
+    </div>
+
+    <dialog id="dlg" class="w-full max-w-3xl rounded-lg border p-0">
+      <div class="p-4 border-b flex justify-between items-center">
+        <div id="traj-detail-title" class="font-semibold">Action detail</div>
+        <form method="dialog"><button class="text-slate-500 hover:text-slate-800">✕</button></form>
+      </div>
+      <div class="p-4 text-sm">
+        <pre id="dlg-content" class="whitespace-pre-wrap text-xs bg-slate-50 p-3 rounded border overflow-x-auto"></pre>
+      </div>
+      <div class="p-3 border-t text-right">
+        <form method="dialog"><button id="traj-close" class="px-3 py-1.5 rounded bg-slate-900 text-white text-sm">Close</button></form>
+      </div>
+    </dialog>
+  </main>
+</body>
+</html>
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+  cat <<'USAGE'
+Usage: scripts/run_cov.sh [PLAN_SPEC] [-- <hl-runner args>]
+
+Environment variables:
+  OUT_DIR   Override run output directory (default: runs/<timestamp>)
+  NETWORK   Override --network flag (default: testnet)
+
+Examples:
+  scripts/run_cov.sh dataset/tasks/hl_perp_basic_01.jsonl:1
+  OUT_DIR=runs/demo NETWORK=mainnet scripts/run_cov.sh dataset/tasks/plan.json -- --builder-code demo123
+USAGE
+  exit 0
+fi
+
+PLAN_SPEC=${1:-dataset/tasks/hl_perp_basic_01.jsonl:1}
+if [[ $# -gt 0 ]]; then
+  shift
+fi
+if [[ "${1:-}" == "--" ]]; then
+  shift
+fi
+
+OUT_DIR=${OUT_DIR:-"runs/$(date +%Y%m%d-%H%M%S)"}
+NETWORK=${NETWORK:-testnet}
+DOMAINS_FILE=${DOMAINS_FILE:-dataset/domains-hl.yaml}
+EVAL_ARGS=${EVAL_ARGS:-}
+
+cargo run -p hl-runner -- \
+  --plan "$PLAN_SPEC" \
+  --out "$OUT_DIR" \
+  --network "$NETWORK" \
+  "$@"
+
+cargo run -p hl-evaluator -- \
+  --input "$OUT_DIR/per_action.jsonl" \
+  --domains "$DOMAINS_FILE" \
+  --out-dir "$OUT_DIR" \
+  $EVAL_ARGS
+
+echo "final score written to $OUT_DIR/eval_score.json"
+cat "$OUT_DIR/eval_score.json"
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'USAGE'
+Usage: scripts/run_cov_matrix.sh [TASK ...] [-- <hl-runner args>]
+
+Runs `scripts/run_cov.sh` across every plan line in the supplied task files.
+If no TASK arguments are provided, all files under dataset/tasks/*.jsonl are
+used. Arguments after `--` are forwarded verbatim to `scripts/run_cov.sh` (and
+therefore `hl-runner`).
+
+Environment variables honoured (same as run_cov.sh):
+  OUT_DIR        base directory for run outputs (default: runs/<timestamp>)
+  NETWORK        network flag passed to hl-runner (default: testnet)
+  DOMAINS_FILE   domains config path (default: dataset/domains-hl.yaml)
+USAGE
+}
+
+TASKS=()
+FORWARD=()
+PARSE_FORWARD=false
+
+for arg in "$@"; do
+  if ${PARSE_FORWARD}; then
+    FORWARD+=("$arg")
+    continue
+  fi
+  case "$arg" in
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    --)
+      PARSE_FORWARD=true
+      ;;
+    *)
+      TASKS+=("$arg")
+      ;;
+  esac
+done
+
+if [[ ${#TASKS[@]} -eq 0 ]]; then
+  mapfile -t TASKS < <(ls dataset/tasks/*.jsonl 2>/dev/null | sort)
+fi
+
+if [[ ${#TASKS[@]} -eq 0 ]]; then
+  echo "no task files found" >&2
+  exit 1
+fi
+
+for task in "${TASKS[@]}"; do
+  if [[ ! -f "$task" ]]; then
+    echo "skipping missing task file: $task" >&2
+    continue
+  fi
+  total=$(rg -c '.' "$task" 2>/dev/null || true)
+  if [[ -z "$total" || "$total" -eq 0 ]]; then
+    echo "no scenarios in $task" >&2
+    continue
+  fi
+  for ((idx=1; idx<=total; idx++)); do
+    echo "==> Running $task:$idx"
+    if [[ ${#FORWARD[@]} -gt 0 ]]; then
+      scripts/run_cov.sh "$task:$idx" -- "${FORWARD[@]}"
+    else
+      scripts/run_cov.sh "$task:$idx"
+    fi
+  done
+done
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+  cat <<'USAGE'
+Usage: scripts/run_hian.sh [CASE_DIR] [-- <hl-runner args>]
+
+Environment variables:
+  OUT_DIR   Override run output directory (default: runs/hian-<timestamp>)
+
+Examples:
+  scripts/run_hian.sh
+  OUT_DIR=runs/hian-demo scripts/run_hian.sh dataset/hian/case_128k -- --effect-timeout-ms 100
+USAGE
+  exit 0
+fi
+
+CASE_DIR=${1:-dataset/hian/case_128k}
+if [[ $# -gt 0 ]]; then
+  shift
+fi
+if [[ "${1:-}" == "--" ]]; then
+  shift
+fi
+
+OUT_DIR=${OUT_DIR:-"runs/hian-$(date +%Y%m%d-%H%M%S)"}
+PLAN_FILE="$OUT_DIR/hian_plan.json"
+mkdir -p "$OUT_DIR"
+
+cat > "$PLAN_FILE" <<'JSON'
+{
+  "steps": [
+    { "usd_class_transfer": { "toPerp": true, "usdc": 7.5 } },
+    {
+      "perp_orders": {
+        "orders": [
+          {
+            "coin": "ETH",
+            "tif": "ALO",
+            "side": "buy",
+            "sz": 0.01,
+            "reduceOnly": false,
+            "px": "mid-1%"
+          }
+        ]
+      }
+    }
+  ]
+}
+JSON
+
+cargo run -p hl-runner -- \
+  --plan "$PLAN_FILE" \
+  --out "$OUT_DIR" \
+  --network local \
+  --demo \
+  "$@"
+
+cargo run -p hl-evaluator -- hian \
+  --ground "$CASE_DIR/ground_truth.json" \
+  --per-action "$OUT_DIR/per_action.jsonl" \
+  --ws-stream "$OUT_DIR/ws_stream.jsonl" \
+  --out-dir "$OUT_DIR"
+
+cat "$OUT_DIR/eval_hian.json"
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+RUN_DIR=${1:-}
+if [[ -z "$RUN_DIR" ]]; then
+  if [[ ! -d runs ]]; then
+    echo "no runs/ directory found" >&2
+    exit 1
+  fi
+  RUN_DIR=$(ls -dt runs/* 2>/dev/null | head -n1 || true)
+  if [[ -z "$RUN_DIR" ]]; then
+    echo "no run directories found under runs/" >&2
+    exit 1
+  fi
+fi
+
+if [[ ! -f "$RUN_DIR/ws_stream.jsonl" ]]; then
+  echo "missing ws_stream.jsonl in $RUN_DIR" >&2
+  exit 1
+fi
+
+LINES=${LINES:-40}
+tail -n "$LINES" "$RUN_DIR/ws_stream.jsonl"
@@ -0,0 +1,33 @@
+from __future__ import annotations
+
+from benchmarks.HyperliquidBench.__main__ import (
+    count_scenarios,
+    expand_scenarios,
+    validate_scenarios,
+)
+from benchmarks.HyperliquidBench.eliza_agent import make_coverage_scenario
+
+
+def test_hyperliquid_scenario_expansion_adds_ten_edge_variants() -> None:
+    base = [make_coverage_scenario(allowed_coins=["ETH", "BTC"], max_steps=3)]
+
+    expanded = expand_scenarios(base)
+
+    assert len(expanded) == 11
+    assert expanded[0].scenario_id == "coverage_smoke"
+    assert expanded[1].scenario_id == "coverage_smoke__edge_01"
+    assert "Edge condition:" in expanded[1].description
+    assert expanded[2].allowed_coins == ["BTC", "ETH"]
+
+
+def test_hyperliquid_scenario_count_and_validate() -> None:
+    base = [make_coverage_scenario(allowed_coins=["ETH", "BTC"], max_steps=3)]
+
+    validate_scenarios(base, include_edge_scenarios=True)
+
+    assert count_scenarios(base, include_edge_scenarios=True) == {
+        "base": 1,
+        "edge": 10,
+        "edge_multiplier": 10,
+        "total": 11,
+    }
@@ -0,0 +1,262 @@
+"""
+HyperliquidBench Type Definitions
+
+Defines all data classes and enums used by the Eliza agent wrapper
+for the HyperliquidBench benchmark.  These types mirror the Rust plan
+schema from ``crates/hl-common/src/plan.rs`` so that the Python agent
+can generate plans in the exact JSON format the Rust runner expects.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from pathlib import Path
+from typing import Optional
+
+
+# ── Plan primitives ─────────────────────────────────────────────────
+
+class OrderSide(str, Enum):
+    """Order side – maps to the Rust ``OrderSide`` enum."""
+    BUY = "buy"
+    SELL = "sell"
+
+
+class PerpTif(str, Enum):
+    """Time-in-force – maps to the Rust ``PerpTif`` enum."""
+    ALO = "ALO"
+    GTC = "GTC"
+    IOC = "IOC"
+
+
+class StepKind(str, Enum):
+    """Discriminator for action-step variants."""
+    PERP_ORDERS = "perp_orders"
+    CANCEL_LAST = "cancel_last"
+    CANCEL_OIDS = "cancel_oids"
+    CANCEL_ALL = "cancel_all"
+    USD_CLASS_TRANSFER = "usd_class_transfer"
+    SET_LEVERAGE = "set_leverage"
+    SLEEP_MS = "sleep_ms"
+
+
+# ── Individual step payloads ────────────────────────────────────────
+
+@dataclass
+class PerpOrder:
+    """Single perpetual order inside a ``PerpOrdersStep``."""
+    coin: str
+    side: OrderSide
+    sz: float
+    px: float | str  # absolute ``float`` or ``"mid±X%"`` string
+    tif: PerpTif = PerpTif.GTC
+    reduce_only: bool = False
+    builder_code: Optional[str] = None
+    cloid: Optional[str] = None
+
+    def to_dict(self) -> dict[str, object]:
+        d: dict[str, object] = {
+            "coin": self.coin,
+            "side": self.side.value,
+            "sz": self.sz,
+            "px": self.px,
+            "tif": self.tif.value,
+            "reduceOnly": self.reduce_only,
+        }
+        if self.builder_code is not None:
+            d["builderCode"] = self.builder_code
+        if self.cloid is not None:
+            d["cloid"] = self.cloid
+        return d
+
+
+@dataclass
+class PerpOrdersStep:
+    """Place one or more perpetual orders."""
+    orders: list[PerpOrder]
+    builder_code: Optional[str] = None
+
+    def to_dict(self) -> dict[str, object]:
+        d: dict[str, object] = {"orders": [o.to_dict() for o in self.orders]}
+        if self.builder_code is not None:
+            d["builderCode"] = self.builder_code
+        return {"perp_orders": d}
+
+
+@dataclass
+class CancelLastStep:
+    """Cancel the most recently placed order."""
+    coin: Optional[str] = None
+
+    def to_dict(self) -> dict[str, object]:
+        inner: dict[str, object] = {}
+        if self.coin is not None:
+            inner["coin"] = self.coin
+        return {"cancel_last": inner}
+
+
+@dataclass
+class CancelOidsStep:
+    """Cancel specific order IDs."""
+    coin: str
+    oids: list[int]
+
+    def to_dict(self) -> dict[str, object]:
+        return {"cancel_oids": {"coin": self.coin, "oids": self.oids}}
+
+
+@dataclass
+class CancelAllStep:
+    """Cancel all resting orders (optionally for one coin)."""
+    coin: Optional[str] = None
+
+    def to_dict(self) -> dict[str, object]:
+        inner: dict[str, object] = {}
+        if self.coin is not None:
+            inner["coin"] = self.coin
+        return {"cancel_all": inner}
+
+
+@dataclass
+class UsdClassTransferStep:
+    """Transfer USDC between spot and perp wallets."""
+    to_perp: bool
+    usdc: float
+
+    def to_dict(self) -> dict[str, object]:
+        return {"usd_class_transfer": {"toPerp": self.to_perp, "usdc": self.usdc}}
+
+
+@dataclass
+class SetLeverageStep:
+    """Set leverage for a given coin."""
+    coin: str
+    leverage: int
+    cross: bool = False
+
+    def to_dict(self) -> dict[str, object]:
+        return {
+            "set_leverage": {
+                "coin": self.coin,
+                "leverage": self.leverage,
+                "cross": self.cross,
+            }
+        }
+
+
+@dataclass
+class SleepStep:
+    """Pause execution for a specified duration."""
+    duration_ms: int
+
+    def to_dict(self) -> dict[str, object]:
+        return {"sleep_ms": {"duration_ms": self.duration_ms}}
+
+
+# Union of all step types
+ActionStep = (
+    PerpOrdersStep
+    | CancelLastStep
+    | CancelOidsStep
+    | CancelAllStep
+    | UsdClassTransferStep
+    | SetLeverageStep
+    | SleepStep
+)
+
+
+@dataclass
+class Plan:
+    """A complete trading plan that the Rust runner can execute."""
+    steps: list[ActionStep]
+
+    def to_dict(self) -> dict[str, list[dict[str, object]]]:
+        return {"steps": [s.to_dict() for s in self.steps]}
+
+
+# ── Scenario / task types ───────────────────────────────────────────
+
+class ScenarioKind(str, Enum):
+    """Kind of benchmark scenario."""
+    COVERAGE = "coverage"
+    HIAN = "hian"
+    CUSTOM = "custom"
+
+
+@dataclass
+class TradingScenario:
+    """A single benchmark task / scenario that the agent must solve."""
+    scenario_id: str
+    kind: ScenarioKind
+    description: str
+    allowed_coins: list[str] = field(default_factory=lambda: ["ETH", "BTC"])
+    max_steps: int = 5
+    builder_code: Optional[str] = None
+    plan_spec: Optional[str] = None  # e.g. ``dataset/tasks/hl_perp_basic_01.jsonl:1``
+    hian_prompt_path: Optional[str] = None
+
+
+# ── Result types ────────────────────────────────────────────────────
+
+@dataclass
+class RunnerResult:
+    """Result from the ``hl-runner`` subprocess."""
+    success: bool
+    out_dir: str
+    run_meta_path: str
+    per_action_path: str
+    stdout: str
+    stderr: str
+    exit_code: int
+
+
+@dataclass
+class EvaluatorResult:
+    """Result from the ``hl-evaluator`` subprocess."""
+    success: bool
+    final_score: float
+    base: float
+    bonus: float
+    penalty: float
+    unique_signatures: list[str]
+    eval_score_path: str
+    stdout: str
+    stderr: str
+    exit_code: int
+
+
+@dataclass
+class BenchmarkResult:
+    """Aggregate result for one scenario."""
+    scenario_id: str
+    plan: Plan
+    runner: RunnerResult
+    evaluator: Optional[EvaluatorResult]
+    error_message: Optional[str] = None
+
+
+# ── Config ──────────────────────────────────────────────────────────
+
+@dataclass
+class HLBenchConfig:
+    """Configuration for the Eliza HyperliquidBench agent."""
+    # Paths (relative to the HyperliquidBench root)
+    bench_root: Path = field(default_factory=lambda: Path(__file__).resolve().parent)
+    dataset_dir: str = "dataset"
+    domains_file: str = "dataset/domains-hl.yaml"
+    runs_dir: str = "runs"
+
+    # Runner settings
+    demo_mode: bool = True
+    network: str = "testnet"
+    builder_code: Optional[str] = None
+    effect_timeout_ms: int = 2000
+
+    # LLM / model settings
+    model_name: str = "gpt-4o"
+    temperature: float = 0.2
+
+    # Agent settings
+    max_iterations: int = 3
+    verbose: bool = False
@@ -0,0 +1,142 @@
+# Meeting, Transcription, and Voice Benchmark Matrix
+
+Research matrix for issue #13352. This file records which external meeting,
+speech, and voice-assistant benchmarks are already covered by elizaOS, which
+ones need legal or infrastructure review, and where an adapter should land.
+
+Status is based on repo inspection plus public source pages on 2026-07-04. The
+source review checked the QMSum, MeetingBank, ELITR-Bench, VoiceCodeBench,
+VoiceBench, and Sierra 3-Bench primary pages directly; the remaining P1/P2 rows
+are conservative until their adapter PRs recheck license/access terms. Do not
+commit raw external data unless the allowed-use column explicitly permits a repo
+fixture and the source license is rechecked in the adapter PR.
+
+## Existing elizaOS Coverage
+
+| Surface | Existing coverage | Gap this matrix should drive |
+| --- | --- | --- |
+| `packages/scenario-runner` | Real `AgentRuntime` scenarios, deterministic/live lanes, JSON reports, native JSONL export. | Meeting transcript and voice task adapters that produce scenario definitions or scenario metadata from external datasets. |
+| `packages/benchmarks/lifeops-bench` | Multi-turn tool-use benchmark with deterministic world-state scoring across Eliza/Hermes/OpenClaw/Cerebras adapters. | Knowledge-grounded and voice-interruption domains inspired by Sierra tau-Knowledge/tau-Voice without copying restricted data. |
+| `packages/benchmarks/meeting-transcription-proof` | Planned adapter-contract scaffold from #13378 / #13359 for transcript, diarization, speaker identity, consent, retention, and meeting note metrics; not pre-existing `develop` coverage at the time of the matrix. | Dataset adapters for QMSum, MeetingBank, ELITR-Bench, TCR, and controlled public meeting slices. |
+| `packages/benchmarks/voice-speaker-validation` | Speaker profile lifecycle, diarization, single-stream, and async identity checks. | Public speaker-count/diarization stress slices and trait-aware regression fixtures. |
+| `packages/benchmarks/registry` | Integrated `voicebench`, `voicebench_quality`, `mmau`, and scoring gates that reject mock results for publishable runs. | New registry entries for VoiceCodeBench, QMSum/MeetingBank smoke slices, and tau-style knowledge/voice methodology. |
+| `packages/training` | Eliza-1 benchmark matrix and voice gates for ASR/TTS/runtime metrics. | Exact structured-token recovery, long-form ASR, dropped-frame/interruption robustness, and voice task pass-at-1 gates. |
+| `plugins/plugin-google` | Google Meet artifact import path with the current `GoogleMeetReport` export; canonical `elizaos.meeting_artifact.v1` is planned adapter/schema work rather than an existing plugin-google export. | Reference-based artifact grading: summary, quote grounding, action items, topic relevance, and privacy/retention checks. |
+
+## Benchmark Matrix
+
+Allowed-use values:
+
+- `repo-fixture`: small derived fixture can be committed after license check.
+- `downloaded-eval`: adapter downloads public data at run time; no raw data in repo.
+- `private-eval`: use only with credentials/access approval; do not redistribute.
+- `training-ok`: candidate training/fine-tuning data after contamination review.
+- `eval-only`: evaluate only; keep out of training.
+- `needs-legal`: license/access is unclear or restricted.
+- `do-not-use`: avoid until legal/product explicitly changes the decision.
+
+| Source | Task/modality | Status in elizaOS | License/access posture | Allowed use | Harness target | Priority | Minimal starting slice |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| MeetBench / MeetAll / MeetMaster: https://github.com/huyuelin/MeetBench-MeetAll.github.io | Meeting-agent QA, meeting summaries, factuality and structure judging; multilingual/multimodal meetings. | New; only indirectly overlaps with meeting-proof and scenario-runner. | Project notes describe MeetAll data as CC BY-NC 4.0 / non-commercial research only. | `needs-legal`, likely `eval-only`/`do-not-use` for commercial repo fixtures. | Scenario-runner live-only meeting-agent scenarios; no raw data in repo. | P2 | Metadata-only adapter proof plus one synthetic eliza-owned fixture that mirrors task shape. |
+| QMSum: https://github.com/Yale-LILY/QMSum | Query-focused meeting summarization over academic/product/committee meetings. | New; maps cleanly to meeting artifact QA. | GitHub repo is MIT; underlying source meeting corpora still need citation/license review per corpus. | `downloaded-eval`; small query/summary fixture only after corpus-level review. | `meeting-transcription-proof` and scenario-runner transcript QA scenarios. | P0 | 10 query-summary pairs with transcript excerpts, exact reference spans, and judge fallback. |
+| MeetingBank / MeetingBank-utils: https://meetingbank.github.io/ and https://github.com/YebowenHu/MeetingBank-utils | City council meeting summarization with transcripts, videos, agendas, and minutes. | New; target adapter scaffold is #13378 / #13359, not existing dataset coverage on `develop`. | Public meeting data; site and Zenodo publish dataset, but city/source terms and video reuse need review. | `downloaded-eval`; repo fixture should be minimal metadata/excerpt only. | Meeting artifact generation and long-context summarization benchmark. | P0 | 5 public meeting sections with transcript excerpt, agenda, reference minutes, and action-item extraction. |
+| ELITR-Bench: https://github.com/utter-project/ELITR-Bench | Long-context meeting transcript QA with ground-truth answers and metadata. | New. | GitHub lists CC BY 4.0; verify upstream ELITR meeting-data terms before raw fixture use. | `downloaded-eval`; `repo-fixture` only for tiny CC-compatible examples. | Scenario-runner long-context QA and retrieval/quote-grounding checks. | P1 | 20 QA rows with answer/reference metadata and no audio. |
+| Topic-Conversation Relevance (TCR): https://github.com/microsoft/topic_conversation | Topic relevance over 1,500 meetings and 15k+ topics. | New. | Paper/repo indicate CC BY 4.0 data, but source-meeting provenance needs review. | `downloaded-eval`; possible small `repo-fixture` after provenance check. | Scenario-runner topic tracking and meeting artifact topic relevance. | P1 | 50 topic/transcript-window pairs with exact-match relevance metrics. |
+| AMI Meeting Corpus: https://groups.inf.ed.ac.uk/ami/corpus/ | Meeting audio/video/transcripts, diarization, summaries. | Partially covered by generic meeting proof metrics, not dataset-specific. | Research corpus with separate license/access terms. | `needs-legal`, likely `private-eval` or `downloaded-eval`. | Meeting transcription, diarization, speaker attribution, summarization. | P1 | License-gated manifest adapter; no committed media. |
+| ICSI Meeting Corpus: https://groups.inf.ed.ac.uk/ami/icsi/ | Meeting speech, transcripts, speaker labels. | Partially covered conceptually only. | Research corpus with access restrictions and citation requirements. | `needs-legal`, likely `private-eval`. | Speaker validation and meeting transcription proof. | P1 | Manifest-only adapter plus documented acquisition step. |
+| CHiME-6: https://chimechallenge.github.io/chime6/ | Dinner-party multi-speaker ASR/diarization under real noise. | Partially covered by voice-speaker-validation abstractions, not dataset. | Challenge data requires agreement and download process. | `private-eval`; no repo data. | ASR/diarization stress lane and Eliza-1 ASR gates. | P1 | 1-hour manifest slice with WER/DER/JER metrics. |
+| SAMSum: https://huggingface.co/datasets/Samsung/samsum | Dialogue summarization, not meeting-specific. | New; lower-fidelity baseline. | HF endpoint returned 401 during the 2026-07-04 link check; treat access and license as gated until rechecked. | `needs-legal`, likely `private-eval`; no repo fixture unless access terms change. | Baseline summarization scorer only, clearly marked non-meeting. | P2 | Metadata-only until access and license are verified. |
+| DialogSum: https://huggingface.co/datasets/knkarthick/dialogsum | Dialogue summarization, not meeting-specific. | New; lower-fidelity baseline. | HF dataset; license must be checked before fixture. | `downloaded-eval`. | Baseline summarization scorer. | P2 | 100 short dialogue summaries. |
+| MediaSum: https://github.com/zcgzcgzcg1/MediaSum | Interview/media transcript summarization. | New. | Dataset terms need review. | `needs-legal`; likely `downloaded-eval`. | Long transcript summarization baseline. | P2 | Metadata-only until license verified. |
+| LongSpeech: https://arxiv.org/html/2601.13539v1 | Long-duration speech benchmark for ASR, ST, summarization, language detection, speaker count, QA, temporal localization, and emotion. | New. | Paper describes dataset; public code/data availability and license need review. | `needs-legal`, likely `downloaded-eval`. | Eliza-1 long-form ASR and audio QA gates. | P2 | 10-minute public subset if released with redistributable terms. |
+| Sierra mu-Bench: https://huggingface.co/datasets/sierra-research/mu-bench | Multilingual transcription from real customer-service calls. | New. | Restricted access, non-commercial/restricted terms and deletion/redistribution limits. | `private-eval`, `eval-only`; no training or repo fixtures. | Private ASR regression lane. | P2 | Access-gated manifest and local-only result schema. |
+| VoiceCodeBench: https://huggingface.co/datasets/besimple-ai/voice-code-bench | Exact structured-token recovery for workplace speech: paths, URLs, numbers, code-like entities. | New. | HF page indicates MIT and small human-recorded WAV set. | `downloaded-eval`; candidate `repo-fixture` for a tiny sample if license confirmed. | Eliza-1 ASR gate and meeting-transcription-proof exact-token metric. | P0 | 30 clips covering URLs, file paths, IDs, and punctuation; report CTEM/TSR. |
+| Picovoice STT benchmark: https://github.com/Picovoice/speech-to-text-benchmark | WER/punctuation over LibriSpeech, TED-LIUM, Common Voice, MLS, VoxPopuli, FLEURS. | New harness; overlaps with ASR metrics. | Framework repo and component datasets have separate licenses. | `downloaded-eval`; dataset-specific gates. | Eliza-1 ASR comparison harness. | P1 | Common Voice or FLEURS 50-row smoke slice with checksums. |
+| LibriSpeech: https://www.openslr.org/12 | Read-speech ASR baseline. | Not directly integrated. | Widely used OpenSLR corpus; verify license before committing slices. | `downloaded-eval`; possible small fixture after review. | Eliza-1 ASR baseline. | P2 | 100 dev-clean utterances. |
+| TED-LIUM: https://www.openslr.org/51 | Lecture/talk ASR baseline. | Not directly integrated. | OpenSLR dataset with its own terms. | `downloaded-eval`. | Long-form ASR and punctuation baseline. | P2 | 30 short segments plus transcript refs. |
+| Common Voice: https://commonvoice.mozilla.org/datasets | Multilingual ASR. | Not directly integrated. | Mozilla Common Voice has versioned licenses; review selected release. | `downloaded-eval`; possible training after contamination review. | Eliza-1 multilingual ASR gates. | P1 | 50 clips per target language. |
+| MLS: https://www.openslr.org/94 | Multilingual read-speech ASR. | Not directly integrated. | OpenSLR dataset terms. | `downloaded-eval`. | Multilingual ASR. | P2 | 50 clips across high-priority locales. |
+| VoxPopuli: https://github.com/facebookresearch/voxpopuli | Multilingual parliamentary speech ASR. | Not directly integrated. | Research dataset; terms vary by source. | `downloaded-eval`, `needs-legal` for fixtures. | Long-form multilingual ASR. | P2 | 50 short segments. |
+| FLEURS: https://huggingface.co/datasets/google/fleurs | Multilingual speech recognition/translation. | Not directly integrated. | HF dataset; verify license/version. | `downloaded-eval`. | Multilingual ASR and language-ID gate. | P1 | 50 clips per launch locale. |
+| VoiceBench: https://github.com/matthewcym/voicebench and https://huggingface.co/datasets/hlt-lab/voicebench | LLM-based voice assistant benchmark across spoken instruction subsets. | Partially covered: `voicebench_quality` is registered; `voicebench` latency harness exists. | HF/GitHub pages indicate Apache-2.0 for public dataset/code; verify exact subset terms. | `downloaded-eval`; selected `repo-fixture` possible for smoke. | Existing `voicebench_quality`; add missing subsets or adapter parity tests. | P0 | Close gap between registered suite list and current HF subsets; document skipped subsets. |
+| VoiceAssistant-Eval: https://github.com/mathllm/VoiceAssistant-Eval and https://huggingface.co/datasets/MathLLMs/VoiceAssistant-Eval | Listening, speaking, and viewing; content quality, speech quality, consistency. | New. | Public HF/GitHub; license needs check before fixtures. | `downloaded-eval`, `needs-legal` for raw media fixtures. | Eliza-1 voice gates and scenario-runner multimodal live lane. | P2 | Metadata-only adapter plus 20 listening/speaking rows after license check. |
+| VocalBench: https://github.com/SJTU-OmniAgent/VocalBench | Semantic, acoustic, chat, latency/RTF dimensions for speech interaction models. | New; overlaps with voicebench latency. | Public repo; data/license needs review. | `downloaded-eval`, `needs-legal`. | Eliza-1 speech-to-speech release gate. | P2 | Latency/RTF metric schema first, data second. |
+| VCB-Bench: https://github.com/Tencent/VCB-Bench | Chinese voice chat benchmark with robustness and multi-turn tasks. | New. | Public repo; license/data terms need review. | `downloaded-eval`, `needs-legal`. | Multilingual voice assistant lane. | P2 | 20 Chinese prompt/audio rows with pass-at-1 task scoring. |
+| MTalk-Bench: https://github.com/FreedomIntelligence/MTalk-Bench | Multi-turn speech-to-speech dialogue with semantic, paralinguistic, ambient, multiparty dimensions. | New. | Research license noted by project; verify terms before use. | `needs-legal`, likely `eval-only`. | Scenario-runner live voice and Eliza-1 full-duplex gates. | P2 | Methodology-only first; no raw data committed. |
+| aiewf-eval: https://github.com/kwindla/aiewf-eval | Multi-turn workflow evals with text, realtime audio, speech-to-speech, tool-use, instruction, and grounding dimensions. | New; methodology overlaps LifeOpsBench and scenario-runner. | Public repo; license needs review. | `downloaded-eval` for code ideas, not data until verified. | LifeOpsBench adapter design and realtime audio scenario design. | P1 | Borrow report/schema concepts; implement an elizaOS-native workflow smoke fixture. |
+| Vox-Profile: https://arxiv.org/html/2505.14648v1 | Speaker/speech trait robustness benchmark. | New; overlaps speaker validation. | Paper/data release status needs review. | `needs-legal`. | Voice-speaker-validation robustness report. | P2 | Trait taxonomy mapped to existing speaker validation metrics. |
+| Picovoice TTS benchmark: https://github.com/Picovoice/text-to-speech-benchmark | Streaming TTS responsiveness under assistant-like generated text. | New harness; overlaps TTS latency gates. | Framework and datasets need license review. | `downloaded-eval`. | Eliza-1 TTS latency/resource gate. | P1 | 20 generated-response prompts with first-audio/p95/resource metrics. |
+| Sierra tau-Knowledge / tau-Voice / 3-Bench: https://sierra.ai/blog/bench-advancing-agent-benchmarking-to-knowledge-and-voice | Knowledge-grounded backend task completion and realistic voice interactions with interruptions/noise/dropped frames. | New as dataset; methodology overlaps LifeOpsBench and Eliza-1 voice gates. | Blog describes benchmark release; public code/data/license must be verified separately. | `needs-legal`; methodology can be used without copying data. | LifeOpsBench knowledge domain and scenario-runner live voice task lane. | P0 methodology, P2 data | Add eliza-owned knowledge-base task and voice-interruption scenario using the same scoring philosophy. |
+| MultiVox: https://arxiv.org/html/2507.10859v1 | Omni voice assistant benchmark with speech plus visual cues and paralinguistic features. | New. | Paper/data release and license need review. | `needs-legal`. | Multimodal scenario-runner and Eliza-1 audio-vision gate. | P2 | Methodology-only until data terms are clear. |
+| SOVA-Bench: https://arxiv.org/abs/2506.02457 | Speech conversational voice assistant benchmark, including understanding and generated speech quality. | New. | Paper/data license needs review. | `needs-legal`. | Eliza-1 speech generation quality and acoustic metrics. | P2 | Metric taxonomy and scoring schema first. |
+| AudioBench / OpenAudioBench: https://github.com/audiollms/audiobench | AudioLLM benchmark across speech, audio-scene, and voice understanding tasks. | Partially covered by registered MMAU and voicebench, but AudioBench itself is new. | Public repo; dataset licenses vary by subset. | `downloaded-eval`, subset-specific legal review. | Benchmark registry audio understanding lane. | P1 | Map only speech/voice subsets first; avoid scene/music until model support is explicit. |
+| Big Bench Audio: https://huggingface.co/blog/big-bench-audio-release | Speech reasoning benchmark adapted from Big Bench Hard, evaluated across speech-to-speech/text paths. | New; overlaps MMAU/VoiceBench reasoning. | Public blog/dataset; license and access need verification. | `needs-legal` until source package checked. | Eliza-1 speech reasoning gate and hosted provider comparison. | P1 | 100-question reasoning slice, pass-at-1 exact answer. |
+| ADU-Bench: https://arxiv.org/abs/2412.05167 | Open-ended audio dialogue understanding, multilingual ambiguity handling. | New. | Paper/data release and license need review. | `needs-legal`. | Scenario-runner voice ambiguity scenarios. | P2 | Methodology and ambiguity taxonomy first. |
+| AIR-Bench: https://aclanthology.org/2024.acl-long.109/ | Generative audio-language benchmark with speech, natural sounds, and music. | New; broader than current voice-only lanes. | Public benchmark; subset licenses need review. | `downloaded-eval`, subset-specific. | Audio understanding registry entry after speech subset triage. | P2 | Speech-only foundation rows, no music/sound scene rows initially. |
+
+## Recommended P0 Implementation Issues
+
+1. VoiceCodeBench exact-token ASR gate: #13358
+   - Target: `packages/training` plus `packages/benchmarks/registry`.
+   - Metrics: CTEM/TSR, URL/path/ID exact recovery, punctuation-critical WER.
+   - Evidence: real Eliza-1 or configured ASR provider run over a downloaded slice, score JSON, and manually reviewed failures.
+
+2. QMSum and MeetingBank meeting artifact adapter: #13359
+   - Target: `packages/benchmarks/meeting-transcription-proof` with scenario-runner export.
+   - Metrics: query answer correctness, quote grounding, action-item extraction, agenda/topic coverage, summary faithfulness.
+   - Evidence: real provider run over a tiny downloaded slice and `elizaos.meeting_artifact.v1` outputs inspected by hand.
+
+3. VoiceBench coverage closeout: #13360
+   - Target: existing `voicebench_quality` and `voicebench` registry entries.
+   - Metrics: current suite coverage vs public VoiceBench subsets, per-suite score, STT provider, judge model, mock-result rejection.
+   - Evidence: real `voicebench_quality` run with non-mock STT and Cerebras judge, plus report JSON.
+
+4. Sierra-style tau-Knowledge/tau-Voice eliza-owned fixtures: #13361
+   - Target: `lifeops-bench` and scenario-runner live voice.
+   - Metrics: deterministic backend state success, knowledge-source grounding, interruption recovery, dropped-frame robustness, task pass-at-1.
+   - Evidence: eliza-owned synthetic domain only; no Sierra data unless license/access is confirmed.
+
+## Source And Repo Review Notes
+
+- QMSum's primary repository describes 1,808 query-summary pairs over 232
+  meetings across Academic, Product, and Committee domains and carries an MIT
+  repository license, but adapter PRs still need corpus-level provenance review.
+- MeetingBank's project page describes 1,366 city-council meetings, more than
+  3,579 hours of video, transcripts, minutes, agendas, and 6,892 segment-level
+  summarization instances; adapters should download by manifest rather than
+  commit raw data.
+- ELITR-Bench includes password-protected JSON archives with manually crafted
+  questions, ground-truth answers, metadata, and generated responses. GitHub
+  reports mixed code/data licenses, including CC BY 4.0 for data, so keep it P1
+  and recheck upstream ELITR terms before fixtures.
+- VoiceCodeBench's Hugging Face card reports MIT licensing, 300 test rows, audio
+  modality, and structured entities; it is the cleanest P0 exact-token ASR slice.
+- VoiceBench's GitHub repository and Hugging Face dataset report Apache-2.0
+  licensing, public data, and 20,554 rows. elizaOS already registers
+  `voicebench` and `voicebench_quality`, so the P0 work is coverage closeout, not
+  a greenfield benchmark.
+- Sierra's 3-Bench blog describes tau-Voice as realistic full-duplex voice with
+  interruptions, noisy/compressed audio, dropped frames, and task pass@1
+  failures. The P0 value is methodology and eliza-owned fixtures until public
+  code/data/license status is reviewed.
+- Local repo inspection found existing benchmark registry coverage for
+  `voicebench`, `voicebench_quality`, `mmau`, and `tau_bench`, plus
+  voice-speaker-validation, LifeOpsBench, plugin-google Meet report imports, and
+  Eliza-1 training/eval surfaces. #13378 is the proposed
+  meeting-transcription-proof adapter-contract scaffold; no direct QMSum,
+  MeetingBank, VoiceCodeBench, ELITR-Bench, or TCR adapter exists on `develop`
+  today.
+
+## Follow-Up Triage
+
+- Legal: MeetBench/MeetAll, mu-Bench, MTalk-Bench, AMI, ICSI, CHiME-6, and any dataset with non-commercial, restricted, or source-corpus terms.
+- Infra: long audio downloads, GPU/ASR providers, and nightly-only release lanes.
+- Product/model: which Eliza-1 gates become blocking versus dashboard-only.
+- Contamination: keep training data and evaluation data separated; do not fine-tune on benchmark rows without explicit approval.
+
+## Verification Notes
+
+This PR is documentation and planning only. It does not add adapters, benchmark
+data, model behavior, or runtime code. Applicable evidence is source review and
+repo coverage inspection; real run evidence belongs to the adapter PRs listed
+above.
@@ -0,0 +1,347 @@
+# Orchestrator Sub-Agent Benchmark Runbook
+
+This runbook is the validation loop for the ElizaOS coding agent, OpenCode,
+Pi Agent, and the swarm/orchestrator layer.
+
+## Current Wiring
+
+- Default code-agent adapter is `elizaos` through
+  `ELIZA_ACP_DEFAULT_AGENT` / `ELIZA_DEFAULT_AGENT_TYPE`.
+- Settings can switch the default to `opencode` or `pi-agent`; with
+  `ELIZA_AGENT_SELECTION_STRATEGY=fixed`, the configured default wins over a
+  planner guess.
+- Benchmark matrix runs override the default with `BENCHMARK_TASK_AGENT`.
+- Sub-agent routing uses existing rooms. Each spawned session receives
+  task-room and optional worktree-room metadata, plus instructions for
+  `QUESTION_FOR_TASK_CREATOR` and `AGENT_COORDINATION`.
+- The coding tool surface should stay narrow: read/search files, edit/write
+  patches, shell/test commands, git/diff inspection, task status, and swarm or
+  parent communication. Avoid loading broad personal-data, connector, media, or
+  unrelated app actions into coding sub-agents.
+
+OpenCode's relevant built-in tool surface is broader than the minimum:
+`read`, `write`, `edit`, `apply_patch`, `glob`, `grep`, shell/bash, LSP,
+todo, task/subagent, question, skill, repo clone/overview, web fetch/search,
+and plugin/MCP tools. The ElizaOS coding profile should start with the narrow
+subset above, then add only tools justified by failed trajectories.
+
+## Matrix Coverage
+
+The executable coverage manifest lives in
+`packages/benchmarks/orchestrator/code_agent_coverage.py`; tests assert that
+its included IDs match `DEFAULT_BENCHMARKS`.
+
+The code-agent matrix currently covers the benchmark families that have a
+thin, repeatable Eliza bridge path and enough structured output for
+head-to-head reporting:
+
+- `swe_bench` — SWE-bench Lite through the shared SWE-bench bridge.
+- `swe_bench_multilingual` — SWE-bench Multilingual through the same bridge,
+  reported separately because it stresses non-Python code repair.
+- `terminal_bench` — terminal task execution.
+- `mind2web` — browser/navigation DOM-action tasks.
+- `visualwebbench` — visual web understanding and grounding.
+- `webshop` — simulated e-commerce web-interaction tasks using Princeton
+  WebShop's reward function.
+- `osworld` — desktop/computer-use tasks.
+
+`swe-bench-pro` is intentionally not a default matrix cell yet. The vendored
+tree uses a separate prediction-gather plus Docker/Modal evaluation workflow,
+so it needs a dedicated adapter before its results would be comparable to the
+other matrix cells.
+
+`nl2repo`, `app-eval`, `vision-language`, AgentBench/MINT, and the
+CLaw/OpenCLaw/QwenClaw suites are not default code-agent matrix cells yet.
+They are relevant adjacent benchmarks, but they either need a dedicated
+Eliza-vs-OpenCode adapter, use a different model/runtime surface, require
+heavy external datasets or LLM judges, or are broader general-agent/regression
+suites rather than comparable coding-agent cells. Add them only after their
+runner can emit the same structured right/wrong, cached-token, input/output
+token, LLM-call, result-path, and trajectory artifacts as the default matrix.
+
+## Secrets
+
+Keep provider keys in the process environment or a secret manager only.
+Do not commit keys into profiles, `.env`, run artifacts, or docs.
+
+For Cerebras gemma-4-31b:
+
+```bash
+export CEREBRAS_API_KEY=...
+```
+
+The matrix harness redacts secret-looking values from `command.json`,
+`stdout.log`, `stderr.log`, JSON, JSONL, text, and Markdown artifacts.
+
+## Smoke Validation
+
+From the repository root:
+
+```bash
+cd packages
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --dry-run \
+  --smoke \
+  --no-docker \
+  --max-tasks 1 \
+  --benchmarks swe_bench,swe_bench_multilingual,terminal_bench,mind2web,visualwebbench,webshop,osworld \
+  --adapters elizaos,opencode,pi-agent \
+  --provider cerebras \
+  --model gemma-4-31b \
+  --run-root /tmp/eliza-code-agent-matrix-smoke
+```
+
+Expected checks:
+
+- one cell for each `(swe_bench|swe_bench_multilingual|terminal_bench|mind2web|visualwebbench|webshop|osworld) x (elizaos|opencode|pi-agent)`;
+- `BENCHMARK_TASK_AGENT` matches the cell adapter;
+- no provider key appears in command metadata;
+- `summary.json` and `summary.md` are written with right/wrong counts,
+  cached-token percent, input/output/total tokens, LLM call count, and the
+  `head_to_head` ElizaOS-vs-OpenCode section.
+- `summary.json.coverage` reports seven included benchmarks and the deferred
+  adjacent suites from `code_agent_coverage.py`.
+
+Before a publishable run, use preflight. It writes `preflight.json` and
+`preflight.md` and exits before running cells:
+
+```bash
+cd packages
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --preflight \
+  --benchmarks swe_bench,swe_bench_multilingual,terminal_bench,mind2web,visualwebbench,webshop,osworld \
+  --adapters elizaos,opencode,pi-agent \
+  --provider cerebras \
+  --model gemma-4-31b \
+  --max-tasks 1
+```
+
+Preflight rejects missing provider credentials for live runs, missing
+benchmark entrypoints and working directories, a missing `opencode` CLI when
+the opencode adapter is selected, and a missing Docker CLI or daemon where a
+selected benchmark requires Docker. Add `--enforce-release-readiness` together
+with `--quality-guardrail-summary /path/to/non-code-quality-guardrail.json` to
+also require a clean non-code quality-guardrail report at preflight time.
+The publishable comparison adapter pair is always `elizaos` target vs
+`opencode` baseline.
+
+## Real Comparison Run
+
+Start small and resumable:
+
+```bash
+cd packages
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --benchmarks swe_bench,swe_bench_multilingual,terminal_bench,mind2web,visualwebbench,webshop,osworld \
+  --adapters elizaos,opencode,pi-agent \
+  --provider cerebras \
+  --model gemma-4-31b \
+  --max-tasks 1 \
+  --timeout-seconds 3600
+```
+
+The seven-benchmark commands run with Docker enabled: `osworld` defaults to
+Docker execution, and preflight rejects `--no-docker` for osworld unless an
+alternate provider (`OSWORLD_PROVIDER_NAME=vmware|virtualbox|aws`) is
+configured. Use `--no-docker` only on subsets that exclude osworld.
+
+Then run three independent passes:
+
+```bash
+for i in 1 2 3; do
+  python -m benchmarks.orchestrator.code_agent_matrix \
+    --benchmarks swe_bench,swe_bench_multilingual,terminal_bench,mind2web,visualwebbench,webshop,osworld \
+    --adapters elizaos,opencode,pi-agent \
+    --provider cerebras \
+    --model gemma-4-31b \
+    --max-tasks 1 \
+    --force
+done
+```
+
+After repeated runs, attach the longitudinal ElizaOS-vs-OpenCode trend by
+re-summarizing the latest run against a previous run's `summary.json`:
+
+```bash
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --summarize benchmark_results/code-agent-matrix/<latest-run> \
+  --compare-summary benchmark_results/code-agent-matrix/<previous-run>/summary.json
+```
+
+To index every run under one browsable HTML overview:
+
+```bash
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --write-run-index benchmark_results/code-agent-matrix/index \
+  --index-scan-root benchmark_results/code-agent-matrix
+```
+
+Each run writes:
+
+- `benchmark_results/code-agent-matrix/<timestamp>/summary.json`
+- `summary.md`
+- per-cell `command.json`, `stdout.log`, `stderr.log`
+- benchmark output JSON
+- trajectories under each cell's `trajectories/`
+
+The top-level summaries are the comparison scorecard: each cell records
+`outcome_metrics.right`, `outcome_metrics.wrong`, `token_metrics.input_tokens`,
+`token_metrics.output_tokens`, `token_metrics.total_tokens`,
+`token_metrics.cached_token_percent`, and `token_metrics.llm_call_count`.
+The `evidence` block records `run_mode`, required provider env-var names,
+whether provider credentials were present, and `publishable_live_evidence`.
+Treat smoke, dry-run, mock-only, missing-credential, and provider-auth-failure
+summaries as validation artifacts, not reportable benchmark evidence.
+The `coverage` block records which included benchmark IDs were selected, which
+included IDs were omitted, any unknown selected IDs, and all deferred adjacent
+suites with reasons. Publishable gates reject summaries or trends that do not
+cover every included benchmark.
+The `head_to_head` section compares `elizaos` against `opencode` by benchmark,
+including accuracy, total-token, and LLM-call deltas plus trajectory directory
+pointers for both cells. Each comparison also includes `triage_hints`, which
+flag target failure class, missing trajectory artifacts, missing token
+telemetry, and higher ElizaOS token or LLM-call usage.
+The `review_queue` section ranks the rows that need attention first, starting
+with inferior ElizaOS rows, then target failures, missing comparisons, missing
+telemetry or trajectories, and finally higher token/call usage.
+For every queued row, `trajectory_reviews` adds prompt-safe target/baseline
+trajectory summaries and token-heavy turn pointers so the first inspection can
+start from the highest-signal turns before reading full trajectories.
+`improvement_backlog` converts the same rows into evidence-scoped hypotheses
+and recommended next actions, so live inferior rows become concrete Eliza
+patch tasks instead of loose notes.
+With `--compare-summary`, the summary gains a `previous_summary_comparison`
+section (and a trend table in `summary.md`): per-benchmark trend status
+(`improved` / `unchanged` / `regressed` / `missing`), target-accuracy deltas,
+accuracy-gap change, and target token / cached-percent / LLM-call deltas.
+Add `--enforce-no-regression` to exit nonzero if ElizaOS target accuracy
+regressed against the compared summary.
+Before publishing results, run with the enforcement stack the tool itself
+recommends in `preflight.json.next_commands` (`release_comparable`):
+`--enforce-live-report --enforce-trajectory-reviews --enforce-report
+--enforce-coverage --enforce-comparable --enforce-required-stats
+--enforce-token-evidence --enforce-efficiency --enforce-release-readiness`.
+These exit nonzero unless the evidence is live provider-backed, covers all
+included benchmark IDs, has no inferior or missing ElizaOS rows, and includes
+coherent right/wrong/total counts, accuracy values that match `right / total`,
+integer input/output/total-token and LLM-call counts,
+`total_tokens == input_tokens + output_tokens`, and cached-token percent for
+both adapters on every compared row.
+
+If claiming code-agent improvements did not sacrifice non-code quality, also
+require a clean non-code quality-guardrail artifact:
+
+```bash
+PYTHONPATH=packages python -m benchmarks.orchestrator validate-latest-readiness \
+  --skip-runtime-gates \
+  --exclude-benchmarks agentbench,mind2web,mint,nl2repo,osworld,standard_humaneval,swe_bench,terminal_bench,vision_language,visualwebbench,webshop \
+  --json > /path/to/non-code-quality-guardrail.json
+
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --summarize benchmark_results/code-agent-matrix/<latest-run> \
+  --quality-guardrail-summary /path/to/non-code-quality-guardrail.json \
+  --enforce-quality-guardrail
+```
+
+A guardrail summary is clean when it is a readiness report with `ok: true` and
+an empty `findings` list. A missing or failing guardrail summary keeps the
+code-agent run non-publishable when `--enforce-quality-guardrail` (or
+`--enforce-release-readiness`) is set.
+
+Summarize an interrupted or completed run without re-executing:
+
+```bash
+cd packages
+python -m benchmarks.orchestrator.code_agent_matrix \
+  --summarize /path/to/benchmark_results/code-agent-matrix/<timestamp>
+```
+
+## Triage Loop
+
+1. Read `summary.json.review_queue` and the Review Queue table in `summary.md`.
+2. For each queued row, open the listed ElizaOS trajectory directory and the
+   matching OpenCode trajectory directory.
+3. Use `summary.json.head_to_head.comparisons[].triage_hints` to prioritize the
+   first suspected cause before reading full logs.
+4. If the head-to-head section is missing because one adapter did not run,
+   compare `summary.json` by adapter and benchmark manually.
+5. Find cases where OpenCode has `pass` or a higher score and ElizaOS has
+   `no_patch`, `patch_apply_failed`, `tests_failed`, `timeout`,
+   `auth_or_provider`, or `stopped_early`.
+6. Read the ElizaOS trajectory and logs first, then the matching OpenCode
+   trajectory.
+7. Classify the cause as prompt/tooling/provider/harness/coordination.
+8. Patch the smallest surface:
+   - prompt/instructions when the agent stopped early or failed to persist a
+     patch;
+   - tool whitelist when it lacks an OpenCode capability that mattered;
+   - provider/env routing when auth or model calls failed;
+   - swarm routing when agents missed coordination or user-question flow.
+9. Re-run only the failed cells with `--force` or a higher `--max-tasks` once
+   the small set is stable.
+
+Do not add broad actions because a single trajectory failed. Add a capability
+only when it is repeatedly needed for coding or terminal tasks and can be
+tested.
+
+For SWE-bench `no_patch` triage, first replay patch extraction against the
+latest ElizaOS telemetry. This separates empty responses from parser misses and
+from diffs that need repository source context:
+
+```bash
+PYTHONPATH=packages python - <<'PY'
+import json
+from collections import Counter
+from pathlib import Path
+from benchmarks.swe_bench import cli
+
+telemetry = Path(
+    "benchmark_results/code-agent-matrix/<run>/swe_bench/elizaos/trajectories/telemetry.jsonl"
+)
+counts = Counter()
+for line in telemetry.read_text(errors="replace").splitlines():
+    obj = json.loads(line)
+    text = obj.get("response_text") or obj.get("response") or ""
+    candidate = cli._extract_patch_candidate(text) if text else ""
+    if not text:
+        counts["empty_response"] += 1
+    elif not candidate:
+        counts["no_candidate"] += 1
+    elif cli._extract_patch(text):
+        counts["valid_without_repo"] += 1
+    else:
+        counts[cli._unified_diff_error(cli._sanitize_patch_text(candidate))] += 1
+print(dict(counts))
+PY
+```
+
+`no_candidate` should be rare. If it is not, patch extraction likely missed a
+model-output shape. If most failures are invalid hunk headers, inspect whether
+`_candidate_context_paths` included the source files needed to repair bare
+model hunk headers before changing the agent prompt.
+
+## Swarm Validation
+
+For orchestrated sub-agent behavior, validate these separately from single
+agent scoring:
+
+- task room receives final status and questions for the creator;
+- worktree room receives coordination messages when agents touch overlapping
+  files;
+- if task and worktree rooms collapse to one room, routing sends one message;
+- sub-agent names/labels are preserved in session metadata and synthetic
+  messages;
+- blocked/question events ping the task creator through the originating
+  channel and wait for the follow-up before continuing;
+- agents include changed files, tests run, risks, and coordination state in
+  final reports.
+
+## Success Bar
+
+ElizaOS is comparable when, over at least three runs on the same task sample:
+
+- it writes patches consistently;
+- it does not underperform OpenCode on every run;
+- failures are explainable from trajectories rather than missing artifacts;
+- any OpenCode-only win is mapped to an actionable ElizaOS change or a known
+  benchmark variance.
@@ -0,0 +1,5 @@
+watch_file .mise.toml
+[[ -e ~/.local/bin/mise ]] || (curl -sf https://mise.run | MISE_QUIET=1 sh)
+~/.local/bin/mise trust 2> /dev/null
+~/.local/bin/mise install -qy
+direnv_load ~/.local/bin/mise direnv exec
@@ -0,0 +1,214 @@
+# Model checkpoints
+*.pth
+*.pt
+
+# Credential files
+evaluation_examples/settings/google/settings.json
+evaluation_examples/settings/googledrive/credentials.json
+evaluation_examples/settings/googledrive/client_secrets.json
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# PyCharm
+**/.idea/**/*
+
+# Mac OS
+.DS_Store
+
+# data
+**/data/**/*
+!**/utils/data/**/*
+
+# tmp files
+**/tmp/**/*
+api_key.py
+tmp.*
+
+## Server logging
+**/.logging/**/*
+
+# DB cache
+**/.db_cache/**/*
+
+**/debugging/**/*
+
+# embedding repo
+instructor-embedding
+
+# plugin cache
+**/static/**/*
+
+# frontend cache
+frontend/node_modules/
+frontend/.next/
+frontend/.idea
+
+tags
+tags-opts
+snapshots
+branch_flag
+branch-config
+*.syncthing.*.tmp
+cache
+version.folder
+at_processing
+
+test.xlsx
+test2.xlsx
+
+# vm info
+.vms
+/vm_data
+docker_vm_data
+vmware_vm_data
+.vmware*
+.aws*
+
+# result
+**/result*/**/*
+
+.vscode
+
+dataimpulse_proxy_config.json
+
+## reference and draft and debug
+reference/
+draft/
+manual_examine.py
+run_human_examine.sh
+quick_start.py
+result_multi_apps_pengxiang_transformers12evaluation_examples/settings/proxy/dataimpulse.json
+evaluation_examples/settings/proxy/dataimpulse.json
+
+# Local test configurations (not for public repo)
+evaluation_examples/spiderman.json
+evaluation_examples/test_50_random_proportional.json
+evaluation_examples/test_chrome.json
@@ -0,0 +1,5 @@
+[tools]
+python = "3.14.4"
+
+[env]
+_.python.venv = { path = ".venv", create = true }
@@ -0,0 +1,137 @@
+# OSWorld — Agent Guide
+
+Multimodal desktop agent benchmark: 369 real computer tasks spanning Chrome,
+LibreOffice, GIMP, VS Code, and more — arXiv:2404.07972. Vendored from
+[xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld) with an elizaOS
+TypeScript bridge agent layered on top. Registered in the suite registry as
+`osworld`.
+
+## Run
+
+```bash
+# Direct — single task via Docker provider (from this directory)
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --result_dir ./results/eliza \
+    --task_id 030eeff7-b492-4218-b312-701ec99ee0cc
+
+# Direct — all tasks, 5 parallel VMs
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --num_envs 5 \
+    --result_dir ./results/eliza
+
+# VMware on macOS
+python scripts/python/run_multienv_eliza.py \
+    --provider_name vmware \
+    --path_to_vm ~/Virtual\ Machines.localized/Ubuntu.vmwarevm/Ubuntu.vmx \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --result_dir ./results/eliza
+
+# Through the suite orchestrator
+python -m benchmarks.orchestrator run --benchmarks osworld --provider <p> --model <m>
+```
+
+## Smoke test (no VM required)
+
+```bash
+# Runs one synthetic in-process task; does not start VMs or the Eliza server
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 1 \
+    --dry_run \
+    --result_dir /tmp/osworld-smoke
+
+# Via orchestrator (passes extra.dry_run=true)
+python -m benchmarks.orchestrator run --benchmarks osworld --provider mock --model mock \
+    --extra '{"dry_run": true}'
+```
+
+## Test the harness
+
+```bash
+# From the OSWorld directory
+pip install -e .
+pytest tests/ -v
+```
+
+## Layout
+
+| Path | Role |
+| --- | --- |
+| `scripts/python/run_multienv_eliza.py` | Primary entrypoint (elizaOS bridge agent, multi-env) |
+| `run.py` | Legacy single-env runner (almost deprecated) |
+| `desktop_env/` | VM provider abstractions (Docker, VMware, VirtualBox, AWS, Azure, GCP) |
+| `desktop_env/evaluators/` | Per-app task evaluators (Chrome, GIMP, LibreOffice, VLC, VS Code, etc.) |
+| `mm_agents/` | Reference agent implementations (upstream; not used by elizaOS path) |
+| `evaluation_examples/` | 369 task config JSON files, organised by domain |
+| `tests/test_run_multienv_eliza.py` | pytest suite for the elizaOS bridge harness |
+| `lib_run_single.py` | Per-task execution loop (shared by all runners) |
+| `lib_results_logger.py` | Structured result/error logging helpers |
+| `pyproject.toml` | Package metadata and dependencies |
+
+## Notes
+
+- Requires a VM provider: Docker (with KVM), VMware, or VirtualBox. No API
+  keys are needed for the benchmark itself, but the agent model needs an LLM key.
+- The elizaOS bridge routes all decisions through the TypeScript benchmark
+  server (`packages/lifeops-bench/src/server.ts`). Set `ELIZA_BENCH_URL`
+  to skip auto-starting it and point at an already-running instance.
+- Results write to `./results/eliza/` by default (gitignored). The orchestrator
+  writes to its own `output_dir` and locates `osworld-eliza-results-*.json`.
+- Scored by `_score_from_osworld_json` in `registry/scores.py`.
+- Observation types: `screenshot`, `a11y_tree`, `screenshot_a11y_tree` (default), `som`.
+- Full setup (VM provisioning, GCP auth, proxy): [SETUP_GUIDELINE.md](SETUP_GUIDELINE.md).
+- Upstream paper and data: [README.md](README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,137 @@
+# OSWorld — Agent Guide
+
+Multimodal desktop agent benchmark: 369 real computer tasks spanning Chrome,
+LibreOffice, GIMP, VS Code, and more — arXiv:2404.07972. Vendored from
+[xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld) with an elizaOS
+TypeScript bridge agent layered on top. Registered in the suite registry as
+`osworld`.
+
+## Run
+
+```bash
+# Direct — single task via Docker provider (from this directory)
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --result_dir ./results/eliza \
+    --task_id 030eeff7-b492-4218-b312-701ec99ee0cc
+
+# Direct — all tasks, 5 parallel VMs
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --num_envs 5 \
+    --result_dir ./results/eliza
+
+# VMware on macOS
+python scripts/python/run_multienv_eliza.py \
+    --provider_name vmware \
+    --path_to_vm ~/Virtual\ Machines.localized/Ubuntu.vmwarevm/Ubuntu.vmx \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 15 \
+    --result_dir ./results/eliza
+
+# Through the suite orchestrator
+python -m benchmarks.orchestrator run --benchmarks osworld --provider <p> --model <m>
+```
+
+## Smoke test (no VM required)
+
+```bash
+# Runs one synthetic in-process task; does not start VMs or the Eliza server
+python scripts/python/run_multienv_eliza.py \
+    --provider_name docker \
+    --observation_type screenshot_a11y_tree \
+    --model gemma-4-31b \
+    --max_steps 1 \
+    --dry_run \
+    --result_dir /tmp/osworld-smoke
+
+# Via orchestrator (passes extra.dry_run=true)
+python -m benchmarks.orchestrator run --benchmarks osworld --provider mock --model mock \
+    --extra '{"dry_run": true}'
+```
+
+## Test the harness
+
+```bash
+# From the OSWorld directory
+pip install -e .
+pytest tests/ -v
+```
+
+## Layout
+
+| Path | Role |
+| --- | --- |
+| `scripts/python/run_multienv_eliza.py` | Primary entrypoint (elizaOS bridge agent, multi-env) |
+| `run.py` | Legacy single-env runner (almost deprecated) |
+| `desktop_env/` | VM provider abstractions (Docker, VMware, VirtualBox, AWS, Azure, GCP) |
+| `desktop_env/evaluators/` | Per-app task evaluators (Chrome, GIMP, LibreOffice, VLC, VS Code, etc.) |
+| `mm_agents/` | Reference agent implementations (upstream; not used by elizaOS path) |
+| `evaluation_examples/` | 369 task config JSON files, organised by domain |
+| `tests/test_run_multienv_eliza.py` | pytest suite for the elizaOS bridge harness |
+| `lib_run_single.py` | Per-task execution loop (shared by all runners) |
+| `lib_results_logger.py` | Structured result/error logging helpers |
+| `pyproject.toml` | Package metadata and dependencies |
+
+## Notes
+
+- Requires a VM provider: Docker (with KVM), VMware, or VirtualBox. No API
+  keys are needed for the benchmark itself, but the agent model needs an LLM key.
+- The elizaOS bridge routes all decisions through the TypeScript benchmark
+  server (`packages/lifeops-bench/src/server.ts`). Set `ELIZA_BENCH_URL`
+  to skip auto-starting it and point at an already-running instance.
+- Results write to `./results/eliza/` by default (gitignored). The orchestrator
+  writes to its own `output_dir` and locates `osworld-eliza-results-*.json`.
+- Scored by `_score_from_osworld_json` in `registry/scores.py`.
+- Observation types: `screenshot`, `a11y_tree`, `screenshot_a11y_tree` (default), `som`.
+- Full setup (VM provisioning, GCP auth, proxy): [SETUP_GUIDELINE.md](SETUP_GUIDELINE.md).
+- Upstream paper and data: [README.md](README.md).
+
+<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
+## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests
+
+> The binding, repo-wide standard is **[AGENTS.md](../../../AGENTS.md)**. Read it.
+> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
+> works **without reading the code**, from the artifacts you attach. This applies to **every**
+> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.
+
+- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
+  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
+  providers/context, the raw model output, every tool/action call, and the result. Then **open
+  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
+  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
+- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
+  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
+  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
+  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
+  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
+  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
+  them is part of your change.
+- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
+  the entire feature or view, start to finish (`bun run test:e2e:record`).
+- **Manually review every artifact the change touches** — never just the green check: client
+  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
+  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
+- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
+  blocker by the **hard path**: build the real architecture, stand up the real
+  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
+  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
+  highest-effort, production-ready version. Keep going until every possibility is exhausted.
+
+Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
+explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
+behavior, **re-capture** evidence; stale proof is worse than none.
+
+**Capture & manually review for this package — benchmark / eval suite:**
+- A **real-model** run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
+- The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
+- The provider matrix actually exercised, and the scoring math validated against a known case.
+- Failure / timeout / partial-output handling in the harness itself.
+<!-- END: evidence-and-e2e-mandate -->
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024 XLANG NLP Lab
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
@@ -0,0 +1,274 @@
+<p align="center">
+  <img src="https://huggingface.co/datasets/xlangai/assets/resolve/main/github_banner_v2.png" alt="Banner">
+</p>
+
+<p align="center">
+  <a href="https://os-world.github.io/">Website</a> •
+  <a href="https://arxiv.org/abs/2404.07972">Paper</a> •
+  <a href="https://timothyxxx.github.io/OSWorld/">Doc</a> •
+  <a href="https://github.com/xlang-ai/OSWorld/tree/main/evaluation_examples">Data</a> •
+  <a href="https://os-world.github.io/explorer.html">Data Viewer</a> •
+  <a href="https://discord.gg/4Gnw7eTEZR">Discord</a> •
+  <a href="https://drive.google.com/file/d/1XlEy49otYDyBlA3O9NbR0BpPfr2TXgaD/view?usp=drive_link">Cache</a>
+</p>
+
+<p align="center">
+    <a href="https://img.shields.io/badge/PRs-Welcome-red">
+        <img src="https://img.shields.io/badge/PRs-Welcome-red">
+    </a>
+    <a href="https://img.shields.io/github/last-commit/xlang-ai/OSWorld?color=green">
+        <img src="https://img.shields.io/github/last-commit/xlang-ai/OSWorld?color=green">
+    </a>
+    <a href="https://opensource.org/licenses/Apache-2.0">
+        <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg">
+    </a>
+    <a href="https://badge.fury.io/py/desktop-env">
+        <img src="https://badge.fury.io/py/desktop-env.svg">
+    </a>
+    <a href="https://pepy.tech/project/desktop-env">
+        <img src="https://static.pepy.tech/badge/desktop-env">
+    </a>
+    <br/>
+</p>
+
+
+## 📢 Updates
+- 2025-07-28: Introducing **OSWorld-Verified**! We have made major updates, fixed several issues reported by the community, with more support for AWS (can reduce evaluation time to within 1 hour through parallelization!), and making the benchmark signals more effective. Check out more in the [report](https://xlang.ai/blog/osworld-verified). We have run new model results in the latest version and updated them on the [official website](https://os-world.github.io/). Please compare your OSWorld results with the new benchmark results when running the latest version.
+- 2025-05-01: If you need pre-downloaded files for init state setup, we downloaded for you [here](https://drive.google.com/file/d/1XlEy49otYDyBlA3O9NbR0BpPfr2TXgaD/view?usp=drive_link).
+- 2024-10-22: We supported Docker🐳 for hosting virtual machines on virtualized platforms. Check below for detailed instructions!
+- 2024-06-15: We refactor the code of environment part to decompose VMware Integration, and start to support other platforms such as VirtualBox, AWS, Azure, etc. Hold tight!
+- 2024-04-11: We released our [paper](https://arxiv.org/abs/2404.07972), [environment and benchmark](https://github.com/xlang-ai/OSWorld), and [project page](https://os-world.github.io/). Check it out!
+
+## 💾 Installation
+### VMware/VirtualBox (Desktop, Laptop, Bare Metal Machine)
+Suppose you are operating on a system that has not been virtualized (e.g. your desktop, laptop, bare metal machine), meaning you are not utilizing a virtualized environment like AWS, Azure, or k8s.
+If this is the case, proceed with the instructions below. However, if you are on a virtualized platform, please refer to the [Docker](https://github.com/xlang-ai/OSWorld?tab=readme-ov-file#docker-server-with-kvm-support-for-the-better) section.
+
+1. First, clone this repository and `cd` into it. Then, install the dependencies listed in `requirements.txt`. It is recommended that you use the latest version of Conda to manage the environment, but you can also choose to manually install the dependencies. Please ensure that the version of Python is >= 3.10.
+```bash
+# Clone the OSWorld repository
+git clone https://github.com/xlang-ai/OSWorld
+
+# Change directory into the cloned repository
+cd OSWorld
+
+# Optional: Create a Conda environment for OSWorld
+# conda create -n osworld python=3.10
+# conda activate osworld
+
+# Install required dependencies
+pip install -r requirements.txt
+```
+
+Alternatively, you can install the environment without any benchmark tasks:
+```bash
+pip install desktop-env
+```
+
+2. Install [VMware Workstation Pro](https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html) (for systems with Apple Chips, you should install [VMware Fusion](https://support.broadcom.com/group/ecx/productdownloads?subfamily=VMware+Fusion)) and configure the `vmrun` command.  The installation process can refer to [How to install VMware Workstation Pro](desktop_env/providers/vmware/INSTALL_VMWARE.md). Verify the successful installation by running the following:
+```bash
+vmrun -T ws list
+```
+If the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines.
+> **Note:** We also support using [VirtualBox](https://www.virtualbox.org/) if you have issues with VMware Pro. However, features such as parallelism and macOS on Apple chips might not be well-supported.
+
+All set! Our setup script will automatically download the necessary virtual machines and configure the environment for you.
+
+### Docker (Server with KVM Support for Better Performance)
+If you are running on a non-bare metal server, or prefer not to use VMware and VirtualBox platforms, we recommend using our Docker support.
+
+#### Prerequisite: Check if your machine supports KVM
+We recommend running the VM with KVM support. To check if your hosting platform supports KVM, run
+```
+egrep -c '(vmx|svm)' /proc/cpuinfo
+```
+on Linux. If the return value is greater than zero, the processor should be able to support KVM.
+> **Note**: macOS hosts generally do not support KVM. You are advised to use VMware if you would like to run OSWorld on macOS.
+
+#### Install Docker
+If your hosting platform supports a graphical user interface (GUI), you may refer to [Install Docker Desktop on Linux](https://docs.docker.com/desktop/install/linux/) or [Install Docker Desktop on Windows](https://docs.docker.com/desktop/install/windows-install/) based on your OS. Otherwise, you may [Install Docker Engine](https://docs.docker.com/engine/install/).
+
+#### Running Experiments
+Add the following arguments when initializing `DesktopEnv`: 
+- `provider_name`: `docker`
+- `os_type`: `Ubuntu` or `Windows`, depending on the OS of the VM
+> **Note**: If the experiment is interrupted abnormally (e.g., by interrupting signals), there may be residual docker containers which could affect system performance over time. Please run `docker stop $(docker ps -q) && docker rm $(docker ps -a -q)` to clean up.
+
+### AWS
+Using cloud services for parallel evaluation can significantly accelerate evaluation efficiency (can reduce evaluation time to within 1 hour through parallelization!) and can even be used as infrastructure for training.
+We provide comprehensive AWS support with a Host-Client architecture that enables large-scale parallel evaluation of OSWorld tasks.
+For detailed setup instructions, see [Setup Guideline](SETUP_GUIDELINE.md) and [AWS Configuration Guide](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/providers/aws/AWS_GUIDELINE.md). 
+
+### Others
+We are working on supporting more 👷. Please hold tight!
+
+
+## 🚀 Quick Start
+Run the following minimal example to interact with the environment:
+
+```bash
+# Basic usage with default settings
+python quickstart.py
+
+# Customize provider and VM path
+python quickstart.py --provider_name vmware --path_to_vm "path/to/your/vm.vmx"
+```
+
+You will see all the logs of the system running normally, including the successful creation of the environment, completion of setup, and successful execution of actions. In the end, you will observe a successful right-click on the screen, which means you are ready to go.
+
+## 🧪 Experiments
+### Agent Baselines
+
+> **⚠️ Important Configuration Requirements:**
+>
+> * **Google Account Tasks**: Some tasks require Google account access and OAuth2.0 configuration. Please refer to [Setup Guideline - Google Account Setup](SETUP_GUIDELINE.md#1-google-account-setup) for detailed setup instructions.
+> * **Proxy Configuration**: Some tasks may require proxy settings to function properly (this depends on the strength of website defenses against your network location). Please refer to [Setup Guideline - Proxy Configuration](SETUP_GUIDELINE.md#2-proxy-configuration).
+> * **Impact of Missing Configuration**: If these configurations are not properly set up, the corresponding tasks will fail to execute correctly, leading to lower evaluation scores.
+
+
+If you wish to run the baseline agent used in our paper, you can execute the following command as an example under the GPT-4o pure-screenshot setting:
+
+Set **OPENAI_API_KEY** environment variable with your API key
+```bash
+export OPENAI_API_KEY='changeme'
+```
+
+Optionally, set **OPENAI_BASE_URL** to use a custom OpenAI-compatible API endpoint
+```bash
+export OPENAI_BASE_URL='http://your-custom-endpoint.com/v1'  # Optional: defaults to https://api.openai.com
+```
+
+Single-threaded execution (deprecated, using `vmware` provider as example)
+```bash
+python run.py \
+    --provider_name vmware \
+    --path_to_vm Ubuntu/Ubuntu.vmx \
+    --headless \
+    --observation_type screenshot \
+    --model gpt-4o \
+    --sleep_after_execution 3 \
+    --max_steps 15 \
+    --result_dir ./results \
+    --client_password password
+```
+
+Parallel execution (example showing switching provider to `docker`)
+```bash
+python scripts/python/run_multienv.py \
+    --provider_name docker \
+    --headless \
+    --observation_type screenshot \
+    --model gpt-4o \
+    --sleep_after_execution 3 \
+    --max_steps 15 \
+    --num_envs 10 \
+    --client_password password
+```
+
+The results, which include screenshots, actions, and video recordings of the agent's task completion, will be saved in the `./results` (or other `result_dir` you specified) directory in this case.
+You can then run the following command to obtain the result:
+
+```bash
+# Basic usage with default parameters
+python show_result.py
+
+# Specify custom parameters
+python show_result.py \
+    --action_space pyautogui \
+    --model gpt-4o \
+    --observation_type screenshot \
+    --result_dir ./results
+
+# Show detailed scores per domain (format: score/total)
+python show_result.py --detailed
+```
+
+The script will display:
+- Per-domain success rates
+- Category-level statistics (Office, Daily, Professional)
+- Overall success rate and total score
+- With `--detailed` flag: compact format showing "score/total" for each domain
+
+### Manual Task Examination
+For manual verification and examination of specific benchmark tasks, you can use the manual examination tool:
+
+```bash
+python scripts/python/manual_examine.py \
+    --headless \
+    --observation_type screenshot \
+    --result_dir ./results_human_examine \
+    --test_all_meta_path evaluation_examples/test_all.json \
+    --domain libreoffice_impress \
+    --example_id a669ef01-ded5-4099-9ea9-25e99b569840 \
+    --max_steps 3
+```
+
+This tool allows you to:
+- Manually execute tasks in the environment
+- Verify task correctness and evaluation metrics
+- Record the execution process with screenshots and videos
+- Examine specific problematic tasks
+
+See `scripts/bash/run_manual_examine.sh` for example task IDs across different domains.
+
+## Evaluation
+### Local Evaluation
+Please start by reading through the [agent interface](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/README.md) and the [environment interface](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/README.md).
+Correctly implement the agent interface and import your customized version in the `run.py` (for single-threaded execution) or `scripts/python/run_multienv.py` / `scripts/python/run_multienv_xxx.py` (for parallel execution) file.
+Afterward, you can execute a command similar to the one in the previous section to run the benchmark on your agent.
+
+### Public Evaluation
+If you want your results to be verified and displayed on the verified leaderboard, you need to schedule a meeting with us (current maintainer: tianbaoxiexxx@gmail.com, yuanmengqi732@gmail.com) to run your agent code on our side and have us report the results. 
+You need to upload and allow us to disclose your agent implementation under the OSWorld framework (you may choose not to expose your model API to the public), along with a report that allows the public to understand what's happening behind the scenes.
+Alternatively, if you are from a trusted institution, you can share your monitoring data and trajectories with us.
+Please carefully follow the [Setup Guideline - Public Evaluation Platform](SETUP_GUIDELINE.md#3-public-evaluation-platform) to get results.
+
+
+## ❓ FAQ
+### What is the username and password for the virtual machines?
+The username and password for the virtual machines are as follows (for provider `vmware`, `virtualbox` and `docker`): we set the account credentials for Ubuntu as `user` / `password`. 
+For cloud service providers like `aws`, to prevent attacks due to weak passwords, we default to `osworld-public-evaluation`. 
+If you make further modifications, remember to set the client_password variable and pass it to DesktopEnv and Agent (if supported) when running experiments. 
+Some features like setting up proxy require the environment to have the client VM password to obtain sudo privileges, and for some OSWorld tasks, the agent needs the password to obtain sudo privileges to complete them.
+
+### How to setup the account and credentials for Google and Google Drive?
+
+See [Setup Guideline - Google Account Setup](SETUP_GUIDELINE.md#1-google-account-setup).
+
+### How can I configure a proxy for the VM (if I'm behind the GFW, or I don't want some of my tasks to be identified as bot and get lower scores)?
+
+See [Setup Guideline - Proxy Configuration](SETUP_GUIDELINE.md#2-proxy-configuration).
+We also provide a pre-configured solution based on DataImpulse, please refer to the [proxy setup section](SETUP_GUIDELINE.md#23-proxy-for-specific-tasks-recommended).
+
+### Open Source Contributors
+
+Thanks to all the contributors!
+
+<a href="https://github.com/xlang-ai/OSWorld/graphs/contributors">
+  <img src="https://stg.contrib.rocks/image?repo=xlang-ai/OSWorld" />
+</a>
+
+
+## 📄 Citation
+If you find this environment useful, please consider citing our work:
+```
+@misc{OSWorld,
+      title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}, 
+      author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},
+      year={2024},
+      eprint={2404.07972},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI}
+}
+```
+
+## Acknowledgement for OSWorld-Verified
+Special thanks to the following institutions that provided feedback and participated in the fixes (as well as institutions that provided feedback during the process): [MoonShot AI, a.k.a. Kimi](https://www.moonshot.ai/)，[Human Data](https://www.hud.so/), [OpenAI](https://openai.com/), [ByteDance Seed TARS](https://seed-tars.com/), [Anthropic](https://www.anthropic.com/), [Simular](https://www.simular.ai/), [HKU Data Intelligence Lab](https://sites.google.com/view/chaoh)
+
+Special thanks to the following students who participated in the specific fixes: [Mengqi Yuan](https://yuanmengqi.github.io/), [Danyang Zhang](https://zdy023.github.io/), [Xinzhuang Xiong](https://thisisxxz.com/),  [Zhennan Shen](https://scholar.google.com/citations?user=JPwg5MwAAAAJ&hl=en), [Zilong Zhou](https://github.com/adlsdztony), Yanxu Chen, [Jiaqi Deng](https://millank0817.github.io/), [Tianbao Xie](https://tianbaoxie.com/), Junda Chen, [Jixuan Chen](https://chenjix.github.io/), [Haoyuan Wu](https://www.linkedin.com/in/haoyuan-wu-240878291/).
+
+Special thanks to the following students who participated in running the re-evaluation: [Mengqi Yuan](https://yuanmengqi.github.io/), [Zilong Zhou](https://github.com/adlsdztony), [Xinyuan Wang](https://xinyuanwangcs.github.io/), [Bowen Wang](https://bowenbryanwang.github.io/).
+
+## You might also be interested
+
+- **OSWorld-MCP**: Benchmarking MCP Tool Invocation in Computer-Use Agents. [Website](https://osworld-mcp.github.io/)
@@ -0,0 +1,452 @@
+# OSWorld Setup and Evaluation Guide
+
+This comprehensive guide covers all aspects of setting up and running OSWorld evaluations, including account configuration, proxy setup, and public evaluation platform deployment.
+
+## Table of Contents
+
+1. [Google Account Setup](#1-google-account-setup)
+2. [Proxy Configuration](#2-proxy-configuration)
+3. [Public Evaluation Platform](#3-public-evaluation-platform)
+
+---
+
+## 1. Google Account Setup
+
+For tasks including Google or Google Drive, you need a real Google account with configured OAuth2.0 secrets.
+
+> **Attention**: To prevent environment reset and result evaluation conflicts caused by multiple people using the same Google account simultaneously, please register a private Google account rather than using a shared one.
+
+### 1.1 Register A Blank Google Account
+
+1. Go to Google website and register a blank new account
+   - You do not need to provide any recovery email or phone for testing purposes
+   - **IGNORE** any security recommendations
+   - Turn **OFF** the [2-Step Verification](https://support.google.com/accounts/answer/1064203?hl=en&co=GENIE.Platform%3DDesktop#:~:text=Open%20your%20Google%20Account.,Select%20Turn%20off.) to avoid failure in environment setup
+
+<p align="center">
+  <img src="assets/googleshutoff.png" width="40%" alt="Shut Off 2-Step Verification">
+</p>
+
+> **Attention**: We strongly recommend registering a new blank account instead of using an existing one to avoid messing up your personal workspace.
+
+2. Copy and rename `settings.json.template` to `settings.json` under `evaluation_examples/settings/google/`. Replace the two fields:
+
+```json
+{
+    "email": "your_google_account@gmail.com",
+    "password": "your_google_account_password"
+}
+```
+
+### 1.2 Create A Google Cloud Project
+
+1. Navigate to [Google Cloud Project Creation](https://console.cloud.google.com/projectcreate) and create a new GCP (see [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project) for detailed steps)
+
+2. Go to the [Google Drive API console](https://console.cloud.google.com/apis/library/drive.googleapis.com?) and enable the Google Drive API for the created project (see [Enable and disable APIs](https://support.google.com/googleapi/answer/6158841?hl=en))
+
+<p align="center">
+  <img src="assets/creategcp.png" width="45%" style="margin-right: 5%;" alt="Create GCP">
+  <img src="assets/enableapi.png" width="45%" alt="Google Drive API">
+</p>
+
+### 1.3 Configure OAuth Consent Screen
+
+Go to [OAuth consent screen](https://console.cloud.google.com/apis/credentials/consent):
+
+1. Select **External** as the User Type and click **CREATE**
+
+<p align="center">
+  <img src="assets/external.png" width="80%" alt="External User Type">
+</p>
+
+2. Fill in the required fields:
+   - **App name**: Any name you prefer
+   - **User support email**: Your Google account email
+   - **Developer contact information**: Your Google account email
+   - Click **SAVE AND CONTINUE**
+
+<p align="center">
+  <img src="assets/appinfo.png" width="80%" alt="App Information">
+</p>
+
+3. Add scopes:
+   - Click **ADD OR REMOVE SCOPES**
+   - Filter and select: `https://www.googleapis.com/auth/drive`
+   - Click **UPDATE** and **SAVE AND CONTINUE**
+
+<p align="center">
+  <img src="assets/addscope.png" width="80%" alt="Add Scopes">
+</p>
+
+4. Add test users:
+   - Click **ADD USERS**
+   - Add your Google account email
+   - Click **SAVE AND CONTINUE**
+
+<p align="center">
+  <img src="assets/adduser.png" width="80%" alt="Add Test Users">
+</p>
+
+### 1.4 Create OAuth2.0 Credentials
+
+1. Go to [Credentials](https://console.cloud.google.com/apis/credentials) page
+2. Click **CREATE CREDENTIALS** → **OAuth client ID**
+3. Select **Desktop app** as Application type
+4. Name it (e.g., "OSWorld Desktop Client")
+5. Click **CREATE**
+
+<p align="center">
+  <img src="assets/createcredential.png" width="80%" alt="Create Credentials">
+</p>
+
+6. Download the JSON file and rename it to `credentials.json`
+7. Place it in `evaluation_examples/settings/google/`
+
+<p align="center">
+  <img src="assets/downloadjson.png" width="80%" alt="Download JSON">
+</p>
+
+### 1.5 Potential Issues
+
+#### Issue 1: Access Blocked During OAuth Flow
+
+**Symptom**: "Access blocked: OSWorld's request is invalid" error
+
+**Solution**: Ensure you've added your Google account as a test user in the OAuth consent screen configuration.
+
+#### Issue 2: Scope Not Granted
+
+**Symptom**: Application doesn't have necessary permissions
+
+**Solution**: Verify that `https://www.googleapis.com/auth/drive` scope is added in the OAuth consent screen.
+
+---
+
+## 2. Proxy Configuration
+
+If you're using OSWorld behind a firewall or need proxy configuration, follow these steps.
+
+### 2.1 Configure Proxy on Host Machine
+
+By default, proxy software usually listens only to localhost (`127.0.0.1`), which cannot be reached from the virtual machine. You need to make your proxy software listen to the VMware network card IP or `0.0.0.0`.
+
+#### Find VM and Host IP Addresses
+
+After launching the VM:
+
+```bash
+# Run this command on host
+# Change ws to fusion if you use VMware Fusion
+vmrun -T ws getGuestIPAddress /path/to/vmx/file
+```
+
+**On Linux (Ubuntu)**:
+```bash
+ip a  # Check IP addresses of each network card
+```
+
+**On Windows**:
+```cmd
+ipconfig  # Check IP addresses of each network card
+```
+
+Look for the VMware network card (usually named `vmnetX` like `vmnet8`). Make sure to use an IP address within the same network segment as the VM.
+
+#### Configure Proxy Software
+
+Configure your proxy software to listen on the VMware network card IP:
+
+<p align="center">
+  <img src="assets/proxysetup.png" width="80%" alt="Proxy Setup">
+</p>
+
+#### Alternative: Port Forwarding
+
+If you cannot change the listening address, set up port forwarding.
+
+**On Linux (Ubuntu)**:
+```bash
+# Forward 192.168.108.1:1080 to 127.0.0.1:1080
+socat TCP-LISTEN:1080,bind=192.168.108.1,fork TCP:127.0.0.1:1080
+```
+
+**On Windows** (with admin privileges):
+```cmd
+netsh interface portproxy add v4tov4 listenport=1080 listenaddress=192.168.108.1 connectport=1080 connectaddress=127.0.0.1
+```
+
+### 2.2 Configure Proxy in Virtual Machine
+
+#### For VMware/VirtualBox
+
+1. Start the VM and log in
+2. Open terminal and edit proxy settings:
+
+```bash
+# Edit environment variables
+sudo nano /etc/environment
+```
+
+3. Add the following lines (replace with your host IP and port):
+
+```bash
+http_proxy="http://192.168.108.1:1080"
+https_proxy="http://192.168.108.1:1080"
+no_proxy="localhost,127.0.0.1"
+```
+
+4. For APT package manager:
+
+```bash
+sudo nano /etc/apt/apt.conf.d/proxy.conf
+```
+
+Add:
+```
+Acquire::http::Proxy "http://192.168.108.1:1080";
+Acquire::https::Proxy "http://192.168.108.1:1080";
+```
+
+5. Reboot the VM or reload environment:
+
+```bash
+source /etc/environment
+```
+
+#### For Docker
+
+When using Docker provider, you can set proxy environment variables:
+
+```python
+env = DesktopEnv(
+    provider_name="docker",
+    # ... other parameters
+)
+```
+
+Set environment variables before running:
+```bash
+export HTTP_PROXY=http://your-proxy:port
+export HTTPS_PROXY=http://your-proxy:port
+```
+
+### 2.3 Proxy for Specific Tasks (Recommended)
+
+OSWorld provides built-in proxy support using DataImpulse or similar services:
+
+1. Register at [DataImpulse](https://dataimpulse.com/)
+2. Purchase a US residential IP package (approximately $1 per 1GB)
+3. Configure credentials in `evaluation_examples/settings/proxy/dataimpulse.json`:
+
+```json
+[
+    {
+        "host": "gw.dataimpulse.com",
+        "port": 823,
+        "username": "your_username",
+        "password": "your_password",
+        "protocol": "http",
+        "provider": "dataimpulse",
+        "type": "residential",
+        "country": "US",
+        "note": "Dataimpulse Residential Proxy"
+    }
+]
+```
+
+OSWorld will automatically use proxy for tasks that need it when `enable_proxy=True` in DesktopEnv.
+
+---
+
+## 3. Public Evaluation Platform
+
+We provide an AWS-based platform for large-scale parallel evaluation of OSWorld tasks.
+
+### 3.1 Architecture Overview
+
+- **Host Instance**: Central controller that stores code, configurations, and manages task execution
+- **Client Instances**: Worker nodes automatically launched to perform tasks in parallel
+
+### 3.2 Platform Deployment
+
+#### Step 1: Launch the Host Instance
+
+1. Create an EC2 instance in AWS console
+2. **Instance type recommendations**:
+   - `t3.medium`: For < 5 parallel environments
+   - `t3.large`: For < 15 parallel environments
+   - `c4.8xlarge`: For 15+ parallel environments
+3. **AMI**: Ubuntu Server 24.04 LTS (HVM), SSD Volume Type
+4. **Storage**: At least 50GB
+5. **Security group**: Open port 8080 for monitor service
+6. **VPC**: Use default (note the VPC ID for later)
+
+#### Step 2: Connect to Host Instance
+
+1. Download the `.pem` key file when creating the instance
+2. Set permissions:
+   ```bash
+   chmod 400 <your_key_file_path>
+   ```
+3. Connect via SSH:
+   ```bash
+   ssh -i <your_key_path> ubuntu@<your_public_dns>
+   ```
+
+#### Step 3: Set Up Host Machine
+
+```bash
+# Clone OSWorld repository
+git clone https://github.com/xlang-ai/OSWorld
+cd OSWorld
+
+# Optional: Create Conda environment
+# conda create -n osworld python=3.10
+# conda activate osworld
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+#### Step 4: Configure AWS Client Machines
+
+##### Security Group Configuration
+
+Create a security group with the following rules:
+
+**Inbound Rules** (8 rules required):
+
+| Type       | Protocol | Port Range | Source         | Description                |
+|------------|----------|------------|----------------|----------------------------|
+| SSH        | TCP      | 22         | 0.0.0.0/0      | SSH access                 |
+| HTTP       | TCP      | 80         | 172.31.0.0/16  | HTTP traffic               |
+| Custom TCP | TCP      | 5000       | 172.31.0.0/16  | OSWorld backend service    |
+| Custom TCP | TCP      | 5910       | 0.0.0.0/0      | NoVNC visualization port   |
+| Custom TCP | TCP      | 8006       | 172.31.0.0/16  | VNC service port           |
+| Custom TCP | TCP      | 8080       | 172.31.0.0/16  | VLC service port           |
+| Custom TCP | TCP      | 8081       | 172.31.0.0/16  | Additional service port    |
+| Custom TCP | TCP      | 9222       | 172.31.0.0/16  | Chrome control port        |
+
+**Outbound Rules** (1 rule required):
+
+| Type        | Protocol | Port Range | Destination | Description                 |
+|-------------|----------|------------|-------------|----------------------------|
+| All traffic | All      | All        | 0.0.0.0/0   | Allow all outbound traffic |
+
+Record the `AWS_SECURITY_GROUP_ID`.
+
+##### VPC and Subnet Configuration
+
+1. Note the **VPC ID** and **Subnet ID** from your host instance
+2. Record the **Subnet ID** as `AWS_SUBNET_ID`
+
+##### AWS Access Keys
+
+1. Go to AWS Console → Security Credentials
+2. Create access key
+3. Record `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
+
+### 3.3 Environment Setup
+
+#### Google Drive Integration (Optional)
+
+Follow [Section 1: Google Account Setup](#1-google-account-setup) above.
+
+**Note**: OSWorld includes 8 Google Drive tasks out of 369 total tasks. You can:
+- Complete setup for all 369 tasks, or
+- Skip Google Drive tasks and evaluate 361 tasks (officially supported)
+
+#### Set Environment Variables
+
+```bash
+# API Keys (if using)
+# export OPENAI_API_KEY="your_openai_api_key"
+# export ANTHROPIC_API_KEY="your_anthropic_api_key"
+
+# AWS Configuration
+export AWS_ACCESS_KEY_ID="your_access_key"
+export AWS_SECRET_ACCESS_KEY="your_security_access_key"
+export AWS_REGION="us-east-1"  # or your preferred region
+export AWS_SECURITY_GROUP_ID="sg-xxxx"
+export AWS_SUBNET_ID="subnet-xxxx"
+```
+
+### 3.4 Running Evaluations
+
+```bash
+# Example: Run OpenAI CUA
+python scripts/python/run_multienv_openaicua.py \
+    --headless \
+    --observation_type screenshot \
+    --model computer-use-preview \
+    --result_dir ./results_operator \
+    --test_all_meta_path evaluation_examples/test_all.json \
+    --region us-east-1 \
+    --max_steps 50 \
+    --num_envs 5 \
+    --client_password osworld-public-evaluation
+
+# Example: Run Claude (via AWS Bedrock)
+python scripts/python/run_multienv_claude.py \
+    --headless \
+    --observation_type screenshot \
+    --action_space claude_computer_use \
+    --model claude-sonnet-4-6 \
+    --result_dir ./results_claude \
+    --test_all_meta_path evaluation_examples/test_all.json \
+    --max_steps 50 \
+    --num_envs 5 \
+    --provider_name aws \
+    --client_password osworld-public-evaluation
+```
+
+**Key Parameters**:
+- `--num_envs`: Number of parallel environments
+- `--max_steps`: Maximum steps per task
+- `--result_dir`: Output directory for results
+- `--test_all_meta_path`: Path to test set metadata
+- `--region`: AWS region
+
+### 3.5 Monitoring and Results
+
+#### Web Monitoring Tool
+
+```bash
+cd monitor
+pip install -r requirements.txt
+python main.py
+```
+
+Access at: `http://<host-public-ip>:8080`
+
+#### VNC Remote Desktop Access
+
+Access VMs via VNC at: `http://<client-public-ip>:5910/vnc.html`
+
+Default password: `osworld-public-evaluation`
+
+### 3.6 Submitting Results
+
+For leaderboard submission, contact:
+- tianbaoxiexxx@gmail.com
+- yuanmengqi732@gmail.com
+
+**Options**:
+1. **Self-reported**: Submit results with monitor data and trajectories
+2. **Verified**: Schedule a meeting to run your agent code on our infrastructure
+
+---
+
+## Additional Resources
+
+- [Main README](README.md) - Project overview and quick start
+- [Installation Guide](README.md#-installation) - Detailed installation instructions
+- [FAQ](README.md#-faq) - Frequently asked questions
+- [Scripts Documentation](scripts/README.md) - Information about run scripts
+
+## Support
+
+If you encounter issues or have questions:
+- Open an issue on [GitHub](https://github.com/xlang-ai/OSWorld/issues)
+- Join our [Discord](https://discord.gg/4Gnw7eTEZR)
+- Email the maintainers (see contact information above)
@@ -0,0 +1 @@
+
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`{"steps":[{"perp_orders":{"orders":[{"coin":"ETH","tif":"Gtc","side":"buy","sz":0.02,"reduceOnly":false,"px":"mid-0.5%"}]}},{"sleep_ms":{"durationMs":150}},{"cancel_all":{"coin":"ETH"}}]}`
				`@@ -0,0 +1 @@`
				`{"steps":[{"usd_class_transfer":{"toPerp":true,"usdc":10.0}},{"set_leverage":{"coin":"ETH","leverage":5,"cross":false}},{"perp_orders":{"orders":[{"coin":"ETH","tif":"Ioc","side":"buy","sz":0.01,"reduceOnly":true,"px":"mid"}]}}]}`