tracer-cloud--opensre/tests/benchmarks

Running a benchmark

Where the corpus lives

• Hugging Face (upstream): tracer-cloud/cloud-ops-bench-dataset
• AWS S3 mirror: s3://cloud-ops-bench-dataset/<hf-revision-sha>/ — the same corpus, revision-pinned, used by the Fargate bench task at startup (faster than HF, no rate limits, no HF_TOKEN needed at runtime). Populated by make mirror-cloudopsbench-s3.

# One-time setup
make install
make download-cloudopsbench-hf      # pull the 452-scenario corpus
echo "ANTHROPIC_API_KEY=sk-..." >> .env   # plus OPENAI_API_KEY, DEEPSEEK_API_KEY as needed

# Smoke run on 5 scenarios (dev mode skips integrity gates, still calls real LLMs)
uv run python -m tests.benchmarks._framework.cli run \
    tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml --dev

# Artifacts land in .bench-results/example/<run-id>/:
#   report.json        ← machine-readable
#   report.md          ← human-readable summary
#   report.html        ← self-contained, open in any browser
#   provenance.json    ← code SHA, config content, env, model versions
#   cases/*.json       ← per-case raw artifacts

Other commands

uv run python -m tests.benchmarks._framework.cli list        # show available adapters
uv run python -m tests.benchmarks._framework.cli validate <config>   # lint config without running
uv run python -m tests.benchmarks._framework.cli report <run_dir>    # re-render md + html from report.json

Production run (real numbers, not dev mode)

Drop --dev. The framework will refuse to start unless a pre-registration file is committed at the path named in your config. See ../../docs/cloudopsbench.mdx for the full guide.

Running from GitHub CI

Trigger from Actions → "Benchmark — run on Fargate" → Run workflow. Fill in the config path and the dev_mode toggle. The workflow launches an ECS task and exits in under a minute — the actual bench runs on Fargate. Watch live logs with aws logs tail /ecs/opensre-bench --follow, or via the AWS Console under ECS → Clusters → opensre-bench → Tasks. Results land in the bench results S3 bucket under runs/<date>-<sha>/ when the task finishes.

One-time setup before the first CI run: add repo secrets ANTHROPIC_API_KEY, OPENAI_API_KEY, DEEPSEEK_API_KEY (only the ones your config needs). Workflow lives at ../../.github/workflows/benchmark-run.yml.