wehub-resource-sync
4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
54 lines
2.5 KiB
Markdown
54 lines
2.5 KiB
Markdown
# Running a benchmark
|
|
|
|
## Where the corpus lives
|
|
|
|
- **Hugging Face (upstream):** [tracer-cloud/cloud-ops-bench-dataset](https://huggingface.co/datasets/tracer-cloud/cloud-ops-bench-dataset)
|
|
- **AWS S3 mirror:** `s3://cloud-ops-bench-dataset/<hf-revision-sha>/` — the same corpus, revision-pinned, used by the Fargate bench task at startup (faster than HF, no rate limits, no `HF_TOKEN` needed at runtime). Populated by `make mirror-cloudopsbench-s3`.
|
|
|
|
```bash
|
|
# One-time setup
|
|
make install
|
|
make download-cloudopsbench-hf # pull the 452-scenario corpus
|
|
echo "ANTHROPIC_API_KEY=sk-..." >> .env # plus OPENAI_API_KEY, DEEPSEEK_API_KEY as needed
|
|
|
|
# Smoke run on 5 scenarios (dev mode skips integrity gates, still calls real LLMs)
|
|
uv run python -m tests.benchmarks._framework.cli run \
|
|
tests/benchmarks/cloudopsbench/configs/cloudopsbench_smoke.yml --dev
|
|
|
|
# Artifacts land in .bench-results/example/<run-id>/:
|
|
# report.json ← machine-readable
|
|
# report.md ← human-readable summary
|
|
# report.html ← self-contained, open in any browser
|
|
# provenance.json ← code SHA, config content, env, model versions
|
|
# cases/*.json ← per-case raw artifacts
|
|
```
|
|
|
|
## Other commands
|
|
|
|
```bash
|
|
uv run python -m tests.benchmarks._framework.cli list # show available adapters
|
|
uv run python -m tests.benchmarks._framework.cli validate <config> # lint config without running
|
|
uv run python -m tests.benchmarks._framework.cli report <run_dir> # re-render md + html from report.json
|
|
```
|
|
|
|
## Production run (real numbers, not dev mode)
|
|
|
|
Drop `--dev`. The framework will refuse to start unless a pre-registration
|
|
file is committed at the path named in your config. See
|
|
[../../docs/cloudopsbench.mdx](../../docs/cloudopsbench.mdx) for the full
|
|
guide.
|
|
|
|
## Running from GitHub CI
|
|
|
|
Trigger from **Actions → "Benchmark — run on Fargate" → Run workflow**. Fill in
|
|
the config path and the dev_mode toggle. The workflow launches an ECS task and
|
|
exits in under a minute — the actual bench runs on Fargate. Watch live logs
|
|
with `aws logs tail /ecs/opensre-bench --follow`, or via the AWS Console under
|
|
ECS → Clusters → opensre-bench → Tasks. Results land in the bench results S3
|
|
bucket under `runs/<date>-<sha>/` when the task finishes.
|
|
|
|
One-time setup before the first CI run: add repo secrets `ANTHROPIC_API_KEY`,
|
|
`OPENAI_API_KEY`, `DEEPSEEK_API_KEY` (only the ones your config needs).
|
|
Workflow lives at
|
|
[../../.github/workflows/benchmark-run.yml](../../.github/workflows/benchmark-run.yml).
|