wehub-resource-sync
4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
69 lines
1.3 KiB
Markdown
69 lines
1.3 KiB
Markdown
# Hermes e2e suites
|
|
|
|
Hermes e2e tests execute the OpenSRE investigation pipeline against fixture-backed
|
|
Hermes evidence with `context_sources="hermes"`.
|
|
|
|
Run only Hermes e2e:
|
|
|
|
```bash
|
|
uv run pytest tests/e2e/hermes -m e2e -v
|
|
```
|
|
|
|
These tests are LLM-credential gated via `has_credentials_for_active_llm_provider()`.
|
|
|
|
|
|
## Surface Attribution (Part 5)
|
|
|
|
Part 5 extends Hermes RCA evaluation with surface attribution coverage.
|
|
|
|
Goals:
|
|
|
|
* identify the failing subsystem family
|
|
* select the closest historical analog scenario
|
|
* generate actionable diagnostic follow-up questions
|
|
|
|
### Synthetic Coverage
|
|
|
|
Primary scenario:
|
|
|
|
* `050-surface-sprawl-unknown-adapter`
|
|
|
|
Supporting components:
|
|
|
|
* `analog_registry.py`
|
|
* `surface_scoring.py`
|
|
* `adapter_tuples.json`
|
|
|
|
### Meta Evaluation
|
|
|
|
Run the attribution meta-suite:
|
|
|
|
```bash
|
|
uv run pytest tests/e2e/hermes/meta/test_surface_sprawl.py -q
|
|
```
|
|
|
|
The current attribution corpus contains 23 deterministic adapter tuples spanning:
|
|
|
|
* messaging
|
|
* provider
|
|
* runtime
|
|
* orchestration
|
|
* memory
|
|
* control
|
|
|
|
The meta-suite validates attribution consistency across all registered tuples.
|
|
|
|
### Benchmarking
|
|
|
|
Generate an offline benchmark snapshot:
|
|
|
|
```bash
|
|
uv run python -m tests.synthetic.hermes_rca.run_suite --offline-only --write-history
|
|
```
|
|
|
|
Generate a benchmark report:
|
|
|
|
```bash
|
|
uv run python -m tests.synthetic.hermes_rca.benchmark_report
|
|
```
|