wehub-resource-sync
4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
37 lines
1.7 KiB
Markdown
37 lines
1.7 KiB
Markdown
# Eval + Terminal Metrics Runbook
|
|
|
|
This runbook defines how to interpret the evaluation-process and interactive-terminal
|
|
analytics emitted by the CLI.
|
|
|
|
## Event Groups
|
|
|
|
- Evaluation lifecycle: `eval_process_started`, `eval_process_completed`,
|
|
`eval_process_failed`, `eval_process_skipped`, `eval_process_parse_failed`
|
|
- Test execution lifecycle: `test_run_started`, `test_run_completed`, `test_run_failed`,
|
|
`test_synthetic_started`, `test_synthetic_completed`, `test_synthetic_failed`
|
|
- Interactive terminal behavior: `terminal_actions_planned`, `terminal_actions_executed`,
|
|
`terminal_turn_summarized`
|
|
|
|
## Core KPIs
|
|
|
|
- `eval_pass_rate`: ratio of successful evals where `overall_pass=true`
|
|
- `eval_latency_p50_ms` / `eval_latency_p95_ms`: latency percentiles from `duration_ms`
|
|
- `eval_parse_error_rate`: parser failures as a percentage of total eval completions/failures
|
|
- `terminal_action_execution_success_rate`: successful deterministic action executions
|
|
- `terminal_fallback_rate`: share of turns that required LLM fallback
|
|
|
|
## Operational Guidance
|
|
|
|
- High `eval_parse_error_rate` generally points to malformed judge output.
|
|
- Rising `eval_latency_p95_ms` with stable p50 suggests intermittent upstream LLM delays.
|
|
- High `terminal_fallback_rate` with low `planned_count` indicates missing deterministic
|
|
action coverage; improve action recognizers before changing LLM prompts.
|
|
- High `planned_count` but low execution success suggests command execution reliability
|
|
issues (shell failures, missing dependencies, timeout thresholds).
|
|
|
|
## Data Contract Source of Truth
|
|
|
|
- Event enum: `platform/analytics/events.py`
|
|
- Capture helpers and KPI query specs: `platform/analytics/cli.py`
|
|
- Provider type constraints and coercion: `platform/analytics/provider.py`
|