wehub-resource-sync
4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
1.7 KiB
1.7 KiB
Eval + Terminal Metrics Runbook
This runbook defines how to interpret the evaluation-process and interactive-terminal analytics emitted by the CLI.
Event Groups
- Evaluation lifecycle:
eval_process_started,eval_process_completed,eval_process_failed,eval_process_skipped,eval_process_parse_failed - Test execution lifecycle:
test_run_started,test_run_completed,test_run_failed,test_synthetic_started,test_synthetic_completed,test_synthetic_failed - Interactive terminal behavior:
terminal_actions_planned,terminal_actions_executed,terminal_turn_summarized
Core KPIs
eval_pass_rate: ratio of successful evals whereoverall_pass=trueeval_latency_p50_ms/eval_latency_p95_ms: latency percentiles fromduration_mseval_parse_error_rate: parser failures as a percentage of total eval completions/failuresterminal_action_execution_success_rate: successful deterministic action executionsterminal_fallback_rate: share of turns that required LLM fallback
Operational Guidance
- High
eval_parse_error_rategenerally points to malformed judge output. - Rising
eval_latency_p95_mswith stable p50 suggests intermittent upstream LLM delays. - High
terminal_fallback_ratewith lowplanned_countindicates missing deterministic action coverage; improve action recognizers before changing LLM prompts. - High
planned_countbut low execution success suggests command execution reliability issues (shell failures, missing dependencies, timeout thresholds).
Data Contract Source of Truth
- Event enum:
platform/analytics/events.py - Capture helpers and KPI query specs:
platform/analytics/cli.py - Provider type constraints and coercion:
platform/analytics/provider.py