yaojingang
68 lines
2.7 KiB
Markdown
68 lines
2.7 KiB
Markdown
# Benchmark Methodology
|
|
|
|
This report makes Yao Meta Skill benchmark claims auditable. It distinguishes self-eval evidence from release gates and names the limits of every comparison.
|
|
|
|
## Benchmark Types
|
|
|
|
| Type | Purpose | Evidence |
|
|
| --- | --- | --- |
|
|
| Self-eval | Fast local regression signal during authoring | Trigger suites, output eval cases, packaging checks |
|
|
| Internal blind eval | Guard against tuning only visible cases | Blind holdout, adversarial holdout, route confusion |
|
|
| External review | Check whether the workflow holds up outside the author context | Human review notes, benchmark scans, release snapshots |
|
|
|
|
## Sample Sources
|
|
|
|
- Public fixtures that can be committed and rerun.
|
|
- Real but anonymized cases when user content can be safely reduced to metadata or synthetic prompts.
|
|
- Failure library entries from prior regressions.
|
|
- Near-neighbor prompts that should not trigger the skill.
|
|
- Output cases with without-skill and with-skill artifacts.
|
|
|
|
## Evaluation Dimensions
|
|
|
|
| Dimension | Evidence |
|
|
| --- | --- |
|
|
| Trigger reliability | should-trigger, should-not-trigger, near-neighbor, route confusion |
|
|
| Output effectiveness | with-skill vs baseline delta, v1 vs v2 delta, assertion grading |
|
|
| Context efficiency | entrypoint size, references split, resource-boundary score |
|
|
| Runtime compatibility | target package structure, metadata, degradation notes |
|
|
| Trust and security | script interface, dependency pinning, permissions, secret scan, package hash |
|
|
| Governance and drift | owner, maturity, review cadence, regression history, promotion decisions |
|
|
| UX and adoption | quickstart, review viewer, report readability, reviewer handoff |
|
|
|
|
## Weighting Rule
|
|
|
|
Any weighted public score must publish:
|
|
|
|
- exact sample count
|
|
- case families
|
|
- weights per dimension
|
|
- commands used to reproduce results
|
|
- failing cases and excluded cases
|
|
- commit hash and generated artifact paths
|
|
|
|
If a score is based on local self-eval only, label it as project-level self-eval rather than external benchmark proof.
|
|
|
|
## Failure Disclosure
|
|
|
|
Every release should keep at least one representative failure when failures exist. The release note should say:
|
|
|
|
- what failed
|
|
- why it mattered
|
|
- whether the fix was trigger, output, runtime, trust, or governance related
|
|
- which test now prevents recurrence
|
|
|
|
## Reproduction
|
|
|
|
Recommended release evidence:
|
|
|
|
```bash
|
|
git rev-parse HEAD
|
|
python3 scripts/run_output_eval.py
|
|
python3 scripts/export_skill_ir.py . --output-json skill-ir/examples/yao-meta-skill.json
|
|
python3 scripts/yao.py benchmark-reproducibility .
|
|
make ci-test
|
|
```
|
|
|
|
Record generated artifacts in `reports/` and avoid comparing different runtime targets as if their capabilities were identical.
|