yaojingang
2.7 KiB
2.7 KiB
Benchmark Methodology
This report makes Yao Meta Skill benchmark claims auditable. It distinguishes self-eval evidence from release gates and names the limits of every comparison.
Benchmark Types
| Type | Purpose | Evidence |
|---|---|---|
| Self-eval | Fast local regression signal during authoring | Trigger suites, output eval cases, packaging checks |
| Internal blind eval | Guard against tuning only visible cases | Blind holdout, adversarial holdout, route confusion |
| External review | Check whether the workflow holds up outside the author context | Human review notes, benchmark scans, release snapshots |
Sample Sources
- Public fixtures that can be committed and rerun.
- Real but anonymized cases when user content can be safely reduced to metadata or synthetic prompts.
- Failure library entries from prior regressions.
- Near-neighbor prompts that should not trigger the skill.
- Output cases with without-skill and with-skill artifacts.
Evaluation Dimensions
| Dimension | Evidence |
|---|---|
| Trigger reliability | should-trigger, should-not-trigger, near-neighbor, route confusion |
| Output effectiveness | with-skill vs baseline delta, v1 vs v2 delta, assertion grading |
| Context efficiency | entrypoint size, references split, resource-boundary score |
| Runtime compatibility | target package structure, metadata, degradation notes |
| Trust and security | script interface, dependency pinning, permissions, secret scan, package hash |
| Governance and drift | owner, maturity, review cadence, regression history, promotion decisions |
| UX and adoption | quickstart, review viewer, report readability, reviewer handoff |
Weighting Rule
Any weighted public score must publish:
- exact sample count
- case families
- weights per dimension
- commands used to reproduce results
- failing cases and excluded cases
- commit hash and generated artifact paths
If a score is based on local self-eval only, label it as project-level self-eval rather than external benchmark proof.
Failure Disclosure
Every release should keep at least one representative failure when failures exist. The release note should say:
- what failed
- why it mattered
- whether the fix was trigger, output, runtime, trust, or governance related
- which test now prevents recurrence
Reproduction
Recommended release evidence:
git rev-parse HEAD
python3 scripts/run_output_eval.py
python3 scripts/export_skill_ir.py . --output-json skill-ir/examples/yao-meta-skill.json
python3 scripts/yao.py benchmark-reproducibility .
make ci-test
Record generated artifacts in reports/ and avoid comparing different runtime targets as if their capabilities were identical.