yaojingang
1.7 KiB
Output Eval Method
Output Eval Lab proves whether a skill improves the final user-facing result, not only whether it routes correctly.
When To Use
Use output evals for production, library, governed, or team-distributed skills. Scaffold skills can start with one smoke case, but production and above should show a positive with-skill vs baseline signal before promotion.
Case Design
Each case should include:
- a real prompt or task shape
- any required input files
- a baseline output that represents doing the task without the skill
- a with-skill output that represents the skill-guided behavior
- assertions that can be checked without subjective guessing
- optional human review notes for taste, completeness, or judgment
Assertion Rules
Prefer assertions that catch material quality:
- required deliverable paths
- required sections or contracts
- required boundary or exclusion language
- required evidence paths
- forbidden generic placeholders
- forbidden unsafe actions
Avoid assertions that only reward wording memorization. If a case can pass by parroting one phrase while failing the real job, the assertion is too narrow.
Score Reading
The first v0 scorecard reports:
- baseline pass rate
- with-skill pass rate
- absolute delta
- failed assertions and failure taxonomy
- recommended next fixes
Production promotion should require the with-skill pass rate to beat baseline and should explain every failed assertion.
Anti-Overfitting
Keep a small public smoke set and a separate holdout set. Rotate real failures into the taxonomy instead of editing only the prompt that failed. Add near-neighbor cases whenever the output looks good but the boundary is still unclear.