yaojingang
48 lines
1.7 KiB
Markdown
48 lines
1.7 KiB
Markdown
# Output Eval Method
|
|
|
|
Output Eval Lab proves whether a skill improves the final user-facing result, not only whether it routes correctly.
|
|
|
|
## When To Use
|
|
|
|
Use output evals for production, library, governed, or team-distributed skills. Scaffold skills can start with one smoke case, but production and above should show a positive with-skill vs baseline signal before promotion.
|
|
|
|
## Case Design
|
|
|
|
Each case should include:
|
|
|
|
- a real prompt or task shape
|
|
- any required input files
|
|
- a baseline output that represents doing the task without the skill
|
|
- a with-skill output that represents the skill-guided behavior
|
|
- assertions that can be checked without subjective guessing
|
|
- optional human review notes for taste, completeness, or judgment
|
|
|
|
## Assertion Rules
|
|
|
|
Prefer assertions that catch material quality:
|
|
|
|
- required deliverable paths
|
|
- required sections or contracts
|
|
- required boundary or exclusion language
|
|
- required evidence paths
|
|
- forbidden generic placeholders
|
|
- forbidden unsafe actions
|
|
|
|
Avoid assertions that only reward wording memorization. If a case can pass by parroting one phrase while failing the real job, the assertion is too narrow.
|
|
|
|
## Score Reading
|
|
|
|
The first v0 scorecard reports:
|
|
|
|
- baseline pass rate
|
|
- with-skill pass rate
|
|
- absolute delta
|
|
- failed assertions and failure taxonomy
|
|
- recommended next fixes
|
|
|
|
Production promotion should require the with-skill pass rate to beat baseline and should explain every failed assertion.
|
|
|
|
## Anti-Overfitting
|
|
|
|
Keep a small public smoke set and a separate holdout set. Rotate real failures into the taxonomy instead of editing only the prompt that failed. Add near-neighbor cases whenever the output looks good but the boundary is still unclear.
|