yaojingang
1.3 KiB
1.3 KiB
Output Quality Scorecard
This v0 scorecard compares static without-skill and with-skill outputs using assertion grading.
- Cases:
5 - Baseline pass rate:
0.0 - With-skill pass rate:
100.0 - Delta:
100.0 - Regressions:
0 - Blind A/B pairs:
5 - Gate pass:
True
Blind review artifacts are generated separately so reviewers can inspect A/B outputs without seeing the answer key. Run output review adjudication after reviewer decisions are recorded; pending cases should stay pending rather than being counted as human agreement.
Case Results
| Case | Baseline | With Skill | Delta | Winner | Failed With-Skill Assertions |
|---|---|---|---|---|---|
| skill-package-contract | 0.0 | 100.0 | 100.0 | with_skill | None |
| output-eval-expectation | 0.0 | 100.0 | 100.0 | with_skill | None |
| ir-before-packaging | 0.0 | 100.0 | 100.0 | with_skill | None |
| near-neighbor-boundary | 0.0 | 100.0 | 100.0 | with_skill | None |
| file-backed-governed-package | 0.0 | 100.0 | 100.0 | with_skill | None |
Failure Taxonomy
- No with-skill assertion failures.
Next Fixes
- Add holdout cases before using this as a release gate.
- Promote repeated failed assertions into the output-risk profile.
- Keep assertions tied to material deliverables, not phrasing trivia.