yaojingang
2.3 KiB
2.3 KiB
Output Execution Runs
This report records how output-eval variants were produced and whether timing or token evidence is observed or estimated.
- Cases:
5 - Variant runs:
10 - Command executed:
10 - Model executed:
0 - Recorded fixtures:
0 - Timing observed:
10 - Token observed:
0 - Token estimated:
10 - Delta:
100.0 - Gate pass:
True
No model-executed runs are recorded yet.
Use python3 scripts/yao.py output-exec --provider-runner openai or --runner-command with a reviewed provider-backed runner to replace recorded fixtures with real model output evidence.
Command runner evidence is present. This proves the eval harness executed an external command, but it is not provider-backed model evidence unless the runner reports model metadata.
Runs
| Case | Variant | Mode | Model | Duration ms | Tokens | Score | Status |
|---|---|---|---|---|---|---|---|
| skill-package-contract | baseline | command | local-output-eval-runner | 28.76 | 33 | 0.0 | pass |
| skill-package-contract | with_skill | command | local-output-eval-runner | 27.9 | 73 | 100.0 | pass |
| output-eval-expectation | baseline | command | local-output-eval-runner | 27.6 | 36 | 0.0 | pass |
| output-eval-expectation | with_skill | command | local-output-eval-runner | 28.21 | 80 | 100.0 | pass |
| ir-before-packaging | baseline | command | local-output-eval-runner | 28.73 | 33 | 0.0 | pass |
| ir-before-packaging | with_skill | command | local-output-eval-runner | 28.14 | 80 | 100.0 | pass |
| near-neighbor-boundary | baseline | command | local-output-eval-runner | 28.71 | 36 | 0.0 | pass |
| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 28.58 | 65 | 100.0 | pass |
| file-backed-governed-package | baseline | command | local-output-eval-runner | 28.49 | 37 | 0.0 | pass |
| file-backed-governed-package | with_skill | command | local-output-eval-runner | 27.46 | 98 | 100.0 | pass |
Next Fixes
- Keep recorded fixtures as reproducible baselines, but do not describe them as model-executed evidence.
- Use
scripts/provider_output_eval_runner.pyfor provider-backed holdout cases when release confidence depends on real generation behavior. - Compare timing, token cost, and assertion deltas before promoting a skill to governed reuse.