Evals
This directory makes trigger quality and packaging quality more reproducible.
Contents:
trigger_cases.json: positive, negative, and near-neighbor promptsbaseline_description.txt: intentionally weaker trigger descriptionimproved_description.txt: current stronger trigger descriptionsample_trigger_report.json: example comparison output using threshold0.35failure-cases.md: current weak spots and regression targetspackaging_expectations.json: required packaging behaviors for supported targets
Use:
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --threshold 0.35
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt --threshold 0.35
python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip
python3 tests/verify_packager_failures.py
Regression scope now includes:
- direct positives
- direct negatives
- near neighbors
- long-context positives
- mixed-intent negatives
- explicit "do not build a skill" negatives