Evals
This directory makes trigger quality and packaging quality more reproducible.
Contents:
trigger_cases.json: full regression set withfamilylabelstrain/,dev/,holdout/: split trigger suites for iterative tuning and final verificationsemantic_config.json: local semantic-intent concepts, exclusions, and weightsbaseline_description.txt: intentionally weaker trigger descriptionimproved_description.txt: current stronger trigger descriptionsample_trigger_report.json: example comparison output using the current recommended thresholdfailure-cases.md: current weak spots and regression targetspackaging_expectations.json: required packaging behaviors for supported targets../reports/: generated suite JSON plus the homepage-visible family summary panel source
Use:
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
python3 scripts/run_eval_suite.py
python3 scripts/render_eval_dashboard.py
python3 tests/verify_failure_regressions.py
python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip
python3 tests/verify_packager_failures.py
Regression scope now includes:
- direct positives
- direct negatives
- near neighbors
- long-context positives
- mixed-intent negatives
- explicit "do not build a skill" negatives
- semantic exclusion cases such as one-off, document-only, and future-outline prompts
- paraphrase families that avoid the original wording while preserving the same trigger intent
- long-context contamination cases where build intent or no-build intent appears after unrelated setup text
- family-based reporting across workflow-to-skill, iterate-existing-skill, document-only, one-off, and future-outline cases
- holdout verification