yaojingang
42 lines
2.0 KiB
Markdown
42 lines
2.0 KiB
Markdown
# Evals
|
|
|
|
This directory makes trigger quality and packaging quality more reproducible.
|
|
|
|
Contents:
|
|
|
|
- `trigger_cases.json`: full regression set with `family` labels
|
|
- `train/`, `dev/`, `holdout/`: split trigger suites for iterative tuning and final verification
|
|
- `semantic_config.json`: local semantic-intent concepts, exclusions, and weights
|
|
- `baseline_description.txt`: intentionally weaker trigger description
|
|
- `improved_description.txt`: current stronger trigger description
|
|
- `sample_trigger_report.json`: example comparison output using the current recommended threshold
|
|
- `failure-cases.md`: current weak spots and regression targets
|
|
- `packaging_expectations.json`: required packaging behaviors for supported targets
|
|
- `../reports/`: generated suite JSON plus the homepage-visible family summary panel source
|
|
|
|
Use:
|
|
|
|
```bash
|
|
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json
|
|
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
|
|
python3 scripts/run_eval_suite.py
|
|
python3 scripts/render_eval_dashboard.py
|
|
python3 tests/verify_failure_regressions.py
|
|
python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip
|
|
python3 tests/verify_packager_failures.py
|
|
```
|
|
|
|
Regression scope now includes:
|
|
|
|
- direct positives
|
|
- direct negatives
|
|
- near neighbors
|
|
- long-context positives
|
|
- mixed-intent negatives
|
|
- explicit "do not build a skill" negatives
|
|
- semantic exclusion cases such as one-off, document-only, and future-outline prompts
|
|
- paraphrase families that avoid the original wording while preserving the same trigger intent
|
|
- long-context contamination cases where build intent or no-build intent appears after unrelated setup text
|
|
- family-based reporting across workflow-to-skill, iterate-existing-skill, document-only, one-off, and future-outline cases
|
|
- holdout verification
|