Files
yao-meta-skill/evals

Evals

This directory makes trigger quality and packaging quality more reproducible.

Contents:

  • trigger_cases.json: full regression set with family labels
  • train/, dev/, holdout/: split trigger suites for iterative tuning and final verification
  • semantic_config.json: local semantic-intent concepts, exclusions, and weights
  • baseline_description.txt: intentionally weaker trigger description
  • improved_description.txt: current stronger trigger description
  • sample_trigger_report.json: example comparison output using the current recommended threshold
  • failure-cases.md: current weak spots and regression targets
  • packaging_expectations.json: required packaging behaviors for supported targets
  • ../reports/: generated suite JSON plus the homepage-visible family summary panel source

Use:

python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
python3 scripts/run_eval_suite.py
python3 scripts/render_eval_dashboard.py
python3 tests/verify_failure_regressions.py
python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip
python3 tests/verify_packager_failures.py

Regression scope now includes:

  • direct positives
  • direct negatives
  • near neighbors
  • long-context positives
  • mixed-intent negatives
  • explicit "do not build a skill" negatives
  • semantic exclusion cases such as one-off, document-only, and future-outline prompts
  • paraphrase families that avoid the original wording while preserving the same trigger intent
  • long-context contamination cases where build intent or no-build intent appears after unrelated setup text
  • family-based reporting across workflow-to-skill, iterate-existing-skill, document-only, one-off, and future-outline cases
  • holdout verification