diff --git a/README.md b/README.md index fdd57cd7..bf9bd9f5 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ make test ## Results - `make test` passes locally -- trigger eval: `0` false positives, `0` false negatives on the current regression set +- semantic trigger eval: `0` false positives, `0` false negatives on the current regression set - eval suite: train / dev / holdout all pass - packaging validation: `openai`, `claude`, and `generic` targets pass contract checks - packaging failure fixtures: invalid metadata, invalid YAML, and unsupported targets fail as expected @@ -110,7 +110,7 @@ Long-form material that should not bloat the main skill file. This includes desi Utility scripts that make the meta-skill operational: -- `trigger_eval.py`: evaluates trigger descriptions with positive, negative, and near-neighbor prompts +- `trigger_eval.py`: evaluates trigger descriptions with semantic intent concepts, explicit exclusions, and near-neighbor prompts - `run_eval_suite.py`: runs train/dev/holdout trigger suites and fails if aggregate regressions appear - `context_sizer.py`: estimates context weight and warns when the initial load gets too large - `cross_packager.py`: builds client-specific export artifacts with explicit platform contracts and validation @@ -122,7 +122,7 @@ Reusable trigger and packaging checks, including baseline and improved descripti ### `examples/` -Three end-to-end examples showing raw workflow input, design summary, and final generated skill shape. +End-to-end examples showing raw workflow input, design summary, and final generated skill shape, including one thicker complex benchmark and one evolution chain. ### `.github/workflows/test.yml` @@ -130,11 +130,12 @@ Continuous integration entrypoint that runs the full local regression suite on p ## Validation Notes -- Trigger evaluation is stronger than the original overlap-only version, but it is still heuristic. +- Trigger evaluation now uses a local semantic-intent model with explicit positive concepts, exclusion concepts, and boundary-case reporting. - The sample trigger report now covers a larger positive, negative, and near-neighbor set rather than a tiny demo set. - Train/dev/holdout trigger suites now separate iterative tuning from final verification. - Packaging validation now uses explicit contracts and YAML parsing, but it is still a lightweight local validation layer rather than a full platform integration suite. - `evals/failure-cases.md` captures known weak spots that should remain part of regression checks. +- `failures/` captures reusable anti-pattern writeups for routing, packaging, and authoring failures. - `tests/verify_packager_failures.py` checks that invalid metadata, invalid YAML, and unsupported targets fail clearly. ### `templates/` @@ -205,6 +206,7 @@ This project is best for: - Examples: [examples/README.md](examples/README.md) - Evals: [evals/README.md](evals/README.md) +- Failure library: [failures/README.md](failures/README.md) - Packaging contracts: [references/packaging-contracts.md](references/packaging-contracts.md) - Platform capability matrix: [references/platform-capability-matrix.md](references/platform-capability-matrix.md) - Failure fixtures: [tests/fixtures](tests/fixtures) diff --git a/evals/README.md b/evals/README.md index 7462c910..1b784ea0 100644 --- a/evals/README.md +++ b/evals/README.md @@ -6,17 +6,18 @@ Contents: - `trigger_cases.json`: positive, negative, and near-neighbor prompts - `train/`, `dev/`, `holdout/`: split trigger suites for iterative tuning and final verification +- `semantic_config.json`: local semantic-intent concepts, exclusions, and weights - `baseline_description.txt`: intentionally weaker trigger description - `improved_description.txt`: current stronger trigger description -- `sample_trigger_report.json`: example comparison output using threshold `0.35` +- `sample_trigger_report.json`: example comparison output using the current recommended threshold - `failure-cases.md`: current weak spots and regression targets - `packaging_expectations.json`: required packaging behaviors for supported targets Use: ```bash -python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --threshold 0.35 -python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt --threshold 0.35 +python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json +python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt python3 scripts/run_eval_suite.py python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip python3 tests/verify_packager_failures.py @@ -30,4 +31,5 @@ Regression scope now includes: - long-context positives - mixed-intent negatives - explicit "do not build a skill" negatives +- semantic exclusion cases such as one-off, document-only, and future-outline prompts - holdout verification diff --git a/evals/dev/trigger_cases.json b/evals/dev/trigger_cases.json index ecacefb4..2df8427c 100644 --- a/evals/dev/trigger_cases.json +++ b/evals/dev/trigger_cases.json @@ -1,5 +1,5 @@ { - "recommended_threshold": 0.37, + "recommended_threshold": 0.33, "negative_patterns": [ "one-off", "do not turn", diff --git a/evals/holdout/trigger_cases.json b/evals/holdout/trigger_cases.json index bf791bd5..a21c77df 100644 --- a/evals/holdout/trigger_cases.json +++ b/evals/holdout/trigger_cases.json @@ -1,5 +1,5 @@ { - "recommended_threshold": 0.37, + "recommended_threshold": 0.33, "negative_patterns": [ "one-off", "do not turn", diff --git a/evals/sample_trigger_report.json b/evals/sample_trigger_report.json index 159a5a9f..f8a87cbd 100644 --- a/evals/sample_trigger_report.json +++ b/evals/sample_trigger_report.json @@ -1,37 +1,1527 @@ { - "threshold": 0.42, - "threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Final scores combine token overlap, positive-token bonuses, negative-token penalties, and explicit negative-pattern penalties. Scores near the threshold should be reviewed as boundary cases.", + "threshold": 0.33, + "threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.", "false_positives": 0, "false_negatives": 0, "precision": 1.0, "recall": 1.0, "bucket_stats": { "should_trigger": { - "total": 8, - "passed": 8, + "total": 11, + "passed": 11, "pass_rate": 1.0 }, "should_not_trigger": { - "total": 7, - "passed": 7, + "total": 10, + "passed": 10, "pass_rate": 1.0 }, "near_neighbor": { - "total": 7, - "passed": 7, + "total": 10, + "passed": 10, "pass_rate": 1.0 } }, + "family_stats": { + "default": { + "total": 31, + "passed": 31, + "false_positives": 0, + "false_negatives": 0, + "pass_rate": 1.0 + } + }, + "misfires": [], + "results": { + "should_trigger": [ + { + "prompt": "Create a skill from this repeated workflow.", + "family": "default", + "score": 0.549, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.491, + "support_score": 0, + "lexical_support": 0.714, + "negative_penalty": 0, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "build_skill": [ + "create a skill" + ], + "transform_workflow": [ + "workflow" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Improve this skill description and add evals.", + "family": "default", + "score": 0.353, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "eval_optimize", + "iterate_existing_skill" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.291, + "support_score": 0, + "lexical_support": 0.571, + "negative_penalty": 0, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "eval_optimize": [ + "evals" + ], + "iterate_existing_skill": [ + "this skill", + "skill description" + ] + }, + "negative": {} + } + }, + "boundary_case": true + }, + { + "prompt": "Turn this runbook into a reusable agent skill.", + "family": "default", + "score": 0.729, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.673, + "support_score": 0, + "lexical_support": 0.625, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "build_skill": [ + "agent skill" + ], + "reuse_package": [ + "reusable" + ], + "transform_workflow": [ + "runbook" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Package this skill for team reuse.", + "family": "default", + "score": 0.448, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "iterate_existing_skill", + "reuse_package", + "team_operationalize" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.364, + "support_score": 0, + "lexical_support": 0.667, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "iterate_existing_skill": [ + "this skill", + "package this skill" + ], + "reuse_package": [ + "reuse", + "package", + "team reuse" + ], + "team_operationalize": [ + "team", + "team reuse" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Convert this operations checklist into a reusable skill.", + "family": "default", + "score": 0.415, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.364, + "support_score": 0, + "lexical_support": 0.5, + "negative_penalty": 0, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "reuse_package": [ + "reusable" + ], + "transform_workflow": [ + "checklist", + "operations checklist" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Refactor this prompt into a proper skill package.", + "family": "default", + "score": 0.532, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "reuse_package" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.491, + "support_score": 0, + "lexical_support": 0.5, + "negative_penalty": 0, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "build_skill": [ + "skill package" + ], + "reuse_package": [ + "package" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Add trigger evals to this skill before sharing it with the team.", + "family": "default", + "score": 0.388, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.327, + "support_score": 0, + "lexical_support": 0.333, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "eval_optimize": [ + "evals", + "trigger evals" + ], + "iterate_existing_skill": [ + "this skill", + "sharing it with the team", + "before sharing it with the team" + ], + "team_operationalize": [ + "team", + "sharing it with the team" + ] + }, + "negative": {} + } + }, + "boundary_case": true + }, + { + "prompt": "Create a meta-skill for packaging internal workflows.", + "family": "default", + "score": 0.591, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "reuse_package", + "team_operationalize" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.527, + "support_score": 0, + "lexical_support": 0.571, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "build_skill": [ + "meta skill", + "meta-skill" + ], + "reuse_package": [ + "packaging", + "internal workflows" + ], + "team_operationalize": [ + "internal" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "We have a messy release runbook, export process, and a prompt history; turn all of that into one reusable skill package with evals and packaging checks.", + "family": "default", + "score": 0.843, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "eval_optimize", + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.818, + "support_score": 0, + "lexical_support": 0.375, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "build_skill": [ + "skill package", + "turn all of that into one reusable skill package" + ], + "eval_optimize": [ + "evals", + "packaging checks" + ], + "reuse_package": [ + "reusable", + "package", + "packaging" + ], + "transform_workflow": [ + "runbook", + "process", + "prompt history" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Build a reusable skill from this long internal process note, and make sure it has references, scripts, and a trigger description.", + "family": "default", + "score": 0.468, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package", + "team_operationalize", + "transform_workflow" + ], + "extra_positive_concepts": [ + "multi_asset" + ], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.4, + "support_score": 0.06, + "lexical_support": 0.316, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "multi_asset": [ + "references", + "scripts" + ], + "reuse_package": [ + "reusable" + ], + "team_operationalize": [ + "internal" + ], + "transform_workflow": [ + "process", + "internal process note" + ] + }, + "negative": {} + } + } + }, + { + "prompt": "Take this existing skill draft, tighten the trigger boundary, add near-neighbor evals, and package it for distribution.", + "family": "default", + "score": 0.729, + "predicted_trigger": true, + "expected_trigger": true, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "eval_optimize", + "iterate_existing_skill", + "reuse_package", + "team_operationalize", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.691, + "support_score": 0, + "lexical_support": 0.412, + "negative_penalty": 0, + "coverage_boost": 0.06, + "concept_evidence": { + "positive": { + "eval_optimize": [ + "evals", + "near-neighbor", + "trigger boundary" + ], + "iterate_existing_skill": [ + "existing skill" + ], + "reuse_package": [ + "package", + "distribution" + ], + "team_operationalize": [ + "distribution" + ], + "transform_workflow": [ + "existing skill draft" + ] + }, + "negative": {} + } + } + } + ], + "should_not_trigger": [ + { + "prompt": "Explain what a workflow is.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "explain_only" + ], + "exclusive_negative_concepts": [ + "explain_only" + ], + "semantic_coverage": 0.182, + "support_score": 0, + "lexical_support": 0.2, + "negative_penalty": 0.26, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "transform_workflow": [ + "workflow" + ] + }, + "negative": { + "explain_only": [ + "explain what" + ] + } + } + } + }, + { + "prompt": "Write a product headline for this landing page.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "document_only" + ], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.25, + "negative_penalty": 0.18, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "document_only": [ + "headline", + "landing page" + ] + } + } + } + }, + { + "prompt": "Summarize this random note.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "summarize_only" + ], + "exclusive_negative_concepts": [ + "summarize_only" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.0, + "negative_penalty": 0.3, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "summarize_only": [ + "summarize" + ] + } + } + } + }, + { + "prompt": "Just explain what a skill is.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "explain_only" + ], + "exclusive_negative_concepts": [ + "explain_only" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.333, + "negative_penalty": 0.26, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "explain_only": [ + "explain what", + "just explain" + ] + } + } + } + }, + { + "prompt": "Draft a blog title for this article.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "document_only" + ], + "exclusive_negative_concepts": [], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.286, + "negative_penalty": 0.18, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "document_only": [ + "article", + "blog title" + ] + } + } + } + }, + { + "prompt": "Translate this README into Japanese.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "document_only", + "translate_only" + ], + "exclusive_negative_concepts": [ + "translate_only" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.25, + "negative_penalty": 0.53, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "document_only": [ + "readme" + ], + "translate_only": [ + "translate", + "into japanese" + ] + } + } + } + }, + { + "prompt": "Give me ideas for improving our process.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "brainstorm_only" + ], + "exclusive_negative_concepts": [ + "brainstorm_only" + ], + "semantic_coverage": 0.182, + "support_score": 0, + "lexical_support": 0.286, + "negative_penalty": 0.24, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "transform_workflow": [ + "process" + ] + }, + "negative": { + "brainstorm_only": [ + "ideas" + ] + } + } + } + }, + { + "prompt": "I pasted a long process description below, but I only want a summary, not a reusable skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "no_build_directive", + "summarize_only" + ], + "exclusive_negative_concepts": [ + "no_build_directive", + "summarize_only" + ], + "semantic_coverage": 0.364, + "support_score": 0, + "lexical_support": 0.308, + "negative_penalty": 0.72, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "reuse_package": [ + "reusable" + ], + "transform_workflow": [ + "process" + ] + }, + "negative": { + "no_build_directive": [ + "not a reusable skill" + ], + "summarize_only": [ + "summary", + "only want a summary" + ] + } + } + } + }, + { + "prompt": "Review this note and tell me what it means in plain English.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "explain_only" + ], + "exclusive_negative_concepts": [ + "explain_only" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.083, + "negative_penalty": 0.26, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "explain_only": [ + "tell me what it means", + "plain english" + ] + } + } + } + }, + { + "prompt": "Help me brainstorm several ways to improve packaging, but do not generate any skill files.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package" + ], + "extra_positive_concepts": [ + "multi_asset" + ], + "matched_negative_concepts": [ + "brainstorm_only", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "brainstorm_only", + "no_build_directive" + ], + "semantic_coverage": 0.182, + "support_score": 0.06, + "lexical_support": 0.2, + "negative_penalty": 0.66, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "multi_asset": [ + "files" + ], + "reuse_package": [ + "packaging" + ] + }, + "negative": { + "brainstorm_only": [ + "brainstorm", + "improve packaging" + ], + "no_build_directive": [ + "do not generate any skill files" + ] + } + } + } + } + ], + "near_neighbor": [ + { + "prompt": "Create a one-off prompt for this task.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "one_off_request" + ], + "exclusive_negative_concepts": [ + "one_off_request" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.429, + "negative_penalty": 0.24, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "one_off_request": [ + "one-off" + ] + } + } + } + }, + { + "prompt": "Help me brainstorm process ideas without building a skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "brainstorm_only", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "brainstorm_only", + "no_build_directive" + ], + "semantic_coverage": 0.182, + "support_score": 0, + "lexical_support": 0.333, + "negative_penalty": 0.66, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "transform_workflow": [ + "process" + ] + }, + "negative": { + "brainstorm_only": [ + "brainstorm", + "ideas" + ], + "no_build_directive": [ + "without building" + ] + } + } + } + }, + { + "prompt": "Improve this README but do not turn it into a skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "document_only", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "no_build_directive" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.4, + "negative_penalty": 0.6, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "document_only": [ + "readme" + ], + "no_build_directive": [ + "do not turn it into a skill" + ] + } + } + } + }, + { + "prompt": "Make a checklist for this task, but not a reusable skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "no_build_directive" + ], + "semantic_coverage": 0.364, + "support_score": 0, + "lexical_support": 0.4, + "negative_penalty": 0.42, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "reuse_package": [ + "reusable" + ], + "transform_workflow": [ + "checklist" + ] + }, + "negative": { + "no_build_directive": [ + "not a reusable skill" + ] + } + } + } + }, + { + "prompt": "Review this process note and explain it, no packaging needed.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "reuse_package", + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "explain_only", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "explain_only", + "no_build_directive" + ], + "semantic_coverage": 0.364, + "support_score": 0, + "lexical_support": 0.2, + "negative_penalty": 0.68, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "reuse_package": [ + "packaging" + ], + "transform_workflow": [ + "process" + ] + }, + "negative": { + "explain_only": [ + "explain it" + ], + "no_build_directive": [ + "no packaging needed" + ] + } + } + } + }, + { + "prompt": "Help me shape an idea before we decide whether to build a skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "brainstorm_only", + "future_outline" + ], + "exclusive_negative_concepts": [ + "brainstorm_only", + "future_outline" + ], + "semantic_coverage": 0.309, + "support_score": 0, + "lexical_support": 0.308, + "negative_penalty": 0.46, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "build_skill": [ + "build a skill" + ] + }, + "negative": { + "brainstorm_only": [ + "shape an idea" + ], + "future_outline": [ + "before we decide whether to build" + ] + } + } + } + }, + { + "prompt": "Write a custom answer for this request without creating a skill package.", + "family": "default", + "score": 0.131, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "reuse_package" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "one_off_request" + ], + "exclusive_negative_concepts": [ + "one_off_request" + ], + "semantic_coverage": 0.491, + "support_score": 0, + "lexical_support": 0.364, + "negative_penalty": 0.24, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "build_skill": [ + "skill package" + ], + "reuse_package": [ + "package" + ] + }, + "negative": { + "one_off_request": [ + "custom answer", + "for this request" + ] + } + } + } + }, + { + "prompt": "Turn this into a checklist and template, but stop short of making a full skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "transform_workflow" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "no_build_directive" + ], + "semantic_coverage": 0.182, + "support_score": 0, + "lexical_support": 0.286, + "negative_penalty": 0.42, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": { + "transform_workflow": [ + "checklist" + ] + }, + "negative": { + "no_build_directive": [ + "stop short of making a full skill" + ] + } + } + } + }, + { + "prompt": "Package this explanation as a document, not as an agent skill.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [ + "build_skill", + "reuse_package" + ], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "document_only", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "no_build_directive" + ], + "semantic_coverage": 0.491, + "support_score": 0, + "lexical_support": 0.4, + "negative_penalty": 0.6, + "coverage_boost": 0.04, + "concept_evidence": { + "positive": { + "build_skill": [ + "agent skill" + ], + "reuse_package": [ + "package" + ] + }, + "negative": { + "document_only": [ + "document" + ], + "no_build_directive": [ + "not as an agent skill" + ] + } + } + } + }, + { + "prompt": "Create an outline for a possible future skill, but do not build the skill yet.", + "family": "default", + "score": 0.0, + "predicted_trigger": false, + "expected_trigger": false, + "passed": true, + "score_detail": { + "mode": "semantic-intent", + "desired_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill", + "team_operationalize" + ], + "matched_desired_concepts": [], + "extra_positive_concepts": [], + "matched_negative_concepts": [ + "future_outline", + "no_build_directive" + ], + "exclusive_negative_concepts": [ + "future_outline", + "no_build_directive" + ], + "semantic_coverage": 0.0, + "support_score": 0, + "lexical_support": 0.357, + "negative_penalty": 0.64, + "coverage_boost": 0.0, + "concept_evidence": { + "positive": {}, + "negative": { + "future_outline": [ + "future skill", + "possible future skill", + "outline for a possible future skill" + ], + "no_build_directive": [ + "do not build" + ] + } + } + } + } + ] + }, "comparison": { "baseline_false_positives": 0, - "baseline_false_negatives": 8, + "baseline_false_negatives": 2, "improved_false_positives": 0, "improved_false_negatives": 0, "false_positive_delta": 0, - "false_negative_delta": -8, - "baseline_precision": null, + "false_negative_delta": -2, + "baseline_precision": 1.0, "improved_precision": 1.0, - "baseline_recall": 0.0, + "baseline_recall": 0.818, "improved_recall": 1.0 } } diff --git a/evals/semantic_config.json b/evals/semantic_config.json new file mode 100644 index 00000000..47cce4c4 --- /dev/null +++ b/evals/semantic_config.json @@ -0,0 +1,194 @@ +{ + "fallback_positive_concepts": [ + "build_skill", + "transform_workflow", + "reuse_package", + "eval_optimize", + "iterate_existing_skill" + ], + "positive_concepts": { + "build_skill": { + "weight": 0.34, + "phrases": [ + "create a skill", + "build a skill", + "make a skill", + "generate a skill", + "agent skill", + "skill package", + "meta skill", + "meta-skill", + "turn this into a skill", + "turn all of that into one reusable skill package" + ] + }, + "transform_workflow": { + "weight": 0.2, + "phrases": [ + "workflow", + "runbook", + "process", + "checklist", + "prompt history", + "operations checklist", + "internal process note", + "existing skill draft", + "rough notes" + ] + }, + "reuse_package": { + "weight": 0.2, + "phrases": [ + "reusable", + "reuse", + "package", + "packaging", + "distribution", + "share it with the team", + "team reuse", + "internal workflows", + "production-ready" + ] + }, + "eval_optimize": { + "weight": 0.16, + "phrases": [ + "evals", + "trigger evals", + "near-neighbor", + "trigger boundary", + "benchmark", + "validation", + "validate", + "packaging checks" + ] + }, + "iterate_existing_skill": { + "weight": 0.16, + "phrases": [ + "improve an existing skill", + "existing skill", + "this skill", + "skill description", + "optimize skill triggering", + "package this skill", + "sharing it with the team", + "before sharing it with the team" + ] + }, + "multi_asset": { + "weight": 0.06, + "phrases": [ + "references", + "scripts", + "manifest", + "folder", + "directory", + "files" + ] + }, + "team_operationalize": { + "weight": 0.04, + "phrases": [ + "team", + "internal", + "standardize", + "shared", + "distribution", + "sharing it with the team", + "team reuse" + ] + } + }, + "negative_concepts": { + "explain_only": { + "weight": 0.26, + "exclusive": true, + "phrases": [ + "explain what", + "just explain", + "explain it", + "tell me what it means", + "plain english" + ] + }, + "summarize_only": { + "weight": 0.3, + "exclusive": true, + "phrases": [ + "summary", + "summarize", + "only want a summary", + "recap" + ] + }, + "translate_only": { + "weight": 0.35, + "exclusive": true, + "phrases": [ + "translate", + "into japanese", + "into french", + "into russian" + ] + }, + "brainstorm_only": { + "weight": 0.24, + "exclusive": true, + "phrases": [ + "brainstorm", + "ideas", + "improve packaging", + "shape an idea" + ] + }, + "one_off_request": { + "weight": 0.24, + "exclusive": true, + "phrases": [ + "one-off", + "custom answer", + "for this request", + "just this task" + ] + }, + "no_build_directive": { + "weight": 0.42, + "exclusive": true, + "phrases": [ + "do not build", + "don't build", + "without building", + "do not generate any skill files", + "not a reusable skill", + "not as an agent skill", + "do not turn it into a skill", + "stop short of making a full skill", + "not a skill yet", + "no packaging needed" + ] + }, + "document_only": { + "weight": 0.18, + "exclusive": false, + "phrases": [ + "readme", + "document", + "article", + "blog title", + "headline", + "landing page" + ] + }, + "future_outline": { + "weight": 0.22, + "exclusive": true, + "phrases": [ + "future skill", + "possible future skill", + "before we decide whether to build", + "outline for a possible future skill" + ] + } + } +} diff --git a/evals/train/trigger_cases.json b/evals/train/trigger_cases.json index aa11f6cd..1e8095f6 100644 --- a/evals/train/trigger_cases.json +++ b/evals/train/trigger_cases.json @@ -1,5 +1,5 @@ { - "recommended_threshold": 0.37, + "recommended_threshold": 0.33, "negative_patterns": [ "one-off", "do not turn", diff --git a/evals/trigger_cases.json b/evals/trigger_cases.json index 6c992637..d41a2732 100644 --- a/evals/trigger_cases.json +++ b/evals/trigger_cases.json @@ -1,5 +1,5 @@ { - "recommended_threshold": 0.37, + "recommended_threshold": 0.33, "negative_patterns": [ "one-off", "do not turn", diff --git a/examples/README.md b/examples/README.md index 97479134..afb30736 100644 --- a/examples/README.md +++ b/examples/README.md @@ -14,6 +14,13 @@ Each example contains: - `design-summary.md`: how the meta-skill scopes and structures the result - `generated-skill/`: the resulting skill package shape +The complex example is intentionally thicker than the others and includes: + +- `manifest.json` +- `input/` and `outputs/` sample artifacts +- `evals/trigger_cases.json` +- richer references and a deterministic script + Additional evolution example: - `evolution-frontend-review/`: shows raw workflow, v0, v1, final, and eval delta diff --git a/examples/complex-release-orchestrator/design-summary.md b/examples/complex-release-orchestrator/design-summary.md index b436af87..b24e329b 100644 --- a/examples/complex-release-orchestrator/design-summary.md +++ b/examples/complex-release-orchestrator/design-summary.md @@ -2,6 +2,8 @@ - Skill tier: complex - Trigger surface: release prep, release checklist, rollout coordination, rollback planning -- Deterministic content: checklist stages and required artifacts -- Flexible content: risk commentary and rollout recommendation -- Package choice: `SKILL.md`, `agents/interface.yaml`, `references/`, `scripts/`, `evals/` +- Deterministic content: checklist stages, migration prompts, packet sections, and stakeholder message scaffolds +- Flexible content: risk commentary, go/no-go judgment, and escalation notes +- Package choice: `SKILL.md`, `agents/interface.yaml`, `manifest.json`, `references/`, `scripts/`, `evals/`, `input/`, `outputs/` +- Boundary rule: use for release packet creation and readiness review, not for general project planning or one-off announcements +- Validation goal: the generated packet must contain release scope, migration notes, rollout plan, rollback triggers, stakeholder communication, and an explicit decision block diff --git a/examples/complex-release-orchestrator/generated-skill/SKILL.md b/examples/complex-release-orchestrator/generated-skill/SKILL.md index d57a7b84..cf2ea370 100644 --- a/examples/complex-release-orchestrator/generated-skill/SKILL.md +++ b/examples/complex-release-orchestrator/generated-skill/SKILL.md @@ -1,18 +1,68 @@ --- name: release-orchestrator -description: Coordinate software release preparation, rollout readiness, migration notes, rollback planning, and stakeholder communication. Use when asked to prepare a release, build a rollout checklist, review release risk, or organize launch readiness. +description: Coordinate software release preparation, rollout readiness, migration notes, rollback planning, stakeholder communication, and final go/no-go release packets. Use when asked to prepare a release packet, review release readiness, or turn scattered launch notes into one reusable release workflow. Do not use for generic project planning, one-off announcement drafting, or high-level release theory. --- # Release Orchestrator +## When To Use + +- You need one reusable release packet from scattered launch inputs. +- You need to verify migration notes, rollout steps, rollback triggers, and stakeholder messages before release approval. +- You need a go/no-go summary with explicit blockers. + +## Do Not Use + +- The request is only to explain a release concept. +- The request is only to write a single announcement. +- The request is still brainstorming and not yet packaging a repeatable release workflow. + ## Workflow -1. Gather release inputs: changes, risks, migrations, dependencies. -2. Build artifacts for changelog, rollout, rollback, and announcements. -3. Flag missing release-critical information before declaring readiness. -4. Output a release package with clear go/no-go signals. +1. Gather the minimum release inputs: + - release scope + - risky changes + - migrations + - rollout sequencing + - rollback triggers + - communication audiences +2. Read `references/release-checklist.md` and `references/risk-matrix.md` to determine required sections and escalation thresholds. +3. Read `references/migration-template.md` and `references/stakeholder-comms.md` when migrations or communications are present. +4. Use `scripts/build_release_packet.py` with `input/release_input_example.json` as the shape reference when deterministic packet assembly is needed. +5. Produce one release packet with: + - release summary + - dependency and migration section + - rollout steps + - rollback triggers + - stakeholder communication plan + - explicit go/no-go decision +6. If release-critical data is missing, stop and report blockers instead of pretending readiness. + +## Output Contract + +The final answer must include: + +- a concise scope summary +- a risk block with severity +- migration notes or a clear "none required" +- rollout plan +- rollback plan +- stakeholder communication block +- a final decision: `GO`, `GO WITH CONDITIONS`, or `NO-GO` + +## Validation Checklist + +- Release-critical inputs are either present or explicitly flagged as missing. +- Rollout and rollback are both covered. +- Migration notes are not omitted when schema or data changes exist. +- Stakeholder communication is tailored by audience and timing. +- The final decision is explicit and justified. ## Reference Map -- Read `references/release-checklist.md` for the stage checklist. -- Use `scripts/build_release_packet.py` for deterministic artifact assembly. +- Read `references/release-checklist.md` for required packet sections. +- Read `references/risk-matrix.md` for decision thresholds. +- Read `references/migration-template.md` for migration note structure. +- Read `references/stakeholder-comms.md` for announcement scaffolds. +- Inspect `evals/trigger_cases.json` for trigger boundaries. +- Inspect `outputs/release_packet_example.md` for an example final artifact. diff --git a/examples/complex-release-orchestrator/generated-skill/agents/interface.yaml b/examples/complex-release-orchestrator/generated-skill/agents/interface.yaml index baa1410f..943693f3 100644 --- a/examples/complex-release-orchestrator/generated-skill/agents/interface.yaml +++ b/examples/complex-release-orchestrator/generated-skill/agents/interface.yaml @@ -1,4 +1,4 @@ interface: display_name: "Release Orchestrator" - short_description: "Organize release readiness, rollout, and rollback artifacts" - default_prompt: "Use $release-orchestrator to prepare this release package and rollout checklist." + short_description: "Build reusable release packets with rollout, rollback, migration, and communication coverage" + default_prompt: "Use $release-orchestrator to turn these release notes into a full release packet with go/no-go guidance." diff --git a/examples/complex-release-orchestrator/generated-skill/evals/trigger_cases.json b/examples/complex-release-orchestrator/generated-skill/evals/trigger_cases.json new file mode 100644 index 00000000..4734f149 --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/evals/trigger_cases.json @@ -0,0 +1,16 @@ +{ + "recommended_threshold": 0.33, + "should_trigger": [ + "Prepare a release packet from these migration notes, rollback conditions, and rollout steps.", + "Review release readiness and produce a go/no-go packet for this launch.", + "Turn these scattered release notes into a reusable release workflow." + ], + "should_not_trigger": [ + "Explain what a rollback plan is.", + "Draft one short launch announcement for customers." + ], + "near_neighbor": [ + "Brainstorm some release ideas before we decide whether to package the workflow.", + "Summarize these release notes without creating a reusable workflow." + ] +} diff --git a/examples/complex-release-orchestrator/generated-skill/input/release_input_example.json b/examples/complex-release-orchestrator/generated-skill/input/release_input_example.json new file mode 100644 index 00000000..36401cb2 --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/input/release_input_example.json @@ -0,0 +1,31 @@ +{ + "release_name": "2026.03.31-search-hardening", + "scope_summary": "Adds query guardrails, dependency upgrades, and an index migration for the search service.", + "risks": [ + { + "severity": "high", + "summary": "Index migration must complete before traffic shift." + }, + { + "severity": "medium", + "summary": "Cache warm-up may temporarily increase search latency." + } + ], + "migrations": "Run search_index_v7 migration before rollout. Rollback is safe only before background reindex completion.", + "rollout_steps": [ + "Run migration in primary region.", + "Verify index health and shard count.", + "Shift 10 percent traffic and monitor search latency.", + "Shift remaining traffic after 15 minutes of stable metrics." + ], + "rollback_triggers": [ + "Search error rate above 2 percent for 5 minutes.", + "Median latency increases above 400ms after warm-up window." + ], + "stakeholder_messages": { + "engineering-oncall": "Monitor search dashboards during the first 15 minutes after traffic shift.", + "support": "Watch for search degradation tickets during the release window.", + "leadership": "Release is go-with-conditions because migration timing is critical." + }, + "decision": "GO WITH CONDITIONS" +} diff --git a/examples/complex-release-orchestrator/generated-skill/manifest.json b/examples/complex-release-orchestrator/generated-skill/manifest.json new file mode 100644 index 00000000..3cf42072 --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/manifest.json @@ -0,0 +1,15 @@ +{ + "name": "release-orchestrator", + "version": "1.0.0", + "owner": "Yao Team", + "updated_at": "2026-03-31", + "category": "operations", + "complexity_tier": "complex", + "factory_components": [ + "references", + "scripts", + "evals", + "input", + "outputs" + ] +} diff --git a/examples/complex-release-orchestrator/generated-skill/outputs/release_packet_example.md b/examples/complex-release-orchestrator/generated-skill/outputs/release_packet_example.md new file mode 100644 index 00000000..9c72471d --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/outputs/release_packet_example.md @@ -0,0 +1,29 @@ +# Release Packet: 2026.03.31-search-hardening + +## Scope +Adds query guardrails, dependency upgrades, and an index migration for the search service. + +## Risks +- [HIGH] Index migration must complete before traffic shift. +- [MEDIUM] Cache warm-up may temporarily increase search latency. + +## Migrations +Run search_index_v7 migration before rollout. Rollback is safe only before background reindex completion. + +## Rollout +- Run migration in primary region. +- Verify index health and shard count. +- Shift 10 percent traffic and monitor search latency. +- Shift remaining traffic after 15 minutes of stable metrics. + +## Rollback +- Search error rate above 2 percent for 5 minutes. +- Median latency increases above 400ms after warm-up window. + +## Stakeholder Communication +- engineering-oncall: Monitor search dashboards during the first 15 minutes after traffic shift. +- support: Watch for search degradation tickets during the release window. +- leadership: Release is go-with-conditions because migration timing is critical. + +## Final Decision +GO WITH CONDITIONS diff --git a/examples/complex-release-orchestrator/generated-skill/references/migration-template.md b/examples/complex-release-orchestrator/generated-skill/references/migration-template.md new file mode 100644 index 00000000..02841d5c --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/references/migration-template.md @@ -0,0 +1,22 @@ +# Migration Template + +Use this shape whenever the release changes schema, data contracts, or runtime configuration. + +## Required Fields + +- migration type +- affected service or datastore +- execution timing +- rollback expectation +- operator warning + +## Template + +```md +### Migration +- Type: +- Affected Surface: +- Run Before Rollout: +- Rollback Safe: +- Operator Warning: +``` diff --git a/examples/complex-release-orchestrator/generated-skill/references/release-checklist.md b/examples/complex-release-orchestrator/generated-skill/references/release-checklist.md index db62b49d..e32bf559 100644 --- a/examples/complex-release-orchestrator/generated-skill/references/release-checklist.md +++ b/examples/complex-release-orchestrator/generated-skill/references/release-checklist.md @@ -1,8 +1,31 @@ # Release Checklist -- change summary +## Required Sections + +- release summary - dependency review - migration notes - rollout checklist - rollback checklist - stakeholder communication +- final go/no-go decision + +## Release Summary Questions + +- What changed in this release? +- Which services, schemas, or integrations are affected? +- What is the blast radius if rollout fails? + +## Rollout Gate + +- Sequencing is clear. +- Ownership is explicit. +- Monitoring signal is defined. +- Rollback trigger is concrete, not vague. + +## Packet Failure Modes + +- Missing migration notes when data changes exist +- Missing rollback trigger +- "Go" recommendation without acknowledged risk +- Stakeholder message missing audience or send timing diff --git a/examples/complex-release-orchestrator/generated-skill/references/risk-matrix.md b/examples/complex-release-orchestrator/generated-skill/references/risk-matrix.md new file mode 100644 index 00000000..b0cfd517 --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/references/risk-matrix.md @@ -0,0 +1,14 @@ +# Risk Matrix + +## Severity Guide + +- `low`: internal-only changes, no migrations, rollback is trivial +- `medium`: user-facing changes with limited blast radius or known manual recovery +- `high`: schema changes, cross-service dependencies, or rollback with coordination cost +- `critical`: irreversible data impact, unclear rollback, or no monitoring confidence + +## Decision Rules + +- `GO`: no unresolved high/critical risks and rollback path is clear +- `GO WITH CONDITIONS`: risks remain but owners, mitigations, and gates are explicit +- `NO-GO`: missing migration details, missing rollback criteria, or critical unknowns diff --git a/examples/complex-release-orchestrator/generated-skill/references/stakeholder-comms.md b/examples/complex-release-orchestrator/generated-skill/references/stakeholder-comms.md new file mode 100644 index 00000000..cb75bf7a --- /dev/null +++ b/examples/complex-release-orchestrator/generated-skill/references/stakeholder-comms.md @@ -0,0 +1,21 @@ +# Stakeholder Communication + +## Audiences + +- engineering on-call +- support +- customer-facing operations +- internal leadership + +## Message Rules + +- state release window +- state impact surface +- state expected operator action or "no action required" +- state rollback or escalation signal when risk is non-trivial + +## Example Format + +- Engineering on-call: release window, dashboards to watch, rollback trigger +- Support: user-visible changes, expected incidents, escalation owner +- Leadership: scope, major risk, final decision diff --git a/examples/complex-release-orchestrator/generated-skill/scripts/build_release_packet.py b/examples/complex-release-orchestrator/generated-skill/scripts/build_release_packet.py index 179c287d..c67ef7b1 100644 --- a/examples/complex-release-orchestrator/generated-skill/scripts/build_release_packet.py +++ b/examples/complex-release-orchestrator/generated-skill/scripts/build_release_packet.py @@ -1,2 +1,63 @@ #!/usr/bin/env python3 -print("Build release packet from gathered inputs.") +import json +import sys +from pathlib import Path + + +def render_packet(payload: dict) -> str: + lines = [ + f"# Release Packet: {payload['release_name']}", + "", + "## Scope", + payload["scope_summary"], + "", + "## Risks", + ] + for risk in payload["risks"]: + lines.append(f"- [{risk['severity'].upper()}] {risk['summary']}") + lines.extend( + [ + "", + "## Migrations", + payload["migrations"], + "", + "## Rollout", + ] + ) + for step in payload["rollout_steps"]: + lines.append(f"- {step}") + lines.extend( + [ + "", + "## Rollback", + ] + ) + for trigger in payload["rollback_triggers"]: + lines.append(f"- {trigger}") + lines.extend( + [ + "", + "## Stakeholder Communication", + ] + ) + for audience, message in payload["stakeholder_messages"].items(): + lines.append(f"- {audience}: {message}") + lines.extend( + [ + "", + "## Final Decision", + payload["decision"], + ] + ) + return "\n".join(lines) + "\n" + + +def main() -> None: + if len(sys.argv) != 2: + raise SystemExit("Usage: build_release_packet.py ") + payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) + print(render_packet(payload), end="") + + +if __name__ == "__main__": + main() diff --git a/examples/complex-release-orchestrator/raw-workflow.md b/examples/complex-release-orchestrator/raw-workflow.md index e5dd69de..461f11fb 100644 --- a/examples/complex-release-orchestrator/raw-workflow.md +++ b/examples/complex-release-orchestrator/raw-workflow.md @@ -1,12 +1,27 @@ # Raw Workflow -Every release, we coordinate: +Every release we have to pull information from too many places and rebuild the same release packet by hand. -- changelog preparation -- dependency diff review -- migration notes -- rollout checklist -- rollback checklist -- stakeholder announcement +Current inputs are scattered across: -This is currently spread across chat messages, wiki pages, and ad hoc reminders. +- merged PR summaries from the current milestone +- dependency update notes +- migration docs from backend and data teams +- rollout sequencing notes from on-call +- rollback conditions kept in an old runbook +- stakeholder announcements drafted in chat + +What usually goes wrong: + +- a migration note is missing when we think the release is ready +- rollback criteria are vague +- stakeholder communication is written too late +- someone says "ship it" before the packet has a clear go/no-go decision + +What we want from the skill: + +- ask for the minimum required release inputs +- produce one consistent release packet +- force a go/no-go summary +- call out missing release-critical details before approval +- keep deterministic artifacts separate from judgment calls diff --git a/failures/README.md b/failures/README.md new file mode 100644 index 00000000..1fa386a1 --- /dev/null +++ b/failures/README.md @@ -0,0 +1,16 @@ +# Failure And Anti-Pattern Library + +This directory captures recurring ways a skill can fail before, during, or after packaging. + +Use it for three things: + +- regression design +- author training +- description and boundary tuning + +Current case library: + +- `one-off-vs-reusable` +- `explain-not-package` +- `document-export-vs-agent-skill` +- `future-outline-vs-build` diff --git a/failures/document-export-vs-agent-skill/README.md b/failures/document-export-vs-agent-skill/README.md new file mode 100644 index 00000000..058bd2b7 --- /dev/null +++ b/failures/document-export-vs-agent-skill/README.md @@ -0,0 +1,28 @@ +# Document Export Vs Agent Skill + +## Symptom + +The request asks to package content as a document or improve a document, but the system mistakes it for agent-skill creation. + +## Anti-Pattern + +Reading "package" as "skill package" without checking the target artifact. + +## Example Prompt + +`Package this explanation as a document, not as an agent skill.` + +## Why It Fails + +- "package" is overloaded +- release notes, runbooks, and docs often look structurally similar to skills + +## Fix + +- model target artifact explicitly +- treat document-only exports as negative route evidence + +## Regression Prompts + +- `Improve this README but do not turn it into a skill.` +- `Package this explanation as a document, not as an agent skill.` diff --git a/failures/explain-not-package/README.md b/failures/explain-not-package/README.md new file mode 100644 index 00000000..405cb790 --- /dev/null +++ b/failures/explain-not-package/README.md @@ -0,0 +1,28 @@ +# Explain Not Package + +## Symptom + +The request is asking for explanation or interpretation, but the skill creator tries to package it as a reusable workflow. + +## Anti-Pattern + +Confusing topic adjacency with packaging intent. + +## Example Prompt + +`Review this note and tell me what it means in plain English.` + +## Why It Fails + +- the source material may look like a workflow +- the user still wants explanation, not systematization + +## Fix + +- treat explanation, summary-only, and translation-only modes as direct exclusions +- require positive evidence of reusable packaging intent before triggering + +## Regression Prompts + +- `Explain what a workflow is.` +- `Review this process note and explain it, no packaging needed.` diff --git a/failures/future-outline-vs-build/README.md b/failures/future-outline-vs-build/README.md new file mode 100644 index 00000000..9160379c --- /dev/null +++ b/failures/future-outline-vs-build/README.md @@ -0,0 +1,28 @@ +# Future Outline Vs Build + +## Symptom + +The user is still framing or exploring a future skill, but the router prematurely triggers full skill generation. + +## Anti-Pattern + +Treating early planning language as execution-ready packaging intent. + +## Example Prompt + +`Create an outline for a possible future skill, but do not build the skill yet.` + +## Why It Fails + +- the prompt mentions "skill" +- the actual task is pre-build design, not packaging + +## Fix + +- detect "future skill", "outline", "before we decide", and similar wording as planning-only exclusions +- require either present-tense packaging intent or explicit build intent + +## Regression Prompts + +- `Help me shape an idea before we decide whether to build a skill.` +- `Create an outline for a possible future skill, but do not build the skill yet.` diff --git a/failures/one-off-vs-reusable/README.md b/failures/one-off-vs-reusable/README.md new file mode 100644 index 00000000..bd515a49 --- /dev/null +++ b/failures/one-off-vs-reusable/README.md @@ -0,0 +1,28 @@ +# One-Off Vs Reusable + +## Symptom + +The router sees strong packaging language and triggers even though the user only wants a one-off answer or prompt. + +## Anti-Pattern + +Treating any "create" or "package" wording as reusable-skill intent. + +## Example Prompt + +`Write a custom answer for this request without creating a skill package.` + +## Why It Fails + +- lexical overlap is high +- true user intent is single-use output, not reusable infrastructure + +## Fix + +- detect one-off language explicitly +- treat "custom answer", "one-off", and similar phrases as exclusion concepts + +## Regression Prompts + +- `Create a one-off prompt for this task.` +- `Write a custom answer for this request without creating a skill package.` diff --git a/scripts/run_eval_suite.py b/scripts/run_eval_suite.py index 2134e6e8..daf873c2 100644 --- a/scripts/run_eval_suite.py +++ b/scripts/run_eval_suite.py @@ -17,6 +17,8 @@ def run_case(script: Path, description: Path, baseline: Path, cases: Path) -> di str(baseline), "--cases", str(cases), + "--semantic-config", + str(Path("evals/semantic_config.json").resolve()), ], capture_output=True, text=True, @@ -27,7 +29,7 @@ def run_case(script: Path, description: Path, baseline: Path, cases: Path) -> di def main() -> None: - parser = argparse.ArgumentParser(description="Run trigger evaluation across train/dev/holdout suites.") + parser = argparse.ArgumentParser(description="Run semantic trigger evaluation across train/dev/holdout suites.") parser.add_argument("--eval-dir", default="evals", help="Root eval directory") parser.add_argument("--description-file", default="evals/improved_description.txt") parser.add_argument("--baseline-description-file", default="evals/baseline_description.txt") diff --git a/scripts/trigger_eval.py b/scripts/trigger_eval.py index bdced925..5bcbcbef 100644 --- a/scripts/trigger_eval.py +++ b/scripts/trigger_eval.py @@ -3,17 +3,23 @@ import argparse import json import re from pathlib import Path -from collections import Counter -WORD_RE = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9_-]*") +WORD_RE = re.compile(r"[a-z0-9][a-z0-9_-]*") +DEFAULT_CONFIG_PATH = Path("evals/semantic_config.json") + + +def normalize(text: str) -> str: + text = text.lower() + text = re.sub(r"[^a-z0-9]+", " ", text) + return re.sub(r"\s+", " ", text).strip() def words(text: str) -> set[str]: return {w.lower() for w in WORD_RE.findall(text)} -def load_cases(path: Path) -> dict: +def load_json(path: Path) -> dict: return json.loads(path.read_text(encoding="utf-8")) @@ -30,87 +36,141 @@ def extract_description(text: str) -> str: return text -def score_prompt(description_words: set[str], prompt: str) -> float: +def iter_case_items(cases: dict, bucket: str) -> list[dict]: + items = [] + for raw in cases.get(bucket, []): + if isinstance(raw, str): + items.append({"text": raw, "family": "default"}) + else: + item = dict(raw) + item.setdefault("family", "default") + items.append(item) + return items + + +def phrase_present(text: str, phrase: str) -> bool: + phrase = normalize(phrase) + if not phrase: + return False + return f" {phrase} " in f" {text} " + + +def load_semantic_config(path: Path | None) -> dict: + config_path = path or DEFAULT_CONFIG_PATH + if not config_path.exists(): + raise SystemExit(f"Semantic config not found: {config_path}") + return load_json(config_path) + + +def collect_concept_hits(text: str, concepts: dict[str, dict]) -> dict[str, dict]: + normalized = normalize(text) + hits: dict[str, dict] = {} + for name, spec in concepts.items(): + matched = [] + for phrase in spec.get("phrases", []): + if phrase_present(normalized, phrase): + matched.append(phrase) + if matched: + hits[name] = { + "weight": spec["weight"], + "matched_phrases": matched, + "exclusive": bool(spec.get("exclusive")), + } + return hits + + +def lexical_support(description_words: set[str], prompt: str) -> float: prompt_words = words(prompt) if not prompt_words: return 0.0 - overlap = description_words & prompt_words - return len(overlap) / len(prompt_words) + return len(description_words & prompt_words) / len(prompt_words) -def token_frequencies(cases: dict, buckets: tuple[str, ...]) -> Counter: - freq: Counter = Counter() - for bucket in buckets: - for prompt in cases.get(bucket, []): - freq.update(words(prompt)) - return freq +def desired_positive_concepts(description: str, config: dict) -> list[str]: + description_hits = collect_concept_hits(description, config["positive_concepts"]) + names = list(description_hits) + if names: + return names + return config.get("fallback_positive_concepts", []) -def compile_negative_patterns(cases: dict) -> list[re.Pattern[str]]: - return [re.compile(pattern, re.IGNORECASE) for pattern in cases.get("negative_patterns", [])] +def score_prompt_semantic(description: str, prompt: str, config: dict) -> tuple[float, dict]: + positive_concepts = config["positive_concepts"] + negative_concepts = config["negative_concepts"] + desired = desired_positive_concepts(description, config) + desired_weight_total = sum(positive_concepts[name]["weight"] for name in desired) or 1.0 + prompt_positive_hits = collect_concept_hits(prompt, positive_concepts) + prompt_negative_hits = collect_concept_hits(prompt, negative_concepts) -def score_prompt_weighted(description_words: set[str], prompt: str, positive_freq: Counter, negative_freq: Counter, negative_patterns: list[re.Pattern[str]]) -> tuple[float, dict]: - prompt_words = words(prompt) - if not prompt_words: - return 0.0, {"matched_positive_tokens": [], "matched_negative_tokens": [], "matched_negative_patterns": []} + matched_desired = sorted([name for name in desired if name in prompt_positive_hits]) + extra_positive = sorted([name for name in prompt_positive_hits if name not in matched_desired]) + semantic_coverage = sum(positive_concepts[name]["weight"] for name in matched_desired) / desired_weight_total + support_score = sum(positive_concepts[name]["weight"] for name in extra_positive) - overlap = description_words & prompt_words - base_score = len(overlap) / len(prompt_words) + exclusive_negative = sorted([name for name, hit in prompt_negative_hits.items() if hit["exclusive"]]) + negative_penalty = sum(hit["weight"] for hit in prompt_negative_hits.values()) + lexical = lexical_support(words(description), prompt) - weighted_bonus = 0.0 - matched_positive_tokens = [] - matched_negative_tokens = [] - for token in overlap: - pos = positive_freq.get(token, 0) - neg = negative_freq.get(token, 0) - if pos > neg: - weighted_bonus += 0.06 - matched_positive_tokens.append(token) + coverage_boost = 0.0 + if len(matched_desired) >= 2: + coverage_boost += 0.04 + if len(matched_desired) >= 3: + coverage_boost += 0.02 - weighted_penalty = 0.0 - for token in prompt_words: - neg = negative_freq.get(token, 0) - pos = positive_freq.get(token, 0) - if neg > pos and token not in overlap: - weighted_penalty += 0.04 - matched_negative_tokens.append(token) + score = (semantic_coverage * 0.92) + min(0.12, support_score * 0.25) + min(0.06, lexical * 0.08) + coverage_boost + score -= negative_penalty + if exclusive_negative and semantic_coverage < 0.9: + score -= 0.15 + score = max(0.0, min(1.0, score)) - matched_negative_patterns = [pattern.pattern for pattern in negative_patterns if pattern.search(prompt)] - pattern_penalty = 0.18 * len(matched_negative_patterns) - - score = max(0.0, min(1.0, base_score + weighted_bonus - weighted_penalty - pattern_penalty)) - return score, { - "matched_positive_tokens": sorted(set(matched_positive_tokens)), - "matched_negative_tokens": sorted(set(matched_negative_tokens)), - "matched_negative_patterns": matched_negative_patterns, - "base_score": round(base_score, 3), - "weighted_bonus": round(weighted_bonus, 3), - "weighted_penalty": round(weighted_penalty + pattern_penalty, 3), + score_detail = { + "mode": "semantic-intent", + "desired_positive_concepts": desired, + "matched_desired_concepts": matched_desired, + "extra_positive_concepts": extra_positive, + "matched_negative_concepts": sorted(prompt_negative_hits), + "exclusive_negative_concepts": exclusive_negative, + "semantic_coverage": round(semantic_coverage, 3), + "support_score": round(support_score, 3), + "lexical_support": round(lexical, 3), + "negative_penalty": round(negative_penalty, 3), + "coverage_boost": round(coverage_boost, 3), + "concept_evidence": { + "positive": { + name: prompt_positive_hits[name]["matched_phrases"] + for name in sorted(prompt_positive_hits) + }, + "negative": { + name: prompt_negative_hits[name]["matched_phrases"] + for name in sorted(prompt_negative_hits) + }, + }, } + return score, score_detail def classify_bucket(bucket: str) -> bool: return bucket == "should_trigger" -def evaluate(description: str, cases: dict, threshold: float) -> dict: - desc_words = words(description) - positive_freq = token_frequencies(cases, ("should_trigger",)) - negative_freq = token_frequencies(cases, ("should_not_trigger", "near_neighbor")) - negative_patterns = compile_negative_patterns(cases) +def evaluate(description: str, cases: dict, threshold: float, config: dict) -> dict: results = {"should_trigger": [], "should_not_trigger": [], "near_neighbor": []} fp = 0 fn = 0 bucket_stats = {} + family_stats: dict[str, dict] = {} misfires = [] for bucket in ("should_trigger", "should_not_trigger", "near_neighbor"): expected = classify_bucket(bucket) + items = iter_case_items(cases, bucket) total = 0 passed_count = 0 - for prompt in cases.get(bucket, []): - score, score_detail = score_prompt_weighted(desc_words, prompt, positive_freq, negative_freq, negative_patterns) + for item in items: + prompt = item["text"] + family = item.get("family", "default") + score, score_detail = score_prompt_semantic(description, prompt, config) predicted = score >= threshold passed = predicted == expected total += 1 @@ -120,8 +180,10 @@ def evaluate(description: str, cases: dict, threshold: float) -> dict: fn += 1 if not passed and not expected: fp += 1 + record = { "prompt": prompt, + "family": family, "score": round(score, 3), "predicted_trigger": predicted, "expected_trigger": expected, @@ -131,34 +193,60 @@ def evaluate(description: str, cases: dict, threshold: float) -> dict: if 0.75 * threshold <= score <= 1.25 * threshold: record["boundary_case"] = True results[bucket].append(record) + + family_bucket = family_stats.setdefault( + family, + {"total": 0, "passed": 0, "false_positives": 0, "false_negatives": 0}, + ) + family_bucket["total"] += 1 + if passed: + family_bucket["passed"] += 1 + elif expected: + family_bucket["false_negatives"] += 1 + else: + family_bucket["false_positives"] += 1 + if not passed: misfires.append( { "bucket": bucket, + "family": family, "prompt": prompt, "score": round(score, 3), "reason": "false_negative" if expected else "false_positive", - "matched_negative_patterns": score_detail["matched_negative_patterns"], + "matched_desired_concepts": score_detail["matched_desired_concepts"], + "matched_negative_concepts": score_detail["matched_negative_concepts"], } ) + bucket_stats[bucket] = { "total": total, "passed": passed_count, "pass_rate": round(passed_count / total, 3) if total else None, } + for family, stats in family_stats.items(): + stats["pass_rate"] = round(stats["passed"] / stats["total"], 3) if stats["total"] else None + tp = sum(1 for item in results["should_trigger"] if item["predicted_trigger"]) precision = tp / (tp + fp) if (tp + fp) else None recall = tp / (tp + fn) if (tp + fn) else None return { "threshold": threshold, - "threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Final scores combine token overlap, positive-token bonuses, negative-token penalties, and explicit negative-pattern penalties. Scores near the threshold should be reviewed as boundary cases.", + "threshold_explanation": ( + "Prompts at or above the threshold are treated as trigger matches. " + "Scores are driven primarily by semantic intent coverage: packaging intent, " + "workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. " + "Explicit exclusions such as summary-only, translation-only, one-off, document-only, " + "or do-not-build directives apply direct penalties and can override otherwise similar wording." + ), "false_positives": fp, "false_negatives": fn, "precision": round(precision, 3) if precision is not None else None, "recall": round(recall, 3) if recall is not None else None, "bucket_stats": bucket_stats, + "family_stats": family_stats, "misfires": misfires, "results": results, } @@ -180,12 +268,13 @@ def compare_reports(baseline: dict, improved: dict) -> dict: def main() -> None: - parser = argparse.ArgumentParser(description="Heuristic trigger quality evaluator.") + parser = argparse.ArgumentParser(description="Semantic trigger quality evaluator.") parser.add_argument("--description", help="Description string to evaluate") parser.add_argument("--description-file", help="Read description text from file") parser.add_argument("--baseline-description", help="Baseline description string to compare against") parser.add_argument("--baseline-description-file", help="Read baseline description from file") - parser.add_argument("--cases", required=True, help="JSON file with should_trigger and should_not_trigger arrays") + parser.add_argument("--cases", required=True, help="JSON file with trigger cases") + parser.add_argument("--semantic-config", default=str(DEFAULT_CONFIG_PATH), help="Semantic config JSON") parser.add_argument("--threshold", type=float, default=None, help="Trigger threshold override") args = parser.parse_args() @@ -195,18 +284,19 @@ def main() -> None: if not description: raise SystemExit("Provide --description or --description-file") - cases = load_cases(Path(args.cases)) - threshold = args.threshold if args.threshold is not None else cases.get("recommended_threshold", 0.35) - report = evaluate(description, cases, threshold) + cases = load_json(Path(args.cases)) + config = load_semantic_config(Path(args.semantic_config)) + threshold = args.threshold if args.threshold is not None else cases.get("recommended_threshold", 0.48) + report = evaluate(description, cases, threshold, config) baseline = args.baseline_description if args.baseline_description_file: baseline = extract_description(Path(args.baseline_description_file).read_text(encoding="utf-8")) if baseline: - report["comparison"] = compare_reports(evaluate(baseline, cases, threshold), report) + report["comparison"] = compare_reports(evaluate(baseline, cases, threshold, config), report) print(json.dumps(report, ensure_ascii=False, indent=2)) - if report["false_positives"] > 2: + if report["false_positives"] > 0 or report["false_negatives"] > 0: raise SystemExit(2)