feat: add semantic trigger eval and failure library
This commit is contained in:
yaojingang
@@ -47,7 +47,7 @@ make test
|
||||
## Results
|
||||
|
||||
- `make test` passes locally
|
||||
- trigger eval: `0` false positives, `0` false negatives on the current regression set
|
||||
- semantic trigger eval: `0` false positives, `0` false negatives on the current regression set
|
||||
- eval suite: train / dev / holdout all pass
|
||||
- packaging validation: `openai`, `claude`, and `generic` targets pass contract checks
|
||||
- packaging failure fixtures: invalid metadata, invalid YAML, and unsupported targets fail as expected
|
||||
@@ -110,7 +110,7 @@ Long-form material that should not bloat the main skill file. This includes desi
|
||||
|
||||
Utility scripts that make the meta-skill operational:
|
||||
|
||||
- `trigger_eval.py`: evaluates trigger descriptions with positive, negative, and near-neighbor prompts
|
||||
- `trigger_eval.py`: evaluates trigger descriptions with semantic intent concepts, explicit exclusions, and near-neighbor prompts
|
||||
- `run_eval_suite.py`: runs train/dev/holdout trigger suites and fails if aggregate regressions appear
|
||||
- `context_sizer.py`: estimates context weight and warns when the initial load gets too large
|
||||
- `cross_packager.py`: builds client-specific export artifacts with explicit platform contracts and validation
|
||||
@@ -122,7 +122,7 @@ Reusable trigger and packaging checks, including baseline and improved descripti
|
||||
|
||||
### `examples/`
|
||||
|
||||
Three end-to-end examples showing raw workflow input, design summary, and final generated skill shape.
|
||||
End-to-end examples showing raw workflow input, design summary, and final generated skill shape, including one thicker complex benchmark and one evolution chain.
|
||||
|
||||
### `.github/workflows/test.yml`
|
||||
|
||||
@@ -130,11 +130,12 @@ Continuous integration entrypoint that runs the full local regression suite on p
|
||||
|
||||
## Validation Notes
|
||||
|
||||
- Trigger evaluation is stronger than the original overlap-only version, but it is still heuristic.
|
||||
- Trigger evaluation now uses a local semantic-intent model with explicit positive concepts, exclusion concepts, and boundary-case reporting.
|
||||
- The sample trigger report now covers a larger positive, negative, and near-neighbor set rather than a tiny demo set.
|
||||
- Train/dev/holdout trigger suites now separate iterative tuning from final verification.
|
||||
- Packaging validation now uses explicit contracts and YAML parsing, but it is still a lightweight local validation layer rather than a full platform integration suite.
|
||||
- `evals/failure-cases.md` captures known weak spots that should remain part of regression checks.
|
||||
- `failures/` captures reusable anti-pattern writeups for routing, packaging, and authoring failures.
|
||||
- `tests/verify_packager_failures.py` checks that invalid metadata, invalid YAML, and unsupported targets fail clearly.
|
||||
|
||||
### `templates/`
|
||||
@@ -205,6 +206,7 @@ This project is best for:
|
||||
|
||||
- Examples: [examples/README.md](examples/README.md)
|
||||
- Evals: [evals/README.md](evals/README.md)
|
||||
- Failure library: [failures/README.md](failures/README.md)
|
||||
- Packaging contracts: [references/packaging-contracts.md](references/packaging-contracts.md)
|
||||
- Platform capability matrix: [references/platform-capability-matrix.md](references/platform-capability-matrix.md)
|
||||
- Failure fixtures: [tests/fixtures](tests/fixtures)
|
||||
|
||||
+5
-3
@@ -6,17 +6,18 @@ Contents:
|
||||
|
||||
- `trigger_cases.json`: positive, negative, and near-neighbor prompts
|
||||
- `train/`, `dev/`, `holdout/`: split trigger suites for iterative tuning and final verification
|
||||
- `semantic_config.json`: local semantic-intent concepts, exclusions, and weights
|
||||
- `baseline_description.txt`: intentionally weaker trigger description
|
||||
- `improved_description.txt`: current stronger trigger description
|
||||
- `sample_trigger_report.json`: example comparison output using threshold `0.35`
|
||||
- `sample_trigger_report.json`: example comparison output using the current recommended threshold
|
||||
- `failure-cases.md`: current weak spots and regression targets
|
||||
- `packaging_expectations.json`: required packaging behaviors for supported targets
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --threshold 0.35
|
||||
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt --threshold 0.35
|
||||
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json
|
||||
python3 scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
|
||||
python3 scripts/run_eval_suite.py
|
||||
python3 scripts/cross_packager.py . --platform openai --platform claude --expectations evals/packaging_expectations.json --zip
|
||||
python3 tests/verify_packager_failures.py
|
||||
@@ -30,4 +31,5 @@ Regression scope now includes:
|
||||
- long-context positives
|
||||
- mixed-intent negatives
|
||||
- explicit "do not build a skill" negatives
|
||||
- semantic exclusion cases such as one-off, document-only, and future-outline prompts
|
||||
- holdout verification
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"recommended_threshold": 0.37,
|
||||
"recommended_threshold": 0.33,
|
||||
"negative_patterns": [
|
||||
"one-off",
|
||||
"do not turn",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"recommended_threshold": 0.37,
|
||||
"recommended_threshold": 0.33,
|
||||
"negative_patterns": [
|
||||
"one-off",
|
||||
"do not turn",
|
||||
|
||||
+1502
-12
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,194 @@
|
||||
{
|
||||
"fallback_positive_concepts": [
|
||||
"build_skill",
|
||||
"transform_workflow",
|
||||
"reuse_package",
|
||||
"eval_optimize",
|
||||
"iterate_existing_skill"
|
||||
],
|
||||
"positive_concepts": {
|
||||
"build_skill": {
|
||||
"weight": 0.34,
|
||||
"phrases": [
|
||||
"create a skill",
|
||||
"build a skill",
|
||||
"make a skill",
|
||||
"generate a skill",
|
||||
"agent skill",
|
||||
"skill package",
|
||||
"meta skill",
|
||||
"meta-skill",
|
||||
"turn this into a skill",
|
||||
"turn all of that into one reusable skill package"
|
||||
]
|
||||
},
|
||||
"transform_workflow": {
|
||||
"weight": 0.2,
|
||||
"phrases": [
|
||||
"workflow",
|
||||
"runbook",
|
||||
"process",
|
||||
"checklist",
|
||||
"prompt history",
|
||||
"operations checklist",
|
||||
"internal process note",
|
||||
"existing skill draft",
|
||||
"rough notes"
|
||||
]
|
||||
},
|
||||
"reuse_package": {
|
||||
"weight": 0.2,
|
||||
"phrases": [
|
||||
"reusable",
|
||||
"reuse",
|
||||
"package",
|
||||
"packaging",
|
||||
"distribution",
|
||||
"share it with the team",
|
||||
"team reuse",
|
||||
"internal workflows",
|
||||
"production-ready"
|
||||
]
|
||||
},
|
||||
"eval_optimize": {
|
||||
"weight": 0.16,
|
||||
"phrases": [
|
||||
"evals",
|
||||
"trigger evals",
|
||||
"near-neighbor",
|
||||
"trigger boundary",
|
||||
"benchmark",
|
||||
"validation",
|
||||
"validate",
|
||||
"packaging checks"
|
||||
]
|
||||
},
|
||||
"iterate_existing_skill": {
|
||||
"weight": 0.16,
|
||||
"phrases": [
|
||||
"improve an existing skill",
|
||||
"existing skill",
|
||||
"this skill",
|
||||
"skill description",
|
||||
"optimize skill triggering",
|
||||
"package this skill",
|
||||
"sharing it with the team",
|
||||
"before sharing it with the team"
|
||||
]
|
||||
},
|
||||
"multi_asset": {
|
||||
"weight": 0.06,
|
||||
"phrases": [
|
||||
"references",
|
||||
"scripts",
|
||||
"manifest",
|
||||
"folder",
|
||||
"directory",
|
||||
"files"
|
||||
]
|
||||
},
|
||||
"team_operationalize": {
|
||||
"weight": 0.04,
|
||||
"phrases": [
|
||||
"team",
|
||||
"internal",
|
||||
"standardize",
|
||||
"shared",
|
||||
"distribution",
|
||||
"sharing it with the team",
|
||||
"team reuse"
|
||||
]
|
||||
}
|
||||
},
|
||||
"negative_concepts": {
|
||||
"explain_only": {
|
||||
"weight": 0.26,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"explain what",
|
||||
"just explain",
|
||||
"explain it",
|
||||
"tell me what it means",
|
||||
"plain english"
|
||||
]
|
||||
},
|
||||
"summarize_only": {
|
||||
"weight": 0.3,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"summary",
|
||||
"summarize",
|
||||
"only want a summary",
|
||||
"recap"
|
||||
]
|
||||
},
|
||||
"translate_only": {
|
||||
"weight": 0.35,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"translate",
|
||||
"into japanese",
|
||||
"into french",
|
||||
"into russian"
|
||||
]
|
||||
},
|
||||
"brainstorm_only": {
|
||||
"weight": 0.24,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"brainstorm",
|
||||
"ideas",
|
||||
"improve packaging",
|
||||
"shape an idea"
|
||||
]
|
||||
},
|
||||
"one_off_request": {
|
||||
"weight": 0.24,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"one-off",
|
||||
"custom answer",
|
||||
"for this request",
|
||||
"just this task"
|
||||
]
|
||||
},
|
||||
"no_build_directive": {
|
||||
"weight": 0.42,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"do not build",
|
||||
"don't build",
|
||||
"without building",
|
||||
"do not generate any skill files",
|
||||
"not a reusable skill",
|
||||
"not as an agent skill",
|
||||
"do not turn it into a skill",
|
||||
"stop short of making a full skill",
|
||||
"not a skill yet",
|
||||
"no packaging needed"
|
||||
]
|
||||
},
|
||||
"document_only": {
|
||||
"weight": 0.18,
|
||||
"exclusive": false,
|
||||
"phrases": [
|
||||
"readme",
|
||||
"document",
|
||||
"article",
|
||||
"blog title",
|
||||
"headline",
|
||||
"landing page"
|
||||
]
|
||||
},
|
||||
"future_outline": {
|
||||
"weight": 0.22,
|
||||
"exclusive": true,
|
||||
"phrases": [
|
||||
"future skill",
|
||||
"possible future skill",
|
||||
"before we decide whether to build",
|
||||
"outline for a possible future skill"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"recommended_threshold": 0.37,
|
||||
"recommended_threshold": 0.33,
|
||||
"negative_patterns": [
|
||||
"one-off",
|
||||
"do not turn",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"recommended_threshold": 0.37,
|
||||
"recommended_threshold": 0.33,
|
||||
"negative_patterns": [
|
||||
"one-off",
|
||||
"do not turn",
|
||||
|
||||
@@ -14,6 +14,13 @@ Each example contains:
|
||||
- `design-summary.md`: how the meta-skill scopes and structures the result
|
||||
- `generated-skill/`: the resulting skill package shape
|
||||
|
||||
The complex example is intentionally thicker than the others and includes:
|
||||
|
||||
- `manifest.json`
|
||||
- `input/` and `outputs/` sample artifacts
|
||||
- `evals/trigger_cases.json`
|
||||
- richer references and a deterministic script
|
||||
|
||||
Additional evolution example:
|
||||
|
||||
- `evolution-frontend-review/`: shows raw workflow, v0, v1, final, and eval delta
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
- Skill tier: complex
|
||||
- Trigger surface: release prep, release checklist, rollout coordination, rollback planning
|
||||
- Deterministic content: checklist stages and required artifacts
|
||||
- Flexible content: risk commentary and rollout recommendation
|
||||
- Package choice: `SKILL.md`, `agents/interface.yaml`, `references/`, `scripts/`, `evals/`
|
||||
- Deterministic content: checklist stages, migration prompts, packet sections, and stakeholder message scaffolds
|
||||
- Flexible content: risk commentary, go/no-go judgment, and escalation notes
|
||||
- Package choice: `SKILL.md`, `agents/interface.yaml`, `manifest.json`, `references/`, `scripts/`, `evals/`, `input/`, `outputs/`
|
||||
- Boundary rule: use for release packet creation and readiness review, not for general project planning or one-off announcements
|
||||
- Validation goal: the generated packet must contain release scope, migration notes, rollout plan, rollback triggers, stakeholder communication, and an explicit decision block
|
||||
|
||||
@@ -1,18 +1,68 @@
|
||||
---
|
||||
name: release-orchestrator
|
||||
description: Coordinate software release preparation, rollout readiness, migration notes, rollback planning, and stakeholder communication. Use when asked to prepare a release, build a rollout checklist, review release risk, or organize launch readiness.
|
||||
description: Coordinate software release preparation, rollout readiness, migration notes, rollback planning, stakeholder communication, and final go/no-go release packets. Use when asked to prepare a release packet, review release readiness, or turn scattered launch notes into one reusable release workflow. Do not use for generic project planning, one-off announcement drafting, or high-level release theory.
|
||||
---
|
||||
|
||||
# Release Orchestrator
|
||||
|
||||
## When To Use
|
||||
|
||||
- You need one reusable release packet from scattered launch inputs.
|
||||
- You need to verify migration notes, rollout steps, rollback triggers, and stakeholder messages before release approval.
|
||||
- You need a go/no-go summary with explicit blockers.
|
||||
|
||||
## Do Not Use
|
||||
|
||||
- The request is only to explain a release concept.
|
||||
- The request is only to write a single announcement.
|
||||
- The request is still brainstorming and not yet packaging a repeatable release workflow.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Gather release inputs: changes, risks, migrations, dependencies.
|
||||
2. Build artifacts for changelog, rollout, rollback, and announcements.
|
||||
3. Flag missing release-critical information before declaring readiness.
|
||||
4. Output a release package with clear go/no-go signals.
|
||||
1. Gather the minimum release inputs:
|
||||
- release scope
|
||||
- risky changes
|
||||
- migrations
|
||||
- rollout sequencing
|
||||
- rollback triggers
|
||||
- communication audiences
|
||||
2. Read `references/release-checklist.md` and `references/risk-matrix.md` to determine required sections and escalation thresholds.
|
||||
3. Read `references/migration-template.md` and `references/stakeholder-comms.md` when migrations or communications are present.
|
||||
4. Use `scripts/build_release_packet.py` with `input/release_input_example.json` as the shape reference when deterministic packet assembly is needed.
|
||||
5. Produce one release packet with:
|
||||
- release summary
|
||||
- dependency and migration section
|
||||
- rollout steps
|
||||
- rollback triggers
|
||||
- stakeholder communication plan
|
||||
- explicit go/no-go decision
|
||||
6. If release-critical data is missing, stop and report blockers instead of pretending readiness.
|
||||
|
||||
## Output Contract
|
||||
|
||||
The final answer must include:
|
||||
|
||||
- a concise scope summary
|
||||
- a risk block with severity
|
||||
- migration notes or a clear "none required"
|
||||
- rollout plan
|
||||
- rollback plan
|
||||
- stakeholder communication block
|
||||
- a final decision: `GO`, `GO WITH CONDITIONS`, or `NO-GO`
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
- Release-critical inputs are either present or explicitly flagged as missing.
|
||||
- Rollout and rollback are both covered.
|
||||
- Migration notes are not omitted when schema or data changes exist.
|
||||
- Stakeholder communication is tailored by audience and timing.
|
||||
- The final decision is explicit and justified.
|
||||
|
||||
## Reference Map
|
||||
|
||||
- Read `references/release-checklist.md` for the stage checklist.
|
||||
- Use `scripts/build_release_packet.py` for deterministic artifact assembly.
|
||||
- Read `references/release-checklist.md` for required packet sections.
|
||||
- Read `references/risk-matrix.md` for decision thresholds.
|
||||
- Read `references/migration-template.md` for migration note structure.
|
||||
- Read `references/stakeholder-comms.md` for announcement scaffolds.
|
||||
- Inspect `evals/trigger_cases.json` for trigger boundaries.
|
||||
- Inspect `outputs/release_packet_example.md` for an example final artifact.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
interface:
|
||||
display_name: "Release Orchestrator"
|
||||
short_description: "Organize release readiness, rollout, and rollback artifacts"
|
||||
default_prompt: "Use $release-orchestrator to prepare this release package and rollout checklist."
|
||||
short_description: "Build reusable release packets with rollout, rollback, migration, and communication coverage"
|
||||
default_prompt: "Use $release-orchestrator to turn these release notes into a full release packet with go/no-go guidance."
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"recommended_threshold": 0.33,
|
||||
"should_trigger": [
|
||||
"Prepare a release packet from these migration notes, rollback conditions, and rollout steps.",
|
||||
"Review release readiness and produce a go/no-go packet for this launch.",
|
||||
"Turn these scattered release notes into a reusable release workflow."
|
||||
],
|
||||
"should_not_trigger": [
|
||||
"Explain what a rollback plan is.",
|
||||
"Draft one short launch announcement for customers."
|
||||
],
|
||||
"near_neighbor": [
|
||||
"Brainstorm some release ideas before we decide whether to package the workflow.",
|
||||
"Summarize these release notes without creating a reusable workflow."
|
||||
]
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"release_name": "2026.03.31-search-hardening",
|
||||
"scope_summary": "Adds query guardrails, dependency upgrades, and an index migration for the search service.",
|
||||
"risks": [
|
||||
{
|
||||
"severity": "high",
|
||||
"summary": "Index migration must complete before traffic shift."
|
||||
},
|
||||
{
|
||||
"severity": "medium",
|
||||
"summary": "Cache warm-up may temporarily increase search latency."
|
||||
}
|
||||
],
|
||||
"migrations": "Run search_index_v7 migration before rollout. Rollback is safe only before background reindex completion.",
|
||||
"rollout_steps": [
|
||||
"Run migration in primary region.",
|
||||
"Verify index health and shard count.",
|
||||
"Shift 10 percent traffic and monitor search latency.",
|
||||
"Shift remaining traffic after 15 minutes of stable metrics."
|
||||
],
|
||||
"rollback_triggers": [
|
||||
"Search error rate above 2 percent for 5 minutes.",
|
||||
"Median latency increases above 400ms after warm-up window."
|
||||
],
|
||||
"stakeholder_messages": {
|
||||
"engineering-oncall": "Monitor search dashboards during the first 15 minutes after traffic shift.",
|
||||
"support": "Watch for search degradation tickets during the release window.",
|
||||
"leadership": "Release is go-with-conditions because migration timing is critical."
|
||||
},
|
||||
"decision": "GO WITH CONDITIONS"
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "release-orchestrator",
|
||||
"version": "1.0.0",
|
||||
"owner": "Yao Team",
|
||||
"updated_at": "2026-03-31",
|
||||
"category": "operations",
|
||||
"complexity_tier": "complex",
|
||||
"factory_components": [
|
||||
"references",
|
||||
"scripts",
|
||||
"evals",
|
||||
"input",
|
||||
"outputs"
|
||||
]
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
# Release Packet: 2026.03.31-search-hardening
|
||||
|
||||
## Scope
|
||||
Adds query guardrails, dependency upgrades, and an index migration for the search service.
|
||||
|
||||
## Risks
|
||||
- [HIGH] Index migration must complete before traffic shift.
|
||||
- [MEDIUM] Cache warm-up may temporarily increase search latency.
|
||||
|
||||
## Migrations
|
||||
Run search_index_v7 migration before rollout. Rollback is safe only before background reindex completion.
|
||||
|
||||
## Rollout
|
||||
- Run migration in primary region.
|
||||
- Verify index health and shard count.
|
||||
- Shift 10 percent traffic and monitor search latency.
|
||||
- Shift remaining traffic after 15 minutes of stable metrics.
|
||||
|
||||
## Rollback
|
||||
- Search error rate above 2 percent for 5 minutes.
|
||||
- Median latency increases above 400ms after warm-up window.
|
||||
|
||||
## Stakeholder Communication
|
||||
- engineering-oncall: Monitor search dashboards during the first 15 minutes after traffic shift.
|
||||
- support: Watch for search degradation tickets during the release window.
|
||||
- leadership: Release is go-with-conditions because migration timing is critical.
|
||||
|
||||
## Final Decision
|
||||
GO WITH CONDITIONS
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
# Migration Template
|
||||
|
||||
Use this shape whenever the release changes schema, data contracts, or runtime configuration.
|
||||
|
||||
## Required Fields
|
||||
|
||||
- migration type
|
||||
- affected service or datastore
|
||||
- execution timing
|
||||
- rollback expectation
|
||||
- operator warning
|
||||
|
||||
## Template
|
||||
|
||||
```md
|
||||
### Migration
|
||||
- Type:
|
||||
- Affected Surface:
|
||||
- Run Before Rollout:
|
||||
- Rollback Safe:
|
||||
- Operator Warning:
|
||||
```
|
||||
+24
-1
@@ -1,8 +1,31 @@
|
||||
# Release Checklist
|
||||
|
||||
- change summary
|
||||
## Required Sections
|
||||
|
||||
- release summary
|
||||
- dependency review
|
||||
- migration notes
|
||||
- rollout checklist
|
||||
- rollback checklist
|
||||
- stakeholder communication
|
||||
- final go/no-go decision
|
||||
|
||||
## Release Summary Questions
|
||||
|
||||
- What changed in this release?
|
||||
- Which services, schemas, or integrations are affected?
|
||||
- What is the blast radius if rollout fails?
|
||||
|
||||
## Rollout Gate
|
||||
|
||||
- Sequencing is clear.
|
||||
- Ownership is explicit.
|
||||
- Monitoring signal is defined.
|
||||
- Rollback trigger is concrete, not vague.
|
||||
|
||||
## Packet Failure Modes
|
||||
|
||||
- Missing migration notes when data changes exist
|
||||
- Missing rollback trigger
|
||||
- "Go" recommendation without acknowledged risk
|
||||
- Stakeholder message missing audience or send timing
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Risk Matrix
|
||||
|
||||
## Severity Guide
|
||||
|
||||
- `low`: internal-only changes, no migrations, rollback is trivial
|
||||
- `medium`: user-facing changes with limited blast radius or known manual recovery
|
||||
- `high`: schema changes, cross-service dependencies, or rollback with coordination cost
|
||||
- `critical`: irreversible data impact, unclear rollback, or no monitoring confidence
|
||||
|
||||
## Decision Rules
|
||||
|
||||
- `GO`: no unresolved high/critical risks and rollback path is clear
|
||||
- `GO WITH CONDITIONS`: risks remain but owners, mitigations, and gates are explicit
|
||||
- `NO-GO`: missing migration details, missing rollback criteria, or critical unknowns
|
||||
@@ -0,0 +1,21 @@
|
||||
# Stakeholder Communication
|
||||
|
||||
## Audiences
|
||||
|
||||
- engineering on-call
|
||||
- support
|
||||
- customer-facing operations
|
||||
- internal leadership
|
||||
|
||||
## Message Rules
|
||||
|
||||
- state release window
|
||||
- state impact surface
|
||||
- state expected operator action or "no action required"
|
||||
- state rollback or escalation signal when risk is non-trivial
|
||||
|
||||
## Example Format
|
||||
|
||||
- Engineering on-call: release window, dashboards to watch, rollback trigger
|
||||
- Support: user-visible changes, expected incidents, escalation owner
|
||||
- Leadership: scope, major risk, final decision
|
||||
+62
-1
@@ -1,2 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
print("Build release packet from gathered inputs.")
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def render_packet(payload: dict) -> str:
|
||||
lines = [
|
||||
f"# Release Packet: {payload['release_name']}",
|
||||
"",
|
||||
"## Scope",
|
||||
payload["scope_summary"],
|
||||
"",
|
||||
"## Risks",
|
||||
]
|
||||
for risk in payload["risks"]:
|
||||
lines.append(f"- [{risk['severity'].upper()}] {risk['summary']}")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Migrations",
|
||||
payload["migrations"],
|
||||
"",
|
||||
"## Rollout",
|
||||
]
|
||||
)
|
||||
for step in payload["rollout_steps"]:
|
||||
lines.append(f"- {step}")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Rollback",
|
||||
]
|
||||
)
|
||||
for trigger in payload["rollback_triggers"]:
|
||||
lines.append(f"- {trigger}")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Stakeholder Communication",
|
||||
]
|
||||
)
|
||||
for audience, message in payload["stakeholder_messages"].items():
|
||||
lines.append(f"- {audience}: {message}")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Final Decision",
|
||||
payload["decision"],
|
||||
]
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if len(sys.argv) != 2:
|
||||
raise SystemExit("Usage: build_release_packet.py <input.json>")
|
||||
payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(render_packet(payload), end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -1,12 +1,27 @@
|
||||
# Raw Workflow
|
||||
|
||||
Every release, we coordinate:
|
||||
Every release we have to pull information from too many places and rebuild the same release packet by hand.
|
||||
|
||||
- changelog preparation
|
||||
- dependency diff review
|
||||
- migration notes
|
||||
- rollout checklist
|
||||
- rollback checklist
|
||||
- stakeholder announcement
|
||||
Current inputs are scattered across:
|
||||
|
||||
This is currently spread across chat messages, wiki pages, and ad hoc reminders.
|
||||
- merged PR summaries from the current milestone
|
||||
- dependency update notes
|
||||
- migration docs from backend and data teams
|
||||
- rollout sequencing notes from on-call
|
||||
- rollback conditions kept in an old runbook
|
||||
- stakeholder announcements drafted in chat
|
||||
|
||||
What usually goes wrong:
|
||||
|
||||
- a migration note is missing when we think the release is ready
|
||||
- rollback criteria are vague
|
||||
- stakeholder communication is written too late
|
||||
- someone says "ship it" before the packet has a clear go/no-go decision
|
||||
|
||||
What we want from the skill:
|
||||
|
||||
- ask for the minimum required release inputs
|
||||
- produce one consistent release packet
|
||||
- force a go/no-go summary
|
||||
- call out missing release-critical details before approval
|
||||
- keep deterministic artifacts separate from judgment calls
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
# Failure And Anti-Pattern Library
|
||||
|
||||
This directory captures recurring ways a skill can fail before, during, or after packaging.
|
||||
|
||||
Use it for three things:
|
||||
|
||||
- regression design
|
||||
- author training
|
||||
- description and boundary tuning
|
||||
|
||||
Current case library:
|
||||
|
||||
- `one-off-vs-reusable`
|
||||
- `explain-not-package`
|
||||
- `document-export-vs-agent-skill`
|
||||
- `future-outline-vs-build`
|
||||
@@ -0,0 +1,28 @@
|
||||
# Document Export Vs Agent Skill
|
||||
|
||||
## Symptom
|
||||
|
||||
The request asks to package content as a document or improve a document, but the system mistakes it for agent-skill creation.
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
Reading "package" as "skill package" without checking the target artifact.
|
||||
|
||||
## Example Prompt
|
||||
|
||||
`Package this explanation as a document, not as an agent skill.`
|
||||
|
||||
## Why It Fails
|
||||
|
||||
- "package" is overloaded
|
||||
- release notes, runbooks, and docs often look structurally similar to skills
|
||||
|
||||
## Fix
|
||||
|
||||
- model target artifact explicitly
|
||||
- treat document-only exports as negative route evidence
|
||||
|
||||
## Regression Prompts
|
||||
|
||||
- `Improve this README but do not turn it into a skill.`
|
||||
- `Package this explanation as a document, not as an agent skill.`
|
||||
@@ -0,0 +1,28 @@
|
||||
# Explain Not Package
|
||||
|
||||
## Symptom
|
||||
|
||||
The request is asking for explanation or interpretation, but the skill creator tries to package it as a reusable workflow.
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
Confusing topic adjacency with packaging intent.
|
||||
|
||||
## Example Prompt
|
||||
|
||||
`Review this note and tell me what it means in plain English.`
|
||||
|
||||
## Why It Fails
|
||||
|
||||
- the source material may look like a workflow
|
||||
- the user still wants explanation, not systematization
|
||||
|
||||
## Fix
|
||||
|
||||
- treat explanation, summary-only, and translation-only modes as direct exclusions
|
||||
- require positive evidence of reusable packaging intent before triggering
|
||||
|
||||
## Regression Prompts
|
||||
|
||||
- `Explain what a workflow is.`
|
||||
- `Review this process note and explain it, no packaging needed.`
|
||||
@@ -0,0 +1,28 @@
|
||||
# Future Outline Vs Build
|
||||
|
||||
## Symptom
|
||||
|
||||
The user is still framing or exploring a future skill, but the router prematurely triggers full skill generation.
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
Treating early planning language as execution-ready packaging intent.
|
||||
|
||||
## Example Prompt
|
||||
|
||||
`Create an outline for a possible future skill, but do not build the skill yet.`
|
||||
|
||||
## Why It Fails
|
||||
|
||||
- the prompt mentions "skill"
|
||||
- the actual task is pre-build design, not packaging
|
||||
|
||||
## Fix
|
||||
|
||||
- detect "future skill", "outline", "before we decide", and similar wording as planning-only exclusions
|
||||
- require either present-tense packaging intent or explicit build intent
|
||||
|
||||
## Regression Prompts
|
||||
|
||||
- `Help me shape an idea before we decide whether to build a skill.`
|
||||
- `Create an outline for a possible future skill, but do not build the skill yet.`
|
||||
@@ -0,0 +1,28 @@
|
||||
# One-Off Vs Reusable
|
||||
|
||||
## Symptom
|
||||
|
||||
The router sees strong packaging language and triggers even though the user only wants a one-off answer or prompt.
|
||||
|
||||
## Anti-Pattern
|
||||
|
||||
Treating any "create" or "package" wording as reusable-skill intent.
|
||||
|
||||
## Example Prompt
|
||||
|
||||
`Write a custom answer for this request without creating a skill package.`
|
||||
|
||||
## Why It Fails
|
||||
|
||||
- lexical overlap is high
|
||||
- true user intent is single-use output, not reusable infrastructure
|
||||
|
||||
## Fix
|
||||
|
||||
- detect one-off language explicitly
|
||||
- treat "custom answer", "one-off", and similar phrases as exclusion concepts
|
||||
|
||||
## Regression Prompts
|
||||
|
||||
- `Create a one-off prompt for this task.`
|
||||
- `Write a custom answer for this request without creating a skill package.`
|
||||
@@ -17,6 +17,8 @@ def run_case(script: Path, description: Path, baseline: Path, cases: Path) -> di
|
||||
str(baseline),
|
||||
"--cases",
|
||||
str(cases),
|
||||
"--semantic-config",
|
||||
str(Path("evals/semantic_config.json").resolve()),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
@@ -27,7 +29,7 @@ def run_case(script: Path, description: Path, baseline: Path, cases: Path) -> di
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Run trigger evaluation across train/dev/holdout suites.")
|
||||
parser = argparse.ArgumentParser(description="Run semantic trigger evaluation across train/dev/holdout suites.")
|
||||
parser.add_argument("--eval-dir", default="evals", help="Root eval directory")
|
||||
parser.add_argument("--description-file", default="evals/improved_description.txt")
|
||||
parser.add_argument("--baseline-description-file", default="evals/baseline_description.txt")
|
||||
|
||||
+153
-63
@@ -3,17 +3,23 @@ import argparse
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from collections import Counter
|
||||
|
||||
|
||||
WORD_RE = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9_-]*")
|
||||
WORD_RE = re.compile(r"[a-z0-9][a-z0-9_-]*")
|
||||
DEFAULT_CONFIG_PATH = Path("evals/semantic_config.json")
|
||||
|
||||
|
||||
def normalize(text: str) -> str:
|
||||
text = text.lower()
|
||||
text = re.sub(r"[^a-z0-9]+", " ", text)
|
||||
return re.sub(r"\s+", " ", text).strip()
|
||||
|
||||
|
||||
def words(text: str) -> set[str]:
|
||||
return {w.lower() for w in WORD_RE.findall(text)}
|
||||
|
||||
|
||||
def load_cases(path: Path) -> dict:
|
||||
def load_json(path: Path) -> dict:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
@@ -30,87 +36,141 @@ def extract_description(text: str) -> str:
|
||||
return text
|
||||
|
||||
|
||||
def score_prompt(description_words: set[str], prompt: str) -> float:
|
||||
def iter_case_items(cases: dict, bucket: str) -> list[dict]:
|
||||
items = []
|
||||
for raw in cases.get(bucket, []):
|
||||
if isinstance(raw, str):
|
||||
items.append({"text": raw, "family": "default"})
|
||||
else:
|
||||
item = dict(raw)
|
||||
item.setdefault("family", "default")
|
||||
items.append(item)
|
||||
return items
|
||||
|
||||
|
||||
def phrase_present(text: str, phrase: str) -> bool:
|
||||
phrase = normalize(phrase)
|
||||
if not phrase:
|
||||
return False
|
||||
return f" {phrase} " in f" {text} "
|
||||
|
||||
|
||||
def load_semantic_config(path: Path | None) -> dict:
|
||||
config_path = path or DEFAULT_CONFIG_PATH
|
||||
if not config_path.exists():
|
||||
raise SystemExit(f"Semantic config not found: {config_path}")
|
||||
return load_json(config_path)
|
||||
|
||||
|
||||
def collect_concept_hits(text: str, concepts: dict[str, dict]) -> dict[str, dict]:
|
||||
normalized = normalize(text)
|
||||
hits: dict[str, dict] = {}
|
||||
for name, spec in concepts.items():
|
||||
matched = []
|
||||
for phrase in spec.get("phrases", []):
|
||||
if phrase_present(normalized, phrase):
|
||||
matched.append(phrase)
|
||||
if matched:
|
||||
hits[name] = {
|
||||
"weight": spec["weight"],
|
||||
"matched_phrases": matched,
|
||||
"exclusive": bool(spec.get("exclusive")),
|
||||
}
|
||||
return hits
|
||||
|
||||
|
||||
def lexical_support(description_words: set[str], prompt: str) -> float:
|
||||
prompt_words = words(prompt)
|
||||
if not prompt_words:
|
||||
return 0.0
|
||||
overlap = description_words & prompt_words
|
||||
return len(overlap) / len(prompt_words)
|
||||
return len(description_words & prompt_words) / len(prompt_words)
|
||||
|
||||
|
||||
def token_frequencies(cases: dict, buckets: tuple[str, ...]) -> Counter:
|
||||
freq: Counter = Counter()
|
||||
for bucket in buckets:
|
||||
for prompt in cases.get(bucket, []):
|
||||
freq.update(words(prompt))
|
||||
return freq
|
||||
def desired_positive_concepts(description: str, config: dict) -> list[str]:
|
||||
description_hits = collect_concept_hits(description, config["positive_concepts"])
|
||||
names = list(description_hits)
|
||||
if names:
|
||||
return names
|
||||
return config.get("fallback_positive_concepts", [])
|
||||
|
||||
|
||||
def compile_negative_patterns(cases: dict) -> list[re.Pattern[str]]:
|
||||
return [re.compile(pattern, re.IGNORECASE) for pattern in cases.get("negative_patterns", [])]
|
||||
def score_prompt_semantic(description: str, prompt: str, config: dict) -> tuple[float, dict]:
|
||||
positive_concepts = config["positive_concepts"]
|
||||
negative_concepts = config["negative_concepts"]
|
||||
desired = desired_positive_concepts(description, config)
|
||||
desired_weight_total = sum(positive_concepts[name]["weight"] for name in desired) or 1.0
|
||||
|
||||
prompt_positive_hits = collect_concept_hits(prompt, positive_concepts)
|
||||
prompt_negative_hits = collect_concept_hits(prompt, negative_concepts)
|
||||
|
||||
def score_prompt_weighted(description_words: set[str], prompt: str, positive_freq: Counter, negative_freq: Counter, negative_patterns: list[re.Pattern[str]]) -> tuple[float, dict]:
|
||||
prompt_words = words(prompt)
|
||||
if not prompt_words:
|
||||
return 0.0, {"matched_positive_tokens": [], "matched_negative_tokens": [], "matched_negative_patterns": []}
|
||||
matched_desired = sorted([name for name in desired if name in prompt_positive_hits])
|
||||
extra_positive = sorted([name for name in prompt_positive_hits if name not in matched_desired])
|
||||
semantic_coverage = sum(positive_concepts[name]["weight"] for name in matched_desired) / desired_weight_total
|
||||
support_score = sum(positive_concepts[name]["weight"] for name in extra_positive)
|
||||
|
||||
overlap = description_words & prompt_words
|
||||
base_score = len(overlap) / len(prompt_words)
|
||||
exclusive_negative = sorted([name for name, hit in prompt_negative_hits.items() if hit["exclusive"]])
|
||||
negative_penalty = sum(hit["weight"] for hit in prompt_negative_hits.values())
|
||||
lexical = lexical_support(words(description), prompt)
|
||||
|
||||
weighted_bonus = 0.0
|
||||
matched_positive_tokens = []
|
||||
matched_negative_tokens = []
|
||||
for token in overlap:
|
||||
pos = positive_freq.get(token, 0)
|
||||
neg = negative_freq.get(token, 0)
|
||||
if pos > neg:
|
||||
weighted_bonus += 0.06
|
||||
matched_positive_tokens.append(token)
|
||||
coverage_boost = 0.0
|
||||
if len(matched_desired) >= 2:
|
||||
coverage_boost += 0.04
|
||||
if len(matched_desired) >= 3:
|
||||
coverage_boost += 0.02
|
||||
|
||||
weighted_penalty = 0.0
|
||||
for token in prompt_words:
|
||||
neg = negative_freq.get(token, 0)
|
||||
pos = positive_freq.get(token, 0)
|
||||
if neg > pos and token not in overlap:
|
||||
weighted_penalty += 0.04
|
||||
matched_negative_tokens.append(token)
|
||||
score = (semantic_coverage * 0.92) + min(0.12, support_score * 0.25) + min(0.06, lexical * 0.08) + coverage_boost
|
||||
score -= negative_penalty
|
||||
if exclusive_negative and semantic_coverage < 0.9:
|
||||
score -= 0.15
|
||||
score = max(0.0, min(1.0, score))
|
||||
|
||||
matched_negative_patterns = [pattern.pattern for pattern in negative_patterns if pattern.search(prompt)]
|
||||
pattern_penalty = 0.18 * len(matched_negative_patterns)
|
||||
|
||||
score = max(0.0, min(1.0, base_score + weighted_bonus - weighted_penalty - pattern_penalty))
|
||||
return score, {
|
||||
"matched_positive_tokens": sorted(set(matched_positive_tokens)),
|
||||
"matched_negative_tokens": sorted(set(matched_negative_tokens)),
|
||||
"matched_negative_patterns": matched_negative_patterns,
|
||||
"base_score": round(base_score, 3),
|
||||
"weighted_bonus": round(weighted_bonus, 3),
|
||||
"weighted_penalty": round(weighted_penalty + pattern_penalty, 3),
|
||||
score_detail = {
|
||||
"mode": "semantic-intent",
|
||||
"desired_positive_concepts": desired,
|
||||
"matched_desired_concepts": matched_desired,
|
||||
"extra_positive_concepts": extra_positive,
|
||||
"matched_negative_concepts": sorted(prompt_negative_hits),
|
||||
"exclusive_negative_concepts": exclusive_negative,
|
||||
"semantic_coverage": round(semantic_coverage, 3),
|
||||
"support_score": round(support_score, 3),
|
||||
"lexical_support": round(lexical, 3),
|
||||
"negative_penalty": round(negative_penalty, 3),
|
||||
"coverage_boost": round(coverage_boost, 3),
|
||||
"concept_evidence": {
|
||||
"positive": {
|
||||
name: prompt_positive_hits[name]["matched_phrases"]
|
||||
for name in sorted(prompt_positive_hits)
|
||||
},
|
||||
"negative": {
|
||||
name: prompt_negative_hits[name]["matched_phrases"]
|
||||
for name in sorted(prompt_negative_hits)
|
||||
},
|
||||
},
|
||||
}
|
||||
return score, score_detail
|
||||
|
||||
|
||||
def classify_bucket(bucket: str) -> bool:
|
||||
return bucket == "should_trigger"
|
||||
|
||||
|
||||
def evaluate(description: str, cases: dict, threshold: float) -> dict:
|
||||
desc_words = words(description)
|
||||
positive_freq = token_frequencies(cases, ("should_trigger",))
|
||||
negative_freq = token_frequencies(cases, ("should_not_trigger", "near_neighbor"))
|
||||
negative_patterns = compile_negative_patterns(cases)
|
||||
def evaluate(description: str, cases: dict, threshold: float, config: dict) -> dict:
|
||||
results = {"should_trigger": [], "should_not_trigger": [], "near_neighbor": []}
|
||||
fp = 0
|
||||
fn = 0
|
||||
bucket_stats = {}
|
||||
family_stats: dict[str, dict] = {}
|
||||
misfires = []
|
||||
|
||||
for bucket in ("should_trigger", "should_not_trigger", "near_neighbor"):
|
||||
expected = classify_bucket(bucket)
|
||||
items = iter_case_items(cases, bucket)
|
||||
total = 0
|
||||
passed_count = 0
|
||||
for prompt in cases.get(bucket, []):
|
||||
score, score_detail = score_prompt_weighted(desc_words, prompt, positive_freq, negative_freq, negative_patterns)
|
||||
for item in items:
|
||||
prompt = item["text"]
|
||||
family = item.get("family", "default")
|
||||
score, score_detail = score_prompt_semantic(description, prompt, config)
|
||||
predicted = score >= threshold
|
||||
passed = predicted == expected
|
||||
total += 1
|
||||
@@ -120,8 +180,10 @@ def evaluate(description: str, cases: dict, threshold: float) -> dict:
|
||||
fn += 1
|
||||
if not passed and not expected:
|
||||
fp += 1
|
||||
|
||||
record = {
|
||||
"prompt": prompt,
|
||||
"family": family,
|
||||
"score": round(score, 3),
|
||||
"predicted_trigger": predicted,
|
||||
"expected_trigger": expected,
|
||||
@@ -131,34 +193,60 @@ def evaluate(description: str, cases: dict, threshold: float) -> dict:
|
||||
if 0.75 * threshold <= score <= 1.25 * threshold:
|
||||
record["boundary_case"] = True
|
||||
results[bucket].append(record)
|
||||
|
||||
family_bucket = family_stats.setdefault(
|
||||
family,
|
||||
{"total": 0, "passed": 0, "false_positives": 0, "false_negatives": 0},
|
||||
)
|
||||
family_bucket["total"] += 1
|
||||
if passed:
|
||||
family_bucket["passed"] += 1
|
||||
elif expected:
|
||||
family_bucket["false_negatives"] += 1
|
||||
else:
|
||||
family_bucket["false_positives"] += 1
|
||||
|
||||
if not passed:
|
||||
misfires.append(
|
||||
{
|
||||
"bucket": bucket,
|
||||
"family": family,
|
||||
"prompt": prompt,
|
||||
"score": round(score, 3),
|
||||
"reason": "false_negative" if expected else "false_positive",
|
||||
"matched_negative_patterns": score_detail["matched_negative_patterns"],
|
||||
"matched_desired_concepts": score_detail["matched_desired_concepts"],
|
||||
"matched_negative_concepts": score_detail["matched_negative_concepts"],
|
||||
}
|
||||
)
|
||||
|
||||
bucket_stats[bucket] = {
|
||||
"total": total,
|
||||
"passed": passed_count,
|
||||
"pass_rate": round(passed_count / total, 3) if total else None,
|
||||
}
|
||||
|
||||
for family, stats in family_stats.items():
|
||||
stats["pass_rate"] = round(stats["passed"] / stats["total"], 3) if stats["total"] else None
|
||||
|
||||
tp = sum(1 for item in results["should_trigger"] if item["predicted_trigger"])
|
||||
precision = tp / (tp + fp) if (tp + fp) else None
|
||||
recall = tp / (tp + fn) if (tp + fn) else None
|
||||
|
||||
return {
|
||||
"threshold": threshold,
|
||||
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Final scores combine token overlap, positive-token bonuses, negative-token penalties, and explicit negative-pattern penalties. Scores near the threshold should be reviewed as boundary cases.",
|
||||
"threshold_explanation": (
|
||||
"Prompts at or above the threshold are treated as trigger matches. "
|
||||
"Scores are driven primarily by semantic intent coverage: packaging intent, "
|
||||
"workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. "
|
||||
"Explicit exclusions such as summary-only, translation-only, one-off, document-only, "
|
||||
"or do-not-build directives apply direct penalties and can override otherwise similar wording."
|
||||
),
|
||||
"false_positives": fp,
|
||||
"false_negatives": fn,
|
||||
"precision": round(precision, 3) if precision is not None else None,
|
||||
"recall": round(recall, 3) if recall is not None else None,
|
||||
"bucket_stats": bucket_stats,
|
||||
"family_stats": family_stats,
|
||||
"misfires": misfires,
|
||||
"results": results,
|
||||
}
|
||||
@@ -180,12 +268,13 @@ def compare_reports(baseline: dict, improved: dict) -> dict:
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Heuristic trigger quality evaluator.")
|
||||
parser = argparse.ArgumentParser(description="Semantic trigger quality evaluator.")
|
||||
parser.add_argument("--description", help="Description string to evaluate")
|
||||
parser.add_argument("--description-file", help="Read description text from file")
|
||||
parser.add_argument("--baseline-description", help="Baseline description string to compare against")
|
||||
parser.add_argument("--baseline-description-file", help="Read baseline description from file")
|
||||
parser.add_argument("--cases", required=True, help="JSON file with should_trigger and should_not_trigger arrays")
|
||||
parser.add_argument("--cases", required=True, help="JSON file with trigger cases")
|
||||
parser.add_argument("--semantic-config", default=str(DEFAULT_CONFIG_PATH), help="Semantic config JSON")
|
||||
parser.add_argument("--threshold", type=float, default=None, help="Trigger threshold override")
|
||||
args = parser.parse_args()
|
||||
|
||||
@@ -195,18 +284,19 @@ def main() -> None:
|
||||
if not description:
|
||||
raise SystemExit("Provide --description or --description-file")
|
||||
|
||||
cases = load_cases(Path(args.cases))
|
||||
threshold = args.threshold if args.threshold is not None else cases.get("recommended_threshold", 0.35)
|
||||
report = evaluate(description, cases, threshold)
|
||||
cases = load_json(Path(args.cases))
|
||||
config = load_semantic_config(Path(args.semantic_config))
|
||||
threshold = args.threshold if args.threshold is not None else cases.get("recommended_threshold", 0.48)
|
||||
report = evaluate(description, cases, threshold, config)
|
||||
|
||||
baseline = args.baseline_description
|
||||
if args.baseline_description_file:
|
||||
baseline = extract_description(Path(args.baseline_description_file).read_text(encoding="utf-8"))
|
||||
if baseline:
|
||||
report["comparison"] = compare_reports(evaluate(baseline, cases, threshold), report)
|
||||
report["comparison"] = compare_reports(evaluate(baseline, cases, threshold, config), report)
|
||||
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
if report["false_positives"] > 2:
|
||||
if report["false_positives"] > 0 or report["false_negatives"] > 0:
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user