yaojingang
6 lines
6.4 KiB
JSON
6 lines
6.4 KiB
JSON
{"id":"skill-package-contract","prompt":"Turn this repeated workflow into a reusable team skill package.","baseline_output":"I can write a prompt for that workflow and include a short checklist.","with_skill_output":"Create a routeable SKILL.md, agents/interface.yaml, reports/skill-overview.html, and a focused eval plan. Keep the root entrypoint lean, move durable guidance into references/, scripts into scripts/, and evidence into reports/.","assertions":[{"id":"has-entrypoint","description":"Output names the routeable Skill entrypoint.","required":["SKILL.md"],"weight":1,"failure_type":"missing_entrypoint"},{"id":"has-interface","description":"Output includes neutral interface metadata.","required":["agents/interface.yaml"],"weight":1,"failure_type":"missing_interface"},{"id":"has-report","description":"Output points reviewer to generated overview report.","required":["reports/skill-overview.html"],"weight":1,"failure_type":"missing_review_artifact"},{"id":"has-resource-boundary","description":"Output preserves entrypoint/resource separation.","required":["references/","scripts/","reports/"],"weight":1,"failure_type":"resource_boundary"}],"human_review":{"expected_winner":"with_skill"}}
|
|
{"id":"output-eval-expectation","prompt":"Upgrade this production skill so we know whether the generated output is better than baseline.","baseline_output":"Add more examples and run the trigger eval again.","with_skill_output":"Add Output Eval Lab cases with baseline_output and with_skill_output, run assertion grading, report with-skill vs baseline pass-rate delta, and record failure taxonomy plus next fixes in reports/output_quality_scorecard.md.","assertions":[{"id":"has-baseline-comparison","description":"Output explicitly compares with-skill and baseline outputs.","required":["with-skill","baseline"],"weight":1,"failure_type":"missing_baseline_delta"},{"id":"has-assertion-grading","description":"Output includes objective assertion grading.","required":["assertion grading"],"weight":1,"failure_type":"missing_assertions"},{"id":"has-scorecard","description":"Output produces a scorecard report path.","required":["reports/output_quality_scorecard.md"],"weight":1,"failure_type":"missing_scorecard"},{"id":"has-failure-taxonomy","description":"Output records failure taxonomy.","required":["failure taxonomy"],"weight":1,"failure_type":"missing_failure_taxonomy"}],"human_review":{"expected_winner":"with_skill"}}
|
|
{"id":"ir-before-packaging","prompt":"Package this skill for OpenAI, Claude, Agent Skills, and generic targets.","baseline_output":"Export folders for each platform and adjust files as needed.","with_skill_output":"Export Skill IR first with job_to_be_done, trigger_surface, workflow, resources, eval_plan, risk, and governance. Then compile or package targets from the IR so OpenAI, Claude, Agent Skills, and generic packages preserve the same capability contract.","assertions":[{"id":"has-ir-first","description":"Output requires Skill IR before packaging.","required":["Skill IR first"],"weight":1,"failure_type":"missing_ir"},{"id":"has-core-ir-fields","description":"Output lists core IR contract fields.","required":["job_to_be_done","trigger_surface","eval_plan","governance"],"weight":1,"failure_type":"incomplete_ir_contract"},{"id":"has-targets","description":"Output names requested runtime targets.","required":["OpenAI","Claude","Agent Skills","generic"],"weight":1,"failure_type":"missing_targets"},{"id":"has-semantic-preservation","description":"Output says targets must preserve the capability contract.","required":["preserve the same capability contract"],"weight":1,"failure_type":"semantic_drift"}],"human_review":{"expected_winner":"with_skill"}}
|
|
{"id":"near-neighbor-boundary","prompt":"I only need a one-off summary of these notes, not a reusable process.","baseline_output":"Create a SKILL.md and a reusable workflow anyway so future notes can use it.","with_skill_output":"Do not create a skill for this one-off request. Treat it as a near-neighbor: answer the summary directly unless the user confirms repeated use, shared ownership, or a reusable output contract.","assertions":[{"id":"declines-skill","description":"Output refuses unnecessary skill creation for one-off work.","required":["Do not create a skill"],"weight":1,"failure_type":"over_trigger"},{"id":"names-near-neighbor","description":"Output labels the request as near-neighbor instead of owned work.","required":["near-neighbor"],"weight":1,"failure_type":"missing_boundary_language"},{"id":"requires-reuse-signal","description":"Output asks for repeat-use evidence before packaging.","required":["repeated use","reusable output contract"],"weight":1,"failure_type":"missing_reuse_gate"}],"human_review":{"expected_winner":"with_skill"},"metadata":{"case_type":"near_neighbor","tier":"production"}}
|
|
{"id":"file-backed-governed-package","prompt":"Turn the attached release brief source into a governed skill package.","input_files":["fixtures/release-brief-source.md"],"baseline_output":"Draft a release announcement with the changelog, support notes, and owner name.","with_skill_output":"Use the file-backed fixture as source evidence, then create a governed skill package with SKILL.md, agents/interface.yaml, owner, review cadence, input_files, output contract, rollback boundary, trust report, and reports/output_quality_scorecard.md. Mark missing launch metrics as missing evidence instead of inventing them.","assertions":[{"id":"uses-file-backed-evidence","description":"Output names file-backed source evidence.","required":["file-backed fixture","input_files"],"weight":1,"failure_type":"missing_file_fixture"},{"id":"has-governance","description":"Output preserves governed ownership metadata.","required":["owner","review cadence"],"weight":1,"failure_type":"missing_governance"},{"id":"has-output-contract","description":"Output preserves output and rollback boundaries.","required":["output contract","rollback boundary"],"weight":1,"failure_type":"missing_contract_boundary"},{"id":"has-trust-and-scorecard","description":"Output requires trust and output scorecard artifacts.","required":["trust report","reports/output_quality_scorecard.md"],"weight":1,"failure_type":"missing_release_evidence"},{"id":"does-not-invent-evidence","description":"Output forbids invented launch evidence.","required":["missing evidence"],"forbidden":["invented launch metrics"],"weight":1,"failure_type":"invented_evidence"}],"human_review":{"expected_winner":"with_skill"},"metadata":{"case_type":"boundary","tier":"governed"}}
|