当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。; Use this skill when the request matches: 将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。
Next
Review the top iteration directions before growing the package.
Use this skill when the request matches: 将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。
+
+
+
+
+
+
Intent questions
+
If this skill worked beautifully, what recurring job would it reliably handle for the user every time? This reveals the real job-to-be-done and gives the package a humane center instead of a guessed prompt shape.
When someone reaches for this skill in the real world, what materials will they actually hand to it? Input shape decides whether references, scripts, or templates are needed.
What finished output should it hand back so the user can immediately keep moving? Outputs should drive the package structure before extra guidance is added.
Which nearby requests should this skill politely refuse so the boundary stays clean? The exclusion list is the fastest route to better trigger quality.
What matters most here: speed, consistency, auditability, portability, governance, or tone/style fit? Constraints decide how much structure, packaging, and review the skill actually needs.
+
+
+
Why this package is strong
+
触发面保持精简，并锚定在 frontmatter description。
已生成 Skill IR，核心语义可先于平台打包被审查和迁移。
已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。
Conflict handling · decision needed The stated preference leans lightweight or speed-first, while the benchmark mix leans toward governance, review, or heavier evaluation structure.
Are the known limits visible before the package deepens?
+
Does the evidence support the borrowed patterns?
+
Should uncertainty become a clarification question instead of more structure?
+
+
+
+
+
+
+
Output risk profile
+
Markdown readability Tables can render as dense grids with weak hierarchy or poor mobile readability.; Long bullets can make the output look complete while hiding the actual decision logic.
Screenshot and visual capture Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.; Missing screenshots can cause the skill to invent visual references instead of declaring the gap.
Citation and footnote clutter Footnote markers or dense citation notes can interrupt the reading flow.; Evidence can be over-attached to obvious statements and under-attached to risky claims.
+
+
+
Self-repair checks
+
Preview whether each table still reads well when columns are narrow.
Convert any table with paragraph-length cells into bullets or cards.
Check that every screenshot reference points to a real provided or generated asset.
Reword any visual instruction that depends on an unseen screen state.
Remove decorative citations that do not support a material claim.
+
+
+
+
+
+
Artifact design profile
+
Design system: metric editorial
+
Report or brief High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.
Review viewer Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix.
Dashboard or metrics page Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
+
+
+
Visual quality gates
+
Keep the first screen useful without requiring the reader to parse every detail.
Use tables only for comparisons; move explanations below the table.
Keep source notes readable without flooding the body with markers.
Make differences visible instead of hiding them in prose.
Separate author-facing recommendations from reviewer-only evidence.
Completeness · 80 / 100 Name missing inputs, outputs, constraints, or success standards before deepening the package.
Clarity · 90 / 100 Replace broad verbs with observable actions and define what done means.
Consistency · 90 / 100 Check that role, task, format, exclusions, and examples do not contradict each other.
Practicality · 95 / 100 Add runnable steps, examples, or verification cues instead of abstract advice.
Specificity · 70 / 100 Anchor wording in the user's audience, domain nouns, and target outcome.
+
+
+
RTF to skill mapping
+
Role Use a taste-aware creator role with clear audience, tone, and originality boundaries.
Task Generate variants, explain selection logic, and preserve the user's distinctive constraints.
Format Return options with rationale, selection criteria, and refinement paths.
+
+
+
+
+
+
Reference coach
+
No GitHub benchmark scan has been attached to this package yet.
+
+
+
Decide before you deepen
+
+
Choose one pattern to borrow on purpose, not three at once.
+
State one thing this skill will not inherit from the benchmark objects.
+
Only deepen the package after that choice is visible in the boundary or execution flow.
+
+
+
+
+
+
+
Reference synthesis
+
Official skill anatomy and context discipline
Borrow now
Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
Avoid
Do not let packaging or platform concerns swallow the core job boundary.
Hypothesis-test-learn loop
Borrow now
Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.
Avoid
Do not create experimental overhead that exceeds the skill's real risk tier.
Boundary-first design
Borrow now
Borrow the discipline of defining what the skill should not own before growing the package.
Avoid
Do not expand execution assets until route boundaries stay clean.
+
+
+
Borrow now
+
Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.
Borrow the discipline of defining what the skill should not own before growing the package.
+
Use the recommendation by default. Only surface the underlying benchmark tradeoffs when intent is uncertain or a real design conflict needs a deliberate call.
+
+
+
+
+
Top three next moves
+
Tighten trigger and exclusions
The package needs clearer near-neighbor exclusions before it grows.
Add 3 to 5 should-trigger and should-not-trigger examples.
Refine the frontmatter description to name the recurring job and non-goals.
Run a first trigger evaluation pass before expanding the package.
Unlocks: Cleaner routing and fewer accidental activations.
Add the first execution asset
The package is still mostly prose. Add one asset that removes repeated manual work.
Move stable procedural guidance into references if users will need it repeatedly.
Create one deterministic helper script if a repeated step can be executed instead of described.
Keep the main SKILL.md compact and route-oriented.
Unlocks: Stronger execution quality without bloating the entrypoint.
Promote from scaffold to production-ready
The first version exists; the next gain usually comes from adding the smallest useful gates.
Decide whether this skill is personal, team-reused, or library-grade.
Add only the gates that match that risk level.
Record lifecycle metadata and review cadence once reuse becomes real.
Unlocks: A clearer path from exploratory package to maintained asset.
+
+
+
+
+
Recent feedback
+
No lightweight feedback captured yet. Use `yao.py feedback` to record quick review notes.
+
+
+
Promotion status
+
No promotion summary is attached to this package yet.
manifest.json — Lifecycle and portability metadata
references — Extended guidance and reusable notes
scripts — Deterministic helpers or local tooling
evals — Trigger and quality checks
reports — Generated evidence and overview artifacts
+
+
+
First-pass review frame
+
+
Does the trigger stay narrow enough for the intended job?
+
Does the archetype match the real reuse level?
+
Are we adding structure faster than we are adding reliability?
+
Should the next step be trigger tightening, execution assets, or portability hardening?
+
+
+
+
+
+
+
Authoring discipline
+
+
Name unresolved assumptions before deepening the package.
+
Keep the package no larger than the recurring job requires.
+
Touch only files that directly support the requested change.
+
Tie every meaningful new artifact to a check or reviewer note.
+
+
+
+
Reviewer guardrails
+
+
Block speculative features that are not backed by real workflow variation.
+
Move unverifiable ideas into next-step candidates instead of baseline structure.
+
Reject decorative folders, reports, or governance that do not reduce risk.
+
Ask for one high-leverage clarification when job, output, or exclusion is still fuzzy.
+
+
+
+
+
+
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review-viewer.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review-viewer.json
new file mode 100644
index 00000000..d956d9a5
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review-viewer.json
@@ -0,0 +1,1921 @@
+{
+ "overview": {
+ "skill_summary": {
+ "name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "display_name": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "maturity": "production",
+ "updated_at": "2026-06-15",
+ "core_value": "把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。",
+ "audience": "Skill 作者、复用团队和后续 reviewer。",
+ "deliverables": [
+ "SKILL.md",
+ "agents/interface.yaml",
+ "reports/skill-ir.json",
+ "reports/compiled_targets.md",
+ "reports/output_quality_scorecard.md",
+ "reports/conformance_matrix.md",
+ "reports/security_trust_report.md",
+ "reports/skill_atlas.html",
+ "reports/registry_audit.md",
+ "reports/package_verification.md",
+ "reports/install_simulation.md",
+ "reports/upgrade_check.md",
+ "reports/adoption_drift_report.md",
+ "reports/review_waivers.md",
+ "reports/review_annotations.md",
+ "reports/review-studio.html",
+ "reports/skill-overview.html"
+ ],
+ "flow": [
+ "输入材料",
+ "Skill 包体",
+ "可复用能力"
+ ]
+ },
+ "scorecard": {
+ "completeness_score": {
+ "label": "完整度",
+ "score": 92,
+ "reasons": [
+ "SKILL.md 已存在，是 Skill 的入口。",
+ "README.md 已存在，便于人工阅读。",
+ "agents/interface.yaml 已存在，便于跨平台适配。",
+ "manifest.json 已存在，生命周期信息可追踪。",
+ "reports/ 已存在，生成证据可以随包体迁移。"
+ ]
+ },
+ "trigger_score": {
+ "label": "触发清晰",
+ "score": 100,
+ "reasons": [
+ "frontmatter description 已存在，具备基础路由面。",
+ "description 有足够长度说明任务边界。",
+ "description 已包含使用场景或排除边界信号。",
+ "evals/ 已存在，可承载触发样例或质量检查。",
+ "intent-confidence 报告已生成，可辅助判断触发稳定性。"
+ ]
+ },
+ "evidence_score": {
+ "label": "证据充分",
+ "score": 60,
+ "reasons": [
+ "已生成 12 / 20 类报告证据。",
+ "skill-ir.json 已存在。",
+ "compiled_targets.json 已存在。",
+ "intent-dialogue.json 已存在。",
+ "证据不足：缺少 output_quality_scorecard.json, conformance_matrix.json, security_trust_report.json。"
+ ]
+ },
+ "maintainability_score": {
+ "label": "可维护性",
+ "score": 85,
+ "reasons": [
+ "SKILL.md 约 864 个词/字。",
+ "入口文件保持克制，可维护性较好。",
+ "references/ 已承载扩展指导。",
+ "scripts/ 证据不足，重复执行逻辑可能仍靠人工。",
+ "evals/ 已承载可迁移检查。"
+ ]
+ },
+ "portability_score": {
+ "label": "可迁移性",
+ "score": 100,
+ "reasons": [
+ "agents/interface.yaml 已存在。",
+ "manifest.json 已存在。",
+ "目标平台或 adapter target 已声明。",
+ "入口文件未发现明显私有绝对路径。"
+ ]
+ },
+ "context_cost": {
+ "label": "上下文成本",
+ "score": 78,
+ "reasons": [
+ "入口约 864 个词/字，references 约 565 个词/字。",
+ "分数越高代表上下文成本越低。",
+ "上下文成本处于可控区间。"
+ ]
+ }
+ },
+ "capability_profile": {
+ "archetype": "production",
+ "task_family": "Creative generation",
+ "maturity": "production",
+ "trigger_strength": "手动触发 + description 路由",
+ "reuse_scope": "跨平台",
+ "matrix": {
+ "execution_certainty": 72,
+ "knowledge_density": 80
+ }
+ },
+ "principle_model": {
+ "nodes": [
+ {
+ "title": "意图澄清",
+ "body": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Ask one focused question and update intent context before adding assets."
+ }
+ ],
+ "layers": [
+ "入口层",
+ "参考层",
+ "脚本层",
+ "评估层",
+ "报告层"
+ ]
+ },
+ "contract_boundary": {
+ "trigger": {
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "inputs": [
+ "用户提供的工作流、提示词、文档、记录或散乱笔记",
+ "期望沉淀的复用场景、排除项、约束和质量标准"
+ ],
+ "outputs": [
+ "可路由的 SKILL.md",
+ "agents/interface.yaml 元数据",
+ "必要的 references、scripts、evals、reports 证据",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ],
+ "should_trigger": [
+ "团队复用前，需要明确触发边界、质量证据和维护责任。",
+ "用户说出类似需求时：当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "已有原始资料，但还没有清晰输入、输出和后续迭代路径。"
+ ],
+ "should_not_trigger": [
+ "只需要一次性回答、没有复用价值的临时请求。",
+ "要求直接执行相邻任务，而不是沉淀或使用这个 Skill。",
+ "缺少必要事实且用户不允许澄清的场景。"
+ ],
+ "boundary_cards": [
+ {
+ "label": "Owned",
+ "body": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。"
+ },
+ {
+ "label": "Adjacent",
+ "body": "相邻任务需要先确认是否应转为独立 Skill。"
+ },
+ {
+ "label": "Excluded",
+ "body": "不替代人工事实核查，也不静默扩大职责。"
+ }
+ ]
+ },
+ "quality_review": {
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Adoption Drift Report，可把本地使用反馈转为下一轮迭代信号。",
+ "已生成 Review Waivers 台账，可记录 reviewer 对 warning 风险的批准、理由和到期时间。",
+ "已生成 Review Annotations 台账，可把 reviewer 批注挂到 gate、文件和行号。"
+ ],
+ "gaps": [
+ "证据充分需要补强：已生成 12 / 20 类报告证据。"
+ ],
+ "recommendations": [
+ "先改触发边界，再扩展工作流。",
+ "只把重复且稳定的步骤沉淀为脚本。",
+ "每次升级后重新生成报告并检查分数原因。"
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.",
+ "Keep the first screen useful without requiring the reader to parse every detail.",
+ "Use tables only for comparisons; move explanations below the table.",
+ "Keep source notes readable without flooding the body with markers."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 85.0,
+ "highlights": [
+ "Primary prompt task family: Creative generation.",
+ "Complexity: complex — multiple inputs, constraints, or task families require tradeoff handling",
+ "Completeness: 80/100.",
+ "Clarity: 90/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 77,
+ "band": "stable-first-pass"
+ },
+ "highlights": [
+ "Stability: stable-first-pass (77/100).",
+ "Owned job: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "Leverage: Clarify the real job boundary — Ask one focused question and update intent context before adding assets.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ ]
+ }
+ },
+ "risk_governance": {
+ "risks": [
+ {
+ "name": "误触发风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "frontmatter description 已存在，具备基础路由面。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "输出漂移风险",
+ "impact": 2,
+ "probability": 2,
+ "signal": "已生成 12 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "证据不足风险",
+ "impact": 3,
+ "probability": 2,
+ "signal": "已生成 12 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "包体膨胀风险",
+ "impact": 2,
+ "probability": 2,
+ "signal": "SKILL.md 约 864 个词/字。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "跨平台迁移风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "agents/interface.yaml 已存在。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ }
+ ],
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "table",
+ "report",
+ "brief"
+ ],
+ "score": 4,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "screenshot",
+ "visual",
+ "screen"
+ ],
+ "score": 3,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "citation",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "copy",
+ "content"
+ ],
+ "score": 2,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ },
+ {
+ "key": "tutorial_quality",
+ "label": "Tutorial quality",
+ "matched_keywords": [
+ "tutorial"
+ ],
+ "score": 1,
+ "risks": [
+ "Generic section headings make the tutorial feel templated instead of fitted to the learner's task.",
+ "Steps may explain what to do without naming the exact check that proves the step worked.",
+ "Examples can become too abstract when the input material is concrete."
+ ],
+ "constraints": [
+ "Write headings from the user's domain nouns and desired outcome, not from generic labels like Overview or Key Points.",
+ "Pair each major step with a visible success check or expected intermediate output.",
+ "Use one concrete worked example before adding broad principles."
+ ],
+ "self_repair": [
+ "Replace generic H2/H3 headings with task-specific headings before final output.",
+ "Scan every numbered step for a missing verification cue."
+ ]
+ }
+ ],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ]
+ },
+ "package_assets": {
+ "entries": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 0
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 34
+ }
+ ],
+ "file_count": 42,
+ "folder_count": 4,
+ "distribution": [
+ {
+ "label": "SKILL.md",
+ "value": 1
+ },
+ {
+ "label": "README.md",
+ "value": 1
+ },
+ {
+ "label": "agents/interface.yaml",
+ "value": 1
+ },
+ {
+ "label": "manifest.json",
+ "value": 1
+ },
+ {
+ "label": "references",
+ "value": 2
+ },
+ {
+ "label": "scripts",
+ "value": 1
+ },
+ {
+ "label": "evals",
+ "value": 2
+ },
+ {
+ "label": "reports",
+ "value": 34
+ }
+ ]
+ },
+ "iteration_roadmap": {
+ "items": [
+ {
+ "title": "Tighten trigger and exclusions",
+ "why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "actions": [
+ "Add 3 to 5 should-trigger and should-not-trigger examples.",
+ "Refine the frontmatter description to name the recurring job and non-goals.",
+ "Run a first trigger evaluation pass before expanding the package."
+ ],
+ "unlocks": "Cleaner routing and fewer accidental activations."
+ },
+ {
+ "title": "Add the first execution asset",
+ "why": "The package is still mostly prose. Add one asset that removes repeated manual work.",
+ "actions": [
+ "Move stable procedural guidance into references if users will need it repeatedly.",
+ "Create one deterministic helper script if a repeated step can be executed instead of described.",
+ "Keep the main SKILL.md compact and route-oriented."
+ ],
+ "unlocks": "Stronger execution quality without bloating the entrypoint."
+ },
+ {
+ "title": "Promote from scaffold to production-ready",
+ "why": "The first version exists; the next gain usually comes from adding the smallest useful gates.",
+ "actions": [
+ "Decide whether this skill is personal, team-reused, or library-grade.",
+ "Add only the gates that match that risk level.",
+ "Record lifecycle metadata and review cadence once reuse becomes real."
+ ],
+ "unlocks": "A clearer path from exploratory package to maintained asset."
+ }
+ ]
+ },
+ "report_contract": {
+ "schema_version": "2.0",
+ "html_report": "reports/skill-overview.html",
+ "language": "zh-CN",
+ "default_language": "zh-CN",
+ "languages": [
+ "zh-CN",
+ "en"
+ ],
+ "layout": "kami-white-audit-v2",
+ "nav_labels": [
+ "技能概述",
+ "总览指标",
+ "能力画像",
+ "原理结构",
+ "契约边界",
+ "质量评估",
+ "风险治理",
+ "包体资产",
+ "迭代路线"
+ ],
+ "nav_labels_en": [
+ "Overview",
+ "Metrics",
+ "Profile",
+ "Principle",
+ "Contract",
+ "Quality",
+ "Risk",
+ "Assets",
+ "Roadmap"
+ ]
+ },
+ "name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "display_name": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "logic_steps": [
+ "读取用户输入，识别目标受众、内容渠道、关键词、品牌边界、禁止表达和必须覆盖的信息。",
+ "对照 `references/brief-structure.md` 生成简报骨架：目标、洞察、角度、关键词、结构、证据、风险和验收标准。",
+ "如果缺少核心输入，先提出最多三个澄清问题，不用假设补全用户没有提供的事实。",
+ "用 `evals/trigger_cases.jsonl` 检查当前请求是否属于 GEO 内容简报场景。",
+ "用 `evals/output_cases.jsonl` 检查输出是否包含输入摘要、内容结构、证据要求、禁区和下一步动作。"
+ ],
+ "usage_steps": [
+ "当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "Use this skill when the request matches: 将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。"
+ ],
+ "package_map": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 0
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 34
+ }
+ ],
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Adoption Drift Report，可把本地使用反馈转为下一轮迭代信号。",
+ "已生成 Review Waivers 台账，可记录 reviewer 对 warning 风险的批准、理由和到期时间。",
+ "已生成 Review Annotations 台账，可把 reviewer 批注挂到 gate、文件和行号。"
+ ],
+ "scenarios": [
+ "团队复用前，需要明确触发边界、质量证据和维护责任。",
+ "用户说出类似需求时：当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "已有原始资料，但还没有清晰输入、输出和后续迭代路径。"
+ ],
+ "trigger_contract": {
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "io_contract": {
+ "inputs": [
+ "用户提供的工作流、提示词、文档、记录或散乱笔记",
+ "期望沉淀的复用场景、排除项、约束和质量标准"
+ ],
+ "outputs": [
+ "可路由的 SKILL.md",
+ "agents/interface.yaml 元数据",
+ "必要的 references、scripts、evals、reports 证据",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ]
+ },
+ "principles": [
+ {
+ "title": "意图澄清",
+ "body": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Ask one focused question and update intent context before adding assets."
+ }
+ ],
+ "roadmap": [
+ {
+ "title": "Tighten trigger and exclusions",
+ "why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "actions": [
+ "Add 3 to 5 should-trigger and should-not-trigger examples.",
+ "Refine the frontmatter description to name the recurring job and non-goals.",
+ "Run a first trigger evaluation pass before expanding the package."
+ ],
+ "unlocks": "Cleaner routing and fewer accidental activations."
+ },
+ {
+ "title": "Add the first execution asset",
+ "why": "The package is still mostly prose. Add one asset that removes repeated manual work.",
+ "actions": [
+ "Move stable procedural guidance into references if users will need it repeatedly.",
+ "Create one deterministic helper script if a repeated step can be executed instead of described.",
+ "Keep the main SKILL.md compact and route-oriented."
+ ],
+ "unlocks": "Stronger execution quality without bloating the entrypoint."
+ },
+ {
+ "title": "Promote from scaffold to production-ready",
+ "why": "The first version exists; the next gain usually comes from adding the smallest useful gates.",
+ "actions": [
+ "Decide whether this skill is personal, team-reused, or library-grade.",
+ "Add only the gates that match that risk level.",
+ "Record lifecycle metadata and review cadence once reuse becomes real."
+ ],
+ "unlocks": "A clearer path from exploratory package to maintained asset."
+ }
+ ],
+ "cards": [],
+ "introduction": [
+ "这份报告用于快速理解新生成 Skill 的定位、原理、触发边界和交付内容。",
+ "先确认重复任务、真实输入形态和可交付输出，再决定是否继续加 references、scripts 或 evals。",
+ "如果需求仍然模糊，优先回到 intent dialogue 收紧边界，再扩展包体结构。"
+ ],
+ "benchmark_highlights": [],
+ "skill_ir": {
+ "schema_version": "2.0.0",
+ "target_count": 5,
+ "trigger_samples": 1,
+ "output_eval_cases": 0
+ },
+ "compiled_targets": {
+ "ok": true,
+ "schema_version": "1.0",
+ "summary": {
+ "target_count": 5,
+ "pass_count": 5,
+ "warn_count": 0,
+ "block_count": 0,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "targets": [
+ {
+ "target": "openai",
+ "status": "pass",
+ "adapter_mode": "metadata-adapter",
+ "degradation_strategy": "metadata-adapter",
+ "native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
+ "permission_enforcement": "metadata-only",
+ "generated_files": [
+ "targets/openai/adapter.json",
+ "targets/openai/agents/openai.yaml"
+ ],
+ "unsupported_features": [
+ "client-native script permission prompts are represented as permission contract metadata"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "claude",
+ "status": "pass",
+ "adapter_mode": "neutral-source-plus-adapter",
+ "degradation_strategy": "neutral-source-plus-adapter",
+ "native_surface": "Claude-compatible neutral source folder with adapter notes",
+ "permission_enforcement": "metadata-fallback",
+ "generated_files": [
+ "targets/claude/adapter.json",
+ "targets/claude/README.md"
+ ],
+ "unsupported_features": [
+ "vendor-native metadata fields are carried as adapter JSON and README notes"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "generic",
+ "status": "pass",
+ "adapter_mode": "agent-skills-compatible",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills compatible neutral package",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "targets/generic/adapter.json"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "agent-skills-compatible",
+ "status": "pass",
+ "adapter_mode": "neutral-agent-skills-source",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills standard source tree",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "SKILL.md",
+ "agents/interface.yaml"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "permission_enforcement": "client-or-workspace-trust",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "unsupported_features": [
+ "VS Code installation scope is documented but not installed by this compiler"
+ ],
+ "warnings": []
+ }
+ ],
+ "failures": [],
+ "warnings": []
+ },
+ "output_quality": {},
+ "output_execution": {
+ "ok": false,
+ "summary": {},
+ "runner": {},
+ "failures": []
+ },
+ "output_blind_review": {
+ "summary": {},
+ "seed": "",
+ "pair_count": 0,
+ "answer_key_separate": false
+ },
+ "output_review_kit": {
+ "ok": false,
+ "summary": {},
+ "artifacts": {},
+ "failures": []
+ },
+ "output_review_adjudication": {
+ "ok": false,
+ "summary": {},
+ "reviewer": "",
+ "reviewed_at": "",
+ "failures": []
+ },
+ "benchmark_reproducibility": {
+ "ok": false,
+ "summary": {},
+ "commit": "",
+ "missing_artifacts": [],
+ "limitations": []
+ },
+ "runtime_conformance": {},
+ "runtime_permissions": {
+ "ok": false,
+ "summary": {},
+ "expected_capabilities": [],
+ "targets": [],
+ "failures": []
+ },
+ "trust_security": {},
+ "skill_atlas": {},
+ "registry_distribution": {
+ "ok": false,
+ "package": {},
+ "failures": [],
+ "warnings": []
+ },
+ "package_verification": {
+ "ok": false,
+ "summary": {},
+ "failures": [],
+ "warnings": []
+ },
+ "install_simulation": {
+ "ok": false,
+ "summary": {},
+ "failures": [],
+ "warnings": []
+ },
+ "upgrade_check": {
+ "ok": false,
+ "summary": {},
+ "upgrade_diff": {},
+ "release_notes": [],
+ "failures": [],
+ "warnings": []
+ },
+ "adoption_drift": {
+ "ok": true,
+ "summary": {
+ "event_count": 0,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "edited_count": 0,
+ "rejected_count": 0,
+ "missed_count": 0,
+ "failed_count": 0,
+ "adoption_rate": 0,
+ "missed_trigger_count": 0,
+ "wrong_trigger_count": 0,
+ "bad_output_count": 0,
+ "script_error_count": 0,
+ "missing_resource_count": 0,
+ "review_overdue_count": 0,
+ "risk_band": "no-data",
+ "event_types": {},
+ "failure_types": {},
+ "source_types": {},
+ "command_counts": {}
+ },
+ "next_iteration_candidates": [
+ {
+ "signal": "no telemetry",
+ "recommendation": "Start with a small metadata-only sample before using telemetry for release decisions."
+ }
+ ],
+ "privacy_contract": {
+ "storage": "local-first",
+ "event_scope": "metadata-only",
+ "raw_content_allowed": false,
+ "raw_event_log_packaged": false,
+ "blocked_fields": [
+ "content",
+ "input",
+ "inputs",
+ "message",
+ "messages",
+ "note",
+ "output",
+ "outputs",
+ "prompt",
+ "raw",
+ "text",
+ "transcript"
+ ]
+ },
+ "failures": []
+ },
+ "review_waivers": {
+ "ok": true,
+ "summary": {
+ "waiver_count": 0,
+ "active_count": 0,
+ "expired_count": 0,
+ "invalid_count": 0,
+ "covered_gate_count": 0,
+ "covered_gate_keys": [],
+ "waiver_candidate_count": 0,
+ "waiverable_open_count": 0,
+ "non_waivable_count": 0
+ },
+ "policy": {
+ "blocker_waivers_allowed": false,
+ "minimum_reason_chars": 20,
+ "expires_required": true,
+ "known_gate_keys": [
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ]
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "review_annotations": {
+ "ok": true,
+ "summary": {
+ "annotation_count": 0,
+ "open_count": 0,
+ "resolved_count": 0,
+ "deferred_count": 0,
+ "open_blocker_count": 0,
+ "open_warning_count": 0,
+ "linked_gate_count": 0,
+ "target_missing_count": 0,
+ "failure_count": 0
+ },
+ "annotations": [],
+ "failures": []
+ },
+ "world_class_evidence_plan": {
+ "ok": false,
+ "summary": {},
+ "tasks": [],
+ "source_audit": {}
+ },
+ "world_class_evidence_ledger": {
+ "ok": false,
+ "summary": {},
+ "entries": [],
+ "source_plan": {}
+ },
+ "synthesis_highlights": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "Borrow the discipline of defining what the skill should not own before growing the package."
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.",
+ "Keep the first screen useful without requiring the reader to parse every detail.",
+ "Use tables only for comparisons; move explanations below the table.",
+ "Keep source notes readable without flooding the body with markers."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 85.0,
+ "highlights": [
+ "Primary prompt task family: Creative generation.",
+ "Complexity: complex — multiple inputs, constraints, or task families require tradeoff handling",
+ "Completeness: 80/100.",
+ "Clarity: 90/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 77,
+ "band": "stable-first-pass"
+ },
+ "highlights": [
+ "Stability: stable-first-pass (77/100).",
+ "Owned job: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "Leverage: Clarify the real job boundary — Ask one focused question and update intent context before adding assets.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ ]
+ },
+ "metadata": {
+ "canonical_format": "agent-skills",
+ "targets": [
+ "openai",
+ "claude",
+ "generic",
+ "vscode"
+ ],
+ "maturity_tier": "production",
+ "skill_archetype": "production",
+ "updated_at": "2026-06-15"
+ }
+ },
+ "intent_confidence": {
+ "score": 10,
+ "band": "low",
+ "gate_passed": false,
+ "strengths": [],
+ "gaps": [
+ {
+ "key": "job_specificity",
+ "label": "Recurring job is still generic",
+ "reason": "The current job statement sounds more like a packaging goal than a concrete repeated task.",
+ "severity": "high"
+ },
+ {
+ "key": "real_inputs",
+ "label": "Real inputs are missing",
+ "reason": "Without real inputs, it is hard to choose assets, scripts, or examples.",
+ "severity": "high"
+ },
+ {
+ "key": "primary_output",
+ "label": "Primary output is missing",
+ "reason": "The package does not yet know what it must hand back.",
+ "severity": "high"
+ },
+ {
+ "key": "exclusions",
+ "label": "Near-neighbor exclusions are missing",
+ "reason": "The route may blur into nearby requests without an exclusion list.",
+ "severity": "high"
+ },
+ {
+ "key": "constraints",
+ "label": "Constraints are missing",
+ "reason": "The package does not yet know which tradeoffs matter most.",
+ "severity": "high"
+ },
+ {
+ "key": "standards",
+ "label": "Quality bar is implied, not explicit",
+ "reason": "The first evaluation target is still underspecified.",
+ "severity": "medium"
+ }
+ ],
+ "follow_up_questions": [
+ {
+ "slot": "job",
+ "question": "If you say it plainly, what concrete repeated task should this skill own every time?",
+ "why": "A skill needs a real recurring job, not only a generic packaging goal.",
+ "list": false,
+ "label": "Recurring job is still generic",
+ "severity": "high"
+ },
+ {
+ "slot": "real_inputs",
+ "question": "What material will people actually hand to this skill in practice?",
+ "why": "Real input shape decides whether references, scripts, or examples are needed.",
+ "list": true,
+ "label": "Real inputs are missing",
+ "severity": "high"
+ },
+ {
+ "slot": "primary_output",
+ "question": "What finished hand-back should this skill return so the next person can keep moving?",
+ "why": "The output is the anchor for package design and review.",
+ "list": false,
+ "label": "Primary output is missing",
+ "severity": "high"
+ }
+ ],
+ "anchor_sentence": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "recommended_action": "Pause before deep authoring and close the highest-leverage gaps first.",
+ "context": {
+ "job": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "real_inputs": [],
+ "primary_output": "",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "exclusions": [],
+ "constraints": [],
+ "standards": [],
+ "correction": "",
+ "user_references": []
+ }
+ },
+ "intent": {
+ "skill_name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "focus": "trigger-and-output",
+ "opening_frame": "Let's start from the real work, the result you care about, and the standards that matter here. We can make the structure clearer after that.",
+ "reference_note": "If you already have examples you admire, bring them in. We will learn the pattern, not copy the source.",
+ "conversation_path": [
+ "Start with the user's own words, not package vocabulary.",
+ "Reflect the job, output, and non-goals back in one clean sentence.",
+ "Only then offer a tiny scaffold if it would help the user move faster."
+ ],
+ "opening_styles": [
+ {
+ "label": "温柔陪伴型",
+ "best_when": "用户想法还散、还在试探，或者需要先被接住。",
+ "message": "我们先不急着把它说成一个很完整的 skill。你就像跟我聊天一样，先说说你最想让它以后稳稳接住哪类重复工作；如果它做得很理想，最后应该交回你一个什么结果。"
+ },
+ {
+ "label": "专业教练型",
+ "best_when": "用户目标比较明确，希望被高效带着走。",
+ "message": "我们先把这件事讲清楚，再决定 skill 怎么设计。你先告诉我三件事：它要接住的重复任务是什么，别人通常会给它什么材料，最后你希望它交付什么结果。"
+ },
+ {
+ "label": "共创伙伴型",
+ "best_when": "用户已经有一些想法，希望一起打磨，不想被填表。",
+ "message": "我们把这次当成一次共创。你先给我一个粗糙版本就行，我先帮你看它真正的核心任务是什么，再一起决定边界、结构和接下来最值的一步。"
+ }
+ ],
+ "optional_scaffold": [
+ "The repeated job it should reliably handle",
+ "The real inputs people will hand to it",
+ "The useful output it should hand back",
+ "What it should clearly refuse"
+ ],
+ "questions": [
+ {
+ "question": "If this skill worked beautifully, what recurring job would it reliably handle for the user every time?",
+ "why": "This reveals the real job-to-be-done and gives the package a humane center instead of a guessed prompt shape."
+ },
+ {
+ "question": "When someone reaches for this skill in the real world, what materials will they actually hand to it?",
+ "why": "Input shape decides whether references, scripts, or templates are needed."
+ },
+ {
+ "question": "What finished output should it hand back so the user can immediately keep moving?",
+ "why": "Outputs should drive the package structure before extra guidance is added."
+ },
+ {
+ "question": "Which nearby requests should this skill politely refuse so the boundary stays clean?",
+ "why": "The exclusion list is the fastest route to better trigger quality."
+ },
+ {
+ "question": "What matters most here: speed, consistency, auditability, portability, governance, or tone/style fit?",
+ "why": "Constraints decide how much structure, packaging, and review the skill actually needs."
+ },
+ {
+ "question": "Do you already have any references you want this skill to learn from, such as a repo, product, page, workflow, or prompt example?",
+ "why": "A good reference can raise the quality bar quickly, but the skill should only borrow patterns and standards, never copy wording or confidential material."
+ },
+ {
+ "question": "What repeated manual step should become a deterministic asset first?",
+ "why": "This usually reveals whether a script or reference should be created next."
+ }
+ ],
+ "output": {
+ "capability_sentence": "Geo Content Brief Skill should turn a recurring request into a reliable reusable output without widening the boundary unnecessarily.",
+ "required_capture": [
+ "recurring job",
+ "real inputs",
+ "required outputs",
+ "exclusions",
+ "constraints",
+ "reference preferences",
+ "first evaluation target"
+ ],
+ "recommended_first_gate": "trigger and boundary"
+ }
+ },
+ "reference": {
+ "skill_name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "scan_focus": [
+ {
+ "label": "Execution pattern",
+ "reason": "There is deterministic logic in scripts, so compare how strong references separate prose from executable steps."
+ },
+ {
+ "label": "Portability pattern",
+ "reason": "The package carries neutral metadata, so scan how good references preserve semantics across targets without forking source."
+ },
+ {
+ "label": "Method pattern",
+ "reason": "Use the core job description as the anchor for comparison: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ }
+ ],
+ "external_references": [],
+ "user_references": [],
+ "local_constraints": [],
+ "borrow_plan": [
+ "External benchmark first: let high-quality public references define the upper bound for method, structure, execution, or portability.",
+ "User references second: use them to understand taste, standards, and directional preferences without copying source phrasing.",
+ "Local fit third: use local assets only to detect naming conflicts, private dependencies, or compatibility constraints.",
+ "Borrow patterns, not prose: extract loops, boundaries, metadata, and operator flow without copying source-specific language.",
+ "Keep the package light: reject any borrowed pattern that increases context cost faster than it increases reliability."
+ ],
+ "priority_rule": "External benchmark objects set the pattern ceiling. User references refine taste and standards. Local files only calibrate fit, risk, and compatibility.",
+ "non_goals": [
+ "Do not copy source prose or branding into the new skill.",
+ "Do not import gates that cost more context than they save.",
+ "Do not use benchmark scanning to justify scope creep.",
+ "Do not let local historical habits outrank stronger public benchmarks."
+ ]
+ },
+ "iteration": {
+ "summary": {
+ "skill_name": "geo-content-brief-skill",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "maturity_tier": "scaffold",
+ "selection_rule": "Pick the three smallest next steps that increase reliability more than they increase context cost.",
+ "recommended_now": "Tighten trigger and exclusions",
+ "recommended_now_why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "defer_for_now": "Promote from scaffold to production-ready"
+ },
+ "directions": [
+ {
+ "priority": 1,
+ "title": "Tighten trigger and exclusions",
+ "why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "actions": [
+ "Add 3 to 5 should-trigger and should-not-trigger examples.",
+ "Refine the frontmatter description to name the recurring job and non-goals.",
+ "Run a first trigger evaluation pass before expanding the package."
+ ],
+ "unlocks": "Cleaner routing and fewer accidental activations.",
+ "do_now": "Do this first.",
+ "wait_on": "Wait to add broader structure until this move clearly improves reliability."
+ },
+ {
+ "priority": 2,
+ "title": "Add the first execution asset",
+ "why": "The package is still mostly prose. Add one asset that removes repeated manual work.",
+ "actions": [
+ "Move stable procedural guidance into references if users will need it repeatedly.",
+ "Create one deterministic helper script if a repeated step can be executed instead of described.",
+ "Keep the main SKILL.md compact and route-oriented."
+ ],
+ "unlocks": "Stronger execution quality without bloating the entrypoint.",
+ "do_now": "Do this after the first move lands cleanly.",
+ "wait_on": "Wait until the package has evidence that this extra structure is justified."
+ },
+ {
+ "priority": 3,
+ "title": "Promote from scaffold to production-ready",
+ "why": "The first version exists; the next gain usually comes from adding the smallest useful gates.",
+ "actions": [
+ "Decide whether this skill is personal, team-reused, or library-grade.",
+ "Add only the gates that match that risk level.",
+ "Record lifecycle metadata and review cadence once reuse becomes real."
+ ],
+ "unlocks": "A clearer path from exploratory package to maintained asset.",
+ "do_now": "Do this after the first move lands cleanly.",
+ "wait_on": "Wait until the package has evidence that this extra structure is justified."
+ }
+ ]
+ },
+ "feedback": {},
+ "baseline": {},
+ "compare": {},
+ "promotion": {},
+ "benchmark": {},
+ "reference_synthesis": {
+ "skill_name": "geo-content-brief-skill",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "intent_confidence": {
+ "score": 10,
+ "band": "low",
+ "gate_passed": false
+ },
+ "github_benchmarks": [],
+ "source_tracks": [
+ {
+ "source_type": "official",
+ "name": "Official skill anatomy and context discipline",
+ "evidence_mode": "curated-pattern-track",
+ "matched_keywords": [
+ "general fit"
+ ],
+ "borrow": "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "avoid": "Do not let packaging or platform concerns swallow the core job boundary.",
+ "why_relevant": "This track matches: general fit."
+ },
+ {
+ "source_type": "research",
+ "name": "Hypothesis-test-learn loop",
+ "evidence_mode": "curated-pattern-track",
+ "matched_keywords": [
+ "general fit"
+ ],
+ "borrow": "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "avoid": "Do not create experimental overhead that exceeds the skill's real risk tier.",
+ "why_relevant": "This track matches: general fit."
+ },
+ {
+ "source_type": "principles",
+ "name": "Boundary-first design",
+ "evidence_mode": "curated-pattern-track",
+ "matched_keywords": [
+ "general fit"
+ ],
+ "borrow": "Borrow the discipline of defining what the skill should not own before growing the package.",
+ "avoid": "Do not expand execution assets until route boundaries stay clean.",
+ "why_relevant": "This track matches: general fit."
+ }
+ ],
+ "synthesis": {
+ "borrow_now": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "Borrow the discipline of defining what the skill should not own before growing the package."
+ ],
+ "avoid_now": [
+ "Do not let packaging or platform concerns swallow the core job boundary.",
+ "Do not create experimental overhead that exceeds the skill's real risk tier.",
+ "Do not expand execution assets until route boundaries stay clean."
+ ],
+ "quality_risers": [
+ "Use GitHub repositories for concrete package and workflow patterns.",
+ "Use curated official or commercial tracks for entrypoint and operator ergonomics.",
+ "Use research tracks to justify the smallest evaluation loop that still catches regressions.",
+ "Use principle tracks to keep the package small, boundary-aware, and outcome-driven."
+ ],
+ "pattern_gate": {
+ "threshold": 2,
+ "source_count": 3,
+ "accepted": [
+ {
+ "name": "Official skill anatomy and context discipline",
+ "source_type": "official",
+ "borrow": "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "avoid": "Do not let packaging or platform concerns swallow the core job boundary.",
+ "gates": {
+ "recurrence": true,
+ "generativity": true,
+ "distinctiveness": false,
+ "boundary": true
+ },
+ "passed": [
+ "recurrence",
+ "generativity",
+ "boundary"
+ ],
+ "missing": [
+ "distinctiveness"
+ ],
+ "score": 3
+ },
+ {
+ "name": "Hypothesis-test-learn loop",
+ "source_type": "research",
+ "borrow": "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "avoid": "Do not create experimental overhead that exceeds the skill's real risk tier.",
+ "gates": {
+ "recurrence": true,
+ "generativity": true,
+ "distinctiveness": false,
+ "boundary": true
+ },
+ "passed": [
+ "recurrence",
+ "generativity",
+ "boundary"
+ ],
+ "missing": [
+ "distinctiveness"
+ ],
+ "score": 3
+ },
+ {
+ "name": "Boundary-first design",
+ "source_type": "principles",
+ "borrow": "Borrow the discipline of defining what the skill should not own before growing the package.",
+ "avoid": "Do not expand execution assets until route boundaries stay clean.",
+ "gates": {
+ "recurrence": true,
+ "generativity": true,
+ "distinctiveness": false,
+ "boundary": true
+ },
+ "passed": [
+ "recurrence",
+ "generativity",
+ "boundary"
+ ],
+ "missing": [
+ "distinctiveness"
+ ],
+ "score": 3
+ }
+ ],
+ "deferred": [],
+ "summary": "3 accepted, 0 deferred using threshold 2/4."
+ },
+ "conflicts": [
+ {
+ "key": "lightweight_vs_governance",
+ "summary": "The stated preference leans lightweight or speed-first, while the benchmark mix leans toward governance, review, or heavier evaluation structure.",
+ "user_preference": "lightweight or speed-first",
+ "benchmark_pressure": "governance or evaluation-heavy patterns"
+ }
+ ],
+ "recommendation": {
+ "summary": "Start by borrowing this pattern: Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts. Avoid this for the first pass: Do not let packaging or platform concerns swallow the core job boundary.",
+ "borrow_now": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed."
+ ],
+ "avoid_for_now": [
+ "Do not let packaging or platform concerns swallow the core job boundary.",
+ "Do not create experimental overhead that exceeds the skill's real risk tier."
+ ],
+ "why": "There is a real design conflict to resolve: The stated preference leans lightweight or speed-first, while the benchmark mix leans toward governance, review, or heavier evaluation structure.",
+ "user_decision_required": true
+ },
+ "visibility": {
+ "mode": "explicit",
+ "user_decision_required": true,
+ "reasons": [
+ "intent_uncertain",
+ "design_conflict"
+ ],
+ "user_note": "Surface the recommendation because intent is still settling or there is a real design conflict that needs a user call.",
+ "reviewer_note": "Keep the full benchmark and synthesis evidence visible for authors and reviewers."
+ },
+ "decision_prompt": "Use the recommendation by default. Only surface the underlying benchmark tradeoffs when intent is uncertain or a real design conflict needs a deliberate call.",
+ "source_mix": {
+ "github_benchmarks": 0,
+ "curated_tracks": 3,
+ "user_references": 0
+ }
+ }
+ },
+ "output_risk": {
+ "skill_name": "geo-content-brief-skill",
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "table",
+ "report",
+ "brief"
+ ],
+ "score": 4,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "screenshot",
+ "visual",
+ "screen"
+ ],
+ "score": 3,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "citation",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "copy",
+ "content"
+ ],
+ "score": 2,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ },
+ {
+ "key": "tutorial_quality",
+ "label": "Tutorial quality",
+ "matched_keywords": [
+ "tutorial"
+ ],
+ "score": 1,
+ "risks": [
+ "Generic section headings make the tutorial feel templated instead of fitted to the learner's task.",
+ "Steps may explain what to do without naming the exact check that proves the step worked.",
+ "Examples can become too abstract when the input material is concrete."
+ ],
+ "constraints": [
+ "Write headings from the user's domain nouns and desired outcome, not from generic labels like Overview or Key Points.",
+ "Pair each major step with a visible success check or expected intermediate output.",
+ "Use one concrete worked example before adding broad principles."
+ ],
+ "self_repair": [
+ "Replace generic H2/H3 headings with task-specific headings before final output.",
+ "Scan every numbered step for a missing verification cue."
+ ]
+ }
+ ],
+ "top_risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims."
+ ],
+ "output_constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability."
+ ],
+ "self_repair_checks": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards.",
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state.",
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ],
+ "reviewer_note": "Use this report before deepening the package and again before approving example outputs."
+ },
+ "artifact_design": {
+ "skill_name": "geo-content-brief-skill",
+ "design_system": "metric editorial",
+ "primary_artifact": {
+ "key": "report",
+ "label": "Report or brief",
+ "direction": "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.",
+ "matched_keywords": [
+ "report",
+ "brief",
+ "简报"
+ ]
+ },
+ "artifact_families": [
+ {
+ "key": "report",
+ "label": "Report or brief",
+ "score": 3,
+ "matched_keywords": [
+ "report",
+ "brief",
+ "简报"
+ ],
+ "direction": "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail."
+ },
+ {
+ "key": "review_viewer",
+ "label": "Review viewer",
+ "score": 2,
+ "matched_keywords": [
+ "review",
+ "审查"
+ ],
+ "direction": "Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix."
+ },
+ {
+ "key": "dashboard",
+ "label": "Dashboard or metrics page",
+ "score": 2,
+ "matched_keywords": [
+ "dashboard",
+ "table"
+ ],
+ "direction": "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation."
+ },
+ {
+ "key": "tutorial",
+ "label": "Tutorial or guide",
+ "score": 1,
+ "matched_keywords": [
+ "tutorial"
+ ],
+ "direction": "Progressive instructional layout with domain-specific section names, short success checks, and examples close to the user's real input."
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot or visual evidence",
+ "score": 1,
+ "matched_keywords": [
+ "screenshot"
+ ],
+ "direction": "Evidence-led visual artifact that records source, viewport, crop intent, and the exact region the reader should inspect."
+ }
+ ],
+ "layout_patterns": [
+ "thesis",
+ "evidence blocks",
+ "decision table",
+ "risks",
+ "next actions",
+ "summary",
+ "variant comparison",
+ "evidence"
+ ],
+ "design_tokens": {
+ "type": [
+ "Use a distinctive display face or serif for major claims when the artifact is editorial.",
+ "Use a restrained sans for dense body text and technical details.",
+ "Use mono only for metadata, paths, commands, labels, and evidence tags."
+ ],
+ "color": [
+ "Choose colors from the artifact's domain, brand, or evidence mood.",
+ "Do not default to Kami parchment, purple gradients, or generic SaaS blue unless the content justifies it.",
+ "Keep accent color limited to decisions, active states, risk, or section anchors."
+ ],
+ "spacing": [
+ "Prefer clear grid rhythm over floating decorative cards.",
+ "Increase whitespace around decisions and shrink whitespace around supporting metadata.",
+ "Split dense content instead of shrinking type or adding scroll traps."
+ ],
+ "components": [
+ "Use cards for grouped evidence, tables for comparisons, callouts for decisions, and timelines for sequence.",
+ "Avoid cards inside cards.",
+ "Keep reviewer-only detail visible but visually quieter than user-facing guidance."
+ ]
+ },
+ "quality_gates": [
+ "Keep the first screen useful without requiring the reader to parse every detail.",
+ "Use tables only for comparisons; move explanations below the table.",
+ "Keep source notes readable without flooding the body with markers.",
+ "Make differences visible instead of hiding them in prose.",
+ "Separate author-facing recommendations from reviewer-only evidence.",
+ "Surface conflicts clearly and keep routine benchmark synthesis quiet.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each."
+ ],
+ "anti_patterns": [
+ "Do not copy Kami's fixed parchment background as a default.",
+ "Do not use generic purple gradients, glass cards, or stock SaaS hero sections unless the content calls for them.",
+ "Do not let Markdown tables become the default shape for every comparison or explanation.",
+ "Do not turn reviewer evidence into user-facing clutter.",
+ "Do not invent screenshots, citations, charts, or UI states."
+ ],
+ "reviewer_note": "Use this profile to judge whether the generated artifacts feel designed for their job, not merely rendered."
+ },
+ "prompt_quality": {
+ "skill_name": "geo-content-brief-skill",
+ "relevance": "prompt-heavy",
+ "primary_task_family": {
+ "key": "creative_generation",
+ "label": "Creative generation",
+ "matched_keywords": [
+ "copy",
+ "content",
+ "内容"
+ ]
+ },
+ "task_families": [
+ {
+ "key": "creative_generation",
+ "label": "Creative generation",
+ "score": 3,
+ "matched_keywords": [
+ "copy",
+ "content",
+ "内容"
+ ],
+ "role_guidance": "Use a taste-aware creator role with clear audience, tone, and originality boundaries.",
+ "task_guidance": "Generate variants, explain selection logic, and preserve the user's distinctive constraints.",
+ "format_guidance": "Return options with rationale, selection criteria, and refinement paths."
+ },
+ {
+ "key": "execution_operation",
+ "label": "Execution operation",
+ "score": 3,
+ "matched_keywords": [
+ "workflow",
+ "execute",
+ "执行"
+ ],
+ "role_guidance": "Use an operator role with explicit boundaries, inputs, outputs, and failure handling.",
+ "task_guidance": "Convert the job into ordered steps with validation checks and stop conditions.",
+ "format_guidance": "Return a runbook-like handoff with commands, checks, owners, and next actions when relevant."
+ },
+ {
+ "key": "prompt_engineering",
+ "label": "Prompt engineering",
+ "score": 3,
+ "matched_keywords": [
+ "prompt",
+ "role",
+ "format"
+ ],
+ "role_guidance": "Use a prompt engineer role only when role design materially improves execution.",
+ "task_guidance": "Map Role, Task, and Format into skill behavior rather than copying a large prompt template.",
+ "format_guidance": "Return a compact prompt contract plus tests, quality matrix, and usage notes."
+ },
+ {
+ "key": "dialogue_interaction",
+ "label": "Dialogue interaction",
+ "score": 2,
+ "matched_keywords": [
+ "dialogue",
+ "访谈"
+ ],
+ "role_guidance": "Use a conversational role that asks only high-leverage questions and remembers the user's goal.",
+ "task_guidance": "Clarify intent, resolve uncertainty, and converge toward a recommendation instead of a long option list.",
+ "format_guidance": "Return concise prompts, decision points, and reviewer-visible assumptions."
+ }
+ ],
+ "complexity": {
+ "band": "complex",
+ "score": 6,
+ "reason": "multiple inputs, constraints, or task families require tradeoff handling"
+ },
+ "need_model": {
+ "explicit_need": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "implicit_need": "The reusable skill needs a stable role, task, and output contract rather than a one-off prompt.",
+ "scenario": "not yet explicit",
+ "user_level": "infer from examples and standards; ask only if it changes output depth",
+ "success_standard": "usable output with clear validation cues"
+ },
+ "rtf_to_skill": {
+ "role": "Use a taste-aware creator role with clear audience, tone, and originality boundaries.",
+ "task": "Generate variants, explain selection logic, and preserve the user's distinctive constraints.",
+ "format": "Return options with rationale, selection criteria, and refinement paths."
+ },
+ "quality_matrix": [
+ {
+ "key": "completeness",
+ "label": "Completeness",
+ "score": 80,
+ "matched_signals": [
+ "input",
+ "output",
+ "constraint",
+ "example",
+ "约束"
+ ],
+ "repair": "Name missing inputs, outputs, constraints, or success standards before deepening the package."
+ },
+ {
+ "key": "clarity",
+ "label": "Clarity",
+ "score": 90,
+ "matched_signals": [
+ "clear",
+ "specific"
+ ],
+ "repair": "Replace broad verbs with observable actions and define what done means."
+ },
+ {
+ "key": "consistency",
+ "label": "Consistency",
+ "score": 90,
+ "matched_signals": [
+ "boundary",
+ "边界"
+ ],
+ "repair": "Check that role, task, format, exclusions, and examples do not contradict each other."
+ },
+ {
+ "key": "practicality",
+ "label": "Practicality",
+ "score": 95,
+ "matched_signals": [
+ "execute",
+ "use",
+ "workflow",
+ "执行"
+ ],
+ "repair": "Add runnable steps, examples, or verification cues instead of abstract advice."
+ },
+ {
+ "key": "specificity",
+ "label": "Specificity",
+ "score": 70,
+ "matched_signals": [],
+ "repair": "Anchor wording in the user's audience, domain nouns, and target outcome."
+ }
+ ],
+ "overall_quality_score": 85.0,
+ "self_repair_checks": [
+ "Check explicit need, implicit need, scenario, user level, and success standard before deepening.",
+ "Map Role, Task, and Format into skill behavior, not decorative prompt labels.",
+ "Ask one focused clarification only when missing information changes the package boundary.",
+ "Add tests or examples for prompt-heavy behavior before treating it as reusable.",
+ "Keep prompt methodology in references and reports instead of bloating SKILL.md."
+ ],
+ "reviewer_note": "Use this profile when the package depends on prompt behavior, role design, output contracts, or conversation quality."
+ },
+ "evidence_readiness": {
+ "score": 57,
+ "checks": [
+ {
+ "label": "Intent clarity",
+ "status": "needs review",
+ "detail": "10/100 intent confidence."
+ },
+ {
+ "label": "Benchmark coverage",
+ "status": "needs evidence",
+ "detail": "0 GitHub benchmark repositories attached."
+ },
+ {
+ "label": "Pattern gate",
+ "status": "ready",
+ "detail": "3 accepted, 0 deferred using threshold 2/4."
+ },
+ {
+ "label": "Conflict handling",
+ "status": "decision needed",
+ "detail": "The stated preference leans lightweight or speed-first, while the benchmark mix leans toward governance, review, or heavier evaluation structure."
+ },
+ {
+ "label": "Output risk profile",
+ "status": "ready",
+ "detail": "5 output risk families attached."
+ },
+ {
+ "label": "Artifact design profile",
+ "status": "ready",
+ "detail": "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail."
+ },
+ {
+ "label": "Prompt quality profile",
+ "status": "ready",
+ "detail": "85.0/100 prompt-facing quality score."
+ }
+ ],
+ "reviewer_note": "Use this section to decide whether the package is ready to deepen or should stay in discovery."
+ }
+}
\ No newline at end of file
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.json
new file mode 100644
index 00000000..36bd1473
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.json
@@ -0,0 +1,24 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "skill_dir": ".previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill",
+ "source": "reports/review_annotations_input.json",
+ "summary": {
+ "annotation_count": 0,
+ "open_count": 0,
+ "resolved_count": 0,
+ "deferred_count": 0,
+ "open_blocker_count": 0,
+ "open_warning_count": 0,
+ "linked_gate_count": 0,
+ "target_missing_count": 0,
+ "failure_count": 0
+ },
+ "annotations": [],
+ "failures": [],
+ "template_written": false,
+ "artifacts": {
+ "json": "reports/review_annotations.json",
+ "markdown": "reports/review_annotations.md"
+ }
+}
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.md b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.md
new file mode 100644
index 00000000..68230eb6
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_annotations.md
@@ -0,0 +1,19 @@
+# Review Annotations
+
+This report renders reviewer annotations attached to Review Studio gates and source/report paths.
+
+- Annotations: `0`
+- Open: `0`
+- Resolved: `0`
+- Deferred: `0`
+- Open blockers: `0`
+- Open warnings: `0`
+
+No reviewer annotations recorded yet.
+
+
+## Review Rule
+
+- Use annotations for reviewer comments tied to a gate or source line.
+- Use waivers only for explicit acceptance of warning-level release risk.
+- Open blocker annotations should block a release decision until resolved or deferred with rationale.
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.json
new file mode 100644
index 00000000..bce91bb1
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.json
@@ -0,0 +1,44 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "skill_dir": ".previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill",
+ "generated_at": "2026-06-15",
+ "summary": {
+ "waiver_count": 0,
+ "active_count": 0,
+ "expired_count": 0,
+ "invalid_count": 0,
+ "covered_gate_count": 0,
+ "covered_gate_keys": [],
+ "waiver_candidate_count": 0,
+ "waiverable_open_count": 0,
+ "non_waivable_count": 0
+ },
+ "policy": {
+ "blocker_waivers_allowed": false,
+ "minimum_reason_chars": 20,
+ "expires_required": true,
+ "known_gate_keys": [
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ]
+ },
+ "waivers": [],
+ "waiver_candidates": [],
+ "failures": [],
+ "warnings": [],
+ "artifacts": {
+ "json": ".previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.json",
+ "markdown": ".previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.md"
+ }
+}
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.md b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.md
new file mode 100644
index 00000000..0e34c96f
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/review_waivers.md
@@ -0,0 +1,34 @@
+# Review Waivers
+
+- OK: `True`
+- Waivers: `0`
+- Active: `0`
+- Expired: `0`
+- Invalid: `0`
+- Covered gates: `none`
+- Waiver candidates: `0`
+- Open waiverable candidates: `0`
+- Non-waivable boundaries: `0`
+
+## Policy
+
+- Blocker waivers allowed: `False`
+- Minimum reason chars: `20`
+- Expiry is required for every waiver.
+- World-class evidence completion cannot be waived; it can only be proven by accepted ledger evidence.
+
+## Waivers
+
+- None
+
+## Candidate Actions
+
+- None
+
+## Failures
+
+- None
+
+## Warnings
+
+- None
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-ir.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-ir.json
new file mode 100644
index 00000000..771b377f
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-ir.json
@@ -0,0 +1,93 @@
+{
+ "schema_version": "2.0.0",
+ "name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "job_to_be_done": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "trigger_surface": {
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "should_trigger": [
+ "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ ],
+ "should_not_trigger": [
+ "one-off adjacent requests that do not match the recurring job",
+ "private local material that was not intentionally included"
+ ],
+ "edge_cases": []
+ },
+ "workflow": {
+ "steps": [
+ "Understand the request.",
+ "Execute the main task.",
+ "Validate the result."
+ ],
+ "decision_points": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ],
+ "failure_modes": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Users start invoking the skill for adjacent one-off or explanation-only requests.",
+ "Outputs remain valid but become generic, cluttered, or weakly aligned with the user's domain.",
+ "Borrowed benchmark patterns no longer fit the local job or add ceremony without payoff.",
+ "Skill usage becomes team-critical while ownership, review cadence, or rollback evidence stays informal."
+ ]
+ },
+ "resources": {
+ "references": [],
+ "scripts": [],
+ "assets": [],
+ "reports": [
+ "reports/intent-context.json",
+ "reports/intent-confidence.json",
+ "reports/intent-confidence.md",
+ "reports/reference-synthesis.json",
+ "reports/reference-synthesis.md",
+ "reports/output-risk-profile.json",
+ "reports/output-risk-profile.md",
+ "reports/artifact-design-profile.json",
+ "reports/artifact-design-profile.md",
+ "reports/prompt-quality-profile.json",
+ "reports/prompt-quality-profile.md",
+ "reports/system-model.json",
+ "reports/system-model.md"
+ ]
+ },
+ "eval_plan": {
+ "trigger": [],
+ "output": [],
+ "adversarial": [],
+ "baseline": "without_skill"
+ },
+ "risk": {
+ "output_risk": "high",
+ "execution_risk": "low",
+ "trust_boundary": "personal"
+ },
+ "governance": {
+ "owner": "Yao Team",
+ "maturity": "scaffold",
+ "review_cadence": "per-release",
+ "review_due": ""
+ },
+ "targets": [
+ "openai",
+ "claude",
+ "generic",
+ "agent-skills-compatible",
+ "vscode"
+ ],
+ "source_files": [
+ "SKILL.md",
+ "manifest.json",
+ "agents/interface.yaml",
+ "reports/intent-context.json",
+ "reports/output-risk-profile.json",
+ "reports/system-model.json"
+ ]
+}
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.html b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.html
new file mode 100644
index 00000000..45f5c88a
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.html
@@ -0,0 +1,986 @@
+
+
+
+
+
+ geo-content-brief-skill Skill 生成审计报告
+
+
+
+ 跳到正文Skip to content
+
+
+
+
+ Skill audit
+
+
+
+
+
+
+
+
+
+
YAO Skill 生成审计报告YAO Skill Generation Audit
+
geo-content-brief-skill
+
技能审计报告Geo Content Brief Skill Audit Report
+
这份报告默认使用中文简体，把新 Skill 的定位、指标、原理、契约、质量、风险、资产和迭代路线整理为一份可审计的 HTML 报告。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
完整度Completeness92SKILL.md 已存在，是 Skill 的入口。SKILL.md exists and acts as the skill entrypoint.触发清晰Trigger clarity100frontmatter description 已存在，具备基础路由面。The frontmatter description exists, giving the skill a basic routing surface.证据充分Evidence depth60已生成 12 / 20 类报告证据。Generated 12 / 20 evidence report types.上下文成本Context cost78入口约 864 个词/字，references 约 565 个词/字。Entrypoint is about 864 words/characters; references are about 565.
+
+
+
+
+
+
技能概述Overview
+
先用一屏说明这个 Skill 是什么、给谁用、交付什么。A first-screen explanation of what this skill is, who it serves, and what it delivers.
+
+
+
+
作用定位Role
+
将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。Turn one-off experience into a reusable, evaluable, and portable skill package.
+
+ 交付流程把用户输入、生成的包体和可复用能力放在一条线上。The delivery flow places user input, generated package, and reusable capability on one path.
+
+
+
+
+
+
+
+
总览指标Metrics
+
分数来自本地文件和 reports 证据，缺失时明确标为证据不足。Scores are derived from package files and reports; missing inputs are shown as evidence gaps.
+
+
+
指标判读Reading
+
先看雷达图判断能力短板，再看每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.
触发强度：手动触发 + description 路由Trigger strength: Manual activation plus description-based routing
复用范围：跨平台Reuse scope: Cross-platform
+
+
+
+
+
+
+
+
+
原理结构Principle
+
说明入口、参考、脚本、评估和报告如何组成一个稳定闭环。Explains how entrypoint, references, scripts, evals, and reports form a stable loop.
+
+
+
分层结构展示入口、参考、脚本、评估和报告如何各司其职。The layered structure shows how entrypoint, references, scripts, evals, and reports each carry a distinct role.
+
+
+
执行流程Execution Flow
+
读取用户输入，识别目标受众、内容渠道、关键词、品牌边界、禁止表达和必须覆盖的信息。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
对照 `references/brief-structure.md` 生成简报骨架：目标、洞察、角度、关键词、结构、证据、风险和验收标准。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
如果缺少核心输入，先提出最多三个澄清问题，不用假设补全用户没有提供的事实。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
用 `evals/trigger_cases.jsonl` 检查当前请求是否属于 GEO 内容简报场景。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
用 `evals/output_cases.jsonl` 检查输出是否包含输入摘要、内容结构、证据要求、禁区和下一步动作。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
+
+
+
调用方式How To Use
+
当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
Use this skill when the request matches: 将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。Use this skill when the request matches the frontmatter description.
+
+
+
+
+
+
+
+
+
+
契约边界Contract
+
把触发、输入、输出和排除场景放在同一屏。Keeps trigger, inputs, outputs, and exclusions on the same screen.
+
+
+
+
触发描述Trigger
+
将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。Skill-specific source text is authored in Chinese; switch to Simplified Chinese for the exact wording.
+
输入材料Inputs
+
用户提供的工作流、提示词、文档、记录或散乱笔记User-provided workflows, prompts, documents, records, or rough notes.
期望沉淀的复用场景、排除项、约束和质量标准The reusable scenario, exclusions, constraints, and quality standards to capture.
只需要一次性回答、没有复用价值的临时请求。One-off requests that do not need reusable skill behavior.
要求直接执行相邻任务，而不是沉淀或使用这个 Skill。Requests to perform an adjacent task directly rather than create or use this skill.
缺少必要事实且用户不允许澄清的场景。Cases that lack required facts and do not allow clarification.
+
+
+
+
+
+
+
+
+
质量评估Quality
+
展示强项、缺口和建议，避免只给分不解释。Shows strengths, gaps, and recommendations instead of scores without explanation.
+
+
+
+
类型Type
证据Evidence
建议Action
+
强项Strength
触发面保持精简，并锚定在 frontmatter description。The trigger surface stays lean and anchored in the frontmatter description.
保留并复用Keep
强项Strength
已生成 Skill IR，核心语义可先于平台打包被审查和迁移。Skill IR is generated so core semantics can be reviewed and migrated before platform packaging.
保留并复用Keep
强项Strength
已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。Target compilation evidence is generated to review how IR maps to OpenAI, Claude, generic, and other target contracts.
先补证据和边界，再增加包体复杂度。Improve evidence and boundaries before adding package complexity.
+
+
+
+
世界证据World Evidence
未生成 world-class ledger；当前报告不会宣称世界级完成。No world-class ledger was generated; this report does not claim world-class completion.
证据待补Evidence pending
待补证据Pending0仍需外部或人工证据接受。External or human evidence still needs acceptance.已接受Accepted0已通过 source check 与提交契约。Passed source checks and submission contract.源检查Source Checks0 / 0通过数 / 总检查数。Passed checks / total checks.
尚未生成 world-class ledger；这里只保留反过度承诺提示。No world-class ledger has been generated; this panel keeps the anti-overclaim guard visible.
+
+
+
+
+
+
+
包体资产Assets
+
让 reviewer 快速确认关键文件、目录和资产分布。Lets reviewers confirm key files, directories, and asset distribution quickly.
+
+
+ 资产分布图展示当前包体的文件和目录重心。The asset distribution chart shows where files and directories are concentrated.
+
+
路径Path
作用Role
类型Type
+
SKILL.md
Skill 入口文件Skill entrypoint
文件file
README.md
人类可读使用说明Human-readable usage guide
文件file
agents/interface.yaml
跨平台接口元数据Neutral interface metadata
文件file
manifest.json
生命周期与打包元数据Lifecycle and portability metadata
文件file
references
扩展指导与复用资料Extended guidance and reusable notes
目录folder
scripts
确定性脚本或本地工具Deterministic helpers or local tooling
目录folder
evals
触发与质量检查Trigger and quality checks
目录folder
reports
生成的证据与总结报告Generated evidence and overview artifacts
目录folder
+
+
+
+
+
+
+
+
+
迭代路线Roadmap
+
把下一步升级收束为少数高价值动作。Keeps next iteration moves focused and actionable.
+
+
+
迭代时间线把下一步升级收束成少数可执行动作。The iteration timeline narrows the next upgrade into a few executable moves.
+
下一步 1Next 1
收紧触发与排除边界Tighten trigger and exclusions
在继续扩展前，需要先把相邻但不应触发的场景说清楚。The package needs clearer near-neighbor exclusions before it grows.
增加 3 到 5 个应触发和不应触发的例子。Add 3 to 5 should-trigger and should-not-trigger examples.
精炼 frontmatter description，明确重复任务和非目标。Refine the frontmatter description to name the recurring job and non-goals.
扩展包体前先跑一轮触发评估。Run a first trigger evaluation pass before expanding the package.
路由更清晰，误触发更少。Cleaner routing and fewer accidental activations.
下一步 2Next 2
补上第一个执行资产Add the first execution asset
当前包体仍偏文本说明，应先增加一个能减少重复人工操作的资产。The package is still mostly prose. Add one asset that removes repeated manual work.
如果用户会反复使用某段流程说明，把它沉淀到 references。Move stable procedural guidance into references if users will need it repeatedly.
如果某个重复步骤可以执行而不是描述，就沉淀成一个确定性 helper script。Create one deterministic helper script if a repeated step can be executed instead of described.
保持主 SKILL.md 简洁，并围绕路由与入口组织。Keep the main SKILL.md compact and route-oriented.
在不膨胀入口文件的前提下提升执行质量。Stronger execution quality without bloating the entrypoint.
下一步 3Next 3
从脚手架推进到生产可用Promote from scaffold to production-ready
第一版已经存在，下一步收益通常来自补上最小但有效的质量门禁。The first version exists; the next gain usually comes from adding the smallest useful gates.
判断这个 Skill 是个人使用、团队复用，还是库级基础能力。Decide whether this skill is personal, team-reused, or library-grade.
只添加与风险等级匹配的质量门禁。Add only the gates that match that risk level.
一旦进入真实复用，就记录生命周期元数据和评审节奏。Record lifecycle metadata and review cadence once reuse becomes real.
更清晰地从探索性包体走向可维护资产。A clearer path from exploratory package to maintained asset.
+
+
+
+
+
+
+
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.json
new file mode 100644
index 00000000..968c3b5b
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/skill-overview.json
@@ -0,0 +1,1060 @@
+{
+ "skill_summary": {
+ "name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "display_name": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "maturity": "production",
+ "updated_at": "2026-06-15",
+ "core_value": "把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。",
+ "audience": "Skill 作者、复用团队和后续 reviewer。",
+ "deliverables": [
+ "SKILL.md",
+ "agents/interface.yaml",
+ "reports/skill-ir.json",
+ "reports/compiled_targets.md",
+ "reports/output_quality_scorecard.md",
+ "reports/conformance_matrix.md",
+ "reports/security_trust_report.md",
+ "reports/skill_atlas.html",
+ "reports/registry_audit.md",
+ "reports/package_verification.md",
+ "reports/install_simulation.md",
+ "reports/upgrade_check.md",
+ "reports/adoption_drift_report.md",
+ "reports/review_waivers.md",
+ "reports/review_annotations.md",
+ "reports/review-studio.html",
+ "reports/skill-interpretation.html",
+ "reports/skill-overview.html"
+ ],
+ "flow": [
+ "输入材料",
+ "Skill 包体",
+ "可复用能力"
+ ]
+ },
+ "scorecard": {
+ "completeness_score": {
+ "label": "完整度",
+ "score": 92,
+ "reasons": [
+ "SKILL.md 已存在，是 Skill 的入口。",
+ "README.md 已存在，便于人工阅读。",
+ "agents/interface.yaml 已存在，便于跨平台适配。",
+ "manifest.json 已存在，生命周期信息可追踪。",
+ "reports/ 已存在，生成证据可以随包体迁移。"
+ ]
+ },
+ "trigger_score": {
+ "label": "触发清晰",
+ "score": 100,
+ "reasons": [
+ "frontmatter description 已存在，具备基础路由面。",
+ "description 有足够长度说明任务边界。",
+ "description 已包含使用场景或排除边界信号。",
+ "evals/ 已存在，可承载触发样例或质量检查。",
+ "intent-confidence 报告已生成，可辅助判断触发稳定性。"
+ ]
+ },
+ "evidence_score": {
+ "label": "证据充分",
+ "score": 60,
+ "reasons": [
+ "已生成 12 / 20 类报告证据。",
+ "skill-ir.json 已存在。",
+ "compiled_targets.json 已存在。",
+ "intent-dialogue.json 已存在。",
+ "证据不足：缺少 output_quality_scorecard.json, conformance_matrix.json, security_trust_report.json。"
+ ]
+ },
+ "maintainability_score": {
+ "label": "可维护性",
+ "score": 85,
+ "reasons": [
+ "SKILL.md 约 864 个词/字。",
+ "入口文件保持克制，可维护性较好。",
+ "references/ 已承载扩展指导。",
+ "scripts/ 证据不足，重复执行逻辑可能仍靠人工。",
+ "evals/ 已承载可迁移检查。"
+ ]
+ },
+ "portability_score": {
+ "label": "可迁移性",
+ "score": 100,
+ "reasons": [
+ "agents/interface.yaml 已存在。",
+ "manifest.json 已存在。",
+ "目标平台或 adapter target 已声明。",
+ "入口文件未发现明显私有绝对路径。"
+ ]
+ },
+ "context_cost": {
+ "label": "上下文成本",
+ "score": 78,
+ "reasons": [
+ "入口约 864 个词/字，references 约 565 个词/字。",
+ "分数越高代表上下文成本越低。",
+ "上下文成本处于可控区间。"
+ ]
+ }
+ },
+ "capability_profile": {
+ "archetype": "production",
+ "task_family": "Creative generation",
+ "maturity": "production",
+ "trigger_strength": "手动触发 + description 路由",
+ "reuse_scope": "跨平台",
+ "matrix": {
+ "execution_certainty": 72,
+ "knowledge_density": 80
+ }
+ },
+ "principle_model": {
+ "nodes": [
+ {
+ "title": "意图澄清",
+ "body": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Ask one focused question and update intent context before adding assets."
+ }
+ ],
+ "layers": [
+ "入口层",
+ "参考层",
+ "脚本层",
+ "评估层",
+ "报告层"
+ ]
+ },
+ "contract_boundary": {
+ "trigger": {
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "inputs": [
+ "用户提供的工作流、提示词、文档、记录或散乱笔记",
+ "期望沉淀的复用场景、排除项、约束和质量标准"
+ ],
+ "outputs": [
+ "可路由的 SKILL.md",
+ "agents/interface.yaml 元数据",
+ "必要的 references、scripts、evals、reports 证据",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ],
+ "should_trigger": [
+ "团队复用前，需要明确触发边界、质量证据和维护责任。",
+ "用户说出类似需求时：当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "已有原始资料，但还没有清晰输入、输出和后续迭代路径。"
+ ],
+ "should_not_trigger": [
+ "只需要一次性回答、没有复用价值的临时请求。",
+ "要求直接执行相邻任务，而不是沉淀或使用这个 Skill。",
+ "缺少必要事实且用户不允许澄清的场景。"
+ ],
+ "boundary_cards": [
+ {
+ "label": "Owned",
+ "body": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。"
+ },
+ {
+ "label": "Adjacent",
+ "body": "相邻任务需要先确认是否应转为独立 Skill。"
+ },
+ {
+ "label": "Excluded",
+ "body": "不替代人工事实核查，也不静默扩大职责。"
+ }
+ ]
+ },
+ "quality_review": {
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Adoption Drift Report，可把本地使用反馈转为下一轮迭代信号。",
+ "已生成 Review Waivers 台账，可记录 reviewer 对 warning 风险的批准、理由和到期时间。",
+ "已生成 Review Annotations 台账，可把 reviewer 批注挂到 gate、文件和行号。"
+ ],
+ "gaps": [
+ "证据充分需要补强：已生成 12 / 20 类报告证据。"
+ ],
+ "recommendations": [
+ "先改触发边界，再扩展工作流。",
+ "只把重复且稳定的步骤沉淀为脚本。",
+ "每次升级后重新生成报告并检查分数原因。"
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.",
+ "Keep the first screen useful without requiring the reader to parse every detail.",
+ "Use tables only for comparisons; move explanations below the table.",
+ "Keep source notes readable without flooding the body with markers."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 85.0,
+ "highlights": [
+ "Primary prompt task family: Creative generation.",
+ "Complexity: complex — multiple inputs, constraints, or task families require tradeoff handling",
+ "Completeness: 80/100.",
+ "Clarity: 90/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 77,
+ "band": "stable-first-pass"
+ },
+ "highlights": [
+ "Stability: stable-first-pass (77/100).",
+ "Owned job: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "Leverage: Clarify the real job boundary — Ask one focused question and update intent context before adding assets.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ ]
+ }
+ },
+ "risk_governance": {
+ "risks": [
+ {
+ "name": "误触发风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "frontmatter description 已存在，具备基础路由面。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "输出漂移风险",
+ "impact": 2,
+ "probability": 2,
+ "signal": "已生成 12 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "证据不足风险",
+ "impact": 3,
+ "probability": 2,
+ "signal": "已生成 12 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "包体膨胀风险",
+ "impact": 2,
+ "probability": 2,
+ "signal": "SKILL.md 约 864 个词/字。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "跨平台迁移风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "agents/interface.yaml 已存在。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ }
+ ],
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "table",
+ "report",
+ "brief"
+ ],
+ "score": 4,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "screenshot",
+ "visual",
+ "screen"
+ ],
+ "score": 3,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "citation",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "copy",
+ "content"
+ ],
+ "score": 2,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ },
+ {
+ "key": "tutorial_quality",
+ "label": "Tutorial quality",
+ "matched_keywords": [
+ "tutorial"
+ ],
+ "score": 1,
+ "risks": [
+ "Generic section headings make the tutorial feel templated instead of fitted to the learner's task.",
+ "Steps may explain what to do without naming the exact check that proves the step worked.",
+ "Examples can become too abstract when the input material is concrete."
+ ],
+ "constraints": [
+ "Write headings from the user's domain nouns and desired outcome, not from generic labels like Overview or Key Points.",
+ "Pair each major step with a visible success check or expected intermediate output.",
+ "Use one concrete worked example before adding broad principles."
+ ],
+ "self_repair": [
+ "Replace generic H2/H3 headings with task-specific headings before final output.",
+ "Scan every numbered step for a missing verification cue."
+ ]
+ }
+ ],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ]
+ },
+ "world_class_readiness": {
+ "ready": false,
+ "decision": "not-generated",
+ "entry_count": 0,
+ "pending_count": 0,
+ "accepted_count": 0,
+ "external_pending_count": 0,
+ "human_pending_count": 0,
+ "source_check_count": 0,
+ "source_pass_count": 0,
+ "conclusion_zh": "未生成 world-class ledger；当前报告不会宣称世界级完成。",
+ "conclusion_en": "No world-class ledger was generated; this report does not claim world-class completion.",
+ "entries": []
+ },
+ "package_assets": {
+ "entries": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 0
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 34
+ }
+ ],
+ "file_count": 42,
+ "folder_count": 4,
+ "distribution": [
+ {
+ "label": "SKILL.md",
+ "value": 1
+ },
+ {
+ "label": "README.md",
+ "value": 1
+ },
+ {
+ "label": "agents/interface.yaml",
+ "value": 1
+ },
+ {
+ "label": "manifest.json",
+ "value": 1
+ },
+ {
+ "label": "references",
+ "value": 2
+ },
+ {
+ "label": "scripts",
+ "value": 1
+ },
+ {
+ "label": "evals",
+ "value": 2
+ },
+ {
+ "label": "reports",
+ "value": 34
+ }
+ ]
+ },
+ "iteration_roadmap": {
+ "items": [
+ {
+ "title": "Tighten trigger and exclusions",
+ "why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "actions": [
+ "Add 3 to 5 should-trigger and should-not-trigger examples.",
+ "Refine the frontmatter description to name the recurring job and non-goals.",
+ "Run a first trigger evaluation pass before expanding the package."
+ ],
+ "unlocks": "Cleaner routing and fewer accidental activations."
+ },
+ {
+ "title": "Add the first execution asset",
+ "why": "The package is still mostly prose. Add one asset that removes repeated manual work.",
+ "actions": [
+ "Move stable procedural guidance into references if users will need it repeatedly.",
+ "Create one deterministic helper script if a repeated step can be executed instead of described.",
+ "Keep the main SKILL.md compact and route-oriented."
+ ],
+ "unlocks": "Stronger execution quality without bloating the entrypoint."
+ },
+ {
+ "title": "Promote from scaffold to production-ready",
+ "why": "The first version exists; the next gain usually comes from adding the smallest useful gates.",
+ "actions": [
+ "Decide whether this skill is personal, team-reused, or library-grade.",
+ "Add only the gates that match that risk level.",
+ "Record lifecycle metadata and review cadence once reuse becomes real."
+ ],
+ "unlocks": "A clearer path from exploratory package to maintained asset."
+ }
+ ]
+ },
+ "report_contract": {
+ "schema_version": "2.0",
+ "html_report": "reports/skill-overview.html",
+ "language": "zh-CN",
+ "default_language": "zh-CN",
+ "languages": [
+ "zh-CN",
+ "en"
+ ],
+ "layout": "kami-white-audit-v2",
+ "nav_labels": [
+ "技能概述",
+ "总览指标",
+ "能力画像",
+ "原理结构",
+ "契约边界",
+ "质量评估",
+ "风险治理",
+ "包体资产",
+ "迭代路线"
+ ],
+ "nav_labels_en": [
+ "Overview",
+ "Metrics",
+ "Profile",
+ "Principle",
+ "Contract",
+ "Quality",
+ "Risk",
+ "Assets",
+ "Roadmap"
+ ]
+ },
+ "name": "geo-content-brief-skill",
+ "title": "Geo Content Brief Skill",
+ "display_name": "Geo Content Brief Skill",
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "logic_steps": [
+ "读取用户输入，识别目标受众、内容渠道、关键词、品牌边界、禁止表达和必须覆盖的信息。",
+ "对照 `references/brief-structure.md` 生成简报骨架：目标、洞察、角度、关键词、结构、证据、风险和验收标准。",
+ "如果缺少核心输入，先提出最多三个澄清问题，不用假设补全用户没有提供的事实。",
+ "用 `evals/trigger_cases.jsonl` 检查当前请求是否属于 GEO 内容简报场景。",
+ "用 `evals/output_cases.jsonl` 检查输出是否包含输入摘要、内容结构、证据要求、禁区和下一步动作。"
+ ],
+ "usage_steps": [
+ "当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "Use this skill when the request matches: 将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。"
+ ],
+ "package_map": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 0
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 2
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 34
+ }
+ ],
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Adoption Drift Report，可把本地使用反馈转为下一轮迭代信号。",
+ "已生成 Review Waivers 台账，可记录 reviewer 对 warning 风险的批准、理由和到期时间。",
+ "已生成 Review Annotations 台账，可把 reviewer 批注挂到 gate、文件和行号。"
+ ],
+ "scenarios": [
+ "团队复用前，需要明确触发边界、质量证据和维护责任。",
+ "用户说出类似需求时：当你需要把访谈纪要、关键词、竞品线索和渠道限制整理成中文 GEO 内容简报时，使用 $geo-content-brief-skill。",
+ "已有原始资料，但还没有清晰输入、输出和后续迭代路径。"
+ ],
+ "trigger_contract": {
+ "description": "将 GEO 内容访谈、关键词笔记、竞品线索和渠道约束整理成可执行的中文内容简报。用于团队复用、选题规划、内容生产交接和质量评审；不用于直接代写完整长文或替代品牌策略判断。",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "io_contract": {
+ "inputs": [
+ "用户提供的工作流、提示词、文档、记录或散乱笔记",
+ "期望沉淀的复用场景、排除项、约束和质量标准"
+ ],
+ "outputs": [
+ "可路由的 SKILL.md",
+ "agents/interface.yaml 元数据",
+ "必要的 references、scripts、evals、reports 证据",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ]
+ },
+ "principles": [
+ {
+ "title": "意图澄清",
+ "body": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。"
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Ask one focused question and update intent context before adding assets."
+ }
+ ],
+ "roadmap": [
+ {
+ "title": "Tighten trigger and exclusions",
+ "why": "The package needs clearer near-neighbor exclusions before it grows.",
+ "actions": [
+ "Add 3 to 5 should-trigger and should-not-trigger examples.",
+ "Refine the frontmatter description to name the recurring job and non-goals.",
+ "Run a first trigger evaluation pass before expanding the package."
+ ],
+ "unlocks": "Cleaner routing and fewer accidental activations."
+ },
+ {
+ "title": "Add the first execution asset",
+ "why": "The package is still mostly prose. Add one asset that removes repeated manual work.",
+ "actions": [
+ "Move stable procedural guidance into references if users will need it repeatedly.",
+ "Create one deterministic helper script if a repeated step can be executed instead of described.",
+ "Keep the main SKILL.md compact and route-oriented."
+ ],
+ "unlocks": "Stronger execution quality without bloating the entrypoint."
+ },
+ {
+ "title": "Promote from scaffold to production-ready",
+ "why": "The first version exists; the next gain usually comes from adding the smallest useful gates.",
+ "actions": [
+ "Decide whether this skill is personal, team-reused, or library-grade.",
+ "Add only the gates that match that risk level.",
+ "Record lifecycle metadata and review cadence once reuse becomes real."
+ ],
+ "unlocks": "A clearer path from exploratory package to maintained asset."
+ }
+ ],
+ "cards": [],
+ "introduction": [
+ "这份报告用于快速理解新生成 Skill 的定位、原理、触发边界和交付内容。",
+ "先确认重复任务、真实输入形态和可交付输出，再决定是否继续加 references、scripts 或 evals。",
+ "如果需求仍然模糊，优先回到 intent dialogue 收紧边界，再扩展包体结构。"
+ ],
+ "benchmark_highlights": [],
+ "skill_ir": {
+ "schema_version": "2.0.0",
+ "source_path": "reports/skill-ir.json",
+ "target_count": 5,
+ "trigger_samples": 1,
+ "output_eval_cases": 0
+ },
+ "compiled_targets": {
+ "ok": true,
+ "schema_version": "1.0",
+ "summary": {
+ "target_count": 5,
+ "pass_count": 5,
+ "warn_count": 0,
+ "block_count": 0,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "targets": [
+ {
+ "target": "openai",
+ "status": "pass",
+ "adapter_mode": "metadata-adapter",
+ "degradation_strategy": "metadata-adapter",
+ "native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
+ "permission_enforcement": "metadata-only",
+ "generated_files": [
+ "targets/openai/adapter.json",
+ "targets/openai/agents/openai.yaml"
+ ],
+ "unsupported_features": [
+ "client-native script permission prompts are represented as permission contract metadata"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "claude",
+ "status": "pass",
+ "adapter_mode": "neutral-source-plus-adapter",
+ "degradation_strategy": "neutral-source-plus-adapter",
+ "native_surface": "Claude-compatible neutral source folder with adapter notes",
+ "permission_enforcement": "metadata-fallback",
+ "generated_files": [
+ "targets/claude/adapter.json",
+ "targets/claude/README.md"
+ ],
+ "unsupported_features": [
+ "vendor-native metadata fields are carried as adapter JSON and README notes"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "generic",
+ "status": "pass",
+ "adapter_mode": "agent-skills-compatible",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills compatible neutral package",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "targets/generic/adapter.json"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "agent-skills-compatible",
+ "status": "pass",
+ "adapter_mode": "neutral-agent-skills-source",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills standard source tree",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "SKILL.md",
+ "agents/interface.yaml"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "permission_enforcement": "client-or-workspace-trust",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "unsupported_features": [
+ "VS Code installation scope is documented but not installed by this compiler"
+ ],
+ "warnings": []
+ }
+ ],
+ "failures": [],
+ "warnings": []
+ },
+ "output_quality": {},
+ "output_execution": {
+ "ok": false,
+ "summary": {},
+ "runner": {},
+ "failures": []
+ },
+ "output_blind_review": {
+ "summary": {},
+ "seed": "",
+ "pair_count": 0,
+ "answer_key_separate": false
+ },
+ "output_review_kit": {
+ "ok": false,
+ "summary": {},
+ "artifacts": {},
+ "failures": []
+ },
+ "output_review_adjudication": {
+ "ok": false,
+ "summary": {},
+ "reviewer": "",
+ "reviewed_at": "",
+ "failures": []
+ },
+ "benchmark_reproducibility": {
+ "ok": false,
+ "summary": {},
+ "commit": "",
+ "missing_artifacts": [],
+ "limitations": []
+ },
+ "runtime_conformance": {},
+ "runtime_permissions": {
+ "ok": false,
+ "summary": {},
+ "expected_capabilities": [],
+ "targets": [],
+ "failures": []
+ },
+ "trust_security": {},
+ "skill_atlas": {},
+ "registry_distribution": {
+ "ok": false,
+ "package": {},
+ "failures": [],
+ "warnings": []
+ },
+ "package_verification": {
+ "ok": false,
+ "summary": {},
+ "failures": [],
+ "warnings": []
+ },
+ "install_simulation": {
+ "ok": false,
+ "summary": {},
+ "failures": [],
+ "warnings": []
+ },
+ "upgrade_check": {
+ "ok": false,
+ "summary": {},
+ "upgrade_diff": {},
+ "release_notes": [],
+ "failures": [],
+ "warnings": []
+ },
+ "adoption_drift": {
+ "ok": true,
+ "summary": {
+ "event_count": 0,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "edited_count": 0,
+ "rejected_count": 0,
+ "missed_count": 0,
+ "failed_count": 0,
+ "adoption_rate": 0,
+ "missed_trigger_count": 0,
+ "wrong_trigger_count": 0,
+ "bad_output_count": 0,
+ "script_error_count": 0,
+ "missing_resource_count": 0,
+ "review_overdue_count": 0,
+ "risk_band": "no-data",
+ "event_types": {},
+ "failure_types": {},
+ "source_types": {},
+ "command_counts": {}
+ },
+ "next_iteration_candidates": [
+ {
+ "signal": "no telemetry",
+ "recommendation": "Start with a small metadata-only sample before using telemetry for release decisions."
+ }
+ ],
+ "privacy_contract": {
+ "storage": "local-first",
+ "event_scope": "metadata-only",
+ "raw_content_allowed": false,
+ "raw_event_log_packaged": false,
+ "blocked_fields": [
+ "content",
+ "input",
+ "inputs",
+ "message",
+ "messages",
+ "note",
+ "output",
+ "outputs",
+ "prompt",
+ "raw",
+ "text",
+ "transcript"
+ ]
+ },
+ "failures": []
+ },
+ "review_waivers": {
+ "ok": true,
+ "summary": {
+ "waiver_count": 0,
+ "active_count": 0,
+ "expired_count": 0,
+ "invalid_count": 0,
+ "covered_gate_count": 0,
+ "covered_gate_keys": [],
+ "waiver_candidate_count": 0,
+ "waiverable_open_count": 0,
+ "non_waivable_count": 0
+ },
+ "policy": {
+ "blocker_waivers_allowed": false,
+ "minimum_reason_chars": 20,
+ "expires_required": true,
+ "known_gate_keys": [
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ]
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "review_annotations": {
+ "ok": true,
+ "summary": {
+ "annotation_count": 0,
+ "open_count": 0,
+ "resolved_count": 0,
+ "deferred_count": 0,
+ "open_blocker_count": 0,
+ "open_warning_count": 0,
+ "linked_gate_count": 0,
+ "target_missing_count": 0,
+ "failure_count": 0
+ },
+ "annotations": [],
+ "failures": []
+ },
+ "world_class_evidence_plan": {
+ "ok": false,
+ "summary": {},
+ "tasks": [],
+ "source_audit": {}
+ },
+ "world_class_evidence_ledger": {
+ "ok": false,
+ "summary": {},
+ "entries": [],
+ "source_plan": {}
+ },
+ "synthesis_highlights": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "Borrow the discipline of defining what the skill should not own before growing the package."
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.",
+ "Keep the first screen useful without requiring the reader to parse every detail.",
+ "Use tables only for comparisons; move explanations below the table.",
+ "Keep source notes readable without flooding the body with markers."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 85.0,
+ "highlights": [
+ "Primary prompt task family: Creative generation.",
+ "Complexity: complex — multiple inputs, constraints, or task families require tradeoff handling",
+ "Completeness: 80/100.",
+ "Clarity: 90/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 77,
+ "band": "stable-first-pass"
+ },
+ "highlights": [
+ "Stability: stable-first-pass (77/100).",
+ "Owned job: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "Leverage: Clarify the real job boundary — Ask one focused question and update intent context before adding assets.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ ]
+ },
+ "metadata": {
+ "canonical_format": "agent-skills",
+ "targets": [
+ "openai",
+ "claude",
+ "generic",
+ "vscode"
+ ],
+ "maturity_tier": "production",
+ "skill_archetype": "production",
+ "updated_at": "2026-06-15"
+ }
+}
\ No newline at end of file
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.json b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.json
new file mode 100644
index 00000000..5a2417a8
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.json
@@ -0,0 +1,166 @@
+{
+ "skill_name": "geo-content-brief-skill",
+ "description": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "systems_doctrine": "Structure drives behavior: improve the boundary, feedback loops, drift watch, and leverage points before adding weight.",
+ "stability": {
+ "score": 77,
+ "band": "stable-first-pass"
+ },
+ "boundary_map": {
+ "owned_job": "将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。",
+ "input_boundary": [
+ "user-provided workflow notes, prompts, docs, or examples"
+ ],
+ "output_boundary": "a reusable skill output",
+ "non_goals": [
+ "one-off adjacent requests that do not match the recurring job",
+ "private local material that was not intentionally included"
+ ],
+ "constraints": [],
+ "standards": [],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ],
+ "maturity_assumption": "scaffold"
+ },
+ "feedback_loops": [
+ {
+ "name": "Intent boundary loop",
+ "signal": "Intent confidence score is 10/100.",
+ "response": "Ask only the highest-leverage clarification before adding package weight.",
+ "evidence": "reports/intent-confidence.md and reports/intent-dialogue.md"
+ },
+ {
+ "name": "Reference synthesis loop",
+ "signal": "Benchmark patterns are useful only after they are abstracted into borrow and avoid guidance.",
+ "response": "Borrow one pattern at a time and keep the rest as reviewer-visible evidence.",
+ "evidence": "reports/reference-synthesis.md",
+ "current_patterns": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.",
+ "Borrow the discipline of defining what the skill should not own before growing the package.",
+ "Do not let packaging or platform concerns swallow the core job boundary.",
+ "Do not create experimental overhead that exceeds the skill's real risk tier."
+ ]
+ },
+ {
+ "name": "Output quality loop",
+ "signal": "Generated output may fail in recurring domain-specific ways.",
+ "response": "Apply predicted output-risk families as self-repair checks before final output.",
+ "evidence": "reports/output-risk-profile.md",
+ "current_risk_families": [
+ "Markdown readability",
+ "Screenshot and visual capture",
+ "Citation and footnote clutter",
+ "Tone and specificity",
+ "Tutorial quality"
+ ]
+ },
+ {
+ "name": "Reviewer feedback loop",
+ "signal": "Human review catches drift that static checks miss.",
+ "response": "Capture lightweight feedback and turn repeated findings into gates or references.",
+ "evidence": "reports/review-viewer.html and feedback records"
+ },
+ {
+ "name": "Lifecycle loop",
+ "signal": "As reuse grows, the skill needs stronger gates, ownership, and regression evidence.",
+ "response": "Promote only when the next gate improves reliability more than context cost.",
+ "evidence": "manifest.json, reports/iteration-directions.md, and governance checks"
+ }
+ ],
+ "drift_watch": [
+ {
+ "name": "Trigger drift",
+ "watch_signal": "Users start invoking the skill for adjacent one-off or explanation-only requests.",
+ "countermeasure": "Add near-neighbor exclusions and route evals before expanding workflow steps.",
+ "cadence": "per trigger or description change"
+ },
+ {
+ "name": "Output drift",
+ "watch_signal": "Outputs remain valid but become generic, cluttered, or weakly aligned with the user's domain.",
+ "countermeasure": "Refresh output-risk and artifact-design profiles, then add one self-repair check.",
+ "cadence": "after the first 3-5 real uses",
+ "risk_families": [
+ "Markdown readability",
+ "Screenshot and visual capture",
+ "Citation and footnote clutter",
+ "Tone and specificity",
+ "Tutorial quality"
+ ]
+ },
+ {
+ "name": "Reference drift",
+ "watch_signal": "Borrowed benchmark patterns no longer fit the local job or add ceremony without payoff.",
+ "countermeasure": "Re-run reference synthesis and keep only patterns that improve the current boundary.",
+ "cadence": "per material benchmark or product assumption change"
+ },
+ {
+ "name": "Governance drift",
+ "watch_signal": "Skill usage becomes team-critical while ownership, review cadence, or rollback evidence stays informal.",
+ "countermeasure": "Promote maturity tier and add reviewer-visible lifecycle evidence.",
+ "cadence": "when reuse becomes real"
+ }
+ ],
+ "failure_pattern_map": [
+ {
+ "family": "Boundary failure",
+ "symptom": "The skill handles nearby requests that were never part of the recurring job.",
+ "repair": "Narrow the description and add explicit non-goals before adding more execution steps."
+ },
+ {
+ "family": "Feedback gap",
+ "symptom": "The skill has rules but no signal telling authors which rule should change after use.",
+ "repair": "Turn repeated reviewer feedback into one eval, one reference note, or one self-repair check."
+ },
+ {
+ "family": "Output degradation",
+ "symptom": "The result is structurally correct but generic, cluttered, or weakly matched to the user's domain.",
+ "repair": "Use output-risk families as pre-final checks.",
+ "current_risk_families": [
+ "Markdown readability",
+ "Screenshot and visual capture",
+ "Citation and footnote clutter",
+ "Tone and specificity",
+ "Tutorial quality"
+ ]
+ },
+ {
+ "family": "Prompt-behavior mismatch",
+ "symptom": "The role, task, and format are copied from a prompt instead of becoming stable skill behavior.",
+ "repair": "Convert reusable role/task/format assumptions into workflow, reports, or references.",
+ "watch_axes": [
+ "Specificity"
+ ]
+ }
+ ],
+ "leverage_points": [
+ {
+ "rank": 1,
+ "point": "Clarify the real job boundary",
+ "why": "Intent uncertainty creates downstream trigger, output, and governance errors.",
+ "move": "Ask one focused question and update intent context before adding assets."
+ },
+ {
+ "rank": 2,
+ "point": "Tune the frontmatter description",
+ "why": "The description is the highest-leverage routing surface.",
+ "move": "Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ },
+ {
+ "rank": 3,
+ "point": "Install output self-repair checks",
+ "why": "The likely failure families are: Markdown readability, Screenshot and visual capture, Citation and footnote clutter.",
+ "move": "Add only the checks that prevent recurring output mistakes."
+ },
+ {
+ "rank": 4,
+ "point": "Borrow one pattern, not a whole product",
+ "why": "External references improve quality when reduced to structure, not copied as surface style.",
+ "move": "Start from: Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts."
+ }
+ ],
+ "reviewer_rule": "Reviewer should ask whether the skill's structure will keep producing the desired behavior after repeated real use."
+}
\ No newline at end of file
diff --git a/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.md b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.md
new file mode 100644
index 00000000..1b59b0a4
--- /dev/null
+++ b/.previews/yao-meta-skill-v2-report-demo/geo-content-brief-skill/reports/system-model.md
@@ -0,0 +1,156 @@
+# System Model
+
+Skill: `geo-content-brief-skill`
+
+- Stability score: `77/100`
+- Stability band: `stable-first-pass`
+- Doctrine: Structure drives behavior: improve the boundary, feedback loops, drift watch, and leverage points before adding weight.
+
+## System Boundary Map
+
+- Owned job: 将 GEO 内容访谈、关键词笔记和渠道约束整理成可执行的中文内容简报 Skill，适用于团队复用、触发边界审查和后续质量评估。
+- Output boundary: a reusable skill output
+- Maturity assumption: `scaffold`
+- Input boundary:
+ - user-provided workflow notes, prompts, docs, or examples
+- Non-goals:
+ - one-off adjacent requests that do not match the recurring job
+ - private local material that was not intentionally included
+- Human judgment boundary:
+ - Ask one focused clarification when the real job, output, or exclusion boundary is unclear.
+ - Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.
+ - Do not silently broaden the skill into adjacent jobs just because the examples are nearby.
+
+## Feedback Loops
+
+### Intent boundary loop
+
+- Signal: Intent confidence score is 10/100.
+- Response: Ask only the highest-leverage clarification before adding package weight.
+- Evidence: reports/intent-confidence.md and reports/intent-dialogue.md
+
+### Reference synthesis loop
+
+- Signal: Benchmark patterns are useful only after they are abstracted into borrow and avoid guidance.
+- Response: Borrow one pattern at a time and keep the rest as reviewer-visible evidence.
+- Evidence: reports/reference-synthesis.md
+- Current patterns:
+ - Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
+ - Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.
+ - Borrow the discipline of defining what the skill should not own before growing the package.
+ - Do not let packaging or platform concerns swallow the core job boundary.
+ - Do not create experimental overhead that exceeds the skill's real risk tier.
+
+### Output quality loop
+
+- Signal: Generated output may fail in recurring domain-specific ways.
+- Response: Apply predicted output-risk families as self-repair checks before final output.
+- Evidence: reports/output-risk-profile.md
+- Current risk families:
+ - Markdown readability
+ - Screenshot and visual capture
+ - Citation and footnote clutter
+ - Tone and specificity
+ - Tutorial quality
+
+### Reviewer feedback loop
+
+- Signal: Human review catches drift that static checks miss.
+- Response: Capture lightweight feedback and turn repeated findings into gates or references.
+- Evidence: reports/review-viewer.html and feedback records
+
+### Lifecycle loop
+
+- Signal: As reuse grows, the skill needs stronger gates, ownership, and regression evidence.
+- Response: Promote only when the next gate improves reliability more than context cost.
+- Evidence: manifest.json, reports/iteration-directions.md, and governance checks
+
+## Delay And Drift Watch
+
+### Trigger drift
+
+- Watch signal: Users start invoking the skill for adjacent one-off or explanation-only requests.
+- Countermeasure: Add near-neighbor exclusions and route evals before expanding workflow steps.
+- Cadence: per trigger or description change
+
+### Output drift
+
+- Watch signal: Outputs remain valid but become generic, cluttered, or weakly aligned with the user's domain.
+- Countermeasure: Refresh output-risk and artifact-design profiles, then add one self-repair check.
+- Cadence: after the first 3-5 real uses
+- Risk families:
+ - Markdown readability
+ - Screenshot and visual capture
+ - Citation and footnote clutter
+ - Tone and specificity
+ - Tutorial quality
+
+### Reference drift
+
+- Watch signal: Borrowed benchmark patterns no longer fit the local job or add ceremony without payoff.
+- Countermeasure: Re-run reference synthesis and keep only patterns that improve the current boundary.
+- Cadence: per material benchmark or product assumption change
+
+### Governance drift
+
+- Watch signal: Skill usage becomes team-critical while ownership, review cadence, or rollback evidence stays informal.
+- Countermeasure: Promote maturity tier and add reviewer-visible lifecycle evidence.
+- Cadence: when reuse becomes real
+
+## Failure Pattern Map
+
+### Boundary failure
+
+- Symptom: The skill handles nearby requests that were never part of the recurring job.
+- Repair: Narrow the description and add explicit non-goals before adding more execution steps.
+
+### Feedback gap
+
+- Symptom: The skill has rules but no signal telling authors which rule should change after use.
+- Repair: Turn repeated reviewer feedback into one eval, one reference note, or one self-repair check.
+
+### Output degradation
+
+- Symptom: The result is structurally correct but generic, cluttered, or weakly matched to the user's domain.
+- Repair: Use output-risk families as pre-final checks.
+- Current Risk Families:
+ - Markdown readability
+ - Screenshot and visual capture
+ - Citation and footnote clutter
+ - Tone and specificity
+ - Tutorial quality
+
+### Prompt-behavior mismatch
+
+- Symptom: The role, task, and format are copied from a prompt instead of becoming stable skill behavior.
+- Repair: Convert reusable role/task/format assumptions into workflow, reports, or references.
+- Watch Axes:
+ - Specificity
+
+## Highest Leverage Moves
+
+### 1. Clarify the real job boundary
+
+- Why: Intent uncertainty creates downstream trigger, output, and governance errors.
+- Move: Ask one focused question and update intent context before adding assets.
+
+### 2. Tune the frontmatter description
+
+- Why: The description is the highest-leverage routing surface.
+- Move: Name the recurring job, expected input, output, and strongest non-goal in compact language.
+
+### 3. Install output self-repair checks
+
+- Why: The likely failure families are: Markdown readability, Screenshot and visual capture, Citation and footnote clutter.
+- Move: Add only the checks that prevent recurring output mistakes.
+
+### 4. Borrow one pattern, not a whole product
+
+- Why: External references improve quality when reduced to structure, not copied as surface style.
+- Move: Start from: Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
+
+## Reviewer Use
+
+- Reviewer should ask whether the skill's structure will keep producing the desired behavior after repeated real use.
+- Prefer changing the system boundary, feedback loop, or leverage point before adding more prose.
+- If a problem repeats, convert it into a named failure pattern and one regression check.
diff --git a/AGENTS.md b/AGENTS.md
index 47d8fee4..b7511deb 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -34,18 +34,60 @@ After source changes that affect scripts, package contents, trust evidence, Revi
GENERATED_AT="${GENERATED_AT:-$(date +%F)}"
python3 scripts/run_output_execution.py --runner-command '["python3","scripts/local_output_eval_runner.py"]'
python3 scripts/compile_skill.py . --generated-at "$GENERATED_AT"
-python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --expectations evals/packaging_expectations.json --output-dir dist --zip
+python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip
python3 scripts/simulate_install.py . --package-dir dist --install-root dist/install-simulation --output-json reports/install_simulation.json --output-md reports/install_simulation.md --generated-at "$GENERATED_AT"
python3 scripts/trust_check.py . --output-json reports/security_trust_report.json --output-md reports/security_trust_report.md
python3 scripts/registry_audit.py . --generated-at "$GENERATED_AT"
python3 scripts/verify_package.py . --package-dir dist --expectations evals/packaging_expectations.json --registry-json reports/registry_audit.json --output-json reports/package_verification.json --output-md reports/package_verification.md --require-zip --generated-at "$GENERATED_AT"
python3 scripts/registry_audit.py . --generated-at "$GENERATED_AT"
python3 scripts/upgrade_check.py . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json --current-package-json reports/registry_audit.json --output-json reports/upgrade_check.json --output-md reports/upgrade_check.md --generated-at "$GENERATED_AT"
+python3 scripts/render_adoption_drift_report.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_architecture_maintainability.py . --generated-at "$GENERATED_AT"
+python3 scripts/python_compat_check.py . --generated-at "$GENERATED_AT"
+python3 scripts/probe_runtime_permissions.py . --package-dir dist
+python3 scripts/render_review_waivers.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_review_annotations.py .
+python3 scripts/build_skill_atlas.py --workspace-root . --output-dir skill_atlas --report-html reports/skill_atlas.html --report-json reports/skill_atlas.json --today "$GENERATED_AT"
+python3 scripts/render_world_class_evidence_plan.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_world_class_evidence_ledger.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_world_class_evidence_intake.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_world_class_submission_review.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_world_class_operator_runbook.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_world_class_claim_guard.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_daily_skillops_report.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_weekly_curator_report.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_skill_os2_audit.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_skill_os2_coverage.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_context_reports.py --generated-at "$GENERATED_AT"
+python3 scripts/render_benchmark_reproducibility.py . --generated-at "$GENERATED_AT"
python3 scripts/render_skill_overview.py .
+python3 scripts/render_skill_interpretation.py .
+python3 scripts/render_review_viewer.py .
+python3 scripts/render_world_class_preflight.py . --generated-at "$GENERATED_AT"
python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json
+python3 scripts/render_evidence_consistency.py . --generated-at "$GENERATED_AT"
```
-Clean test-only scratch directories after verification with `rm -rf tests/tmp_*`. Do not clean unrelated untracked files.
+For final release evidence, commit source and generated package evidence first, then run the clean-lock reports from a clean worktree:
+
+```bash
+python3 scripts/render_context_reports.py --generated-at "$GENERATED_AT"
+python3 scripts/render_benchmark_reproducibility.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_daily_skillops_report.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_weekly_curator_report.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_skill_overview.py .
+python3 scripts/render_skill_interpretation.py .
+python3 scripts/render_review_viewer.py .
+python3 scripts/render_world_class_preflight.py . --generated-at "$GENERATED_AT"
+python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json
+python3 scripts/render_evidence_consistency.py . --generated-at "$GENERATED_AT"
+```
+
+If `reports/benchmark_reproducibility.json` reports `release_lock_ready: false`, do not commit that benchmark as release evidence. Restore the transient dirty-lock reports, commit the source/generated evidence that caused the dirty state, and regenerate the clean-lock reports on the resulting clean tree.
+
+Local sync into `~/.agents/skills.disabled/yao-meta-skill` or `~/.agents/skills/yao-meta-skill` must keep the install preflight enabled unless the user explicitly requests a diagnostic bypass. `make sync-local-install` and `make sync-active-install` rebuild the package first, then `scripts/sync_local_install.py` refuses to copy files when install simulation or installer permission enforcement fails.
+
+Clean test-only scratch directories after verification with `find tests -maxdepth 1 \( -name 'tmp' -o -name 'tmp_*' \) -type d -exec rm -rf {} +`. Do not clean unrelated untracked files.
## Boundaries
@@ -60,11 +102,28 @@ Clean test-only scratch directories after verification with `rm -rf tests/tmp_*`
- `scripts/yao.py`: unified CLI orchestration. Keep command behavior stable; move pure config and side-effect-free helpers into small internal modules.
- `scripts/render_skill_overview.py`: v2 bilingual skill overview report. Preserve `reports/skill-overview.html` / `.json`, `body data-report-lang="zh-CN"`, default Simplified Chinese, English switch, and inline-chart/no-external-dependency behavior.
+- `scripts/skill_report_layout.py` plus `assets/skill-overview.css` / `assets/skill-overview.js`: overview report layout contract and inline assets. Keep the generated report single-file at render time, but keep long CSS/JS source in `assets/` instead of embedding it inside Python.
- `scripts/render_review_studio.py`: Review Studio gate orchestration. Keep gate scoring, evidence links, and action generation separate from layout helpers.
-- `scripts/review_studio_layout.py`: Review Studio static layout and CSS contract.
+- `scripts/review_studio_layout.py` plus `assets/review-studio.css`: Review Studio static layout and CSS contract. Keep the generated report single-file at render time, but keep long CSS source in `assets/` instead of embedding it inside Python.
- `scripts/review_studio_formatting.py`: Review Studio dictionary-to-panel formatting and Chinese metric labels.
+- `scripts/review_studio_gates.py`: Review Studio gate evaluation, release decision scoring, and gate status labels.
+- `scripts/render_skill_os2_audit.py`: requirement-by-requirement Skill OS 2.0 completion audit. Keep local evidence, human-required gaps, and external-required gaps separate so reports do not overclaim world-class readiness.
+- `scripts/render_world_class_evidence_plan.py`: executable evidence task plan for the remaining world-class readiness gaps. Keep provider, human, native-permission, and real-client telemetry evidence requirements concrete without marking planned work as complete.
+- `scripts/render_world_class_evidence_ledger.py`: machine-checkable acceptance ledger for the remaining world-class evidence gaps. Keep anti-overclaim guards explicit so planned work, metadata fallbacks, pending review, and local command runners never count as final evidence.
+- `scripts/render_world_class_evidence_intake.py`: intake validator for external and human world-class evidence packets. Real submissions must reference concrete local aggregate artifacts with matching SHA-256 digests; templates may stay hash-free and must not count as evidence.
+- `scripts/world_class_evidence_contract.py`: shared intake contract and artifact-integrity validator. Keep ledger, intake, and submission review aligned so source evidence cannot be accepted without a valid real submission and matching artifact SHA-256 checks.
+- `scripts/render_world_class_submission_review.py`: read-only queue for external and human evidence packets after intake validation. Keep it from accepting evidence; it may only compare packet validity, source evidence checks, and ledger state.
+- `scripts/render_world_class_operator_runbook.py`: operator-facing world-class evidence runbook. Keep it as coordination guidance only; it must not accept evidence or flip world-class readiness.
+- `scripts/render_world_class_preflight.py` plus `scripts/world_class_preflight_layout.py`: operator-facing collection preflight for world-class evidence. Keep data assembly and CLI emission in the renderer, keep HTML layout in the layout helper, keep environment and external prerequisite checks redacted, and never let preflight count as accepted evidence.
+- `scripts/render_benchmark_reproducibility.py`: release-facing benchmark reproducibility manifest. Keep methodology sections, required artifacts, failure disclosure, reproduction commands, and world-class limitations machine-checkable.
- `scripts/skill_report_model.py`, `scripts/skill_report_metrics.py`, `scripts/skill_report_charts.py`: skill overview data model, scoring, and inline SVG chart generation.
- `scripts/yao_cli_config.py`: CLI target maps, archetype heuristics, diagnosis copy, and side-effect-free shaping helpers.
+- `scripts/yao_cli_parser.py`: CLI argparse command surface, flags, choices, and command handler binding.
+- `scripts/yao_cli_telemetry.py`: opt-in metadata-only CLI run telemetry. Keep it free of prompt, argument, output, transcript, note, or message capture.
+- `scripts/import_telemetry_events.py`: external telemetry importer. Validate the whole input before appending events, and keep raw prompt/output/transcript/message/note fields blocked.
+- `scripts/emit_telemetry_event.py`: external client telemetry emitter. It may append one normalized metadata event to a local spool, but must never accept or write raw prompt, output, transcript, message, note, argument, or private content.
+- `scripts/render_telemetry_hook_recipes.py`: client hook recipe report. Keep recipes metadata-only, mark native auto-capture as unclaimed unless a real client integration exists, and preserve dry-run commands for Browser/Chrome/IDE/wrapper adapters.
+- `scripts/telemetry_native_host.py`: Browser/Chrome Native Messaging telemetry bridge. Preserve length-prefixed stdio behavior, raw-content blocking, and launcher/manifest generation tests.
New helper modules that are imported by CLI/report scripts but are not standalone commands must declare:
diff --git a/Makefile b/Makefile
index 659bca15..c3871e02 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@ PYTHON ?= python3
LOCAL_SKILL_INSTALL_DIR ?= $(HOME)/.agents/skills.disabled/yao-meta-skill
ACTIVE_SKILL_INSTALL_DIR ?= $(HOME)/.agents/skills/yao-meta-skill
-.PHONY: eval eval-suite route-scorecard route-confusion-check description-optimization judge-blind-eval description-optimization-check promotion-check yao-cli-check skill-overview-check skill-report-metrics-check skill-report-charts-check skill-ir-check compiler-check output-eval-check output-execution-check output-review-adjudication-check runtime-conformance-check runtime-permission-check trust-check skill-atlas-check registry-audit-check package-verify-check install-simulation-check upgrade-check review-viewer-check review-studio-check feedback-check adoption-drift-check review-waivers-check review-annotations-check baseline-compare-check reference-scan-check github-benchmark-scan-check intent-confidence-check reference-synthesis-check output-risk-profile-check artifact-design-profile-check prompt-quality-profile-check system-model-check iteration-directions-check description-drift-history iteration-ledger results-panel regression-history context-reports portability-report portability-check failure-regression-check package-check package-failure-check security-boundary-check local-install-sync-check snapshot-check validate lint governance-check resource-boundary-check quality-check sync-local-install sync-active-install test ci-test clean
+.PHONY: eval eval-suite route-scorecard route-confusion-check description-optimization judge-blind-eval description-optimization-check promotion-check python-compat-check architecture-maintainability-check yao-cli-check yao-cli-world-class-check skill-overview-check skill-interpretation-check skill-report-metrics-check skill-report-charts-check html-rendering-check skill-ir-check compiler-check output-eval-check output-execution-check output-review-kit-check output-review-adjudication-check runtime-conformance-check runtime-permission-check trust-check skill-atlas-check registry-audit-check package-verify-check install-simulation-check upgrade-check review-viewer-check review-studio-check skill-os2-audit-check skill-os2-coverage-check world-class-evidence-check world-class-ledger-check world-class-intake-check world-class-submission-kit-check world-class-preflight-check world-class-submission-review-check world-class-runbook-check world-class-claim-guard-check benchmark-reproducibility-check evidence-consistency-check feedback-check adaptation-safety-check skillops-opportunity-check daily-skillops-check weekly-curator-check adoption-drift-check telemetry-import-check telemetry-emit-check telemetry-hooks-check telemetry-native-host-check review-waivers-check review-annotations-check baseline-compare-check reference-scan-check github-benchmark-scan-check intent-confidence-check reference-synthesis-check output-risk-profile-check artifact-design-profile-check prompt-quality-profile-check system-model-check iteration-directions-check description-drift-history iteration-ledger results-panel regression-history context-reports portability-report portability-check failure-regression-check package-check package-failure-check security-boundary-check local-install-sync-check snapshot-check validate lint governance-check resource-boundary-check quality-check sync-local-install sync-active-install test ci-test clean
eval:
$(PYTHON) scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
@@ -28,20 +28,36 @@ description-optimization-check:
promotion-check:
$(PYTHON) tests/verify_promotion_checker.py
+python-compat-check:
+ $(PYTHON) tests/verify_python_compat_check.py
+
+architecture-maintainability-check:
+ $(PYTHON) tests/verify_architecture_maintainability.py
+
yao-cli-check:
$(PYTHON) tests/verify_yao_cli.py
+yao-cli-world-class-check:
+ $(PYTHON) tests/verify_yao_cli_world_class.py
+
skill-overview-check:
$(PYTHON) tests/verify_skill_overview.py
+skill-interpretation-check:
+ $(PYTHON) tests/verify_skill_interpretation.py
+
skill-report-metrics-check:
$(PYTHON) tests/verify_skill_report_metrics.py
skill-report-charts-check:
$(PYTHON) tests/verify_skill_report_charts.py
+html-rendering-check:
+ $(PYTHON) tests/verify_html_rendering.py
+
skill-ir-check:
$(PYTHON) tests/verify_skill_ir.py
+ $(PYTHON) tests/verify_skill_ir_paths.py
compiler-check:
$(PYTHON) tests/verify_compile_skill.py
@@ -52,6 +68,9 @@ output-eval-check:
output-execution-check:
$(PYTHON) tests/verify_output_execution_runs.py
+output-review-kit-check:
+ $(PYTHON) tests/verify_output_review_kit.py
+
output-review-adjudication-check:
$(PYTHON) tests/verify_output_review_adjudication.py
@@ -85,12 +104,72 @@ review-viewer-check:
review-studio-check:
$(PYTHON) tests/verify_review_studio.py
+skill-os2-audit-check:
+ $(PYTHON) tests/verify_skill_os2_audit.py
+
+skill-os2-coverage-check:
+ $(PYTHON) tests/verify_skill_os2_coverage.py
+
+world-class-evidence-check:
+ $(PYTHON) tests/verify_world_class_evidence_plan.py
+
+world-class-ledger-check:
+ $(PYTHON) tests/verify_world_class_evidence_ledger.py
+
+world-class-intake-check:
+ $(PYTHON) tests/verify_world_class_evidence_intake.py
+
+world-class-submission-kit-check:
+ $(PYTHON) tests/verify_world_class_submission_kit.py
+
+world-class-preflight-check:
+ $(PYTHON) tests/verify_world_class_preflight.py
+
+world-class-submission-review-check:
+ $(PYTHON) tests/verify_world_class_submission_review.py
+
+world-class-runbook-check:
+ $(PYTHON) tests/verify_world_class_operator_runbook.py
+
+world-class-claim-guard-check:
+ $(PYTHON) tests/verify_world_class_claim_guard.py
+
+benchmark-reproducibility-check:
+ $(PYTHON) tests/verify_benchmark_reproducibility.py
+
+evidence-consistency-check:
+ $(PYTHON) tests/verify_evidence_consistency.py
+
feedback-check:
$(PYTHON) tests/verify_feedback.py
+adaptation-safety-check:
+ $(PYTHON) tests/verify_adaptation_safety.py
+
+skillops-opportunity-check:
+ $(PYTHON) tests/verify_skillops_opportunity.py
+
+daily-skillops-check:
+ $(PYTHON) tests/verify_daily_skillops.py
+
+weekly-curator-check:
+ $(PYTHON) tests/verify_weekly_curator.py
+
adoption-drift-check:
$(PYTHON) tests/verify_adoption_drift.py
+telemetry-import-check:
+ $(PYTHON) tests/verify_telemetry_import.py
+
+telemetry-emit-check:
+ $(PYTHON) tests/verify_telemetry_emit.py
+
+telemetry-hooks-check:
+ $(PYTHON) tests/verify_telemetry_hooks.py
+
+telemetry-native-host-check:
+ $(PYTHON) tests/verify_telemetry_native_host.py
+
review-waivers-check:
$(PYTHON) tests/verify_review_waivers.py
@@ -152,7 +231,7 @@ failure-regression-check:
$(PYTHON) tests/verify_failure_regressions.py
package-check:
- $(PYTHON) scripts/cross_packager.py . --platform openai --platform claude --platform generic --expectations evals/packaging_expectations.json --output-dir dist --zip
+ $(PYTHON) scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip
package-failure-check:
$(PYTHON) tests/verify_packager_failures.py
@@ -181,17 +260,17 @@ resource-boundary-check:
quality-check:
$(PYTHON) tests/verify_quality_checks.py
-sync-local-install:
+sync-local-install: package-check
$(PYTHON) scripts/sync_local_install.py --install-dir "$(LOCAL_SKILL_INSTALL_DIR)"
-sync-active-install:
+sync-active-install: package-check
$(PYTHON) scripts/sync_local_install.py --install-dir "$(ACTIVE_SKILL_INSTALL_DIR)"
-test: eval eval-suite route-scorecard route-confusion-check description-optimization description-optimization-check promotion-check yao-cli-check skill-overview-check skill-report-metrics-check skill-report-charts-check skill-ir-check compiler-check output-eval-check output-execution-check output-review-adjudication-check runtime-conformance-check runtime-permission-check trust-check skill-atlas-check registry-audit-check package-verify-check install-simulation-check upgrade-check review-viewer-check review-studio-check feedback-check adoption-drift-check review-waivers-check review-annotations-check baseline-compare-check reference-scan-check github-benchmark-scan-check intent-confidence-check reference-synthesis-check output-risk-profile-check artifact-design-profile-check prompt-quality-profile-check system-model-check iteration-directions-check description-drift-history iteration-ledger regression-history context-reports portability-report portability-check failure-regression-check package-check package-failure-check security-boundary-check local-install-sync-check snapshot-check validate lint governance-check resource-boundary-check quality-check
+test: eval eval-suite route-scorecard route-confusion-check description-optimization description-optimization-check promotion-check python-compat-check architecture-maintainability-check yao-cli-check skill-overview-check skill-interpretation-check skill-report-metrics-check skill-report-charts-check html-rendering-check skill-ir-check compiler-check output-eval-check output-execution-check output-review-adjudication-check runtime-conformance-check runtime-permission-check trust-check skill-atlas-check registry-audit-check package-verify-check install-simulation-check upgrade-check review-viewer-check review-studio-check skill-os2-audit-check skill-os2-coverage-check world-class-evidence-check world-class-ledger-check world-class-intake-check world-class-submission-kit-check world-class-preflight-check world-class-submission-review-check world-class-runbook-check world-class-claim-guard-check benchmark-reproducibility-check evidence-consistency-check feedback-check adaptation-safety-check skillops-opportunity-check daily-skillops-check weekly-curator-check adoption-drift-check telemetry-import-check telemetry-emit-check telemetry-hooks-check telemetry-native-host-check review-waivers-check review-annotations-check baseline-compare-check reference-scan-check github-benchmark-scan-check intent-confidence-check reference-synthesis-check output-risk-profile-check artifact-design-profile-check prompt-quality-profile-check system-model-check iteration-directions-check description-drift-history iteration-ledger regression-history context-reports portability-report portability-check failure-regression-check package-check package-failure-check security-boundary-check local-install-sync-check snapshot-check validate lint governance-check resource-boundary-check quality-check
ci-test:
$(PYTHON) scripts/ci_test.py
clean:
- rm -rf dist tests/tmp tests/tmp_snapshot tests/tmp_cli tests/tmp_skill_overview tests/tmp_skill_report_metrics tests/tmp_skill_report_charts tests/tmp_skill_ir tests/tmp_compile_skill tests/tmp_output_eval tests/tmp_output_execution tests/tmp_output_review_adjudication tests/tmp_conformance tests/tmp_runtime_permission tests/tmp_trust tests/tmp_skill_atlas tests/tmp_registry tests/tmp_package_verification tests/tmp_install_simulation tests/tmp_upgrade_check tests/tmp_reference_scan tests/tmp_iteration_directions tests/tmp_review_viewer tests/tmp_review_studio tests/tmp_feedback tests/tmp_adoption_drift tests/tmp_review_waivers tests/tmp_review_annotations tests/tmp_github_benchmark_scan tests/tmp_intent_confidence tests/tmp_reference_synthesis tests/tmp_output_risk_profile tests/tmp_artifact_design_profile tests/tmp_prompt_quality_profile tests/tmp_system_model tests/tmp_security tests/tmp_baseline_compare.json tests/tmp_baseline_compare.md
+ rm -rf dist tests/tmp tests/tmp_snapshot tests/tmp_cli tests/tmp_python_compat tests/tmp_architecture_maintainability tests/tmp_skill_overview tests/tmp_skill_interpretation tests/tmp_skill_report_metrics tests/tmp_skill_report_charts tests/tmp_skill_ir tests/tmp_compile_skill tests/tmp_output_eval tests/tmp_output_execution tests/tmp_output_review_adjudication tests/tmp_conformance tests/tmp_runtime_permission tests/tmp_trust tests/tmp_skill_atlas tests/tmp_registry tests/tmp_package_verification tests/tmp_install_simulation tests/tmp_upgrade_check tests/tmp_reference_scan tests/tmp_iteration_directions tests/tmp_review_viewer tests/tmp_review_studio tests/tmp_skill_os2_audit tests/tmp_skill_os2_coverage tests/tmp_world_class_evidence tests/tmp_world_class_evidence_ledger tests/tmp_world_class_evidence_intake tests/tmp_world_class_submission_review tests/tmp_world_class_operator_runbook tests/tmp_world_class_claim_guard tests/tmp_benchmark_reproducibility tests/tmp_evidence_consistency tests/tmp_feedback tests/tmp_daily_skillops tests/tmp_weekly_curator tests/tmp_adoption_drift tests/tmp_telemetry_import tests/tmp_telemetry_emit tests/tmp_telemetry_hooks tests/tmp_telemetry_native_host tests/tmp_review_waivers tests/tmp_review_annotations tests/tmp_github_benchmark_scan tests/tmp_intent_confidence tests/tmp_reference_synthesis tests/tmp_output_risk_profile tests/tmp_artifact_design_profile tests/tmp_prompt_quality_profile tests/tmp_system_model tests/tmp_security tests/tmp_baseline_compare.json tests/tmp_baseline_compare.md
find . -type d -name __pycache__ -prune -exec rm -rf {} +
diff --git a/README.md b/README.md
index bda3dc80..2367431f 100644
--- a/README.md
+++ b/README.md
@@ -23,15 +23,32 @@ It turns rough workflows, transcripts, prompts, notes, and runbooks into reusabl
- a silent-by-default GitHub benchmark scan plus reference synthesis that studies top public repositories and world-class pattern tracks, then surfaces only real conflicts or uncertainty to the user
- a generated visual HTML overview for each newly initialized skill
- a Review Studio 2.0 HTML gate page that combines intent, trigger, output eval, context, runtime, trust, atlas, adoption drift, reviewer waivers, reviewer annotations, release evidence, and per-warning fix actions
+- a Skill OS 2.0 audit that maps each world-class requirement to current evidence, human-required gaps, and external-required gaps
+- a Skill OS 2.0 blueprint coverage report that maps the upgrade plan's core modules and recommended PRs to concrete artifacts, commands, and tests
+- a world-class evidence plan that turns remaining provider, human, native-permission, and real-client telemetry gaps into executable evidence tasks
+- a world-class evidence ledger that records which external and human evidence is accepted or still pending without treating planned work as proof
+- a world-class evidence intake contract that validates external and human evidence packets for provenance, privacy, artifact refs, and anti-overclaim rules before ledger review
+- a redacted world-class preflight report that checks local files, environment readiness, human/external prerequisites, and source blockers before operators collect evidence
+- a world-class submission review queue that compares evidence packets, intake validation, source artifacts, and ledger state without accepting evidence
+- a world-class operator runbook that gives reviewers the exact commands, artifacts, and collection checklist needed to close remaining evidence gaps
+- a world-class claim guard that scans public claim surfaces and blocks premature completed/true claims while the evidence ledger still has pending external or human evidence
+- a benchmark reproducibility manifest that checks methodology sections, required artifacts, failure disclosure, and reproduction commands
+- an evidence consistency gate that compares generated reports against each other so benchmark, overview, interpretation, adoption, world-class ledger, coverage, and Review Studio facts do not drift silently
- Output Eval Lab evidence with assertion grading, execution/timing/token evidence, a blind A/B review pack, a separate answer key, and reviewer adjudication reports
- a runtime permission probe report that checks packaged target adapters for explicit permission metadata, native-enforcement flags, metadata fallback notes, and residual risks
+- a Python compatibility gate that catches supported-runtime syntax hazards before they reach GitHub Actions or packaged distribution
- a side-by-side HTML review studio for first-pass human review
- an artifact design profile that defines visual direction, layout patterns, and quality gates for reports, tutorials, dashboards, screenshots, and review pages
- a prompt quality profile that abstracts need modeling, RTF mapping, complexity, and quality checks into reviewer-visible evidence instead of bloating `SKILL.md`
- a systems-thinking model that maps boundaries, feedback loops, drift risks, recurring failure patterns, and highest-leverage quality moves
- three high-value next iteration directions after the first package is created
- a lightweight feedback log that does not require a full promotion cycle
-- a local-first metadata-only adoption and drift report that turns real usage signals into next iteration candidates
+- a local-first metadata-only adoption and drift report that turns real usage signals into next iteration candidates, with optional `yao.py` CLI run capture, external client event emit hooks, hook recipes, and JSONL import that record command names and outcomes without arguments or raw content
+- an explicit-source adaptive proposal loop that summarizes redacted repeated user preferences and generates approval-gated adaptation proposals without scanning private logs or writing source files
+- a SkillOps opportunity scorer and decision policy that ranks redacted repeated signals, maps them to report-only, AGENTS update, existing-skill patch, or eval-addition actions, and keeps every durable write approval-gated
+- a weekly SkillOps curator report that aggregates daily opportunities, Skill Atlas portfolio signals, release lock state, and world-class evidence gaps into a proposal-only maintenance queue
+- a Browser/Chrome Native Messaging telemetry host that can receive length-prefixed metadata-only client events and generate a local launcher plus manifest without storing raw content
+- a Skill Atlas drift layer that reads aggregate adoption reports and surfaces portfolio-level drift signals without packaging raw telemetry logs
- a baseline compare report for with-skill vs baseline review
- a conversation-style, archetype-aware quickstart that steers new packages toward scaffold, production, library, or governed fits
- Skill IR as the platform-neutral semantic contract, plus compiler reports and client-specific adapters
@@ -89,7 +106,7 @@ Weighted score formula: `sum(score / 10 * weight)`.
2. Start with a short, human intent dialogue so the real job, outputs, exclusions, constraints, and standards are explicit.
3. Let `quickstart` clarify intent first, then run silent benchmark scan and reference synthesis; it only surfaces explicit questions when intent is still unclear or when there is a real design conflict.
4. Use the archetype-aware `quickstart` or the full authoring flow to generate or improve the package in scaffold, production, library, or governed mode.
-5. Review the generated `reports/skill-overview.html` first for the bilingual HTML skill report. It defaults to Simplified Chinese and provides an English switch in the top right. Then open `reports/review-studio.html` to inspect release blockers, permission approvals, and evidence paths in one page before adding more structure.
+5. Review the generated `reports/skill-interpretation.html` first for the bilingual interpretation report. It defaults to Simplified Chinese and provides an English switch in the top right. Then open `reports/skill-overview.html` for the audit scorecard and `reports/review-studio.html` to inspect release blockers, permission approvals, and evidence paths in one page before adding more structure.
Or use the unified authoring CLI:
@@ -100,32 +117,56 @@ python3 scripts/yao.py reference-scan my-skill \
--external-reference "World Class Method::method::Borrow a tight evaluation loop.::Do not copy heavy process." \
--user-reference "A product or repo I admire::taste::Learn the clarity and operating standard.::Do not copy wording." \
--local-constraint "Current Library Naming::structure::Keep naming aligned with the local skill library.::Do not inherit private references."
+python3 scripts/yao.py skill-interpretation my-skill
python3 scripts/yao.py review-viewer my-skill
python3 scripts/yao.py review-studio my-skill
python3 scripts/yao.py artifact-design-profile my-skill
python3 scripts/yao.py prompt-quality-profile my-skill
python3 scripts/yao.py system-model my-skill
python3 scripts/yao.py feedback my-skill --note "Tighten exclusions before adding scripts." --rating 4 --category boundary
+python3 scripts/yao.py adapt-scan my-skill --source ./curated-user-signals.jsonl
+python3 scripts/yao.py adapt-propose my-skill
+python3 scripts/yao.py daily-skillops my-skill --source ./curated-user-signals.jsonl
+python3 scripts/yao.py weekly-curator my-skill
python3 scripts/yao.py adoption-drift my-skill --record-event skill_activation --activation-type explicit --outcome accepted
+YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate my-skill
+python3 scripts/yao.py telemetry-emit my-skill --event skill_activation --activation-type explicit --outcome accepted --command browser-extension
+python3 scripts/yao.py telemetry-hooks my-skill
+python3 scripts/telemetry_native_host.py my-skill --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/
+python3 scripts/yao.py telemetry-import my-skill --input-jsonl /tmp/external-client-events.jsonl --command browser-extension
python3 scripts/yao.py review-waivers my-skill --add-waiver --gate-key trust-report --reviewer "Yao Team" --reason "Known warning accepted for this release with bounded follow-up." --expires-at 2026-09-30
python3 scripts/yao.py review-waivers my-skill --add-waiver --gate-key permission-gates --reviewer "Yao Team" --reason "Permission warning accepted only for this non-governed release window." --expires-at 2026-09-30
python3 scripts/yao.py review-annotations my-skill --add-annotation --gate-key output-lab --target-path reports/output_quality_scorecard.md --line 1 --body "Clarify recorded fixture vs model-executed evidence before release."
python3 scripts/yao.py baseline-compare
python3 scripts/yao.py check-update
python3 scripts/yao.py skill-ir . --output-json skill-ir/examples/yao-meta-skill.json
-python3 scripts/yao.py compile-skill . --target openai --target claude --target generic
+python3 scripts/yao.py compile-skill . --target openai --target claude --target generic --target vscode
python3 scripts/yao.py package . --platform generic --output-dir dist
python3 scripts/yao.py output-eval
python3 scripts/yao.py output-exec
python3 scripts/yao.py output-review
python3 scripts/yao.py conformance .
python3 scripts/yao.py trust .
+python3 scripts/yao.py python-compat .
python3 scripts/yao.py runtime-permissions . --package-dir dist
python3 scripts/yao.py skill-atlas --workspace-root .
python3 scripts/yao.py registry-audit .
python3 scripts/yao.py package-verify . --package-dir dist --require-zip
python3 scripts/yao.py install-simulate . --package-dir dist
python3 scripts/yao.py upgrade-check . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json
+python3 scripts/yao.py world-class-evidence .
+SUBMISSIONS_DIR="${SUBMISSIONS_DIR:-evidence/world_class/submissions}"
+python3 scripts/yao.py world-class-preflight . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR"
+# Alternative: prefill artifact SHA-256 digests while keeping drafts template-only.
+python3 scripts/yao.py world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR" --prefill-artifacts
+python3 scripts/yao.py world-class-intake . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-submission-review . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-ledger . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-runbook . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-claim-guard .
+python3 scripts/yao.py benchmark-reproducibility .
+python3 scripts/yao.py evidence-consistency .
```
## Local Development Source
@@ -140,7 +181,7 @@ Sync the current source into the disabled mirror:
make sync-local-install
```
-The sync command copies Git-tracked files plus new source files in code and guidance directories such as `scripts/`, `tests/`, `references/`, and `docs/`. It skips untracked business-skill folders and untracked private reports by default, so local experiments do not leak into the mirror.
+The sync command first rebuilds the package and runs install preflight against `dist/yao-meta-skill.zip`. It refuses to sync when package extraction, adapter readability, or installer permission enforcement fails. After the preflight passes, it copies Git-tracked files plus new source files in code and guidance directories such as `scripts/`, `tests/`, `references/`, and `docs/`. It skips untracked business-skill folders and untracked private reports by default, so local experiments do not leak into the mirror.
Restore an active global Codex install only when you intentionally want this skill discoverable outside the development workspace:
@@ -177,7 +218,7 @@ python3 scripts/context_sizer.py .
python3 scripts/resource_boundary_check.py .
python3 scripts/governance_check.py . --require-manifest
python3 scripts/compile_skill.py .
-python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --expectations evals/packaging_expectations.json --zip
+python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --zip
python3 scripts/probe_runtime_permissions.py . --package-dir dist
python3 tests/verify_packager_failures.py
```
@@ -199,7 +240,7 @@ python3 scripts/yao.py review --target root
python3 scripts/yao.py release-snapshot --target root --label release-candidate
python3 scripts/yao.py skill-ir . --output-json skill-ir/examples/yao-meta-skill.json
python3 scripts/yao.py compile-skill .
-python3 scripts/yao.py package . --platform openai --platform claude --platform generic --output-dir dist --zip
+python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
python3 scripts/yao.py runtime-permissions . --package-dir dist
python3 scripts/yao.py package-verify . --package-dir dist --require-zip
python3 scripts/yao.py test
@@ -247,9 +288,9 @@ The homepage panel below is generated from the current eval suite so the family-
Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/family_summary.md](reports/family_summary.md)
-- packaging validation: `openai`, `claude`, and `generic` targets pass contract checks and carry IR provenance, semantic parity metadata, and target-native behavior contracts
-- target compiler validation: `openai`, `claude`, `generic`, and Agent Skills compatible contracts are compiled from Skill IR with generated-file mappings, adapter modes, native surfaces, permission enforcement notes, and unsupported-feature notes
-- runtime permission probes: `openai`, `claude`, and `generic` adapters expose explicit permission contracts; current targets report `0` native-enforcement adapters and `3` metadata fallbacks with residual risks visible to reviewers
+- packaging validation: `openai`, `claude`, `generic`, and `vscode` targets pass contract checks and carry IR provenance, semantic parity metadata, and target-native behavior contracts
+- target compiler validation: `openai`, `claude`, `generic`, Agent Skills compatible, and VS Code / Copilot contracts are compiled from Skill IR with generated-file mappings, adapter modes, native surfaces, permission enforcement notes, and unsupported-feature notes
+- runtime permission probes: `openai`, `claude`, `generic`, and `vscode` adapters expose explicit permission contracts; current targets report `0` native-enforcement adapters and `4` metadata fallbacks with residual risks visible to reviewers
- portability score: `100/100` with neutral activation, execution, trust, and degradation metadata preserved across all exported targets
- description optimization suite: root, team frontend review, and governed incident command pass blind and adversarial holdout gates; governed incident command still carries one visible holdout miss, and adversarial calibration plus family drift are now tracked separately
- judge-backed blind eval: root, team frontend review, and governed incident command now pass an independent rubric judge on blind holdout prompts
@@ -258,8 +299,8 @@ Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/fa
- governance and resource-boundary checks are part of the default test path
- root governance maturity score: `90/100`; governed benchmark example: `95/100`
- CJK-aware trigger matching is now covered by explicit Chinese build, packaging, eval, and near-neighbor cases
-- context budgets: root `987/1000`, complex benchmark `790/1000`, governed benchmark `760/1000`
-- quality density: root `131.7`, complex benchmark `164.6`, governed benchmark `171.1`
+- context budgets: root `944/1000`, complex benchmark `790/1000`, governed benchmark `760/1000`
+- quality density: root `137.7`, complex benchmark `164.6`, governed benchmark `171.1`
- regression milestones are tracked in [reports/regression_history.md](reports/regression_history.md)
- description drift history is tracked in [reports/description_drift_history.md](reports/description_drift_history.md)
- route confusion is tracked in [reports/route_scorecard.md](reports/route_scorecard.md)
@@ -269,8 +310,18 @@ Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/fa
- lightweight with-skill vs baseline comparison is published in [reports/baseline-compare.md](reports/baseline-compare.md)
- Review Studio 2.0 gate evidence is published in [reports/review-studio.html](reports/review-studio.html)
- Review Studio fix actions are embedded in [reports/review-studio.json](reports/review-studio.json)
+- Skill OS 2.0 blueprint coverage is published in [reports/skill_os2_coverage.md](reports/skill_os2_coverage.md)
- reviewer waiver evidence is published in [reports/review_waivers.md](reports/review_waivers.md)
+- remaining world-class evidence tasks are published in [reports/world_class_evidence_plan.md](reports/world_class_evidence_plan.md)
+- current world-class evidence acceptance state is published in [reports/world_class_evidence_ledger.md](reports/world_class_evidence_ledger.md)
+- world-class evidence intake readiness is published in [reports/world_class_evidence_intake.md](reports/world_class_evidence_intake.md)
+- world-class submission review queue is published in [reports/world_class_submission_review.md](reports/world_class_submission_review.md)
+- world-class operator runbook is published in [reports/world_class_operator_runbook.md](reports/world_class_operator_runbook.md) and [reports/world_class_operator_runbook.html](reports/world_class_operator_runbook.html)
+- world-class public claim guard status is published in [reports/world_class_claim_guard.md](reports/world_class_claim_guard.md)
+- benchmark reproducibility evidence is published in [reports/benchmark_reproducibility.md](reports/benchmark_reproducibility.md)
+- cross-report evidence consistency is published in [reports/evidence_consistency.md](reports/evidence_consistency.md)
- target compiler evidence is published in [reports/compiled_targets.md](reports/compiled_targets.md)
+- Python runtime compatibility evidence is published in [reports/python_compatibility.md](reports/python_compatibility.md)
- registry package metadata and audit status are published in [reports/registry_audit.md](reports/registry_audit.md)
- package archive verification is published in [reports/package_verification.md](reports/package_verification.md)
- temporary local install simulation is published in [reports/install_simulation.md](reports/install_simulation.md)
@@ -385,7 +436,7 @@ Utility scripts that make the meta-skill operational:
- `run_description_optimization_suite.py`: runs description optimization across the root skill and governed examples, then writes reusable reports and optional drift snapshots with calibration and family summaries
- `promotion_checker.py`: applies promotion policy to current description candidates, writes promotion decisions, builds candidate registries, and emits iteration bundles with review stubs
- `create_iteration_snapshot.py`: freezes the current promotion decision into a versioned release snapshot with review, route, and context evidence
-- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, review, release-snapshot, workspace-flow, report, skill-ir, compile-skill, output-exec, output-review, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint
+- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, python-compat, review, release-snapshot, workspace-flow, report, skill-report, skill-interpretation, skill-ir, compile-skill, output-exec, output-review, skill-os2-audit, skill-os2-coverage, world-class-evidence, world-class-ledger, world-class-intake, world-class-preflight, world-class-submission-kit, world-class-submission-review, world-class-runbook, world-class-claim-guard, benchmark-reproducibility, evidence-consistency, adapt-scan, adapt-propose, adapt-apply, daily-skillops, weekly-curator, telemetry-emit, telemetry-hooks, telemetry-import, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint
- `render_description_drift_history.py`: turns description-optimization snapshots into a readable drift-history report
- `build_confusion_matrix.py`: scores route confusion across tracked sibling skills and `no_route` cases, then writes a route scorecard and optional milestone snapshot
- `render_iteration_ledger.py`: compresses regression milestones, description optimization drift, and route scorecards into one iteration-facing ledger
@@ -394,9 +445,25 @@ Utility scripts that make the meta-skill operational:
- `governance_check.py`: validates owner, review cadence, lifecycle stage, and maturity metadata
- `render_context_reports.py`: generates root and example context-budget reports plus a shared context summary
- `render_regression_history.py`: turns milestone snapshots into a readable regression history report
+- `render_skill_os2_audit.py`: renders a requirement-by-requirement Skill OS 2.0 audit that separates landed local evidence from human-required and external-required gaps
+- `render_skill_os2_coverage.py`: maps the Skill OS 2.0 upgrade blueprint to local artifacts, commands, tests, and remaining evidence boundaries
+- `render_daily_skillops_report.py`: renders an explicit-source Daily SkillOps operations report that summarizes redacted user patterns, proposal-only adaptations, approval state, release evidence, and world-class evidence gaps without scanning private logs or applying patches
+- `render_weekly_curator_report.py`: renders a weekly SkillOps curator report from generated daily reports, Skill Atlas, benchmark lock, evidence consistency, and world-class ledger state without scanning private logs or applying patches
+- `skillops_opportunity.py`: scores redacted SkillOps opportunities and maps them to approval-gated action types such as report-only, AGENTS update, existing-skill patch, or eval addition
+- `render_world_class_evidence_plan.py`: renders executable evidence tasks for remaining world-class gaps without treating planned external work as completed evidence
+- `render_world_class_evidence_ledger.py`: renders a machine-checkable ledger for current world-class evidence acceptance, anti-overclaim guards, provenance requirements, and privacy contracts
+- `render_world_class_evidence_intake.py`: validates world-class external and human evidence packets against provenance, privacy, artifact, and anti-overclaim requirements before ledger review
+- `render_world_class_preflight.py`: renders redacted collection preflight checks for pending provider, human, native-permission, and native-client evidence without accepting evidence
+- `render_world_class_submission_review.py`: renders a read-only queue that compares submissions, intake validation, source evidence, and ledger state without accepting evidence
+- `render_world_class_operator_runbook.py`: renders an operator-facing checklist and command map for collecting pending world-class evidence without accepting evidence
+- `render_world_class_claim_guard.py`: scans README, docs, and reports for premature world-class completion claims while accepted evidence is still pending
+- `render_benchmark_reproducibility.py`: renders methodology, artifact, failure-disclosure, and reproduction-command evidence for public benchmark claims
+- `render_evidence_consistency.py`: compares generated report facts across benchmark reproducibility, overview, interpretation, adoption drift, world-class ledger, coverage, and Review Studio artifacts
+- `python_compat_check.py`: checks Python source for supported-runtime compatibility hazards such as Python 3.11 f-string expression backslashes
- `cross_packager.py`: builds client-specific export artifacts from Skill IR plus neutral metadata, with explicit platform contracts and validation
- `render_portability_report.py`: scores cross-environment portability from neutral metadata, degradation rules, and consumer validation coverage
- `render_skill_overview.py`: generates the white-background bilingual HTML skill audit report with sticky four-character Chinese navigation, top-right language switch, v2 scorecard, inline SVG charts, contract boundary, quality review, risk governance, assets, and iteration roadmap
+- `render_skill_interpretation.py`: renders `reports/skill-interpretation.html/json` as the first-class post-creation interpretation report while reusing the Skill Overview v2 model and Kami white layout
- `export_skill_ir.py`: exports the 2.0 platform-neutral Skill IR contract from `SKILL.md`, manifest, interface metadata, evals, resources, and reports
- `compile_skill.py`: compiles Skill IR into target-specific semantic contracts, generated-file maps, adapter modes, target-native behavior contracts, preserved semantics, warnings, and unsupported-feature notes
- `run_output_eval.py`: runs the Output Eval Lab v0 with static with-skill vs baseline assertion grading, blind A/B review pack generation, and separate answer key artifacts
@@ -406,12 +473,17 @@ Utility scripts that make the meta-skill operational:
- `render_review_annotations.py`: records reviewer annotations tied to Review Studio gates, source/report paths, and optional line numbers, with open blocker annotations reflected in Review Studio decisions
- `run_conformance_suite.py`: verifies runtime conformance for OpenAI, Claude, Agent Skills, VS Code/Copilot-style, and generic targets
- `trust_check.py`: generates the trust/security report for scripts, dependencies, secret risk, bounded network host policy, execution-level `--help` smoke checks, permission inputs, trust metadata, and stable source-contract integrity
-- `build_skill_atlas.py`: builds the Skill Atlas catalog, route-overlap matrix, dependency graph, stale report, owner gaps, and HTML overview for a multi-skill workspace
+- `build_skill_atlas.py`: builds the Skill Atlas catalog, route-overlap matrix, dependency graph, stale report, owner gaps, aggregate drift signals, and HTML overview for a multi-skill workspace
- `registry_audit.py`: builds registry package metadata and audits version, owner, license, checksum, Skill IR source, and compatibility matrix
- `verify_package.py`: verifies generated package manifests, target adapters, zip archive safety, archive checksum, and registry parity
- `simulate_install.py`: extracts a generated zip into a temporary skill root and verifies entrypoint, manifest, interface, reports, and adapters can be loaded
- `upgrade_check.py`: compares current and previous registry package metadata, recommends a version bump, and blocks incompatible upgrade claims
- `render_adoption_drift_report.py`: records metadata-only local telemetry and renders adoption, missed-trigger, bad-output, script-error, and review-drift signals without packaging raw event logs
+- `import_telemetry_events.py`: imports external metadata-only telemetry JSONL after whole-file privacy validation, then refreshes the aggregate adoption drift report
+- `emit_telemetry_event.py`: emits one metadata-only external client event into a local spool for later `telemetry-import`, with dry-run validation and raw-content field blocking
+- `render_telemetry_hook_recipes.py`: renders Browser, Chrome, VS Code, CLI wrapper, and provider-adapter telemetry hook recipes with dry-run commands and explicit native-integration caveats
+- `telemetry_native_host.py`: receives Browser/Chrome Native Messaging length-prefixed JSON events, rejects raw-content fields, appends metadata-only events, and writes local launcher/manifest files for operator installation
+- `yao_cli_telemetry.py`: opt-in metadata-only `yao.py` run capture for command name, source, outcome, and failure class without command arguments or raw content
- `render_review_waivers.py`: validates human reviewer risk approvals with gate keys, reasons, expiry dates, and blocker-safe waiver policy
- `init_skill.py`, `lint_skill.py`, `validate_skill.py`, `diff_eval.py`: minimal authoring toolchain
- `check_update.py`: checks GitHub for a newer `VERSION` or remote manifest version and reports a reinstall hint without modifying local files
diff --git a/SKILL.md b/SKILL.md
index 34d49012..89809d3b 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -1,9 +1,8 @@
---
name: yao-meta-skill
-description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
metadata:
author: Yao Team
- philosophy: "structured design, evaluation loop, template ergonomics, operational packaging"
---
# Yao Meta Skill
@@ -16,44 +15,39 @@ metadata:
## Modes
-- `Scaffold`: exploratory or personal.
-- `Production`: team reuse.
-- `Library`: shared infrastructure.
-
-Mode rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).
+- `Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.
+- Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).
## Compact Workflow
-1. Decide whether the request should become a skill and choose the lightest fit.
-2. Capture job, output, exclusions, constraints, and standards.
-3. Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.
-4. Write the `description` early and test route quality before expanding the package.
-5. Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.
-6. Add only folders and gates that earn their keep.
-7. Surface the top three next iteration directions.
+1. For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.
+2. Capture job, output, exclusions, constraints, standards, and the lightest fit.
+3. Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
+4. Write `description` early, test route quality, then add only earned folders and gates.
+5. Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful.
-Core playbooks: [Method](references/skill-engineering-method.md), [Intent](references/intent-dialogue.md), [Reference Scan](references/reference-scan.md), [Skill IR](references/skill-ir-method.md), [Output Eval](references/output-eval-method.md), [Registry](references/distribution-registry-method.md), [Telemetry](references/telemetry-drift-method.md), [Waivers](references/review-waiver-method.md), [Review Studio](references/review-studio-method.md).
+Playbooks: [Method](references/skill-engineering-method.md), [Intent](references/intent-dialogue.md), [Skill IR](references/skill-ir-method.md), [Output Eval](references/output-eval-method.md), [Review Studio](references/review-studio-method.md).
## Skill OS 2.0 Gates
-For production, library, governed, or team-distributed work, use Skill IR, target compiler contracts, trigger + output eval, Skill Atlas, runtime conformance, trust report, registry metadata, package verification, install simulation, upgrade check, adoption/drift telemetry, reviewer waiver evidence, and Review Studio before release.
+For production, library, governed, or team-distributed work, run Skill IR, target compiler, trigger + output eval, Skill Atlas, conformance, trust, registry/package/install, upgrade, drift, waiver, and Review Studio gates before release.
+
+## Governed Package Boundary
+
+For file-backed, release-critical, or governed packages, name `input_files` as `file-backed fixture` evidence; include `owner`, `review cadence`, `input_files`, `output contract`, `rollback boundary`; require `trust report` and `reports/output_quality_scorecard.md`; mark unavailable telemetry, approvals, metrics, or benchmarks as `missing evidence`; do not fabricate evidence.
+
+Preserve audit labels literally when they apply: `file-backed fixture`, `input_files`, `output contract`, `rollback boundary`, `trust report`, `reports/output_quality_scorecard.md`, `missing evidence`.
## First-Turn Style
-When the skill first activates:
-
-- start from the user's work and desired outcome before structure
-- ask only `2-3` key questions unless the user already gave enough detail
-- avoid cold field lists; surface benchmark choices only when uncertainty or conflict needs a call
-
-Chinese conversations should sound soft and companion-like rather than procedural.
-
-Opening patterns: [Intent Dialogue](references/intent-dialogue.md).
+- Start from the user's work/outcome before structure.
+- Ask only `2-3` key questions unless enough detail exists.
+- In Chinese, sound soft and companion-like; use [Intent Dialogue](references/intent-dialogue.md).
## Output Contract
-Unless the user asks otherwise, produce a working skill directory with `SKILL.md`, aligned `agents/interface.yaml`, justified assets, and a short summary of boundary, exclusions, gates, and next steps.
+Unless asked otherwise, produce `SKILL.md`, aligned `agents/interface.yaml`, justified assets, and a short summary of boundary, exclusions, gates, and next steps.
## Reference Map
-Primary references: [Method](references/skill-engineering-method.md), [Artifact Design](references/artifact-design-doctrine.md), [Systems Thinking](references/systems-thinking-doctrine.md), [Governance](references/governance.md).
+Primary: [Method](references/skill-engineering-method.md), [Artifact Design](references/artifact-design-doctrine.md), [Systems Thinking](references/systems-thinking-doctrine.md), [Governance](references/governance.md), [SkillOps Decision](references/skillops-decision-policy.md).
diff --git a/agents/interface.yaml b/agents/interface.yaml
index 599591a9..35912dba 100644
--- a/agents/interface.yaml
+++ b/agents/interface.yaml
@@ -8,6 +8,7 @@ compatibility:
- "openai"
- "claude"
- "generic"
+ - "vscode"
activation:
mode: "manual"
paths: []
@@ -22,3 +23,4 @@ compatibility:
openai: "metadata-adapter"
claude: "neutral-source-plus-adapter"
generic: "neutral-source"
+ vscode: "agent-skills-source-with-vscode-notes"
diff --git a/assets/review-studio.css b/assets/review-studio.css
new file mode 100644
index 00000000..20bdd878
--- /dev/null
+++ b/assets/review-studio.css
@@ -0,0 +1,749 @@
+:root {
+ --ink: #1B365D;
+ --text: #24201d;
+ --muted: #746d66;
+ --line: #e7ded2;
+ --soft: #faf8f5;
+ --pass: #1e6b52;
+ --warn: #9a6718;
+ --block: #9b2c2c;
+ }
+ * { box-sizing: border-box; }
+ html { scroll-behavior: smooth; }
+ body {
+ margin: 0;
+ background: #ffffff;
+ color: var(--text);
+ font-family: Georgia, "Times New Roman", "Songti SC", serif;
+ line-height: 1.58;
+ }
+ nav {
+ position: sticky;
+ top: 0;
+ z-index: 10;
+ display: flex;
+ gap: 4px;
+ justify-content: center;
+ flex-wrap: wrap;
+ padding: 10px 16px;
+ background: rgba(255,255,255,0.94);
+ border-bottom: 1px solid var(--line);
+ backdrop-filter: blur(10px);
+ }
+ nav a {
+ color: var(--ink);
+ text-decoration: none;
+ font-size: 14px;
+ padding: 7px 10px;
+ border-radius: 6px;
+ }
+ nav a:hover { background: var(--soft); }
+ main { max-width: 1180px; margin: 0 auto; padding: 44px 28px 76px; }
+ header { border-bottom: 1px solid var(--line); padding-bottom: 28px; margin-bottom: 28px; }
+ .eyebrow { color: var(--ink); font-size: 14px; letter-spacing: .08em; text-transform: uppercase; }
+ h1, h2, h3 { color: var(--text); font-weight: 500; margin: 0; letter-spacing: 0; }
+ h1 { font-size: clamp(34px, 5vw, 64px); line-height: 1.03; max-width: 920px; margin-top: 12px; }
+ h2 { font-size: 30px; margin-bottom: 14px; }
+ h3 { font-size: 19px; }
+ p { margin: 0; }
+ .lede { max-width: 820px; color: var(--muted); font-size: 20px; margin-top: 18px; }
+ .decision {
+ display: inline-flex;
+ align-items: baseline;
+ gap: 12px;
+ margin-top: 24px;
+ padding: 12px 16px;
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ color: var(--ink);
+ background: var(--soft);
+ }
+ .decision strong { font-size: 28px; }
+ section { padding: 30px 0; border-bottom: 1px solid var(--line); scroll-margin-top: 76px; }
+ .metrics, .gates {
+ display: grid;
+ grid-template-columns: repeat(5, minmax(0, 1fr));
+ gap: 14px;
+ }
+ .gates { grid-template-columns: repeat(4, minmax(0, 1fr)); }
+ .metric, .gate {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 16px;
+ background: #fff;
+ min-width: 0;
+ }
+ .metric span, .gate span { display: block; color: var(--muted); font-size: 13px; }
+ .metric strong { display: block; color: var(--ink); font-size: 34px; line-height: 1.1; margin: 8px 0; }
+ .metric p, .gate p, .gate footer, .issues span, .evidence span { color: var(--muted); font-size: 14px; overflow-wrap: anywhere; }
+ .gate { display: flex; flex-direction: column; gap: 10px; }
+ .gate.pass { border-top: 4px solid var(--pass); }
+ .gate.warn { border-top: 4px solid var(--warn); }
+ .gate.block { border-top: 4px solid var(--block); }
+ .gate footer { border-top: 1px solid var(--line); padding-top: 10px; }
+ a { color: var(--ink); }
+ .twocol {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr) minmax(0, 1fr);
+ gap: 22px;
+ align-items: start;
+ }
+ .panel {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ }
+ .panel p { color: var(--muted); }
+ .kv-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 10px 12px;
+ margin: 2px 0 0;
+ }
+ .kv-grid div {
+ min-width: 0;
+ padding: 9px 0 0;
+ border-top: 1px solid var(--line);
+ }
+ .kv-grid dt {
+ color: var(--muted);
+ font-size: 13px;
+ }
+ .kv-grid dd {
+ margin: 2px 0 0;
+ color: var(--ink);
+ font-size: 15px;
+ overflow-wrap: anywhere;
+ }
+ .issues, .evidence {
+ list-style: none;
+ padding: 0;
+ margin: 0;
+ display: grid;
+ gap: 12px;
+ }
+ .issues li, .evidence li {
+ border-left: 3px solid var(--line);
+ padding-left: 12px;
+ display: grid;
+ gap: 3px;
+ }
+ .muted { color: var(--muted); }
+ .actions-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 16px;
+ }
+ .annotations-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 16px;
+ }
+ .action-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 10px;
+ min-width: 0;
+ }
+ .action-card.warn { border-left: 4px solid var(--warn); }
+ .action-card.block { border-left: 4px solid var(--block); }
+ .action-card.with-evidence {
+ grid-column: 1 / -1;
+ }
+ .action-evidence-panel {
+ display: grid;
+ gap: 12px;
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ margin-top: 2px;
+ }
+ .action-evidence-panel h4 {
+ margin: 0;
+ color: var(--ink);
+ font-size: 16px;
+ font-weight: 500;
+ }
+ .action-evidence-panel p {
+ color: var(--muted);
+ font-size: 14px;
+ }
+ .action-evidence-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 14px;
+ }
+ .action-evidence-item {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 14px;
+ display: grid;
+ gap: 10px;
+ min-width: 0;
+ background: var(--soft);
+ }
+ .action-evidence-item.pending { border-left: 4px solid var(--warn); }
+ .action-evidence-item.accepted { border-left: 4px solid var(--pass); }
+ .action-evidence-item h4,
+ .action-evidence-item h5 {
+ margin: 0;
+ color: var(--ink);
+ font-weight: 500;
+ }
+ .action-evidence-item h4 { font-size: 16px; }
+ .action-evidence-item h5 { font-size: 14px; }
+ .action-evidence-item section {
+ border-top: 1px solid var(--line);
+ padding-top: 10px;
+ display: grid;
+ gap: 8px;
+ }
+ .action-command-details,
+ .action-phase-details,
+ .action-runbook-details,
+ .action-collection-details {
+ border-top: 1px solid var(--line);
+ padding-top: 10px;
+ display: grid;
+ gap: 8px;
+ }
+ .action-command-details summary,
+ .action-phase-details summary,
+ .action-runbook-details summary,
+ .action-collection-details summary {
+ cursor: pointer;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .action-command-details[open] summary,
+ .action-phase-details[open] summary,
+ .action-runbook-details[open] summary,
+ .action-collection-details[open] summary {
+ margin-bottom: 8px;
+ }
+ .action-collection-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 12px;
+ }
+ .action-collection-grid section {
+ border-top: 1px solid var(--line);
+ padding-top: 8px;
+ display: grid;
+ gap: 6px;
+ }
+ .action-collection-grid ul,
+ .action-collection-grid ol {
+ margin: 0;
+ padding-left: 18px;
+ display: grid;
+ gap: 5px;
+ }
+ .action-collection-grid li {
+ color: var(--muted);
+ font-size: 13px;
+ overflow-wrap: anywhere;
+ }
+ .action-artifact-roles {
+ padding: 0;
+ margin: 0;
+ list-style: none;
+ display: grid;
+ gap: 8px;
+ }
+ .action-artifact-roles li {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ background: #fff;
+ padding: 10px;
+ display: grid;
+ gap: 4px;
+ min-width: 0;
+ }
+ .action-artifact-roles strong {
+ color: var(--ink);
+ font-weight: 500;
+ overflow-wrap: anywhere;
+ }
+ .action-artifact-roles span,
+ .action-artifact-roles code,
+ .action-artifact-roles small {
+ color: var(--muted);
+ font-size: 12px;
+ overflow-wrap: anywhere;
+ white-space: normal;
+ }
+ .action-evidence-item dl {
+ display: grid;
+ grid-template-columns: 76px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .action-evidence-item dt {
+ color: var(--ink);
+ font-size: 14px;
+ }
+ .action-evidence-item dd {
+ margin: 0;
+ min-width: 0;
+ }
+ .action-evidence-checks,
+ .action-phase-list,
+ .action-command-list,
+ .action-repair-list,
+ .action-runbook-list {
+ margin: 0;
+ display: grid;
+ gap: 8px;
+ }
+ .action-evidence-checks,
+ .action-phase-list,
+ .action-repair-list,
+ .action-command-list {
+ padding: 0;
+ list-style: none;
+ }
+ .action-runbook-list {
+ padding-left: 20px;
+ }
+ .action-evidence-check,
+ .action-phase-row,
+ .action-repair-row,
+ .action-command-list li {
+ display: grid;
+ gap: 3px;
+ min-width: 0;
+ border-top: 1px solid var(--line);
+ padding-top: 8px;
+ }
+ .action-evidence-check.blocked { border-left: 3px solid var(--warn); padding-left: 8px; }
+ .action-evidence-check.pass { border-left: 3px solid var(--pass); padding-left: 8px; }
+ .action-phase-row.blocked { border-left: 3px solid var(--warn); padding-left: 8px; }
+ .action-phase-row.ready { border-left: 3px solid var(--pass); padding-left: 8px; }
+ .action-repair-row.blocked { border-left: 3px solid var(--warn); padding-left: 8px; }
+ .action-repair-row.ready { border-left: 3px solid var(--pass); padding-left: 8px; }
+ .action-evidence-check span,
+ .action-phase-row span,
+ .action-repair-row span,
+ .action-command-list span {
+ color: var(--ink);
+ font-size: 13px;
+ }
+ .action-phase-row strong,
+ .action-repair-row strong {
+ color: var(--ink);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .action-phase-row em,
+ .action-repair-row em {
+ color: var(--muted);
+ font-size: 12px;
+ font-style: normal;
+ overflow-wrap: anywhere;
+ }
+ .action-evidence-check code,
+ .action-evidence-check small,
+ .action-phase-row code,
+ .action-phase-row small,
+ .action-repair-row code,
+ .action-repair-row small,
+ .action-command-list code,
+ .action-runbook-list li {
+ color: var(--muted);
+ font-size: 13px;
+ overflow-wrap: anywhere;
+ }
+ .action-command-list code {
+ display: block;
+ white-space: normal;
+ }
+ .annotation-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 10px;
+ min-width: 0;
+ }
+ .annotation-card.warning { border-left: 4px solid var(--warn); }
+ .annotation-card.blocker { border-left: 4px solid var(--block); }
+ .annotation-card.resolved { opacity: .72; }
+ .world-evidence-grid {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 16px;
+ margin-top: 16px;
+ }
+ .world-evidence-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 12px;
+ min-width: 0;
+ }
+ .world-evidence-card.pending { border-left: 4px solid var(--warn); }
+ .world-evidence-card.accepted { border-left: 4px solid var(--pass); }
+ .world-evidence-card span,
+ .world-evidence-card p,
+ .world-evidence-card dd,
+ .world-evidence-card li {
+ color: var(--muted);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .world-evidence-card dl {
+ display: grid;
+ grid-template-columns: 86px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .world-evidence-card dt { color: var(--ink); font-size: 14px; }
+ .world-evidence-card dd { margin: 0; }
+ .world-evidence-columns {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 12px;
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .world-evidence-columns h4 {
+ margin: 0 0 6px;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .world-evidence-columns ul {
+ margin: 0;
+ padding-left: 18px;
+ }
+ .world-source-panel {
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .world-runbook-panel {
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .world-runbook-panel h4,
+ .world-source-panel h4 {
+ margin: 0 0 8px;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .world-runbook-panel ul {
+ margin: 0;
+ padding-left: 18px;
+ }
+ .world-source-checks {
+ list-style: none;
+ padding: 0;
+ margin: 0;
+ display: grid;
+ gap: 8px;
+ }
+ .world-source-check {
+ display: grid;
+ grid-template-columns: minmax(0, .7fr) minmax(0, .6fr) minmax(0, 1fr);
+ gap: 10px;
+ align-items: start;
+ min-width: 0;
+ padding: 9px 0 0;
+ border-top: 1px solid var(--line);
+ }
+ .world-source-check span {
+ color: var(--ink);
+ font-size: 14px;
+ }
+ .world-source-check code,
+ .world-source-check small {
+ color: var(--muted);
+ font-size: 13px;
+ overflow-wrap: anywhere;
+ }
+ .world-source-check.pass { border-left: 3px solid var(--pass); padding-left: 8px; }
+ .world-source-check.blocked { border-left: 3px solid var(--warn); padding-left: 8px; }
+ .output-review-grid {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 16px;
+ margin-top: 16px;
+ }
+ .output-review-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 12px;
+ min-width: 0;
+ }
+ .output-review-card.awaiting-decision,
+ .output-review-card.fix-decision { border-left: 4px solid var(--warn); }
+ .output-review-card.adjudicated { border-left: 4px solid var(--pass); }
+ .output-review-card span,
+ .output-review-card p,
+ .output-review-card dd,
+ .output-review-card li {
+ color: var(--muted);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .output-review-card dl {
+ display: grid;
+ grid-template-columns: 86px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .output-review-card dt { color: var(--ink); font-size: 14px; }
+ .output-review-card dd { margin: 0; min-width: 0; }
+ .output-review-steps {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 12px;
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .output-review-steps h4 {
+ margin: 0 0 6px;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .output-review-steps ul {
+ margin: 0;
+ padding-left: 18px;
+ }
+ .output-review-commands,
+ .output-review-fields {
+ list-style: none;
+ padding: 0;
+ display: grid;
+ gap: 8px;
+ }
+ .output-review-commands li,
+ .output-review-fields li {
+ display: grid;
+ gap: 2px;
+ min-width: 0;
+ padding: 8px 0 0;
+ border-top: 1px solid var(--line);
+ }
+ .output-review-commands code {
+ overflow-wrap: anywhere;
+ white-space: normal;
+ display: block;
+ }
+ .world-intake-grid {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 16px;
+ margin-top: 16px;
+ }
+ .world-intake-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 12px;
+ min-width: 0;
+ }
+ .world-intake-card.awaiting-submission,
+ .world-intake-card.fix-submission,
+ .world-intake-card.fix-template { border-left: 4px solid var(--warn); }
+ .world-intake-card.ready-for-ledger-review,
+ .world-intake-card.accepted { border-left: 4px solid var(--pass); }
+ .world-intake-card span,
+ .world-intake-card p,
+ .world-intake-card dd,
+ .world-intake-card li {
+ color: var(--muted);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .world-intake-card dl {
+ display: grid;
+ grid-template-columns: 86px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .world-intake-card dt { color: var(--ink); font-size: 14px; }
+ .world-intake-card dd { margin: 0; min-width: 0; }
+ .world-intake-steps {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 12px;
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .world-intake-steps h4 {
+ margin: 0 0 6px;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .world-intake-steps ul {
+ margin: 0;
+ padding-left: 18px;
+ }
+ .world-intake-commands {
+ list-style: none;
+ padding: 0;
+ display: grid;
+ gap: 8px;
+ }
+ .world-intake-commands li {
+ display: grid;
+ gap: 2px;
+ min-width: 0;
+ padding: 8px 0 0;
+ border-top: 1px solid var(--line);
+ }
+ .world-intake-commands code {
+ overflow-wrap: anywhere;
+ white-space: normal;
+ display: block;
+ }
+ .section-subtitle {
+ margin-top: 22px;
+ margin-bottom: 12px;
+ color: var(--ink);
+ }
+ .waiver-summary { margin-top: 16px; }
+ .waiver-candidate-grid {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 16px;
+ }
+ .waiver-card {
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ padding: 18px;
+ background: #fff;
+ display: grid;
+ gap: 12px;
+ min-width: 0;
+ }
+ .waiver-card.allowed { border-left: 4px solid var(--warn); }
+ .waiver-card.blocked { border-left: 4px solid var(--block); }
+ .waiver-card span,
+ .waiver-card p,
+ .waiver-card dd,
+ .waiver-card li {
+ color: var(--muted);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .waiver-card dl {
+ display: grid;
+ grid-template-columns: 86px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .waiver-card dt {
+ color: var(--ink);
+ font-size: 14px;
+ }
+ .waiver-card dd { margin: 0; min-width: 0; }
+ .waiver-review {
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .waiver-review h4 {
+ margin: 0 0 6px;
+ color: var(--ink);
+ font-size: 14px;
+ font-weight: 500;
+ }
+ .waiver-review ul {
+ margin: 0;
+ padding-left: 18px;
+ }
+ .waiver-command {
+ display: grid;
+ gap: 4px;
+ border-top: 1px solid var(--line);
+ padding-top: 12px;
+ }
+ .waiver-command span {
+ color: var(--ink);
+ font-size: 14px;
+ }
+ .waiver-command code {
+ display: block;
+ max-width: 100%;
+ overflow-wrap: anywhere;
+ white-space: normal;
+ }
+ .action-card span,
+ .annotation-card span,
+ .action-card small,
+ .annotation-card small,
+ .action-card footer,
+ .annotation-card footer,
+ .action-card dd {
+ color: var(--muted);
+ font-size: 14px;
+ overflow-wrap: anywhere;
+ }
+ .annotation-card dd { color: var(--muted); font-size: 14px; overflow-wrap: anywhere; }
+ .action-card dl, .annotation-card dl {
+ display: grid;
+ grid-template-columns: 80px minmax(0, 1fr);
+ gap: 6px 10px;
+ margin: 0;
+ }
+ .action-card dt, .annotation-card dt { color: var(--ink); font-size: 14px; }
+ .action-card dd, .annotation-card dd { margin: 0; }
+ .source-ref-list {
+ list-style: none;
+ padding: 0;
+ margin: 0;
+ display: grid;
+ gap: 6px;
+ }
+ .source-ref-list li {
+ display: grid;
+ gap: 2px;
+ min-width: 0;
+ padding: 8px 0 0;
+ border-top: 1px solid var(--line);
+ }
+ .source-ref-list a,
+ .source-ref-list span {
+ overflow-wrap: anywhere;
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 13px;
+ }
+ .source-ref-list small {
+ font-size: 12px;
+ color: var(--muted);
+ }
+ .source-ref-list blockquote {
+ margin: 2px 0 0;
+ padding: 7px 9px;
+ border-left: 3px solid var(--line);
+ background: var(--soft);
+ color: var(--muted);
+ font-size: 12px;
+ overflow-wrap: anywhere;
+ }
+ code {
+ font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
+ font-size: 13px;
+ }
+ @media (max-width: 980px) {
+ .metrics, .gates, .twocol, .actions-grid, .annotations-grid, .action-evidence-grid, .action-evidence-item dl, .action-collection-grid, .output-review-grid, .output-review-steps, .world-evidence-grid, .world-evidence-columns, .world-source-check, .world-intake-grid, .world-intake-steps, .waiver-candidate-grid, .waiver-card dl, .kv-grid { grid-template-columns: 1fr; }
+ main { padding: 32px 18px 60px; }
+ nav { justify-content: flex-start; overflow-x: auto; flex-wrap: nowrap; }
+ nav a { flex: 0 0 auto; }
+ }
diff --git a/assets/review-viewer.css b/assets/review-viewer.css
new file mode 100644
index 00000000..11abdb53
--- /dev/null
+++ b/assets/review-viewer.css
@@ -0,0 +1,176 @@
+:root {
+ --text: #111111;
+ --muted: #666666;
+ --line: #e8e8e8;
+ --soft: #f6f6f4;
+ --white: #ffffff;
+}
+* { box-sizing: border-box; }
+body {
+ margin: 0;
+ background: var(--white);
+ color: var(--text);
+ font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
+ line-height: 1.6;
+}
+.page {
+ max-width: 1120px;
+ margin: 0 auto;
+ padding: 48px 32px 72px;
+}
+.hero {
+ padding-bottom: 28px;
+ border-bottom: 1px solid var(--line);
+ margin-bottom: 28px;
+}
+h1, h2, h3 {
+ margin: 0 0 12px;
+ letter-spacing: -0.02em;
+ font-weight: 600;
+}
+h1 { font-size: 40px; line-height: 1.08; }
+h2 { font-size: 22px; margin-top: 34px; }
+h3 { font-size: 16px; }
+p, li, span { font-size: 15px; }
+.lede {
+ max-width: 860px;
+ font-size: 18px;
+ color: var(--muted);
+ margin: 0 0 18px;
+}
+.meta {
+ display: flex;
+ gap: 10px;
+ flex-wrap: wrap;
+ margin-top: 16px;
+}
+.meta span {
+ border: 1px solid var(--line);
+ padding: 6px 10px;
+ background: var(--soft);
+}
+.arch-grid {
+ display: grid;
+ grid-template-columns: repeat(5, minmax(0, 1fr));
+ gap: 12px;
+ margin-top: 16px;
+}
+.arch-step, .panel, .direction-card, .baseline-box {
+ border: 1px solid var(--line);
+ background: var(--white);
+}
+.arch-step {
+ padding: 14px;
+ min-height: 132px;
+}
+.step-label {
+ font-size: 12px;
+ text-transform: uppercase;
+ letter-spacing: 0.08em;
+ color: var(--muted);
+ margin-bottom: 10px;
+}
+.step-detail {
+ font-size: 14px;
+}
+.grid {
+ display: grid;
+ grid-template-columns: 1.1fr 0.9fr;
+ gap: 18px;
+ margin-top: 16px;
+}
+.panel {
+ padding: 18px;
+}
+.panel ul {
+ margin: 0;
+ padding-left: 18px;
+}
+.direction-grid {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 16px;
+ margin-top: 16px;
+}
+.variant-grid {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 16px;
+ margin-top: 16px;
+}
+.direction-card {
+ padding: 18px;
+}
+.direction-card ul {
+ margin: 12px 0;
+ padding-left: 18px;
+}
+.minor {
+ color: var(--muted);
+ font-size: 13px;
+}
+.variant-card {
+ border: 1px solid var(--line);
+ background: var(--white);
+ padding: 18px;
+}
+.variant-head {
+ display: flex;
+ justify-content: space-between;
+ gap: 12px;
+ align-items: baseline;
+}
+.variant-head span {
+ color: var(--muted);
+ font-size: 12px;
+ text-transform: uppercase;
+ letter-spacing: 0.06em;
+}
+.variant-description {
+ margin: 14px 0;
+ padding-left: 14px;
+ border-left: 2px solid var(--line);
+ color: var(--text);
+}
+.variant-metrics {
+ display: flex;
+ flex-wrap: wrap;
+ gap: 8px;
+ margin-bottom: 14px;
+}
+.variant-metrics span {
+ border: 1px solid var(--line);
+ background: var(--soft);
+ padding: 6px 10px;
+ font-size: 12px;
+}
+.variant-cues p {
+ margin: 8px 0 6px;
+}
+.variant-cues ul {
+ margin: 0 0 12px;
+ padding-left: 18px;
+}
+table {
+ width: 100%;
+ border-collapse: collapse;
+ margin-top: 14px;
+ font-size: 14px;
+}
+th, td {
+ border-top: 1px solid var(--line);
+ text-align: left;
+ padding: 10px 8px;
+ vertical-align: top;
+}
+th {
+ font-size: 12px;
+ text-transform: uppercase;
+ letter-spacing: 0.06em;
+ color: var(--muted);
+}
+@media (max-width: 1000px) {
+ .arch-grid, .direction-grid, .variant-grid, .grid {
+ grid-template-columns: 1fr;
+ }
+}
diff --git a/assets/skill-overview.css b/assets/skill-overview.css
new file mode 100644
index 00000000..b626f5e1
--- /dev/null
+++ b/assets/skill-overview.css
@@ -0,0 +1,735 @@
+:root {
+ --paper: #ffffff;
+ --wash: #f8fafc;
+ --wash-strong: #f2f5f8;
+ --line: #e6e0d4;
+ --line-soft: #eee9df;
+ --brand: #1B365D;
+ --brand-soft: #EEF3F8;
+ --brand-mid: #315982;
+ --ink: #151515;
+ --text: #2f2d29;
+ --muted: #68625a;
+ --faint: #8b857b;
+ --success: #2f6f5e;
+ --warn: #8a5a19;
+ --serif: "TsangerJinKai02", "Source Han Serif SC", "Noto Serif CJK SC", "Songti SC", "STSong", Charter, Georgia, serif;
+ --mono: "JetBrains Mono", "SF Mono", ui-monospace, Menlo, Consolas, monospace;
+ --shadow-soft: 0 1px 2px rgba(27, 54, 93, 0.06), 0 16px 44px rgba(27, 54, 93, 0.08);
+ }
+ * { box-sizing: border-box; }
+ html { scroll-behavior: smooth; }
+ body {
+ margin: 0;
+ background: #ffffff;
+ color: var(--ink);
+ font-family: var(--serif);
+ line-height: 1.62;
+ letter-spacing: 0;
+ -webkit-font-smoothing: antialiased;
+ -moz-osx-font-smoothing: grayscale;
+ text-rendering: optimizeLegibility;
+ }
+ body[data-report-lang="zh-CN"] [data-lang="en"],
+ body[data-report-lang="en"] [data-lang="zh-CN"] { display: none !important; }
+ .skip-link {
+ position: fixed;
+ left: 18px;
+ top: 10px;
+ z-index: 40;
+ transform: translateY(-140%);
+ padding: 8px 12px;
+ border-radius: 8px;
+ background: var(--brand);
+ color: #fff;
+ text-decoration: none;
+ transition: transform 160ms cubic-bezier(0.16, 1, 0.3, 1);
+ }
+ .skip-link:focus-visible { transform: translateY(0); outline: 2px solid var(--brand-mid); outline-offset: 2px; }
+ .topbar {
+ position: sticky;
+ top: 0;
+ z-index: 20;
+ background: rgba(255, 255, 255, 0.96);
+ border-bottom: 1px solid var(--line);
+ }
+ .progress-track {
+ height: 2px;
+ background: transparent;
+ }
+ .progress-bar {
+ display: block;
+ width: 100%;
+ height: 100%;
+ background: var(--brand);
+ transition: transform 160ms cubic-bezier(0.16, 1, 0.3, 1);
+ transform-origin: left center;
+ transform: scaleX(0);
+ }
+ .topbar-inner {
+ max-width: 1240px;
+ margin: 0 auto;
+ padding: 9px 28px;
+ display: grid;
+ grid-template-columns: minmax(0, 1fr) auto;
+ gap: 18px;
+ align-items: center;
+ }
+ .nav-shell {
+ display: grid;
+ grid-template-columns: auto minmax(0, 1fr);
+ gap: 18px;
+ align-items: center;
+ min-width: 0;
+ }
+ .report-mark {
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 11px;
+ line-height: 1;
+ text-transform: uppercase;
+ white-space: nowrap;
+ }
+ .report-nav {
+ display: flex;
+ gap: 4px;
+ overflow-x: auto;
+ scrollbar-width: none;
+ min-width: 0;
+ }
+ .report-nav::-webkit-scrollbar { display: none; }
+ .report-nav a {
+ flex: 0 0 auto;
+ min-width: 76px;
+ min-height: 40px;
+ display: inline-flex;
+ align-items: center;
+ justify-content: center;
+ padding: 0 10px;
+ color: var(--brand);
+ text-decoration: none;
+ text-align: center;
+ font-size: 13px;
+ border-radius: 8px;
+ white-space: nowrap;
+ transition-property: background-color, color, transform;
+ transition-duration: 160ms;
+ transition-timing-function: cubic-bezier(0.16, 1, 0.3, 1);
+ }
+ @media (hover: hover) {
+ .report-nav a:hover { background: var(--brand-soft); }
+ }
+ .report-nav a:focus-visible {
+ outline: 2px solid var(--brand-mid);
+ outline-offset: 2px;
+ background: var(--brand-soft);
+ }
+ .report-nav a[aria-current="true"] {
+ background: var(--brand);
+ color: #ffffff;
+ }
+ .report-nav a[aria-current="true"] span { color: #ffffff; }
+ .language-switch {
+ display: inline-flex;
+ gap: 3px;
+ padding: 3px;
+ border: 1px solid var(--line);
+ border-radius: 8px;
+ background: var(--paper);
+ box-shadow: 0 1px 2px rgba(27, 54, 93, 0.05);
+ }
+ .language-switch button {
+ appearance: none;
+ min-width: 42px;
+ min-height: 34px;
+ border: 0;
+ border-radius: 6px;
+ padding: 0 10px;
+ background: transparent;
+ color: var(--muted);
+ font: inherit;
+ font-size: 13px;
+ cursor: pointer;
+ transition-property: background-color, color, transform;
+ transition-duration: 160ms;
+ transition-timing-function: cubic-bezier(0.16, 1, 0.3, 1);
+ }
+ .language-switch button:active { transform: scale(0.96); }
+ .language-switch button:focus-visible { outline: 2px solid var(--brand-mid); outline-offset: 2px; }
+ .language-switch button[aria-pressed="true"] {
+ background: var(--brand-soft);
+ color: var(--brand);
+ }
+ .wrap {
+ max-width: 1240px;
+ margin: 0 auto;
+ padding: 58px 28px 92px;
+ }
+ .hero {
+ padding: 16px 0 34px;
+ }
+ .hero-grid {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr) 360px;
+ gap: 44px;
+ align-items: end;
+ }
+ .eyebrow, .report-mark {
+ letter-spacing: 0;
+ }
+ .eyebrow {
+ margin: 0 0 12px;
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 12px;
+ text-transform: uppercase;
+ }
+ .slug {
+ margin: 0 0 18px;
+ color: var(--faint);
+ font-family: var(--mono);
+ font-size: 13px;
+ overflow-wrap: anywhere;
+ }
+ h1, h2, h3 {
+ margin: 0;
+ font-weight: 500;
+ letter-spacing: 0;
+ text-wrap: balance;
+ }
+ h1 {
+ max-width: 8em;
+ color: var(--ink);
+ font-size: 4rem;
+ line-height: 1.02;
+ }
+ .lead {
+ max-width: 760px;
+ margin: 20px 0 0;
+ color: var(--text);
+ font-size: 1.08rem;
+ text-wrap: pretty;
+ }
+ .hero-meta, .badges {
+ display: flex;
+ flex-wrap: wrap;
+ gap: 8px;
+ margin-top: 20px;
+ }
+ .hero-meta span, .badges span, .tag {
+ display: inline-flex;
+ align-items: center;
+ min-height: 30px;
+ padding: 5px 10px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 13px;
+ font-variant-numeric: tabular-nums;
+ }
+ .hero-card {
+ position: relative;
+ padding: 22px 22px 24px;
+ border-radius: 8px;
+ background: linear-gradient(180deg, #ffffff 0%, #fbfaf7 100%);
+ box-shadow: var(--shadow-soft);
+ border: 1px solid var(--line-soft);
+ }
+ .hero-card::before {
+ content: "";
+ position: absolute;
+ top: 18px;
+ right: 18px;
+ width: 38px;
+ height: 2px;
+ background: var(--brand);
+ }
+ .score-strip {
+ display: grid;
+ grid-template-columns: repeat(4, minmax(0, 1fr));
+ gap: 10px;
+ margin-top: 32px;
+ padding-top: 20px;
+ border-top: 1px solid var(--line);
+ }
+ .score-chip {
+ padding: 14px 14px 12px;
+ border-radius: 8px;
+ background: var(--wash);
+ border: 1px solid var(--line-soft);
+ }
+ .score-chip span {
+ display: block;
+ color: var(--muted);
+ font-size: 12px;
+ }
+ .score-chip strong {
+ display: block;
+ margin: 5px 0 8px;
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 1.65rem;
+ font-weight: 500;
+ line-height: 1;
+ font-variant-numeric: tabular-nums;
+ }
+ .score-chip i {
+ display: block;
+ height: 3px;
+ border-radius: 999px;
+ background: linear-gradient(90deg, var(--brand) var(--score), #dfe6ee var(--score));
+ }
+ .score-chip small {
+ display: block;
+ margin-top: 8px;
+ color: var(--muted);
+ font-size: 12px;
+ line-height: 1.45;
+ }
+ section {
+ scroll-margin-top: 78px;
+ padding-top: 44px;
+ margin-top: 44px;
+ border-top: 1px solid var(--line);
+ }
+ section.hero {
+ scroll-margin-top: 0;
+ padding-top: 16px;
+ margin-top: 0;
+ border-top: 0;
+ }
+ .section-head {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 22px;
+ align-items: start;
+ }
+ h2 {
+ color: var(--ink);
+ font-size: 1.5rem;
+ line-height: 1.18;
+ }
+ h2::before {
+ content: "";
+ display: block;
+ width: 32px;
+ height: 2px;
+ margin-bottom: 12px;
+ background: var(--brand);
+ }
+ h3 {
+ color: var(--ink);
+ font-size: 1.02rem;
+ line-height: 1.3;
+ margin-bottom: 10px;
+ }
+ .section-head > div:first-child p {
+ margin: 12px 0 0;
+ color: var(--muted);
+ max-width: 68ch;
+ text-wrap: pretty;
+ }
+ .section-body {
+ margin-top: 24px;
+ }
+ .two-col, .metric-grid, .chart-grid {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 16px;
+ align-items: stretch;
+ }
+ .quality-panels {
+ margin-top: 16px;
+ }
+ .metrics-flow {
+ display: grid;
+ gap: 30px;
+ }
+ .metrics-primary {
+ display: grid;
+ grid-template-columns: minmax(360px, 0.92fr) minmax(0, 1.08fr);
+ gap: 22px;
+ align-items: start;
+ }
+ .metrics-primary .chart-figure {
+ min-height: 0;
+ display: grid;
+ align-content: center;
+ }
+ .metrics-primary .chart-figure svg {
+ max-height: 520px;
+ }
+ .metrics-note {
+ display: grid;
+ align-content: start;
+ gap: 14px;
+ }
+ .metrics-note p {
+ margin: 0;
+ max-width: none;
+ color: var(--text);
+ }
+ .metric-summary-list {
+ margin: 0;
+ padding: 0;
+ list-style: none;
+ display: grid;
+ gap: 10px;
+ }
+ .metric-summary-list li {
+ display: grid;
+ grid-template-columns: auto minmax(0, 1fr) auto;
+ gap: 8px 10px;
+ align-items: baseline;
+ padding: 10px 0;
+ border-bottom: 1px solid var(--line-soft);
+ }
+ .metric-summary-list li:last-child { border-bottom: 0; }
+ .metric-summary-list b {
+ color: var(--ink);
+ font-weight: 500;
+ }
+ .metric-summary-list em {
+ color: var(--brand);
+ font-family: var(--mono);
+ font-style: normal;
+ font-variant-numeric: tabular-nums;
+ }
+ .metric-summary-list small {
+ grid-column: 2 / 4;
+ color: var(--muted);
+ line-height: 1.5;
+ overflow-wrap: break-word;
+ }
+ .metric-detail-section {
+ display: grid;
+ gap: 14px;
+ }
+ .detail-section-kicker {
+ display: inline-flex;
+ width: fit-content;
+ min-height: 28px;
+ align-items: center;
+ padding: 3px 8px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 12px;
+ }
+ .metric-status {
+ display: inline-flex;
+ align-items: center;
+ min-height: 24px;
+ padding: 2px 7px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 12px;
+ }
+ .metric-detail-grid {
+ grid-template-columns: repeat(2, minmax(min(100%, 420px), 1fr));
+ align-items: stretch;
+ }
+ .list, .compact-list, .step-list {
+ margin: 0;
+ padding-left: 1.15em;
+ display: grid;
+ gap: 8px;
+ }
+ .list li::marker, .compact-list li::marker, .step-list li::marker { color: var(--brand); }
+ .compact-list {
+ gap: 6px;
+ font-size: 13.5px;
+ color: var(--muted);
+ line-height: 1.55;
+ }
+ .compact-list li, .list li, .step-list li {
+ min-width: 0;
+ overflow-wrap: break-word;
+ }
+ .panel, .metric-card, .roadmap-item {
+ background: #ffffff;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ padding: 22px;
+ box-shadow: 0 1px 2px rgba(27, 54, 93, 0.04);
+ min-width: 0;
+ }
+ .metric-card {
+ display: grid;
+ grid-template-columns: minmax(112px, 0.28fr) minmax(0, 1fr);
+ gap: 18px;
+ align-content: start;
+ min-height: 0;
+ }
+ .metric-card-head {
+ display: grid;
+ gap: 10px;
+ align-content: start;
+ }
+ .metric-card strong {
+ display: block;
+ margin: 0;
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 2rem;
+ font-weight: 500;
+ line-height: 1;
+ font-variant-numeric: tabular-nums;
+ }
+ .metric-label {
+ color: var(--muted);
+ font-size: 13px;
+ line-height: 1.45;
+ overflow-wrap: break-word;
+ }
+ .metric-card-body {
+ min-width: 0;
+ }
+ .metric-card-body .compact-list {
+ gap: 7px;
+ max-width: 76ch;
+ }
+ .metric-card-body .compact-list li {
+ overflow-wrap: break-word;
+ word-break: normal;
+ }
+ .chart-figure {
+ margin: 0;
+ padding: 18px;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ background: #ffffff;
+ box-shadow: 0 1px 2px rgba(27, 54, 93, 0.04);
+ }
+ .chart-figure svg {
+ width: 100%;
+ height: auto;
+ display: block;
+ overflow: visible;
+ }
+ .chart-figure text {
+ fill: var(--ink);
+ font-family: var(--serif);
+ font-size: 13px;
+ }
+ .chart-title {
+ fill: var(--brand);
+ font-size: 16px;
+ font-weight: 500;
+ }
+ .chart-line {
+ fill: none;
+ stroke: var(--brand);
+ stroke-width: 2;
+ }
+ figcaption {
+ margin-top: 12px;
+ padding-top: 10px;
+ border-top: 1px solid var(--line-soft);
+ color: var(--muted);
+ font-size: 13px;
+ line-height: 1.5;
+ text-wrap: pretty;
+ }
+ table {
+ width: 100%;
+ border-collapse: collapse;
+ font-size: 14px;
+ }
+ th, td {
+ padding: 12px 10px;
+ text-align: left;
+ border-bottom: 1px solid var(--line-soft);
+ vertical-align: top;
+ }
+ th {
+ color: var(--brand);
+ font-weight: 500;
+ font-size: 12px;
+ }
+ td:first-child, th:first-child { width: 96px; }
+ .world-readiness {
+ margin-top: 16px;
+ display: grid;
+ gap: 16px;
+ }
+ .section-head > .world-readiness { grid-column: 1; }
+ .world-readiness-head {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr) auto;
+ gap: 16px;
+ align-items: start;
+ }
+ .world-readiness-head p {
+ margin-top: 8px;
+ max-width: 72ch;
+ }
+ .world-status {
+ display: inline-flex;
+ align-items: center;
+ min-height: 30px;
+ padding: 4px 10px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 13px;
+ white-space: nowrap;
+ }
+ .evidence-kpis {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 10px;
+ }
+ .evidence-kpi {
+ padding: 14px;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ background: var(--wash);
+ min-width: 0;
+ }
+ .evidence-kpi span {
+ display: block;
+ color: var(--muted);
+ font-size: 12px;
+ }
+ .evidence-kpi strong {
+ display: block;
+ margin-top: 4px;
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 1.5rem;
+ font-weight: 500;
+ line-height: 1;
+ font-variant-numeric: tabular-nums;
+ }
+ .evidence-kpi small {
+ display: block;
+ margin-top: 8px;
+ color: var(--muted);
+ line-height: 1.45;
+ overflow-wrap: break-word;
+ }
+ .evidence-list {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 10px;
+ }
+ .evidence-item {
+ padding: 14px;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ background: #ffffff;
+ min-width: 0;
+ }
+ .evidence-item span {
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 11px;
+ text-transform: uppercase;
+ }
+ .evidence-item h4 {
+ margin: 4px 0 8px;
+ color: var(--ink);
+ font-size: 1rem;
+ font-weight: 500;
+ line-height: 1.3;
+ }
+ .evidence-item h5 {
+ margin: 12px 0 6px;
+ color: var(--brand);
+ font-size: 12px;
+ font-weight: 500;
+ }
+ .evidence-item p {
+ margin: 0;
+ max-width: none;
+ color: var(--muted);
+ overflow-wrap: break-word;
+ }
+ .blocked-checks {
+ margin: 0;
+ padding-left: 1.05em;
+ color: var(--muted);
+ font-size: 13px;
+ line-height: 1.45;
+ }
+ .blocked-checks li {
+ overflow-wrap: break-word;
+ }
+ .blocked-checks li::marker {
+ color: var(--brand);
+ }
+ .blocked-checks-empty {
+ font-size: 13px;
+ }
+ .roadmap {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 16px;
+ }
+ .step {
+ display: inline-flex;
+ margin-bottom: 12px;
+ min-height: 28px;
+ align-items: center;
+ padding: 3px 8px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 12px;
+ }
+ .unlock {
+ margin-top: 14px;
+ color: var(--muted);
+ font-size: 13px;
+ }
+ @media (max-width: 980px) {
+ .topbar-inner {
+ grid-template-columns: minmax(0, 1fr) auto;
+ padding: 8px 16px;
+ }
+ .nav-shell { grid-template-columns: 1fr; gap: 6px; }
+ .report-mark { display: none; }
+ .report-nav a { min-width: 72px; }
+ .hero-grid, .section-head { grid-template-columns: 1fr; }
+ .section-head > .world-readiness { grid-column: auto; }
+ .hero-grid { gap: 24px; }
+ .score-strip { grid-template-columns: repeat(2, minmax(0, 1fr)); }
+ .metrics-primary, .metric-detail-grid, .two-col, .metric-grid, .chart-grid, .roadmap { grid-template-columns: 1fr; }
+ .evidence-list { grid-template-columns: 1fr; }
+ .metric-card { grid-template-columns: 1fr; }
+ h1 { font-size: 2.75rem; }
+ }
+ @media (max-width: 540px) {
+ .wrap { padding: 34px 16px 72px; }
+ .topbar-inner { gap: 10px; }
+ .report-nav {
+ display: flex;
+ gap: 2px;
+ }
+ .report-nav a {
+ min-width: 66px;
+ min-height: 38px;
+ padding: 0 8px;
+ font-size: 12px;
+ }
+ .language-switch button { min-width: 38px; min-height: 32px; padding: 0 8px; }
+ .hero { padding-top: 8px; }
+ h1 { max-width: 8em; font-size: 2.35rem; }
+ .lead { font-size: 1rem; }
+ .hero-meta span, .badges span { font-size: 12px; }
+ .score-strip { grid-template-columns: 1fr; }
+ section { margin-top: 34px; padding-top: 34px; }
+ .panel, .metric-card, .roadmap-item, .chart-figure, .hero-card { padding: 18px; }
+ .world-readiness-head, .evidence-kpis { grid-template-columns: 1fr; }
+ .metric-card { grid-template-columns: 1fr; gap: 10px; }
+ .metric-card strong { margin-bottom: 2px; }
+ .metric-summary-list li { grid-template-columns: auto minmax(0, 1fr) auto; }
+ table { font-size: 13px; }
+ th, td { padding: 10px 7px; }
+ }
+ @media (prefers-reduced-motion: reduce) {
+ html { scroll-behavior: auto; }
+ *, *::before, *::after { transition-duration: 0.01ms !important; }
+ }
diff --git a/assets/skill-overview.js b/assets/skill-overview.js
new file mode 100644
index 00000000..eaf9ef4f
--- /dev/null
+++ b/assets/skill-overview.js
@@ -0,0 +1,37 @@
+(function () {
+ var buttons = Array.prototype.slice.call(document.querySelectorAll("[data-set-lang]"));
+ var navLinks = Array.prototype.slice.call(document.querySelectorAll(".report-nav a"));
+ var sections = navLinks
+ .map(function (link) { return document.querySelector(link.getAttribute("href")); })
+ .filter(Boolean);
+ var progressBar = document.querySelector(".progress-bar");
+ function setLanguage(lang) {
+ document.body.setAttribute("data-report-lang", lang);
+ document.documentElement.setAttribute("lang", lang);
+ buttons.forEach(function (button) {
+ button.setAttribute("aria-pressed", button.getAttribute("data-set-lang") === lang ? "true" : "false");
+ });
+ }
+ function updateProgress() {
+ var scrollTop = window.scrollY || document.documentElement.scrollTop;
+ var height = Math.max(1, document.documentElement.scrollHeight - window.innerHeight);
+ var pct = Math.min(100, Math.max(0, (scrollTop / height) * 100));
+ if (progressBar) progressBar.style.transform = "scaleX(" + pct / 100 + ")";
+ var active = sections[0];
+ sections.forEach(function (section) {
+ if (section.getBoundingClientRect().top <= 110) active = section;
+ });
+ navLinks.forEach(function (link) {
+ link.setAttribute("aria-current", link.getAttribute("href") === "#" + active.id ? "true" : "false");
+ });
+ }
+ buttons.forEach(function (button) {
+ button.addEventListener("click", function () {
+ setLanguage(button.getAttribute("data-set-lang"));
+ });
+ });
+ window.addEventListener("scroll", updateProgress, { passive: true });
+ window.addEventListener("resize", updateProgress);
+ setLanguage("zh-CN");
+ updateProgress();
+ })();
diff --git a/docs/migration-v2.md b/docs/migration-v2.md
index 0b4a349b..7189997b 100644
--- a/docs/migration-v2.md
+++ b/docs/migration-v2.md
@@ -41,6 +41,8 @@ This document tracks the first migration path from Yao Meta Skill 1.x to the Ski
python3 scripts/build_skill_atlas.py --workspace-root . --output-dir skill_atlas --report-html reports/skill_atlas.html --report-json reports/skill_atlas.json
```
+ This also writes `skill_atlas/drift_signals.json` from aggregate `reports/adoption_drift_report.json` files. Do not migrate or publish raw `reports/telemetry_events.jsonl` logs.
+
5. Keep legacy trigger eval gates intact while output eval coverage grows.
6. Move adapter-specific logic toward future compiler commands after IR fields are stable.
diff --git a/evals/adaptation/user_signals.example.jsonl b/evals/adaptation/user_signals.example.jsonl
new file mode 100644
index 00000000..ed06eda9
--- /dev/null
+++ b/evals/adaptation/user_signals.example.jsonl
@@ -0,0 +1,8 @@
+{"text":"新生成的 Skill 报告默认使用中文简体，并在右上角提供英文切换。"}
+{"text":"HTML 报告需要双语能力，但默认内容应该保持中文简体。"}
+{"text":"报告排版采用白底 Kami 风格，图表、模块和导航都要清晰。"}
+{"text":"Skill 概览报告需要更多图表模块，避免右侧文本挤在一起。"}
+{"text":"自适应升级必须先生成提案，不能直接自动修改源文件。"}
+{"text":"用户偏好扫描必须由用户提供明确路径，不要默认扫描私人日志。"}
+{"text":"每次升级都需要测试、覆盖报告和可审计证据，推送前要跑 CI。"}
+{"text":"涉及 GitHub 推送时，要保留证据链，避免把计划当作完成证明。"}
diff --git a/evals/packaging_expectations.json b/evals/packaging_expectations.json
index 374c9059..f39792d4 100644
--- a/evals/packaging_expectations.json
+++ b/evals/packaging_expectations.json
@@ -1,5 +1,5 @@
{
- "required_targets": ["openai", "claude", "generic"],
+ "required_targets": ["openai", "claude", "generic", "vscode"],
"required_fields": [
"name",
"description",
@@ -30,5 +30,6 @@
],
"openai_required_files": ["targets/openai/adapter.json", "targets/openai/agents/openai.yaml"],
"claude_required_files": ["targets/claude/adapter.json", "targets/claude/README.md"],
- "generic_required_files": ["targets/generic/adapter.json"]
+ "generic_required_files": ["targets/generic/adapter.json"],
+ "vscode_required_files": ["targets/vscode/adapter.json", "targets/vscode/README.md"]
}
diff --git a/evidence/world_class/README.md b/evidence/world_class/README.md
new file mode 100644
index 00000000..a6013f8d
--- /dev/null
+++ b/evidence/world_class/README.md
@@ -0,0 +1,45 @@
+# World-Class Evidence Intake
+
+This directory defines the intake contract for external and human evidence required before `yao-meta-skill` can honestly claim public world-class completion.
+
+The templates in `templates/` are review aids only. They do not count as accepted evidence. Real submissions belong in `evidence/world_class/submissions/`, which is intentionally gitignored by default so provider metadata, reviewer identity, or client integration notes can be reviewed before anything is committed.
+
+Run:
+
+```bash
+SUBMISSIONS_DIR="${SUBMISSIONS_DIR:-evidence/world_class/submissions}"
+python3 scripts/yao.py world-class-preflight . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR"
+# Alternative: prefill artifact SHA-256 digests while keeping drafts template-only.
+python3 scripts/yao.py world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR" --prefill-artifacts
+python3 scripts/yao.py world-class-intake . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-submission-review . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-ledger . --submissions-dir "$SUBMISSIONS_DIR"
+python3 scripts/yao.py world-class-runbook . --submissions-dir "$SUBMISSIONS_DIR"
+```
+
+The intake validator checks:
+
+- the evidence key matches the current world-class ledger
+- the category and source type match the expected human or external evidence path
+- artifact references are declared
+- real submissions reference concrete files inside the skill directory and include a matching SHA-256 digest for each artifact
+- real submissions use the canonical `.json` filename expected by the ledger
+- credentials, secrets, raw user content, and raw provider prompts are explicitly excluded
+- raw prompt, output, transcript, message, credential, secret, token, and API-key fields are rejected even when nested
+- human adjudication packets must preserve reviewer identity, review date, A/B winner, confidence, and a required rationale before answer-key reveal can count
+- planned work, local command-only output, and metadata fallback are not claimed as completion evidence
+
+Run `world-class-preflight` before assigning external or human work. It checks local files, redacted environment readiness, human/external prerequisites, and source-evidence blockers without accepting evidence or printing secrets.
+
+The generated intake report also includes an `operator_checklist` for each pending evidence item. Use it to find the template path, target submission path, preparation command, validation command, required provenance, success checks, and privacy boundary before asking a reviewer or external operator to submit evidence.
+
+The submission kit command creates editable JSON drafts plus a local README for an external operator or human reviewer. Use `--prefill-artifacts` when you want the kit to insert SHA-256 digests for currently available local aggregate artifacts. Prefill is operator convenience only: those drafts still keep `template_only: true` and do not count as evidence until the real run or review exists, the packet is edited truthfully, every artifact ref points to a local aggregate evidence file with a matching `sha256`, and `world-class-intake` validates it.
+
+The submission kit separates artifact rows into `submission-ref` and `supporting-evidence`. `submission-ref` rows are the concrete paths expected in a real packet's `artifact_refs`; `supporting-evidence` rows help reviewers audit the packet and do not all need to be copied into the submission.
+
+The submission review command renders a read-only queue that compares valid packets with the source evidence checks and current ledger state. It is for reviewer triage only; it does not accept evidence or make the world-class claim true.
+
+The operator runbook is the step-by-step coordination cockpit for finishing the remaining real-world work. It includes a Coordination Plan that separates user-required provider, reviewer, native-client, and telemetry actions from assistant-run commands, plus a Release Gate that mirrors the ledger, claim guard, benchmark, Review Studio, evidence consistency, and final CI checks. The runbook is operational guidance only: planning rows, submission drafts, and release-gate rows keep `counts_as_completion: false`.
+
+Accepted intake means "ready for ledger review", not evidence that the final public claim is ready. The ledger remains the source of truth for `ready_to_claim_world_class`.
diff --git a/evidence/world_class/intake.schema.json b/evidence/world_class/intake.schema.json
new file mode 100644
index 00000000..8194e251
--- /dev/null
+++ b/evidence/world_class/intake.schema.json
@@ -0,0 +1,440 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://example.local/yao-meta-skill/world-class-evidence-intake.schema.json",
+ "title": "Yao World-Class Evidence Intake",
+ "description": "Real submissions must use the canonical .json filename and the runtime validator rejects nested raw prompt, output, transcript, message, credential, secret, token, and API-key fields.",
+ "type": "object",
+ "required": [
+ "schema_version",
+ "evidence_key",
+ "template_only",
+ "category",
+ "source_type",
+ "submitted_by",
+ "submitted_at",
+ "summary",
+ "artifact_refs",
+ "provenance",
+ "privacy",
+ "anti_overclaim",
+ "attestation"
+ ],
+ "properties": {
+ "schema_version": {
+ "const": "1.0"
+ },
+ "evidence_key": {
+ "enum": [
+ "provider-holdout",
+ "human-adjudication",
+ "native-permission-enforcement",
+ "native-client-telemetry"
+ ]
+ },
+ "template_only": {
+ "type": "boolean"
+ },
+ "category": {
+ "enum": [
+ "human",
+ "external"
+ ]
+ },
+ "source_type": {
+ "type": "string",
+ "minLength": 3
+ },
+ "submitted_by": {
+ "type": "string",
+ "minLength": 1
+ },
+ "submitted_at": {
+ "type": "string",
+ "minLength": 1
+ },
+ "summary": {
+ "type": "string",
+ "minLength": 1
+ },
+ "artifact_refs": {
+ "type": "array",
+ "minItems": 1,
+ "items": {
+ "type": "object",
+ "required": [
+ "path",
+ "kind",
+ "contains_raw_content"
+ ],
+ "properties": {
+ "path": {
+ "type": "string",
+ "minLength": 1
+ },
+ "kind": {
+ "type": "string",
+ "minLength": 1
+ },
+ "contains_raw_content": {
+ "type": "boolean"
+ },
+ "sha256": {
+ "type": "string",
+ "pattern": "^[0-9a-fA-F]{64}$"
+ },
+ "note": {
+ "type": "string"
+ }
+ },
+ "additionalProperties": true
+ }
+ },
+ "provenance": {
+ "type": "object",
+ "additionalProperties": true
+ },
+ "privacy": {
+ "type": "object",
+ "required": [
+ "raw_user_content_included",
+ "raw_provider_prompt_included",
+ "credentials_included",
+ "secrets_included"
+ ],
+ "properties": {
+ "raw_user_content_included": {
+ "type": "boolean"
+ },
+ "raw_provider_prompt_included": {
+ "type": "boolean"
+ },
+ "credentials_included": {
+ "type": "boolean"
+ },
+ "secrets_included": {
+ "type": "boolean"
+ },
+ "notes": {
+ "type": "string"
+ }
+ },
+ "additionalProperties": true
+ },
+ "anti_overclaim": {
+ "type": "object",
+ "required": [
+ "planned_work_counts_as_evidence",
+ "metadata_fallback_counts_as_native_enforcement",
+ "pending_review_counts_as_human_decision",
+ "local_command_runner_counts_as_provider_model"
+ ],
+ "properties": {
+ "planned_work_counts_as_evidence": {
+ "const": false
+ },
+ "metadata_fallback_counts_as_native_enforcement": {
+ "const": false
+ },
+ "pending_review_counts_as_human_decision": {
+ "const": false
+ },
+ "local_command_runner_counts_as_provider_model": {
+ "const": false
+ }
+ },
+ "additionalProperties": false
+ },
+ "attestation": {
+ "type": "object",
+ "required": [
+ "real_external_or_human_evidence",
+ "reviewer_or_operator_identity_present",
+ "artifact_refs_reviewed",
+ "privacy_contract_satisfied",
+ "ledger_reviewer_approved",
+ "ledger_reviewer",
+ "ledger_reviewed_at"
+ ],
+ "properties": {
+ "real_external_or_human_evidence": {
+ "type": "boolean"
+ },
+ "reviewer_or_operator_identity_present": {
+ "type": "boolean"
+ },
+ "artifact_refs_reviewed": {
+ "type": "boolean"
+ },
+ "privacy_contract_satisfied": {
+ "type": "boolean"
+ },
+ "ledger_reviewer_approved": {
+ "type": "boolean"
+ },
+ "ledger_reviewer": {
+ "type": "string"
+ },
+ "ledger_reviewed_at": {
+ "type": "string"
+ },
+ "notes": {
+ "type": "string"
+ }
+ },
+ "additionalProperties": true
+ }
+ },
+ "allOf": [
+ {
+ "if": {
+ "properties": {
+ "template_only": {
+ "const": false
+ }
+ },
+ "required": [
+ "template_only"
+ ]
+ },
+ "then": {
+ "properties": {
+ "submitted_at": {
+ "pattern": "^\\d{4}-\\d{2}-\\d{2}(?:T\\d{2}:\\d{2}:\\d{2}Z)?$"
+ },
+ "submitted_by": {
+ "not": {
+ "enum": [
+ "operator with provider credentials",
+ "human reviewer",
+ "target client or installer integrator",
+ "Browser/Chrome/IDE/provider client integrator"
+ ]
+ }
+ },
+ "artifact_refs": {
+ "items": {
+ "required": [
+ "path",
+ "kind",
+ "contains_raw_content",
+ "sha256"
+ ]
+ }
+ }
+ }
+ }
+ },
+ {
+ "if": {
+ "properties": {
+ "template_only": {
+ "const": false
+ },
+ "evidence_key": {
+ "const": "provider-holdout"
+ }
+ },
+ "required": [
+ "template_only",
+ "evidence_key"
+ ]
+ },
+ "then": {
+ "properties": {
+ "artifact_refs": {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/output_execution_runs.json"
+ },
+ "kind": {
+ "const": "aggregate-report"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ }
+ }
+ }
+ },
+ {
+ "if": {
+ "properties": {
+ "template_only": {
+ "const": false
+ },
+ "evidence_key": {
+ "const": "human-adjudication"
+ }
+ },
+ "required": [
+ "template_only",
+ "evidence_key"
+ ]
+ },
+ "then": {
+ "properties": {
+ "artifact_refs": {
+ "allOf": [
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/output_review_adjudication.json"
+ },
+ "kind": {
+ "const": "adjudication-report"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ },
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/output_review_decisions.json"
+ },
+ "kind": {
+ "const": "review-decisions"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "if": {
+ "properties": {
+ "template_only": {
+ "const": false
+ },
+ "evidence_key": {
+ "const": "native-permission-enforcement"
+ }
+ },
+ "required": [
+ "template_only",
+ "evidence_key"
+ ]
+ },
+ "then": {
+ "properties": {
+ "artifact_refs": {
+ "allOf": [
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/runtime_permission_probes.json"
+ },
+ "kind": {
+ "const": "runtime-probe-report"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ },
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/install_simulation.json"
+ },
+ "kind": {
+ "const": "installer-enforcement-report"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ }
+ ]
+ }
+ }
+ }
+ },
+ {
+ "if": {
+ "properties": {
+ "template_only": {
+ "const": false
+ },
+ "evidence_key": {
+ "const": "native-client-telemetry"
+ }
+ },
+ "required": [
+ "template_only",
+ "evidence_key"
+ ]
+ },
+ "then": {
+ "properties": {
+ "artifact_refs": {
+ "allOf": [
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/adoption_drift_report.json"
+ },
+ "kind": {
+ "const": "adoption-drift-report"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ },
+ {
+ "contains": {
+ "type": "object",
+ "properties": {
+ "path": {
+ "const": "reports/telemetry_hook_recipes.json"
+ },
+ "kind": {
+ "const": "hook-recipes"
+ }
+ },
+ "required": [
+ "path",
+ "kind"
+ ]
+ }
+ }
+ ]
+ }
+ }
+ }
+ }
+ ],
+ "additionalProperties": true
+}
diff --git a/evidence/world_class/templates/human-adjudication.intake.json b/evidence/world_class/templates/human-adjudication.intake.json
new file mode 100644
index 00000000..f6cfd728
--- /dev/null
+++ b/evidence/world_class/templates/human-adjudication.intake.json
@@ -0,0 +1,64 @@
+{
+ "schema_version": "1.0",
+ "evidence_key": "human-adjudication",
+ "template_only": true,
+ "category": "human",
+ "source_type": "blind-ab-review",
+ "submitted_by": "human reviewer",
+ "submitted_at": "YYYY-MM-DD",
+ "summary": "Real blind A/B reviewer decisions recorded before the answer key was opened.",
+ "artifact_refs": [
+ {
+ "path": "reports/output_review_adjudication.json",
+ "kind": "adjudication-report",
+ "contains_raw_content": false,
+ "note": "Must show pending_count == 0, invalid_decision_count == 0, blind_review_attested == true, and review_integrity.blind_pack_sha256."
+ },
+ {
+ "path": "reports/output_review_decisions.json",
+ "kind": "review-decisions",
+ "contains_raw_content": false,
+ "note": "Reviewer identity, reviewed_at, winner_variant, confidence, required reason, reviewer_attestation, and review_integrity only."
+ }
+ ],
+ "provenance": {
+ "reviewer": "name or team handle",
+ "reviewed_at": "YYYY-MM-DD",
+ "blind_pack_path": "reports/output_blind_review_pack.md",
+ "blind_pack_sha256": "sha256 from reports/output_review_adjudication.json review_integrity.blind_pack_sha256",
+ "answer_key_opened_after_decisions": true,
+ "blind_review_completed_before_answer_key": true,
+ "decision_fields": [
+ "case_id",
+ "winner_variant",
+ "confidence",
+ "reason",
+ "reviewer_attestation",
+ "review_integrity"
+ ],
+ "reviewer_reason_required": true
+ },
+ "privacy": {
+ "raw_user_content_included": false,
+ "raw_provider_prompt_included": false,
+ "credentials_included": false,
+ "secrets_included": false,
+ "notes": "Reviewer reasons must be based on the visible rubric and must not include private customer data."
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "attestation": {
+ "real_external_or_human_evidence": false,
+ "reviewer_or_operator_identity_present": false,
+ "artifact_refs_reviewed": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "notes": "Copy this template into submissions/ after a real reviewer completes blind decisions."
+ }
+}
diff --git a/evidence/world_class/templates/native-client-telemetry.intake.json b/evidence/world_class/templates/native-client-telemetry.intake.json
new file mode 100644
index 00000000..9dfdfa27
--- /dev/null
+++ b/evidence/world_class/templates/native-client-telemetry.intake.json
@@ -0,0 +1,53 @@
+{
+ "schema_version": "1.0",
+ "evidence_key": "native-client-telemetry",
+ "template_only": true,
+ "category": "external",
+ "source_type": "native-client-telemetry",
+ "submitted_by": "Browser/Chrome/IDE/provider client integrator",
+ "submitted_at": "YYYY-MM-DD",
+ "summary": "Metadata-only production events imported from a real external client into the local drift loop.",
+ "artifact_refs": [
+ {
+ "path": "reports/adoption_drift_report.json",
+ "kind": "adoption-drift-report",
+ "contains_raw_content": false,
+ "note": "Must show summary.source_types.external > 0 and adoption_sample_count > 0."
+ },
+ {
+ "path": "reports/telemetry_hook_recipes.json",
+ "kind": "hook-recipes",
+ "contains_raw_content": false,
+ "note": "Documents local-first metadata-only client integration."
+ }
+ ],
+ "provenance": {
+ "client": "Browser/Chrome/IDE/provider client",
+ "native_host_manifest": "/local/path/not/committed",
+ "event_source": "external",
+ "metadata_only": true
+ },
+ "privacy": {
+ "raw_user_content_included": false,
+ "raw_provider_prompt_included": false,
+ "credentials_included": false,
+ "secrets_included": false,
+ "notes": "Never commit telemetry JSONL with raw prompts, outputs, transcripts, notes, or messages."
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "attestation": {
+ "real_external_or_human_evidence": false,
+ "reviewer_or_operator_identity_present": false,
+ "artifact_refs_reviewed": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "notes": "Copy this template into submissions/ after real client events have been imported."
+ }
+}
diff --git a/evidence/world_class/templates/native-permission-enforcement.intake.json b/evidence/world_class/templates/native-permission-enforcement.intake.json
new file mode 100644
index 00000000..e16db354
--- /dev/null
+++ b/evidence/world_class/templates/native-permission-enforcement.intake.json
@@ -0,0 +1,61 @@
+{
+ "schema_version": "1.0",
+ "evidence_key": "native-permission-enforcement",
+ "template_only": true,
+ "category": "external",
+ "source_type": "runtime-permission-guard",
+ "submitted_by": "target client or installer integrator",
+ "submitted_at": "YYYY-MM-DD",
+ "summary": "A real target or installer guard enforces approved high-permission capabilities at runtime.",
+ "artifact_refs": [
+ {
+ "path": "reports/runtime_permission_probes.json",
+ "kind": "runtime-probe-report",
+ "contains_raw_content": false,
+ "note": "Must show native_enforcement_count > 0 and failure_count == 0 for target-native completion; installer_enforcement_pass_count is supporting local evidence only."
+ },
+ {
+ "path": "reports/install_simulation.json",
+ "kind": "installer-enforcement-report",
+ "contains_raw_content": false,
+ "note": "Shows local package installer enforcement. This does not replace target-native or external runtime guard evidence."
+ },
+ {
+ "path": "dist/targets//adapter.json",
+ "kind": "target-adapter",
+ "contains_raw_content": false,
+ "note": "Adapter may claim native enforcement only when the target really enforces it."
+ }
+ ],
+ "provenance": {
+ "target": "openai|claude|generic|vscode|other",
+ "guard_location": "client or installer component",
+ "guard_scope": "target-client-native",
+ "guard_blocks_undeclared_capability": true,
+ "metadata_fallback_retained_for_other_targets": true,
+ "notes": "Use target-client-native for client-enforced guards or external-installer-runtime-guard for a real external installer guard."
+ },
+ "privacy": {
+ "raw_user_content_included": false,
+ "raw_provider_prompt_included": false,
+ "credentials_included": false,
+ "secrets_included": false,
+ "notes": "Runtime guard proof should avoid environment dumps."
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "attestation": {
+ "real_external_or_human_evidence": false,
+ "reviewer_or_operator_identity_present": false,
+ "artifact_refs_reviewed": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "notes": "Copy this template into submissions/ only after a real client or installer guard exists."
+ }
+}
diff --git a/evidence/world_class/templates/provider-holdout.intake.json b/evidence/world_class/templates/provider-holdout.intake.json
new file mode 100644
index 00000000..631982b8
--- /dev/null
+++ b/evidence/world_class/templates/provider-holdout.intake.json
@@ -0,0 +1,50 @@
+{
+ "schema_version": "1.0",
+ "evidence_key": "provider-holdout",
+ "template_only": true,
+ "category": "external",
+ "source_type": "provider-output-eval",
+ "submitted_by": "operator with provider credentials",
+ "submitted_at": "YYYY-MM-DD",
+ "summary": "Provider-backed holdout run with model, timing, token metadata, and aggregate output hashes.",
+ "artifact_refs": [
+ {
+ "path": "reports/output_execution_runs.json",
+ "kind": "aggregate-report",
+ "contains_raw_content": false,
+ "note": "Must show model_executed_count > 0 and token_observed_count > 0."
+ }
+ ],
+ "provenance": {
+ "provider": "",
+ "model": "",
+ "run_command": "python3 scripts/yao.py output-exec --provider-runner --provider-model --timeout-seconds 60",
+ "credential_env": "",
+ "model_env": "YAO_OUTPUT_EVAL_MODEL",
+ "model_default": "",
+ "credential_material_committed": false
+ },
+ "privacy": {
+ "raw_user_content_included": false,
+ "raw_provider_prompt_included": false,
+ "credentials_included": false,
+ "secrets_included": false,
+ "notes": "Commit aggregate metadata and hashes only."
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "attestation": {
+ "real_external_or_human_evidence": false,
+ "reviewer_or_operator_identity_present": false,
+ "artifact_refs_reviewed": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "notes": "Copy this template into submissions/ and set all booleans truthfully after the real provider run."
+ }
+}
diff --git a/examples/complex-release-orchestrator/generated-skill/reports/context_budget.json b/examples/complex-release-orchestrator/generated-skill/reports/context_budget.json
index 5c287256..660f62f5 100644
--- a/examples/complex-release-orchestrator/generated-skill/reports/context_budget.json
+++ b/examples/complex-release-orchestrator/generated-skill/reports/context_budget.json
@@ -6,10 +6,47 @@
"context_budget_tier": "production",
"context_budget_limit": 1000,
"skill_body_tokens": 718,
- "other_text_tokens": 1946,
+ "other_text_tokens": 1842,
"estimated_initial_load_tokens": 790,
- "estimated_total_text_tokens": 2664,
- "relevant_file_count": 12,
+ "estimated_total_text_tokens": 2560,
+ "deferred_resource_tokens": 1657,
+ "deferred_resource_warn_threshold": 120000,
+ "deferred_resource_dirs": [
+ {
+ "path": "references",
+ "estimated_tokens": 554,
+ "file_count": 4
+ },
+ {
+ "path": "scripts",
+ "estimated_tokens": 367,
+ "file_count": 1
+ },
+ {
+ "path": "input",
+ "estimated_tokens": 315,
+ "file_count": 1
+ },
+ {
+ "path": "outputs",
+ "estimated_tokens": 263,
+ "file_count": 1
+ },
+ {
+ "path": "evals",
+ "estimated_tokens": 158,
+ "file_count": 1
+ }
+ ],
+ "large_deferred_resource_dirs": [],
+ "deferred_resource_governance": {
+ "status": "not-required",
+ "large_dir_count": 0,
+ "governed_large_dir_count": 0,
+ "directories": [],
+ "summary": "No large deferred resource directory exceeds the per-dir threshold."
+ },
+ "relevant_file_count": 11,
"unused_resource_dirs": [],
"quality_signal_points": 130,
"quality_density": 164.6
diff --git a/examples/governed-incident-command/generated-skill/reports/context_budget.json b/examples/governed-incident-command/generated-skill/reports/context_budget.json
index 98b114e9..c0471b88 100644
--- a/examples/governed-incident-command/generated-skill/reports/context_budget.json
+++ b/examples/governed-incident-command/generated-skill/reports/context_budget.json
@@ -6,10 +6,47 @@
"context_budget_tier": "production",
"context_budget_limit": 1000,
"skill_body_tokens": 658,
- "other_text_tokens": 1603,
+ "other_text_tokens": 1499,
"estimated_initial_load_tokens": 760,
- "estimated_total_text_tokens": 2261,
- "relevant_file_count": 13,
+ "estimated_total_text_tokens": 2157,
+ "deferred_resource_tokens": 1030,
+ "deferred_resource_warn_threshold": 120000,
+ "deferred_resource_dirs": [
+ {
+ "path": "scripts",
+ "estimated_tokens": 278,
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "estimated_tokens": 266,
+ "file_count": 3
+ },
+ {
+ "path": "outputs",
+ "estimated_tokens": 203,
+ "file_count": 1
+ },
+ {
+ "path": "input",
+ "estimated_tokens": 153,
+ "file_count": 1
+ },
+ {
+ "path": "evals",
+ "estimated_tokens": 130,
+ "file_count": 1
+ }
+ ],
+ "large_deferred_resource_dirs": [],
+ "deferred_resource_governance": {
+ "status": "not-required",
+ "large_dir_count": 0,
+ "governed_large_dir_count": 0,
+ "directories": [],
+ "summary": "No large deferred resource directory exceeds the per-dir threshold."
+ },
+ "relevant_file_count": 12,
"unused_resource_dirs": [],
"quality_signal_points": 130,
"quality_density": 171.1
diff --git a/examples/governed-incident-command/optimization/reports/description_optimization.json b/examples/governed-incident-command/optimization/reports/description_optimization.json
index 6af793f3..8707247d 100644
--- a/examples/governed-incident-command/optimization/reports/description_optimization.json
+++ b/examples/governed-incident-command/optimization/reports/description_optimization.json
@@ -8,19 +8,19 @@
"estimated_tokens": 37,
"dev": {
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"near_neighbor_pass_rate": 1.0,
"should_not_trigger_pass_rate": 1.0
},
"selection_key": [
0,
- 1,
+ 0,
+ -1.0,
-1.0,
-1.0,
-1.0,
- -0.667,
37
],
"holdout": {
@@ -39,14 +39,14 @@
"threshold": 0.48,
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.",
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"bucket_stats": {
"should_trigger": {
"total": 3,
- "passed": 2,
- "pass_rate": 0.667
+ "passed": 3,
+ "pass_rate": 1.0
},
"should_not_trigger": {
"total": 3,
@@ -69,10 +69,10 @@
},
"governed_workflow": {
"total": 2,
- "passed": 1,
+ "passed": 2,
"false_positives": 0,
- "false_negatives": 1,
- "pass_rate": 0.5
+ "false_negatives": 0,
+ "pass_rate": 1.0
},
"explain_only": {
"total": 1,
@@ -110,21 +110,7 @@
"pass_rate": 1.0
}
},
- "misfires": [
- {
- "bucket": "should_trigger",
- "family": "governed_workflow",
- "prompt": "Package this incident process into a maintained operational skill with severity, owners, and communication blocks.",
- "score": 0.392,
- "reason": "false_negative",
- "matched_desired_concepts": [
- "governed_workflow",
- "reusable_packaging",
- "severity_assessment"
- ],
- "matched_negative_concepts": []
- }
- ],
+ "misfires": [],
"results": {
"should_trigger": [
{
@@ -212,7 +198,8 @@
"outage workflow"
],
"incident_packet": [
- "incident review skill"
+ "incident review skill",
+ "incident review"
],
"reusable_packaging": [
"reusable",
@@ -221,7 +208,8 @@
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
@@ -231,10 +219,10 @@
{
"prompt": "Package this incident process into a maintained operational skill with severity, owners, and communication blocks.",
"family": "governed_workflow",
- "score": 0.392,
- "predicted_trigger": false,
+ "score": 0.539,
+ "predicted_trigger": true,
"expected_trigger": true,
- "passed": false,
+ "passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
@@ -246,6 +234,7 @@
"reusable_packaging"
],
"matched_desired_concepts": [
+ "comms_and_owners",
"governed_workflow",
"reusable_packaging",
"severity_assessment"
@@ -253,13 +242,16 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.32,
+ "semantic_coverage": 0.48,
"support_score": 0,
"lexical_support": 0.467,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
+ "comms_and_owners": [
+ "owners"
+ ],
"governed_workflow": [
"maintained",
"operational skill"
@@ -394,7 +386,7 @@
{
"prompt": "Summarize this incident but do not package it as a reusable skill.",
"family": "summary_only",
- "score": 0.07,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
@@ -412,12 +404,16 @@
"reusable_packaging"
],
"extra_positive_concepts": [],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"semantic_coverage": 0.04,
"support_score": 0,
"lexical_support": 0.417,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
"positive": {
@@ -427,7 +423,11 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
}
},
@@ -467,7 +467,8 @@
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"reusable_packaging": [
"workflow"
@@ -488,14 +489,14 @@
"threshold": 0.48,
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.",
"false_positives": 0,
- "false_negatives": 2,
- "precision": null,
- "recall": 0.0,
+ "false_negatives": 1,
+ "precision": 1.0,
+ "recall": 0.5,
"bucket_stats": {
"should_trigger": {
"total": 2,
- "passed": 0,
- "pass_rate": 0.0
+ "passed": 1,
+ "pass_rate": 0.5
},
"should_not_trigger": {
"total": 2,
@@ -511,10 +512,10 @@
"family_stats": {
"governed_workflow": {
"total": 1,
- "passed": 0,
+ "passed": 1,
"false_positives": 0,
- "false_negatives": 1,
- "pass_rate": 0.0
+ "false_negatives": 0,
+ "pass_rate": 1.0
},
"packet_assembly": {
"total": 1,
@@ -553,19 +554,6 @@
}
},
"misfires": [
- {
- "bucket": "should_trigger",
- "family": "governed_workflow",
- "prompt": "Standardize this incident review process into a governed packet workflow with severity rationale and stakeholder updates.",
- "score": 0.399,
- "reason": "false_negative",
- "matched_desired_concepts": [
- "governed_workflow",
- "reusable_packaging",
- "severity_assessment"
- ],
- "matched_negative_concepts": []
- },
{
"bucket": "should_trigger",
"family": "packet_assembly",
@@ -584,10 +572,10 @@
{
"prompt": "Standardize this incident review process into a governed packet workflow with severity rationale and stakeholder updates.",
"family": "governed_workflow",
- "score": 0.399,
- "predicted_trigger": false,
+ "score": 0.694,
+ "predicted_trigger": true,
"expected_trigger": true,
- "passed": false,
+ "passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
@@ -600,13 +588,14 @@
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"reusable_packaging",
"severity_assessment"
],
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.32,
+ "semantic_coverage": 0.64,
"support_score": 0,
"lexical_support": 0.562,
"negative_penalty": 0,
@@ -614,19 +603,23 @@
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "standardize"
+ ],
+ "incident_packet": [
+ "incident review"
],
"reusable_packaging": [
"workflow"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Build a reusable incident command skill from alerts, timelines, and responder notes.",
@@ -829,19 +822,19 @@
"estimated_tokens": 37,
"dev": {
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"near_neighbor_pass_rate": 1.0,
"should_not_trigger_pass_rate": 1.0
},
"selection_key": [
0,
- 1,
+ 0,
+ -1.0,
-1.0,
-1.0,
-1.0,
- -0.667,
37
],
"holdout": {
@@ -857,14 +850,14 @@
"threshold": 0.48,
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.",
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"bucket_stats": {
"should_trigger": {
"total": 3,
- "passed": 2,
- "pass_rate": 0.667
+ "passed": 3,
+ "pass_rate": 1.0
},
"should_not_trigger": {
"total": 3,
@@ -880,10 +873,10 @@
"family_stats": {
"packet_assembly": {
"total": 1,
- "passed": 0,
+ "passed": 1,
"false_positives": 0,
- "false_negatives": 1,
- "pass_rate": 0.0
+ "false_negatives": 0,
+ "pass_rate": 1.0
},
"governed_workflow": {
"total": 2,
@@ -928,49 +921,38 @@
"pass_rate": 1.0
}
},
- "misfires": [
- {
- "bucket": "should_trigger",
- "family": "packet_assembly",
- "prompt": "Turn these alerts, incident notes, and responder transcripts into a governed incident command packet.",
- "score": 0.348,
- "reason": "false_negative",
- "matched_desired_concepts": [
- "governed_workflow"
- ],
- "matched_negative_concepts": []
- }
- ],
+ "misfires": [],
"results": {
"should_trigger": [
{
"prompt": "Turn these alerts, incident notes, and responder transcripts into a governed incident command packet.",
"family": "packet_assembly",
- "score": 0.348,
- "predicted_trigger": false,
+ "score": 0.617,
+ "predicted_trigger": true,
"expected_trigger": true,
- "passed": false,
+ "passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
- "governed_workflow"
+ "governed_workflow",
+ "incident_packet"
],
"extra_positive_concepts": [
- "incident_inputs",
- "incident_packet"
+ "incident_inputs"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.227,
- "support_score": 0.52,
+ "semantic_coverage": 0.553,
+ "support_score": 0.2,
"lexical_support": 0.231,
"negative_penalty": 0,
- "coverage_boost": 0.0,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"governed_workflow": [
@@ -992,39 +974,41 @@
{
"prompt": "Create a reusable incident review skill from this outage workflow with severity assessment and stakeholder communication.",
"family": "governed_workflow",
- "score": 0.521,
+ "score": 0.73,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
"incident_inputs",
- "incident_packet",
"reusable_packaging"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
- "support_score": 0.56,
+ "semantic_coverage": 0.658,
+ "support_score": 0.24,
"lexical_support": 0.312,
"negative_penalty": 0,
- "coverage_boost": 0.0,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"incident_inputs": [
"outage workflow"
],
"incident_packet": [
- "incident review skill"
+ "incident review skill",
+ "incident review"
],
"reusable_packaging": [
"reusable",
@@ -1033,29 +1017,31 @@
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Package this incident process into a maintained operational skill with severity, owners, and communication blocks.",
"family": "governed_workflow",
- "score": 0.651,
+ "score": 0.619,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
+ "comms_and_owners",
"governed_workflow",
"severity_assessment"
],
@@ -1064,13 +1050,16 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.579,
"support_score": 0.04,
"lexical_support": 0.2,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
+ "comms_and_owners": [
+ "owners"
+ ],
"governed_workflow": [
"maintained",
"operational skill"
@@ -1092,13 +1081,14 @@
{
"prompt": "Explain what incident severity means.",
"family": "explain_only",
- "score": 0.408,
+ "score": 0.25,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1109,7 +1099,7 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0,
"lexical_support": 0.4,
"negative_penalty": 0,
@@ -1122,8 +1112,7 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Draft one short outage update for customers.",
@@ -1135,6 +1124,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1172,6 +1162,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1196,13 +1187,14 @@
{
"prompt": "Summarize this incident but do not package it as a reusable skill.",
"family": "summary_only",
- "score": 0.017,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1211,12 +1203,16 @@
"extra_positive_concepts": [
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"semantic_coverage": 0.0,
"support_score": 0.04,
"lexical_support": 0.083,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
"positive": {
@@ -1226,7 +1222,11 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
}
},
@@ -1240,6 +1240,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1256,7 +1257,7 @@
"exclusive_negative_concepts": [
"brainstorm_only"
],
- "semantic_coverage": 0.227,
+ "semantic_coverage": 0.132,
"support_score": 0.04,
"lexical_support": 0.286,
"negative_penalty": 0.24,
@@ -1264,7 +1265,8 @@
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"reusable_packaging": [
"workflow"
@@ -1365,19 +1367,21 @@
{
"prompt": "Standardize this incident review process into a governed packet workflow with severity rationale and stakeholder updates.",
"family": "governed_workflow",
- "score": 0.66,
+ "score": 0.821,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -1385,21 +1389,26 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.789,
"support_score": 0.04,
"lexical_support": 0.312,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "standardize"
+ ],
+ "incident_packet": [
+ "incident review"
],
"reusable_packaging": [
"workflow"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
@@ -1416,6 +1425,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1458,6 +1468,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1487,6 +1498,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1526,6 +1538,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1561,6 +1574,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1587,14 +1601,14 @@
"threshold": 0.48,
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.",
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"bucket_stats": {
"should_trigger": {
"total": 3,
- "passed": 2,
- "pass_rate": 0.667
+ "passed": 3,
+ "pass_rate": 1.0
},
"should_not_trigger": {
"total": 3,
@@ -1610,10 +1624,10 @@
"family_stats": {
"packet_assembly": {
"total": 1,
- "passed": 0,
+ "passed": 1,
"false_positives": 0,
- "false_negatives": 1,
- "pass_rate": 0.0
+ "false_negatives": 0,
+ "pass_rate": 1.0
},
"governed_workflow": {
"total": 2,
@@ -1658,49 +1672,38 @@
"pass_rate": 1.0
}
},
- "misfires": [
- {
- "bucket": "should_trigger",
- "family": "packet_assembly",
- "prompt": "Turn these alerts, incident notes, and responder transcripts into a governed incident command packet.",
- "score": 0.348,
- "reason": "false_negative",
- "matched_desired_concepts": [
- "governed_workflow"
- ],
- "matched_negative_concepts": []
- }
- ],
+ "misfires": [],
"results": {
"should_trigger": [
{
"prompt": "Turn these alerts, incident notes, and responder transcripts into a governed incident command packet.",
"family": "packet_assembly",
- "score": 0.348,
- "predicted_trigger": false,
+ "score": 0.617,
+ "predicted_trigger": true,
"expected_trigger": true,
- "passed": false,
+ "passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
- "governed_workflow"
+ "governed_workflow",
+ "incident_packet"
],
"extra_positive_concepts": [
- "incident_inputs",
- "incident_packet"
+ "incident_inputs"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.227,
- "support_score": 0.52,
+ "semantic_coverage": 0.553,
+ "support_score": 0.2,
"lexical_support": 0.231,
"negative_penalty": 0,
- "coverage_boost": 0.0,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"governed_workflow": [
@@ -1722,39 +1725,41 @@
{
"prompt": "Create a reusable incident review skill from this outage workflow with severity assessment and stakeholder communication.",
"family": "governed_workflow",
- "score": 0.521,
+ "score": 0.73,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
"incident_inputs",
- "incident_packet",
"reusable_packaging"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
- "support_score": 0.56,
+ "semantic_coverage": 0.658,
+ "support_score": 0.24,
"lexical_support": 0.312,
"negative_penalty": 0,
- "coverage_boost": 0.0,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"incident_inputs": [
"outage workflow"
],
"incident_packet": [
- "incident review skill"
+ "incident review skill",
+ "incident review"
],
"reusable_packaging": [
"reusable",
@@ -1763,29 +1768,31 @@
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Package this incident process into a maintained operational skill with severity, owners, and communication blocks.",
"family": "governed_workflow",
- "score": 0.651,
+ "score": 0.619,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
+ "comms_and_owners",
"governed_workflow",
"severity_assessment"
],
@@ -1794,13 +1801,16 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.579,
"support_score": 0.04,
"lexical_support": 0.2,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
+ "comms_and_owners": [
+ "owners"
+ ],
"governed_workflow": [
"maintained",
"operational skill"
@@ -1822,13 +1832,14 @@
{
"prompt": "Explain what incident severity means.",
"family": "explain_only",
- "score": 0.408,
+ "score": 0.25,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1839,7 +1850,7 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0,
"lexical_support": 0.4,
"negative_penalty": 0,
@@ -1852,8 +1863,7 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Draft one short outage update for customers.",
@@ -1865,6 +1875,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1902,6 +1913,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1926,13 +1938,14 @@
{
"prompt": "Summarize this incident but do not package it as a reusable skill.",
"family": "summary_only",
- "score": 0.017,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1941,12 +1954,16 @@
"extra_positive_concepts": [
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"semantic_coverage": 0.0,
"support_score": 0.04,
"lexical_support": 0.083,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
"positive": {
@@ -1956,7 +1973,11 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
}
},
@@ -1970,6 +1991,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -1986,7 +2008,7 @@
"exclusive_negative_concepts": [
"brainstorm_only"
],
- "semantic_coverage": 0.227,
+ "semantic_coverage": 0.132,
"support_score": 0.04,
"lexical_support": 0.286,
"negative_penalty": 0.24,
@@ -1994,7 +2016,8 @@
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"reusable_packaging": [
"workflow"
@@ -2095,19 +2118,21 @@
{
"prompt": "Standardize this incident review process into a governed packet workflow with severity rationale and stakeholder updates.",
"family": "governed_workflow",
- "score": 0.66,
+ "score": 0.821,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -2115,21 +2140,26 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.789,
"support_score": 0.04,
"lexical_support": 0.312,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "standardize"
+ ],
+ "incident_packet": [
+ "incident review"
],
"reusable_packaging": [
"workflow"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
@@ -2146,6 +2176,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2188,6 +2219,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2217,6 +2249,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2256,6 +2289,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2291,6 +2325,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2387,13 +2422,14 @@
{
"prompt": "Turn these alert logs and responder notes into a governed incident packet with severity rationale and owner actions.",
"family": "blind_packet_assembly",
- "score": 1.0,
+ "score": 0.994,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2401,22 +2437,22 @@
"matched_desired_concepts": [
"comms_and_owners",
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
- "extra_positive_concepts": [
- "incident_packet"
- ],
+ "extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
"semantic_coverage": 1.0,
- "support_score": 0.32,
+ "support_score": 0,
"lexical_support": 0.176,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -2435,19 +2471,21 @@
{
"prompt": "Standardize this outage review, run severity assessment, and assemble incident communication into a reusable incident workflow.",
"family": "blind_governed_workflow",
- "score": 0.804,
+ "score": 0.645,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"comms_and_owners",
+ "governed_workflow",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -2455,23 +2493,27 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.773,
+ "semantic_coverage": 0.579,
"support_score": 0.04,
"lexical_support": 0.533,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -2483,13 +2525,14 @@
{
"prompt": "Explain how severity levels are decided during incidents.",
"family": "blind_explain_only",
- "score": 0.386,
+ "score": 0.228,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2500,7 +2543,7 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0,
"lexical_support": 0.125,
"negative_penalty": 0,
@@ -2513,8 +2556,7 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Write a single incident update for customers from these notes.",
@@ -2526,6 +2568,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2550,13 +2593,14 @@
{
"prompt": "Summarize the outage timeline, but do not package a governed workflow.",
"family": "blind_summary_only",
- "score": 0.276,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2568,17 +2612,22 @@
"incident_inputs",
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
- "semantic_coverage": 0.227,
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "semantic_coverage": 0.132,
"support_score": 0.24,
"lexical_support": 0.091,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -2588,7 +2637,11 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
}
},
@@ -2602,6 +2655,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2706,13 +2760,14 @@
{
"prompt": "Turn these alert logs and responder notes into a governed incident packet with severity rationale and owner actions.",
"family": "blind_packet_assembly",
- "score": 1.0,
+ "score": 0.994,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2720,22 +2775,22 @@
"matched_desired_concepts": [
"comms_and_owners",
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
- "extra_positive_concepts": [
- "incident_packet"
- ],
+ "extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
"semantic_coverage": 1.0,
- "support_score": 0.32,
+ "support_score": 0,
"lexical_support": 0.176,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -2754,19 +2809,21 @@
{
"prompt": "Standardize this outage review, run severity assessment, and assemble incident communication into a reusable incident workflow.",
"family": "blind_governed_workflow",
- "score": 0.804,
+ "score": 0.645,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"comms_and_owners",
+ "governed_workflow",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -2774,23 +2831,27 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.773,
+ "semantic_coverage": 0.579,
"support_score": 0.04,
"lexical_support": 0.533,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -2802,13 +2863,14 @@
{
"prompt": "Explain how severity levels are decided during incidents.",
"family": "blind_explain_only",
- "score": 0.386,
+ "score": 0.228,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2819,7 +2881,7 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0,
"lexical_support": 0.125,
"negative_penalty": 0,
@@ -2832,8 +2894,7 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Write a single incident update for customers from these notes.",
@@ -2845,6 +2906,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2869,13 +2931,14 @@
{
"prompt": "Summarize the outage timeline, but do not package a governed workflow.",
"family": "blind_summary_only",
- "score": 0.276,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2887,17 +2950,22 @@
"incident_inputs",
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
- "semantic_coverage": 0.227,
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "semantic_coverage": 0.132,
"support_score": 0.24,
"lexical_support": 0.091,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -2907,7 +2975,11 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
}
},
@@ -2921,6 +2993,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -2955,14 +3028,14 @@
"threshold": 0.48,
"threshold_explanation": "Prompts at or above the threshold are treated as trigger matches. Scores are driven primarily by semantic intent coverage: packaging intent, workflow-to-skill transformation intent, reuse/distribution intent, and eval intent. Explicit exclusions such as summary-only, translation-only, one-off, document-only, or do-not-build directives apply direct penalties and can override otherwise similar wording.",
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.5,
+ "recall": 1.0,
"bucket_stats": {
"should_trigger": {
"total": 2,
- "passed": 1,
- "pass_rate": 0.5
+ "passed": 2,
+ "pass_rate": 1.0
},
"should_not_trigger": {
"total": 2,
@@ -2985,10 +3058,10 @@
},
"blind_governed_workflow": {
"total": 1,
- "passed": 0,
+ "passed": 1,
"false_positives": 0,
- "false_negatives": 1,
- "pass_rate": 0.0
+ "false_negatives": 0,
+ "pass_rate": 1.0
},
"blind_explain_only": {
"total": 1,
@@ -3019,21 +3092,7 @@
"pass_rate": 1.0
}
},
- "misfires": [
- {
- "bucket": "should_trigger",
- "family": "blind_governed_workflow",
- "prompt": "Standardize this outage review, run severity assessment, and assemble incident communication into a reusable incident workflow.",
- "score": 0.458,
- "reason": "false_negative",
- "matched_desired_concepts": [
- "comms_and_owners",
- "reusable_packaging",
- "severity_assessment"
- ],
- "matched_negative_concepts": []
- }
- ],
+ "misfires": [],
"results": {
"should_trigger": [
{
@@ -3070,7 +3129,8 @@
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -3089,10 +3149,10 @@
{
"prompt": "Standardize this outage review, run severity assessment, and assemble incident communication into a reusable incident workflow.",
"family": "blind_governed_workflow",
- "score": 0.458,
- "predicted_trigger": false,
+ "score": 0.55,
+ "predicted_trigger": true,
"expected_trigger": true,
- "passed": false,
+ "passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
@@ -3105,13 +3165,14 @@
],
"matched_desired_concepts": [
"comms_and_owners",
+ "governed_workflow",
"reusable_packaging",
"severity_assessment"
],
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.38,
+ "semantic_coverage": 0.48,
"support_score": 0,
"lexical_support": 0.6,
"negative_penalty": 0,
@@ -3121,13 +3182,17 @@
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -3212,7 +3277,7 @@
{
"prompt": "Summarize the outage timeline, but do not package a governed workflow.",
"family": "blind_summary_only",
- "score": 0.395,
+ "score": 0.005,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
@@ -3232,17 +3297,22 @@
"reusable_packaging"
],
"extra_positive_concepts": [],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"semantic_coverage": 0.34,
"support_score": 0,
"lexical_support": 0.273,
- "negative_penalty": 0,
+ "negative_penalty": 0.24,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -3252,10 +3322,13 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Brainstorm how incident handling could improve next quarter without building the packet.",
@@ -3380,11 +3453,13 @@
"judge_detail": {
"mode": "judge-rubric",
"focused_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"strong_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners"
],
@@ -3403,7 +3478,8 @@
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -3429,7 +3505,8 @@
"mode": "judge-rubric",
"focused_positive_concepts": [
"severity_assessment",
- "comms_and_owners"
+ "comms_and_owners",
+ "governed_workflow"
],
"strong_positive_concepts": [
"severity_assessment",
@@ -3446,22 +3523,26 @@
"nonexclusive_negative_concepts": [],
"exclusion_hits": [],
"generic_exclusion_hits": [],
- "positive_vote": 5,
+ "positive_vote": 6,
"negative_vote": 0,
- "margin": 5,
- "confidence": 0.96,
+ "margin": 6,
+ "confidence": 1.0,
"concept_evidence": {
"positive": {
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -3556,20 +3637,23 @@
"input_hits": [],
"artifact_hits": [],
"capability_overlap": 1,
- "exclusive_negative_concepts": [],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"nonexclusive_negative_concepts": [],
"exclusion_hits": [],
"generic_exclusion_hits": [
"summarize"
],
"positive_vote": 1,
- "negative_vote": 1,
- "margin": 0,
- "confidence": 0.5,
+ "negative_vote": 3,
+ "margin": -2,
+ "confidence": 0.58,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -3579,10 +3663,13 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Brainstorm how incident handling could improve next quarter without building the packet.",
@@ -3624,7 +3711,7 @@
},
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -3708,11 +3795,13 @@
"judge_detail": {
"mode": "judge-rubric",
"focused_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"strong_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners"
],
@@ -3731,7 +3820,8 @@
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -3757,7 +3847,8 @@
"mode": "judge-rubric",
"focused_positive_concepts": [
"severity_assessment",
- "comms_and_owners"
+ "comms_and_owners",
+ "governed_workflow"
],
"strong_positive_concepts": [
"severity_assessment",
@@ -3774,22 +3865,26 @@
"nonexclusive_negative_concepts": [],
"exclusion_hits": [],
"generic_exclusion_hits": [],
- "positive_vote": 5,
+ "positive_vote": 6,
"negative_vote": 0,
- "margin": 5,
- "confidence": 0.96,
+ "margin": 6,
+ "confidence": 1.0,
"concept_evidence": {
"positive": {
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -3884,20 +3979,23 @@
"input_hits": [],
"artifact_hits": [],
"capability_overlap": 1,
- "exclusive_negative_concepts": [],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"nonexclusive_negative_concepts": [],
"exclusion_hits": [],
"generic_exclusion_hits": [
"summarize"
],
"positive_vote": 1,
- "negative_vote": 1,
- "margin": 0,
- "confidence": 0.5,
+ "negative_vote": 3,
+ "margin": -2,
+ "confidence": 0.58,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -3907,10 +4005,13 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Brainstorm how incident handling could improve next quarter without building the packet.",
@@ -3952,7 +4053,7 @@
},
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -4061,7 +4162,8 @@
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -4088,6 +4190,7 @@
"focused_positive_concepts": [
"severity_assessment",
"comms_and_owners",
+ "governed_workflow",
"reusable_packaging"
],
"strong_positive_concepts": [
@@ -4114,13 +4217,17 @@
"comms_and_owners": [
"incident communication"
],
+ "governed_workflow": [
+ "standardize"
+ ],
"reusable_packaging": [
"reusable",
"workflow"
],
"severity_assessment": [
"severity assessment",
- "severity"
+ "severity",
+ "outage review"
]
},
"negative": {}
@@ -4219,20 +4326,23 @@
"input_hits": [],
"artifact_hits": [],
"capability_overlap": 1,
- "exclusive_negative_concepts": [],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
"nonexclusive_negative_concepts": [],
"exclusion_hits": [],
"generic_exclusion_hits": [
"summarize"
],
"positive_vote": 3,
- "negative_vote": 1,
- "margin": 2,
- "confidence": 0.66,
+ "negative_vote": 3,
+ "margin": 0,
+ "confidence": 0.58,
"concept_evidence": {
"positive": {
"governed_workflow": [
- "governed"
+ "governed",
+ "governed workflow"
],
"incident_inputs": [
"timeline"
@@ -4242,9 +4352,14 @@
"package"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "do not package"
+ ]
+ }
}
- }
+ },
+ "boundary_case": true
},
{
"prompt": "Brainstorm how incident handling could improve next quarter without building the packet.",
@@ -4286,7 +4401,7 @@
},
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.69,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -4371,6 +4486,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4378,23 +4494,24 @@
"matched_desired_concepts": [
"comms_and_owners",
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
- "incident_inputs",
- "incident_packet"
+ "incident_inputs"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
"semantic_coverage": 1.0,
- "support_score": 0.52,
+ "support_score": 0.2,
"lexical_support": 0.263,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -4419,19 +4536,21 @@
{
"prompt": "Treat this outage as a maintained operational skill: standardize the incident review, assess severity, and package communications.",
"family": "adversarial_operational_skill_route",
- "score": 0.654,
+ "score": 0.815,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -4439,23 +4558,28 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.789,
"support_score": 0.04,
"lexical_support": 0.235,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"governed_workflow": [
+ "standardize",
"maintained",
"operational skill"
],
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"skill",
"package"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
@@ -4474,6 +4598,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4502,13 +4627,14 @@
{
"prompt": "Explain incident severity and summarize the outage timeline for training.",
"family": "adversarial_explain_and_summary_collision",
- "score": 0.032,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4525,7 +4651,7 @@
"exclusive_negative_concepts": [
"explain_only"
],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0.2,
"lexical_support": 0.2,
"negative_penalty": 0.26,
@@ -4552,39 +4678,61 @@
{
"prompt": "Sketch a future incident review playbook for later, but do not package a reusable workflow.",
"family": "adversarial_future_playbook_not_packet",
- "score": 0.021,
+ "score": 0.277,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
- "matched_desired_concepts": [],
+ "matched_desired_concepts": [
+ "incident_packet",
+ "severity_assessment"
+ ],
"extra_positive_concepts": [
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
- "semantic_coverage": 0.0,
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "semantic_coverage": 0.658,
"support_score": 0.04,
"lexical_support": 0.143,
- "negative_penalty": 0,
- "coverage_boost": 0.0,
+ "negative_penalty": 0.24,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"reusable",
"workflow",
"package"
+ ],
+ "severity_assessment": [
+ "incident review"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "sketch a future",
+ "for later",
+ "do not package",
+ "do not package a reusable workflow"
+ ]
+ }
}
- }
+ },
+ "boundary_case": true
},
{
"prompt": "Troubleshoot the outage and fix the service before we talk about reusable incident packets.",
@@ -4596,6 +4744,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4713,6 +4862,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4720,23 +4870,24 @@
"matched_desired_concepts": [
"comms_and_owners",
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
- "incident_inputs",
- "incident_packet"
+ "incident_inputs"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
"semantic_coverage": 1.0,
- "support_score": 0.52,
+ "support_score": 0.2,
"lexical_support": 0.263,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -4761,19 +4912,21 @@
{
"prompt": "Treat this outage as a maintained operational skill: standardize the incident review, assess severity, and package communications.",
"family": "adversarial_operational_skill_route",
- "score": 0.654,
+ "score": 0.815,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"severity_assessment"
],
"extra_positive_concepts": [
@@ -4781,23 +4934,28 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.636,
+ "semantic_coverage": 0.789,
"support_score": 0.04,
"lexical_support": 0.235,
"negative_penalty": 0,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
"governed_workflow": [
+ "standardize",
"maintained",
"operational skill"
],
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"skill",
"package"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
@@ -4816,6 +4974,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4844,13 +5003,14 @@
{
"prompt": "Explain incident severity and summarize the outage timeline for training.",
"family": "adversarial_explain_and_summary_collision",
- "score": 0.032,
+ "score": 0.0,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -4867,7 +5027,7 @@
"exclusive_negative_concepts": [
"explain_only"
],
- "semantic_coverage": 0.409,
+ "semantic_coverage": 0.237,
"support_score": 0.2,
"lexical_support": 0.2,
"negative_penalty": 0.26,
@@ -4894,39 +5054,61 @@
{
"prompt": "Sketch a future incident review playbook for later, but do not package a reusable workflow.",
"family": "adversarial_future_playbook_not_packet",
- "score": 0.021,
+ "score": 0.277,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
],
- "matched_desired_concepts": [],
+ "matched_desired_concepts": [
+ "incident_packet",
+ "severity_assessment"
+ ],
"extra_positive_concepts": [
"reusable_packaging"
],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
- "semantic_coverage": 0.0,
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "semantic_coverage": 0.658,
"support_score": 0.04,
"lexical_support": 0.143,
- "negative_penalty": 0,
- "coverage_boost": 0.0,
+ "negative_penalty": 0.24,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"reusable",
"workflow",
"package"
+ ],
+ "severity_assessment": [
+ "incident review"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "sketch a future",
+ "for later",
+ "do not package",
+ "do not package a reusable workflow"
+ ]
+ }
}
- }
+ },
+ "boundary_case": true
},
{
"prompt": "Troubleshoot the outage and fix the service before we talk about reusable incident packets.",
@@ -4938,6 +5120,7 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
+ "incident_packet",
"severity_assessment",
"comms_and_owners",
"governed_workflow"
@@ -5080,7 +5263,8 @@
"concept_evidence": {
"positive": {
"comms_and_owners": [
- "owner"
+ "owner",
+ "owner actions"
],
"governed_workflow": [
"governed"
@@ -5105,7 +5289,7 @@
{
"prompt": "Treat this outage as a maintained operational skill: standardize the incident review, assess severity, and package communications.",
"family": "adversarial_operational_skill_route",
- "score": 0.387,
+ "score": 0.682,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
@@ -5121,13 +5305,14 @@
],
"matched_desired_concepts": [
"governed_workflow",
+ "incident_packet",
"reusable_packaging",
"severity_assessment"
],
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.32,
+ "semantic_coverage": 0.64,
"support_score": 0,
"lexical_support": 0.412,
"negative_penalty": 0,
@@ -5135,21 +5320,25 @@
"concept_evidence": {
"positive": {
"governed_workflow": [
+ "standardize",
"maintained",
"operational skill"
],
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"skill",
"package"
],
"severity_assessment": [
- "severity"
+ "severity",
+ "incident review"
]
},
"negative": {}
}
- },
- "boundary_case": true
+ }
}
],
"should_not_trigger": [
@@ -5246,7 +5435,7 @@
{
"prompt": "Sketch a future incident review playbook for later, but do not package a reusable workflow.",
"family": "adversarial_future_playbook_not_packet",
- "score": 0.071,
+ "score": 0.201,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
@@ -5261,25 +5450,44 @@
"reusable_packaging"
],
"matched_desired_concepts": [
- "reusable_packaging"
+ "incident_packet",
+ "reusable_packaging",
+ "severity_assessment"
],
"extra_positive_concepts": [],
- "matched_negative_concepts": [],
- "exclusive_negative_concepts": [],
- "semantic_coverage": 0.04,
+ "matched_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "exclusive_negative_concepts": [
+ "brainstorm_only"
+ ],
+ "semantic_coverage": 0.54,
"support_score": 0,
"lexical_support": 0.429,
- "negative_penalty": 0,
- "coverage_boost": 0.0,
+ "negative_penalty": 0.24,
+ "coverage_boost": 0.06,
"concept_evidence": {
"positive": {
+ "incident_packet": [
+ "incident review"
+ ],
"reusable_packaging": [
"reusable",
"workflow",
"package"
+ ],
+ "severity_assessment": [
+ "incident review"
]
},
- "negative": {}
+ "negative": {
+ "brainstorm_only": [
+ "sketch a future",
+ "for later",
+ "do not package",
+ "do not package a reusable workflow"
+ ]
+ }
}
}
},
@@ -5341,19 +5549,19 @@
"estimated_tokens": 37,
"dev": {
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.667,
+ "recall": 1.0,
"near_neighbor_pass_rate": 1.0,
"should_not_trigger_pass_rate": 1.0
},
"selection_key": [
0,
- 1,
+ 0,
+ -1.0,
-1.0,
-1.0,
-1.0,
- -0.667,
37
],
"holdout": {
@@ -5365,6 +5573,102 @@
"should_not_trigger_pass_rate": 1.0
}
},
+ {
+ "id": "balanced",
+ "label": "Balanced",
+ "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
+ "strategy": "balanced_template",
+ "estimated_tokens": 54,
+ "dev": {
+ "false_positives": 0,
+ "false_negatives": 0,
+ "precision": 1.0,
+ "recall": 1.0,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ },
+ "selection_key": [
+ 0,
+ 0,
+ -1.0,
+ -1.0,
+ -1.0,
+ -1.0,
+ 54
+ ],
+ "holdout": {
+ "false_positives": 0,
+ "false_negatives": 1,
+ "precision": 1.0,
+ "recall": 0.5,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ }
+ },
+ {
+ "id": "boundary",
+ "label": "Boundary",
+ "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
+ "strategy": "boundary_template",
+ "estimated_tokens": 78,
+ "dev": {
+ "false_positives": 0,
+ "false_negatives": 0,
+ "precision": 1.0,
+ "recall": 1.0,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ },
+ "selection_key": [
+ 0,
+ 0,
+ -1.0,
+ -1.0,
+ -1.0,
+ -1.0,
+ 78
+ ],
+ "holdout": {
+ "false_positives": 0,
+ "false_negatives": 1,
+ "precision": 1.0,
+ "recall": 0.5,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ }
+ },
+ {
+ "id": "artifact_aware",
+ "label": "Artifact Aware",
+ "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
+ "strategy": "artifact_template",
+ "estimated_tokens": 78,
+ "dev": {
+ "false_positives": 0,
+ "false_negatives": 0,
+ "precision": 1.0,
+ "recall": 1.0,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ },
+ "selection_key": [
+ 0,
+ 0,
+ -1.0,
+ -1.0,
+ -1.0,
+ -1.0,
+ 78
+ ],
+ "holdout": {
+ "false_positives": 0,
+ "false_negatives": 1,
+ "precision": 1.0,
+ "recall": 0.5,
+ "near_neighbor_pass_rate": 1.0,
+ "should_not_trigger_pass_rate": 1.0
+ }
+ },
{
"id": "guardrail",
"label": "Guardrail",
@@ -5388,102 +5692,6 @@
-0.667,
51
],
- "holdout": {
- "false_positives": 1,
- "false_negatives": 1,
- "precision": 0.5,
- "recall": 0.5,
- "near_neighbor_pass_rate": 0.5,
- "should_not_trigger_pass_rate": 1.0
- }
- },
- {
- "id": "balanced",
- "label": "Balanced",
- "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
- "strategy": "balanced_template",
- "estimated_tokens": 54,
- "dev": {
- "false_positives": 0,
- "false_negatives": 1,
- "precision": 1.0,
- "recall": 0.667,
- "near_neighbor_pass_rate": 1.0,
- "should_not_trigger_pass_rate": 1.0
- },
- "selection_key": [
- 0,
- 1,
- -1.0,
- -1.0,
- -1.0,
- -0.667,
- 54
- ],
- "holdout": {
- "false_positives": 0,
- "false_negatives": 1,
- "precision": 1.0,
- "recall": 0.5,
- "near_neighbor_pass_rate": 1.0,
- "should_not_trigger_pass_rate": 1.0
- }
- },
- {
- "id": "boundary",
- "label": "Boundary",
- "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
- "strategy": "boundary_template",
- "estimated_tokens": 78,
- "dev": {
- "false_positives": 0,
- "false_negatives": 1,
- "precision": 1.0,
- "recall": 0.667,
- "near_neighbor_pass_rate": 1.0,
- "should_not_trigger_pass_rate": 1.0
- },
- "selection_key": [
- 0,
- 1,
- -1.0,
- -1.0,
- -1.0,
- -0.667,
- 78
- ],
- "holdout": {
- "false_positives": 0,
- "false_negatives": 1,
- "precision": 1.0,
- "recall": 0.5,
- "near_neighbor_pass_rate": 1.0,
- "should_not_trigger_pass_rate": 1.0
- }
- },
- {
- "id": "artifact_aware",
- "label": "Artifact Aware",
- "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
- "strategy": "artifact_template",
- "estimated_tokens": 78,
- "dev": {
- "false_positives": 0,
- "false_negatives": 1,
- "precision": 1.0,
- "recall": 0.667,
- "near_neighbor_pass_rate": 1.0,
- "should_not_trigger_pass_rate": 1.0
- },
- "selection_key": [
- 0,
- 1,
- -1.0,
- -1.0,
- -1.0,
- -0.667,
- 78
- ],
"holdout": {
"false_positives": 0,
"false_negatives": 1,
@@ -5508,15 +5716,15 @@
"comparison": {
"winner_vs_current_dev": {
"baseline_false_positives": 0,
- "baseline_false_negatives": 1,
+ "baseline_false_negatives": 0,
"improved_false_positives": 0,
- "improved_false_negatives": 1,
+ "improved_false_negatives": 0,
"false_positive_delta": 0,
"false_negative_delta": 0,
"baseline_precision": 1.0,
"improved_precision": 1.0,
- "baseline_recall": 0.667,
- "improved_recall": 0.667
+ "baseline_recall": 1.0,
+ "improved_recall": 1.0
},
"winner_vs_current_holdout": {
"baseline_false_positives": 0,
@@ -5543,6 +5751,18 @@
"improved_recall": 1.0
},
"winner_vs_baseline_dev": {
+ "baseline_false_positives": 0,
+ "baseline_false_negatives": 0,
+ "improved_false_positives": 0,
+ "improved_false_negatives": 0,
+ "false_positive_delta": 0,
+ "false_negative_delta": 0,
+ "baseline_precision": 1.0,
+ "improved_precision": 1.0,
+ "baseline_recall": 1.0,
+ "improved_recall": 1.0
+ },
+ "winner_vs_baseline_holdout": {
"baseline_false_positives": 0,
"baseline_false_negatives": 1,
"improved_false_positives": 0,
@@ -5551,31 +5771,19 @@
"false_negative_delta": 0,
"baseline_precision": 1.0,
"improved_precision": 1.0,
- "baseline_recall": 0.667,
- "improved_recall": 0.667
- },
- "winner_vs_baseline_holdout": {
- "baseline_false_positives": 0,
- "baseline_false_negatives": 2,
- "improved_false_positives": 0,
- "improved_false_negatives": 1,
- "false_positive_delta": 0,
- "false_negative_delta": -1,
- "baseline_precision": null,
- "improved_precision": 1.0,
- "baseline_recall": 0.0,
+ "baseline_recall": 0.5,
"improved_recall": 0.5
},
"winner_vs_baseline_blind_holdout": {
"baseline_false_positives": 0,
- "baseline_false_negatives": 1,
+ "baseline_false_negatives": 0,
"improved_false_positives": 0,
"improved_false_negatives": 0,
"false_positive_delta": 0,
- "false_negative_delta": -1,
+ "false_negative_delta": 0,
"baseline_precision": 1.0,
"improved_precision": 1.0,
- "baseline_recall": 0.5,
+ "baseline_recall": 1.0,
"improved_recall": 1.0
},
"winner_vs_current_judge_blind_holdout": {
@@ -5648,15 +5856,15 @@
},
"baseline": {
"false_positives": 0,
- "false_negatives": 2,
- "precision": null,
- "recall": 0.0,
+ "false_negatives": 1,
+ "precision": 1.0,
+ "recall": 0.5,
"near_neighbor_pass_rate": 1.0,
"should_not_trigger_pass_rate": 1.0
},
"winner_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.37,
+ "mean_positive_score": 0.45,
"mean_non_trigger_score": 0.019,
"mean_near_neighbor_score": 0.037,
"min_positive_score": 0.08,
@@ -5671,7 +5879,7 @@
},
"current_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.37,
+ "mean_positive_score": 0.45,
"mean_non_trigger_score": 0.019,
"mean_near_neighbor_score": 0.037,
"min_positive_score": 0.08,
@@ -5686,7 +5894,7 @@
},
"baseline_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.36,
+ "mean_positive_score": 0.507,
"mean_non_trigger_score": 0.059,
"mean_near_neighbor_score": 0.119,
"min_positive_score": 0.321,
@@ -5695,8 +5903,8 @@
"threshold_margin": -0.159,
"positive_threshold_buffer": -0.159,
"negative_threshold_buffer": 0.26,
- "boundary_case_count": 1,
- "boundary_case_rate": 0.167,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
"risk_band": "overlap"
},
"winner_family_health": {
@@ -5733,13 +5941,8 @@
},
"baseline_family_health": {
"family_count": 6,
- "clean_family_count": 4,
+ "clean_family_count": 5,
"failing_families": [
- {
- "family": "governed_workflow",
- "errors": 1,
- "pass_rate": 0.0
- },
{
"family": "packet_assembly",
"errors": 1,
@@ -5772,56 +5975,56 @@
},
"baseline": {
"false_positives": 0,
- "false_negatives": 1,
+ "false_negatives": 0,
"precision": 1.0,
- "recall": 0.5,
+ "recall": 1.0,
"near_neighbor_pass_rate": 1.0,
"should_not_trigger_pass_rate": 1.0
},
"winner_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.902,
- "mean_non_trigger_score": 0.168,
- "mean_near_neighbor_score": 0.138,
- "min_positive_score": 0.804,
- "max_non_trigger_score": 0.386,
- "score_gap": 0.418,
- "threshold_margin": 0.094,
- "positive_threshold_buffer": 0.324,
- "negative_threshold_buffer": 0.094,
- "boundary_case_count": 1,
- "boundary_case_rate": 0.167,
+ "mean_positive_score": 0.82,
+ "mean_non_trigger_score": 0.059,
+ "mean_near_neighbor_score": 0.0,
+ "min_positive_score": 0.645,
+ "max_non_trigger_score": 0.228,
+ "score_gap": 0.417,
+ "threshold_margin": 0.165,
+ "positive_threshold_buffer": 0.165,
+ "negative_threshold_buffer": 0.252,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"current_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.902,
- "mean_non_trigger_score": 0.168,
- "mean_near_neighbor_score": 0.138,
- "min_positive_score": 0.804,
- "max_non_trigger_score": 0.386,
- "score_gap": 0.418,
- "threshold_margin": 0.094,
- "positive_threshold_buffer": 0.324,
- "negative_threshold_buffer": 0.094,
- "boundary_case_count": 1,
- "boundary_case_rate": 0.167,
+ "mean_positive_score": 0.82,
+ "mean_non_trigger_score": 0.059,
+ "mean_near_neighbor_score": 0.0,
+ "min_positive_score": 0.645,
+ "max_non_trigger_score": 0.228,
+ "score_gap": 0.417,
+ "threshold_margin": 0.165,
+ "positive_threshold_buffer": 0.165,
+ "negative_threshold_buffer": 0.252,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"baseline_calibration": {
"threshold": 0.48,
- "mean_positive_score": 0.625,
- "mean_non_trigger_score": 0.151,
- "mean_near_neighbor_score": 0.198,
- "min_positive_score": 0.458,
- "max_non_trigger_score": 0.395,
- "score_gap": 0.063,
- "threshold_margin": -0.022,
- "positive_threshold_buffer": -0.022,
- "negative_threshold_buffer": 0.085,
- "boundary_case_count": 2,
- "boundary_case_rate": 0.333,
- "risk_band": "overlap"
+ "mean_positive_score": 0.671,
+ "mean_non_trigger_score": 0.053,
+ "mean_near_neighbor_score": 0.003,
+ "min_positive_score": 0.55,
+ "max_non_trigger_score": 0.176,
+ "score_gap": 0.374,
+ "threshold_margin": 0.07,
+ "positive_threshold_buffer": 0.07,
+ "negative_threshold_buffer": 0.304,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "watch"
},
"winner_family_health": {
"family_count": 6,
@@ -5845,18 +6048,12 @@
},
"baseline_family_health": {
"family_count": 6,
- "clean_family_count": 5,
- "failing_families": [
- {
- "family": "blind_governed_workflow",
- "errors": 1,
- "pass_rate": 0.0
- }
- ],
+ "clean_family_count": 6,
+ "failing_families": [],
"weakest_family": {
- "family": "blind_governed_workflow",
- "pass_rate": 0.0,
- "errors": 1
+ "family": "blind_summary_only",
+ "pass_rate": 1.0,
+ "errors": 0
}
}
},
@@ -5870,7 +6067,7 @@
"should_not_trigger_pass_rate": 1.0,
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -5883,7 +6080,7 @@
"should_not_trigger_pass_rate": 1.0,
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -5896,7 +6093,7 @@
"should_not_trigger_pass_rate": 1.0,
"judge_summary": {
"agreement_rate": 1.0,
- "mean_confidence": 0.69,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -5961,49 +6158,49 @@
},
"winner_calibration": {
"threshold": 0.34,
- "mean_positive_score": 0.827,
- "mean_non_trigger_score": 0.027,
- "mean_near_neighbor_score": 0.011,
- "min_positive_score": 0.654,
- "max_non_trigger_score": 0.056,
- "score_gap": 0.598,
- "threshold_margin": 0.284,
- "positive_threshold_buffer": 0.314,
- "negative_threshold_buffer": 0.284,
- "boundary_case_count": 0,
- "boundary_case_rate": 0.0,
- "risk_band": "healthy"
- },
- "current_calibration": {
- "threshold": 0.34,
- "mean_positive_score": 0.827,
- "mean_non_trigger_score": 0.027,
- "mean_near_neighbor_score": 0.011,
- "min_positive_score": 0.654,
- "max_non_trigger_score": 0.056,
- "score_gap": 0.598,
- "threshold_margin": 0.284,
- "positive_threshold_buffer": 0.314,
- "negative_threshold_buffer": 0.284,
- "boundary_case_count": 0,
- "boundary_case_rate": 0.0,
- "risk_band": "healthy"
- },
- "baseline_calibration": {
- "threshold": 0.34,
- "mean_positive_score": 0.689,
- "mean_non_trigger_score": 0.074,
- "mean_near_neighbor_score": 0.035,
- "min_positive_score": 0.387,
- "max_non_trigger_score": 0.215,
- "score_gap": 0.172,
- "threshold_margin": 0.047,
- "positive_threshold_buffer": 0.047,
- "negative_threshold_buffer": 0.125,
+ "mean_positive_score": 0.907,
+ "mean_non_trigger_score": 0.083,
+ "mean_near_neighbor_score": 0.139,
+ "min_positive_score": 0.815,
+ "max_non_trigger_score": 0.277,
+ "score_gap": 0.538,
+ "threshold_margin": 0.063,
+ "positive_threshold_buffer": 0.475,
+ "negative_threshold_buffer": 0.063,
"boundary_case_count": 1,
"boundary_case_rate": 0.167,
"risk_band": "watch"
},
+ "current_calibration": {
+ "threshold": 0.34,
+ "mean_positive_score": 0.907,
+ "mean_non_trigger_score": 0.083,
+ "mean_near_neighbor_score": 0.139,
+ "min_positive_score": 0.815,
+ "max_non_trigger_score": 0.277,
+ "score_gap": 0.538,
+ "threshold_margin": 0.063,
+ "positive_threshold_buffer": 0.475,
+ "negative_threshold_buffer": 0.063,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "watch"
+ },
+ "baseline_calibration": {
+ "threshold": 0.34,
+ "mean_positive_score": 0.836,
+ "mean_non_trigger_score": 0.107,
+ "mean_near_neighbor_score": 0.101,
+ "min_positive_score": 0.682,
+ "max_non_trigger_score": 0.215,
+ "score_gap": 0.467,
+ "threshold_margin": 0.125,
+ "positive_threshold_buffer": 0.342,
+ "negative_threshold_buffer": 0.125,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
+ "risk_band": "healthy"
+ },
"winner_family_health": {
"family_count": 6,
"clean_family_count": 6,
@@ -6040,8 +6237,8 @@
"winner_label": "Current",
"winner_tokens": 37,
"current_tokens": 37,
- "winner_dev_total_errors": 1,
- "current_dev_total_errors": 1,
+ "winner_dev_total_errors": 0,
+ "current_dev_total_errors": 0,
"winner_holdout_total_errors": 1,
"current_holdout_total_errors": 1,
"winner_blind_holdout_total_errors": 0,
@@ -6051,13 +6248,13 @@
"winner_adversarial_holdout_total_errors": 0,
"current_adversarial_holdout_total_errors": 0,
"winner_judge_blind_agreement_rate": 1.0,
- "winner_adversarial_risk_band": "healthy",
- "winner_adversarial_score_gap": 0.598,
+ "winner_adversarial_risk_band": "watch",
+ "winner_adversarial_score_gap": 0.538,
"candidate_count": 5,
"baseline_tokens": 93,
- "baseline_dev_total_errors": 1,
- "baseline_holdout_total_errors": 2,
- "baseline_blind_holdout_total_errors": 1,
+ "baseline_dev_total_errors": 0,
+ "baseline_holdout_total_errors": 1,
+ "baseline_blind_holdout_total_errors": 0,
"baseline_judge_blind_holdout_total_errors": 0,
"baseline_adversarial_holdout_total_errors": 0
}
diff --git a/examples/governed-incident-command/optimization/reports/description_optimization.md b/examples/governed-incident-command/optimization/reports/description_optimization.md
index 154dab1a..874d859a 100644
--- a/examples/governed-incident-command/optimization/reports/description_optimization.md
+++ b/examples/governed-incident-command/optimization/reports/description_optimization.md
@@ -14,18 +14,18 @@ Build governed incident command packets. Use when asked to standardize incident
| Candidate | Tokens | Dev FP | Dev FN | Dev Near | Holdout FP | Holdout FN |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| `Current` | 37 | 0 | 1 | 1.0 | 0 | 1 |
-| `Guardrail` | 51 | 0 | 1 | 1.0 | 1 | 1 |
-| `Balanced` | 54 | 0 | 1 | 1.0 | 0 | 1 |
-| `Boundary` | 78 | 0 | 1 | 1.0 | 0 | 1 |
-| `Artifact Aware` | 78 | 0 | 1 | 1.0 | 0 | 1 |
+| `Current` | 37 | 0 | 0 | 1.0 | 0 | 1 |
+| `Balanced` | 54 | 0 | 0 | 1.0 | 0 | 1 |
+| `Boundary` | 78 | 0 | 0 | 1.0 | 0 | 1 |
+| `Artifact Aware` | 78 | 0 | 0 | 1.0 | 0 | 1 |
+| `Guardrail` | 51 | 0 | 1 | 1.0 | 0 | 1 |
## Acceptance Gates
| Gate | Winner FP | Winner FN | Current FP | Current FN | Baseline FP | Baseline FN |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| Holdout | 0 | 1 | 0 | 1 | 0 | 2 |
-| Blind Holdout | 0 | 0 | 0 | 0 | 0 | 1 |
+| Holdout | 0 | 1 | 0 | 1 | 0 | 1 |
+| Blind Holdout | 0 | 0 | 0 | 0 | 0 | 0 |
| Judge Blind Holdout | 0 | 0 | 0 | 0 | 0 | 0 |
| Adversarial Holdout | 0 | 0 | 0 | 0 | 0 | 0 |
@@ -34,21 +34,21 @@ Build governed incident command packets. Use when asked to standardize incident
| Gate | Winner Gap | Winner Risk | Winner Boundary Rate | Current Gap | Baseline Gap |
| --- | ---: | --- | ---: | ---: | ---: |
| Holdout | 0.023 | overlap | 0.0 | 0.023 | 0.101 |
-| Blind Holdout | 0.418 | healthy | 0.167 | 0.418 | 0.063 |
-| Adversarial Holdout | 0.598 | healthy | 0.0 | 0.598 | 0.172 |
+| Blind Holdout | 0.417 | healthy | 0.0 | 0.417 | 0.374 |
+| Adversarial Holdout | 0.538 | watch | 0.167 | 0.538 | 0.467 |
## Judge Blind Summary
| Gate | Winner Agreement | Winner Mean Confidence | Current Agreement | Baseline Agreement |
| --- | ---: | ---: | ---: | ---: |
-| Judge Blind Holdout | 1.0 | 0.657 | 1.0 | 1.0 |
+| Judge Blind Holdout | 1.0 | 0.677 | 1.0 | 1.0 |
## Family Health
| Gate | Winner Clean Families | Winner Weakest Family | Current Clean Families | Baseline Clean Families |
| --- | --- | --- | --- | --- |
-| Holdout | 5/6 | packet_assembly (1 errors) | 5/6 | 4/6 |
-| Blind Holdout | 6/6 | blind_summary_only (0 errors) | 6/6 | 5/6 |
+| Holdout | 5/6 | packet_assembly (1 errors) | 5/6 | 5/6 |
+| Blind Holdout | 6/6 | blind_summary_only (0 errors) | 6/6 | 6/6 |
| Judge Blind Holdout | 6/6 | blind_summary_only (0 errors) | 6/6 | 6/6 |
| Adversarial Holdout | 6/6 | adversarial_single_update_collision (0 errors) | 6/6 | 6/6 |
diff --git a/examples/governed-incident-command/optimization/semantic_config.json b/examples/governed-incident-command/optimization/semantic_config.json
index 8cd90699..20cdcf21 100644
--- a/examples/governed-incident-command/optimization/semantic_config.json
+++ b/examples/governed-incident-command/optimization/semantic_config.json
@@ -37,9 +37,12 @@
"weight": 0.32,
"phrases": [
"incident command packet",
+ "incident command packets",
"governed incident packet",
"incident packet assembly",
- "incident review skill"
+ "incident review skill",
+ "incident review",
+ "standardize incident review"
]
},
"incident_inputs": {
@@ -59,6 +62,8 @@
"severity assessment",
"severity",
"impact scope",
+ "incident review",
+ "outage review",
"sev"
]
},
@@ -68,6 +73,8 @@
"stakeholder update",
"incident communication",
"owner",
+ "owners",
+ "owner actions",
"action ownership",
"next actions"
]
@@ -76,6 +83,9 @@
"weight": 0.1,
"phrases": [
"governed",
+ "governed workflow",
+ "reusable governed workflow",
+ "standardize",
"reviewed",
"audited",
"maintained",
@@ -128,9 +138,13 @@
"exclusive": true,
"phrases": [
"brainstorm",
+ "sketch a future",
+ "for later",
"possible response ideas",
"improve incident handling",
- "ideas for later"
+ "ideas for later",
+ "do not package",
+ "do not package a reusable workflow"
]
},
"summary_only": {
diff --git a/manifest.json b/manifest.json
index 91767255..317a1a11 100644
--- a/manifest.json
+++ b/manifest.json
@@ -12,7 +12,8 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
],
"factory_components": [
"templates",
diff --git a/references/autonomous-adaptation.md b/references/autonomous-adaptation.md
new file mode 100644
index 00000000..6f3684dd
--- /dev/null
+++ b/references/autonomous-adaptation.md
@@ -0,0 +1,65 @@
+# Autonomous Adaptation Method
+
+This reference defines the safe foundation for adaptive self-iteration.
+
+## Scope
+
+Adaptive iteration is proposal-only until a human explicitly approves a patch application workflow. The current implementation may:
+
+- read one user-provided local source file;
+- redact sensitive text before storing evidence excerpts;
+- summarize repeated preferences and operational signals;
+- produce adaptation proposals with target files, risks, tests, and rollback plans.
+- draft a pending approval ledger entry from a reviewed patch, including patch SHA-256, target files, target baseline hashes, verifier commands, and rollback metadata.
+- dry-run an approved patch through `adapt-apply`, after patch hash, approval, target allowlist, and target baseline hash checks pass.
+- apply a patch only when the operator passes `--apply` and the approval ledger names the reviewer, reason, patch hash, target files, target file SHA-256 baselines, verification commands, and rollback plan.
+- automatically reverse an applied patch when `--run-verification` fails, unless the operator explicitly passes `--no-rollback-on-failure`.
+
+It must not:
+
+- scan shell history, browser history, chat logs, mail, or private folders by default;
+- infer permanent user memory from a single comment;
+- write source files as part of scan or proposal generation;
+- write source files through `adapt-apply` without explicit `--apply`;
+- apply a patch whose target files are outside both the proposal and approval allowlists;
+- apply a patch when an approved target file has changed since the reviewer recorded its baseline SHA-256;
+- leave a failed verified apply in place by default;
+- count proposals as completed implementation evidence.
+
+## Flow
+
+1. `adapt-scan` reads an explicit source path and writes `reports/user_patterns.json` plus `reports/user_patterns.md`.
+2. `adapt-propose` reads the pattern report and writes `reports/adaptation_proposals.json` plus `reports/adaptation_proposals.md`.
+3. A reviewer decides whether any proposal is worth implementing.
+4. `adapt-apply --write-template` creates `reports/adaptation_approval_ledger.json` and `reports/adaptation_regression_report.json` so the review surface exists before any patch is applied.
+5. `adapt-apply --prepare-approval --proposal-id --patch-file ` drafts a `pending-review` approval entry. It does not approve or apply the patch.
+6. A human reviewer changes the draft decision to `approved`, fills reviewer, reason, approval date, and optional expiry, then keeps the generated patch and target baseline hashes intact.
+7. `adapt-apply --proposal-id --patch-file ` defaults to a dry-run and records patch, target, approval, regression, and rollback evidence.
+8. `adapt-apply --apply --run-verification` may write files only after approval, patch hash, allowlist, target baseline hash, `git apply --check`, and safe regression command checks pass.
+9. If a verification command fails after a patch is applied, `adapt-apply` runs `git apply -R ` by default and records `failed-rolled-back` plus rollback evidence in `reports/adaptation_regression_report.json`.
+
+## Evidence Standard
+
+Each proposal should include:
+
+- the repeated pattern that triggered it;
+- redacted excerpts, never unredacted raw content;
+- target files and change intent;
+- risk level and boundary;
+- verification commands;
+- rollback plan;
+- a clear `proposal-only` status.
+
+Each approved application should include:
+
+- reviewer, reason, approval date, and optional expiry;
+- exact patch SHA-256;
+- target file allowlist;
+- target file SHA-256 baselines for every patch target, or `__absent__` for approved new files;
+- regression commands restricted to local `make` targets or local Python verifier scripts;
+- rollback command or plan.
+- rollback result if regression failed after an apply attempt.
+
+## Review Boundary
+
+The adaptive loop improves iteration quality, but it does not replace normal review. Any proposal touching trigger behavior, reports, packaging, telemetry, privacy, or governance must still pass the same tests and release gates as a manually designed change. `adapt-apply` evidence proves that an approved patch path was checked or applied; it does not make world-class external or human evidence complete.
diff --git a/references/distribution-registry-method.md b/references/distribution-registry-method.md
index 68bb0521..4511b58e 100644
--- a/references/distribution-registry-method.md
+++ b/references/distribution-registry-method.md
@@ -25,7 +25,7 @@ Do not publish a team package when registry audit reports missing version, hash,
For an installable archive, first build the distribution and run package verification:
```bash
-python3 scripts/yao.py package . --platform openai --platform claude --platform generic --output-dir dist --zip
+python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
python3 scripts/yao.py package-verify . --package-dir dist --require-zip
python3 scripts/yao.py install-simulate . --package-dir dist
python3 scripts/yao.py registry-audit .
@@ -36,6 +36,8 @@ Do not claim archive readiness when package verification reports unsafe zip path
Do not claim install readiness when install simulation cannot extract the archive into a temporary skill root, load `SKILL.md` frontmatter, read `manifest.json`, read `agents/interface.yaml`, find the overview and Review Studio reports, or load each generated adapter.
+Do not sync a local or active install from source until the same package has passed install preflight. `scripts/sync_local_install.py` must run install simulation against the configured package directory and fail before copying files when any target/capability pair lacks active permission approval or target-specific enforcement evidence. Use `--skip-install-preflight` only for isolated diagnostics, not for release or active install.
+
Do not include raw `reports/telemetry_events.jsonl` in a distributed package. Include only aggregate adoption drift reports, and block release review when telemetry contains raw prompts, outputs, transcripts, notes, or messages.
Review waiver evidence may be distributed as `reports/review_waivers.md/json` because it is metadata-only reviewer accountability. Do not store raw prompts, outputs, transcripts, credentials, or private customer detail in waiver reasons.
@@ -53,5 +55,6 @@ A reviewer should be able to answer:
5. Which reports prove trust and runtime readiness?
6. Was the installable archive verified, and which checksum identifies it?
7. Was the archive install-simulated in a temporary local skill root?
-8. What changed since the previous package, and does the declared version bump match the recommended bump?
-9. Are adoption and drift signals summarized without packaging raw local telemetry?
+8. Did local or active install sync preserve that preflight and installer permission gate?
+9. What changed since the previous package, and does the declared version bump match the recommended bump?
+10. Are adoption and drift signals summarized without packaging raw local telemetry?
diff --git a/references/operating-modes.md b/references/operating-modes.md
index 766b4d1e..a6e3c9de 100644
--- a/references/operating-modes.md
+++ b/references/operating-modes.md
@@ -68,11 +68,37 @@ Minimum gates:
- `trigger_eval.py`
- `cross_packager.py` for requested targets
+## Governed
+
+Use when:
+
+- the skill affects incident, release, compliance, security, or organizational standards
+- external distribution, public claims, or high-permission scripts require reviewable evidence
+- wrong output or wrong activation can cause operational, legal, trust, or reputational harm
+
+Default deliverables:
+
+- everything required for Library
+- explicit owner, lifecycle, review cadence, and expiry-aware approvals
+- trust/security reports for scripts, dependencies, permissions, secrets, and package hash
+- output eval evidence with blind review status and reviewer-visible boundaries
+- world-class or public-claim evidence ledger when public readiness is claimed
+
+Minimum gates:
+
+- Library gates
+- `trust_check.py`
+- runtime permission probes for packaged adapters
+- review waiver ledger for accepted warning-level risk
+- Review Studio before release
+- claim guard before public world-class language
+
## Escalation Rules
- stay in Scaffold unless reuse is clearly real
- move to Production when team reuse or route confusion matters
-- move to Library when the skill becomes shared infrastructure or a governed asset
+- move to Library when the skill becomes shared infrastructure
+- move to Governed when the skill needs explicit risk ownership, high-permission review, or public-claim evidence
## Context Discipline
diff --git a/references/output-eval-method.md b/references/output-eval-method.md
index d9d999c3..dfb8d443 100644
--- a/references/output-eval-method.md
+++ b/references/output-eval-method.md
@@ -69,6 +69,16 @@ python3 scripts/yao.py output-exec --runner-command '["python3","scripts/local_o
This verifies the command-runner contract, timing capture, grading path, and failure handling. It must not be described as provider-backed model evidence.
+For provider-backed evidence, use the bundled provider runner with real credentials:
+
+```bash
+YAO_OUTPUT_EVAL_MODEL=gpt-4.1-mini \
+OPENAI_API_KEY=... \
+python3 scripts/yao.py output-exec --provider-runner openai
+```
+
+The provider runner calls an OpenAI Responses API compatible endpoint, reads input files relative to `evals/output/`, returns `execution_kind: "model"`, and records observed token usage when the provider returns usage fields. If the API key or model is missing, the runner must fail instead of falling back to fixtures or pretending model evidence exists. Use `--provider-base-url` only for reviewed compatible endpoints; non-default HTTPS hosts require `--allow-custom-base-url`, and plain HTTP is allowed only with `--allow-insecure-localhost` for local test servers.
+
## Blind A/B Review
Every output eval run should also generate:
@@ -81,7 +91,7 @@ The review pack must hide whether Variant A or Variant B came from the baseline
## Reviewer Adjudication
-After blind review, record reviewer choices in `reports/output_review_decisions.json` and run:
+After blind review, record reviewer choices in `reports/output_review_decisions.json` with `reviewer`, `reviewed_at`, `winner_variant`, optional `confidence`, and a required rubric-based `reason`, then run:
```bash
python3 scripts/adjudicate_output_review.py --write-template
@@ -90,10 +100,13 @@ python3 scripts/yao.py output-review
The adjudication report writes:
+- `reports/output_review_decisions.json`
- `reports/output_review_adjudication.json`
- `reports/output_review_adjudication.md`
-When no reviewer decisions exist, the report should say the cases are pending. Do not count pending cases as human agreement. Only a real `winner_variant` of `A` or `B` should contribute to agreement rate, disagreement count, and reviewer judgment count.
+When no reviewer decisions exist, the report should say the cases are pending and Review Studio should link to the decisions template. Do not count pending cases as human agreement. Only a real `winner_variant` of `A` or `B` with reviewer metadata and a non-empty `reason` should contribute to agreement rate, disagreement count, and reviewer judgment count.
+
+The adjudication report must preserve blind-review integrity: pending and invalid decisions should show the expected winner as hidden. Only reveal `expected_winner_variant` after a valid reviewer decision with rationale exists for that case.
## Anti-Overfitting
diff --git a/references/platform-capability-matrix.md b/references/platform-capability-matrix.md
index 101763db..57f5fa94 100644
--- a/references/platform-capability-matrix.md
+++ b/references/platform-capability-matrix.md
@@ -8,6 +8,7 @@ This matrix describes the current packaging targets and their support level.
| `claude` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior | Generates `targets/claude/README.md` plus adapter metadata |
| `generic` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior | Uses neutral adapter metadata only |
| `agent-skills-compatible` | Neutral source | Yes | Yes | Source-compatible | Yes | activation, execution, trust, permissions, degradation, native behavior | Keeps canonical `SKILL.md` plus `agents/interface.yaml` source shape |
+| `vscode` | Yes | Yes | Yes | Yes | Yes | activation, execution, trust, permissions, degradation, native behavior, install scope | Generates `targets/vscode/README.md` plus adapter metadata for VS Code / Copilot Agent Skills review |
## Current Support Model
@@ -15,6 +16,7 @@ This matrix describes the current packaging targets and their support level.
- `claude`: lightweight compatibility adapter with an explicit compiler contract and fallback notes.
- `generic`: lowest-friction export for neutral Agent Skills consumers.
- `agent-skills-compatible`: canonical source shape with compiler evidence for review and distribution.
+- `vscode`: VS Code / Copilot Agent Skills adapter that preserves the neutral source package and documents user/project scope plus workspace-trust review notes.
- runtime permission probes currently report metadata fallback for generated targets; no target is claimed as native-enforced until a client or installer integration can actually enforce the permission model.
## Portable Semantics
diff --git a/references/review-studio-method.md b/references/review-studio-method.md
index da67a4cb..4674fbd4 100644
--- a/references/review-studio-method.md
+++ b/references/review-studio-method.md
@@ -40,7 +40,7 @@ Production, library, and governed reviews should also show a blind A/B review pa
When `reports/output_execution_runs.json` exists, Review Studio should show the number of variant runs, command-executed runs, model-executed runs, recorded fixtures, timing-observed runs, and token-estimated runs. Recorded fixtures are valid reproducibility evidence, but they must not be described as model-executed output evidence.
-When `reports/output_review_adjudication.json` exists, Review Studio should show reviewed pairs and pending pairs. Pending reviewer decisions are acceptable as an explicit state, but they must not be counted as agreement or human review evidence. Invalid adjudication records should block release because they make the blind review audit untrustworthy.
+When `reports/output_review_adjudication.json` exists, Review Studio should show reviewed pairs and pending pairs. Pending reviewer decisions are acceptable as an explicit state, but they must not be counted as agreement or human review evidence. For production, library, and governed packages, pending reviewer decisions should keep the Output Lab in `warn` until reviewer decisions are recorded or the warning is explicitly accepted in the waiver ledger. Invalid adjudication records should block release because they make the blind review audit untrustworthy.
The Operations Loop must never display raw telemetry logs. It should link only to `reports/adoption_drift_report.md`; privacy or schema violations are blockers.
@@ -65,10 +65,12 @@ Each action must include:
- `evidence`
- `verification_command`
-`source_refs` must be structured entries with relative `path`, human label, kind, existence flag, best-effort line number, and relative link when the file exists. They should point to the smallest useful report or source file, not just a broad directory.
+`source_refs` must be structured entries with relative `path`, human label, kind, existence flag, best-effort line number, matched pattern, short source excerpt, and relative link when the file exists. They should point to the smallest useful report or source file, not just a broad directory. The HTML page should render the excerpt next to the link so reviewers can understand why a line anchor matters before opening the full artifact.
The HTML page should render these actions before the detailed supporting sections so a reviewer can move directly from warning to fix. Action entries do not change gate count or score; they make the current decision more operational.
+For `world-class-evidence`, the action should also expose an evidence-step card for every pending evidence key. Each card should show the submission path, template path, blocked source checks, command handoff, first runbook steps, provenance requirements, success checks, evidence artifacts, and privacy boundary. These cards are collection guidance only; they must not count as accepted evidence or change world-class readiness.
+
## Review Annotations
`reports/review_annotations.json` is the structured ledger, and `reports/review_annotations.md` is the human-readable review note surface. Each annotation should include:
diff --git a/references/review-waiver-method.md b/references/review-waiver-method.md
index 5872d830..84d3fc78 100644
--- a/references/review-waiver-method.md
+++ b/references/review-waiver-method.md
@@ -25,6 +25,16 @@ Every waiver must include:
- `evidence`: optional path or note that explains the decision.
- `scope`: default `current-release`.
+## Gate Key Policy
+
+The waiver ledger must track the Review Studio gate universe explicitly:
+
+- `review_studio_gate_keys`: every gate Review Studio can render.
+- `waiverable_gate_keys`: warning gates that may receive bounded human acceptance.
+- `non_waivable_gate_keys`: gates that must not be accepted through a waiver.
+
+When Review Studio adds or renames a gate, update the waiver gate policy and tests in the same change. `review-waivers` and `world-class-evidence` stay non-waivable: the first is the waiver mechanism itself, and the second can only be satisfied by accepted ledger evidence.
+
## Release Semantics
- Invalid waiver records block Review Studio.
@@ -55,3 +65,12 @@ python3 scripts/yao.py review-waivers . \
For a non-governed release where `permission-gates` is only a warning, the same command can name `--gate-key permission-gates`. Governed releases must instead provide reviewer, scope, reason, expiry, evidence, and target-enforcement fields in `security/permission_policy.json`.
Review Studio reads `reports/review_waivers.json` and links to `reports/review_waivers.md`.
+
+## Candidate Actions
+
+The waiver report also surfaces current candidate actions from local evidence:
+
+- waiverable warning candidates, such as an `output-lab` warning caused by pending reviewer decisions or missing provider-backed runs
+- non-waivable boundaries, especially `world-class-evidence`, where pending ledger evidence cannot be converted into completion by a waiver
+
+A waiver can make a bounded warning auditable for a release window. It cannot count as provider-backed evidence, human adjudication, native runtime enforcement, external telemetry, or public world-class readiness.
diff --git a/references/skill-atlas-method.md b/references/skill-atlas-method.md
index a0f0455a..9c580434 100644
--- a/references/skill-atlas-method.md
+++ b/references/skill-atlas-method.md
@@ -16,6 +16,7 @@ Single-skill quality is not enough for a team library. A skill portfolio also ne
- Flag missing owner or review metadata.
- Flag stale skills based on `updated_at` and `review_cadence`.
- Extract no-route opportunities from failure notes.
+- Read aggregate adoption drift reports and flag telemetry drift without reading raw telemetry logs.
## Scope Policy
@@ -23,6 +24,12 @@ Atlas keeps a full catalog, but release gates should distinguish actionable libr
Use `skill_atlas/policy.json` to mark path prefixes as non-actionable when they are intentionally retained as examples, evolution snapshots, embedded generated skills, or validator fixtures. Non-actionable items still appear in the full report, route matrix, stale list, and owner gap list, but Review Studio should use the actionable counts for release readiness.
+## Telemetry Link
+
+Atlas may read each skill's aggregate `reports/adoption_drift_report.json` to surface portfolio drift signals such as no telemetry for production/library/governed skills, missed triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not read or package `reports/telemetry_events.jsonl`; raw telemetry remains local-only evidence owned by the skill.
+
+Write drift output to `skill_atlas/drift_signals.json`. Non-actionable scopes stay visible in that file and in the HTML report, but only actionable drift signals should affect release readiness.
+
## Reviewer Gate
-Use Atlas before promoting a single skill into a shared library. If an actionable route collision, missing owner, or stale governed skill appears, fix the portfolio boundary before adding more local complexity to one skill. Non-actionable issues should stay visible as evidence, not as release blockers.
+Use Atlas before promoting a single skill into a shared library. If an actionable route collision, missing owner, stale governed skill, or telemetry drift signal appears, fix the portfolio boundary before adding more local complexity to one skill. Non-actionable issues should stay visible as evidence, not as release blockers.
diff --git a/references/skillops-decision-policy.md b/references/skillops-decision-policy.md
new file mode 100644
index 00000000..9e3e7471
--- /dev/null
+++ b/references/skillops-decision-policy.md
@@ -0,0 +1,53 @@
+# SkillOps Decision Policy
+
+Use this policy when turning explicit-source conversation evidence into SkillOps opportunities, proposals, or release work. The goal is to make repeated user signals actionable without letting automation write durable instructions, skills, scripts, or evals without review.
+
+## Decision Order
+
+1. Classify the signal as no action, report only, Memory, AGENTS.md, existing Skill patch, candidate Skill, script, eval, report, merge, split, or archive.
+2. Prefer the smallest durable surface that fixes the repeated friction.
+3. Require evidence for every write action.
+4. Require a proposal or approval ledger entry before any source-file write.
+5. Map every proposed change to at least one verification command.
+
+## Score Bands
+
+SkillOps opportunities use a `0-100` score. Scores are advisory and never bypass approval.
+
+| Score | Decision |
+| ---: | --- |
+| `85-100` | Ready for approval review |
+| `70-84` | Proposal review |
+| `50-69` | Observe more evidence |
+| `0-49` | Report only or no action |
+
+High-risk items stay proposal-only even when their score is high.
+
+## Action Mapping
+
+| Pattern | Default Action | Durable Surface |
+| --- | --- | --- |
+| `language_default` | Patch existing skill | Report template or artifact doctrine |
+| `report_ui` | Patch existing skill | Report renderer, artifact doctrine, visual test |
+| `approval_safety` | AGENTS update | Governance guidance or approval policy |
+| `delivery_format` | Patch existing skill | CLI output, README, generated summary copy |
+| `evidence_testing` | Add eval | Focused regression, report-quality, or release gate |
+| Unknown pattern | Report only | Manual review queue |
+
+## Safety Rules
+
+- Do not scan private logs implicitly; only use explicit user-supplied sources.
+- Do not store raw conversation content in reports; use redacted excerpts and aggregate counts.
+- Do not write source files from a daily report run.
+- Do not count SkillOps reports as public world-class evidence.
+- Do not treat planned work, draft submissions, or generated proposals as accepted evidence.
+
+## Verification
+
+Every implementation that changes this policy should run:
+
+```bash
+python3 tests/verify_skillops_opportunity.py
+python3 tests/verify_daily_skillops.py
+python3 tests/verify_yao_cli.py
+```
diff --git a/references/telemetry-drift-method.md b/references/telemetry-drift-method.md
index 86331084..32e1bf1d 100644
--- a/references/telemetry-drift-method.md
+++ b/references/telemetry-drift-method.md
@@ -17,6 +17,8 @@ The local event stream is `reports/telemetry_events.jsonl`. It is intentionally
"event": "skill_activation",
"skill": "example-skill",
"version": "2.0.0",
+ "source": "yao_cli",
+ "command": "quickstart",
"activation_type": "implicit",
"outcome": "accepted",
"failure_type": "none",
@@ -26,10 +28,125 @@ The local event stream is `reports/telemetry_events.jsonl`. It is intentionally
Allowed events: `skill_activation`, `skill_output`, `script_run`, `review_event`.
+Allowed sources: `manual`, `yao_cli`, `external`, `unknown`.
+
Allowed outcomes: `accepted`, `edited`, `rejected`, `missed`, `failed`, `reviewed`, `unknown`.
Allowed failure types: `wrong_trigger`, `under_trigger`, `bad_output`, `missing_resource`, `script_error`, `review_overdue`, `none`.
+`source` and `command` are metadata fields. They may identify that `yao.py` ran `quickstart`, `validate`, `output-exec`, or another subcommand, but they must not include arguments, prompt text, file content, model output, transcripts, or reviewer notes.
+
+## CLI Capture
+
+`scripts/yao.py` can record metadata-only `script_run` events automatically. It is opt-in to keep release evidence reproducible and avoid surprising local writes:
+
+```bash
+YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate .
+```
+
+Optional destination override:
+
+```bash
+YAO_CLI_TELEMETRY=1 \
+YAO_CLI_TELEMETRY_EVENTS=/tmp/yao-telemetry.jsonl \
+python3 scripts/yao.py output-exec
+```
+
+Equivalent global flags are available before the subcommand:
+
+```bash
+python3 scripts/yao.py --record-cli-telemetry validate .
+python3 scripts/yao.py --no-cli-telemetry validate .
+```
+
+Successful CLI runs record `event=script_run`, `source=yao_cli`, `outcome=accepted`, and `failure_type=none`. Failed CLI runs record `outcome=failed` and `failure_type=script_error`. The command name is normalized to the subcommand only; command arguments are never recorded.
+
+## External Client Emit
+
+External clients, browser extensions, editor adapters, or wrapper scripts can emit one sanitized event at a time into a local spool before importing it into the aggregate drift report:
+
+```bash
+python3 scripts/yao.py telemetry-emit . \
+ --event skill_activation \
+ --activation-type explicit \
+ --outcome accepted \
+ --command browser-extension
+```
+
+By default this writes to `.yao/telemetry_spool/external_events.jsonl`. Use `--output-jsonl` when a client needs a different local handoff path:
+
+```bash
+python3 scripts/yao.py telemetry-emit . \
+ --output-jsonl /tmp/external-client-events.jsonl \
+ --event skill_output \
+ --activation-type manual \
+ --outcome edited \
+ --command browser-plugin
+```
+
+Use `--dry-run` to validate a proposed event without writing to the spool. The emitter uses the same metadata-only contract as import: no prompt, input, output, transcript, message, note, raw text, arguments, or unknown fields are accepted.
+
+After a client finishes a batch, import the spool:
+
+```bash
+python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
+```
+
+## Client Hook Recipes
+
+Use `telemetry-hooks` to generate auditable Browser, Chrome, VS Code, CLI wrapper, and provider-adapter hook recipes:
+
+```bash
+python3 scripts/yao.py telemetry-hooks .
+```
+
+The report is written to:
+
+- `reports/telemetry_hook_recipes.json`
+- `reports/telemetry_hook_recipes.md`
+
+Each recipe includes a dry-run command, an emit command, the target local spool, trigger points, and the privacy contract. The report intentionally sets `native_auto_capture=false`; it proves the local hook contract and metadata-only command shape, not that a host client is already natively integrated.
+
+## Browser Native Host
+
+`scripts/telemetry_native_host.py` implements the local side of Browser/Chrome Native Messaging. It accepts length-prefixed JSON messages on stdio, validates them with the same metadata-only telemetry contract, appends accepted events to the local spool, and rejects raw prompt/output/transcript/message/note fields.
+
+Smoke-test one message without Browser installation:
+
+```bash
+python3 scripts/telemetry_native_host.py . \
+ --message-json '{"event":"skill_activation","activation_type":"explicit","outcome":"accepted","failure_type":"none","command":"chrome-native-host"}'
+```
+
+Generate a local launcher and Chrome native messaging manifest for an operator-installed extension:
+
+```bash
+python3 scripts/telemetry_native_host.py . \
+ --write-launcher /tmp/yao-telemetry-host.sh \
+ --write-manifest /tmp/yao-telemetry-host.json \
+ --allowed-origin chrome-extension://aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/
+```
+
+This is an executable native host bridge and manifest generator. It still does not prove that a specific Browser/Chrome extension is installed or sending events in the user's environment.
+
+## External Client Import
+
+External clients, browser extensions, editor adapters, or wrapper scripts may hand off already-sanitized JSONL through `telemetry-import`:
+
+```bash
+python3 scripts/yao.py telemetry-import . \
+ --input-jsonl /tmp/external-client-events.jsonl \
+ --command browser-extension
+```
+
+The importer defaults missing `source` to `external` and missing `command` to `external-client`. It validates the entire JSONL file before writing anything. If any line includes a raw content field, unsupported source, unsupported outcome, unsupported failure type, unknown field, malformed JSON, or an unsafe command name, the whole import is rejected and the existing local event stream is left untouched.
+
+Use `--dry-run` to validate an external batch without writing `reports/telemetry_events.jsonl` or refreshing aggregate reports:
+
+```bash
+python3 scripts/yao.py telemetry-import . --input-jsonl /tmp/external-client-events.jsonl --dry-run
+```
+
## Privacy Rule
The raw JSONL event log is local evidence and should not be distributed in skill packages. The distributable artifact is the aggregate report:
@@ -48,13 +165,18 @@ Package builders should exclude `reports/telemetry_events.jsonl`. The root repos
## Iteration Loop
-1. Capture metadata-only events locally.
+1. Capture metadata-only events locally, either manually with `adoption-drift --record-event`, automatically with opt-in `yao.py` CLI capture, through `telemetry-emit` client hooks, through generated `telemetry-hooks` client recipes, or through validated external JSONL import.
2. Render `reports/adoption_drift_report.md`.
3. Convert missed triggers into trigger eval cases.
4. Convert bad outputs into Output Eval assertions and failure taxonomy entries.
5. Convert script errors into non-interactive smoke tests.
6. Feed review-overdue signals back into Skill Atlas and owner review.
+7. Let Skill Atlas read only `reports/adoption_drift_report.json` and publish portfolio-level `skill_atlas/drift_signals.json`.
## Review Studio Role
Review Studio should show the aggregate telemetry gate as an operating loop, not as raw logs. A blocker means the telemetry contract was violated. A warning means the evidence is absent or the drift signal needs a follow-up case.
+
+## Skill Atlas Role
+
+Skill Atlas uses aggregate adoption drift reports to rank portfolio work. It should surface no-data warnings for actionable production/library/governed skills, and drift warnings for missed triggers, wrong triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not inspect raw JSONL telemetry or use non-actionable example/fixture signals as release blockers.
diff --git a/references/trust-security-method.md b/references/trust-security-method.md
index ad6c9f54..a55e0d88 100644
--- a/references/trust-security-method.md
+++ b/references/trust-security-method.md
@@ -64,6 +64,7 @@ Permission approval validates reviewer intent. Runtime permission probes validat
- Run `python3 scripts/probe_runtime_permissions.py . --package-dir dist` after `cross_packager.py`.
- The probe writes `reports/runtime_permission_probes.json` and `reports/runtime_permission_probes.md`.
- A passing probe requires every target adapter to carry `permission_contract`, `target_permission_contract`, declared capabilities, a native-enforcement boolean, representation notes, and operator notes.
+- When `reports/install_simulation.json` matches the same package directory, the probe also reports installer enforcement counts from the install simulation. This proves the local package installer gate is wired, but it does not count as target-client native enforcement.
- If a target has no native enforcement, the probe must mark an explicit metadata fallback and keep residual risk reviewer-visible.
- Review Studio surfaces this as the `permission-runtime` gate.
diff --git a/references/user-memory-policy.md b/references/user-memory-policy.md
new file mode 100644
index 00000000..fd363856
--- /dev/null
+++ b/references/user-memory-policy.md
@@ -0,0 +1,35 @@
+# User Memory Policy
+
+This skill treats user preference memory as local, explicit, and reviewable.
+
+## Principles
+
+- **Explicit source only**: adaptive scans require a user-provided file path.
+- **Local first**: no network access is needed for preference extraction.
+- **No implicit private logs**: shell history, browser history, mail, and hidden chat logs are blocked by default.
+- **Repeated signals only**: one-off statements are recorded as discarded signals unless they meet the configured support threshold.
+- **Redacted evidence**: stored excerpts must remove secrets, tokens, email addresses, and local absolute paths.
+- **Proposal before patch**: preference memory can generate proposals, not automatic source edits.
+
+## Allowed Inputs
+
+Recommended inputs are curated JSONL, Markdown, or text files prepared for review. JSONL records should use a field such as `text`, `message`, `content`, `excerpt`, `prompt`, `note`, or `body`.
+
+## Blocked By Default
+
+The scanner refuses common shell history files such as `.zsh_history`, `.bash_history`, and `.fish_history` unless an explicit override is added for a controlled test. Even with an override, the output remains redacted and proposal-only.
+
+## Retention
+
+Generated reports store only summarized patterns and short redacted excerpts. They should not be treated as a full transcript, chat archive, or durable personal memory store.
+
+## Upgrade Path
+
+A future patch-application stage must add:
+
+- human approval ledger;
+- allowlisted target files;
+- dry-run diffs;
+- regression command execution;
+- rollback artifacts;
+- reviewer-visible audit trail.
diff --git a/registry/index.json b/registry/index.json
index e4bc9b50..94e14ef1 100644
--- a/registry/index.json
+++ b/registry/index.json
@@ -12,10 +12,11 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
],
"package_metadata": "registry/packages/yao-meta-skill.json",
- "package_sha256": "742dc905b0d221c4f336e741d168861edaf7b75e97fe1e89f7d454c1cabf928f"
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257"
}
]
}
diff --git a/registry/packages/yao-meta-skill.json b/registry/packages/yao-meta-skill.json
index 4fa9e8c6..28c2a608 100644
--- a/registry/packages/yao-meta-skill.json
+++ b/registry/packages/yao-meta-skill.json
@@ -2,12 +2,13 @@
"schema_version": "2.0",
"name": "yao-meta-skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"targets": [
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
],
"maturity": "governed",
"owner": "Yao Team",
@@ -15,8 +16,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
- "package_sha256": "742dc905b0d221c4f336e741d168861edaf7b75e97fe1e89f7d454c1cabf928f",
- "archive_sha256": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9"
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2"
},
"compatibility": {
"openai": "pass",
@@ -47,7 +48,7 @@
},
"distribution": {
"archive_verified": true,
- "archive_sha256": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
diff --git a/reports/adaptation_approval_ledger.json b/reports/adaptation_approval_ledger.json
new file mode 100644
index 00000000..78e9d2c7
--- /dev/null
+++ b/reports/adaptation_approval_ledger.json
@@ -0,0 +1,63 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-15T00:00:00Z",
+ "summary": {
+ "approval_count": 0,
+ "active_approval_count": 0,
+ "pending_review_count": 0,
+ "applied_count": 0,
+ "rollback_count": 0
+ },
+ "approval_contract": {
+ "approval_required": true,
+ "patch_sha256_required": true,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required": true,
+ "approval_draft_supported": true,
+ "dry_run_default": true,
+ "writes_repository_files_only_with_apply": true,
+ "rollback_required": true
+ },
+ "entries": [],
+ "approval_count": 0,
+ "active_approval_count": 0,
+ "pending_review_count": 0,
+ "applied_count": 0,
+ "rollback_count": 0,
+ "approval_required": true,
+ "patch_sha256_required": true,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required": true,
+ "approval_draft_supported": true,
+ "dry_run_default": true,
+ "writes_repository_files_only_with_apply": true,
+ "rollback_required": true,
+ "report_contract": {
+ "schema_version": "1.0",
+ "contract": "adaptation-approval-ledger",
+ "top_level_mirrors_summary": true,
+ "top_level_mirrors_approval_contract": true,
+ "summary_fields": [
+ "approval_count",
+ "active_approval_count",
+ "pending_review_count",
+ "applied_count",
+ "rollback_count"
+ ],
+ "approval_contract_fields": [
+ "approval_required",
+ "patch_sha256_required",
+ "allowlisted_targets_required",
+ "target_file_sha256_required",
+ "approval_draft_supported",
+ "dry_run_default",
+ "writes_repository_files_only_with_apply",
+ "rollback_required"
+ ],
+ "source_of_truth": [
+ "summary",
+ "approval_contract"
+ ]
+ }
+}
diff --git a/reports/adaptation_proposals.json b/reports/adaptation_proposals.json
new file mode 100644
index 00000000..55892afa
--- /dev/null
+++ b/reports/adaptation_proposals.json
@@ -0,0 +1,248 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-17T08:58:54Z",
+ "skill_dir": ".",
+ "source_patterns": "reports/user_patterns.json",
+ "pattern_count": 5,
+ "proposal_count": 5,
+ "apply_supported": true,
+ "failure_count": 0,
+ "proposal_only": true,
+ "approval_required": true,
+ "writes_repository_files": false,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required_for_apply": true,
+ "approval_draft_supported": true,
+ "rollback_required_for_apply": true,
+ "apply_command_available": true,
+ "summary": {
+ "pattern_count": 5,
+ "proposal_count": 5,
+ "apply_supported": true,
+ "failure_count": 0
+ },
+ "proposal_contract": {
+ "proposal_only": true,
+ "approval_required": true,
+ "writes_repository_files": false,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required_for_apply": true,
+ "approval_draft_supported": true,
+ "rollback_required_for_apply": true,
+ "apply_command_available": true
+ },
+ "report_contract": {
+ "schema_version": "1.0",
+ "contract": "adaptation-proposals",
+ "top_level_mirrors_summary": true,
+ "top_level_mirrors_proposal_contract": true,
+ "summary_fields": [
+ "pattern_count",
+ "proposal_count",
+ "apply_supported",
+ "failure_count"
+ ],
+ "proposal_contract_fields": [
+ "proposal_only",
+ "approval_required",
+ "writes_repository_files",
+ "allowlisted_targets_required",
+ "target_file_sha256_required_for_apply",
+ "approval_draft_supported",
+ "rollback_required_for_apply",
+ "apply_command_available"
+ ],
+ "source_of_truth": [
+ "summary",
+ "proposal_contract"
+ ]
+ },
+ "proposals": [
+ {
+ "proposal_id": "adapt-18c7517f3d",
+ "pattern_id": "language_default",
+ "title": "Keep reports Chinese-first with optional English",
+ "change_type": "report-default-language",
+ "status": "proposal-only",
+ "requires_approval": true,
+ "write_allowed_without_approval": false,
+ "risk_level": "low",
+ "reason": "2 redacted records matched repeated report-language signals.",
+ "support_count": 2,
+ "target_files": [
+ "scripts/render_skill_overview.py",
+ "references/artifact-design-doctrine.md"
+ ],
+ "suggested_changes": [
+ "Keep user-facing report copy Simplified Chinese by default.",
+ "Expose English through the existing language switch instead of mixing languages in the default view."
+ ],
+ "verification_commands": [
+ "python3 tests/verify_skill_overview.py"
+ ],
+ "rollback_plan": "Revert report language template changes and rerun the overview verifier.",
+ "evidence_refs": [
+ {
+ "record_id": "line-1",
+ "excerpt": "新生成的 Skill 报告默认使用中文简体，并在右上角提供英文切换。"
+ },
+ {
+ "record_id": "line-2",
+ "excerpt": "HTML 报告需要双语能力，但默认内容应该保持中文简体。"
+ }
+ ]
+ },
+ {
+ "proposal_id": "adapt-fbfe921ba5",
+ "pattern_id": "report_ui",
+ "title": "Improve report layout, visual hierarchy, and chart readability",
+ "change_type": "artifact-ui-polish",
+ "status": "proposal-only",
+ "requires_approval": true,
+ "write_allowed_without_approval": false,
+ "risk_level": "medium",
+ "reason": "5 redacted records matched repeated artifact-design signals.",
+ "support_count": 5,
+ "target_files": [
+ "scripts/render_skill_overview.py",
+ "references/artifact-design-doctrine.md",
+ "tests/verify_skill_overview.py"
+ ],
+ "suggested_changes": [
+ "Prefer vertical narrative sections with limited two-column layouts only when content has enough width.",
+ "Keep charts inline SVG, with captions and stable responsive constraints."
+ ],
+ "verification_commands": [
+ "python3 tests/verify_skill_overview.py",
+ "python3 tests/verify_skill_report_charts.py"
+ ],
+ "rollback_plan": "Restore the previous report renderer and regenerate the demo report.",
+ "evidence_refs": [
+ {
+ "record_id": "line-1",
+ "excerpt": "新生成的 Skill 报告默认使用中文简体，并在右上角提供英文切换。"
+ },
+ {
+ "record_id": "line-2",
+ "excerpt": "HTML 报告需要双语能力，但默认内容应该保持中文简体。"
+ },
+ {
+ "record_id": "line-3",
+ "excerpt": "报告排版采用白底 Kami 风格，图表、模块和导航都要清晰。"
+ }
+ ]
+ },
+ {
+ "proposal_id": "adapt-59d219a1fb",
+ "pattern_id": "approval_safety",
+ "title": "Keep adaptive iteration approval-gated",
+ "change_type": "privacy-governance",
+ "status": "proposal-only",
+ "requires_approval": true,
+ "write_allowed_without_approval": false,
+ "risk_level": "low",
+ "reason": "2 redacted records matched repeated governance signals.",
+ "support_count": 2,
+ "target_files": [
+ "references/user-memory-policy.md",
+ "references/autonomous-adaptation.md",
+ "schemas/adaptation-proposal.schema.json"
+ ],
+ "suggested_changes": [
+ "Require explicit source paths for memory scans.",
+ "Generate proposals before any source patching.",
+ "Reserve automatic apply for a future approval ledger and rollback implementation."
+ ],
+ "verification_commands": [
+ "python3 tests/verify_adaptation_safety.py"
+ ],
+ "rollback_plan": "Remove the adaptive proposal artifacts and keep feedback/adoption drift as the only iteration inputs.",
+ "evidence_refs": [
+ {
+ "record_id": "line-5",
+ "excerpt": "自适应升级必须先生成提案，不能直接自动修改源文件。"
+ },
+ {
+ "record_id": "line-6",
+ "excerpt": "用户偏好扫描必须由用户提供明确路径，不要默认扫描私人日志。"
+ }
+ ]
+ },
+ {
+ "proposal_id": "adapt-457baca160",
+ "pattern_id": "delivery_format",
+ "title": "Make generated artifact paths explicit in CLI output",
+ "change_type": "artifact-discoverability",
+ "status": "proposal-only",
+ "requires_approval": true,
+ "write_allowed_without_approval": false,
+ "risk_level": "low",
+ "reason": "2 redacted records matched repeated artifact-format signals.",
+ "support_count": 2,
+ "target_files": [
+ "scripts/yao.py",
+ "README.md"
+ ],
+ "suggested_changes": [
+ "Include stable report paths in command output.",
+ "Document which artifacts are meant for human review."
+ ],
+ "verification_commands": [
+ "python3 tests/verify_yao_cli.py"
+ ],
+ "rollback_plan": "Revert CLI copy/documentation changes and keep artifact paths unchanged.",
+ "evidence_refs": [
+ {
+ "record_id": "line-2",
+ "excerpt": "HTML 报告需要双语能力，但默认内容应该保持中文简体。"
+ },
+ {
+ "record_id": "line-6",
+ "excerpt": "用户偏好扫描必须由用户提供明确路径，不要默认扫描私人日志。"
+ }
+ ]
+ },
+ {
+ "proposal_id": "adapt-abfee25d3a",
+ "pattern_id": "evidence_testing",
+ "title": "Attach tests and evidence refresh to each upgrade",
+ "change_type": "quality-gate",
+ "status": "proposal-only",
+ "requires_approval": true,
+ "write_allowed_without_approval": false,
+ "risk_level": "medium",
+ "reason": "2 redacted records matched repeated quality-gate signals.",
+ "support_count": 2,
+ "target_files": [
+ "tests/verify_adaptation_safety.py",
+ "scripts/render_skill_os2_coverage.py",
+ "reports/skill_os2_coverage.json"
+ ],
+ "suggested_changes": [
+ "Add focused verifier coverage for every new adaptive behavior.",
+ "Refresh Skill OS 2.0 coverage so planned, partial, and covered states remain visible."
+ ],
+ "verification_commands": [
+ "python3 tests/verify_adaptation_safety.py",
+ "python3 tests/verify_skill_os2_coverage.py"
+ ],
+ "rollback_plan": "Revert the new verifier and coverage status updates, then regenerate coverage reports.",
+ "evidence_refs": [
+ {
+ "record_id": "line-7",
+ "excerpt": "每次升级都需要测试、覆盖报告和可审计证据，推送前要跑 CI。"
+ },
+ {
+ "record_id": "line-8",
+ "excerpt": "涉及 GitHub 推送时，要保留证据链，避免把计划当作完成证明。"
+ }
+ ]
+ }
+ ],
+ "failures": [],
+ "artifacts": {
+ "json": "reports/adaptation_proposals.json",
+ "markdown": "reports/adaptation_proposals.md"
+ }
+}
diff --git a/reports/adaptation_proposals.md b/reports/adaptation_proposals.md
new file mode 100644
index 00000000..86f10bec
--- /dev/null
+++ b/reports/adaptation_proposals.md
@@ -0,0 +1,119 @@
+# Adaptation Proposals
+
+- Generated at: `2026-06-17T08:58:54Z`
+- Pattern report: `reports/user_patterns.json`
+- Proposal only: `true`
+- Writes repository files: `false`
+- Proposals: `5`
+
+## Keep reports Chinese-first with optional English
+
+- ID: `adapt-18c7517f3d`
+- Status: `proposal-only`
+- Pattern: `language_default`
+- Risk: `low`
+- Requires approval: `true`
+- Reason: 2 redacted records matched repeated report-language signals.
+- Target files:
+ - `scripts/render_skill_overview.py`
+ - `references/artifact-design-doctrine.md`
+- Suggested changes:
+ - Keep user-facing report copy Simplified Chinese by default.
+ - Expose English through the existing language switch instead of mixing languages in the default view.
+- Verification:
+ - `python3 tests/verify_skill_overview.py`
+- Rollback: Revert report language template changes and rerun the overview verifier.
+- Redacted evidence refs:
+ - `line-1`: 新生成的 Skill 报告默认使用中文简体，并在右上角提供英文切换。
+ - `line-2`: HTML 报告需要双语能力，但默认内容应该保持中文简体。
+
+## Improve report layout, visual hierarchy, and chart readability
+
+- ID: `adapt-fbfe921ba5`
+- Status: `proposal-only`
+- Pattern: `report_ui`
+- Risk: `medium`
+- Requires approval: `true`
+- Reason: 5 redacted records matched repeated artifact-design signals.
+- Target files:
+ - `scripts/render_skill_overview.py`
+ - `references/artifact-design-doctrine.md`
+ - `tests/verify_skill_overview.py`
+- Suggested changes:
+ - Prefer vertical narrative sections with limited two-column layouts only when content has enough width.
+ - Keep charts inline SVG, with captions and stable responsive constraints.
+- Verification:
+ - `python3 tests/verify_skill_overview.py`
+ - `python3 tests/verify_skill_report_charts.py`
+- Rollback: Restore the previous report renderer and regenerate the demo report.
+- Redacted evidence refs:
+ - `line-1`: 新生成的 Skill 报告默认使用中文简体，并在右上角提供英文切换。
+ - `line-2`: HTML 报告需要双语能力，但默认内容应该保持中文简体。
+ - `line-3`: 报告排版采用白底 Kami 风格，图表、模块和导航都要清晰。
+
+## Keep adaptive iteration approval-gated
+
+- ID: `adapt-59d219a1fb`
+- Status: `proposal-only`
+- Pattern: `approval_safety`
+- Risk: `low`
+- Requires approval: `true`
+- Reason: 2 redacted records matched repeated governance signals.
+- Target files:
+ - `references/user-memory-policy.md`
+ - `references/autonomous-adaptation.md`
+ - `schemas/adaptation-proposal.schema.json`
+- Suggested changes:
+ - Require explicit source paths for memory scans.
+ - Generate proposals before any source patching.
+ - Reserve automatic apply for a future approval ledger and rollback implementation.
+- Verification:
+ - `python3 tests/verify_adaptation_safety.py`
+- Rollback: Remove the adaptive proposal artifacts and keep feedback/adoption drift as the only iteration inputs.
+- Redacted evidence refs:
+ - `line-5`: 自适应升级必须先生成提案，不能直接自动修改源文件。
+ - `line-6`: 用户偏好扫描必须由用户提供明确路径，不要默认扫描私人日志。
+
+## Make generated artifact paths explicit in CLI output
+
+- ID: `adapt-457baca160`
+- Status: `proposal-only`
+- Pattern: `delivery_format`
+- Risk: `low`
+- Requires approval: `true`
+- Reason: 2 redacted records matched repeated artifact-format signals.
+- Target files:
+ - `scripts/yao.py`
+ - `README.md`
+- Suggested changes:
+ - Include stable report paths in command output.
+ - Document which artifacts are meant for human review.
+- Verification:
+ - `python3 tests/verify_yao_cli.py`
+- Rollback: Revert CLI copy/documentation changes and keep artifact paths unchanged.
+- Redacted evidence refs:
+ - `line-2`: HTML 报告需要双语能力，但默认内容应该保持中文简体。
+ - `line-6`: 用户偏好扫描必须由用户提供明确路径，不要默认扫描私人日志。
+
+## Attach tests and evidence refresh to each upgrade
+
+- ID: `adapt-abfee25d3a`
+- Status: `proposal-only`
+- Pattern: `evidence_testing`
+- Risk: `medium`
+- Requires approval: `true`
+- Reason: 2 redacted records matched repeated quality-gate signals.
+- Target files:
+ - `tests/verify_adaptation_safety.py`
+ - `scripts/render_skill_os2_coverage.py`
+ - `reports/skill_os2_coverage.json`
+- Suggested changes:
+ - Add focused verifier coverage for every new adaptive behavior.
+ - Refresh Skill OS 2.0 coverage so planned, partial, and covered states remain visible.
+- Verification:
+ - `python3 tests/verify_adaptation_safety.py`
+ - `python3 tests/verify_skill_os2_coverage.py`
+- Rollback: Revert the new verifier and coverage status updates, then regenerate coverage reports.
+- Redacted evidence refs:
+ - `line-7`: 每次升级都需要测试、覆盖报告和可审计证据，推送前要跑 CI。
+ - `line-8`: 涉及 GitHub 推送时，要保留证据链，避免把计划当作完成证明。
diff --git a/reports/adaptation_regression_report.json b/reports/adaptation_regression_report.json
new file mode 100644
index 00000000..a0308041
--- /dev/null
+++ b/reports/adaptation_regression_report.json
@@ -0,0 +1,87 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-15T00:00:00Z",
+ "skill_dir": ".",
+ "summary": {
+ "apply_supported": true,
+ "attempt_count": 0,
+ "approval_draft_count": 0,
+ "applied_count": 0,
+ "dry_run_count": 0,
+ "rollback_count": 0,
+ "regression_run_count": 0,
+ "regression_pass_count": 0,
+ "failure_count": 0
+ },
+ "apply_contract": {
+ "approval_required": true,
+ "patch_sha256_required": true,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required": true,
+ "approval_draft_supported": true,
+ "dry_run_default": true,
+ "writes_repository_files_only_with_apply": true,
+ "rollback_required": true,
+ "safe_regression_commands_only": true,
+ "rollback_on_failure_default": true
+ },
+ "attempts": [],
+ "failures": [],
+ "artifacts": {
+ "json": "reports/adaptation_regression_report.json",
+ "approval_ledger": "reports/adaptation_approval_ledger.json"
+ },
+ "apply_supported": true,
+ "attempt_count": 0,
+ "approval_draft_count": 0,
+ "applied_count": 0,
+ "dry_run_count": 0,
+ "rollback_count": 0,
+ "regression_run_count": 0,
+ "regression_pass_count": 0,
+ "failure_count": 0,
+ "approval_required": true,
+ "patch_sha256_required": true,
+ "allowlisted_targets_required": true,
+ "target_file_sha256_required": true,
+ "approval_draft_supported": true,
+ "dry_run_default": true,
+ "writes_repository_files_only_with_apply": true,
+ "rollback_required": true,
+ "safe_regression_commands_only": true,
+ "rollback_on_failure_default": true,
+ "report_contract": {
+ "schema_version": "1.0",
+ "contract": "adaptation-regression-report",
+ "top_level_mirrors_summary": true,
+ "top_level_mirrors_apply_contract": true,
+ "summary_fields": [
+ "apply_supported",
+ "attempt_count",
+ "approval_draft_count",
+ "applied_count",
+ "dry_run_count",
+ "rollback_count",
+ "regression_run_count",
+ "regression_pass_count",
+ "failure_count"
+ ],
+ "apply_contract_fields": [
+ "approval_required",
+ "patch_sha256_required",
+ "allowlisted_targets_required",
+ "target_file_sha256_required",
+ "approval_draft_supported",
+ "dry_run_default",
+ "writes_repository_files_only_with_apply",
+ "rollback_required",
+ "safe_regression_commands_only",
+ "rollback_on_failure_default"
+ ],
+ "source_of_truth": [
+ "summary",
+ "apply_contract"
+ ]
+ }
+}
diff --git a/reports/adaptation_regression_report.md b/reports/adaptation_regression_report.md
new file mode 100644
index 00000000..cdcaa683
--- /dev/null
+++ b/reports/adaptation_regression_report.md
@@ -0,0 +1,11 @@
+# Adaptation Regression Report
+
+- generated_at: `2026-06-15T00:00:00Z`
+- apply_supported: `true`
+- attempts: `0`
+- applied: `0`
+- dry runs: `0`
+- rollbacks: `0`
+- failures: `0`
+
+This report proves the adaptive apply harness behavior. It does not count proposals as applied changes.
diff --git a/reports/adoption_drift_report.json b/reports/adoption_drift_report.json
index 7f931d0b..da4a16c2 100644
--- a/reports/adoption_drift_report.json
+++ b/reports/adoption_drift_report.json
@@ -1,7 +1,7 @@
{
"ok": true,
"schema_version": "2.0",
- "generated_at": "2026-06-13T10:57:52Z",
+ "generated_at": "2026-06-17T08:58:53Z",
"skill_dir": ".",
"privacy_contract": {
"storage": "local-first",
@@ -25,13 +25,14 @@
},
"summary": {
"event_count": 1,
- "activation_count": 1,
- "accepted_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
"edited_count": 0,
"rejected_count": 0,
"missed_count": 0,
"failed_count": 0,
- "adoption_rate": 100.0,
+ "adoption_rate": 0,
"missed_trigger_count": 0,
"wrong_trigger_count": 0,
"bad_output_count": 0,
@@ -40,36 +41,43 @@
"review_overdue_count": 0,
"risk_band": "low",
"event_types": {
- "skill_activation": 1
+ "review_event": 1
},
- "failure_types": {}
+ "failure_types": {},
+ "source_types": {
+ "manual": 1
+ },
+ "command_counts": {}
},
"adoption_by_skill": [
{
"skill": "yao-meta-skill",
"events": 1,
- "accepted": 1,
+ "adoption_events": 0,
+ "accepted": 0,
"edited": 0,
"rejected": 0,
"missed": 0,
- "adoption_rate": 100.0
+ "adoption_rate": 0
}
],
"next_iteration_candidates": [],
"recent_events": [
{
- "event": "skill_activation",
+ "command": "unknown",
+ "event": "review_event",
"skill": "yao-meta-skill",
+ "source": "manual",
"version": "1.1.0",
- "activation_type": "explicit",
- "outcome": "accepted",
+ "activation_type": "manual",
+ "outcome": "reviewed",
"failure_type": "none",
- "timestamp": "2026-06-13T10:00:00Z"
+ "timestamp": "2026-06-13T12:00:00Z"
}
],
"failures": [],
"artifacts": {
- "events_jsonl": "tests/tmp_review_studio/telemetry_events.jsonl",
+ "events_jsonl": "reports/telemetry_events.jsonl",
"json": "reports/adoption_drift_report.json",
"markdown": "reports/adoption_drift_report.md"
}
diff --git a/reports/adoption_drift_report.md b/reports/adoption_drift_report.md
index 743dd610..91b31983 100644
--- a/reports/adoption_drift_report.md
+++ b/reports/adoption_drift_report.md
@@ -5,8 +5,9 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs,
## Summary
- Events: `1`
-- Activation events: `1`
-- Adoption rate: `100.0`
+- Adoption samples: `0`
+- Activation events: `0`
+- Adoption rate: `0`
- Missed trigger signals: `0`
- Bad output signals: `0`
- Script error signals: `0`
@@ -22,9 +23,9 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs,
## Adoption By Skill
-| Skill | Events | Accepted | Edited | Rejected | Missed | Adoption Rate |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| `yao-meta-skill` | 1 | 1 | 0 | 0 | 0 | 100.0 |
+| Skill | Events | Adoption Samples | Accepted | Edited | Rejected | Missed | Adoption Rate |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| `yao-meta-skill` | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
## Next Iteration Candidates
@@ -32,4 +33,4 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs,
## Recent Metadata Events
-- `2026-06-13T10:00:00Z` `yao-meta-skill` event=`skill_activation` activation=`explicit` outcome=`accepted` failure=`none`
+- `2026-06-13T12:00:00Z` `yao-meta-skill` event=`review_event` source=`manual` command=`unknown` activation=`manual` outcome=`reviewed` failure=`none`
diff --git a/reports/architecture_maintainability.json b/reports/architecture_maintainability.json
new file mode 100644
index 00000000..6951d03a
--- /dev/null
+++ b/reports/architecture_maintainability.json
@@ -0,0 +1,181 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-17",
+ "skill_dir": ".",
+ "summary": {
+ "python_file_count": 229,
+ "script_file_count": 153,
+ "test_file_count": 76,
+ "internal_module_count": 69,
+ "cli_script_count": 86,
+ "command_handler_count": 68,
+ "entrypoint_command_handler_count": 18,
+ "command_module_count": 6,
+ "warn_line_threshold": 900,
+ "watch_line_threshold": 720,
+ "early_watch_line_threshold": 600,
+ "block_line_threshold": 1500,
+ "largest_file_lines": 719,
+ "watchlist_count": 0,
+ "early_watchlist_count": 6,
+ "hotspot_count": 0,
+ "blocker_count": 0,
+ "decision": "pass"
+ },
+ "largest_files": [
+ {
+ "path": "tests/verify_evidence_consistency.py",
+ "lines": 719,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "tests/verify_world_class_evidence_intake.py",
+ "lines": 703,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "tests/verify_yao_cli.py",
+ "lines": 700,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "scripts/render_benchmark_reproducibility.py",
+ "lines": 685,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/render_evidence_consistency.py",
+ "lines": 676,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/render_world_class_operator_runbook.py",
+ "lines": 651,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "tests/verify_output_review_adjudication.py",
+ "lines": 599,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "scripts/render_skill_overview.py",
+ "lines": 588,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "tests/verify_world_class_evidence_ledger.py",
+ "lines": 587,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "scripts/build_skill_atlas.py",
+ "lines": 586,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/optimize_description.py",
+ "lines": 585,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/trust_check.py",
+ "lines": 582,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": false,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ }
+ ],
+ "watchlist": [],
+ "early_watchlist": [
+ {
+ "path": "tests/verify_evidence_consistency.py",
+ "lines": 719,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "tests/verify_world_class_evidence_intake.py",
+ "lines": 703,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "tests/verify_yao_cli.py",
+ "lines": 700,
+ "kind": "test",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands."
+ },
+ {
+ "path": "scripts/render_benchmark_reproducibility.py",
+ "lines": 685,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/render_evidence_consistency.py",
+ "lines": 676,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ },
+ {
+ "path": "scripts/render_world_class_operator_runbook.py",
+ "lines": 651,
+ "kind": "cli-script",
+ "severity": "pass",
+ "early_watch": true,
+ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates."
+ }
+ ],
+ "hotspots": [],
+ "actions": [],
+ "artifacts": {
+ "json": "reports/architecture_maintainability.json",
+ "markdown": "reports/architecture_maintainability.md"
+ }
+}
diff --git a/reports/architecture_maintainability.md b/reports/architecture_maintainability.md
new file mode 100644
index 00000000..c615e840
--- /dev/null
+++ b/reports/architecture_maintainability.md
@@ -0,0 +1,66 @@
+# Architecture Maintainability
+
+Generated at: `2026-06-17`
+
+## Summary
+
+- decision: `pass`
+- python files: `229`
+- scripts: `153`
+- tests: `76`
+- internal modules: `69`
+- CLI scripts: `86`
+- Yao CLI command handlers: `68`
+- entrypoint command handlers: `18`
+- command modules: `6`
+- largest file lines: `719`
+- early watch threshold lines: `600`
+- early watchlist: `6`
+- watch threshold lines: `720`
+- watchlist: `0`
+- hotspots: `0`
+- blockers: `0`
+
+This report keeps maintainability risk visible before the Meta Skill grows more gates, renderers, and CLI commands.
+
+## Hotspots
+
+No file-size hotspots found.
+
+## Watchlist
+
+No near-threshold files found.
+
+## Early Watchlist
+
+| File | Lines | Kind | Recommended next split |
+| --- | ---: | --- | --- |
+| `tests/verify_evidence_consistency.py` | `719` | `test` | Break broad integration assertions into focused verifier helpers when the next behavior change lands. |
+| `tests/verify_world_class_evidence_intake.py` | `703` | `test` | Break broad integration assertions into focused verifier helpers when the next behavior change lands. |
+| `tests/verify_yao_cli.py` | `700` | `test` | Break broad integration assertions into focused verifier helpers when the next behavior change lands. |
+| `scripts/render_benchmark_reproducibility.py` | `685` | `cli-script` | Watch this file before adding new responsibilities; extract a helper module when one concern dominates. |
+| `scripts/render_evidence_consistency.py` | `676` | `cli-script` | Watch this file before adding new responsibilities; extract a helper module when one concern dominates. |
+| `scripts/render_world_class_operator_runbook.py` | `651` | `cli-script` | Watch this file before adding new responsibilities; extract a helper module when one concern dominates. |
+
+## Largest Files
+
+| File | Lines | Kind | Severity |
+| --- | ---: | --- | --- |
+| `tests/verify_evidence_consistency.py` | `719` | `test` | `pass` |
+| `tests/verify_world_class_evidence_intake.py` | `703` | `test` | `pass` |
+| `tests/verify_yao_cli.py` | `700` | `test` | `pass` |
+| `scripts/render_benchmark_reproducibility.py` | `685` | `cli-script` | `pass` |
+| `scripts/render_evidence_consistency.py` | `676` | `cli-script` | `pass` |
+| `scripts/render_world_class_operator_runbook.py` | `651` | `cli-script` | `pass` |
+| `tests/verify_output_review_adjudication.py` | `599` | `test` | `pass` |
+| `scripts/render_skill_overview.py` | `588` | `cli-script` | `pass` |
+| `tests/verify_world_class_evidence_ledger.py` | `587` | `test` | `pass` |
+| `scripts/build_skill_atlas.py` | `586` | `cli-script` | `pass` |
+| `scripts/optimize_description.py` | `585` | `cli-script` | `pass` |
+| `scripts/trust_check.py` | `582` | `cli-script` | `pass` |
+
+## Release Rule
+
+- `block` hotspots should be split before governed release.
+- `warn` hotspots can ship only when Review Studio keeps them visible and a reviewer accepts the modularization plan.
+- Do not split a file only for line count; split when a stable responsibility boundary is clear.
diff --git a/reports/artifact-design-profile.json b/reports/artifact-design-profile.json
index 2e76f3c6..c32e5dc7 100644
--- a/reports/artifact-design-profile.json
+++ b/reports/artifact-design-profile.json
@@ -2,16 +2,40 @@
"skill_name": "yao-meta-skill",
"design_system": "metric editorial",
"primary_artifact": {
- "key": "code_or_cli",
- "label": "Code, CLI, or implementation guide",
- "direction": "Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.",
+ "key": "dashboard",
+ "label": "Dashboard or metrics page",
+ "direction": "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
"matched_keywords": [
- "code",
- "script",
- "command"
+ "metric",
+ "score",
+ "table",
+ "scorecard"
]
},
"artifact_families": [
+ {
+ "key": "dashboard",
+ "label": "Dashboard or metrics page",
+ "score": 4,
+ "matched_keywords": [
+ "metric",
+ "score",
+ "table",
+ "scorecard"
+ ],
+ "direction": "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation."
+ },
+ {
+ "key": "review_viewer",
+ "label": "Review viewer",
+ "score": 3,
+ "matched_keywords": [
+ "review",
+ "viewer",
+ "audit"
+ ],
+ "direction": "Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix."
+ },
{
"key": "code_or_cli",
"label": "Code, CLI, or implementation guide",
@@ -33,25 +57,6 @@
],
"direction": "High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail."
},
- {
- "key": "review_viewer",
- "label": "Review viewer",
- "score": 2,
- "matched_keywords": [
- "review",
- "viewer"
- ],
- "direction": "Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix."
- },
- {
- "key": "dashboard",
- "label": "Dashboard or metrics page",
- "score": 1,
- "matched_keywords": [
- "table"
- ],
- "direction": "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation."
- },
{
"key": "visual_capture",
"label": "Screenshot or visual evidence",
@@ -63,14 +68,14 @@
}
],
"layout_patterns": [
- "prerequisites",
- "commands",
- "expected output",
- "failure handling",
- "rollback or cleanup",
- "thesis",
- "evidence blocks",
- "decision table"
+ "metric board",
+ "ranked signals",
+ "comparison rows",
+ "interpretation",
+ "action queue",
+ "summary",
+ "variant comparison",
+ "evidence"
],
"design_tokens": {
"type": [
@@ -95,14 +100,14 @@
]
},
"quality_gates": [
- "Name the working directory and required inputs for commands.",
- "Mark destructive, networked, or external side-effect operations.",
- "Prefer the smallest runnable snippet over broad framework scaffolding.",
- "Keep the first screen useful without requiring the reader to parse every detail.",
- "Use tables only for comparisons; move explanations below the table.",
- "Keep source notes readable without flooding the body with markers.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities.",
"Make differences visible instead of hiding them in prose.",
- "Separate author-facing recommendations from reviewer-only evidence."
+ "Separate author-facing recommendations from reviewer-only evidence.",
+ "Surface conflicts clearly and keep routine benchmark synthesis quiet.",
+ "Name the working directory and required inputs for commands.",
+ "Mark destructive, networked, or external side-effect operations."
],
"anti_patterns": [
"Do not copy Kami's fixed parchment background as a default.",
diff --git a/reports/artifact-design-profile.md b/reports/artifact-design-profile.md
index 4b708892..f4fd2531 100644
--- a/reports/artifact-design-profile.md
+++ b/reports/artifact-design-profile.md
@@ -5,12 +5,22 @@ Design system: `metric editorial`
## Primary Artifact Direction
-**Code, CLI, or implementation guide**
+**Dashboard or metrics page**
-Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.
+Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
## Matched Artifact Families
+### Dashboard or metrics page
+- Matched keywords: metric, score, table, scorecard
+- Score: `4`
+- Direction: Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
+
+### Review viewer
+- Matched keywords: review, viewer, audit
+- Score: `3`
+- Direction: Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix.
+
### Code, CLI, or implementation guide
- Matched keywords: code, script, command
- Score: `3`
@@ -21,16 +31,6 @@ Execution-focused technical artifact with environment assumptions, copyable comm
- Score: `2`
- Direction: High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.
-### Review viewer
-- Matched keywords: review, viewer
-- Score: `2`
-- Direction: Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix.
-
-### Dashboard or metrics page
-- Matched keywords: table
-- Score: `1`
-- Direction: Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
-
### Screenshot or visual evidence
- Matched keywords: screenshot
- Score: `1`
@@ -38,14 +38,14 @@ Execution-focused technical artifact with environment assumptions, copyable comm
## Layout Patterns To Prefer
-- prerequisites
-- commands
-- expected output
-- failure handling
-- rollback or cleanup
-- thesis
-- evidence blocks
-- decision table
+- metric board
+- ranked signals
+- comparison rows
+- interpretation
+- action queue
+- summary
+- variant comparison
+- evidence
## Design Tokens
@@ -71,14 +71,14 @@ Execution-focused technical artifact with environment assumptions, copyable comm
## Quality Gates
-- Name the working directory and required inputs for commands.
-- Mark destructive, networked, or external side-effect operations.
-- Prefer the smallest runnable snippet over broad framework scaffolding.
-- Keep the first screen useful without requiring the reader to parse every detail.
-- Use tables only for comparisons; move explanations below the table.
-- Keep source notes readable without flooding the body with markers.
+- Avoid paragraph-heavy table cells.
+- Keep charts tied to one analytical question each.
+- Preserve stable color meaning across metrics and entities.
- Make differences visible instead of hiding them in prose.
- Separate author-facing recommendations from reviewer-only evidence.
+- Surface conflicts clearly and keep routine benchmark synthesis quiet.
+- Name the working directory and required inputs for commands.
+- Mark destructive, networked, or external side-effect operations.
## Anti-Patterns
diff --git a/reports/baseline-compare.json b/reports/baseline-compare.json
index 274a6350..0896a070 100644
--- a/reports/baseline-compare.json
+++ b/reports/baseline-compare.json
@@ -1,10 +1,10 @@
{
"summary": {
"target_count": 3,
- "baseline_total_errors": 8,
- "current_total_errors": 5,
- "winner_total_errors": 5,
- "winner_vs_baseline_gain": 3,
+ "baseline_total_errors": 5,
+ "current_total_errors": 4,
+ "winner_total_errors": 4,
+ "winner_vs_baseline_gain": 1,
"winner_vs_current_gain": 0
},
"comparisons": [
@@ -20,7 +20,7 @@
"adversarial_errors": 0
},
"current": {
- "tokens": 65,
+ "tokens": 53,
"dev_errors": 0,
"holdout_errors": 0,
"blind_errors": 0,
@@ -28,7 +28,7 @@
"adversarial_errors": 0
},
"winner": {
- "tokens": 65,
+ "tokens": 53,
"dev_errors": 0,
"holdout_errors": 0,
"blind_errors": 0,
@@ -79,15 +79,15 @@
"winner_label": "Current",
"baseline": {
"tokens": 93,
- "dev_errors": 1,
- "holdout_errors": 2,
- "blind_errors": 1,
+ "dev_errors": 0,
+ "holdout_errors": 1,
+ "blind_errors": 0,
"judge_blind_errors": 0,
"adversarial_errors": 0
},
"current": {
"tokens": 37,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"blind_errors": 0,
"judge_blind_errors": 0,
@@ -95,15 +95,15 @@
},
"winner": {
"tokens": 37,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"blind_errors": 0,
"judge_blind_errors": 0,
"adversarial_errors": 0
},
"delta": {
- "current_vs_baseline": 2,
- "winner_vs_baseline": 2,
+ "current_vs_baseline": 0,
+ "winner_vs_baseline": 0,
"winner_vs_current": 0
}
}
diff --git a/reports/baseline-compare.md b/reports/baseline-compare.md
index ba90da2d..4b8ece0e 100644
--- a/reports/baseline-compare.md
+++ b/reports/baseline-compare.md
@@ -3,10 +3,10 @@
A lightweight with-skill vs baseline comparison across tracked optimization targets.
- Targets: `3`
-- Baseline total errors: `8`
-- Current total errors: `5`
-- Winner total errors: `5`
-- Winner vs baseline gain: `3`
+- Baseline total errors: `5`
+- Current total errors: `4`
+- Winner total errors: `4`
+- Winner vs baseline gain: `1`
- Winner vs current gain: `0`
## Target Breakdown
@@ -15,4 +15,4 @@ A lightweight with-skill vs baseline comparison across tracked optimization targ
| --- | ---: | ---: | ---: | --- |
| root | 1 | 0 | 0 | Current |
| team-frontend-review | 3 | 3 | 3 | Current |
-| governed-incident-command | 4 | 2 | 2 | Current |
+| governed-incident-command | 1 | 1 | 1 | Current |
diff --git a/reports/benchmark_methodology.md b/reports/benchmark_methodology.md
index 91fa5ff0..7922a2d5 100644
--- a/reports/benchmark_methodology.md
+++ b/reports/benchmark_methodology.md
@@ -60,6 +60,7 @@ Recommended release evidence:
git rev-parse HEAD
python3 scripts/run_output_eval.py
python3 scripts/export_skill_ir.py . --output-json skill-ir/examples/yao-meta-skill.json
+python3 scripts/yao.py benchmark-reproducibility .
make ci-test
```
diff --git a/reports/benchmark_reproducibility.json b/reports/benchmark_reproducibility.json
new file mode 100644
index 00000000..74cf5b71
--- /dev/null
+++ b/reports/benchmark_reproducibility.json
@@ -0,0 +1,488 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-17",
+ "skill_dir": ".",
+ "commit": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "git_status": {
+ "available": true,
+ "dirty": true,
+ "changed_file_count": 29,
+ "generated_dirty": true,
+ "generated_changed_file_count": 29,
+ "source_dirty": false,
+ "source_changed_file_count": 0,
+ "sample": [
+ " M registry/index.json",
+ " M registry/packages/yao-meta-skill.json",
+ " M reports/adaptation_proposals.json",
+ " M reports/adaptation_proposals.md",
+ " M reports/adoption_drift_report.json",
+ " M reports/architecture_maintainability.json",
+ " M reports/architecture_maintainability.md",
+ " M reports/benchmark_reproducibility.json",
+ " M reports/benchmark_reproducibility.md",
+ " M reports/context_budget.json",
+ " M reports/context_budget.md",
+ " M reports/context_budget_summary.json"
+ ],
+ "source_sample": [],
+ "generated_sample": [
+ " M registry/index.json",
+ " M registry/packages/yao-meta-skill.json",
+ " M reports/adaptation_proposals.json",
+ " M reports/adaptation_proposals.md",
+ " M reports/adoption_drift_report.json",
+ " M reports/architecture_maintainability.json",
+ " M reports/architecture_maintainability.md",
+ " M reports/benchmark_reproducibility.json",
+ " M reports/benchmark_reproducibility.md",
+ " M reports/context_budget.json",
+ " M reports/context_budget.md",
+ " M reports/context_budget_summary.json"
+ ],
+ "generated_dirty_prefixes": [
+ "dist/",
+ "registry/index.json",
+ "registry/packages/",
+ "reports/",
+ "skill_atlas/",
+ "skill-ir/examples/"
+ ],
+ "scope": "generation-time status before this report is written"
+ },
+ "summary": {
+ "reproducibility_ready": true,
+ "release_lock_ready": true,
+ "methodology_complete": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "evidence_bundle_sha256": "55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0",
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "output_case_count": 5,
+ "failure_disclosure_count": 3,
+ "command_count": 23,
+ "command_executed_count": 10,
+ "timing_observed_count": 10,
+ "model_executed_count": 10,
+ "token_observed_count": 10,
+ "human_review_complete": false,
+ "provider_evidence_complete": true,
+ "world_class_ready": false,
+ "world_class_open_gap_count": 3,
+ "world_class_task_count": 4,
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3,
+ "working_tree_dirty": true,
+ "changed_file_count": 29,
+ "source_tree_dirty": false,
+ "source_changed_file_count": 0,
+ "generated_tree_dirty": true,
+ "generated_changed_file_count": 29
+ },
+ "beta_test_release": {
+ "ready": true,
+ "scope": "beta/public test release without superiority, fully-reviewed, or world-class claims",
+ "blockers": [],
+ "allowed_deferred_evidence": [
+ {
+ "key": "provider-holdout",
+ "label": "Provider Holdout",
+ "reason": "Provider-backed source evidence exists, but formal ledger submission and reviewer acceptance are still pending before public claims."
+ },
+ {
+ "key": "human-adjudication",
+ "label": "Human Adjudication",
+ "reason": "Human adjudication evidence is still pending; deferred for beta/public testing and still required before superiority, fully-reviewed, or world-class claims."
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "reason": "Native enforcement proof is still pending; deferred for beta/public testing and still required before world-class claims."
+ },
+ {
+ "key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "reason": "Real client telemetry is still pending; deferred for beta/public testing and still required before world-class claims."
+ }
+ ],
+ "policy": "Human blind-review, native permission enforcement, real client telemetry, and ledger acceptance may be deferred for beta/public testing, but public claims must remain blocked until those evidence entries are accepted.",
+ "required_wording": "Use beta, public test, or technical preview wording; do not claim world-class readiness, fully reviewed quality, or proven superiority over baseline."
+ },
+ "public_claim": {
+ "ready": false,
+ "scope": "public benchmark or world-class readiness claim",
+ "blockers": [
+ "human blind-review adjudication is incomplete",
+ "world-class evidence is not accepted yet (3 open gaps, 4 ledger pending)",
+ "world-class source checks are not all accepted (11/19 pass, 8 blocked)"
+ ],
+ "policy": "Local reproducibility can pass before public claims; public claims require provider evidence, human adjudication, clean release lock, accepted world-class evidence, and complete source checks."
+ },
+ "release_lock": {
+ "ready": true,
+ "commit": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "status_scope": "generation-time status before this report is written",
+ "source_changed_file_count": 0,
+ "generated_changed_file_count": 29,
+ "reason": "only generated evidence artifacts were dirty at generation time"
+ },
+ "evidence_bundle": {
+ "algorithm": "sha256(path,label,exists,artifact_sha256)",
+ "artifact_count": 25,
+ "existing_count": 25,
+ "missing_count": 0,
+ "missing_paths": [],
+ "sha256": "55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0"
+ },
+ "methodology": {
+ "path": "reports/benchmark_methodology.md",
+ "exists": true,
+ "sections": [
+ {
+ "heading": "## Benchmark Types",
+ "exists": true
+ },
+ {
+ "heading": "## Sample Sources",
+ "exists": true
+ },
+ {
+ "heading": "## Evaluation Dimensions",
+ "exists": true
+ },
+ {
+ "heading": "## Weighting Rule",
+ "exists": true
+ },
+ {
+ "heading": "## Failure Disclosure",
+ "exists": true
+ },
+ {
+ "heading": "## Reproduction",
+ "exists": true
+ }
+ ],
+ "missing_sections": []
+ },
+ "artifacts_checked": [
+ {
+ "label": "methodology",
+ "path": "reports/benchmark_methodology.md",
+ "exists": true,
+ "bytes": 2715,
+ "sha256": "57025e0123ce5d10401c5bff376d2eeeac7943c83897ae1ad3fb22cadf790f92"
+ },
+ {
+ "label": "failure_disclosure",
+ "path": "evals/failure-cases.md",
+ "exists": true,
+ "bytes": 889,
+ "sha256": "28833c0d4a217d612879d193fb5de199880dd5b3093ab5757e4315600fa4fb08"
+ },
+ {
+ "label": "output_cases",
+ "path": "evals/output/cases.jsonl",
+ "exists": true,
+ "bytes": 6555,
+ "sha256": "a6ae9685711620d7203b73ace4412194ae287689f945b5139ec8ad75b9eefe04"
+ },
+ {
+ "label": "output_schema",
+ "path": "evals/output/schema.json",
+ "exists": true,
+ "bytes": 2193,
+ "sha256": "8ee340c95064260c5e952be614e19841ac676162c6bf01d21b107e38cb04e0b9"
+ },
+ {
+ "label": "output_scorecard",
+ "path": "reports/output_quality_scorecard.json",
+ "exists": true,
+ "bytes": 25530,
+ "sha256": "0806258a8e084b27e112537faff0de64a8519ca90cfdc78b57c0e4c08a514cca"
+ },
+ {
+ "label": "output_execution",
+ "path": "reports/output_execution_runs.json",
+ "exists": true,
+ "bytes": 8449,
+ "sha256": "4df66b63d2e717a559a6881b966fb1b92def7329cc89aa48ee0d8596f87402e5"
+ },
+ {
+ "label": "blind_review",
+ "path": "reports/output_blind_review_pack.json",
+ "exists": true,
+ "bytes": 7804,
+ "sha256": "bbe2db8ec2776fe289cd7d6bb78d48c2b8baa106ad37f79c15e43669b10c9390"
+ },
+ {
+ "label": "review_adjudication",
+ "path": "reports/output_review_adjudication.json",
+ "exists": true,
+ "bytes": 14084,
+ "sha256": "91fd88dd9b0f8876f68027a87893d1852dac59772613345bbbfabba658c9227e"
+ },
+ {
+ "label": "trigger_scorecard",
+ "path": "reports/route_scorecard.json",
+ "exists": true,
+ "bytes": 16950,
+ "sha256": "a649547a7d3d5fe02262ca1b188ed4022d8f557770be425f7b7cf762aa76533a"
+ },
+ {
+ "label": "runtime_conformance",
+ "path": "reports/conformance_matrix.json",
+ "exists": true,
+ "bytes": 10342,
+ "sha256": "97f9ba949c23a60b00e9ba2ff279ca03ba517845cc7c62aa8a42645d58006c7e"
+ },
+ {
+ "label": "trust_report",
+ "path": "reports/security_trust_report.json",
+ "exists": true,
+ "bytes": 137787,
+ "sha256": "bbff1f1bee0ce6f7014efcd4622ba11c70aacfc81f2e69e225411df7004dd616"
+ },
+ {
+ "label": "python_compatibility",
+ "path": "reports/python_compatibility.json",
+ "exists": true,
+ "bytes": 30127,
+ "sha256": "d31dc03c74f17242f4163b8cb3f23120fe3d0712edd08fa183cb94f8120d491c"
+ },
+ {
+ "label": "registry_audit",
+ "path": "reports/registry_audit.json",
+ "exists": true,
+ "bytes": 3137,
+ "sha256": "1feb80c7cdc833a655cc0e44150f39f9f6d0633465abaa1ae29ebe7cb0649065"
+ },
+ {
+ "label": "package_verification",
+ "path": "reports/package_verification.json",
+ "exists": true,
+ "bytes": 19338,
+ "sha256": "7aa6c25e8154d246ace49f7ab9159fa85f96d1d4ebb7e2bb03c7b223260a908b"
+ },
+ {
+ "label": "install_simulation",
+ "path": "reports/install_simulation.json",
+ "exists": true,
+ "bytes": 8758,
+ "sha256": "c1f1122264b868c4998482c580ef94333575c6ff7c5c9c297188b225f69a131d"
+ },
+ {
+ "label": "skill_os2_audit",
+ "path": "reports/skill_os2_audit.json",
+ "exists": true,
+ "bytes": 13824,
+ "sha256": "6362e79a8cfd3ff54b1a7ca32f37de8ed1cda500b953fdf1074928c6d2d62379"
+ },
+ {
+ "label": "world_class_evidence_plan",
+ "path": "reports/world_class_evidence_plan.json",
+ "exists": true,
+ "bytes": 23314,
+ "sha256": "8eb5e1fbaa7580f107fd9920d13a13b239edb72d50b6959fc8f6dd43150f07d7"
+ },
+ {
+ "label": "world_class_evidence_ledger",
+ "path": "reports/world_class_evidence_ledger.json",
+ "exists": true,
+ "bytes": 26277,
+ "sha256": "6038b6cd6e0667f25cfd69a5e40afa314d9efc3a93753a2a3a0509ae9b516a15"
+ },
+ {
+ "label": "world_class_evidence_intake",
+ "path": "reports/world_class_evidence_intake.json",
+ "exists": true,
+ "bytes": 20922,
+ "sha256": "aa262da0ecee7232974ce061a51c8966c310de0b0be9f91eb82bb28bf1c7c0a8"
+ },
+ {
+ "label": "world_class_evidence_preflight",
+ "path": "reports/world_class_evidence_preflight.json",
+ "exists": true,
+ "bytes": 94341,
+ "sha256": "95cb56581224aab06be84d0b338d6ba9aa4aef0b1ebd480801f1e2226b873b0d"
+ },
+ {
+ "label": "world_class_submission_review",
+ "path": "reports/world_class_submission_review.json",
+ "exists": true,
+ "bytes": 17295,
+ "sha256": "12fbb84926294c4bd05af526fcbe0983db26e449d2c305006094baf27332a0ba"
+ },
+ {
+ "label": "world_class_operator_runbook",
+ "path": "reports/world_class_operator_runbook.json",
+ "exists": true,
+ "bytes": 87257,
+ "sha256": "f321dc118ee931a1e1510559e5e64774168a58b0446ae62d232009eb42e3b384"
+ },
+ {
+ "label": "world_class_operator_runbook_markdown",
+ "path": "reports/world_class_operator_runbook.md",
+ "exists": true,
+ "bytes": 25549,
+ "sha256": "858ee11bbacb143f648e9c00735bc248a0b9f3c9303dc9e0ad5c0a65d9d6e07f"
+ },
+ {
+ "label": "world_class_operator_runbook_html",
+ "path": "reports/world_class_operator_runbook.html",
+ "exists": true,
+ "bytes": 34702,
+ "sha256": "31b24658c59ee1dee25aec378e72033b042af88dfdf40c325d361d0c9bec2ca7"
+ },
+ {
+ "label": "world_class_claim_guard",
+ "path": "reports/world_class_claim_guard.json",
+ "exists": true,
+ "bytes": 18596,
+ "sha256": "abe7f7d60c0025e140373fadaefbed4063f285c140c74b9c7cfb464e354bb526"
+ }
+ ],
+ "missing_artifacts": [],
+ "reproduction_commands": [
+ {
+ "label": "source commit",
+ "command": "git rev-parse HEAD",
+ "evidence": "git commit hash"
+ },
+ {
+ "label": "trigger eval",
+ "command": "make eval-suite",
+ "evidence": "reports/eval_suite.json"
+ },
+ {
+ "label": "output eval",
+ "command": "python3 scripts/yao.py output-eval",
+ "evidence": "reports/output_quality_scorecard.json"
+ },
+ {
+ "label": "output execution",
+ "command": "python3 scripts/yao.py output-exec --runner-command '[\"python3\",\"scripts/local_output_eval_runner.py\"]'",
+ "evidence": "reports/output_execution_runs.json"
+ },
+ {
+ "label": "blind review adjudication",
+ "command": "python3 scripts/yao.py output-review",
+ "evidence": "reports/output_review_adjudication.json"
+ },
+ {
+ "label": "skill ir",
+ "command": "python3 scripts/yao.py skill-ir . --output-json skill-ir/examples/yao-meta-skill.json",
+ "evidence": "skill-ir/examples/yao-meta-skill.json"
+ },
+ {
+ "label": "runtime conformance",
+ "command": "python3 scripts/yao.py conformance .",
+ "evidence": "reports/conformance_matrix.json"
+ },
+ {
+ "label": "trust report",
+ "command": "python3 scripts/yao.py trust .",
+ "evidence": "reports/security_trust_report.json"
+ },
+ {
+ "label": "python compatibility",
+ "command": "python3 scripts/yao.py python-compat .",
+ "evidence": "reports/python_compatibility.json"
+ },
+ {
+ "label": "package",
+ "command": "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip",
+ "evidence": "dist/yao-meta-skill.zip"
+ },
+ {
+ "label": "package verify",
+ "command": "python3 scripts/yao.py package-verify . --package-dir dist --require-zip",
+ "evidence": "reports/package_verification.json"
+ },
+ {
+ "label": "install simulate",
+ "command": "python3 scripts/yao.py install-simulate . --package-dir dist",
+ "evidence": "reports/install_simulation.json"
+ },
+ {
+ "label": "registry audit",
+ "command": "python3 scripts/yao.py registry-audit .",
+ "evidence": "reports/registry_audit.json"
+ },
+ {
+ "label": "skill os audit",
+ "command": "python3 scripts/yao.py skill-os2-audit .",
+ "evidence": "reports/skill_os2_audit.json"
+ },
+ {
+ "label": "world-class evidence plan",
+ "command": "python3 scripts/yao.py world-class-evidence .",
+ "evidence": "reports/world_class_evidence_plan.json"
+ },
+ {
+ "label": "world-class evidence ledger",
+ "command": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "evidence": "reports/world_class_evidence_ledger.json"
+ },
+ {
+ "label": "world-class evidence intake",
+ "command": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "evidence": "reports/world_class_evidence_intake.json"
+ },
+ {
+ "label": "world-class evidence preflight",
+ "command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "evidence": "reports/world_class_evidence_preflight.json"
+ },
+ {
+ "label": "world-class submission review",
+ "command": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "evidence": "reports/world_class_submission_review.json"
+ },
+ {
+ "label": "world-class operator runbook",
+ "command": "python3 scripts/yao.py world-class-runbook . --submissions-dir evidence/world_class/submissions",
+ "evidence": "reports/world_class_operator_runbook.json"
+ },
+ {
+ "label": "world-class claim guard",
+ "command": "python3 scripts/yao.py world-class-claim-guard .",
+ "evidence": "reports/world_class_claim_guard.json"
+ },
+ {
+ "label": "evidence consistency",
+ "command": "python3 scripts/yao.py evidence-consistency .",
+ "evidence": "reports/evidence_consistency.json"
+ },
+ {
+ "label": "full ci",
+ "command": "make ci-test",
+ "evidence": "CI target output"
+ }
+ ],
+ "failure_disclosure": {
+ "path": "evals/failure-cases.md",
+ "case_count": 3,
+ "policy": "Keep representative failures visible and tied to regression checks."
+ },
+ "limitations": [
+ "The git commit and dirty flags are generation-time context; release lock is blocked by source changes, while generated evidence artifacts are tracked separately.",
+ "Provider-backed model holdout source evidence is complete, but ledger acceptance still requires a valid independently reviewed submission packet.",
+ "Pending blind-review decisions are visible but do not count as human adjudication.",
+ "World-class readiness remains false until external and human evidence gaps close.",
+ "Beta/public testing may proceed without human blind-review only when wording avoids superiority, fully-reviewed, or world-class claims."
+ ],
+ "artifacts": {
+ "json": "reports/benchmark_reproducibility.json",
+ "markdown": "reports/benchmark_reproducibility.md"
+ }
+}
diff --git a/reports/benchmark_reproducibility.md b/reports/benchmark_reproducibility.md
new file mode 100644
index 00000000..f0a94816
--- /dev/null
+++ b/reports/benchmark_reproducibility.md
@@ -0,0 +1,181 @@
+# Benchmark Reproducibility
+
+Generated at: `2026-06-17`
+Commit: `e79050ed12d057811d7d13ab7d8bcc5a06af76d4`
+Working tree dirty at generation: `true`
+Source tree dirty at generation: `false`
+Generated evidence dirty at generation: `true`
+Evidence bundle SHA256: `55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0`
+
+## Summary
+
+- reproducibility ready: `true`
+- release lock ready: `true`
+- methodology complete: `true`
+- required artifacts: `25`
+- missing artifacts: `0`
+- source contract sha256: `5a99ef3133b0`
+- archive sha256: `7db44f059844`
+- output cases: `5`
+- disclosed failure cases: `3`
+- reproduction commands: `23`
+- provider evidence complete: `true`
+- human review complete: `false`
+- world-class ready: `false`
+- world-class source checks: `11` pass / `19` total; `8` blocked
+- beta test ready: `true`
+- beta test blockers: `0`
+- beta deferred evidence: `4`
+- public claim ready: `false`
+- public claim blockers: `3`
+- changed files at generation: `29`
+- source changed files at generation: `0`
+- generated changed files at generation: `29`
+
+This report proves local benchmark reproducibility only. It keeps external provider and human-review gaps visible instead of counting them as complete. The git commit and dirty samples are generation-time context; the evidence bundle SHA is the durable anchor for the artifacts listed below.
+
+## Beta Test Boundary
+
+- ready: `true`
+- scope: beta/public test release without superiority, fully-reviewed, or world-class claims
+- policy: Human blind-review, native permission enforcement, real client telemetry, and ledger acceptance may be deferred for beta/public testing, but public claims must remain blocked until those evidence entries are accepted.
+- required wording: Use beta, public test, or technical preview wording; do not claim world-class readiness, fully reviewed quality, or proven superiority over baseline.
+
+| Blocker |
+| --- |
+| none |
+
+| Deferred evidence | Reason |
+| --- | --- |
+| `provider-holdout` | Provider-backed source evidence exists, but formal ledger submission and reviewer acceptance are still pending before public claims. |
+| `human-adjudication` | Human adjudication evidence is still pending; deferred for beta/public testing and still required before superiority, fully-reviewed, or world-class claims. |
+| `native-permission-enforcement` | Native enforcement proof is still pending; deferred for beta/public testing and still required before world-class claims. |
+| `native-client-telemetry` | Real client telemetry is still pending; deferred for beta/public testing and still required before world-class claims. |
+
+## Public Claim Boundary
+
+- ready: `false`
+- scope: public benchmark or world-class readiness claim
+- policy: Local reproducibility can pass before public claims; public claims require provider evidence, human adjudication, clean release lock, accepted world-class evidence, and complete source checks.
+
+| Blocker |
+| --- |
+| human blind-review adjudication is incomplete |
+| world-class evidence is not accepted yet (3 open gaps, 4 ledger pending) |
+| world-class source checks are not all accepted (11/19 pass, 8 blocked) |
+
+## Release Lock
+
+- ready: `true`
+- reason: only generated evidence artifacts were dirty at generation time
+- status scope: generation-time status before this report is written
+
+## Evidence Bundle
+
+- algorithm: `sha256(path,label,exists,artifact_sha256)`
+- artifacts: `25` / `25`
+- sha256: `55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0`
+
+## Methodology Sections
+
+| Section | Status |
+| --- | --- |
+| `## Benchmark Types` | present |
+| `## Sample Sources` | present |
+| `## Evaluation Dimensions` | present |
+| `## Weighting Rule` | present |
+| `## Failure Disclosure` | present |
+| `## Reproduction` | present |
+
+## Required Artifacts
+
+| Label | Path | Status | SHA256 |
+| --- | --- | --- | --- |
+| methodology | `reports/benchmark_methodology.md` | present | `57025e0123ce` |
+| failure_disclosure | `evals/failure-cases.md` | present | `28833c0d4a21` |
+| output_cases | `evals/output/cases.jsonl` | present | `a6ae96857116` |
+| output_schema | `evals/output/schema.json` | present | `8ee340c95064` |
+| output_scorecard | `reports/output_quality_scorecard.json` | present | `0806258a8e08` |
+| output_execution | `reports/output_execution_runs.json` | present | `4df66b63d2e7` |
+| blind_review | `reports/output_blind_review_pack.json` | present | `bbe2db8ec277` |
+| review_adjudication | `reports/output_review_adjudication.json` | present | `91fd88dd9b0f` |
+| trigger_scorecard | `reports/route_scorecard.json` | present | `a649547a7d3d` |
+| runtime_conformance | `reports/conformance_matrix.json` | present | `97f9ba949c23` |
+| trust_report | `reports/security_trust_report.json` | present | `bbff1f1bee0c` |
+| python_compatibility | `reports/python_compatibility.json` | present | `d31dc03c74f1` |
+| registry_audit | `reports/registry_audit.json` | present | `1feb80c7cdc8` |
+| package_verification | `reports/package_verification.json` | present | `7aa6c25e8154` |
+| install_simulation | `reports/install_simulation.json` | present | `c1f1122264b8` |
+| skill_os2_audit | `reports/skill_os2_audit.json` | present | `6362e79a8cfd` |
+| world_class_evidence_plan | `reports/world_class_evidence_plan.json` | present | `8eb5e1fbaa75` |
+| world_class_evidence_ledger | `reports/world_class_evidence_ledger.json` | present | `6038b6cd6e06` |
+| world_class_evidence_intake | `reports/world_class_evidence_intake.json` | present | `aa262da0ecee` |
+| world_class_evidence_preflight | `reports/world_class_evidence_preflight.json` | present | `95cb56581224` |
+| world_class_submission_review | `reports/world_class_submission_review.json` | present | `12fbb8492629` |
+| world_class_operator_runbook | `reports/world_class_operator_runbook.json` | present | `f321dc118ee9` |
+| world_class_operator_runbook_markdown | `reports/world_class_operator_runbook.md` | present | `858ee11bbacb` |
+| world_class_operator_runbook_html | `reports/world_class_operator_runbook.html` | present | `31b24658c59e` |
+| world_class_claim_guard | `reports/world_class_claim_guard.json` | present | `abe7f7d60c00` |
+
+## Reproduction Commands
+
+- `git rev-parse HEAD`
+ - evidence: `git commit hash`
+- `make eval-suite`
+ - evidence: `reports/eval_suite.json`
+- `python3 scripts/yao.py output-eval`
+ - evidence: `reports/output_quality_scorecard.json`
+- `python3 scripts/yao.py output-exec --runner-command '["python3","scripts/local_output_eval_runner.py"]'`
+ - evidence: `reports/output_execution_runs.json`
+- `python3 scripts/yao.py output-review`
+ - evidence: `reports/output_review_adjudication.json`
+- `python3 scripts/yao.py skill-ir . --output-json skill-ir/examples/yao-meta-skill.json`
+ - evidence: `skill-ir/examples/yao-meta-skill.json`
+- `python3 scripts/yao.py conformance .`
+ - evidence: `reports/conformance_matrix.json`
+- `python3 scripts/yao.py trust .`
+ - evidence: `reports/security_trust_report.json`
+- `python3 scripts/yao.py python-compat .`
+ - evidence: `reports/python_compatibility.json`
+- `python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip`
+ - evidence: `dist/yao-meta-skill.zip`
+- `python3 scripts/yao.py package-verify . --package-dir dist --require-zip`
+ - evidence: `reports/package_verification.json`
+- `python3 scripts/yao.py install-simulate . --package-dir dist`
+ - evidence: `reports/install_simulation.json`
+- `python3 scripts/yao.py registry-audit .`
+ - evidence: `reports/registry_audit.json`
+- `python3 scripts/yao.py skill-os2-audit .`
+ - evidence: `reports/skill_os2_audit.json`
+- `python3 scripts/yao.py world-class-evidence .`
+ - evidence: `reports/world_class_evidence_plan.json`
+- `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
+ - evidence: `reports/world_class_evidence_ledger.json`
+- `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
+ - evidence: `reports/world_class_evidence_intake.json`
+- `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
+ - evidence: `reports/world_class_evidence_preflight.json`
+- `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
+ - evidence: `reports/world_class_submission_review.json`
+- `python3 scripts/yao.py world-class-runbook . --submissions-dir evidence/world_class/submissions`
+ - evidence: `reports/world_class_operator_runbook.json`
+- `python3 scripts/yao.py world-class-claim-guard .`
+ - evidence: `reports/world_class_claim_guard.json`
+- `python3 scripts/yao.py evidence-consistency .`
+ - evidence: `reports/evidence_consistency.json`
+- `make ci-test`
+ - evidence: `CI target output`
+
+## Failure Disclosure
+
+- path: `evals/failure-cases.md`
+- disclosed cases: `3`
+- policy: Keep representative failures visible and tied to regression checks.
+
+## Limits
+
+- The git commit and dirty flags are generation-time context; release lock is blocked by source changes, while generated evidence artifacts are tracked separately.
+- Provider-backed model holdout source evidence is complete, but ledger acceptance still requires a valid independently reviewed submission packet.
+- Pending blind-review decisions are visible but do not count as human adjudication.
+- World-class readiness remains false until external and human evidence gaps close.
+- Beta/public testing may proceed without human blind-review only when wording avoids superiority, fully-reviewed, or world-class claims.
diff --git a/reports/candidate_registry.json b/reports/candidate_registry.json
index 166fea3c..192a53c8 100644
--- a/reports/candidate_registry.json
+++ b/reports/candidate_registry.json
@@ -3,9 +3,9 @@
{
"name": "yao-meta-skill",
"winner_label": "Current",
- "winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
- "winner_tokens": 65,
- "current_tokens": 65,
+ "winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "winner_tokens": 53,
+ "current_tokens": 53,
"winner_holdout_fp": 0,
"winner_holdout_fn": 0,
"current_holdout_fp": 0,
@@ -36,44 +36,44 @@
"calibration": {
"holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.673,
- "mean_non_trigger_score": 0.0,
- "mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.468,
- "max_non_trigger_score": 0.0,
- "score_gap": 0.468,
- "threshold_margin": 0.138,
- "positive_threshold_buffer": 0.138,
- "negative_threshold_buffer": 0.33,
+ "mean_positive_score": 0.908,
+ "mean_non_trigger_score": 0.003,
+ "mean_near_neighbor_score": 0.006,
+ "min_positive_score": 0.767,
+ "max_non_trigger_score": 0.023,
+ "score_gap": 0.744,
+ "threshold_margin": 0.307,
+ "positive_threshold_buffer": 0.437,
+ "negative_threshold_buffer": 0.307,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"blind_holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.399,
- "mean_non_trigger_score": 0.013,
- "mean_near_neighbor_score": 0.013,
- "min_positive_score": 0.392,
- "max_non_trigger_score": 0.028,
- "score_gap": 0.364,
- "threshold_margin": 0.062,
- "positive_threshold_buffer": 0.062,
- "negative_threshold_buffer": 0.302,
- "boundary_case_count": 2,
- "boundary_case_rate": 0.333,
- "risk_band": "watch"
+ "mean_positive_score": 0.669,
+ "mean_non_trigger_score": 0.116,
+ "mean_near_neighbor_score": 0.152,
+ "min_positive_score": 0.665,
+ "max_non_trigger_score": 0.305,
+ "score_gap": 0.36,
+ "threshold_margin": 0.025,
+ "positive_threshold_buffer": 0.335,
+ "negative_threshold_buffer": 0.025,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "tight"
},
"adversarial_holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.838,
+ "mean_positive_score": 1.0,
"mean_non_trigger_score": 0.0,
"mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.834,
+ "min_positive_score": 1.0,
"max_non_trigger_score": 0.0,
- "score_gap": 0.834,
+ "score_gap": 1.0,
"threshold_margin": 0.33,
- "positive_threshold_buffer": 0.504,
+ "positive_threshold_buffer": 0.67,
"negative_threshold_buffer": 0.33,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
@@ -164,7 +164,7 @@
"route_metrics": {
"precision": 1.0,
"recall": 1.0,
- "average_margin": 0.611,
+ "average_margin": 0.701,
"misroute_count": 0,
"ambiguous_case_count": 0
}
@@ -187,8 +187,8 @@
"label": "Current",
"ranking_state": "selected_by_dev",
"promotion_state": "kept_current",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
- "tokens": 65,
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "tokens": 53,
"dev_errors": 0,
"holdout_errors": 0,
"reason_tags": []
@@ -218,7 +218,8 @@
"dev_errors": 2,
"holdout_errors": 0,
"reason_tags": [
- "weaker_dev_fit"
+ "weaker_dev_fit",
+ "longer_without_gain"
]
},
{
@@ -232,7 +233,8 @@
"dev_errors": 2,
"holdout_errors": 0,
"reason_tags": [
- "weaker_dev_fit"
+ "weaker_dev_fit",
+ "longer_without_gain"
]
},
{
@@ -532,13 +534,13 @@
"current_holdout_fp": 0,
"current_holdout_fn": 1,
"baseline_holdout_fp": 0,
- "baseline_holdout_fn": 2,
+ "baseline_holdout_fn": 1,
"winner_blind_holdout_fp": 0,
"winner_blind_holdout_fn": 0,
"current_blind_holdout_fp": 0,
"current_blind_holdout_fn": 0,
"baseline_blind_holdout_fp": 0,
- "baseline_blind_holdout_fn": 1,
+ "baseline_blind_holdout_fn": 0,
"winner_blind_holdout_total_errors": 0,
"winner_judge_blind_holdout_fp": 0,
"winner_judge_blind_holdout_fn": 0,
@@ -557,7 +559,7 @@
"calibration": {
"holdout": {
"threshold": 0.48,
- "mean_positive_score": 0.37,
+ "mean_positive_score": 0.45,
"mean_non_trigger_score": 0.019,
"mean_near_neighbor_score": 0.037,
"min_positive_score": 0.08,
@@ -572,49 +574,49 @@
},
"blind_holdout": {
"threshold": 0.48,
- "mean_positive_score": 0.902,
- "mean_non_trigger_score": 0.168,
- "mean_near_neighbor_score": 0.138,
- "min_positive_score": 0.804,
- "max_non_trigger_score": 0.386,
- "score_gap": 0.418,
- "threshold_margin": 0.094,
- "positive_threshold_buffer": 0.324,
- "negative_threshold_buffer": 0.094,
- "boundary_case_count": 1,
- "boundary_case_rate": 0.167,
+ "mean_positive_score": 0.82,
+ "mean_non_trigger_score": 0.059,
+ "mean_near_neighbor_score": 0.0,
+ "min_positive_score": 0.645,
+ "max_non_trigger_score": 0.228,
+ "score_gap": 0.417,
+ "threshold_margin": 0.165,
+ "positive_threshold_buffer": 0.165,
+ "negative_threshold_buffer": 0.252,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"adversarial_holdout": {
"threshold": 0.34,
- "mean_positive_score": 0.827,
- "mean_non_trigger_score": 0.027,
- "mean_near_neighbor_score": 0.011,
- "min_positive_score": 0.654,
- "max_non_trigger_score": 0.056,
- "score_gap": 0.598,
- "threshold_margin": 0.284,
- "positive_threshold_buffer": 0.314,
- "negative_threshold_buffer": 0.284,
- "boundary_case_count": 0,
- "boundary_case_rate": 0.0,
- "risk_band": "healthy"
+ "mean_positive_score": 0.907,
+ "mean_non_trigger_score": 0.083,
+ "mean_near_neighbor_score": 0.139,
+ "min_positive_score": 0.815,
+ "max_non_trigger_score": 0.277,
+ "score_gap": 0.538,
+ "threshold_margin": 0.063,
+ "positive_threshold_buffer": 0.475,
+ "negative_threshold_buffer": 0.063,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "watch"
}
},
"judge_blind": {
"winner": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
},
"current": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
},
"baseline": {
"agreement_rate": 1.0,
- "mean_confidence": 0.69,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
@@ -677,7 +679,8 @@
"causes": [
"no_candidate_outperformed_current",
"current_holdout_gap_present",
- "current_holdout_risk"
+ "current_holdout_risk",
+ "adversarial_watch_risk"
],
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
"gate_status": {
@@ -693,7 +696,7 @@
"route_metrics": {
"precision": 1.0,
"recall": 1.0,
- "average_margin": 0.659,
+ "average_margin": 0.586,
"misroute_count": 0,
"ambiguous_case_count": 0
}
@@ -707,8 +710,8 @@
"promotion_state": "reference",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, and stakeholder requirements. Use when asked to standardize incident review, severity assessment, incident communication, or incident packet assembly into one reusable operational skill. Do not use for generic debugging, one-off incident explanations, or brainstorming possible response ideas.",
"tokens": 93,
- "dev_errors": 1,
- "holdout_errors": 2
+ "dev_errors": 0,
+ "holdout_errors": 1
},
{
"name": "governed-incident-command",
@@ -718,25 +721,10 @@
"promotion_state": "kept_current",
"description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 37,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": []
},
- {
- "name": "governed-incident-command",
- "role": "candidate",
- "label": "Guardrail",
- "ranking_state": "not_selected",
- "promotion_state": "blocked",
- "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
- "tokens": 51,
- "dev_errors": 1,
- "holdout_errors": 2,
- "reason_tags": [
- "visible_holdout_regression",
- "longer_without_gain"
- ]
- },
{
"name": "governed-incident-command",
"role": "candidate",
@@ -745,7 +733,7 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 54,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": [
"longer_without_gain"
@@ -759,7 +747,7 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
"tokens": 78,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": [
"longer_without_gain"
@@ -773,9 +761,24 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 78,
+ "dev_errors": 0,
+ "holdout_errors": 1,
+ "reason_tags": [
+ "longer_without_gain"
+ ]
+ },
+ {
+ "name": "governed-incident-command",
+ "role": "candidate",
+ "label": "Guardrail",
+ "ranking_state": "not_selected",
+ "promotion_state": "blocked",
+ "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
+ "tokens": 51,
"dev_errors": 1,
"holdout_errors": 1,
"reason_tags": [
+ "weaker_dev_fit",
"longer_without_gain"
]
}
diff --git a/reports/candidate_registry.md b/reports/candidate_registry.md
index 1dd6df26..d6eae6ea 100644
--- a/reports/candidate_registry.md
+++ b/reports/candidate_registry.md
@@ -3,10 +3,10 @@
| Target | Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors | Reason Tags |
| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
| `yao-meta-skill` | baseline | `Baseline` | reference | reference | 8 | 1 | 0 | - |
-| `yao-meta-skill` | current | `Current` | selected_by_dev | kept_current | 65 | 0 | 0 | - |
+| `yao-meta-skill` | current | `Current` | selected_by_dev | kept_current | 53 | 0 | 0 | - |
| `yao-meta-skill` | candidate | `Minimal` | not_selected | blocked | 41 | 2 | 0 | weaker_dev_fit |
-| `yao-meta-skill` | candidate | `Guardrail` | not_selected | blocked | 56 | 2 | 0 | weaker_dev_fit |
-| `yao-meta-skill` | candidate | `Balanced` | not_selected | blocked | 60 | 2 | 0 | weaker_dev_fit |
+| `yao-meta-skill` | candidate | `Guardrail` | not_selected | blocked | 56 | 2 | 0 | weaker_dev_fit, longer_without_gain |
+| `yao-meta-skill` | candidate | `Balanced` | not_selected | blocked | 60 | 2 | 0 | weaker_dev_fit, longer_without_gain |
| `yao-meta-skill` | candidate | `Artifact Aware` | not_selected | blocked | 77 | 2 | 0 | weaker_dev_fit, longer_without_gain |
| `yao-meta-skill` | candidate | `Boundary` | not_selected | blocked | 83 | 2 | 0 | weaker_dev_fit, longer_without_gain |
| `team-frontend-review` | baseline | `Baseline` | reference | reference | 52 | 3 | 0 | - |
@@ -15,9 +15,9 @@
| `team-frontend-review` | candidate | `Balanced` | not_selected | blocked | 64 | 3 | 0 | longer_without_gain |
| `team-frontend-review` | candidate | `Artifact Aware` | not_selected | blocked | 84 | 3 | 0 | longer_without_gain |
| `team-frontend-review` | candidate | `Boundary` | not_selected | blocked | 90 | 3 | 0 | longer_without_gain |
-| `governed-incident-command` | baseline | `Baseline` | reference | reference | 93 | 1 | 2 | - |
-| `governed-incident-command` | current | `Current` | selected_by_dev | kept_current | 37 | 1 | 1 | - |
-| `governed-incident-command` | candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 2 | visible_holdout_regression, longer_without_gain |
-| `governed-incident-command` | candidate | `Balanced` | not_selected | blocked | 54 | 1 | 1 | longer_without_gain |
-| `governed-incident-command` | candidate | `Boundary` | not_selected | blocked | 78 | 1 | 1 | longer_without_gain |
-| `governed-incident-command` | candidate | `Artifact Aware` | not_selected | blocked | 78 | 1 | 1 | longer_without_gain |
+| `governed-incident-command` | baseline | `Baseline` | reference | reference | 93 | 0 | 1 | - |
+| `governed-incident-command` | current | `Current` | selected_by_dev | kept_current | 37 | 0 | 1 | - |
+| `governed-incident-command` | candidate | `Balanced` | not_selected | blocked | 54 | 0 | 1 | longer_without_gain |
+| `governed-incident-command` | candidate | `Boundary` | not_selected | blocked | 78 | 0 | 1 | longer_without_gain |
+| `governed-incident-command` | candidate | `Artifact Aware` | not_selected | blocked | 78 | 0 | 1 | longer_without_gain |
+| `governed-incident-command` | candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 1 | weaker_dev_fit, longer_without_gain |
diff --git a/reports/compiled_targets.json b/reports/compiled_targets.json
index fc077654..f9f14ed0 100644
--- a/reports/compiled_targets.json
+++ b/reports/compiled_targets.json
@@ -1,11 +1,11 @@
{
"schema_version": "1.0",
"ok": true,
- "generated_at": "2026-06-13",
+ "generated_at": "2026-06-17",
"skill_dir": ".",
"summary": {
- "target_count": 4,
- "pass_count": 4,
+ "target_count": 5,
+ "pass_count": 5,
"warn_count": 0,
"block_count": 0,
"failure_count": 0,
@@ -30,17 +30,18 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
]
},
"compiled_contract": {
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"job_to_be_done": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
"trigger": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"should_trigger": [
"Create a skill from this repeated workflow.",
"Turn this runbook into a reusable agent skill.",
@@ -76,21 +77,18 @@
},
"workflow": {
"steps": [
- "Decide whether the request should become a skill and choose the lightest fit.",
- "Capture job, output, exclusions, constraints, and standards.",
- "Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.",
- "Write the `description` early and test route quality before expanding the package.",
- "Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.",
- "Add only folders and gates that earn their keep.",
- "Surface the top three next iteration directions."
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
],
"decision_points": [
"Route by frontmatter `description`.",
"Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.",
"Use the lightest reliable process.",
- "`Scaffold`: exploratory or personal.",
- "`Production`: team reuse.",
- "`Library`: shared infrastructure.",
+ "`Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.",
+ "Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).",
"Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
"Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
"Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
@@ -112,6 +110,7 @@
"references": [
"references/artifact-design-doctrine.md",
"references/authoring-discipline.md",
+ "references/autonomous-adaptation.md",
"references/distribution-registry-method.md",
"references/eval-playbook.md",
"references/gate-selection.md",
@@ -139,39 +138,71 @@
"references/skill-atlas-method.md",
"references/skill-engineering-method.md",
"references/skill-ir-method.md",
+ "references/skillops-decision-policy.md",
"references/systems-thinking-doctrine.md",
"references/telemetry-drift-method.md",
- "references/trust-security-method.md"
+ "references/trust-security-method.md",
+ "references/user-memory-policy.md"
],
"scripts": [
+ "scripts/adaptation_patch_safety.py",
+ "scripts/adaptation_report_contracts.py",
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
+ "scripts/build_skill_atlas_layout.py",
+ "scripts/build_skill_atlas_opportunities.py",
"scripts/check_update.py",
"scripts/ci_test.py",
"scripts/collect_feedback.py",
"scripts/compile_skill.py",
+ "scripts/compile_skill_targets.py",
"scripts/context_sizer.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/cross_packager_contracts.py",
+ "scripts/description_optimizer_reporting.py",
"scripts/diff_eval.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/evidence_consistency_artifact_roles.py",
+ "scripts/evidence_consistency_core.py",
+ "scripts/evidence_consistency_phase_queue.py",
+ "scripts/evidence_consistency_release.py",
+ "scripts/evidence_consistency_skill_os2_review.py",
+ "scripts/evidence_consistency_world_class.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
"scripts/governance_check.py",
+ "scripts/html_rendering.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
"scripts/judge_blind_eval.py",
"scripts/lint_skill.py",
"scripts/local_output_eval_runner.py",
"scripts/optimize_description.py",
+ "scripts/output_review_privacy.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/provider_output_eval_runner.py",
+ "scripts/python_compat_check.py",
+ "scripts/reference_synthesis_markdown.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -186,32 +217,93 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/resource_boundary_check.py",
+ "scripts/review_studio_action_evidence.py",
+ "scripts/review_studio_actions.py",
+ "scripts/review_studio_data.py",
"scripts/review_studio_formatting.py",
+ "scripts/review_studio_gate_contract.py",
+ "scripts/review_studio_gate_helpers.py",
+ "scripts/review_studio_gates.py",
"scripts/review_studio_layout.py",
+ "scripts/review_studio_output_review.py",
+ "scripts/review_studio_panels.py",
+ "scripts/review_studio_skillops.py",
+ "scripts/review_studio_waivers.py",
+ "scripts/review_studio_world_class.py",
+ "scripts/review_viewer_data.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_eval_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/skill_ir_paths.py",
+ "scripts/skill_os2_coverage_markdown.py",
"scripts/skill_report_charts.py",
+ "scripts/skill_report_i18n.py",
"scripts/skill_report_layout.py",
"scripts/skill_report_metrics.py",
"scripts/skill_report_model.py",
+ "scripts/skill_report_sections.py",
+ "scripts/skill_report_sources.py",
+ "scripts/skill_report_world_class.py",
+ "scripts/skillops_opportunity.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trigger_eval.py",
"scripts/trust_check.py",
+ "scripts/trust_check_scripts.py",
"scripts/upgrade_check.py",
"scripts/validate_skill.py",
"scripts/verify_package.py",
+ "scripts/world_class_evidence_contract.py",
+ "scripts/world_class_human_evidence.py",
+ "scripts/world_class_native_permission_evidence.py",
+ "scripts/world_class_native_telemetry_evidence.py",
+ "scripts/world_class_operator_runbook_coordination.py",
+ "scripts/world_class_phase_queue.py",
+ "scripts/world_class_preflight_layout.py",
+ "scripts/world_class_preflight_markdown.py",
+ "scripts/world_class_provider_evidence.py",
+ "scripts/world_class_repair_checklist.py",
+ "scripts/world_class_source_checks.py",
+ "scripts/world_class_submission_kit_rendering.py",
+ "scripts/world_class_submission_matrix.py",
"scripts/yao.py",
- "scripts/yao_cli_config.py"
+ "scripts/yao_cli_adaptation_commands.py",
+ "scripts/yao_cli_config.py",
+ "scripts/yao_cli_create_commands.py",
+ "scripts/yao_cli_distribution_commands.py",
+ "scripts/yao_cli_output_commands.py",
+ "scripts/yao_cli_parser.py",
+ "scripts/yao_cli_parser_evidence.py",
+ "scripts/yao_cli_parser_operations.py",
+ "scripts/yao_cli_report_commands.py",
+ "scripts/yao_cli_runtime.py",
+ "scripts/yao_cli_telemetry.py"
],
"assets": [
+ "assets/review-studio.css",
+ "assets/review-viewer.css",
+ "assets/skill-overview.css",
+ "assets/skill-overview.js",
"templates/basic_skill.md.j2",
"templates/complex_skill.md.j2"
],
@@ -251,6 +343,8 @@
"reports/security_trust_report.md",
"reports/runtime_permission_probes.json",
"reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
"reports/skill_atlas.json",
"reports/skill_atlas.html",
"reports/skill-os-2-review.md",
@@ -259,15 +353,16 @@
"reports/governance_score.json"
],
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
}
},
"eval_plan": {
"trigger": [
"evals/README.md",
+ "evals/adaptation/user_signals.example.jsonl",
"evals/adversarial/trigger_cases.json",
"evals/blind_holdout/trigger_cases.json",
"evals/confusion/route_cases.json",
@@ -304,7 +399,7 @@
],
"baseline": "without_skill",
"counts": {
- "trigger": 26,
+ "trigger": 27,
"output": 3,
"adversarial": 1,
"baseline": 1
@@ -338,7 +433,8 @@
"adapter_targets": [
"openai",
"claude",
- "generic"
+ "generic",
+ "vscode"
],
"canonical_format": "agent-skills"
},
@@ -355,18 +451,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -375,18 +473,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -401,15 +510,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -418,15 +541,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -441,22 +569,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -476,9 +605,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -490,17 +619,17 @@
"native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
"activation": {
"policy": "Use frontmatter description for catalog routing and targets/openai/agents/openai.yaml for display name, default prompt, and compatibility metadata.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/openai/adapter.json",
@@ -509,7 +638,7 @@
},
"scripts": {
"strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -580,18 +709,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -600,18 +731,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -626,15 +768,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -643,15 +799,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -666,22 +827,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -699,9 +861,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -713,17 +875,17 @@
"native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
"activation": {
"policy": "Use frontmatter description for catalog routing and targets/openai/agents/openai.yaml for display name, default prompt, and compatibility metadata.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/openai/adapter.json",
@@ -732,7 +894,7 @@
},
"scripts": {
"strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -819,17 +981,18 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
]
},
"compiled_contract": {
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"job_to_be_done": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
"trigger": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"should_trigger": [
"Create a skill from this repeated workflow.",
"Turn this runbook into a reusable agent skill.",
@@ -865,21 +1028,18 @@
},
"workflow": {
"steps": [
- "Decide whether the request should become a skill and choose the lightest fit.",
- "Capture job, output, exclusions, constraints, and standards.",
- "Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.",
- "Write the `description` early and test route quality before expanding the package.",
- "Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.",
- "Add only folders and gates that earn their keep.",
- "Surface the top three next iteration directions."
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
],
"decision_points": [
"Route by frontmatter `description`.",
"Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.",
"Use the lightest reliable process.",
- "`Scaffold`: exploratory or personal.",
- "`Production`: team reuse.",
- "`Library`: shared infrastructure.",
+ "`Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.",
+ "Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).",
"Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
"Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
"Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
@@ -901,6 +1061,7 @@
"references": [
"references/artifact-design-doctrine.md",
"references/authoring-discipline.md",
+ "references/autonomous-adaptation.md",
"references/distribution-registry-method.md",
"references/eval-playbook.md",
"references/gate-selection.md",
@@ -928,39 +1089,71 @@
"references/skill-atlas-method.md",
"references/skill-engineering-method.md",
"references/skill-ir-method.md",
+ "references/skillops-decision-policy.md",
"references/systems-thinking-doctrine.md",
"references/telemetry-drift-method.md",
- "references/trust-security-method.md"
+ "references/trust-security-method.md",
+ "references/user-memory-policy.md"
],
"scripts": [
+ "scripts/adaptation_patch_safety.py",
+ "scripts/adaptation_report_contracts.py",
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
+ "scripts/build_skill_atlas_layout.py",
+ "scripts/build_skill_atlas_opportunities.py",
"scripts/check_update.py",
"scripts/ci_test.py",
"scripts/collect_feedback.py",
"scripts/compile_skill.py",
+ "scripts/compile_skill_targets.py",
"scripts/context_sizer.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/cross_packager_contracts.py",
+ "scripts/description_optimizer_reporting.py",
"scripts/diff_eval.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/evidence_consistency_artifact_roles.py",
+ "scripts/evidence_consistency_core.py",
+ "scripts/evidence_consistency_phase_queue.py",
+ "scripts/evidence_consistency_release.py",
+ "scripts/evidence_consistency_skill_os2_review.py",
+ "scripts/evidence_consistency_world_class.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
"scripts/governance_check.py",
+ "scripts/html_rendering.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
"scripts/judge_blind_eval.py",
"scripts/lint_skill.py",
"scripts/local_output_eval_runner.py",
"scripts/optimize_description.py",
+ "scripts/output_review_privacy.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/provider_output_eval_runner.py",
+ "scripts/python_compat_check.py",
+ "scripts/reference_synthesis_markdown.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -975,32 +1168,93 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/resource_boundary_check.py",
+ "scripts/review_studio_action_evidence.py",
+ "scripts/review_studio_actions.py",
+ "scripts/review_studio_data.py",
"scripts/review_studio_formatting.py",
+ "scripts/review_studio_gate_contract.py",
+ "scripts/review_studio_gate_helpers.py",
+ "scripts/review_studio_gates.py",
"scripts/review_studio_layout.py",
+ "scripts/review_studio_output_review.py",
+ "scripts/review_studio_panels.py",
+ "scripts/review_studio_skillops.py",
+ "scripts/review_studio_waivers.py",
+ "scripts/review_studio_world_class.py",
+ "scripts/review_viewer_data.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_eval_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/skill_ir_paths.py",
+ "scripts/skill_os2_coverage_markdown.py",
"scripts/skill_report_charts.py",
+ "scripts/skill_report_i18n.py",
"scripts/skill_report_layout.py",
"scripts/skill_report_metrics.py",
"scripts/skill_report_model.py",
+ "scripts/skill_report_sections.py",
+ "scripts/skill_report_sources.py",
+ "scripts/skill_report_world_class.py",
+ "scripts/skillops_opportunity.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trigger_eval.py",
"scripts/trust_check.py",
+ "scripts/trust_check_scripts.py",
"scripts/upgrade_check.py",
"scripts/validate_skill.py",
"scripts/verify_package.py",
+ "scripts/world_class_evidence_contract.py",
+ "scripts/world_class_human_evidence.py",
+ "scripts/world_class_native_permission_evidence.py",
+ "scripts/world_class_native_telemetry_evidence.py",
+ "scripts/world_class_operator_runbook_coordination.py",
+ "scripts/world_class_phase_queue.py",
+ "scripts/world_class_preflight_layout.py",
+ "scripts/world_class_preflight_markdown.py",
+ "scripts/world_class_provider_evidence.py",
+ "scripts/world_class_repair_checklist.py",
+ "scripts/world_class_source_checks.py",
+ "scripts/world_class_submission_kit_rendering.py",
+ "scripts/world_class_submission_matrix.py",
"scripts/yao.py",
- "scripts/yao_cli_config.py"
+ "scripts/yao_cli_adaptation_commands.py",
+ "scripts/yao_cli_config.py",
+ "scripts/yao_cli_create_commands.py",
+ "scripts/yao_cli_distribution_commands.py",
+ "scripts/yao_cli_output_commands.py",
+ "scripts/yao_cli_parser.py",
+ "scripts/yao_cli_parser_evidence.py",
+ "scripts/yao_cli_parser_operations.py",
+ "scripts/yao_cli_report_commands.py",
+ "scripts/yao_cli_runtime.py",
+ "scripts/yao_cli_telemetry.py"
],
"assets": [
+ "assets/review-studio.css",
+ "assets/review-viewer.css",
+ "assets/skill-overview.css",
+ "assets/skill-overview.js",
"templates/basic_skill.md.j2",
"templates/complex_skill.md.j2"
],
@@ -1040,6 +1294,8 @@
"reports/security_trust_report.md",
"reports/runtime_permission_probes.json",
"reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
"reports/skill_atlas.json",
"reports/skill_atlas.html",
"reports/skill-os-2-review.md",
@@ -1048,15 +1304,16 @@
"reports/governance_score.json"
],
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
}
},
"eval_plan": {
"trigger": [
"evals/README.md",
+ "evals/adaptation/user_signals.example.jsonl",
"evals/adversarial/trigger_cases.json",
"evals/blind_holdout/trigger_cases.json",
"evals/confusion/route_cases.json",
@@ -1093,7 +1350,7 @@
],
"baseline": "without_skill",
"counts": {
- "trigger": 26,
+ "trigger": 27,
"output": 3,
"adversarial": 1,
"baseline": 1
@@ -1127,7 +1384,8 @@
"adapter_targets": [
"openai",
"claude",
- "generic"
+ "generic",
+ "vscode"
],
"canonical_format": "agent-skills"
},
@@ -1144,18 +1402,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -1164,18 +1424,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -1190,15 +1461,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -1207,15 +1492,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -1230,22 +1520,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -1265,9 +1556,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -1279,17 +1570,17 @@
"native_surface": "Claude-compatible neutral source folder with adapter notes",
"activation": {
"policy": "Use SKILL.md frontmatter description as the primary activation contract and adapter.json for review metadata.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/claude/adapter.json",
@@ -1298,7 +1589,7 @@
},
"scripts": {
"strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -1369,18 +1660,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -1389,18 +1682,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -1415,15 +1719,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -1432,15 +1750,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -1455,22 +1778,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -1488,9 +1812,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -1502,17 +1826,17 @@
"native_surface": "Claude-compatible neutral source folder with adapter notes",
"activation": {
"policy": "Use SKILL.md frontmatter description as the primary activation contract and adapter.json for review metadata.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/claude/adapter.json",
@@ -1521,7 +1845,7 @@
},
"scripts": {
"strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -1608,17 +1932,18 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
]
},
"compiled_contract": {
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"job_to_be_done": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
"trigger": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"should_trigger": [
"Create a skill from this repeated workflow.",
"Turn this runbook into a reusable agent skill.",
@@ -1654,21 +1979,18 @@
},
"workflow": {
"steps": [
- "Decide whether the request should become a skill and choose the lightest fit.",
- "Capture job, output, exclusions, constraints, and standards.",
- "Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.",
- "Write the `description` early and test route quality before expanding the package.",
- "Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.",
- "Add only folders and gates that earn their keep.",
- "Surface the top three next iteration directions."
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
],
"decision_points": [
"Route by frontmatter `description`.",
"Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.",
"Use the lightest reliable process.",
- "`Scaffold`: exploratory or personal.",
- "`Production`: team reuse.",
- "`Library`: shared infrastructure.",
+ "`Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.",
+ "Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).",
"Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
"Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
"Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
@@ -1690,6 +2012,7 @@
"references": [
"references/artifact-design-doctrine.md",
"references/authoring-discipline.md",
+ "references/autonomous-adaptation.md",
"references/distribution-registry-method.md",
"references/eval-playbook.md",
"references/gate-selection.md",
@@ -1717,39 +2040,71 @@
"references/skill-atlas-method.md",
"references/skill-engineering-method.md",
"references/skill-ir-method.md",
+ "references/skillops-decision-policy.md",
"references/systems-thinking-doctrine.md",
"references/telemetry-drift-method.md",
- "references/trust-security-method.md"
+ "references/trust-security-method.md",
+ "references/user-memory-policy.md"
],
"scripts": [
+ "scripts/adaptation_patch_safety.py",
+ "scripts/adaptation_report_contracts.py",
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
+ "scripts/build_skill_atlas_layout.py",
+ "scripts/build_skill_atlas_opportunities.py",
"scripts/check_update.py",
"scripts/ci_test.py",
"scripts/collect_feedback.py",
"scripts/compile_skill.py",
+ "scripts/compile_skill_targets.py",
"scripts/context_sizer.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/cross_packager_contracts.py",
+ "scripts/description_optimizer_reporting.py",
"scripts/diff_eval.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/evidence_consistency_artifact_roles.py",
+ "scripts/evidence_consistency_core.py",
+ "scripts/evidence_consistency_phase_queue.py",
+ "scripts/evidence_consistency_release.py",
+ "scripts/evidence_consistency_skill_os2_review.py",
+ "scripts/evidence_consistency_world_class.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
"scripts/governance_check.py",
+ "scripts/html_rendering.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
"scripts/judge_blind_eval.py",
"scripts/lint_skill.py",
"scripts/local_output_eval_runner.py",
"scripts/optimize_description.py",
+ "scripts/output_review_privacy.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/provider_output_eval_runner.py",
+ "scripts/python_compat_check.py",
+ "scripts/reference_synthesis_markdown.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -1764,32 +2119,93 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/resource_boundary_check.py",
+ "scripts/review_studio_action_evidence.py",
+ "scripts/review_studio_actions.py",
+ "scripts/review_studio_data.py",
"scripts/review_studio_formatting.py",
+ "scripts/review_studio_gate_contract.py",
+ "scripts/review_studio_gate_helpers.py",
+ "scripts/review_studio_gates.py",
"scripts/review_studio_layout.py",
+ "scripts/review_studio_output_review.py",
+ "scripts/review_studio_panels.py",
+ "scripts/review_studio_skillops.py",
+ "scripts/review_studio_waivers.py",
+ "scripts/review_studio_world_class.py",
+ "scripts/review_viewer_data.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_eval_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/skill_ir_paths.py",
+ "scripts/skill_os2_coverage_markdown.py",
"scripts/skill_report_charts.py",
+ "scripts/skill_report_i18n.py",
"scripts/skill_report_layout.py",
"scripts/skill_report_metrics.py",
"scripts/skill_report_model.py",
+ "scripts/skill_report_sections.py",
+ "scripts/skill_report_sources.py",
+ "scripts/skill_report_world_class.py",
+ "scripts/skillops_opportunity.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trigger_eval.py",
"scripts/trust_check.py",
+ "scripts/trust_check_scripts.py",
"scripts/upgrade_check.py",
"scripts/validate_skill.py",
"scripts/verify_package.py",
+ "scripts/world_class_evidence_contract.py",
+ "scripts/world_class_human_evidence.py",
+ "scripts/world_class_native_permission_evidence.py",
+ "scripts/world_class_native_telemetry_evidence.py",
+ "scripts/world_class_operator_runbook_coordination.py",
+ "scripts/world_class_phase_queue.py",
+ "scripts/world_class_preflight_layout.py",
+ "scripts/world_class_preflight_markdown.py",
+ "scripts/world_class_provider_evidence.py",
+ "scripts/world_class_repair_checklist.py",
+ "scripts/world_class_source_checks.py",
+ "scripts/world_class_submission_kit_rendering.py",
+ "scripts/world_class_submission_matrix.py",
"scripts/yao.py",
- "scripts/yao_cli_config.py"
+ "scripts/yao_cli_adaptation_commands.py",
+ "scripts/yao_cli_config.py",
+ "scripts/yao_cli_create_commands.py",
+ "scripts/yao_cli_distribution_commands.py",
+ "scripts/yao_cli_output_commands.py",
+ "scripts/yao_cli_parser.py",
+ "scripts/yao_cli_parser_evidence.py",
+ "scripts/yao_cli_parser_operations.py",
+ "scripts/yao_cli_report_commands.py",
+ "scripts/yao_cli_runtime.py",
+ "scripts/yao_cli_telemetry.py"
],
"assets": [
+ "assets/review-studio.css",
+ "assets/review-viewer.css",
+ "assets/skill-overview.css",
+ "assets/skill-overview.js",
"templates/basic_skill.md.j2",
"templates/complex_skill.md.j2"
],
@@ -1829,6 +2245,8 @@
"reports/security_trust_report.md",
"reports/runtime_permission_probes.json",
"reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
"reports/skill_atlas.json",
"reports/skill_atlas.html",
"reports/skill-os-2-review.md",
@@ -1837,15 +2255,16 @@
"reports/governance_score.json"
],
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
}
},
"eval_plan": {
"trigger": [
"evals/README.md",
+ "evals/adaptation/user_signals.example.jsonl",
"evals/adversarial/trigger_cases.json",
"evals/blind_holdout/trigger_cases.json",
"evals/confusion/route_cases.json",
@@ -1882,7 +2301,7 @@
],
"baseline": "without_skill",
"counts": {
- "trigger": 26,
+ "trigger": 27,
"output": 3,
"adversarial": 1,
"baseline": 1
@@ -1916,7 +2335,8 @@
"adapter_targets": [
"openai",
"claude",
- "generic"
+ "generic",
+ "vscode"
],
"canonical_format": "agent-skills"
},
@@ -1933,18 +2353,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -1953,18 +2375,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -1979,15 +2412,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -1996,15 +2443,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -2019,22 +2471,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -2054,9 +2507,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -2068,17 +2521,17 @@
"native_surface": "Agent Skills compatible neutral package",
"activation": {
"policy": "Use SKILL.md name and description; consumers decide automatic or manual activation.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/generic/adapter.json"
@@ -2086,7 +2539,7 @@
},
"scripts": {
"strategy": "Expose script and permission metadata for downstream clients or installers to enforce.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -2151,18 +2604,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -2171,18 +2626,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -2197,15 +2663,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -2214,15 +2694,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -2237,22 +2722,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -2270,9 +2756,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -2284,17 +2770,17 @@
"native_surface": "Agent Skills compatible neutral package",
"activation": {
"policy": "Use SKILL.md name and description; consumers decide automatic or manual activation.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"targets/generic/adapter.json"
@@ -2302,7 +2788,7 @@
},
"scripts": {
"strategy": "Expose script and permission metadata for downstream clients or installers to enforce.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -2381,17 +2867,18 @@
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
]
},
"compiled_contract": {
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"job_to_be_done": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
"trigger": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"should_trigger": [
"Create a skill from this repeated workflow.",
"Turn this runbook into a reusable agent skill.",
@@ -2427,21 +2914,18 @@
},
"workflow": {
"steps": [
- "Decide whether the request should become a skill and choose the lightest fit.",
- "Capture job, output, exclusions, constraints, and standards.",
- "Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.",
- "Write the `description` early and test route quality before expanding the package.",
- "Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.",
- "Add only folders and gates that earn their keep.",
- "Surface the top three next iteration directions."
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
],
"decision_points": [
"Route by frontmatter `description`.",
"Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.",
"Use the lightest reliable process.",
- "`Scaffold`: exploratory or personal.",
- "`Production`: team reuse.",
- "`Library`: shared infrastructure.",
+ "`Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.",
+ "Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).",
"Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
"Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
"Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
@@ -2463,6 +2947,7 @@
"references": [
"references/artifact-design-doctrine.md",
"references/authoring-discipline.md",
+ "references/autonomous-adaptation.md",
"references/distribution-registry-method.md",
"references/eval-playbook.md",
"references/gate-selection.md",
@@ -2490,39 +2975,71 @@
"references/skill-atlas-method.md",
"references/skill-engineering-method.md",
"references/skill-ir-method.md",
+ "references/skillops-decision-policy.md",
"references/systems-thinking-doctrine.md",
"references/telemetry-drift-method.md",
- "references/trust-security-method.md"
+ "references/trust-security-method.md",
+ "references/user-memory-policy.md"
],
"scripts": [
+ "scripts/adaptation_patch_safety.py",
+ "scripts/adaptation_report_contracts.py",
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
+ "scripts/build_skill_atlas_layout.py",
+ "scripts/build_skill_atlas_opportunities.py",
"scripts/check_update.py",
"scripts/ci_test.py",
"scripts/collect_feedback.py",
"scripts/compile_skill.py",
+ "scripts/compile_skill_targets.py",
"scripts/context_sizer.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/cross_packager_contracts.py",
+ "scripts/description_optimizer_reporting.py",
"scripts/diff_eval.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/evidence_consistency_artifact_roles.py",
+ "scripts/evidence_consistency_core.py",
+ "scripts/evidence_consistency_phase_queue.py",
+ "scripts/evidence_consistency_release.py",
+ "scripts/evidence_consistency_skill_os2_review.py",
+ "scripts/evidence_consistency_world_class.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
"scripts/governance_check.py",
+ "scripts/html_rendering.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
"scripts/judge_blind_eval.py",
"scripts/lint_skill.py",
"scripts/local_output_eval_runner.py",
"scripts/optimize_description.py",
+ "scripts/output_review_privacy.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/provider_output_eval_runner.py",
+ "scripts/python_compat_check.py",
+ "scripts/reference_synthesis_markdown.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -2537,32 +3054,93 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/resource_boundary_check.py",
+ "scripts/review_studio_action_evidence.py",
+ "scripts/review_studio_actions.py",
+ "scripts/review_studio_data.py",
"scripts/review_studio_formatting.py",
+ "scripts/review_studio_gate_contract.py",
+ "scripts/review_studio_gate_helpers.py",
+ "scripts/review_studio_gates.py",
"scripts/review_studio_layout.py",
+ "scripts/review_studio_output_review.py",
+ "scripts/review_studio_panels.py",
+ "scripts/review_studio_skillops.py",
+ "scripts/review_studio_waivers.py",
+ "scripts/review_studio_world_class.py",
+ "scripts/review_viewer_data.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_eval_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/skill_ir_paths.py",
+ "scripts/skill_os2_coverage_markdown.py",
"scripts/skill_report_charts.py",
+ "scripts/skill_report_i18n.py",
"scripts/skill_report_layout.py",
"scripts/skill_report_metrics.py",
"scripts/skill_report_model.py",
+ "scripts/skill_report_sections.py",
+ "scripts/skill_report_sources.py",
+ "scripts/skill_report_world_class.py",
+ "scripts/skillops_opportunity.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trigger_eval.py",
"scripts/trust_check.py",
+ "scripts/trust_check_scripts.py",
"scripts/upgrade_check.py",
"scripts/validate_skill.py",
"scripts/verify_package.py",
+ "scripts/world_class_evidence_contract.py",
+ "scripts/world_class_human_evidence.py",
+ "scripts/world_class_native_permission_evidence.py",
+ "scripts/world_class_native_telemetry_evidence.py",
+ "scripts/world_class_operator_runbook_coordination.py",
+ "scripts/world_class_phase_queue.py",
+ "scripts/world_class_preflight_layout.py",
+ "scripts/world_class_preflight_markdown.py",
+ "scripts/world_class_provider_evidence.py",
+ "scripts/world_class_repair_checklist.py",
+ "scripts/world_class_source_checks.py",
+ "scripts/world_class_submission_kit_rendering.py",
+ "scripts/world_class_submission_matrix.py",
"scripts/yao.py",
- "scripts/yao_cli_config.py"
+ "scripts/yao_cli_adaptation_commands.py",
+ "scripts/yao_cli_config.py",
+ "scripts/yao_cli_create_commands.py",
+ "scripts/yao_cli_distribution_commands.py",
+ "scripts/yao_cli_output_commands.py",
+ "scripts/yao_cli_parser.py",
+ "scripts/yao_cli_parser_evidence.py",
+ "scripts/yao_cli_parser_operations.py",
+ "scripts/yao_cli_report_commands.py",
+ "scripts/yao_cli_runtime.py",
+ "scripts/yao_cli_telemetry.py"
],
"assets": [
+ "assets/review-studio.css",
+ "assets/review-viewer.css",
+ "assets/skill-overview.css",
+ "assets/skill-overview.js",
"templates/basic_skill.md.j2",
"templates/complex_skill.md.j2"
],
@@ -2602,6 +3180,8 @@
"reports/security_trust_report.md",
"reports/runtime_permission_probes.json",
"reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
"reports/skill_atlas.json",
"reports/skill_atlas.html",
"reports/skill-os-2-review.md",
@@ -2610,15 +3190,16 @@
"reports/governance_score.json"
],
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
}
},
"eval_plan": {
"trigger": [
"evals/README.md",
+ "evals/adaptation/user_signals.example.jsonl",
"evals/adversarial/trigger_cases.json",
"evals/blind_holdout/trigger_cases.json",
"evals/confusion/route_cases.json",
@@ -2655,7 +3236,7 @@
],
"baseline": "without_skill",
"counts": {
- "trigger": 26,
+ "trigger": 27,
"output": 3,
"adversarial": 1,
"baseline": 1
@@ -2689,7 +3270,8 @@
"adapter_targets": [
"openai",
"claude",
- "generic"
+ "generic",
+ "vscode"
],
"canonical_format": "agent-skills"
},
@@ -2706,18 +3288,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -2726,18 +3310,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -2752,15 +3347,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -2769,15 +3378,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -2792,22 +3406,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -2827,9 +3442,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -2841,17 +3456,17 @@
"native_surface": "Agent Skills standard source tree",
"activation": {
"policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Keep optional directories as relative resources next to SKILL.md.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"SKILL.md",
@@ -2860,7 +3475,7 @@
},
"scripts": {
"strategy": "Scripts remain local optional resources and should advertise --help when executable.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -2924,18 +3539,20 @@
"capabilities": {
"network": {
"required": true,
- "script_count": 2,
+ "script_count": 3,
"scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"review_reason": "Outbound hosts must match security/network_policy.json."
},
"file_write": {
"required": true,
- "script_count": 47,
+ "script_count": 73,
"scripts": [
"scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
"scripts/build_confusion_matrix.py",
"scripts/build_skill_atlas.py",
"scripts/check_update.py",
@@ -2944,18 +3561,29 @@
"scripts/compile_skill.py",
"scripts/create_iteration_snapshot.py",
"scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
"scripts/export_skill_ir.py",
"scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
"scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
"scripts/probe_runtime_permissions.py",
"scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
"scripts/registry_audit.py",
"scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
"scripts/render_artifact_design_profile.py",
"scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
"scripts/render_description_drift_history.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/render_intent_confidence.py",
"scripts/render_intent_dialogue.py",
"scripts/render_iteration_directions.py",
@@ -2970,15 +3598,29 @@
"scripts/render_review_studio.py",
"scripts/render_review_viewer.py",
"scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
"scripts/render_skill_overview.py",
"scripts/render_social_preview.py",
"scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
"scripts/run_conformance_suite.py",
"scripts/run_description_optimization_suite.py",
"scripts/run_output_eval.py",
"scripts/run_output_execution.py",
"scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
"scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
"scripts/trust_check.py",
"scripts/upgrade_check.py",
"scripts/verify_package.py"
@@ -2987,15 +3629,20 @@
},
"subprocess": {
"required": true,
- "script_count": 7,
+ "script_count": 12,
"scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
"scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
"scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
"scripts/run_eval_suite.py",
"scripts/run_output_execution.py",
"scripts/sync_local_install.py",
"scripts/trust_check.py",
- "scripts/yao.py"
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
],
"review_reason": "Scripts spawn local commands and need operator review."
},
@@ -3010,22 +3657,23 @@
"source": "security/network_policy.json",
"covered_scripts": [
"scripts/check_update.py",
- "scripts/github_benchmark_scan.py"
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
],
"missing_scripts": [],
"mismatches": []
},
"help_smoke": {
"enabled": true,
- "checked_count": 59,
+ "checked_count": 86,
"failed_count": 0,
"failed_scripts": []
},
"trust_summary": {
"secret_findings": 0,
- "network_script_count": 2,
- "file_write_script_count": 47,
- "subprocess_script_count": 7,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
"interactive_script_count": 0,
"help_smoke_failed_count": 0
}
@@ -3043,9 +3691,9 @@
"subprocess"
],
"capability_counts": {
- "network": 2,
- "file_write": 47,
- "subprocess": 7,
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
"interactive": 0
},
"evidence": "reports/security_trust_report.json",
@@ -3057,17 +3705,17 @@
"native_surface": "Agent Skills standard source tree",
"activation": {
"policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
- "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"manual_activation_supported": true,
"automatic_activation_note": "Depends on the target client route/catalog implementation."
},
"resources": {
"strategy": "Keep optional directories as relative resources next to SKILL.md.",
"counts": {
- "references": 32,
- "scripts": 66,
- "assets": 2,
- "reports": 41
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
},
"generated_files": [
"SKILL.md",
@@ -3076,7 +3724,7 @@
},
"scripts": {
"strategy": "Scripts remain local optional resources and should advertise --help when executable.",
- "script_count": 66,
+ "script_count": 153,
"help_smoke_failed_count": 0
},
"permissions": {
@@ -3135,6 +3783,951 @@
"unsupported_features": [],
"warnings": [],
"failures": []
+ },
+ {
+ "schema_version": "1.0",
+ "target": "vscode",
+ "status": "pass",
+ "compiler": {
+ "name": "yao-skill-ir-compiler",
+ "schema_version": "1.0",
+ "source": "skill-ir",
+ "ir_source": "skill-ir/examples/yao-meta-skill.json",
+ "ir_schema_version": "2.0.0"
+ },
+ "source": {
+ "skill_dir": ".",
+ "canonical_metadata": "agents/interface.yaml",
+ "declared_targets": [
+ "openai",
+ "claude",
+ "generic",
+ "agent-skills-compatible",
+ "vscode"
+ ]
+ },
+ "compiled_contract": {
+ "name": "yao-meta-skill",
+ "title": "Yao Meta Skill",
+ "version": "1.1.0",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "job_to_be_done": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
+ "trigger": {
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "should_trigger": [
+ "Create a skill from this repeated workflow.",
+ "Turn this runbook into a reusable agent skill.",
+ "Convert this process note into a reusable skill package for the team.",
+ "Improve this skill description and add evals.",
+ "Refactor this prompt into a proper skill package.",
+ "Package this skill for team reuse.",
+ "Create a meta-skill for packaging internal workflows.",
+ "Standardize this recurring workflow as a shareable skill package with references."
+ ],
+ "should_not_trigger": [
+ "Explain what a workflow is.",
+ "Just explain what a skill is.",
+ "Summarize this random note.",
+ "Translate this README into Japanese.",
+ "Write a product headline for this landing page.",
+ "Draft a blog title for this article.",
+ "Rewrite this paragraph more clearly, but do not package anything.",
+ "Explain the difference between a runbook and a workflow.",
+ "one-off writing, translation, explanation, or brainstorming requests that do not need a reusable skill",
+ "general code review or debugging unless the user is packaging that workflow as a skill"
+ ],
+ "edge_cases": [
+ "Create a one-off prompt for this task.",
+ "Write a custom answer for this request without creating a skill package.",
+ "Help me brainstorm process ideas without building a skill.",
+ "Improve this README but do not turn it into a skill.",
+ "Make a checklist for this task, but not a reusable skill.",
+ "Create a reusable-looking checklist, but keep it as a plain note instead of a skill.",
+ "Review this process note and explain it, no packaging needed.",
+ "Turn this into a checklist and template, but stop short of making a full skill."
+ ]
+ },
+ "workflow": {
+ "steps": [
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
+ ],
+ "decision_points": [
+ "Route by frontmatter `description`.",
+ "Keep `SKILL.md` lean; put guidance in `references/`, logic in `scripts/`, and evidence in `reports/`.",
+ "Use the lightest reliable process.",
+ "`Scaffold`: exploratory/personal. `Production`: team reuse. `Library`: shared infrastructure. `Governed`: high-trust, policy-sensitive, or release-critical.",
+ "Rules: [Method](references/skill-engineering-method.md), [Operating Modes](references/operating-modes.md), [Resource Boundaries](references/resource-boundaries.md).",
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ],
+ "failure_modes": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Users start invoking the skill for adjacent one-off or explanation-only requests.",
+ "Outputs remain valid but become generic, cluttered, or weakly aligned with the user's domain.",
+ "Borrowed benchmark patterns no longer fit the local job or add ceremony without payoff.",
+ "Skill usage becomes team-critical while ownership, review cadence, or rollback evidence stays informal."
+ ]
+ },
+ "resources": {
+ "references": [
+ "references/artifact-design-doctrine.md",
+ "references/authoring-discipline.md",
+ "references/autonomous-adaptation.md",
+ "references/distribution-registry-method.md",
+ "references/eval-playbook.md",
+ "references/gate-selection.md",
+ "references/governance.md",
+ "references/human-review-template.md",
+ "references/intent-dialogue.md",
+ "references/iteration-philosophy.md",
+ "references/non-skill-decision-tree.md",
+ "references/operating-modes.md",
+ "references/output-eval-method.md",
+ "references/output-quality-risk.md",
+ "references/output-visual-quality.md",
+ "references/packaging-contracts.md",
+ "references/pattern-extraction-doctrine.md",
+ "references/platform-capability-matrix.md",
+ "references/prompt-engineering-doctrine.md",
+ "references/qa-ladder.md",
+ "references/reference-scan.md",
+ "references/regression-cause-taxonomy.md",
+ "references/resource-boundaries.md",
+ "references/review-studio-method.md",
+ "references/review-waiver-method.md",
+ "references/runtime-conformance-method.md",
+ "references/skill-archetypes.md",
+ "references/skill-atlas-method.md",
+ "references/skill-engineering-method.md",
+ "references/skill-ir-method.md",
+ "references/skillops-decision-policy.md",
+ "references/systems-thinking-doctrine.md",
+ "references/telemetry-drift-method.md",
+ "references/trust-security-method.md",
+ "references/user-memory-policy.md"
+ ],
+ "scripts": [
+ "scripts/adaptation_patch_safety.py",
+ "scripts/adaptation_report_contracts.py",
+ "scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
+ "scripts/build_confusion_matrix.py",
+ "scripts/build_skill_atlas.py",
+ "scripts/build_skill_atlas_layout.py",
+ "scripts/build_skill_atlas_opportunities.py",
+ "scripts/check_update.py",
+ "scripts/ci_test.py",
+ "scripts/collect_feedback.py",
+ "scripts/compile_skill.py",
+ "scripts/compile_skill_targets.py",
+ "scripts/context_sizer.py",
+ "scripts/create_iteration_snapshot.py",
+ "scripts/cross_packager.py",
+ "scripts/cross_packager_contracts.py",
+ "scripts/description_optimizer_reporting.py",
+ "scripts/diff_eval.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/evidence_consistency_artifact_roles.py",
+ "scripts/evidence_consistency_core.py",
+ "scripts/evidence_consistency_phase_queue.py",
+ "scripts/evidence_consistency_release.py",
+ "scripts/evidence_consistency_skill_os2_review.py",
+ "scripts/evidence_consistency_world_class.py",
+ "scripts/export_skill_ir.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/governance_check.py",
+ "scripts/html_rendering.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
+ "scripts/init_skill.py",
+ "scripts/judge_blind_eval.py",
+ "scripts/lint_skill.py",
+ "scripts/local_output_eval_runner.py",
+ "scripts/optimize_description.py",
+ "scripts/output_review_privacy.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
+ "scripts/probe_runtime_permissions.py",
+ "scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/provider_output_eval_runner.py",
+ "scripts/python_compat_check.py",
+ "scripts/reference_synthesis_markdown.py",
+ "scripts/registry_audit.py",
+ "scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
+ "scripts/render_artifact_design_profile.py",
+ "scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
+ "scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
+ "scripts/render_description_drift_history.py",
+ "scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
+ "scripts/render_intent_confidence.py",
+ "scripts/render_intent_dialogue.py",
+ "scripts/render_iteration_directions.py",
+ "scripts/render_iteration_ledger.py",
+ "scripts/render_output_risk_profile.py",
+ "scripts/render_portability_report.py",
+ "scripts/render_prompt_quality_profile.py",
+ "scripts/render_reference_scan.py",
+ "scripts/render_reference_synthesis.py",
+ "scripts/render_regression_history.py",
+ "scripts/render_review_annotations.py",
+ "scripts/render_review_studio.py",
+ "scripts/render_review_viewer.py",
+ "scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
+ "scripts/render_skill_overview.py",
+ "scripts/render_social_preview.py",
+ "scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
+ "scripts/resource_boundary_check.py",
+ "scripts/review_studio_action_evidence.py",
+ "scripts/review_studio_actions.py",
+ "scripts/review_studio_data.py",
+ "scripts/review_studio_formatting.py",
+ "scripts/review_studio_gate_contract.py",
+ "scripts/review_studio_gate_helpers.py",
+ "scripts/review_studio_gates.py",
+ "scripts/review_studio_layout.py",
+ "scripts/review_studio_output_review.py",
+ "scripts/review_studio_panels.py",
+ "scripts/review_studio_skillops.py",
+ "scripts/review_studio_waivers.py",
+ "scripts/review_studio_world_class.py",
+ "scripts/review_viewer_data.py",
+ "scripts/run_conformance_suite.py",
+ "scripts/run_description_optimization_suite.py",
+ "scripts/run_eval_suite.py",
+ "scripts/run_output_eval.py",
+ "scripts/run_output_execution.py",
+ "scripts/simulate_install.py",
+ "scripts/skill_ir_paths.py",
+ "scripts/skill_os2_coverage_markdown.py",
+ "scripts/skill_report_charts.py",
+ "scripts/skill_report_i18n.py",
+ "scripts/skill_report_layout.py",
+ "scripts/skill_report_metrics.py",
+ "scripts/skill_report_model.py",
+ "scripts/skill_report_sections.py",
+ "scripts/skill_report_sources.py",
+ "scripts/skill_report_world_class.py",
+ "scripts/skillops_opportunity.py",
+ "scripts/summarize_user_signals.py",
+ "scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
+ "scripts/trigger_eval.py",
+ "scripts/trust_check.py",
+ "scripts/trust_check_scripts.py",
+ "scripts/upgrade_check.py",
+ "scripts/validate_skill.py",
+ "scripts/verify_package.py",
+ "scripts/world_class_evidence_contract.py",
+ "scripts/world_class_human_evidence.py",
+ "scripts/world_class_native_permission_evidence.py",
+ "scripts/world_class_native_telemetry_evidence.py",
+ "scripts/world_class_operator_runbook_coordination.py",
+ "scripts/world_class_phase_queue.py",
+ "scripts/world_class_preflight_layout.py",
+ "scripts/world_class_preflight_markdown.py",
+ "scripts/world_class_provider_evidence.py",
+ "scripts/world_class_repair_checklist.py",
+ "scripts/world_class_source_checks.py",
+ "scripts/world_class_submission_kit_rendering.py",
+ "scripts/world_class_submission_matrix.py",
+ "scripts/yao.py",
+ "scripts/yao_cli_adaptation_commands.py",
+ "scripts/yao_cli_config.py",
+ "scripts/yao_cli_create_commands.py",
+ "scripts/yao_cli_distribution_commands.py",
+ "scripts/yao_cli_output_commands.py",
+ "scripts/yao_cli_parser.py",
+ "scripts/yao_cli_parser_evidence.py",
+ "scripts/yao_cli_parser_operations.py",
+ "scripts/yao_cli_report_commands.py",
+ "scripts/yao_cli_runtime.py",
+ "scripts/yao_cli_telemetry.py"
+ ],
+ "assets": [
+ "assets/review-studio.css",
+ "assets/review-viewer.css",
+ "assets/skill-overview.css",
+ "assets/skill-overview.js",
+ "templates/basic_skill.md.j2",
+ "templates/complex_skill.md.j2"
+ ],
+ "reports": [
+ "reports/benchmark_methodology.md",
+ "reports/intent-context.json",
+ "reports/intent-confidence.json",
+ "reports/intent-confidence.md",
+ "reports/reference-synthesis.json",
+ "reports/reference-synthesis.md",
+ "reports/output-risk-profile.json",
+ "reports/output-risk-profile.md",
+ "reports/artifact-design-profile.json",
+ "reports/artifact-design-profile.md",
+ "reports/prompt-quality-profile.json",
+ "reports/prompt-quality-profile.md",
+ "reports/system-model.json",
+ "reports/system-model.md",
+ "reports/iteration-directions.json",
+ "reports/iteration-directions.md",
+ "reports/skill-overview.json",
+ "reports/skill-overview.html",
+ "reports/output_quality_scorecard.json",
+ "reports/output_quality_scorecard.md",
+ "reports/output_execution_runs.json",
+ "reports/output_execution_runs.md",
+ "reports/output_blind_review_pack.json",
+ "reports/output_blind_review_pack.md",
+ "reports/output_blind_answer_key.json",
+ "reports/output_review_adjudication.json",
+ "reports/output_review_adjudication.md",
+ "reports/review_annotations.json",
+ "reports/review_annotations.md",
+ "reports/conformance_matrix.json",
+ "reports/conformance_matrix.md",
+ "reports/security_trust_report.json",
+ "reports/security_trust_report.md",
+ "reports/runtime_permission_probes.json",
+ "reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
+ "reports/skill_atlas.json",
+ "reports/skill_atlas.html",
+ "reports/skill-os-2-review.md",
+ "reports/portability_score.json",
+ "reports/portability_score.md",
+ "reports/governance_score.json"
+ ],
+ "counts": {
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
+ }
+ },
+ "eval_plan": {
+ "trigger": [
+ "evals/README.md",
+ "evals/adaptation/user_signals.example.jsonl",
+ "evals/adversarial/trigger_cases.json",
+ "evals/blind_holdout/trigger_cases.json",
+ "evals/confusion/route_cases.json",
+ "evals/dev/trigger_cases.json",
+ "evals/failure-cases.md",
+ "evals/history/2026-03-31-context-first-pack.json",
+ "evals/history/2026-03-31-family-suite-expansion.json",
+ "evals/history/2026-03-31-governance-quality-gates.json",
+ "evals/history/2026-03-31-governed-assets-and-history.json",
+ "evals/history/2026-04-01-route-scorecard-foundation.json",
+ "evals/history/README.md",
+ "evals/history/description_optimization/2026-03-31-description-optimization-suite.json",
+ "evals/history/description_optimization/2026-04-01-adversarial-calibration-and-family-drift.json",
+ "evals/history/description_optimization/2026-04-01-blind-holdout-and-drift-history.json",
+ "evals/history/description_optimization/2026-04-01-judge-backed-blind-eval.json",
+ "evals/history/description_optimization/README.md",
+ "evals/holdout/trigger_cases.json",
+ "evals/output/cases.jsonl",
+ "evals/output/fixtures/release-brief-source.md",
+ "evals/output/schema.json",
+ "evals/packaging_expectations.json",
+ "evals/promotion_policy.md",
+ "evals/semantic_config.json",
+ "evals/train/trigger_cases.json",
+ "evals/trigger_cases.json"
+ ],
+ "output": [
+ "evals/output/cases.jsonl",
+ "evals/output/fixtures/release-brief-source.md",
+ "evals/output/schema.json"
+ ],
+ "adversarial": [
+ "evals/adversarial/trigger_cases.json"
+ ],
+ "baseline": "without_skill",
+ "counts": {
+ "trigger": 27,
+ "output": 3,
+ "adversarial": 1,
+ "baseline": 1
+ }
+ },
+ "risk": {
+ "output_risk": "high",
+ "execution_risk": "medium",
+ "trust_boundary": "external"
+ },
+ "governance": {
+ "owner": "Yao Team",
+ "maturity": "governed",
+ "review_cadence": "quarterly",
+ "review_due": ""
+ },
+ "runtime": {
+ "activation": {
+ "mode": "manual",
+ "paths": []
+ },
+ "execution": {
+ "context": "inline",
+ "shell": "bash"
+ },
+ "trust": {
+ "source_tier": "local",
+ "remote_inline_execution": "forbid",
+ "remote_metadata_policy": "allow-metadata-only"
+ },
+ "adapter_targets": [
+ "openai",
+ "claude",
+ "generic",
+ "vscode"
+ ],
+ "canonical_format": "agent-skills"
+ },
+ "permissions": {
+ "schema_version": "1.0",
+ "source": "reports/security_trust_report.json",
+ "source_available": true,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "review_required": true,
+ "capabilities": {
+ "network": {
+ "required": true,
+ "script_count": 3,
+ "scripts": [
+ "scripts/check_update.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
+ ],
+ "review_reason": "Outbound hosts must match security/network_policy.json."
+ },
+ "file_write": {
+ "required": true,
+ "script_count": 73,
+ "scripts": [
+ "scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/build_confusion_matrix.py",
+ "scripts/build_skill_atlas.py",
+ "scripts/check_update.py",
+ "scripts/ci_test.py",
+ "scripts/collect_feedback.py",
+ "scripts/compile_skill.py",
+ "scripts/create_iteration_snapshot.py",
+ "scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/export_skill_ir.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
+ "scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
+ "scripts/probe_runtime_permissions.py",
+ "scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
+ "scripts/registry_audit.py",
+ "scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
+ "scripts/render_artifact_design_profile.py",
+ "scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
+ "scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
+ "scripts/render_description_drift_history.py",
+ "scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
+ "scripts/render_intent_confidence.py",
+ "scripts/render_intent_dialogue.py",
+ "scripts/render_iteration_directions.py",
+ "scripts/render_iteration_ledger.py",
+ "scripts/render_output_risk_profile.py",
+ "scripts/render_portability_report.py",
+ "scripts/render_prompt_quality_profile.py",
+ "scripts/render_reference_scan.py",
+ "scripts/render_reference_synthesis.py",
+ "scripts/render_regression_history.py",
+ "scripts/render_review_annotations.py",
+ "scripts/render_review_studio.py",
+ "scripts/render_review_viewer.py",
+ "scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
+ "scripts/render_skill_overview.py",
+ "scripts/render_social_preview.py",
+ "scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
+ "scripts/run_conformance_suite.py",
+ "scripts/run_description_optimization_suite.py",
+ "scripts/run_output_eval.py",
+ "scripts/run_output_execution.py",
+ "scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
+ "scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
+ "scripts/trust_check.py",
+ "scripts/upgrade_check.py",
+ "scripts/verify_package.py"
+ ],
+ "review_reason": "Scripts write local files or generated artifacts."
+ },
+ "subprocess": {
+ "required": true,
+ "script_count": 12,
+ "scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
+ "scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
+ "scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
+ "scripts/run_eval_suite.py",
+ "scripts/run_output_execution.py",
+ "scripts/sync_local_install.py",
+ "scripts/trust_check.py",
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
+ ],
+ "review_reason": "Scripts spawn local commands and need operator review."
+ },
+ "interactive": {
+ "required": false,
+ "script_count": 0,
+ "scripts": [],
+ "review_reason": ""
+ }
+ },
+ "network_policy": {
+ "source": "security/network_policy.json",
+ "covered_scripts": [
+ "scripts/check_update.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
+ ],
+ "missing_scripts": [],
+ "mismatches": []
+ },
+ "help_smoke": {
+ "enabled": true,
+ "checked_count": 86,
+ "failed_count": 0,
+ "failed_scripts": []
+ },
+ "trust_summary": {
+ "secret_findings": 0,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
+ "interactive_script_count": 0,
+ "help_smoke_failed_count": 0
+ }
+ },
+ "source_files_count": 6,
+ "target": "vscode",
+ "target_permission_contract": {
+ "schema_version": "1.0",
+ "target": "vscode",
+ "permission_model": "vscode-workspace-trust-plus-metadata",
+ "native_enforcement": false,
+ "representation": "targets/vscode/adapter.json::target_permission_contract and targets/vscode/README.md install notes",
+ "review_required": true,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "capability_counts": {
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
+ "interactive": 0
+ },
+ "evidence": "reports/security_trust_report.json",
+ "operator_note": "VS Code target relies on project or user skill installation plus VS Code workspace trust; Yao preserves permission metadata for reviewer and installer checks."
+ },
+ "target_native_contract": {
+ "schema_version": "1.0",
+ "target": "vscode",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "activation": {
+ "policy": "Use folder name plus SKILL.md name/description; keep description under platform limits.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "manual_activation_supported": true,
+ "automatic_activation_note": "Depends on the target client route/catalog implementation."
+ },
+ "resources": {
+ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.",
+ "counts": {
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
+ },
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ]
+ },
+ "scripts": {
+ "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.",
+ "script_count": 153,
+ "help_smoke_failed_count": 0
+ },
+ "permissions": {
+ "enforcement": "client-or-workspace-trust",
+ "native_enforcement": false,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "review_required": true
+ },
+ "review": {
+ "artifacts": [
+ "SKILL.md",
+ "agents/interface.yaml",
+ "reports/review-studio.html"
+ ],
+ "fallback_behavior": "If VS Code scope is not installed, use the neutral Agent Skills source.",
+ "unsupported_native_features": [
+ "automatic VS Code installation",
+ "VS Code installation scope is documented but not installed by this compiler"
+ ]
+ },
+ "install_scope": "VS Code user or project skills directory"
+ },
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "target_runtime": {
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "metadata_mapping": {
+ "name": "folder-name-and-SKILL.md::frontmatter.name",
+ "description": "SKILL.md::frontmatter.description",
+ "interface": "agents/interface.yaml",
+ "permissions": "targets/vscode/adapter.json::target_permission_contract",
+ "install_scope": "targets/vscode/README.md"
+ },
+ "preserved_semantics": [
+ "trigger",
+ "workflow",
+ "resources",
+ "eval-plan",
+ "risk",
+ "governance",
+ "runtime",
+ "trust",
+ "permissions"
+ ],
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope"
+ }
+ },
+ "permission_contract": {
+ "schema_version": "1.0",
+ "source": "reports/security_trust_report.json",
+ "source_available": true,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "review_required": true,
+ "capabilities": {
+ "network": {
+ "required": true,
+ "script_count": 3,
+ "scripts": [
+ "scripts/check_update.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
+ ],
+ "review_reason": "Outbound hosts must match security/network_policy.json."
+ },
+ "file_write": {
+ "required": true,
+ "script_count": 73,
+ "scripts": [
+ "scripts/adjudicate_output_review.py",
+ "scripts/apply_adaptation.py",
+ "scripts/build_confusion_matrix.py",
+ "scripts/build_skill_atlas.py",
+ "scripts/check_update.py",
+ "scripts/ci_test.py",
+ "scripts/collect_feedback.py",
+ "scripts/compile_skill.py",
+ "scripts/create_iteration_snapshot.py",
+ "scripts/cross_packager.py",
+ "scripts/emit_telemetry_event.py",
+ "scripts/export_skill_ir.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/import_output_review_decisions.py",
+ "scripts/import_telemetry_events.py",
+ "scripts/init_skill.py",
+ "scripts/prepare_output_review_kit.py",
+ "scripts/prepare_world_class_submission_kit.py",
+ "scripts/probe_runtime_permissions.py",
+ "scripts/promotion_checker.py",
+ "scripts/propose_adaptation.py",
+ "scripts/python_compat_check.py",
+ "scripts/registry_audit.py",
+ "scripts/render_adoption_drift_report.py",
+ "scripts/render_architecture_maintainability.py",
+ "scripts/render_artifact_design_profile.py",
+ "scripts/render_baseline_compare.py",
+ "scripts/render_benchmark_reproducibility.py",
+ "scripts/render_context_reports.py",
+ "scripts/render_daily_skillops_report.py",
+ "scripts/render_description_drift_history.py",
+ "scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
+ "scripts/render_intent_confidence.py",
+ "scripts/render_intent_dialogue.py",
+ "scripts/render_iteration_directions.py",
+ "scripts/render_iteration_ledger.py",
+ "scripts/render_output_risk_profile.py",
+ "scripts/render_portability_report.py",
+ "scripts/render_prompt_quality_profile.py",
+ "scripts/render_reference_scan.py",
+ "scripts/render_reference_synthesis.py",
+ "scripts/render_regression_history.py",
+ "scripts/render_review_annotations.py",
+ "scripts/render_review_studio.py",
+ "scripts/render_review_viewer.py",
+ "scripts/render_review_waivers.py",
+ "scripts/render_skill_interpretation.py",
+ "scripts/render_skill_os2_audit.py",
+ "scripts/render_skill_os2_coverage.py",
+ "scripts/render_skill_overview.py",
+ "scripts/render_social_preview.py",
+ "scripts/render_system_model.py",
+ "scripts/render_telemetry_hook_recipes.py",
+ "scripts/render_weekly_curator_report.py",
+ "scripts/render_world_class_claim_guard.py",
+ "scripts/render_world_class_evidence_intake.py",
+ "scripts/render_world_class_evidence_ledger.py",
+ "scripts/render_world_class_evidence_plan.py",
+ "scripts/render_world_class_operator_runbook.py",
+ "scripts/render_world_class_preflight.py",
+ "scripts/render_world_class_submission_review.py",
+ "scripts/run_conformance_suite.py",
+ "scripts/run_description_optimization_suite.py",
+ "scripts/run_output_eval.py",
+ "scripts/run_output_execution.py",
+ "scripts/simulate_install.py",
+ "scripts/summarize_user_signals.py",
+ "scripts/sync_local_install.py",
+ "scripts/telemetry_native_host.py",
+ "scripts/trust_check.py",
+ "scripts/upgrade_check.py",
+ "scripts/verify_package.py"
+ ],
+ "review_reason": "Scripts write local files or generated artifacts."
+ },
+ "subprocess": {
+ "required": true,
+ "script_count": 12,
+ "scripts": [
+ "scripts/apply_adaptation.py",
+ "scripts/benchmark_release_lock.py",
+ "scripts/ci_test.py",
+ "scripts/render_benchmark_reproducibility.py",
+ "scripts/render_eval_dashboard.py",
+ "scripts/render_evidence_consistency.py",
+ "scripts/run_eval_suite.py",
+ "scripts/run_output_execution.py",
+ "scripts/sync_local_install.py",
+ "scripts/trust_check.py",
+ "scripts/yao.py",
+ "scripts/yao_cli_runtime.py"
+ ],
+ "review_reason": "Scripts spawn local commands and need operator review."
+ },
+ "interactive": {
+ "required": false,
+ "script_count": 0,
+ "scripts": [],
+ "review_reason": ""
+ }
+ },
+ "network_policy": {
+ "source": "security/network_policy.json",
+ "covered_scripts": [
+ "scripts/check_update.py",
+ "scripts/github_benchmark_scan.py",
+ "scripts/provider_output_eval_runner.py"
+ ],
+ "missing_scripts": [],
+ "mismatches": []
+ },
+ "help_smoke": {
+ "enabled": true,
+ "checked_count": 86,
+ "failed_count": 0,
+ "failed_scripts": []
+ },
+ "trust_summary": {
+ "secret_findings": 0,
+ "network_script_count": 3,
+ "file_write_script_count": 73,
+ "subprocess_script_count": 12,
+ "interactive_script_count": 0,
+ "help_smoke_failed_count": 0
+ }
+ },
+ "target_permission_contract": {
+ "schema_version": "1.0",
+ "target": "vscode",
+ "permission_model": "vscode-workspace-trust-plus-metadata",
+ "native_enforcement": false,
+ "representation": "targets/vscode/adapter.json::target_permission_contract and targets/vscode/README.md install notes",
+ "review_required": true,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "capability_counts": {
+ "network": 3,
+ "file_write": 73,
+ "subprocess": 12,
+ "interactive": 0
+ },
+ "evidence": "reports/security_trust_report.json",
+ "operator_note": "VS Code target relies on project or user skill installation plus VS Code workspace trust; Yao preserves permission metadata for reviewer and installer checks."
+ },
+ "target_native_contract": {
+ "schema_version": "1.0",
+ "target": "vscode",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "activation": {
+ "policy": "Use folder name plus SKILL.md name/description; keep description under platform limits.",
+ "trigger_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "manual_activation_supported": true,
+ "automatic_activation_note": "Depends on the target client route/catalog implementation."
+ },
+ "resources": {
+ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.",
+ "counts": {
+ "references": 35,
+ "scripts": 153,
+ "assets": 6,
+ "reports": 43
+ },
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ]
+ },
+ "scripts": {
+ "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.",
+ "script_count": 153,
+ "help_smoke_failed_count": 0
+ },
+ "permissions": {
+ "enforcement": "client-or-workspace-trust",
+ "native_enforcement": false,
+ "declared_capabilities": [
+ "network",
+ "file_write",
+ "subprocess"
+ ],
+ "review_required": true
+ },
+ "review": {
+ "artifacts": [
+ "SKILL.md",
+ "agents/interface.yaml",
+ "reports/review-studio.html"
+ ],
+ "fallback_behavior": "If VS Code scope is not installed, use the neutral Agent Skills source.",
+ "unsupported_native_features": [
+ "automatic VS Code installation",
+ "VS Code installation scope is documented but not installed by this compiler"
+ ]
+ },
+ "install_scope": "VS Code user or project skills directory"
+ },
+ "target_transform": {
+ "target": "vscode",
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "metadata_mapping": {
+ "name": "folder-name-and-SKILL.md::frontmatter.name",
+ "description": "SKILL.md::frontmatter.description",
+ "interface": "agents/interface.yaml",
+ "permissions": "targets/vscode/adapter.json::target_permission_contract",
+ "install_scope": "targets/vscode/README.md"
+ },
+ "preserved_semantics": [
+ "trigger",
+ "workflow",
+ "resources",
+ "eval-plan",
+ "risk",
+ "governance",
+ "runtime",
+ "trust",
+ "permissions"
+ ],
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "permission_representation": "targets/vscode/adapter.json::target_permission_contract and targets/vscode/README.md install notes",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "activation_policy": "Use folder name plus SKILL.md name/description; keep description under platform limits.",
+ "resource_strategy": "Install as project or user scoped skill source, preserving relative references and scripts.",
+ "script_strategy": "Scripts require workspace trust and operator/client approval outside this compiler.",
+ "permission_enforcement": "client-or-workspace-trust"
+ },
+ "unsupported_features": [
+ "VS Code installation scope is documented but not installed by this compiler"
+ ],
+ "warnings": [],
+ "failures": []
}
],
"failures": [],
diff --git a/reports/compiled_targets.md b/reports/compiled_targets.md
index ea2ddcb5..f6a11f9b 100644
--- a/reports/compiled_targets.md
+++ b/reports/compiled_targets.md
@@ -1,8 +1,8 @@
# Compiled Targets
- OK: `True`
-- Targets: `4`
-- Pass: `4`
+- Targets: `5`
+- Pass: `5`
- Warn: `0`
- Block: `0`
@@ -14,6 +14,7 @@
| `claude` | `pass` | Claude-compatible neutral source folder with adapter notes | `neutral-source-plus-adapter` | `network, file_write, subprocess` | `neutral-source-plus-adapter` | targets/claude/adapter.json, targets/claude/README.md |
| `generic` | `pass` | Agent Skills compatible neutral package | `agent-skills-compatible` | `network, file_write, subprocess` | `neutral-source` | targets/generic/adapter.json |
| `agent-skills-compatible` | `pass` | Agent Skills standard source tree | `neutral-agent-skills-source` | `network, file_write, subprocess` | `neutral-source` | SKILL.md, agents/interface.yaml |
+| `vscode` | `pass` | VS Code/Copilot Agent Skills project or user scope | `vscode-agent-skills-adapter` | `network, file_write, subprocess` | `agent-skills-source-with-vscode-notes` | targets/vscode/adapter.json, targets/vscode/README.md |
## Native Behavior Contracts
@@ -53,6 +54,15 @@
- Permission enforcement: `consumer-enforced-or-metadata-only`; native enforcement `False`
- Review artifacts: SKILL.md, agents/interface.yaml, reports/review-studio.html
+### vscode
+
+- Native surface: VS Code/Copilot Agent Skills project or user scope
+- Activation: Use folder name plus SKILL.md name/description; keep description under platform limits.
+- Resources: Install as project or user scoped skill source, preserving relative references and scripts.
+- Scripts: Scripts require workspace trust and operator/client approval outside this compiler.
+- Permission enforcement: `client-or-workspace-trust`; native enforcement `False`
+- Review artifacts: SKILL.md, agents/interface.yaml, reports/review-studio.html
+
## Failures
diff --git a/reports/conformance_matrix.json b/reports/conformance_matrix.json
index 20e9bcea..722a8ef7 100644
--- a/reports/conformance_matrix.json
+++ b/reports/conformance_matrix.json
@@ -32,6 +32,7 @@
"Skill IR description matches frontmatter",
"references resource resolves: references/artifact-design-doctrine.md",
"references resource resolves: references/authoring-discipline.md",
+ "references resource resolves: references/autonomous-adaptation.md",
"references resource resolves: references/distribution-registry-method.md",
"references resource resolves: references/eval-playbook.md",
"references resource resolves: references/gate-selection.md",
@@ -40,8 +41,7 @@
"references resource resolves: references/intent-dialogue.md",
"references resource resolves: references/iteration-philosophy.md",
"references resource resolves: references/non-skill-decision-tree.md",
- "references resource resolves: references/operating-modes.md",
- "references resource resolves: references/output-eval-method.md"
+ "references resource resolves: references/operating-modes.md"
],
"failures": [],
"warnings": []
@@ -76,6 +76,7 @@
"Skill IR description matches frontmatter",
"references resource resolves: references/artifact-design-doctrine.md",
"references resource resolves: references/authoring-discipline.md",
+ "references resource resolves: references/autonomous-adaptation.md",
"references resource resolves: references/distribution-registry-method.md",
"references resource resolves: references/eval-playbook.md",
"references resource resolves: references/gate-selection.md",
@@ -84,8 +85,7 @@
"references resource resolves: references/intent-dialogue.md",
"references resource resolves: references/iteration-philosophy.md",
"references resource resolves: references/non-skill-decision-tree.md",
- "references resource resolves: references/operating-modes.md",
- "references resource resolves: references/output-eval-method.md"
+ "references resource resolves: references/operating-modes.md"
],
"failures": [],
"warnings": []
@@ -99,7 +99,7 @@
"frontmatter description exists",
"description length <= 1024",
"name is runtime-safe",
- "directory name matches skill name",
+ "package identity derives from skill name",
"manifest.name exists",
"manifest.version exists",
"manifest.owner exists",
@@ -120,6 +120,7 @@
"Skill IR description matches frontmatter",
"references resource resolves: references/artifact-design-doctrine.md",
"references resource resolves: references/authoring-discipline.md",
+ "references resource resolves: references/autonomous-adaptation.md",
"references resource resolves: references/distribution-registry-method.md",
"references resource resolves: references/eval-playbook.md",
"references resource resolves: references/gate-selection.md",
@@ -128,8 +129,7 @@
"references resource resolves: references/intent-dialogue.md",
"references resource resolves: references/iteration-philosophy.md",
"references resource resolves: references/non-skill-decision-tree.md",
- "references resource resolves: references/operating-modes.md",
- "references resource resolves: references/output-eval-method.md"
+ "references resource resolves: references/operating-modes.md"
],
"failures": [],
"warnings": [
@@ -145,14 +145,14 @@
"frontmatter description exists",
"description length <= 1024",
"name is runtime-safe",
- "directory name matches skill name",
+ "package identity derives from skill name",
"manifest.name exists",
"manifest.version exists",
"manifest.owner exists",
"manifest.status exists",
"manifest.maturity_tier exists",
"manifest.review_cadence exists",
- "manifest declares vscode via agent-skills-compatible",
+ "manifest declares vscode",
"interface.display_name exists",
"interface.short_description exists",
"interface.default_prompt exists",
@@ -166,6 +166,7 @@
"Skill IR description matches frontmatter",
"references resource resolves: references/artifact-design-doctrine.md",
"references resource resolves: references/authoring-discipline.md",
+ "references resource resolves: references/autonomous-adaptation.md",
"references resource resolves: references/distribution-registry-method.md",
"references resource resolves: references/eval-playbook.md",
"references resource resolves: references/gate-selection.md",
@@ -174,8 +175,7 @@
"references resource resolves: references/intent-dialogue.md",
"references resource resolves: references/iteration-philosophy.md",
"references resource resolves: references/non-skill-decision-tree.md",
- "references resource resolves: references/operating-modes.md",
- "references resource resolves: references/output-eval-method.md"
+ "references resource resolves: references/operating-modes.md"
],
"failures": [],
"warnings": [
@@ -212,6 +212,7 @@
"Skill IR description matches frontmatter",
"references resource resolves: references/artifact-design-doctrine.md",
"references resource resolves: references/authoring-discipline.md",
+ "references resource resolves: references/autonomous-adaptation.md",
"references resource resolves: references/distribution-registry-method.md",
"references resource resolves: references/eval-playbook.md",
"references resource resolves: references/gate-selection.md",
@@ -220,8 +221,7 @@
"references resource resolves: references/intent-dialogue.md",
"references resource resolves: references/iteration-philosophy.md",
"references resource resolves: references/non-skill-decision-tree.md",
- "references resource resolves: references/operating-modes.md",
- "references resource resolves: references/output-eval-method.md"
+ "references resource resolves: references/operating-modes.md"
],
"failures": [],
"warnings": []
diff --git a/reports/context_budget.json b/reports/context_budget.json
index 49e96a06..f7f9638e 100644
--- a/reports/context_budget.json
+++ b/reports/context_budget.json
@@ -5,13 +5,70 @@
"stats": {
"context_budget_tier": "production",
"context_budget_limit": 1000,
- "skill_body_tokens": 811,
- "other_text_tokens": 869504,
- "estimated_initial_load_tokens": 987,
- "estimated_total_text_tokens": 870315,
- "relevant_file_count": 327,
+ "skill_body_tokens": 805,
+ "other_text_tokens": 1156781,
+ "estimated_initial_load_tokens": 998,
+ "estimated_total_text_tokens": 1157586,
+ "deferred_resource_tokens": 525864,
+ "deferred_resource_warn_threshold": 120000,
+ "deferred_resource_dirs": [
+ {
+ "path": "scripts",
+ "estimated_tokens": 464561,
+ "file_count": 153
+ },
+ {
+ "path": "references",
+ "estimated_tokens": 28292,
+ "file_count": 35
+ },
+ {
+ "path": "evals",
+ "estimated_tokens": 21835,
+ "file_count": 27
+ },
+ {
+ "path": "assets",
+ "estimated_tokens": 11176,
+ "file_count": 4
+ }
+ ],
+ "large_deferred_resource_dirs": [
+ {
+ "path": "scripts",
+ "estimated_tokens": 464561,
+ "file_count": 153
+ }
+ ],
+ "deferred_resource_governance": {
+ "status": "governed",
+ "large_dir_count": 1,
+ "governed_large_dir_count": 1,
+ "directories": [
+ {
+ "status": "governed",
+ "evidence": [
+ "reports/security_trust_report.json",
+ "reports/architecture_maintainability.json",
+ "reports/python_compatibility.json"
+ ],
+ "reasons": [
+ "trust report covers scripts",
+ "architecture report has no script hotspots or blockers",
+ "Python compatibility report has no issues"
+ ],
+ "missing": [],
+ "path": "scripts",
+ "estimated_tokens": 464561,
+ "file_count": 153,
+ "rationale": "Script resources are deterministic deferred tools, not initial-load prompt context."
+ }
+ ],
+ "summary": "Large deferred resources are indexed and backed by evidence."
+ },
+ "relevant_file_count": 712,
"unused_resource_dirs": [],
- "quality_signal_points": 130,
- "quality_density": 131.7
+ "quality_signal_points": 140,
+ "quality_density": 140.3
}
}
diff --git a/reports/context_budget.md b/reports/context_budget.md
index 9d678e58..46bab4f9 100644
--- a/reports/context_budget.md
+++ b/reports/context_budget.md
@@ -1,9 +1,9 @@
# Context Budget Summary
-| Target | Path | Tier | Limit | Initial | SKILL | Quality Density | Unused Dirs | Status |
-| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
-| root | `.` | `production` | 1000 | 987 | 811 | 131.7 | - | ok |
-| complex-release-orchestrator | `examples/complex-release-orchestrator/generated-skill` | `production` | 1000 | 790 | 718 | 164.6 | - | ok |
-| governed-incident-command | `examples/governed-incident-command/generated-skill` | `production` | 1000 | 760 | 658 | 171.1 | - | ok |
+| Target | Path | Tier | Limit | Initial | SKILL | Deferred | Resource Governance | Large Deferred Dirs | Quality Density | Unused Dirs | Status |
+| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | ---: | --- | --- |
+| root | `.` | `production` | 1000 | 998 | 805 | 525864 | `governed` | scripts:464561 | 140.3 | - | ok |
+| complex-release-orchestrator | `examples/complex-release-orchestrator/generated-skill` | `production` | 1000 | 790 | 718 | 1657 | `not-required` | - | 164.6 | - | ok |
+| governed-incident-command | `examples/governed-incident-command/generated-skill` | `production` | 1000 | 760 | 658 | 1030 | `not-required` | - | 171.1 | - | ok |
Per-target JSON reports are written beside each target and under `reports/`.
diff --git a/reports/context_budget_summary.json b/reports/context_budget_summary.json
index ac17bd28..3a08c3b6 100644
--- a/reports/context_budget_summary.json
+++ b/reports/context_budget_summary.json
@@ -6,9 +6,43 @@
"path": ".",
"budget_tier": "production",
"budget_limit": 1000,
- "initial_tokens": 987,
- "skill_body_tokens": 811,
- "quality_density": 131.7,
+ "initial_tokens": 998,
+ "skill_body_tokens": 805,
+ "deferred_resource_tokens": 525864,
+ "large_deferred_resource_dirs": [
+ {
+ "path": "scripts",
+ "estimated_tokens": 464561,
+ "file_count": 153
+ }
+ ],
+ "deferred_resource_governance": {
+ "status": "governed",
+ "large_dir_count": 1,
+ "governed_large_dir_count": 1,
+ "directories": [
+ {
+ "status": "governed",
+ "evidence": [
+ "reports/security_trust_report.json",
+ "reports/architecture_maintainability.json",
+ "reports/python_compatibility.json"
+ ],
+ "reasons": [
+ "trust report covers scripts",
+ "architecture report has no script hotspots or blockers",
+ "Python compatibility report has no issues"
+ ],
+ "missing": [],
+ "path": "scripts",
+ "estimated_tokens": 464561,
+ "file_count": 153,
+ "rationale": "Script resources are deterministic deferred tools, not initial-load prompt context."
+ }
+ ],
+ "summary": "Large deferred resources are indexed and backed by evidence."
+ },
+ "quality_density": 140.3,
"unused_resource_dirs": [],
"ok": true,
"warnings": []
@@ -20,6 +54,15 @@
"budget_limit": 1000,
"initial_tokens": 790,
"skill_body_tokens": 718,
+ "deferred_resource_tokens": 1657,
+ "large_deferred_resource_dirs": [],
+ "deferred_resource_governance": {
+ "status": "not-required",
+ "large_dir_count": 0,
+ "governed_large_dir_count": 0,
+ "directories": [],
+ "summary": "No large deferred resource directory exceeds the per-dir threshold."
+ },
"quality_density": 164.6,
"unused_resource_dirs": [],
"ok": true,
@@ -32,6 +75,15 @@
"budget_limit": 1000,
"initial_tokens": 760,
"skill_body_tokens": 658,
+ "deferred_resource_tokens": 1030,
+ "large_deferred_resource_dirs": [],
+ "deferred_resource_governance": {
+ "status": "not-required",
+ "large_dir_count": 0,
+ "governed_large_dir_count": 0,
+ "directories": [],
+ "summary": "No large deferred resource directory exceeds the per-dir threshold."
+ },
"quality_density": 171.1,
"unused_resource_dirs": [],
"ok": true,
diff --git a/reports/description_optimization.json b/reports/description_optimization.json
index dc4d7662..08f61621 100644
--- a/reports/description_optimization.json
+++ b/reports/description_optimization.json
@@ -1,11 +1,11 @@
{
- "current_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "current_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"current_candidate": {
"id": "current",
"label": "Current",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"strategy": "current",
- "estimated_tokens": 65,
+ "estimated_tokens": 53,
"dev": {
"false_positives": 0,
"false_negatives": 0,
@@ -21,7 +21,7 @@
-1.0,
-1.0,
-1.0,
- 65
+ 53
],
"holdout": {
"false_positives": 0,
@@ -2148,9 +2148,9 @@
"winner": {
"id": "current",
"label": "Current",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"strategy": "current",
- "estimated_tokens": 65,
+ "estimated_tokens": 53,
"dev": {
"false_positives": 0,
"false_negatives": 0,
@@ -2166,7 +2166,7 @@
-1.0,
-1.0,
-1.0,
- 65
+ 53
],
"holdout": {
"false_positives": 0,
@@ -2321,18 +2321,16 @@
{
"prompt": "Convert this operations checklist into a reusable skill.",
"family": "workflow_to_skill",
- "score": 0.415,
+ "score": 0.673,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2342,9 +2340,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.5,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -2364,33 +2362,32 @@
{
"prompt": "Add trigger evals to this skill before sharing it with the team.",
"family": "iterate_existing_skill",
- "score": 0.394,
+ "score": 0.407,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.327,
- "support_score": 0,
- "lexical_support": 0.417,
+ "semantic_coverage": 0.333,
+ "support_score": 0.16,
+ "lexical_support": 0.25,
"negative_penalty": 0,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"eval_optimize": [
@@ -2415,32 +2412,31 @@
{
"prompt": "We have a messy release runbook, export process, and a prompt history; turn all of that into one reusable skill package with evals and packaging checks.",
"family": "complex_multi_asset",
- "score": 0.843,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.375,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -2471,18 +2467,16 @@
{
"prompt": "Turn these workflow fragments, transcripts, and prompts into one production-ready skill with scripts and references.",
"family": "complex_multi_asset",
- "score": 0.422,
+ "score": 0.7,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2494,7 +2488,7 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0.06,
"lexical_support": 0.4,
"negative_penalty": 0,
@@ -2520,31 +2514,30 @@
{
"prompt": "Tighten the trigger boundary on this existing skill and prepare it for team distribution.",
"family": "iterate_existing_skill",
- "score": 0.557,
+ "score": 0.742,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"reuse_package",
"team_operationalize"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.509,
- "support_score": 0,
+ "semantic_coverage": 0.667,
+ "support_score": 0.16,
"lexical_support": 0.357,
"negative_penalty": 0,
"coverage_boost": 0.06,
@@ -2571,18 +2564,16 @@
{
"prompt": "Turn the SOP, transcript, and cleanup notes into a maintained skill for the team library.",
"family": "paraphrase_trigger",
- "score": 0.474,
+ "score": 0.763,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2593,9 +2584,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.4,
+ "semantic_coverage": 0.733,
"support_score": 0,
- "lexical_support": 0.571,
+ "lexical_support": 0.357,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -2619,32 +2610,31 @@
{
"prompt": "Codify this workflow into a shared capability package and validate the route boundary before distribution.",
"family": "paraphrase_trigger",
- "score": 0.868,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.855,
- "support_score": 0,
+ "semantic_coverage": 1.0,
+ "support_score": 0.34,
"lexical_support": 0.267,
"negative_penalty": 0,
"coverage_boost": 0.06,
@@ -2677,35 +2667,33 @@
{
"prompt": "After the long handoff notes below, build a skill package for team reuse. The materials include a transcript, rough notes, and an onboarding flow, and the result should include references, scripts, and validation.",
"family": "long_context_trigger",
- "score": 0.889,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
"extra_positive_concepts": [
+ "build_skill",
"multi_asset"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.855,
- "support_score": 0.06,
- "lexical_support": 0.346,
+ "semantic_coverage": 1.0,
+ "support_score": 0.4,
+ "lexical_support": 0.231,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -2752,11 +2740,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2769,9 +2755,9 @@
"exclusive_negative_concepts": [
"brainstorm_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.286,
+ "lexical_support": 0.143,
"negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -2798,11 +2784,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -2839,11 +2823,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2860,9 +2842,9 @@
"brainstorm_only",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0.06,
- "lexical_support": 0.2,
+ "lexical_support": 0.133,
"negative_penalty": 0.66,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -2896,11 +2878,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -2913,7 +2893,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.25,
+ "lexical_support": 0.167,
"negative_penalty": 0.35,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -2937,11 +2917,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -2954,9 +2932,9 @@
"exclusive_negative_concepts": [
"summarize_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.111,
+ "lexical_support": 0.222,
"negative_penalty": 0.3,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -2983,11 +2961,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3002,9 +2978,9 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.174,
+ "lexical_support": 0.087,
"negative_penalty": 0.68,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -3029,37 +3005,36 @@
{
"prompt": "I pasted a very long process note. For now I only need a recap and a short summary, not a skill package.",
"family": "long_context_summary_only",
- "score": 0.256,
+ "score": 0.306,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"summarize_only"
],
"exclusive_negative_concepts": [
"summarize_only"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
- "lexical_support": 0.333,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
+ "lexical_support": 0.222,
"negative_penalty": 0.3,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -3094,11 +3069,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3114,7 +3087,7 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.2,
"negative_penalty": 0.68,
@@ -3149,11 +3122,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3166,9 +3137,9 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.357,
+ "lexical_support": 0.143,
"negative_penalty": 0.42,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -3195,18 +3166,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -3214,11 +3184,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.5,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.3,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -3249,18 +3219,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "iterate_existing_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [
"future_outline",
"no_build_directive"
@@ -3269,11 +3238,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.327,
- "support_score": 0,
+ "semantic_coverage": 0.333,
+ "support_score": 0.16,
"lexical_support": 0.267,
"negative_penalty": 0.64,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"iterate_existing_skill": [
@@ -3304,18 +3273,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -3323,11 +3291,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.5,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.286,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -3359,18 +3327,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"future_outline",
@@ -3380,11 +3347,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.238,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.19,
"negative_penalty": 0.82,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -3423,18 +3390,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"future_outline",
"no_build_directive"
@@ -3443,11 +3409,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
"lexical_support": 0.263,
"negative_penalty": 0.64,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -3588,18 +3554,16 @@
{
"prompt": "Build a reusable skill from this long internal process note, and make sure it has references, scripts, and a trigger description.",
"family": "complex_multi_asset",
- "score": 0.468,
+ "score": 0.767,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3612,9 +3576,9 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.4,
+ "semantic_coverage": 0.733,
"support_score": 0.06,
- "lexical_support": 0.316,
+ "lexical_support": 0.211,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -3641,33 +3605,32 @@
{
"prompt": "Take this existing skill draft, tighten the trigger boundary, add near-neighbor evals, and package it for distribution.",
"family": "iterate_existing_skill",
- "score": 0.727,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.691,
- "support_score": 0,
- "lexical_support": 0.389,
+ "semantic_coverage": 1.0,
+ "support_score": 0.16,
+ "lexical_support": 0.333,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -3698,18 +3661,16 @@
{
"prompt": "Turn this recurring onboarding process into a reusable skill library entry with validation notes.",
"family": "workflow_to_skill",
- "score": 0.568,
+ "score": 0.936,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3720,9 +3681,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.509,
+ "semantic_coverage": 0.933,
"support_score": 0,
- "lexical_support": 0.5,
+ "lexical_support": 0.214,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -3745,34 +3706,32 @@
{
"prompt": "Codify this quarterly release routine into a reusable skill package with scripts, references, and packaging checks.",
"family": "paraphrase_trigger",
- "score": 0.858,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
"extra_positive_concepts": [
+ "build_skill",
"multi_asset"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0.06,
- "lexical_support": 0.375,
+ "semantic_coverage": 0.933,
+ "support_score": 0.4,
+ "lexical_support": 0.312,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -3803,32 +3762,31 @@
{
"prompt": "Below is a noisy transcript plus release notes and prompt history. Ignore the chatter and turn this playbook into an agent capability package for the team library.",
"family": "long_context_trigger",
- "score": 0.744,
+ "score": 0.839,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.709,
- "support_score": 0,
- "lexical_support": 0.4,
+ "semantic_coverage": 0.733,
+ "support_score": 0.34,
+ "lexical_support": 0.24,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -3865,11 +3823,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3885,9 +3841,9 @@
"no_build_directive",
"summarize_only"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.286,
+ "lexical_support": 0.143,
"negative_penalty": 0.72,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -3921,11 +3877,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3941,7 +3895,7 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.091,
"negative_penalty": 0.68,
@@ -3977,11 +3931,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -3995,7 +3947,7 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
"lexical_support": 0.222,
"negative_penalty": 0.6,
@@ -4029,19 +3981,18 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"no_build_directive",
"translate_only"
@@ -4050,11 +4001,11 @@
"no_build_directive",
"translate_only"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
- "lexical_support": 0.25,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
+ "lexical_support": 0.188,
"negative_penalty": 0.77,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -4092,17 +4043,15 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
- "matched_desired_concepts": [
+ "matched_desired_concepts": [],
+ "extra_positive_concepts": [
"build_skill"
],
- "extra_positive_concepts": [],
"matched_negative_concepts": [
"brainstorm_only",
"future_outline"
@@ -4111,9 +4060,9 @@
"brainstorm_only",
"future_outline"
],
- "semantic_coverage": 0.309,
- "support_score": 0,
- "lexical_support": 0.308,
+ "semantic_coverage": 0.0,
+ "support_score": 0.34,
+ "lexical_support": 0.077,
"negative_penalty": 0.46,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -4143,11 +4092,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -4162,7 +4109,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.357,
+ "lexical_support": 0.214,
"negative_penalty": 0.64,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -4190,18 +4137,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -4209,11 +4155,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.615,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.308,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -4239,26 +4185,25 @@
{
"prompt": "From the long workflow below, create a template only and a checklist for discussion only. No skill package.",
"family": "long_context_near_neighbor",
- "score": 0.0,
+ "score": 0.023,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -4266,11 +4211,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
"lexical_support": 0.438,
"negative_penalty": 0.6,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -4443,18 +4388,16 @@
{
"prompt": "Convert this operations checklist into a reusable skill.",
"family": "workflow_to_skill",
- "score": 0.415,
+ "score": 0.673,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -4464,9 +4407,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.5,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -4486,33 +4429,32 @@
{
"prompt": "Add trigger evals to this skill before sharing it with the team.",
"family": "iterate_existing_skill",
- "score": 0.394,
+ "score": 0.407,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.327,
- "support_score": 0,
- "lexical_support": 0.417,
+ "semantic_coverage": 0.333,
+ "support_score": 0.16,
+ "lexical_support": 0.25,
"negative_penalty": 0,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"eval_optimize": [
@@ -4537,32 +4479,31 @@
{
"prompt": "We have a messy release runbook, export process, and a prompt history; turn all of that into one reusable skill package with evals and packaging checks.",
"family": "complex_multi_asset",
- "score": 0.843,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.375,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -4593,18 +4534,16 @@
{
"prompt": "Turn these workflow fragments, transcripts, and prompts into one production-ready skill with scripts and references.",
"family": "complex_multi_asset",
- "score": 0.422,
+ "score": 0.7,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -4616,7 +4555,7 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0.06,
"lexical_support": 0.4,
"negative_penalty": 0,
@@ -4642,31 +4581,30 @@
{
"prompt": "Tighten the trigger boundary on this existing skill and prepare it for team distribution.",
"family": "iterate_existing_skill",
- "score": 0.557,
+ "score": 0.742,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"reuse_package",
"team_operationalize"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.509,
- "support_score": 0,
+ "semantic_coverage": 0.667,
+ "support_score": 0.16,
"lexical_support": 0.357,
"negative_penalty": 0,
"coverage_boost": 0.06,
@@ -4693,18 +4631,16 @@
{
"prompt": "Turn the SOP, transcript, and cleanup notes into a maintained skill for the team library.",
"family": "paraphrase_trigger",
- "score": 0.474,
+ "score": 0.763,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -4715,9 +4651,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.4,
+ "semantic_coverage": 0.733,
"support_score": 0,
- "lexical_support": 0.571,
+ "lexical_support": 0.357,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -4741,32 +4677,31 @@
{
"prompt": "Codify this workflow into a shared capability package and validate the route boundary before distribution.",
"family": "paraphrase_trigger",
- "score": 0.868,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.855,
- "support_score": 0,
+ "semantic_coverage": 1.0,
+ "support_score": 0.34,
"lexical_support": 0.267,
"negative_penalty": 0,
"coverage_boost": 0.06,
@@ -4799,35 +4734,33 @@
{
"prompt": "After the long handoff notes below, build a skill package for team reuse. The materials include a transcript, rough notes, and an onboarding flow, and the result should include references, scripts, and validation.",
"family": "long_context_trigger",
- "score": 0.889,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
"extra_positive_concepts": [
+ "build_skill",
"multi_asset"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.855,
- "support_score": 0.06,
- "lexical_support": 0.346,
+ "semantic_coverage": 1.0,
+ "support_score": 0.4,
+ "lexical_support": 0.231,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -4874,11 +4807,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -4891,9 +4822,9 @@
"exclusive_negative_concepts": [
"brainstorm_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.286,
+ "lexical_support": 0.143,
"negative_penalty": 0.24,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -4920,11 +4851,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -4961,11 +4890,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -4982,9 +4909,9 @@
"brainstorm_only",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0.06,
- "lexical_support": 0.2,
+ "lexical_support": 0.133,
"negative_penalty": 0.66,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -5018,11 +4945,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -5035,7 +4960,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.25,
+ "lexical_support": 0.167,
"negative_penalty": 0.35,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -5059,11 +4984,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5076,9 +4999,9 @@
"exclusive_negative_concepts": [
"summarize_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.111,
+ "lexical_support": 0.222,
"negative_penalty": 0.3,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -5105,11 +5028,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5124,9 +5045,9 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.174,
+ "lexical_support": 0.087,
"negative_penalty": 0.68,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -5151,37 +5072,36 @@
{
"prompt": "I pasted a very long process note. For now I only need a recap and a short summary, not a skill package.",
"family": "long_context_summary_only",
- "score": 0.256,
+ "score": 0.306,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"summarize_only"
],
"exclusive_negative_concepts": [
"summarize_only"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
- "lexical_support": 0.333,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
+ "lexical_support": 0.222,
"negative_penalty": 0.3,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -5216,11 +5136,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5236,7 +5154,7 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.2,
"negative_penalty": 0.68,
@@ -5271,11 +5189,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5288,9 +5204,9 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.357,
+ "lexical_support": 0.143,
"negative_penalty": 0.42,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -5317,18 +5233,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -5336,11 +5251,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.5,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.3,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -5371,18 +5286,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "iterate_existing_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [
"future_outline",
"no_build_directive"
@@ -5391,11 +5305,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.327,
- "support_score": 0,
+ "semantic_coverage": 0.333,
+ "support_score": 0.16,
"lexical_support": 0.267,
"negative_penalty": 0.64,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"iterate_existing_skill": [
@@ -5426,18 +5340,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -5445,11 +5358,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.5,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.286,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -5481,18 +5394,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"future_outline",
@@ -5502,11 +5414,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.238,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.19,
"negative_penalty": 0.82,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -5545,18 +5457,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"future_outline",
"no_build_directive"
@@ -5565,11 +5476,11 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
"lexical_support": 0.263,
"negative_penalty": 0.64,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -5710,18 +5621,16 @@
{
"prompt": "Build a reusable skill from this long internal process note, and make sure it has references, scripts, and a trigger description.",
"family": "complex_multi_asset",
- "score": 0.468,
+ "score": 0.767,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5734,9 +5643,9 @@
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.4,
+ "semantic_coverage": 0.733,
"support_score": 0.06,
- "lexical_support": 0.316,
+ "lexical_support": 0.211,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -5763,33 +5672,32 @@
{
"prompt": "Take this existing skill draft, tighten the trigger boundary, add near-neighbor evals, and package it for distribution.",
"family": "iterate_existing_skill",
- "score": 0.727,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
"eval_optimize",
- "iterate_existing_skill",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "iterate_existing_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.691,
- "support_score": 0,
- "lexical_support": 0.389,
+ "semantic_coverage": 1.0,
+ "support_score": 0.16,
+ "lexical_support": 0.333,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -5820,18 +5728,16 @@
{
"prompt": "Turn this recurring onboarding process into a reusable skill library entry with validation notes.",
"family": "workflow_to_skill",
- "score": 0.568,
+ "score": 0.936,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -5842,9 +5748,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.509,
+ "semantic_coverage": 0.933,
"support_score": 0,
- "lexical_support": 0.5,
+ "lexical_support": 0.214,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -5867,34 +5773,32 @@
{
"prompt": "Codify this quarterly release routine into a reusable skill package with scripts, references, and packaging checks.",
"family": "paraphrase_trigger",
- "score": 0.858,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
"extra_positive_concepts": [
+ "build_skill",
"multi_asset"
],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0.06,
- "lexical_support": 0.375,
+ "semantic_coverage": 0.933,
+ "support_score": 0.4,
+ "lexical_support": 0.312,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -5925,32 +5829,31 @@
{
"prompt": "Below is a noisy transcript plus release notes and prompt history. Ignore the chatter and turn this playbook into an agent capability package for the team library.",
"family": "long_context_trigger",
- "score": 0.744,
+ "score": 0.839,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"team_operationalize",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.709,
- "support_score": 0,
- "lexical_support": 0.4,
+ "semantic_coverage": 0.733,
+ "support_score": 0.34,
+ "lexical_support": 0.24,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -5987,11 +5890,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6007,9 +5908,9 @@
"no_build_directive",
"summarize_only"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.286,
+ "lexical_support": 0.143,
"negative_penalty": 0.72,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -6043,11 +5944,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6063,7 +5962,7 @@
"explain_only",
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.091,
"negative_penalty": 0.68,
@@ -6099,11 +5998,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6117,7 +6014,7 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
"lexical_support": 0.222,
"negative_penalty": 0.6,
@@ -6151,19 +6048,18 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"no_build_directive",
"translate_only"
@@ -6172,11 +6068,11 @@
"no_build_directive",
"translate_only"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
- "lexical_support": 0.25,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
+ "lexical_support": 0.188,
"negative_penalty": 0.77,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -6214,17 +6110,15 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
- "matched_desired_concepts": [
+ "matched_desired_concepts": [],
+ "extra_positive_concepts": [
"build_skill"
],
- "extra_positive_concepts": [],
"matched_negative_concepts": [
"brainstorm_only",
"future_outline"
@@ -6233,9 +6127,9 @@
"brainstorm_only",
"future_outline"
],
- "semantic_coverage": 0.309,
- "support_score": 0,
- "lexical_support": 0.308,
+ "semantic_coverage": 0.0,
+ "support_score": 0.34,
+ "lexical_support": 0.077,
"negative_penalty": 0.46,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -6265,11 +6159,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -6284,7 +6176,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.357,
+ "lexical_support": 0.214,
"negative_penalty": 0.64,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -6312,18 +6204,17 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -6331,11 +6222,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.491,
- "support_score": 0,
- "lexical_support": 0.615,
+ "semantic_coverage": 0.333,
+ "support_score": 0.34,
+ "lexical_support": 0.308,
"negative_penalty": 0.6,
- "coverage_boost": 0.04,
+ "coverage_boost": 0.0,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -6361,26 +6252,25 @@
{
"prompt": "From the long workflow below, create a template only and a checklist for discussion only. No skill package.",
"family": "long_context_near_neighbor",
- "score": 0.0,
+ "score": 0.023,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [
"document_only",
"no_build_directive"
@@ -6388,11 +6278,11 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.673,
- "support_score": 0,
+ "semantic_coverage": 0.667,
+ "support_score": 0.34,
"lexical_support": 0.438,
"negative_penalty": 0.6,
- "coverage_boost": 0.06,
+ "coverage_boost": 0.04,
"concept_evidence": {
"positive": {
"build_skill": [
@@ -6495,18 +6385,16 @@
{
"prompt": "Operationalize this support escalation routine as a maintained skill and keep trigger eval coverage.",
"family": "blind_operationalize",
- "score": 0.392,
+ "score": 0.665,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6516,9 +6404,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.214,
+ "lexical_support": 0.143,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -6533,24 +6421,21 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Take these SOP scraps, prompt notes, and transcript excerpts and formalize them into a reusable capability package.",
"family": "blind_multi_asset",
- "score": 0.405,
+ "score": 0.673,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6560,9 +6445,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.375,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -6578,26 +6463,23 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
}
],
"should_not_trigger": [
{
"prompt": "Turn this workflow note into a wiki reference only, do not create any skill files, and keep it out of agent execution.",
"family": "blind_document_only",
- "score": 0.028,
+ "score": 0.16,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6610,9 +6492,9 @@
"document_only"
],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0.06,
- "lexical_support": 0.318,
+ "lexical_support": 0.227,
"negative_penalty": 0.18,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -6642,11 +6524,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -6684,11 +6564,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -6703,7 +6581,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0.06,
- "lexical_support": 0.308,
+ "lexical_support": 0.231,
"negative_penalty": 0.22,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -6723,18 +6601,16 @@
{
"prompt": "Review this process and suggest improvements for later; no packaging yet.",
"family": "blind_brainstorm_only",
- "score": 0.026,
+ "score": 0.305,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6748,7 +6624,7 @@
"exclusive_negative_concepts": [
"future_outline"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.273,
"negative_penalty": 0.22,
@@ -6768,7 +6644,8 @@
]
}
}
- }
+ },
+ "boundary_case": true
}
]
}
@@ -6847,18 +6724,16 @@
{
"prompt": "Operationalize this support escalation routine as a maintained skill and keep trigger eval coverage.",
"family": "blind_operationalize",
- "score": 0.392,
+ "score": 0.665,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6868,9 +6743,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.214,
+ "lexical_support": 0.143,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -6885,24 +6760,21 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
},
{
"prompt": "Take these SOP scraps, prompt notes, and transcript excerpts and formalize them into a reusable capability package.",
"family": "blind_multi_asset",
- "score": 0.405,
+ "score": 0.673,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6912,9 +6784,9 @@
"extra_positive_concepts": [],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
- "lexical_support": 0.375,
+ "lexical_support": 0.25,
"negative_penalty": 0,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -6930,26 +6802,23 @@
},
"negative": {}
}
- },
- "boundary_case": true
+ }
}
],
"should_not_trigger": [
{
"prompt": "Turn this workflow note into a wiki reference only, do not create any skill files, and keep it out of agent execution.",
"family": "blind_document_only",
- "score": 0.028,
+ "score": 0.16,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -6962,9 +6831,9 @@
"document_only"
],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0.06,
- "lexical_support": 0.318,
+ "lexical_support": 0.227,
"negative_penalty": 0.18,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -6994,11 +6863,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -7036,11 +6903,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -7055,7 +6920,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0.06,
- "lexical_support": 0.308,
+ "lexical_support": 0.231,
"negative_penalty": 0.22,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -7075,18 +6940,16 @@
{
"prompt": "Review this process and suggest improvements for later; no packaging yet.",
"family": "blind_brainstorm_only",
- "score": 0.026,
+ "score": 0.305,
"predicted_trigger": false,
"expected_trigger": false,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -7100,7 +6963,7 @@
"exclusive_negative_concepts": [
"future_outline"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0,
"lexical_support": 0.273,
"negative_penalty": 0.22,
@@ -7120,7 +6983,8 @@
]
}
}
- }
+ },
+ "boundary_case": true
}
]
}
@@ -8586,32 +8450,31 @@
{
"prompt": "Ignore the noisy meeting chatter and operationalize this support escalation workflow into a maintained skill package with trigger hardening.",
"family": "adversarial_noisy_operationalize",
- "score": 0.834,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.263,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.211,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -8638,32 +8501,31 @@
{
"prompt": "Codify this routine into a reusable capability from docs, transcripts, and prompt history, then stress test the route boundary.",
"family": "adversarial_multi_asset_boundary",
- "score": 0.842,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.368,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.263,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -8698,11 +8560,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -8719,9 +8579,9 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0.06,
- "lexical_support": 0.429,
+ "lexical_support": 0.357,
"negative_penalty": 0.6,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -8757,11 +8617,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -8779,9 +8637,9 @@
"summarize_only",
"translate_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.188,
+ "lexical_support": 0.125,
"negative_penalty": 1.25,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -8820,11 +8678,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -8839,7 +8695,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.25,
+ "lexical_support": 0.2,
"negative_penalty": 0.64,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -8868,11 +8724,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -8889,9 +8743,9 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.182,
+ "lexical_support": 0.273,
"negative_penalty": 0.88,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -8991,32 +8845,31 @@
{
"prompt": "Ignore the noisy meeting chatter and operationalize this support escalation workflow into a maintained skill package with trigger hardening.",
"family": "adversarial_noisy_operationalize",
- "score": 0.834,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.263,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.211,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -9043,32 +8896,31 @@
{
"prompt": "Codify this routine into a reusable capability from docs, transcripts, and prompt history, then stress test the route boundary.",
"family": "adversarial_multi_asset_boundary",
- "score": 0.842,
+ "score": 1.0,
"predicted_trigger": true,
"expected_trigger": true,
"passed": true,
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
- "build_skill",
"eval_optimize",
"reuse_package",
"transform_workflow"
],
- "extra_positive_concepts": [],
+ "extra_positive_concepts": [
+ "build_skill"
+ ],
"matched_negative_concepts": [],
"exclusive_negative_concepts": [],
- "semantic_coverage": 0.818,
- "support_score": 0,
- "lexical_support": 0.368,
+ "semantic_coverage": 0.933,
+ "support_score": 0.34,
+ "lexical_support": 0.263,
"negative_penalty": 0,
"coverage_boost": 0.06,
"concept_evidence": {
@@ -9103,11 +8955,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -9124,9 +8974,9 @@
"exclusive_negative_concepts": [
"no_build_directive"
],
- "semantic_coverage": 0.364,
+ "semantic_coverage": 0.667,
"support_score": 0.06,
- "lexical_support": 0.429,
+ "lexical_support": 0.357,
"negative_penalty": 0.6,
"coverage_boost": 0.04,
"concept_evidence": {
@@ -9162,11 +9012,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -9184,9 +9032,9 @@
"summarize_only",
"translate_only"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.188,
+ "lexical_support": 0.125,
"negative_penalty": 1.25,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -9225,11 +9073,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [],
@@ -9244,7 +9090,7 @@
],
"semantic_coverage": 0.0,
"support_score": 0,
- "lexical_support": 0.25,
+ "lexical_support": 0.2,
"negative_penalty": 0.64,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -9273,11 +9119,9 @@
"score_detail": {
"mode": "semantic-intent",
"desired_positive_concepts": [
- "build_skill",
"transform_workflow",
"reuse_package",
"eval_optimize",
- "iterate_existing_skill",
"team_operationalize"
],
"matched_desired_concepts": [
@@ -9294,9 +9138,9 @@
"future_outline",
"no_build_directive"
],
- "semantic_coverage": 0.182,
+ "semantic_coverage": 0.333,
"support_score": 0,
- "lexical_support": 0.182,
+ "lexical_support": 0.273,
"negative_penalty": 0.88,
"coverage_boost": 0.0,
"concept_evidence": {
@@ -9725,9 +9569,9 @@
{
"id": "current",
"label": "Current",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"strategy": "current",
- "estimated_tokens": 65,
+ "estimated_tokens": 53,
"dev": {
"false_positives": 0,
"false_negatives": 0,
@@ -9743,7 +9587,7 @@
-1.0,
-1.0,
-1.0,
- 65
+ 53
],
"holdout": {
"false_positives": 0,
@@ -10077,30 +9921,30 @@
},
"winner_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.673,
- "mean_non_trigger_score": 0.0,
- "mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.468,
- "max_non_trigger_score": 0.0,
- "score_gap": 0.468,
- "threshold_margin": 0.138,
- "positive_threshold_buffer": 0.138,
- "negative_threshold_buffer": 0.33,
+ "mean_positive_score": 0.908,
+ "mean_non_trigger_score": 0.003,
+ "mean_near_neighbor_score": 0.006,
+ "min_positive_score": 0.767,
+ "max_non_trigger_score": 0.023,
+ "score_gap": 0.744,
+ "threshold_margin": 0.307,
+ "positive_threshold_buffer": 0.437,
+ "negative_threshold_buffer": 0.307,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"current_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.673,
- "mean_non_trigger_score": 0.0,
- "mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.468,
- "max_non_trigger_score": 0.0,
- "score_gap": 0.468,
- "threshold_margin": 0.138,
- "positive_threshold_buffer": 0.138,
- "negative_threshold_buffer": 0.33,
+ "mean_positive_score": 0.908,
+ "mean_non_trigger_score": 0.003,
+ "mean_near_neighbor_score": 0.006,
+ "min_positive_score": 0.767,
+ "max_non_trigger_score": 0.023,
+ "score_gap": 0.744,
+ "threshold_margin": 0.307,
+ "positive_threshold_buffer": 0.437,
+ "negative_threshold_buffer": 0.307,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
"risk_band": "healthy"
@@ -10178,33 +10022,33 @@
},
"winner_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.399,
- "mean_non_trigger_score": 0.013,
- "mean_near_neighbor_score": 0.013,
- "min_positive_score": 0.392,
- "max_non_trigger_score": 0.028,
- "score_gap": 0.364,
- "threshold_margin": 0.062,
- "positive_threshold_buffer": 0.062,
- "negative_threshold_buffer": 0.302,
- "boundary_case_count": 2,
- "boundary_case_rate": 0.333,
- "risk_band": "watch"
+ "mean_positive_score": 0.669,
+ "mean_non_trigger_score": 0.116,
+ "mean_near_neighbor_score": 0.152,
+ "min_positive_score": 0.665,
+ "max_non_trigger_score": 0.305,
+ "score_gap": 0.36,
+ "threshold_margin": 0.025,
+ "positive_threshold_buffer": 0.335,
+ "negative_threshold_buffer": 0.025,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "tight"
},
"current_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.399,
- "mean_non_trigger_score": 0.013,
- "mean_near_neighbor_score": 0.013,
- "min_positive_score": 0.392,
- "max_non_trigger_score": 0.028,
- "score_gap": 0.364,
- "threshold_margin": 0.062,
- "positive_threshold_buffer": 0.062,
- "negative_threshold_buffer": 0.302,
- "boundary_case_count": 2,
- "boundary_case_rate": 0.333,
- "risk_band": "watch"
+ "mean_positive_score": 0.669,
+ "mean_non_trigger_score": 0.116,
+ "mean_near_neighbor_score": 0.152,
+ "min_positive_score": 0.665,
+ "max_non_trigger_score": 0.305,
+ "score_gap": 0.36,
+ "threshold_margin": 0.025,
+ "positive_threshold_buffer": 0.335,
+ "negative_threshold_buffer": 0.025,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "tight"
},
"baseline_calibration": {
"threshold": 0.33,
@@ -10353,14 +10197,14 @@
},
"winner_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.838,
+ "mean_positive_score": 1.0,
"mean_non_trigger_score": 0.0,
"mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.834,
+ "min_positive_score": 1.0,
"max_non_trigger_score": 0.0,
- "score_gap": 0.834,
+ "score_gap": 1.0,
"threshold_margin": 0.33,
- "positive_threshold_buffer": 0.504,
+ "positive_threshold_buffer": 0.67,
"negative_threshold_buffer": 0.33,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
@@ -10368,14 +10212,14 @@
},
"current_calibration": {
"threshold": 0.33,
- "mean_positive_score": 0.838,
+ "mean_positive_score": 1.0,
"mean_non_trigger_score": 0.0,
"mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.834,
+ "min_positive_score": 1.0,
"max_non_trigger_score": 0.0,
- "score_gap": 0.834,
+ "score_gap": 1.0,
"threshold_margin": 0.33,
- "positive_threshold_buffer": 0.504,
+ "positive_threshold_buffer": 0.67,
"negative_threshold_buffer": 0.33,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
@@ -10430,8 +10274,8 @@
},
"summary": {
"winner_label": "Current",
- "winner_tokens": 65,
- "current_tokens": 65,
+ "winner_tokens": 53,
+ "current_tokens": 53,
"winner_dev_total_errors": 0,
"current_dev_total_errors": 0,
"winner_holdout_total_errors": 0,
@@ -10444,7 +10288,7 @@
"current_adversarial_holdout_total_errors": 0,
"winner_judge_blind_agreement_rate": 1.0,
"winner_adversarial_risk_band": "healthy",
- "winner_adversarial_score_gap": 0.834,
+ "winner_adversarial_score_gap": 1.0,
"candidate_count": 6,
"baseline_tokens": 8,
"baseline_dev_total_errors": 1,
diff --git a/reports/description_optimization.md b/reports/description_optimization.md
index 3081aa26..3f86caf9 100644
--- a/reports/description_optimization.md
+++ b/reports/description_optimization.md
@@ -2,19 +2,19 @@
Winner: `Current`
-- current tokens: `65`
-- winner tokens: `65`
+- current tokens: `53`
+- winner tokens: `53`
- baseline tokens: `8`
## Winner
-Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
## Candidate Ranking
| Candidate | Tokens | Dev FP | Dev FN | Dev Near | Holdout FP | Holdout FN |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| `Current` | 65 | 0 | 0 | 1.0 | 0 | 0 |
+| `Current` | 53 | 0 | 0 | 1.0 | 0 | 0 |
| `Minimal` | 41 | 1 | 1 | 1.0 | 0 | 0 |
| `Guardrail` | 56 | 1 | 1 | 1.0 | 0 | 0 |
| `Balanced` | 60 | 1 | 1 | 1.0 | 0 | 0 |
@@ -34,9 +34,9 @@ Create, refactor, evaluate, and package agent skills from workflows, prompts, tr
| Gate | Winner Gap | Winner Risk | Winner Boundary Rate | Current Gap | Baseline Gap |
| --- | ---: | --- | ---: | ---: | ---: |
-| Holdout | 0.468 | healthy | 0.0 | 0.468 | 0.416 |
-| Blind Holdout | 0.364 | watch | 0.333 | 0.364 | 0.368 |
-| Adversarial Holdout | 0.834 | healthy | 0.0 | 0.834 | 0.845 |
+| Holdout | 0.744 | healthy | 0.0 | 0.744 | 0.416 |
+| Blind Holdout | 0.36 | tight | 0.167 | 0.36 | 0.368 |
+| Adversarial Holdout | 1.0 | healthy | 0.0 | 1.0 | 0.845 |
## Judge Blind Summary
diff --git a/reports/description_optimization_suite.json b/reports/description_optimization_suite.json
index b80cda28..46800ec1 100644
--- a/reports/description_optimization_suite.json
+++ b/reports/description_optimization_suite.json
@@ -3,9 +3,9 @@
{
"name": "yao-meta-skill",
"winner_label": "Current",
- "winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
- "winner_tokens": 65,
- "current_tokens": 65,
+ "winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "winner_tokens": 53,
+ "current_tokens": 53,
"winner_holdout_fp": 0,
"winner_holdout_fn": 0,
"current_holdout_fp": 0,
@@ -36,44 +36,44 @@
"calibration": {
"holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.673,
- "mean_non_trigger_score": 0.0,
- "mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.468,
- "max_non_trigger_score": 0.0,
- "score_gap": 0.468,
- "threshold_margin": 0.138,
- "positive_threshold_buffer": 0.138,
- "negative_threshold_buffer": 0.33,
+ "mean_positive_score": 0.908,
+ "mean_non_trigger_score": 0.003,
+ "mean_near_neighbor_score": 0.006,
+ "min_positive_score": 0.767,
+ "max_non_trigger_score": 0.023,
+ "score_gap": 0.744,
+ "threshold_margin": 0.307,
+ "positive_threshold_buffer": 0.437,
+ "negative_threshold_buffer": 0.307,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"blind_holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.399,
- "mean_non_trigger_score": 0.013,
- "mean_near_neighbor_score": 0.013,
- "min_positive_score": 0.392,
- "max_non_trigger_score": 0.028,
- "score_gap": 0.364,
- "threshold_margin": 0.062,
- "positive_threshold_buffer": 0.062,
- "negative_threshold_buffer": 0.302,
- "boundary_case_count": 2,
- "boundary_case_rate": 0.333,
- "risk_band": "watch"
+ "mean_positive_score": 0.669,
+ "mean_non_trigger_score": 0.116,
+ "mean_near_neighbor_score": 0.152,
+ "min_positive_score": 0.665,
+ "max_non_trigger_score": 0.305,
+ "score_gap": 0.36,
+ "threshold_margin": 0.025,
+ "positive_threshold_buffer": 0.335,
+ "negative_threshold_buffer": 0.025,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "tight"
},
"adversarial_holdout": {
"threshold": 0.33,
- "mean_positive_score": 0.838,
+ "mean_positive_score": 1.0,
"mean_non_trigger_score": 0.0,
"mean_near_neighbor_score": 0.0,
- "min_positive_score": 0.834,
+ "min_positive_score": 1.0,
"max_non_trigger_score": 0.0,
- "score_gap": 0.834,
+ "score_gap": 1.0,
"threshold_margin": 0.33,
- "positive_threshold_buffer": 0.504,
+ "positive_threshold_buffer": 0.67,
"negative_threshold_buffer": 0.33,
"boundary_case_count": 0,
"boundary_case_rate": 0.0,
@@ -295,13 +295,13 @@
"current_holdout_fp": 0,
"current_holdout_fn": 1,
"baseline_holdout_fp": 0,
- "baseline_holdout_fn": 2,
+ "baseline_holdout_fn": 1,
"winner_blind_holdout_fp": 0,
"winner_blind_holdout_fn": 0,
"current_blind_holdout_fp": 0,
"current_blind_holdout_fn": 0,
"baseline_blind_holdout_fp": 0,
- "baseline_blind_holdout_fn": 1,
+ "baseline_blind_holdout_fn": 0,
"winner_blind_holdout_total_errors": 0,
"winner_judge_blind_holdout_fp": 0,
"winner_judge_blind_holdout_fn": 0,
@@ -320,7 +320,7 @@
"calibration": {
"holdout": {
"threshold": 0.48,
- "mean_positive_score": 0.37,
+ "mean_positive_score": 0.45,
"mean_non_trigger_score": 0.019,
"mean_near_neighbor_score": 0.037,
"min_positive_score": 0.08,
@@ -335,49 +335,49 @@
},
"blind_holdout": {
"threshold": 0.48,
- "mean_positive_score": 0.902,
- "mean_non_trigger_score": 0.168,
- "mean_near_neighbor_score": 0.138,
- "min_positive_score": 0.804,
- "max_non_trigger_score": 0.386,
- "score_gap": 0.418,
- "threshold_margin": 0.094,
- "positive_threshold_buffer": 0.324,
- "negative_threshold_buffer": 0.094,
- "boundary_case_count": 1,
- "boundary_case_rate": 0.167,
+ "mean_positive_score": 0.82,
+ "mean_non_trigger_score": 0.059,
+ "mean_near_neighbor_score": 0.0,
+ "min_positive_score": 0.645,
+ "max_non_trigger_score": 0.228,
+ "score_gap": 0.417,
+ "threshold_margin": 0.165,
+ "positive_threshold_buffer": 0.165,
+ "negative_threshold_buffer": 0.252,
+ "boundary_case_count": 0,
+ "boundary_case_rate": 0.0,
"risk_band": "healthy"
},
"adversarial_holdout": {
"threshold": 0.34,
- "mean_positive_score": 0.827,
- "mean_non_trigger_score": 0.027,
- "mean_near_neighbor_score": 0.011,
- "min_positive_score": 0.654,
- "max_non_trigger_score": 0.056,
- "score_gap": 0.598,
- "threshold_margin": 0.284,
- "positive_threshold_buffer": 0.314,
- "negative_threshold_buffer": 0.284,
- "boundary_case_count": 0,
- "boundary_case_rate": 0.0,
- "risk_band": "healthy"
+ "mean_positive_score": 0.907,
+ "mean_non_trigger_score": 0.083,
+ "mean_near_neighbor_score": 0.139,
+ "min_positive_score": 0.815,
+ "max_non_trigger_score": 0.277,
+ "score_gap": 0.538,
+ "threshold_margin": 0.063,
+ "positive_threshold_buffer": 0.475,
+ "negative_threshold_buffer": 0.063,
+ "boundary_case_count": 1,
+ "boundary_case_rate": 0.167,
+ "risk_band": "watch"
}
},
"judge_blind": {
"winner": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
},
"current": {
"agreement_rate": 1.0,
- "mean_confidence": 0.657,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
},
"baseline": {
"agreement_rate": 1.0,
- "mean_confidence": 0.69,
+ "mean_confidence": 0.677,
"rubric_version": "blind-v1"
}
},
diff --git a/reports/description_optimization_suite.md b/reports/description_optimization_suite.md
index a93a7ecb..6344d2fa 100644
--- a/reports/description_optimization_suite.md
+++ b/reports/description_optimization_suite.md
@@ -2,9 +2,9 @@
| Target | Winner | Winner Tokens | Holdout FP | Holdout FN | Blind FP | Blind FN | Judge Blind Errors | Adv FP | Adv FN | Adv Gap | Adv Risk | Status |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- |
-| `yao-meta-skill` | `Current` | 65 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.834 | healthy | ok |
+| `yao-meta-skill` | `Current` | 53 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1.0 | healthy | ok |
| `team-frontend-review` | `Current` | 50 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.509 | healthy | ok |
-| `governed-incident-command` | `Current` | 37 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0.598 | healthy | ok |
+| `governed-incident-command` | `Current` | 37 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0.538 | watch | ok |
## Family Coverage
diff --git a/reports/evidence_consistency.json b/reports/evidence_consistency.json
new file mode 100644
index 00000000..ed224071
--- /dev/null
+++ b/reports/evidence_consistency.json
@@ -0,0 +1,3611 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-17",
+ "skill_dir": ".",
+ "summary": {
+ "check_count": 39,
+ "pass_count": 39,
+ "warn_count": 0,
+ "fail_count": 0,
+ "decision": "consistent"
+ },
+ "status_counts": {
+ "pass": 39,
+ "warn": 0,
+ "fail": 0
+ },
+ "checks": [
+ {
+ "key": "required-report-artifacts",
+ "label": "Required report artifacts are readable",
+ "status": "pass",
+ "expected": "all required JSON and Markdown reports exist and parse",
+ "actual": "all readable",
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json",
+ "reports/adoption_drift_report.json",
+ "reports/world_class_evidence_ledger.json",
+ "reports/world_class_evidence_plan.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_preflight.json",
+ "reports/world_class_submission_review.json",
+ "reports/world_class_operator_runbook.json",
+ "reports/skill_os2_coverage.json",
+ "reports/review-studio.json",
+ "reports/package_verification.json",
+ "reports/install_simulation.json",
+ "reports/security_trust_report.json",
+ "reports/context_budget.json",
+ "reports/world_class_claim_guard.json",
+ "reports/skill-os-2-review.md"
+ ],
+ "detail": "The consistency gate can only be trusted when every source JSON report parses and every source Markdown report is readable."
+ },
+ {
+ "key": "release-evidence-flow-covers-first-class-reports",
+ "label": "Release evidence flow covers first-class reports",
+ "status": "pass",
+ "expected": {
+ "AGENTS.md": true,
+ "source_refresh_section": true,
+ "clean_lock_section": true,
+ "source_refresh_commands": {
+ "python3 scripts/run_output_execution.py --runner-command '[\"python3\",\"scripts/local_output_eval_runner.py\"]'": true,
+ "python3 scripts/compile_skill.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip": true,
+ "python3 scripts/simulate_install.py . --package-dir dist --install-root dist/install-simulation --output-json reports/install_simulation.json --output-md reports/install_simulation.md --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/trust_check.py . --output-json reports/security_trust_report.json --output-md reports/security_trust_report.md": true,
+ "python3 scripts/registry_audit.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/verify_package.py . --package-dir dist --expectations evals/packaging_expectations.json --registry-json reports/registry_audit.json --output-json reports/package_verification.json --output-md reports/package_verification.md --require-zip --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/upgrade_check.py . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json --current-package-json reports/registry_audit.json --output-json reports/upgrade_check.json --output-md reports/upgrade_check.md --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_adoption_drift_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_architecture_maintainability.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/python_compat_check.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/probe_runtime_permissions.py . --package-dir dist": true,
+ "python3 scripts/render_review_waivers.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_annotations.py .": true,
+ "python3 scripts/build_skill_atlas.py --workspace-root . --output-dir skill_atlas --report-html reports/skill_atlas.html --report-json reports/skill_atlas.json --today \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_plan.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_ledger.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_intake.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_submission_review.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_operator_runbook.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_claim_guard.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_daily_skillops_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_weekly_curator_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_os2_audit.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_os2_coverage.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_context_reports.py --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_benchmark_reproducibility.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_overview.py .": true,
+ "python3 scripts/render_skill_interpretation.py .": true,
+ "python3 scripts/render_review_viewer.py .": true,
+ "python3 scripts/render_world_class_preflight.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json": true,
+ "python3 scripts/render_evidence_consistency.py . --generated-at \"$GENERATED_AT\"": true
+ },
+ "clean_lock_commands": {
+ "python3 scripts/render_context_reports.py --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_benchmark_reproducibility.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_daily_skillops_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_weekly_curator_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_overview.py .": true,
+ "python3 scripts/render_skill_interpretation.py .": true,
+ "python3 scripts/render_review_viewer.py .": true,
+ "python3 scripts/render_world_class_preflight.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json": true,
+ "python3 scripts/render_evidence_consistency.py . --generated-at \"$GENERATED_AT\"": true
+ }
+ },
+ "actual": {
+ "AGENTS.md": true,
+ "source_refresh_section": true,
+ "clean_lock_section": true,
+ "source_refresh_commands": {
+ "python3 scripts/run_output_execution.py --runner-command '[\"python3\",\"scripts/local_output_eval_runner.py\"]'": true,
+ "python3 scripts/compile_skill.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip": true,
+ "python3 scripts/simulate_install.py . --package-dir dist --install-root dist/install-simulation --output-json reports/install_simulation.json --output-md reports/install_simulation.md --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/trust_check.py . --output-json reports/security_trust_report.json --output-md reports/security_trust_report.md": true,
+ "python3 scripts/registry_audit.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/verify_package.py . --package-dir dist --expectations evals/packaging_expectations.json --registry-json reports/registry_audit.json --output-json reports/package_verification.json --output-md reports/package_verification.md --require-zip --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/upgrade_check.py . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json --current-package-json reports/registry_audit.json --output-json reports/upgrade_check.json --output-md reports/upgrade_check.md --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_adoption_drift_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_architecture_maintainability.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/python_compat_check.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/probe_runtime_permissions.py . --package-dir dist": true,
+ "python3 scripts/render_review_waivers.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_annotations.py .": true,
+ "python3 scripts/build_skill_atlas.py --workspace-root . --output-dir skill_atlas --report-html reports/skill_atlas.html --report-json reports/skill_atlas.json --today \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_plan.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_ledger.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_evidence_intake.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_submission_review.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_operator_runbook.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_world_class_claim_guard.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_daily_skillops_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_weekly_curator_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_os2_audit.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_os2_coverage.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_context_reports.py --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_benchmark_reproducibility.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_overview.py .": true,
+ "python3 scripts/render_skill_interpretation.py .": true,
+ "python3 scripts/render_review_viewer.py .": true,
+ "python3 scripts/render_world_class_preflight.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json": true,
+ "python3 scripts/render_evidence_consistency.py . --generated-at \"$GENERATED_AT\"": true
+ },
+ "clean_lock_commands": {
+ "python3 scripts/render_context_reports.py --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_benchmark_reproducibility.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_daily_skillops_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_weekly_curator_report.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_skill_overview.py .": true,
+ "python3 scripts/render_skill_interpretation.py .": true,
+ "python3 scripts/render_review_viewer.py .": true,
+ "python3 scripts/render_world_class_preflight.py . --generated-at \"$GENERATED_AT\"": true,
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json": true,
+ "python3 scripts/render_evidence_consistency.py . --generated-at \"$GENERATED_AT\"": true
+ }
+ },
+ "paths": [
+ "AGENTS.md",
+ "reports/output_execution_runs.json",
+ "reports/install_simulation.json",
+ "reports/security_trust_report.json",
+ "reports/registry_audit.json",
+ "reports/package_verification.json",
+ "reports/upgrade_check.json",
+ "reports/adoption_drift_report.json",
+ "reports/architecture_maintainability.json",
+ "reports/python_compatibility.json",
+ "reports/runtime_permission_probes.json",
+ "reports/review_waivers.json",
+ "reports/review_annotations.json",
+ "reports/skill_atlas.json",
+ "reports/skill_os2_audit.json",
+ "reports/skill_os2_coverage.json",
+ "reports/context_budget.json",
+ "reports/context_budget_summary.json",
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json",
+ "reports/review-viewer.json",
+ "reports/world_class_evidence_preflight.json",
+ "reports/skillops/daily",
+ "reports/skillops/weekly",
+ "reports/review-studio.json",
+ "reports/evidence_consistency.json"
+ ],
+ "detail": "Release refresh and clean-lock instructions must regenerate every first-class report before evidence consistency can be trusted."
+ },
+ {
+ "key": "review-studio-gate-action-mirror",
+ "label": "Review Studio gates mirror review actions",
+ "status": "pass",
+ "expected": {
+ "non_pass_gate_keys": [
+ "output-lab",
+ "review-waivers",
+ "world-class-evidence"
+ ],
+ "action_gate_keys": [
+ "output-lab",
+ "review-waivers",
+ "world-class-evidence"
+ ],
+ "pass_gate_action_ids_empty": true,
+ "gate_action_mirrors": {
+ "output-lab": {
+ "review_action_id": "review-action-output-lab",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-output-lab",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ },
+ "review-waivers": {
+ "review_action_id": "review-action-review-waivers",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-review-waivers",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ },
+ "world-class-evidence": {
+ "review_action_id": "review-action-world-class-evidence",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-world-class-evidence",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ }
+ }
+ },
+ "actual": {
+ "non_pass_gate_keys": [
+ "output-lab",
+ "review-waivers",
+ "world-class-evidence"
+ ],
+ "action_gate_keys": [
+ "output-lab",
+ "review-waivers",
+ "world-class-evidence"
+ ],
+ "pass_gate_action_ids_empty": true,
+ "gate_action_mirrors": {
+ "output-lab": {
+ "review_action_id": "review-action-output-lab",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-output-lab",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ },
+ "review-waivers": {
+ "review_action_id": "review-action-review-waivers",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-review-waivers",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ },
+ "world-class-evidence": {
+ "review_action_id": "review-action-world-class-evidence",
+ "review_action_status": "warn",
+ "title_present": true,
+ "next_step_present": true,
+ "reason_present": true,
+ "source_ref_present": true,
+ "verification_command_present": true,
+ "top_level_action_id": "review-action-world-class-evidence",
+ "top_level_action_status": "warn",
+ "top_level_title_present": true,
+ "top_level_next_step_present": true,
+ "top_level_reason_present": true,
+ "top_level_source_ref_present": true,
+ "top_level_verification_command_present": true
+ }
+ }
+ },
+ "paths": [
+ "reports/review-studio.json"
+ ],
+ "detail": "Every non-pass Review Studio gate must have exactly one top-level review action and a gate-local action summary with source refs and a verification command."
+ },
+ {
+ "key": "review-studio-context-budget-mirror",
+ "label": "Review Studio mirrors context budget governance",
+ "status": "pass",
+ "expected": {
+ "status": "pass",
+ "detail": "initial load 998/1000; deferred 525864/120000; top deferred scripts 464561; resource governance governed; quality density 140.3",
+ "evidence": "reports/context_budget.json"
+ },
+ "actual": {
+ "status": "pass",
+ "detail": "initial load 998/1000; deferred 525864/120000; top deferred scripts 464561; resource governance governed; quality density 140.3",
+ "evidence": "reports/context_budget.json"
+ },
+ "paths": [
+ "reports/context_budget.json",
+ "reports/review-studio.json"
+ ],
+ "detail": "Review Studio must not keep stale context warnings after context reports prove large deferred resources are governed."
+ },
+ {
+ "key": "benchmark-release-lock-self-consistency",
+ "label": "Benchmark release lock matches source dirty state",
+ "status": "pass",
+ "expected": true,
+ "actual": true,
+ "paths": [
+ "reports/benchmark_reproducibility.json"
+ ],
+ "detail": "The benchmark release lock must be blocked by source changes, while generated evidence artifacts are tracked as generation context."
+ },
+ {
+ "key": "benchmark-clean-worktree-release-lock",
+ "label": "Clean worktree keeps a clean benchmark release lock",
+ "status": "pass",
+ "expected": "checked only when git is available and the current worktree is clean",
+ "actual": {
+ "available": true,
+ "clean": false,
+ "changed_file_count": 27
+ },
+ "paths": [
+ "reports/benchmark_reproducibility.json"
+ ],
+ "detail": "Dirty or non-git worktrees cannot prove final release-lock freshness, so this check is advisory until the final clean-lock pass."
+ },
+ {
+ "key": "skill-ir-evidence-path-contract",
+ "label": "Human-facing reports expose the canonical Skill IR artifact",
+ "status": "pass",
+ "expected": {
+ "overview_source_path": "skill-ir/examples/yao-meta-skill.json",
+ "interpretation_source_path": "skill-ir/examples/yao-meta-skill.json",
+ "review_studio_evidence_path": "skill-ir/examples/yao-meta-skill.json",
+ "overview_deliverable": true,
+ "interpretation_deliverable": true,
+ "exists": true,
+ "schema_version": "2.0.0"
+ },
+ "actual": {
+ "overview_source_path": "skill-ir/examples/yao-meta-skill.json",
+ "interpretation_source_path": "skill-ir/examples/yao-meta-skill.json",
+ "review_studio_evidence_path": "skill-ir/examples/yao-meta-skill.json",
+ "overview_deliverable": true,
+ "interpretation_deliverable": true,
+ "exists": true,
+ "schema_version": "2.0.0"
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json",
+ "reports/review-studio.json",
+ "skill-ir/examples/yao-meta-skill.json"
+ ],
+ "detail": "Skill IR is the 2.0 platform-neutral semantic source, so user-facing reports must link to the artifact that actually exists."
+ },
+ {
+ "key": "overview-benchmark-commit",
+ "label": "overview embeds the benchmark commit",
+ "status": "pass",
+ "expected": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "actual": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-overview.json"
+ ],
+ "detail": "Human-facing reports must point to the same benchmark release-lock commit."
+ },
+ {
+ "key": "overview-benchmark-summary",
+ "label": "overview embeds benchmark summary fields",
+ "status": "pass",
+ "expected": {
+ "release_lock_ready": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3
+ },
+ "actual": {
+ "release_lock_ready": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3
+ },
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-overview.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "overview-adoption-summary",
+ "label": "overview embeds adoption drift summary fields",
+ "status": "pass",
+ "expected": {
+ "event_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "adoption_rate": 0,
+ "risk_band": "low",
+ "event_types": {
+ "review_event": 1
+ },
+ "source_types": {
+ "manual": 1
+ }
+ },
+ "actual": {
+ "event_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "adoption_rate": 0,
+ "risk_band": "low",
+ "event_types": {
+ "review_event": 1
+ },
+ "source_types": {
+ "manual": 1
+ }
+ },
+ "paths": [
+ "reports/adoption_drift_report.json",
+ "reports/skill-overview.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "overview-world-class-ledger-summary",
+ "label": "overview embeds world-class ledger summary fields",
+ "status": "pass",
+ "expected": {
+ "ledger_entry_count": 4,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "actual": {
+ "ledger_entry_count": 4,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/skill-overview.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "overview-world-class-readiness",
+ "label": "overview derives readiness from the ledger",
+ "status": "pass",
+ "expected": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "pending_count": 4,
+ "accepted_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11
+ },
+ "actual": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "pending_count": 4,
+ "accepted_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/skill-overview.json"
+ ],
+ "detail": "Readiness summaries must be derived from the evidence ledger, not hand-maintained copy."
+ },
+ {
+ "key": "interpretation-benchmark-commit",
+ "label": "interpretation embeds the benchmark commit",
+ "status": "pass",
+ "expected": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "actual": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "Human-facing reports must point to the same benchmark release-lock commit."
+ },
+ {
+ "key": "interpretation-benchmark-summary",
+ "label": "interpretation embeds benchmark summary fields",
+ "status": "pass",
+ "expected": {
+ "release_lock_ready": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3
+ },
+ "actual": {
+ "release_lock_ready": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3
+ },
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "interpretation-adoption-summary",
+ "label": "interpretation embeds adoption drift summary fields",
+ "status": "pass",
+ "expected": {
+ "event_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "adoption_rate": 0,
+ "risk_band": "low",
+ "event_types": {
+ "review_event": 1
+ },
+ "source_types": {
+ "manual": 1
+ }
+ },
+ "actual": {
+ "event_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "adoption_rate": 0,
+ "risk_band": "low",
+ "event_types": {
+ "review_event": 1
+ },
+ "source_types": {
+ "manual": 1
+ }
+ },
+ "paths": [
+ "reports/adoption_drift_report.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "interpretation-world-class-ledger-summary",
+ "label": "interpretation embeds world-class ledger summary fields",
+ "status": "pass",
+ "expected": {
+ "ledger_entry_count": 4,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "actual": {
+ "ledger_entry_count": 4,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "Selected summary fields must match exactly across generated reports."
+ },
+ {
+ "key": "interpretation-world-class-readiness",
+ "label": "interpretation derives readiness from the ledger",
+ "status": "pass",
+ "expected": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "pending_count": 4,
+ "accepted_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11
+ },
+ "actual": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "pending_count": 4,
+ "accepted_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "Readiness summaries must be derived from the evidence ledger, not hand-maintained copy."
+ },
+ {
+ "key": "overview-interpretation-lockstep-scorecard",
+ "label": "Overview and interpretation share scorecard",
+ "status": "pass",
+ "expected": {
+ "completeness_score": {
+ "label": "完整度",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 已存在，是 Skill 的入口。",
+ "README.md 已存在，便于人工阅读。",
+ "agents/interface.yaml 已存在，便于跨平台适配。",
+ "manifest.json 已存在，生命周期信息可追踪。",
+ "reports/ 已存在，生成证据可以随包体迁移。"
+ ]
+ },
+ "trigger_score": {
+ "label": "触发清晰",
+ "score": 100,
+ "reasons": [
+ "frontmatter description 已存在，具备基础路由面。",
+ "description 有足够长度说明任务边界。",
+ "description 已包含使用场景或排除边界信号。",
+ "evals/ 已存在，可承载触发样例或质量检查。",
+ "intent-confidence 报告已生成，可辅助判断触发稳定性。"
+ ]
+ },
+ "evidence_score": {
+ "label": "证据充分",
+ "score": 100,
+ "reasons": [
+ "已生成 20 / 20 类报告证据。",
+ "skill-ir.json 已存在。",
+ "compiled_targets.json 已存在。",
+ "intent-dialogue.json 已存在。"
+ ]
+ },
+ "maintainability_score": {
+ "label": "可维护性",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 约 339 个词/字。",
+ "入口文件保持克制，可维护性较好。",
+ "references/ 已承载扩展指导。",
+ "scripts/ 已承载确定性逻辑。",
+ "evals/ 已承载可迁移检查。"
+ ]
+ },
+ "portability_score": {
+ "label": "可迁移性",
+ "score": 100,
+ "reasons": [
+ "agents/interface.yaml 已存在。",
+ "manifest.json 已存在。",
+ "目标平台或 adapter target 已声明。",
+ "入口文件未发现明显私有绝对路径。"
+ ]
+ },
+ "context_cost": {
+ "label": "上下文成本",
+ "score": 42,
+ "reasons": [
+ "入口约 339 个词/字，references 约 16934 个词/字。",
+ "分数越高代表上下文成本越低。",
+ "上下文成本偏高，建议压缩入口或拆分 references。"
+ ]
+ }
+ },
+ "actual": {
+ "completeness_score": {
+ "label": "完整度",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 已存在，是 Skill 的入口。",
+ "README.md 已存在，便于人工阅读。",
+ "agents/interface.yaml 已存在，便于跨平台适配。",
+ "manifest.json 已存在，生命周期信息可追踪。",
+ "reports/ 已存在，生成证据可以随包体迁移。"
+ ]
+ },
+ "trigger_score": {
+ "label": "触发清晰",
+ "score": 100,
+ "reasons": [
+ "frontmatter description 已存在，具备基础路由面。",
+ "description 有足够长度说明任务边界。",
+ "description 已包含使用场景或排除边界信号。",
+ "evals/ 已存在，可承载触发样例或质量检查。",
+ "intent-confidence 报告已生成，可辅助判断触发稳定性。"
+ ]
+ },
+ "evidence_score": {
+ "label": "证据充分",
+ "score": 100,
+ "reasons": [
+ "已生成 20 / 20 类报告证据。",
+ "skill-ir.json 已存在。",
+ "compiled_targets.json 已存在。",
+ "intent-dialogue.json 已存在。"
+ ]
+ },
+ "maintainability_score": {
+ "label": "可维护性",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 约 339 个词/字。",
+ "入口文件保持克制，可维护性较好。",
+ "references/ 已承载扩展指导。",
+ "scripts/ 已承载确定性逻辑。",
+ "evals/ 已承载可迁移检查。"
+ ]
+ },
+ "portability_score": {
+ "label": "可迁移性",
+ "score": 100,
+ "reasons": [
+ "agents/interface.yaml 已存在。",
+ "manifest.json 已存在。",
+ "目标平台或 adapter target 已声明。",
+ "入口文件未发现明显私有绝对路径。"
+ ]
+ },
+ "context_cost": {
+ "label": "上下文成本",
+ "score": 42,
+ "reasons": [
+ "入口约 339 个词/字，references 约 16934 个词/字。",
+ "分数越高代表上下文成本越低。",
+ "上下文成本偏高，建议压缩入口或拆分 references。"
+ ]
+ }
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-capability-profile",
+ "label": "Overview and interpretation share capability_profile",
+ "status": "pass",
+ "expected": {
+ "archetype": "governed",
+ "task_family": "Execution operation",
+ "maturity": "governed",
+ "trigger_strength": "手动触发 + description 路由",
+ "reuse_scope": "跨平台",
+ "matrix": {
+ "execution_certainty": 72,
+ "knowledge_density": 80
+ }
+ },
+ "actual": {
+ "archetype": "governed",
+ "task_family": "Execution operation",
+ "maturity": "governed",
+ "trigger_strength": "手动触发 + description 路由",
+ "reuse_scope": "跨平台",
+ "matrix": {
+ "execution_certainty": 72,
+ "knowledge_density": 80
+ }
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-principle-model",
+ "label": "Overview and interpretation share principle_model",
+ "status": "pass",
+ "expected": {
+ "nodes": [
+ {
+ "title": "意图澄清",
+ "body": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse."
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ }
+ ],
+ "layers": [
+ "入口层",
+ "参考层",
+ "脚本层",
+ "评估层",
+ "报告层"
+ ]
+ },
+ "actual": {
+ "nodes": [
+ {
+ "title": "意图澄清",
+ "body": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse."
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ }
+ ],
+ "layers": [
+ "入口层",
+ "参考层",
+ "脚本层",
+ "评估层",
+ "报告层"
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-contract-boundary",
+ "label": "Overview and interpretation share contract_boundary",
+ "status": "pass",
+ "expected": {
+ "trigger": {
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "inputs": [
+ "rough workflow notes, SOPs, runbooks, prompts, transcripts, documents, or repeated task descriptions",
+ "an existing skill directory that needs refactor, evaluation, packaging, or governance hardening",
+ "target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution",
+ "benchmark references, local constraints, desired maturity tier, and review standards"
+ ],
+ "outputs": [
+ "A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ],
+ "should_trigger": [
+ "把重复流程沉淀为可复用的 agent skill。",
+ "把分散提示词、对话记录或操作规范整理为稳定能力。",
+ "团队复用前，需要明确触发边界、质量证据和维护责任。"
+ ],
+ "should_not_trigger": [
+ "只需要一次性回答、没有复用价值的临时请求。",
+ "要求直接执行相邻任务，而不是沉淀或使用这个 Skill。",
+ "缺少必要事实且用户不允许澄清的场景。"
+ ],
+ "boundary_cards": [
+ {
+ "label": "Owned",
+ "body": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
+ },
+ {
+ "label": "Adjacent",
+ "body": "相邻任务需要先确认是否应转为独立 Skill。"
+ },
+ {
+ "label": "Excluded",
+ "body": "不替代人工事实核查，也不静默扩大职责。"
+ }
+ ]
+ },
+ "actual": {
+ "trigger": {
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "inputs": [
+ "rough workflow notes, SOPs, runbooks, prompts, transcripts, documents, or repeated task descriptions",
+ "an existing skill directory that needs refactor, evaluation, packaging, or governance hardening",
+ "target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution",
+ "benchmark references, local constraints, desired maturity tier, and review standards"
+ ],
+ "outputs": [
+ "A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ],
+ "should_trigger": [
+ "把重复流程沉淀为可复用的 agent skill。",
+ "把分散提示词、对话记录或操作规范整理为稳定能力。",
+ "团队复用前，需要明确触发边界、质量证据和维护责任。"
+ ],
+ "should_not_trigger": [
+ "只需要一次性回答、没有复用价值的临时请求。",
+ "要求直接执行相邻任务，而不是沉淀或使用这个 Skill。",
+ "缺少必要事实且用户不允许澄清的场景。"
+ ],
+ "boundary_cards": [
+ {
+ "label": "Owned",
+ "body": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
+ },
+ {
+ "label": "Adjacent",
+ "body": "相邻任务需要先确认是否应转为独立 Skill。"
+ },
+ {
+ "label": "Excluded",
+ "body": "不替代人工事实核查，也不静默扩大职责。"
+ }
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-quality-review",
+ "label": "Overview and interpretation share quality_review",
+ "status": "pass",
+ "expected": {
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Output Eval Lab scorecard，可比较 with-skill 与 baseline 输出质量。",
+ "已生成 Output Execution Runs，可区分记录样本、命令执行和模型执行证据。",
+ "已生成 Output Review Adjudication，可记录盲评决策、一致率和待评审项。"
+ ],
+ "gaps": [
+ "上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。"
+ ],
+ "recommendations": [
+ "先改触发边界，再扩展工作流。",
+ "只把重复且稳定的步骤沉淀为脚本。",
+ "每次升级后重新生成报告并检查分数原因。"
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 89.0,
+ "highlights": [
+ "Primary prompt task family: Execution operation.",
+ "Complexity: expert — multiple task families plus governance, evaluation, or expert-level constraints",
+ "Completeness: 100/100.",
+ "Clarity: 85/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 100,
+ "band": "system-ready"
+ },
+ "highlights": [
+ "Stability: system-ready (100/100).",
+ "Owned job: Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language.",
+ "Leverage: Install output self-repair checks — Add only the checks that prevent recurring output mistakes."
+ ]
+ }
+ },
+ "actual": {
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Output Eval Lab scorecard，可比较 with-skill 与 baseline 输出质量。",
+ "已生成 Output Execution Runs，可区分记录样本、命令执行和模型执行证据。",
+ "已生成 Output Review Adjudication，可记录盲评决策、一致率和待评审项。"
+ ],
+ "gaps": [
+ "上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。"
+ ],
+ "recommendations": [
+ "先改触发边界，再扩展工作流。",
+ "只把重复且稳定的步骤沉淀为脚本。",
+ "每次升级后重新生成报告并检查分数原因。"
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 89.0,
+ "highlights": [
+ "Primary prompt task family: Execution operation.",
+ "Complexity: expert — multiple task families plus governance, evaluation, or expert-level constraints",
+ "Completeness: 100/100.",
+ "Clarity: 85/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 100,
+ "band": "system-ready"
+ },
+ "highlights": [
+ "Stability: system-ready (100/100).",
+ "Owned job: Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language.",
+ "Leverage: Install output self-repair checks — Add only the checks that prevent recurring output mistakes."
+ ]
+ }
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-risk-governance",
+ "label": "Overview and interpretation share risk_governance",
+ "status": "pass",
+ "expected": {
+ "risks": [
+ {
+ "name": "误触发风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "frontmatter description 已存在，具备基础路由面。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "输出漂移风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "证据不足风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "包体膨胀风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "SKILL.md 约 339 个词/字。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "跨平台迁移风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "agents/interface.yaml 已存在。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ }
+ ],
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "report",
+ "doc"
+ ],
+ "score": 3,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "source",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "capture"
+ ],
+ "score": 1,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "code_or_command_safety",
+ "label": "Code and command safety",
+ "matched_keywords": [
+ "script"
+ ],
+ "score": 1,
+ "risks": [
+ "Commands can omit environment assumptions, working directory, or rollback notes.",
+ "Code snippets can look runnable while missing required inputs.",
+ "Error handling can be either absent or over-engineered."
+ ],
+ "constraints": [
+ "Name the working directory, required inputs, and expected output for each command.",
+ "Mark destructive or external side-effect operations explicitly.",
+ "Prefer the smallest runnable snippet over broad framework code."
+ ],
+ "self_repair": [
+ "Scan each command for cwd, input, output, and side-effect assumptions.",
+ "Remove speculative error handling that is not tied to a real failure mode."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "summary"
+ ],
+ "score": 1,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ }
+ ],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ]
+ },
+ "actual": {
+ "risks": [
+ {
+ "name": "误触发风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "frontmatter description 已存在，具备基础路由面。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "输出漂移风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "证据不足风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "包体膨胀风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "SKILL.md 约 339 个词/字。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "跨平台迁移风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "agents/interface.yaml 已存在。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ }
+ ],
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "report",
+ "doc"
+ ],
+ "score": 3,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "source",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "capture"
+ ],
+ "score": 1,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "code_or_command_safety",
+ "label": "Code and command safety",
+ "matched_keywords": [
+ "script"
+ ],
+ "score": 1,
+ "risks": [
+ "Commands can omit environment assumptions, working directory, or rollback notes.",
+ "Code snippets can look runnable while missing required inputs.",
+ "Error handling can be either absent or over-engineered."
+ ],
+ "constraints": [
+ "Name the working directory, required inputs, and expected output for each command.",
+ "Mark destructive or external side-effect operations explicitly.",
+ "Prefer the smallest runnable snippet over broad framework code."
+ ],
+ "self_repair": [
+ "Scan each command for cwd, input, output, and side-effect assumptions.",
+ "Remove speculative error handling that is not tied to a real failure mode."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "summary"
+ ],
+ "score": 1,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ }
+ ],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-world-class-readiness",
+ "label": "Overview and interpretation share world_class_readiness",
+ "status": "pass",
+ "expected": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "entry_count": 4,
+ "pending_count": 4,
+ "accepted_count": 0,
+ "external_pending_count": 3,
+ "human_pending_count": 1,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "conclusion_zh": "世界级证据尚未完成：4 项待补，0 项已接受。",
+ "conclusion_en": "World-class evidence is not complete: 4 pending, 0 accepted.",
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label_zh": "提供商留出",
+ "label_en": "Provider Holdout",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "缺少真实 provider 模型运行和 token metadata。",
+ "summary_en": "Missing a real provider model run and token metadata.",
+ "blocked_checks": []
+ },
+ {
+ "key": "human-adjudication",
+ "label_zh": "人工盲评",
+ "label_en": "Human Adjudication",
+ "category": "human",
+ "category_zh": "人工证据",
+ "category_en": "Human evidence",
+ "status": "pending",
+ "summary_zh": "盲评 pair 仍待真实 reviewer 决策。",
+ "summary_en": "Blind-review pairs still need real reviewer decisions.",
+ "blocked_checks": [
+ "No pending decisions",
+ "Judgments complete",
+ "Reviewer metadata"
+ ]
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label_zh": "原生权限",
+ "label_en": "Native Permission",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "原生 runtime enforcement 仍待目标客户端或外部安装器证明。",
+ "summary_en": "Native runtime enforcement still needs target-client or external-installer proof.",
+ "blocked_checks": [
+ "Native enforcement"
+ ]
+ },
+ {
+ "key": "native-client-telemetry",
+ "label_zh": "原生遥测",
+ "label_en": "Native Telemetry",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "真实外部客户端 metadata-only 事件仍未导入。",
+ "summary_en": "Real external-client metadata-only events have not been imported yet.",
+ "blocked_checks": [
+ "External events",
+ "Adoption sample"
+ ]
+ }
+ ]
+ },
+ "actual": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "entry_count": 4,
+ "pending_count": 4,
+ "accepted_count": 0,
+ "external_pending_count": 3,
+ "human_pending_count": 1,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "conclusion_zh": "世界级证据尚未完成：4 项待补，0 项已接受。",
+ "conclusion_en": "World-class evidence is not complete: 4 pending, 0 accepted.",
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label_zh": "提供商留出",
+ "label_en": "Provider Holdout",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "缺少真实 provider 模型运行和 token metadata。",
+ "summary_en": "Missing a real provider model run and token metadata.",
+ "blocked_checks": []
+ },
+ {
+ "key": "human-adjudication",
+ "label_zh": "人工盲评",
+ "label_en": "Human Adjudication",
+ "category": "human",
+ "category_zh": "人工证据",
+ "category_en": "Human evidence",
+ "status": "pending",
+ "summary_zh": "盲评 pair 仍待真实 reviewer 决策。",
+ "summary_en": "Blind-review pairs still need real reviewer decisions.",
+ "blocked_checks": [
+ "No pending decisions",
+ "Judgments complete",
+ "Reviewer metadata"
+ ]
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label_zh": "原生权限",
+ "label_en": "Native Permission",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "原生 runtime enforcement 仍待目标客户端或外部安装器证明。",
+ "summary_en": "Native runtime enforcement still needs target-client or external-installer proof.",
+ "blocked_checks": [
+ "Native enforcement"
+ ]
+ },
+ {
+ "key": "native-client-telemetry",
+ "label_zh": "原生遥测",
+ "label_en": "Native Telemetry",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "真实外部客户端 metadata-only 事件仍未导入。",
+ "summary_en": "Real external-client metadata-only events have not been imported yet.",
+ "blocked_checks": [
+ "External events",
+ "Adoption sample"
+ ]
+ }
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-package-assets",
+ "label": "Overview and interpretation share package_assets",
+ "status": "pass",
+ "expected": {
+ "entries": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 35
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 153
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 29
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 235
+ }
+ ],
+ "file_count": 456,
+ "folder_count": 4,
+ "distribution": [
+ {
+ "label": "SKILL.md",
+ "value": 1
+ },
+ {
+ "label": "README.md",
+ "value": 1
+ },
+ {
+ "label": "agents/interface.yaml",
+ "value": 1
+ },
+ {
+ "label": "manifest.json",
+ "value": 1
+ },
+ {
+ "label": "references",
+ "value": 35
+ },
+ {
+ "label": "scripts",
+ "value": 153
+ },
+ {
+ "label": "evals",
+ "value": 29
+ },
+ {
+ "label": "reports",
+ "value": 235
+ }
+ ]
+ },
+ "actual": {
+ "entries": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 35
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 153
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 29
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 235
+ }
+ ],
+ "file_count": 456,
+ "folder_count": 4,
+ "distribution": [
+ {
+ "label": "SKILL.md",
+ "value": 1
+ },
+ {
+ "label": "README.md",
+ "value": 1
+ },
+ {
+ "label": "agents/interface.yaml",
+ "value": 1
+ },
+ {
+ "label": "manifest.json",
+ "value": 1
+ },
+ {
+ "label": "references",
+ "value": 35
+ },
+ {
+ "label": "scripts",
+ "value": 153
+ },
+ {
+ "label": "evals",
+ "value": 29
+ },
+ {
+ "label": "reports",
+ "value": 235
+ }
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-interpretation-lockstep-iteration-roadmap",
+ "label": "Overview and interpretation share iteration_roadmap",
+ "status": "pass",
+ "expected": {
+ "items": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
+ {
+ "title": "Borrow one proven pattern on purpose",
+ "why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
+ "actions": [
+ "Read the strongest pattern from obra/superpowers.",
+ "Decide whether to borrow method, structure, execution, or portability, but only one of them first.",
+ "Record what you will not borrow so the package stays light."
+ ],
+ "unlocks": "A cleaner package shape with less accidental over-design."
+ },
+ {
+ "title": "Harden portability semantics",
+ "why": "The skill already signals reuse across environments, so contract clarity matters early.",
+ "actions": [
+ "Confirm activation mode, execution context, and trust assumptions.",
+ "Add or review degradation strategy for non-native targets.",
+ "Package the skill once to verify adapter expectations."
+ ],
+ "unlocks": "Safer cross-environment reuse with less target drift."
+ }
+ ]
+ },
+ "actual": {
+ "items": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
+ {
+ "title": "Borrow one proven pattern on purpose",
+ "why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
+ "actions": [
+ "Read the strongest pattern from obra/superpowers.",
+ "Decide whether to borrow method, structure, execution, or portability, but only one of them first.",
+ "Record what you will not borrow so the package stays light."
+ ],
+ "unlocks": "A cleaner package shape with less accidental over-design."
+ },
+ {
+ "title": "Harden portability semantics",
+ "why": "The skill already signals reuse across environments, so contract clarity matters early.",
+ "actions": [
+ "Confirm activation mode, execution context, and trust assumptions.",
+ "Add or review degradation strategy for non-native targets.",
+ "Package the skill once to verify adapter expectations."
+ ],
+ "unlocks": "Safer cross-environment reuse with less target drift."
+ }
+ ]
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json"
+ ],
+ "detail": "The first-class interpretation report must stay in lockstep with the canonical overview model."
+ },
+ {
+ "key": "overview-html-contract",
+ "label": "overview has a stable HTML contract",
+ "status": "pass",
+ "expected": {
+ "schema_version": "2.0",
+ "default_language": "zh-CN",
+ "layout": "kami-white-audit-v2",
+ "html_report": "reports/skill-overview.html",
+ "html_exists": true
+ },
+ "actual": {
+ "schema_version": "2.0",
+ "default_language": "zh-CN",
+ "layout": "kami-white-audit-v2",
+ "html_report": "reports/skill-overview.html",
+ "html_exists": true
+ },
+ "paths": [
+ "reports/skill-overview.json",
+ "reports/skill-overview.html"
+ ],
+ "detail": "Report output paths and language defaults are part of the user-facing contract."
+ },
+ {
+ "key": "interpretation-html-contract",
+ "label": "interpretation has a stable HTML contract",
+ "status": "pass",
+ "expected": {
+ "schema_version": "2.0",
+ "default_language": "zh-CN",
+ "layout": "kami-white-audit-v2",
+ "html_report": "reports/skill-interpretation.html",
+ "html_exists": true
+ },
+ "actual": {
+ "schema_version": "2.0",
+ "default_language": "zh-CN",
+ "layout": "kami-white-audit-v2",
+ "html_report": "reports/skill-interpretation.html",
+ "html_exists": true
+ },
+ "paths": [
+ "reports/skill-interpretation.json",
+ "reports/skill-interpretation.html"
+ ],
+ "detail": "Report output paths and language defaults are part of the user-facing contract."
+ },
+ {
+ "key": "coverage-world-class-boundary",
+ "label": "Coverage report mirrors world-class evidence boundary",
+ "status": "pass",
+ "expected": {
+ "world_class_evidence_pending_count": 4,
+ "public_world_class_ready": false
+ },
+ "actual": {
+ "world_class_evidence_pending_count": 4,
+ "public_world_class_ready": false
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/skill_os2_coverage.json"
+ ],
+ "detail": "Blueprint coverage can be locally complete while public world-class evidence remains pending."
+ },
+ {
+ "key": "benchmark-world-class-boundary",
+ "label": "Benchmark report mirrors world-class evidence boundary",
+ "status": "pass",
+ "expected": {
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "public_claim_ready": false
+ },
+ "actual": {
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "public_claim_ready": false
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/benchmark_reproducibility.json"
+ ],
+ "detail": "Benchmark reproducibility must not overstate public claim readiness."
+ },
+ {
+ "key": "benchmark-beta-public-claim-split",
+ "label": "Benchmark separates beta testing from public claims",
+ "status": "pass",
+ "expected": {
+ "beta_test_ready": true,
+ "public_claim_ready": false,
+ "human_review_complete": false,
+ "beta_release_ready": true,
+ "beta_release_scope": "beta/public test release without superiority, fully-reviewed, or world-class claims",
+ "deferred_evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "deferred_human_review": true
+ },
+ "actual": {
+ "beta_test_ready": true,
+ "public_claim_ready": false,
+ "human_review_complete": false,
+ "beta_release_ready": true,
+ "beta_release_scope": "beta/public test release without superiority, fully-reviewed, or world-class claims",
+ "deferred_evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "deferred_human_review": true
+ },
+ "paths": [
+ "reports/benchmark_reproducibility.json",
+ "reports/world_class_evidence_ledger.json"
+ ],
+ "detail": "Beta/public testing may proceed with human blind review deferred, but public world-class or superiority claims must remain blocked."
+ },
+ {
+ "key": "preflight-world-class-boundary",
+ "label": "Preflight mirrors ledger without accepting evidence",
+ "status": "pass",
+ "expected": {
+ "pending_count": 4,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "preflight_counts_as_evidence": false,
+ "credential_value_exposed": false
+ },
+ "actual": {
+ "pending_count": 4,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "ready_to_claim_world_class": false,
+ "preflight_counts_as_evidence": false,
+ "credential_value_exposed": false
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/world_class_evidence_preflight.json"
+ ],
+ "detail": "Collection preflight may help operators gather evidence, but it must not print secrets or change world-class readiness."
+ },
+ {
+ "key": "preflight-submission-kit-handoff",
+ "label": "Preflight exposes a safe submission-kit handoff",
+ "status": "pass",
+ "expected": {
+ "directory": "evidence/world_class/submissions",
+ "drafts_count_as_evidence": false,
+ "preflight_counts_submission_as_completion": false,
+ "html_report": "reports/world_class_evidence_preflight.html",
+ "html_exists": true,
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard .",
+ "artifact_prefill_counts_as_evidence": false,
+ "artifact_role_source": "world-class-submission-kit",
+ "artifact_role_counts_as_evidence": false,
+ "artifact_role_prefill_counts_as_evidence": false,
+ "submission_ref_role_present": true,
+ "supporting_evidence_role_present": true,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false,
+ "submission_ref_total_present": true,
+ "supporting_evidence_total_present": true
+ },
+ "actual": {
+ "directory": "evidence/world_class/submissions",
+ "drafts_count_as_evidence": false,
+ "preflight_counts_submission_as_completion": false,
+ "html_report": "reports/world_class_evidence_preflight.html",
+ "html_exists": true,
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard .",
+ "artifact_prefill_counts_as_evidence": false,
+ "artifact_role_source": "world-class-submission-kit",
+ "artifact_role_counts_as_evidence": false,
+ "artifact_role_prefill_counts_as_evidence": false,
+ "submission_ref_role_present": true,
+ "supporting_evidence_role_present": true,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false,
+ "submission_ref_total_present": true,
+ "supporting_evidence_total_present": true
+ },
+ "paths": [
+ "reports/world_class_evidence_preflight.json",
+ "reports/world_class_evidence_preflight.html"
+ ],
+ "detail": "Preflight must give operators the exact draft, SHA-prefill, intake, review, ledger, and claim-guard commands without letting drafts, prefill, or submissions count as accepted evidence."
+ },
+ {
+ "key": "review-studio-preflight-artifact-role-handoff",
+ "label": "Review Studio mirrors preflight artifact roles",
+ "status": "pass",
+ "expected": {
+ "provider-holdout": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 1,
+ "submission_ref_ready_count": 1,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "human-adjudication": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 8,
+ "supporting_evidence_ready_count": 8,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "native-permission-enforcement": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 11,
+ "supporting_evidence_ready_count": 11,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "native-client-telemetry": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ }
+ },
+ "actual": {
+ "provider-holdout": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 1,
+ "submission_ref_ready_count": 1,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "human-adjudication": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 8,
+ "supporting_evidence_ready_count": 8,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "native-permission-enforcement": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 11,
+ "supporting_evidence_ready_count": 11,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ },
+ "native-client-telemetry": {
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "submission_ref_copy_to_artifact_refs": true,
+ "supporting_evidence_copy_to_artifact_refs": false
+ }
+ },
+ "paths": [
+ "reports/world_class_evidence_preflight.json",
+ "reports/review-studio.json"
+ ],
+ "detail": "The Review Studio world-class action card must carry the same submission-ref versus supporting-evidence contract as the preflight handoff."
+ },
+ {
+ "key": "world-class-phase-queue-consistency",
+ "label": "World-class phase queues mirror repair rows",
+ "status": "pass",
+ "expected": {
+ "summary": {
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 12,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false
+ },
+ "top_level_phase_queue": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 8,
+ "row_count": 8,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "item_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ "phase_queue_counts_as_completion": false,
+ "operator_runbook_summary": {
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 12,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false
+ },
+ "operator_runbook_top_level_phase_queue": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 8,
+ "row_count": 8,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "operator_runbook_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ "review_studio_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ },
+ "actual": {
+ "summary": {
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 12,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false
+ },
+ "top_level_phase_queue": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 8,
+ "row_count": 8,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "operator_runbook_summary": {
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 12,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false
+ },
+ "operator_runbook_top_level_phase_queue": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 8,
+ "row_count": 8,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "item_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ "operator_runbook_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ "review_studio_phase_queues": {
+ "provider-holdout": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "human-adjudication": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-permission-enforcement": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ],
+ "native-client-telemetry": [
+ {
+ "phase": "unblock-access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ },
+ {
+ "phase": "collect-source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ "phase_queue_counts_as_completion": false
+ },
+ "paths": [
+ "reports/world_class_evidence_preflight.json",
+ "reports/world_class_operator_runbook.json",
+ "reports/review-studio.json"
+ ],
+ "detail": "Phase queues must be derived from repair rows in preflight and mirrored into the operator runbook and Review Studio without counting queue guidance as completion evidence."
+ },
+ {
+ "key": "review-studio-no-overclaim",
+ "label": "Review Studio does not overclaim pending world-class evidence",
+ "status": "pass",
+ "expected": false,
+ "actual": false,
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/review-studio.json"
+ ],
+ "detail": "When world-class evidence is pending, Review Studio must stay in a review or warning posture."
+ },
+ {
+ "key": "claim-guard-package-runtime-surface",
+ "label": "Claim guard covers package and runtime claim surfaces",
+ "status": "pass",
+ "expected": {
+ "overclaim_guard_active": true,
+ "violation_count": 0,
+ "ledger_ready_to_claim_world_class": false,
+ "ledger_pending_count": 4,
+ "metadata_covers_json": true,
+ "package_surface_minimum": true,
+ "claim_surface_covers_package": true,
+ "required_surfaces": {
+ "README.md": true,
+ "SKILL.md": true,
+ "manifest.json": true,
+ "agents/interface.yaml": true,
+ "evidence/world_class/README.md": true,
+ "security/permission_policy.json": true,
+ "reports/world_class_evidence_ledger.json": true,
+ "dist/manifest.json": true,
+ "dist/targets/openai/adapter.json": true
+ },
+ "prohibited_surfaces": []
+ },
+ "actual": {
+ "overclaim_guard_active": true,
+ "violation_count": 0,
+ "ledger_ready_to_claim_world_class": false,
+ "ledger_pending_count": 4,
+ "metadata_covers_json": true,
+ "package_surface_minimum": true,
+ "claim_surface_covers_package": true,
+ "required_surfaces": {
+ "README.md": true,
+ "SKILL.md": true,
+ "manifest.json": true,
+ "agents/interface.yaml": true,
+ "evidence/world_class/README.md": true,
+ "security/permission_policy.json": true,
+ "reports/world_class_evidence_ledger.json": true,
+ "dist/manifest.json": true,
+ "dist/targets/openai/adapter.json": true
+ },
+ "prohibited_surfaces": []
+ },
+ "paths": [
+ "reports/world_class_claim_guard.json",
+ "manifest.json",
+ "agents/interface.yaml",
+ "dist/manifest.json",
+ "dist/targets/openai/adapter.json",
+ "evidence/world_class/README.md",
+ "security/permission_policy.json",
+ "reports/world_class_evidence_ledger.json"
+ ],
+ "detail": "The overclaim guard must scan package manifests, adapter metadata, security policy, and ledger surfaces before public readiness can be trusted."
+ },
+ {
+ "key": "world-class-evidence-workflow-coverage",
+ "label": "World-class evidence workflows cover every pending ledger entry",
+ "status": "pass",
+ "expected": {
+ "keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "plan_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "intake_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "submission_review_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_runbook_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_coordination_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_coordination_counts_as_completion": false,
+ "operator_release_gate_ready": false,
+ "operator_release_gate_counts_as_completion": false,
+ "review_studio_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "intake_ready_to_claim_world_class": false,
+ "submission_review_ready_to_claim_world_class": false,
+ "submission_review_counts_as_completion": false,
+ "operator_runbook_ready_to_claim_world_class": false,
+ "operator_runbook_counts_as_completion": false,
+ "next_actions_present": {
+ "human-adjudication": true,
+ "native-client-telemetry": true,
+ "native-permission-enforcement": true,
+ "provider-holdout": true
+ },
+ "commands": {
+ "human-adjudication": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "native-client-telemetry": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "native-permission-enforcement": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "provider-holdout": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ }
+ }
+ },
+ "actual": {
+ "keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "plan_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "intake_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "submission_review_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_runbook_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_coordination_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "operator_coordination_counts_as_completion": false,
+ "operator_release_gate_ready": false,
+ "operator_release_gate_counts_as_completion": false,
+ "review_studio_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "intake_ready_to_claim_world_class": false,
+ "submission_review_ready_to_claim_world_class": false,
+ "submission_review_counts_as_completion": false,
+ "operator_runbook_ready_to_claim_world_class": false,
+ "operator_runbook_counts_as_completion": false,
+ "next_actions_present": {
+ "human-adjudication": true,
+ "native-client-telemetry": true,
+ "native-permission-enforcement": true,
+ "provider-holdout": true
+ },
+ "commands": {
+ "human-adjudication": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "native-client-telemetry": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "native-permission-enforcement": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ },
+ "provider-holdout": {
+ "intake": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "operator_runbook": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ },
+ "review_studio": {
+ "prepare_submission": true,
+ "validate_intake": true,
+ "submission_review": true,
+ "refresh_ledger": true,
+ "guard_claim": true
+ }
+ }
+ }
+ },
+ "paths": [
+ "reports/world_class_evidence_ledger.json",
+ "reports/world_class_evidence_plan.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_submission_review.json",
+ "reports/world_class_operator_runbook.json",
+ "reports/review-studio.json"
+ ],
+ "detail": "Every pending world-class evidence key must have matching plan, intake, submission review, operator runbook, and Review Studio actions without counting planned work as completion."
+ },
+ {
+ "key": "skill-os-2-review-current-evidence",
+ "label": "Skill OS 2.0 review summary mirrors current evidence",
+ "status": "pass",
+ "expected": [
+ "score `91`",
+ "`16` gates",
+ "`3` warnings",
+ "`67` declared internal modules",
+ "`86 / 86` CLI help smoke checks passing across `153` scripts",
+ "`689` zip entries",
+ "archive with `689` entries",
+ "`12` installer permission checks enforced",
+ "`0` permission failures",
+ "`25` required artifacts",
+ "`23` reproduction commands",
+ "initial load `998/1000`",
+ "target count is `82`"
+ ],
+ "actual": "all present",
+ "paths": [
+ "reports/skill-os-2-review.md",
+ "reports/review-studio.json",
+ "reports/package_verification.json",
+ "reports/install_simulation.json",
+ "reports/security_trust_report.json",
+ "reports/context_budget.json",
+ "reports/benchmark_reproducibility.json",
+ "scripts/ci_test.py"
+ ],
+ "detail": "Manual 2.0 review summaries must not drift from generated gate, package, trust, context, benchmark, or CI evidence."
+ }
+ ],
+ "artifacts": {
+ "json": "reports/evidence_consistency.json",
+ "markdown": "reports/evidence_consistency.md"
+ }
+}
diff --git a/reports/evidence_consistency.md b/reports/evidence_consistency.md
new file mode 100644
index 00000000..013fa854
--- /dev/null
+++ b/reports/evidence_consistency.md
@@ -0,0 +1,57 @@
+# Evidence Consistency
+
+Generated at: `2026-06-17`
+
+## Summary
+
+- decision: `consistent`
+- checks: `39`
+- pass: `39`
+- warn: `0`
+- fail: `0`
+
+This gate compares generated evidence reports against each other. It does not create provider, human, native-client, or permission-enforcement evidence; it only catches drift between reports that already exist.
+
+## Checks
+
+| Check | Status | Detail | Paths |
+| --- | --- | --- | --- |
+| Required report artifacts are readable | `pass` | The consistency gate can only be trusted when every source JSON report parses and every source Markdown report is readable. | `reports/benchmark_reproducibility.json`, `reports/skill-overview.json`, `reports/skill-interpretation.json`, `reports/adoption_drift_report.json`, `reports/world_class_evidence_ledger.json`, `reports/world_class_evidence_plan.json`, `reports/world_class_evidence_intake.json`, `reports/world_class_evidence_preflight.json`, `reports/world_class_submission_review.json`, `reports/world_class_operator_runbook.json`, `reports/skill_os2_coverage.json`, `reports/review-studio.json`, `reports/package_verification.json`, `reports/install_simulation.json`, `reports/security_trust_report.json`, `reports/context_budget.json`, `reports/world_class_claim_guard.json`, `reports/skill-os-2-review.md` |
+| Release evidence flow covers first-class reports | `pass` | Release refresh and clean-lock instructions must regenerate every first-class report before evidence consistency can be trusted. | `AGENTS.md`, `reports/output_execution_runs.json`, `reports/install_simulation.json`, `reports/security_trust_report.json`, `reports/registry_audit.json`, `reports/package_verification.json`, `reports/upgrade_check.json`, `reports/adoption_drift_report.json`, `reports/architecture_maintainability.json`, `reports/python_compatibility.json`, `reports/runtime_permission_probes.json`, `reports/review_waivers.json`, `reports/review_annotations.json`, `reports/skill_atlas.json`, `reports/skill_os2_audit.json`, `reports/skill_os2_coverage.json`, `reports/context_budget.json`, `reports/context_budget_summary.json`, `reports/benchmark_reproducibility.json`, `reports/skill-overview.json`, `reports/skill-interpretation.json`, `reports/review-viewer.json`, `reports/world_class_evidence_preflight.json`, `reports/skillops/daily`, `reports/skillops/weekly`, `reports/review-studio.json`, `reports/evidence_consistency.json` |
+| Review Studio gates mirror review actions | `pass` | Every non-pass Review Studio gate must have exactly one top-level review action and a gate-local action summary with source refs and a verification command. | `reports/review-studio.json` |
+| Review Studio mirrors context budget governance | `pass` | Review Studio must not keep stale context warnings after context reports prove large deferred resources are governed. | `reports/context_budget.json`, `reports/review-studio.json` |
+| Benchmark release lock matches source dirty state | `pass` | The benchmark release lock must be blocked by source changes, while generated evidence artifacts are tracked as generation context. | `reports/benchmark_reproducibility.json` |
+| Clean worktree keeps a clean benchmark release lock | `pass` | Dirty or non-git worktrees cannot prove final release-lock freshness, so this check is advisory until the final clean-lock pass. | `reports/benchmark_reproducibility.json` |
+| Human-facing reports expose the canonical Skill IR artifact | `pass` | Skill IR is the 2.0 platform-neutral semantic source, so user-facing reports must link to the artifact that actually exists. | `reports/skill-overview.json`, `reports/skill-interpretation.json`, `reports/review-studio.json`, `skill-ir/examples/yao-meta-skill.json` |
+| overview embeds the benchmark commit | `pass` | Human-facing reports must point to the same benchmark release-lock commit. | `reports/benchmark_reproducibility.json`, `reports/skill-overview.json` |
+| overview embeds benchmark summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/benchmark_reproducibility.json`, `reports/skill-overview.json` |
+| overview embeds adoption drift summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/adoption_drift_report.json`, `reports/skill-overview.json` |
+| overview embeds world-class ledger summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/world_class_evidence_ledger.json`, `reports/skill-overview.json` |
+| overview derives readiness from the ledger | `pass` | Readiness summaries must be derived from the evidence ledger, not hand-maintained copy. | `reports/world_class_evidence_ledger.json`, `reports/skill-overview.json` |
+| interpretation embeds the benchmark commit | `pass` | Human-facing reports must point to the same benchmark release-lock commit. | `reports/benchmark_reproducibility.json`, `reports/skill-interpretation.json` |
+| interpretation embeds benchmark summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/benchmark_reproducibility.json`, `reports/skill-interpretation.json` |
+| interpretation embeds adoption drift summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/adoption_drift_report.json`, `reports/skill-interpretation.json` |
+| interpretation embeds world-class ledger summary fields | `pass` | Selected summary fields must match exactly across generated reports. | `reports/world_class_evidence_ledger.json`, `reports/skill-interpretation.json` |
+| interpretation derives readiness from the ledger | `pass` | Readiness summaries must be derived from the evidence ledger, not hand-maintained copy. | `reports/world_class_evidence_ledger.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share scorecard | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share capability_profile | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share principle_model | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share contract_boundary | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share quality_review | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share risk_governance | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share world_class_readiness | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share package_assets | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| Overview and interpretation share iteration_roadmap | `pass` | The first-class interpretation report must stay in lockstep with the canonical overview model. | `reports/skill-overview.json`, `reports/skill-interpretation.json` |
+| overview has a stable HTML contract | `pass` | Report output paths and language defaults are part of the user-facing contract. | `reports/skill-overview.json`, `reports/skill-overview.html` |
+| interpretation has a stable HTML contract | `pass` | Report output paths and language defaults are part of the user-facing contract. | `reports/skill-interpretation.json`, `reports/skill-interpretation.html` |
+| Coverage report mirrors world-class evidence boundary | `pass` | Blueprint coverage can be locally complete while public world-class evidence remains pending. | `reports/world_class_evidence_ledger.json`, `reports/skill_os2_coverage.json` |
+| Benchmark report mirrors world-class evidence boundary | `pass` | Benchmark reproducibility must not overstate public claim readiness. | `reports/world_class_evidence_ledger.json`, `reports/benchmark_reproducibility.json` |
+| Benchmark separates beta testing from public claims | `pass` | Beta/public testing may proceed with human blind review deferred, but public world-class or superiority claims must remain blocked. | `reports/benchmark_reproducibility.json`, `reports/world_class_evidence_ledger.json` |
+| Preflight mirrors ledger without accepting evidence | `pass` | Collection preflight may help operators gather evidence, but it must not print secrets or change world-class readiness. | `reports/world_class_evidence_ledger.json`, `reports/world_class_evidence_preflight.json` |
+| Preflight exposes a safe submission-kit handoff | `pass` | Preflight must give operators the exact draft, SHA-prefill, intake, review, ledger, and claim-guard commands without letting drafts, prefill, or submissions count as accepted evidence. | `reports/world_class_evidence_preflight.json`, `reports/world_class_evidence_preflight.html` |
+| Review Studio mirrors preflight artifact roles | `pass` | The Review Studio world-class action card must carry the same submission-ref versus supporting-evidence contract as the preflight handoff. | `reports/world_class_evidence_preflight.json`, `reports/review-studio.json` |
+| World-class phase queues mirror repair rows | `pass` | Phase queues must be derived from repair rows in preflight and mirrored into the operator runbook and Review Studio without counting queue guidance as completion evidence. | `reports/world_class_evidence_preflight.json`, `reports/world_class_operator_runbook.json`, `reports/review-studio.json` |
+| Review Studio does not overclaim pending world-class evidence | `pass` | When world-class evidence is pending, Review Studio must stay in a review or warning posture. | `reports/world_class_evidence_ledger.json`, `reports/review-studio.json` |
+| Claim guard covers package and runtime claim surfaces | `pass` | The overclaim guard must scan package manifests, adapter metadata, security policy, and ledger surfaces before public readiness can be trusted. | `reports/world_class_claim_guard.json`, `manifest.json`, `agents/interface.yaml`, `dist/manifest.json`, `dist/targets/openai/adapter.json`, `evidence/world_class/README.md`, `security/permission_policy.json`, `reports/world_class_evidence_ledger.json` |
+| World-class evidence workflows cover every pending ledger entry | `pass` | Every pending world-class evidence key must have matching plan, intake, submission review, operator runbook, and Review Studio actions without counting planned work as completion. | `reports/world_class_evidence_ledger.json`, `reports/world_class_evidence_plan.json`, `reports/world_class_evidence_intake.json`, `reports/world_class_submission_review.json`, `reports/world_class_operator_runbook.json`, `reports/review-studio.json` |
+| Skill OS 2.0 review summary mirrors current evidence | `pass` | Manual 2.0 review summaries must not drift from generated gate, package, trust, context, benchmark, or CI evidence. | `reports/skill-os-2-review.md`, `reports/review-studio.json`, `reports/package_verification.json`, `reports/install_simulation.json`, `reports/security_trust_report.json`, `reports/context_budget.json`, `reports/benchmark_reproducibility.json`, `scripts/ci_test.py` |
diff --git a/reports/install_simulation.json b/reports/install_simulation.json
index 1d67799e..07155151 100644
--- a/reports/install_simulation.json
+++ b/reports/install_simulation.json
@@ -3,17 +3,21 @@
"schema_version": "2.0",
"generated_at": "2026-06-13",
"skill_dir": ".",
- "package_dir": "dist",
- "install_root": "dist/install-simulation/simulate-yao-meta-skill",
- "installed_skill_dir": "dist/install-simulation/simulate-yao-meta-skill/yao-meta-skill",
+ "package_dir": "tests/tmp_review_studio/dist",
+ "install_root": "tests/tmp_review_studio/install-root/simulate-yao-meta-skill",
+ "installed_skill_dir": "tests/tmp_review_studio/install-root/simulate-yao-meta-skill/yao-meta-skill",
"summary": {
"archive_present": true,
- "archive_entry_count": 488,
+ "archive_entry_count": 696,
"archive_extracted": true,
"entrypoint_loaded": true,
"manifest_loaded": true,
"interface_loaded": true,
- "adapter_count": 3,
+ "adapter_count": 4,
+ "installer_permission_enforced_count": 12,
+ "installer_permission_failure_count": 0,
+ "permission_target_count": 4,
+ "permission_capability_count": 3,
"install_root_is_temp": false,
"failure_count": 0,
"warning_count": 0
@@ -22,7 +26,7 @@
{
"id": "archive-present",
"status": "pass",
- "detail": "Package archive exists: dist/yao-meta-skill.zip"
+ "detail": "Package archive exists: tests/tmp_review_studio/dist/yao-meta-skill.zip"
},
{
"id": "archive-safe-paths",
@@ -108,14 +112,169 @@
"id": "adapter-openai-name",
"status": "pass",
"detail": "openai adapter name matches package manifest"
+ },
+ {
+ "id": "adapter-vscode",
+ "status": "pass",
+ "detail": "vscode adapter is readable after package install simulation"
+ },
+ {
+ "id": "adapter-vscode-name",
+ "status": "pass",
+ "detail": "vscode adapter name matches package manifest"
+ },
+ {
+ "id": "permission-policy-load",
+ "status": "pass",
+ "detail": "Installed permission policy is readable"
+ },
+ {
+ "id": "permission-claude-contract",
+ "status": "pass",
+ "detail": "claude adapter exposes target permission contract for installer enforcement"
+ },
+ {
+ "id": "permission-claude-file_write-approved",
+ "status": "pass",
+ "detail": "claude capability file_write has active reviewer approval"
+ },
+ {
+ "id": "permission-claude-file_write-target-enforcement",
+ "status": "pass",
+ "detail": "claude capability file_write has target enforcement note"
+ },
+ {
+ "id": "permission-claude-network-approved",
+ "status": "pass",
+ "detail": "claude capability network has active reviewer approval"
+ },
+ {
+ "id": "permission-claude-network-target-enforcement",
+ "status": "pass",
+ "detail": "claude capability network has target enforcement note"
+ },
+ {
+ "id": "permission-claude-subprocess-approved",
+ "status": "pass",
+ "detail": "claude capability subprocess has active reviewer approval"
+ },
+ {
+ "id": "permission-claude-subprocess-target-enforcement",
+ "status": "pass",
+ "detail": "claude capability subprocess has target enforcement note"
+ },
+ {
+ "id": "permission-generic-contract",
+ "status": "pass",
+ "detail": "generic adapter exposes target permission contract for installer enforcement"
+ },
+ {
+ "id": "permission-generic-file_write-approved",
+ "status": "pass",
+ "detail": "generic capability file_write has active reviewer approval"
+ },
+ {
+ "id": "permission-generic-file_write-target-enforcement",
+ "status": "pass",
+ "detail": "generic capability file_write has target enforcement note"
+ },
+ {
+ "id": "permission-generic-network-approved",
+ "status": "pass",
+ "detail": "generic capability network has active reviewer approval"
+ },
+ {
+ "id": "permission-generic-network-target-enforcement",
+ "status": "pass",
+ "detail": "generic capability network has target enforcement note"
+ },
+ {
+ "id": "permission-generic-subprocess-approved",
+ "status": "pass",
+ "detail": "generic capability subprocess has active reviewer approval"
+ },
+ {
+ "id": "permission-generic-subprocess-target-enforcement",
+ "status": "pass",
+ "detail": "generic capability subprocess has target enforcement note"
+ },
+ {
+ "id": "permission-openai-contract",
+ "status": "pass",
+ "detail": "openai adapter exposes target permission contract for installer enforcement"
+ },
+ {
+ "id": "permission-openai-file_write-approved",
+ "status": "pass",
+ "detail": "openai capability file_write has active reviewer approval"
+ },
+ {
+ "id": "permission-openai-file_write-target-enforcement",
+ "status": "pass",
+ "detail": "openai capability file_write has target enforcement note"
+ },
+ {
+ "id": "permission-openai-network-approved",
+ "status": "pass",
+ "detail": "openai capability network has active reviewer approval"
+ },
+ {
+ "id": "permission-openai-network-target-enforcement",
+ "status": "pass",
+ "detail": "openai capability network has target enforcement note"
+ },
+ {
+ "id": "permission-openai-subprocess-approved",
+ "status": "pass",
+ "detail": "openai capability subprocess has active reviewer approval"
+ },
+ {
+ "id": "permission-openai-subprocess-target-enforcement",
+ "status": "pass",
+ "detail": "openai capability subprocess has target enforcement note"
+ },
+ {
+ "id": "permission-vscode-contract",
+ "status": "pass",
+ "detail": "vscode adapter exposes target permission contract for installer enforcement"
+ },
+ {
+ "id": "permission-vscode-file_write-approved",
+ "status": "pass",
+ "detail": "vscode capability file_write has active reviewer approval"
+ },
+ {
+ "id": "permission-vscode-file_write-target-enforcement",
+ "status": "pass",
+ "detail": "vscode capability file_write has target enforcement note"
+ },
+ {
+ "id": "permission-vscode-network-approved",
+ "status": "pass",
+ "detail": "vscode capability network has active reviewer approval"
+ },
+ {
+ "id": "permission-vscode-network-target-enforcement",
+ "status": "pass",
+ "detail": "vscode capability network has target enforcement note"
+ },
+ {
+ "id": "permission-vscode-subprocess-approved",
+ "status": "pass",
+ "detail": "vscode capability subprocess has active reviewer approval"
+ },
+ {
+ "id": "permission-vscode-subprocess-target-enforcement",
+ "status": "pass",
+ "detail": "vscode capability subprocess has target enforcement note"
}
],
"failures": [],
"warnings": [],
"artifacts": {
- "archive": "dist/yao-meta-skill.zip",
- "package_manifest": "dist/manifest.json",
- "json": "reports/install_simulation.json",
- "markdown": "reports/install_simulation.md"
+ "archive": "tests/tmp_review_studio/dist/yao-meta-skill.zip",
+ "package_manifest": "tests/tmp_review_studio/dist/manifest.json",
+ "json": "tests/tmp_review_studio/install_simulation.json",
+ "markdown": "tests/tmp_review_studio/install_simulation.md"
}
}
diff --git a/reports/install_simulation.md b/reports/install_simulation.md
index 7a06607a..b0d16dae 100644
--- a/reports/install_simulation.md
+++ b/reports/install_simulation.md
@@ -1,12 +1,14 @@
# Install Simulation
- OK: `True`
-- Package directory: `dist`
+- Package directory: `tests/tmp_review_studio/dist`
- Archive extracted: `True`
- Entrypoint loaded: `True`
- Manifest loaded: `True`
- Interface loaded: `True`
-- Adapters readable: `3`
+- Adapters readable: `4`
+- Installer permissions enforced: `12`
+- Installer permission failures: `0`
- Failures: `0`
- Warnings: `0`
@@ -14,7 +16,7 @@
| Check | Status | Detail |
| --- | --- | --- |
-| `archive-present` | `pass` | Package archive exists: dist/yao-meta-skill.zip |
+| `archive-present` | `pass` | Package archive exists: tests/tmp_review_studio/dist/yao-meta-skill.zip |
| `archive-safe-paths` | `pass` | Archive has no absolute or parent-traversal entries |
| `single-top-level` | `pass` | Archive top-level directory is yao-meta-skill |
| `entrypoint-load` | `pass` | Installed SKILL.md frontmatter is readable |
@@ -32,6 +34,37 @@
| `adapter-generic-name` | `pass` | generic adapter name matches package manifest |
| `adapter-openai` | `pass` | openai adapter is readable after package install simulation |
| `adapter-openai-name` | `pass` | openai adapter name matches package manifest |
+| `adapter-vscode` | `pass` | vscode adapter is readable after package install simulation |
+| `adapter-vscode-name` | `pass` | vscode adapter name matches package manifest |
+| `permission-policy-load` | `pass` | Installed permission policy is readable |
+| `permission-claude-contract` | `pass` | claude adapter exposes target permission contract for installer enforcement |
+| `permission-claude-file_write-approved` | `pass` | claude capability file_write has active reviewer approval |
+| `permission-claude-file_write-target-enforcement` | `pass` | claude capability file_write has target enforcement note |
+| `permission-claude-network-approved` | `pass` | claude capability network has active reviewer approval |
+| `permission-claude-network-target-enforcement` | `pass` | claude capability network has target enforcement note |
+| `permission-claude-subprocess-approved` | `pass` | claude capability subprocess has active reviewer approval |
+| `permission-claude-subprocess-target-enforcement` | `pass` | claude capability subprocess has target enforcement note |
+| `permission-generic-contract` | `pass` | generic adapter exposes target permission contract for installer enforcement |
+| `permission-generic-file_write-approved` | `pass` | generic capability file_write has active reviewer approval |
+| `permission-generic-file_write-target-enforcement` | `pass` | generic capability file_write has target enforcement note |
+| `permission-generic-network-approved` | `pass` | generic capability network has active reviewer approval |
+| `permission-generic-network-target-enforcement` | `pass` | generic capability network has target enforcement note |
+| `permission-generic-subprocess-approved` | `pass` | generic capability subprocess has active reviewer approval |
+| `permission-generic-subprocess-target-enforcement` | `pass` | generic capability subprocess has target enforcement note |
+| `permission-openai-contract` | `pass` | openai adapter exposes target permission contract for installer enforcement |
+| `permission-openai-file_write-approved` | `pass` | openai capability file_write has active reviewer approval |
+| `permission-openai-file_write-target-enforcement` | `pass` | openai capability file_write has target enforcement note |
+| `permission-openai-network-approved` | `pass` | openai capability network has active reviewer approval |
+| `permission-openai-network-target-enforcement` | `pass` | openai capability network has target enforcement note |
+| `permission-openai-subprocess-approved` | `pass` | openai capability subprocess has active reviewer approval |
+| `permission-openai-subprocess-target-enforcement` | `pass` | openai capability subprocess has target enforcement note |
+| `permission-vscode-contract` | `pass` | vscode adapter exposes target permission contract for installer enforcement |
+| `permission-vscode-file_write-approved` | `pass` | vscode capability file_write has active reviewer approval |
+| `permission-vscode-file_write-target-enforcement` | `pass` | vscode capability file_write has target enforcement note |
+| `permission-vscode-network-approved` | `pass` | vscode capability network has active reviewer approval |
+| `permission-vscode-network-target-enforcement` | `pass` | vscode capability network has target enforcement note |
+| `permission-vscode-subprocess-approved` | `pass` | vscode capability subprocess has active reviewer approval |
+| `permission-vscode-subprocess-target-enforcement` | `pass` | vscode capability subprocess has target enforcement note |
## Failures
diff --git a/reports/iteration_bundles/governed-incident-command/bundle.json b/reports/iteration_bundles/governed-incident-command/bundle.json
index cab8ebee..98501ad8 100644
--- a/reports/iteration_bundles/governed-incident-command/bundle.json
+++ b/reports/iteration_bundles/governed-incident-command/bundle.json
@@ -7,7 +7,8 @@
"cause_tags": [
"no_candidate_outperformed_current",
"current_holdout_gap_present",
- "current_holdout_risk"
+ "current_holdout_risk",
+ "adversarial_watch_risk"
],
"gate_status": {
"visible_holdout_non_regression": true,
@@ -28,8 +29,8 @@
"promotion_state": "reference",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, and stakeholder requirements. Use when asked to standardize incident review, severity assessment, incident communication, or incident packet assembly into one reusable operational skill. Do not use for generic debugging, one-off incident explanations, or brainstorming possible response ideas.",
"tokens": 93,
- "dev_errors": 1,
- "holdout_errors": 2
+ "dev_errors": 0,
+ "holdout_errors": 1
},
{
"name": "governed-incident-command",
@@ -39,25 +40,10 @@
"promotion_state": "kept_current",
"description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 37,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": []
},
- {
- "name": "governed-incident-command",
- "role": "candidate",
- "label": "Guardrail",
- "ranking_state": "not_selected",
- "promotion_state": "blocked",
- "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
- "tokens": 51,
- "dev_errors": 1,
- "holdout_errors": 2,
- "reason_tags": [
- "visible_holdout_regression",
- "longer_without_gain"
- ]
- },
{
"name": "governed-incident-command",
"role": "candidate",
@@ -66,7 +52,7 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 54,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": [
"longer_without_gain"
@@ -80,7 +66,7 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
"tokens": 78,
- "dev_errors": 1,
+ "dev_errors": 0,
"holdout_errors": 1,
"reason_tags": [
"longer_without_gain"
@@ -94,9 +80,24 @@
"promotion_state": "blocked",
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
"tokens": 78,
+ "dev_errors": 0,
+ "holdout_errors": 1,
+ "reason_tags": [
+ "longer_without_gain"
+ ]
+ },
+ {
+ "name": "governed-incident-command",
+ "role": "candidate",
+ "label": "Guardrail",
+ "ranking_state": "not_selected",
+ "promotion_state": "blocked",
+ "description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
+ "tokens": 51,
"dev_errors": 1,
"holdout_errors": 1,
"reason_tags": [
+ "weaker_dev_fit",
"longer_without_gain"
]
}
diff --git a/reports/iteration_bundles/governed-incident-command/bundle.md b/reports/iteration_bundles/governed-incident-command/bundle.md
index bcb085aa..4ea7fd31 100644
--- a/reports/iteration_bundles/governed-incident-command/bundle.md
+++ b/reports/iteration_bundles/governed-incident-command/bundle.md
@@ -10,6 +10,7 @@
- `no_candidate_outperformed_current`
- `current_holdout_gap_present`
- `current_holdout_risk`
+- `adversarial_watch_risk`
## Gate Status
@@ -28,12 +29,12 @@
| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |
| --- | --- | --- | --- | ---: | ---: | ---: |
-| baseline | `Baseline` | reference | reference | 93 | 1 | 2 |
-| current | `Current` | selected_by_dev | kept_current | 37 | 1 | 1 |
-| candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 2 |
-| candidate | `Balanced` | not_selected | blocked | 54 | 1 | 1 |
-| candidate | `Boundary` | not_selected | blocked | 78 | 1 | 1 |
-| candidate | `Artifact Aware` | not_selected | blocked | 78 | 1 | 1 |
+| baseline | `Baseline` | reference | reference | 93 | 0 | 1 |
+| current | `Current` | selected_by_dev | kept_current | 37 | 0 | 1 |
+| candidate | `Balanced` | not_selected | blocked | 54 | 0 | 1 |
+| candidate | `Boundary` | not_selected | blocked | 78 | 0 | 1 |
+| candidate | `Artifact Aware` | not_selected | blocked | 78 | 0 | 1 |
+| candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 1 |
## Human Review Stub
diff --git a/reports/iteration_bundles/yao-meta-skill/bundle.json b/reports/iteration_bundles/yao-meta-skill/bundle.json
index 47d9cd6a..c071b070 100644
--- a/reports/iteration_bundles/yao-meta-skill/bundle.json
+++ b/reports/iteration_bundles/yao-meta-skill/bundle.json
@@ -35,8 +35,8 @@
"label": "Current",
"ranking_state": "selected_by_dev",
"promotion_state": "kept_current",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
- "tokens": 65,
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "tokens": 53,
"dev_errors": 0,
"holdout_errors": 0,
"reason_tags": []
@@ -66,7 +66,8 @@
"dev_errors": 2,
"holdout_errors": 0,
"reason_tags": [
- "weaker_dev_fit"
+ "weaker_dev_fit",
+ "longer_without_gain"
]
},
{
@@ -80,7 +81,8 @@
"dev_errors": 2,
"holdout_errors": 0,
"reason_tags": [
- "weaker_dev_fit"
+ "weaker_dev_fit",
+ "longer_without_gain"
]
},
{
@@ -116,8 +118,8 @@
],
"review_template": {
"target": "yao-meta-skill",
- "current_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
- "candidate_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "current_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "candidate_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"focus": [
"no_candidate_outperformed_current"
]
diff --git a/reports/iteration_bundles/yao-meta-skill/bundle.md b/reports/iteration_bundles/yao-meta-skill/bundle.md
index 0abafe75..b207ee61 100644
--- a/reports/iteration_bundles/yao-meta-skill/bundle.md
+++ b/reports/iteration_bundles/yao-meta-skill/bundle.md
@@ -27,7 +27,7 @@
| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |
| --- | --- | --- | --- | ---: | ---: | ---: |
| baseline | `Baseline` | reference | reference | 8 | 1 | 0 |
-| current | `Current` | selected_by_dev | kept_current | 65 | 0 | 0 |
+| current | `Current` | selected_by_dev | kept_current | 53 | 0 | 0 |
| candidate | `Minimal` | not_selected | blocked | 41 | 2 | 0 |
| candidate | `Guardrail` | not_selected | blocked | 56 | 2 | 0 |
| candidate | `Balanced` | not_selected | blocked | 60 | 2 | 0 |
@@ -37,8 +37,8 @@
## Human Review Stub
- target: yao-meta-skill
-- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
-- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
+- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
- review focus: no_candidate_outperformed_current
## Artifact Paths
diff --git a/reports/iteration_bundles/yao-meta-skill/review.md b/reports/iteration_bundles/yao-meta-skill/review.md
index 7559bd45..69c18c9d 100644
--- a/reports/iteration_bundles/yao-meta-skill/review.md
+++ b/reports/iteration_bundles/yao-meta-skill/review.md
@@ -1,8 +1,8 @@
# Human Review
- target: yao-meta-skill
-- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
-- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
+- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
- suggested focus: no_candidate_outperformed_current
Use the shared template in `references/human-review-template.md` to complete the final decision.
diff --git a/reports/iteration_ledger.md b/reports/iteration_ledger.md
index 0d875ca4..acf95708 100644
--- a/reports/iteration_ledger.md
+++ b/reports/iteration_ledger.md
@@ -33,7 +33,7 @@
| --- | --- | --- | --- | --- |
| `yao-meta-skill` | `keep_current` | `Current` | no_candidate_outperformed_current | Keep the current description and open a new candidate only when fresh route evidence appears. |
| `team-frontend-review` | `keep_current` | `Current` | no_candidate_outperformed_current, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
-| `governed-incident-command` | `keep_current` | `Current` | no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
+| `governed-incident-command` | `keep_current` | `Current` | no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk, adversarial_watch_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
## Current Route Scorecard
@@ -43,7 +43,7 @@
| Route | Precision | Recall | Avg Margin |
| --- | ---: | ---: | ---: |
-| `yao-meta-skill` | 1.0 | 1.0 | 0.611 |
+| `yao-meta-skill` | 1.0 | 1.0 | 0.701 |
| `team-frontend-review` | 1.0 | 1.0 | 0.805 |
-| `governed-incident-command` | 1.0 | 1.0 | 0.659 |
+| `governed-incident-command` | 1.0 | 1.0 | 0.586 |
| `no_route` | 1.0 | 1.0 | - |
diff --git a/reports/output_execution_runs.json b/reports/output_execution_runs.json
index 6cb2055f..4ce4e7cf 100644
--- a/reports/output_execution_runs.json
+++ b/reports/output_execution_runs.json
@@ -6,22 +6,32 @@
"mode": "command",
"command": [
"python3",
- "scripts/local_output_eval_runner.py"
+ "scripts/provider_output_eval_runner.py",
+ "--provider",
+ "deepseek",
+ "--model",
+ "deepseek-v4-flash",
+ "--api-format",
+ "chat-completions",
+ "--thinking",
+ "disabled",
+ "--api-key-env",
+ "DEEPSEEK_API_KEY"
],
- "timeout_seconds": 30.0
+ "timeout_seconds": 180.0
},
"summary": {
"case_count": 5,
"variant_run_count": 10,
"command_executed_count": 10,
- "model_executed_count": 0,
+ "model_executed_count": 10,
"recorded_fixture_count": 0,
"timing_observed_count": 10,
- "token_observed_count": 0,
- "token_estimated_count": 10,
- "baseline_pass_rate": 0.0,
- "with_skill_pass_rate": 100.0,
- "delta": 100.0,
+ "token_observed_count": 10,
+ "token_estimated_count": 0,
+ "baseline_pass_rate": 23.67,
+ "with_skill_pass_rate": 43.67,
+ "delta": 20.0,
"regression_count": 0,
"failure_count": 0,
"gate_pass": true
@@ -31,17 +41,17 @@
"case_id": "skill-package-contract",
"variant": "baseline",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 23.11,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 5980.01,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 16,
- "output_tokens": 17,
- "total_tokens": 33,
- "estimated": true
+ "input_tokens": 100,
+ "output_tokens": 384,
+ "total_tokens": 484,
+ "estimated": false
},
"score": 0.0,
"passed_count": 0,
@@ -52,47 +62,49 @@
"has-report",
"has-resource-boundary"
],
- "output_sha256": "ece26a6edbbbea86a93faaf9f0eadf88efbae12f1c3ad81da9376cf2a889d101",
+ "output_sha256": "1aa12d65d55863581eef67aaa173b3a9962524a91533693044cb340a342b760b",
"failure": ""
},
{
"case_id": "skill-package-contract",
"variant": "with_skill",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 23.56,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 11217.19,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 16,
- "output_tokens": 57,
- "total_tokens": 73,
- "estimated": true
+ "input_tokens": 850,
+ "output_tokens": 998,
+ "total_tokens": 1848,
+ "estimated": false
},
- "score": 100.0,
- "passed_count": 4,
- "failed_count": 0,
- "failed_assertions": [],
- "output_sha256": "8bee0131e17fdbe87e48457130ac6aeaa55ea30c919a45e77109852609f33eba",
+ "score": 75.0,
+ "passed_count": 3,
+ "failed_count": 1,
+ "failed_assertions": [
+ "has-report"
+ ],
+ "output_sha256": "ac2cb822520b3099150ea1df07de59b9c6df382b7251792e0662631a60989d89",
"failure": ""
},
{
"case_id": "output-eval-expectation",
"variant": "baseline",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 24.17,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 3178.95,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 24,
- "output_tokens": 12,
- "total_tokens": 36,
- "estimated": true
+ "input_tokens": 107,
+ "output_tokens": 129,
+ "total_tokens": 236,
+ "estimated": false
},
"score": 0.0,
"passed_count": 0,
@@ -103,183 +115,192 @@
"has-scorecard",
"has-failure-taxonomy"
],
- "output_sha256": "98fa332fb493b5dc14656a9d65a6c0a9c25ff61a14b71b647aad89cbdbd3ab35",
+ "output_sha256": "a128ff4dbd7870e287ac651b6cab6b7887e347d6477e5bd8c6f40cd9479539f0",
"failure": ""
},
{
"case_id": "output-eval-expectation",
"variant": "with_skill",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 23.84,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 4611.78,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 24,
- "output_tokens": 56,
- "total_tokens": 80,
- "estimated": true
+ "input_tokens": 857,
+ "output_tokens": 264,
+ "total_tokens": 1121,
+ "estimated": false
},
- "score": 100.0,
- "passed_count": 4,
- "failed_count": 0,
- "failed_assertions": [],
- "output_sha256": "8efdbfefa6a36603682b7063b1753f9f939b3810c082d29686e297906d9f51c8",
+ "score": 25.0,
+ "passed_count": 1,
+ "failed_count": 3,
+ "failed_assertions": [
+ "has-baseline-comparison",
+ "has-assertion-grading",
+ "has-failure-taxonomy"
+ ],
+ "output_sha256": "4df98d20e2199e56dae2618b6a64e59d5c96484a4f71ae0ebc674b755cf533c3",
"failure": ""
},
{
"case_id": "ir-before-packaging",
"variant": "baseline",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 22.51,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 6772.59,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 18,
- "output_tokens": 15,
- "total_tokens": 33,
- "estimated": true
+ "input_tokens": 104,
+ "output_tokens": 661,
+ "total_tokens": 765,
+ "estimated": false
},
- "score": 0.0,
- "passed_count": 0,
- "failed_count": 4,
+ "score": 25.0,
+ "passed_count": 1,
+ "failed_count": 3,
"failed_assertions": [
"has-ir-first",
"has-core-ir-fields",
- "has-targets",
"has-semantic-preservation"
],
- "output_sha256": "07a4aa9ee9e48341d4b84d5cb9b6414901aa2c8841340d67447e187240c6c4c2",
+ "output_sha256": "1696075b4f884ff8b478786b5565aff73ad73235936eb39c2ed615c0746861d2",
"failure": ""
},
{
"case_id": "ir-before-packaging",
"variant": "with_skill",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 26.03,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 17471.19,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 18,
- "output_tokens": 62,
- "total_tokens": 80,
- "estimated": true
+ "input_tokens": 854,
+ "output_tokens": 2132,
+ "total_tokens": 2986,
+ "estimated": false
},
- "score": 100.0,
- "passed_count": 4,
- "failed_count": 0,
- "failed_assertions": [],
- "output_sha256": "707fde419b3d61e921561b1c4a8523e1f7b53b1acd5758a47ce21d9ed711fde0",
- "failure": ""
- },
- {
- "case_id": "near-neighbor-boundary",
- "variant": "baseline",
- "status": "pass",
- "execution_mode": "command",
- "model_executed": false,
- "command_executed": true,
- "duration_ms": 28.48,
- "provider": "local-output-eval-runner",
- "model": "",
- "usage": {
- "input_tokens": 17,
- "output_tokens": 19,
- "total_tokens": 36,
- "estimated": true
- },
- "score": 0.0,
- "passed_count": 0,
+ "score": 25.0,
+ "passed_count": 1,
"failed_count": 3,
+ "failed_assertions": [
+ "has-ir-first",
+ "has-core-ir-fields",
+ "has-semantic-preservation"
+ ],
+ "output_sha256": "4f87633e53f360948d6403126d49ad3dabcb1b47aff596918650719b7840abcf",
+ "failure": ""
+ },
+ {
+ "case_id": "near-neighbor-boundary",
+ "variant": "baseline",
+ "status": "pass",
+ "execution_mode": "model",
+ "model_executed": true,
+ "command_executed": true,
+ "duration_ms": 3091.33,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
+ "usage": {
+ "input_tokens": 105,
+ "output_tokens": 93,
+ "total_tokens": 198,
+ "estimated": false
+ },
+ "score": 33.33,
+ "passed_count": 1,
+ "failed_count": 2,
"failed_assertions": [
"declines-skill",
- "names-near-neighbor",
"requires-reuse-signal"
],
- "output_sha256": "648ebaa6e09486d05f33fa6639dc973b029d115158588fd192efbe15ccaaeac4",
+ "output_sha256": "56c8f48f3b2b9963adad3afdb1b6a2bed6b162067506edee1e0f1acd32668a91",
"failure": ""
},
{
"case_id": "near-neighbor-boundary",
"variant": "with_skill",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 23.17,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 4152.85,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 17,
- "output_tokens": 48,
- "total_tokens": 65,
- "estimated": true
+ "input_tokens": 855,
+ "output_tokens": 160,
+ "total_tokens": 1015,
+ "estimated": false
},
- "score": 100.0,
- "passed_count": 3,
- "failed_count": 0,
- "failed_assertions": [],
- "output_sha256": "17a3a5286ad573dea86e51f82c04b596babaecb92cde8879f09e7c519205e2e1",
+ "score": 33.33,
+ "passed_count": 1,
+ "failed_count": 2,
+ "failed_assertions": [
+ "declines-skill",
+ "requires-reuse-signal"
+ ],
+ "output_sha256": "d2d85a2aae5db5c4c7e6e0db5bdd865157a90018afe302b0a703940cc84ba68c",
"failure": ""
},
{
"case_id": "file-backed-governed-package",
"variant": "baseline",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 23.94,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 4750.01,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 17,
- "output_tokens": 20,
- "total_tokens": 37,
- "estimated": true
+ "input_tokens": 200,
+ "output_tokens": 302,
+ "total_tokens": 502,
+ "estimated": false
},
- "score": 0.0,
- "passed_count": 0,
- "failed_count": 5,
+ "score": 60.0,
+ "passed_count": 3,
+ "failed_count": 2,
"failed_assertions": [
"uses-file-backed-evidence",
- "has-governance",
- "has-output-contract",
- "has-trust-and-scorecard",
- "does-not-invent-evidence"
+ "has-trust-and-scorecard"
],
- "output_sha256": "6e5827219459dee34143e4f11663415db80c3111291f75fb6c2115c7bda4cf30",
+ "output_sha256": "d41880b0c2f7e60ee6acb2cb1f4acb9377fd63de72b8e026e6259a962b89af0b",
"failure": ""
},
{
"case_id": "file-backed-governed-package",
"variant": "with_skill",
"status": "pass",
- "execution_mode": "command",
- "model_executed": false,
+ "execution_mode": "model",
+ "model_executed": true,
"command_executed": true,
- "duration_ms": 22.69,
- "provider": "local-output-eval-runner",
- "model": "",
+ "duration_ms": 9406.25,
+ "provider": "deepseek",
+ "model": "deepseek-v4-flash",
"usage": {
- "input_tokens": 17,
- "output_tokens": 81,
- "total_tokens": 98,
- "estimated": true
+ "input_tokens": 950,
+ "output_tokens": 660,
+ "total_tokens": 1610,
+ "estimated": false
},
- "score": 100.0,
- "passed_count": 5,
- "failed_count": 0,
- "failed_assertions": [],
- "output_sha256": "bcfdf036666d406abbc945fc42b93eb74d7074118893e540253c7d8715baad73",
+ "score": 60.0,
+ "passed_count": 3,
+ "failed_count": 2,
+ "failed_assertions": [
+ "has-governance",
+ "has-output-contract"
+ ],
+ "output_sha256": "c670ae2ef81a819a9a0f1b0c046e82ba4c649f56dcbe96a45a6938c7f214fba4",
"failure": ""
}
],
diff --git a/reports/output_execution_runs.md b/reports/output_execution_runs.md
index d2e3be45..576e77c2 100644
--- a/reports/output_execution_runs.md
+++ b/reports/output_execution_runs.md
@@ -5,37 +5,33 @@ This report records how output-eval variants were produced and whether timing or
- Cases: `5`
- Variant runs: `10`
- Command executed: `10`
-- Model executed: `0`
+- Model executed: `10`
- Recorded fixtures: `0`
- Timing observed: `10`
-- Token observed: `0`
-- Token estimated: `10`
-- Delta: `100.0`
+- Token observed: `10`
+- Token estimated: `0`
+- Delta: `20.0`
- Gate pass: `True`
-No model-executed runs are recorded yet.
-
-Use `--runner-command` with a provider-backed runner to replace recorded fixtures with real model output evidence.
-
Command runner evidence is present. This proves the eval harness executed an external command, but it is not provider-backed model evidence unless the runner reports model metadata.
## Runs
| Case | Variant | Mode | Model | Duration ms | Tokens | Score | Status |
| --- | --- | --- | --- | ---: | ---: | ---: | --- |
-| skill-package-contract | baseline | command | local-output-eval-runner | 23.11 | 33 | 0.0 | pass |
-| skill-package-contract | with_skill | command | local-output-eval-runner | 23.56 | 73 | 100.0 | pass |
-| output-eval-expectation | baseline | command | local-output-eval-runner | 24.17 | 36 | 0.0 | pass |
-| output-eval-expectation | with_skill | command | local-output-eval-runner | 23.84 | 80 | 100.0 | pass |
-| ir-before-packaging | baseline | command | local-output-eval-runner | 22.51 | 33 | 0.0 | pass |
-| ir-before-packaging | with_skill | command | local-output-eval-runner | 26.03 | 80 | 100.0 | pass |
-| near-neighbor-boundary | baseline | command | local-output-eval-runner | 28.48 | 36 | 0.0 | pass |
-| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 23.17 | 65 | 100.0 | pass |
-| file-backed-governed-package | baseline | command | local-output-eval-runner | 23.94 | 37 | 0.0 | pass |
-| file-backed-governed-package | with_skill | command | local-output-eval-runner | 22.69 | 98 | 100.0 | pass |
+| skill-package-contract | baseline | model | deepseek-v4-flash | 5980.01 | 484 | 0.0 | pass |
+| skill-package-contract | with_skill | model | deepseek-v4-flash | 11217.19 | 1848 | 75.0 | pass |
+| output-eval-expectation | baseline | model | deepseek-v4-flash | 3178.95 | 236 | 0.0 | pass |
+| output-eval-expectation | with_skill | model | deepseek-v4-flash | 4611.78 | 1121 | 25.0 | pass |
+| ir-before-packaging | baseline | model | deepseek-v4-flash | 6772.59 | 765 | 25.0 | pass |
+| ir-before-packaging | with_skill | model | deepseek-v4-flash | 17471.19 | 2986 | 25.0 | pass |
+| near-neighbor-boundary | baseline | model | deepseek-v4-flash | 3091.33 | 198 | 33.33 | pass |
+| near-neighbor-boundary | with_skill | model | deepseek-v4-flash | 4152.85 | 1015 | 33.33 | pass |
+| file-backed-governed-package | baseline | model | deepseek-v4-flash | 4750.01 | 502 | 60.0 | pass |
+| file-backed-governed-package | with_skill | model | deepseek-v4-flash | 9406.25 | 1610 | 60.0 | pass |
## Next Fixes
- Keep recorded fixtures as reproducible baselines, but do not describe them as model-executed evidence.
-- Add provider-backed command runners for holdout cases when release confidence depends on real generation behavior.
+- Use `scripts/provider_output_eval_runner.py` for provider-backed holdout cases when release confidence depends on real generation behavior.
- Compare timing, token cost, and assertion deltas before promoting a skill to governed reuse.
diff --git a/reports/output_review_adjudication.json b/reports/output_review_adjudication.json
index 0d58d328..c8ebdbe5 100644
--- a/reports/output_review_adjudication.json
+++ b/reports/output_review_adjudication.json
@@ -8,12 +8,50 @@
"agreement_count": 0,
"disagreement_count": 0,
"invalid_decision_count": 0,
+ "answer_revealed_count": 0,
+ "pending_answer_hidden_count": 5,
"agreement_rate": null,
"needs_review": true,
- "failure_count": 0
+ "reviewer_metadata_present": false,
+ "reason_required": true,
+ "blind_review_attested": false,
+ "blind_review_completed_before_answer_key": false,
+ "answer_key_not_opened_before_decisions": false,
+ "raw_content_excluded_attested": true,
+ "reviewer_reason_required_attested": true,
+ "ready_for_human_evidence": false,
+ "failure_count": 0,
+ "reviewer_checklist_count": 5,
+ "reviewer_checklist_pending_count": 5,
+ "reviewer_checklist_invalid_count": 0,
+ "reviewer_checklist_ready_count": 0
},
"reviewer": "",
"reviewed_at": "",
+ "review_integrity": {
+ "blind_pack_sha256": "788604c67fa31178c9bba06a52731adcbee9c1e72c00767cf7950643812a6e5c",
+ "case_count": 5,
+ "case_ids": [
+ "skill-package-contract",
+ "output-eval-expectation",
+ "ir-before-packaging",
+ "near-neighbor-boundary",
+ "file-backed-governed-package"
+ ],
+ "prompt_sha256_by_case": {
+ "skill-package-contract": "95405d805bd516d0998b69d31ece2c1a465b0aa99f04d603628a3939028133b5",
+ "output-eval-expectation": "3829e97814ca269ad8e400ec984c56ddcd54b79448d3846c15e7f5ff44c2edc4",
+ "ir-before-packaging": "d00333fa41f2e28dc7505f25f259aefb82e65b2df1abda2822686c9a6f07bb80",
+ "near-neighbor-boundary": "2d0585e9fa6d03e28786f5e8f0c090781eb209afd344d25e2f91203c49a656d0",
+ "file-backed-governed-package": "c79a554b573517682da48879c1ef54399841d9ebf570088a2174dc38de39cc7d"
+ }
+ },
+ "reviewer_attestation": {
+ "blind_review_completed_before_answer_key": false,
+ "answer_key_not_opened_before_decisions": false,
+ "raw_content_excluded": true,
+ "reviewer_reason_required": true
+ },
"artifacts": {
"blind_pack": "reports/output_blind_review_pack.json",
"answer_key": "reports/output_blind_answer_key.json",
@@ -26,47 +64,214 @@
{
"case_id": "skill-package-contract",
"status": "pending",
- "expected_winner_variant": "A",
+ "expected_winner_variant": "",
+ "expected_revealed": false,
"reviewer_winner_variant": "",
"confidence": null,
"reason": "",
- "prompt": "Turn this repeated workflow into a reusable team skill package."
+ "prompt_sha256": "95405d805bd516d0998b69d31ece2c1a465b0aa99f04d603628a3939028133b5"
},
{
"case_id": "output-eval-expectation",
"status": "pending",
- "expected_winner_variant": "A",
+ "expected_winner_variant": "",
+ "expected_revealed": false,
"reviewer_winner_variant": "",
"confidence": null,
"reason": "",
- "prompt": "Upgrade this production skill so we know whether the generated output is better than baseline."
+ "prompt_sha256": "3829e97814ca269ad8e400ec984c56ddcd54b79448d3846c15e7f5ff44c2edc4"
},
{
"case_id": "ir-before-packaging",
"status": "pending",
- "expected_winner_variant": "B",
+ "expected_winner_variant": "",
+ "expected_revealed": false,
"reviewer_winner_variant": "",
"confidence": null,
"reason": "",
- "prompt": "Package this skill for OpenAI, Claude, Agent Skills, and generic targets."
+ "prompt_sha256": "d00333fa41f2e28dc7505f25f259aefb82e65b2df1abda2822686c9a6f07bb80"
},
{
"case_id": "near-neighbor-boundary",
"status": "pending",
- "expected_winner_variant": "B",
+ "expected_winner_variant": "",
+ "expected_revealed": false,
"reviewer_winner_variant": "",
"confidence": null,
"reason": "",
- "prompt": "I only need a one-off summary of these notes, not a reusable process."
+ "prompt_sha256": "2d0585e9fa6d03e28786f5e8f0c090781eb209afd344d25e2f91203c49a656d0"
},
{
"case_id": "file-backed-governed-package",
"status": "pending",
- "expected_winner_variant": "B",
+ "expected_winner_variant": "",
+ "expected_revealed": false,
"reviewer_winner_variant": "",
"confidence": null,
"reason": "",
- "prompt": "Turn the attached release brief source into a governed skill package."
+ "prompt_sha256": "c79a554b573517682da48879c1ef54399841d9ebf570088a2174dc38de39cc7d"
+ }
+ ],
+ "reviewer_checklist": [
+ {
+ "case_id": "skill-package-contract",
+ "readiness": "awaiting-decision",
+ "blocking_reason": "Reviewer has not selected A or B yet; answer key remains hidden.",
+ "status": "pending",
+ "reviewer_winner_variant": "",
+ "answer_key_visible": false,
+ "prompt_sha256": "95405d805bd516d0998b69d31ece2c1a465b0aa99f04d603628a3939028133b5",
+ "blind_pack_path": "reports/output_blind_review_pack.json",
+ "decisions_path": "reports/output_review_decisions.json",
+ "commands": {
+ "prepare_review_kit": "python3 scripts/yao.py output-review-kit",
+ "write_template": "python3 scripts/adjudicate_output_review.py --write-template",
+ "import_decisions": "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "adjudicate": "python3 scripts/yao.py output-review",
+ "refresh_review_studio": "python3 scripts/yao.py review-studio ."
+ },
+ "required_fields": {
+ "winner_variant": "A or B after reading only the blind review pack.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required rationale; do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer": "Human reviewer name or review group at the decision-file top level.",
+ "reviewed_at": "Review date or timestamp at the decision-file top level.",
+ "reviewer_attestation.blind_review_completed_before_answer_key": "True only after the reviewer has completed choices before opening the answer key.",
+ "reviewer_attestation.answer_key_not_opened_before_decisions": "True only when the answer key was not opened before decisions were recorded."
+ },
+ "privacy_contract": [
+ "Do not paste raw private user data into the decision reason.",
+ "Do not open the answer key before reviewer choices are recorded.",
+ "Leave winner_variant blank when the reviewer is not ready to decide."
+ ]
+ },
+ {
+ "case_id": "output-eval-expectation",
+ "readiness": "awaiting-decision",
+ "blocking_reason": "Reviewer has not selected A or B yet; answer key remains hidden.",
+ "status": "pending",
+ "reviewer_winner_variant": "",
+ "answer_key_visible": false,
+ "prompt_sha256": "3829e97814ca269ad8e400ec984c56ddcd54b79448d3846c15e7f5ff44c2edc4",
+ "blind_pack_path": "reports/output_blind_review_pack.json",
+ "decisions_path": "reports/output_review_decisions.json",
+ "commands": {
+ "prepare_review_kit": "python3 scripts/yao.py output-review-kit",
+ "write_template": "python3 scripts/adjudicate_output_review.py --write-template",
+ "import_decisions": "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "adjudicate": "python3 scripts/yao.py output-review",
+ "refresh_review_studio": "python3 scripts/yao.py review-studio ."
+ },
+ "required_fields": {
+ "winner_variant": "A or B after reading only the blind review pack.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required rationale; do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer": "Human reviewer name or review group at the decision-file top level.",
+ "reviewed_at": "Review date or timestamp at the decision-file top level.",
+ "reviewer_attestation.blind_review_completed_before_answer_key": "True only after the reviewer has completed choices before opening the answer key.",
+ "reviewer_attestation.answer_key_not_opened_before_decisions": "True only when the answer key was not opened before decisions were recorded."
+ },
+ "privacy_contract": [
+ "Do not paste raw private user data into the decision reason.",
+ "Do not open the answer key before reviewer choices are recorded.",
+ "Leave winner_variant blank when the reviewer is not ready to decide."
+ ]
+ },
+ {
+ "case_id": "ir-before-packaging",
+ "readiness": "awaiting-decision",
+ "blocking_reason": "Reviewer has not selected A or B yet; answer key remains hidden.",
+ "status": "pending",
+ "reviewer_winner_variant": "",
+ "answer_key_visible": false,
+ "prompt_sha256": "d00333fa41f2e28dc7505f25f259aefb82e65b2df1abda2822686c9a6f07bb80",
+ "blind_pack_path": "reports/output_blind_review_pack.json",
+ "decisions_path": "reports/output_review_decisions.json",
+ "commands": {
+ "prepare_review_kit": "python3 scripts/yao.py output-review-kit",
+ "write_template": "python3 scripts/adjudicate_output_review.py --write-template",
+ "import_decisions": "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "adjudicate": "python3 scripts/yao.py output-review",
+ "refresh_review_studio": "python3 scripts/yao.py review-studio ."
+ },
+ "required_fields": {
+ "winner_variant": "A or B after reading only the blind review pack.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required rationale; do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer": "Human reviewer name or review group at the decision-file top level.",
+ "reviewed_at": "Review date or timestamp at the decision-file top level.",
+ "reviewer_attestation.blind_review_completed_before_answer_key": "True only after the reviewer has completed choices before opening the answer key.",
+ "reviewer_attestation.answer_key_not_opened_before_decisions": "True only when the answer key was not opened before decisions were recorded."
+ },
+ "privacy_contract": [
+ "Do not paste raw private user data into the decision reason.",
+ "Do not open the answer key before reviewer choices are recorded.",
+ "Leave winner_variant blank when the reviewer is not ready to decide."
+ ]
+ },
+ {
+ "case_id": "near-neighbor-boundary",
+ "readiness": "awaiting-decision",
+ "blocking_reason": "Reviewer has not selected A or B yet; answer key remains hidden.",
+ "status": "pending",
+ "reviewer_winner_variant": "",
+ "answer_key_visible": false,
+ "prompt_sha256": "2d0585e9fa6d03e28786f5e8f0c090781eb209afd344d25e2f91203c49a656d0",
+ "blind_pack_path": "reports/output_blind_review_pack.json",
+ "decisions_path": "reports/output_review_decisions.json",
+ "commands": {
+ "prepare_review_kit": "python3 scripts/yao.py output-review-kit",
+ "write_template": "python3 scripts/adjudicate_output_review.py --write-template",
+ "import_decisions": "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "adjudicate": "python3 scripts/yao.py output-review",
+ "refresh_review_studio": "python3 scripts/yao.py review-studio ."
+ },
+ "required_fields": {
+ "winner_variant": "A or B after reading only the blind review pack.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required rationale; do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer": "Human reviewer name or review group at the decision-file top level.",
+ "reviewed_at": "Review date or timestamp at the decision-file top level.",
+ "reviewer_attestation.blind_review_completed_before_answer_key": "True only after the reviewer has completed choices before opening the answer key.",
+ "reviewer_attestation.answer_key_not_opened_before_decisions": "True only when the answer key was not opened before decisions were recorded."
+ },
+ "privacy_contract": [
+ "Do not paste raw private user data into the decision reason.",
+ "Do not open the answer key before reviewer choices are recorded.",
+ "Leave winner_variant blank when the reviewer is not ready to decide."
+ ]
+ },
+ {
+ "case_id": "file-backed-governed-package",
+ "readiness": "awaiting-decision",
+ "blocking_reason": "Reviewer has not selected A or B yet; answer key remains hidden.",
+ "status": "pending",
+ "reviewer_winner_variant": "",
+ "answer_key_visible": false,
+ "prompt_sha256": "c79a554b573517682da48879c1ef54399841d9ebf570088a2174dc38de39cc7d",
+ "blind_pack_path": "reports/output_blind_review_pack.json",
+ "decisions_path": "reports/output_review_decisions.json",
+ "commands": {
+ "prepare_review_kit": "python3 scripts/yao.py output-review-kit",
+ "write_template": "python3 scripts/adjudicate_output_review.py --write-template",
+ "import_decisions": "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "adjudicate": "python3 scripts/yao.py output-review",
+ "refresh_review_studio": "python3 scripts/yao.py review-studio ."
+ },
+ "required_fields": {
+ "winner_variant": "A or B after reading only the blind review pack.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required rationale; do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer": "Human reviewer name or review group at the decision-file top level.",
+ "reviewed_at": "Review date or timestamp at the decision-file top level.",
+ "reviewer_attestation.blind_review_completed_before_answer_key": "True only after the reviewer has completed choices before opening the answer key.",
+ "reviewer_attestation.answer_key_not_opened_before_decisions": "True only when the answer key was not opened before decisions were recorded."
+ },
+ "privacy_contract": [
+ "Do not paste raw private user data into the decision reason.",
+ "Do not open the answer key before reviewer choices are recorded.",
+ "Leave winner_variant blank when the reviewer is not ready to decide."
+ ]
}
],
"failures": []
diff --git a/reports/output_review_adjudication.md b/reports/output_review_adjudication.md
index 6c12de42..9b757c8d 100644
--- a/reports/output_review_adjudication.md
+++ b/reports/output_review_adjudication.md
@@ -7,20 +7,198 @@ This report adjudicates reviewer choices from the blind A/B output review pack a
- Pending: `5`
- Agreement rate: `n/a`
- Invalid decisions: `0`
+- Answer keys revealed: `0`
+- Pending/invalid answers hidden: `5`
+- Reviewer checklist: `0` ready / `5` total
+- Reviewer metadata present: `false`
+- Blind review attested: `false`
+- Raw content excluded: `true`
+- Ready for human evidence: `false`
No reviewer decisions recorded yet.
Generate a template with `--write-template`, fill `winner_variant` with `A` or `B`, then rerun adjudication.
+Expected winners stay hidden until a valid reviewer decision is recorded.
## Case Adjudication
| Case | Reviewer | Expected | Status | Confidence | Reason |
| --- | --- | --- | --- | ---: | --- |
-| skill-package-contract | pending | A | pending | | |
-| output-eval-expectation | pending | A | pending | | |
-| ir-before-packaging | pending | B | pending | | |
-| near-neighbor-boundary | pending | B | pending | | |
-| file-backed-governed-package | pending | B | pending | | |
+| skill-package-contract | pending | hidden | pending | | |
+| output-eval-expectation | pending | hidden | pending | | |
+| ir-before-packaging | pending | hidden | pending | | |
+| near-neighbor-boundary | pending | hidden | pending | | |
+| file-backed-governed-package | pending | hidden | pending | | |
+
+## Reviewer Checklist
+
+| Case | Readiness | Answer key | Decision file |
+| --- | --- | --- | --- |
+| `skill-package-contract` | `awaiting-decision` | `hidden` | `reports/output_review_decisions.json` |
+| `output-eval-expectation` | `awaiting-decision` | `hidden` | `reports/output_review_decisions.json` |
+| `ir-before-packaging` | `awaiting-decision` | `hidden` | `reports/output_review_decisions.json` |
+| `near-neighbor-boundary` | `awaiting-decision` | `hidden` | `reports/output_review_decisions.json` |
+| `file-backed-governed-package` | `awaiting-decision` | `hidden` | `reports/output_review_decisions.json` |
+
+### skill-package-contract
+
+- readiness: `awaiting-decision`
+- blocking reason: Reviewer has not selected A or B yet; answer key remains hidden.
+- answer key visible: `false`
+- blind pack: `reports/output_blind_review_pack.json`
+- decisions: `reports/output_review_decisions.json`
+
+#### Commands
+
+- prepare_review_kit: `python3 scripts/yao.py output-review-kit`
+- write_template: `python3 scripts/adjudicate_output_review.py --write-template`
+- import_decisions: `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`
+- adjudicate: `python3 scripts/yao.py output-review`
+- refresh_review_studio: `python3 scripts/yao.py review-studio .`
+
+#### Required Fields
+
+- winner_variant: A or B after reading only the blind review pack.
+- confidence: Optional number from 0 to 1.
+- reason: Required rationale; do not reveal baseline or with-skill labels before adjudication.
+- reviewer: Human reviewer name or review group at the decision-file top level.
+- reviewed_at: Review date or timestamp at the decision-file top level.
+- reviewer_attestation.blind_review_completed_before_answer_key: True only after the reviewer has completed choices before opening the answer key.
+- reviewer_attestation.answer_key_not_opened_before_decisions: True only when the answer key was not opened before decisions were recorded.
+
+#### Privacy Contract
+
+- Do not paste raw private user data into the decision reason.
+- Do not open the answer key before reviewer choices are recorded.
+- Leave winner_variant blank when the reviewer is not ready to decide.
+
+### output-eval-expectation
+
+- readiness: `awaiting-decision`
+- blocking reason: Reviewer has not selected A or B yet; answer key remains hidden.
+- answer key visible: `false`
+- blind pack: `reports/output_blind_review_pack.json`
+- decisions: `reports/output_review_decisions.json`
+
+#### Commands
+
+- prepare_review_kit: `python3 scripts/yao.py output-review-kit`
+- write_template: `python3 scripts/adjudicate_output_review.py --write-template`
+- import_decisions: `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`
+- adjudicate: `python3 scripts/yao.py output-review`
+- refresh_review_studio: `python3 scripts/yao.py review-studio .`
+
+#### Required Fields
+
+- winner_variant: A or B after reading only the blind review pack.
+- confidence: Optional number from 0 to 1.
+- reason: Required rationale; do not reveal baseline or with-skill labels before adjudication.
+- reviewer: Human reviewer name or review group at the decision-file top level.
+- reviewed_at: Review date or timestamp at the decision-file top level.
+- reviewer_attestation.blind_review_completed_before_answer_key: True only after the reviewer has completed choices before opening the answer key.
+- reviewer_attestation.answer_key_not_opened_before_decisions: True only when the answer key was not opened before decisions were recorded.
+
+#### Privacy Contract
+
+- Do not paste raw private user data into the decision reason.
+- Do not open the answer key before reviewer choices are recorded.
+- Leave winner_variant blank when the reviewer is not ready to decide.
+
+### ir-before-packaging
+
+- readiness: `awaiting-decision`
+- blocking reason: Reviewer has not selected A or B yet; answer key remains hidden.
+- answer key visible: `false`
+- blind pack: `reports/output_blind_review_pack.json`
+- decisions: `reports/output_review_decisions.json`
+
+#### Commands
+
+- prepare_review_kit: `python3 scripts/yao.py output-review-kit`
+- write_template: `python3 scripts/adjudicate_output_review.py --write-template`
+- import_decisions: `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`
+- adjudicate: `python3 scripts/yao.py output-review`
+- refresh_review_studio: `python3 scripts/yao.py review-studio .`
+
+#### Required Fields
+
+- winner_variant: A or B after reading only the blind review pack.
+- confidence: Optional number from 0 to 1.
+- reason: Required rationale; do not reveal baseline or with-skill labels before adjudication.
+- reviewer: Human reviewer name or review group at the decision-file top level.
+- reviewed_at: Review date or timestamp at the decision-file top level.
+- reviewer_attestation.blind_review_completed_before_answer_key: True only after the reviewer has completed choices before opening the answer key.
+- reviewer_attestation.answer_key_not_opened_before_decisions: True only when the answer key was not opened before decisions were recorded.
+
+#### Privacy Contract
+
+- Do not paste raw private user data into the decision reason.
+- Do not open the answer key before reviewer choices are recorded.
+- Leave winner_variant blank when the reviewer is not ready to decide.
+
+### near-neighbor-boundary
+
+- readiness: `awaiting-decision`
+- blocking reason: Reviewer has not selected A or B yet; answer key remains hidden.
+- answer key visible: `false`
+- blind pack: `reports/output_blind_review_pack.json`
+- decisions: `reports/output_review_decisions.json`
+
+#### Commands
+
+- prepare_review_kit: `python3 scripts/yao.py output-review-kit`
+- write_template: `python3 scripts/adjudicate_output_review.py --write-template`
+- import_decisions: `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`
+- adjudicate: `python3 scripts/yao.py output-review`
+- refresh_review_studio: `python3 scripts/yao.py review-studio .`
+
+#### Required Fields
+
+- winner_variant: A or B after reading only the blind review pack.
+- confidence: Optional number from 0 to 1.
+- reason: Required rationale; do not reveal baseline or with-skill labels before adjudication.
+- reviewer: Human reviewer name or review group at the decision-file top level.
+- reviewed_at: Review date or timestamp at the decision-file top level.
+- reviewer_attestation.blind_review_completed_before_answer_key: True only after the reviewer has completed choices before opening the answer key.
+- reviewer_attestation.answer_key_not_opened_before_decisions: True only when the answer key was not opened before decisions were recorded.
+
+#### Privacy Contract
+
+- Do not paste raw private user data into the decision reason.
+- Do not open the answer key before reviewer choices are recorded.
+- Leave winner_variant blank when the reviewer is not ready to decide.
+
+### file-backed-governed-package
+
+- readiness: `awaiting-decision`
+- blocking reason: Reviewer has not selected A or B yet; answer key remains hidden.
+- answer key visible: `false`
+- blind pack: `reports/output_blind_review_pack.json`
+- decisions: `reports/output_review_decisions.json`
+
+#### Commands
+
+- prepare_review_kit: `python3 scripts/yao.py output-review-kit`
+- write_template: `python3 scripts/adjudicate_output_review.py --write-template`
+- import_decisions: `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`
+- adjudicate: `python3 scripts/yao.py output-review`
+- refresh_review_studio: `python3 scripts/yao.py review-studio .`
+
+#### Required Fields
+
+- winner_variant: A or B after reading only the blind review pack.
+- confidence: Optional number from 0 to 1.
+- reason: Required rationale; do not reveal baseline or with-skill labels before adjudication.
+- reviewer: Human reviewer name or review group at the decision-file top level.
+- reviewed_at: Review date or timestamp at the decision-file top level.
+- reviewer_attestation.blind_review_completed_before_answer_key: True only after the reviewer has completed choices before opening the answer key.
+- reviewer_attestation.answer_key_not_opened_before_decisions: True only when the answer key was not opened before decisions were recorded.
+
+#### Privacy Contract
+
+- Do not paste raw private user data into the decision reason.
+- Do not open the answer key before reviewer choices are recorded.
+- Leave winner_variant blank when the reviewer is not ready to decide.
## Next Fixes
diff --git a/reports/output_review_decisions.json b/reports/output_review_decisions.json
new file mode 100644
index 00000000..64a178be
--- /dev/null
+++ b/reports/output_review_decisions.json
@@ -0,0 +1,67 @@
+{
+ "schema_version": "1.0",
+ "reviewer": "",
+ "reviewed_at": "",
+ "review_integrity": {
+ "blind_pack_sha256": "788604c67fa31178c9bba06a52731adcbee9c1e72c00767cf7950643812a6e5c",
+ "case_count": 5,
+ "case_ids": [
+ "skill-package-contract",
+ "output-eval-expectation",
+ "ir-before-packaging",
+ "near-neighbor-boundary",
+ "file-backed-governed-package"
+ ],
+ "prompt_sha256_by_case": {
+ "skill-package-contract": "95405d805bd516d0998b69d31ece2c1a465b0aa99f04d603628a3939028133b5",
+ "output-eval-expectation": "3829e97814ca269ad8e400ec984c56ddcd54b79448d3846c15e7f5ff44c2edc4",
+ "ir-before-packaging": "d00333fa41f2e28dc7505f25f259aefb82e65b2df1abda2822686c9a6f07bb80",
+ "near-neighbor-boundary": "2d0585e9fa6d03e28786f5e8f0c090781eb209afd344d25e2f91203c49a656d0",
+ "file-backed-governed-package": "c79a554b573517682da48879c1ef54399841d9ebf570088a2174dc38de39cc7d"
+ }
+ },
+ "reviewer_attestation": {
+ "blind_review_completed_before_answer_key": false,
+ "answer_key_not_opened_before_decisions": false,
+ "raw_content_excluded": true,
+ "reviewer_reason_required": true
+ },
+ "decision_contract": {
+ "winner_variant": "Use A or B after reading the blind review pack. Leave blank when pending.",
+ "confidence": "Optional number from 0 to 1.",
+ "reason": "Required reviewer rationale. Do not reveal baseline or with-skill labels before adjudication.",
+ "reviewer_attestation": "Set blind_review_completed_before_answer_key and answer_key_not_opened_before_decisions to true only after a real blind review."
+ },
+ "decisions": [
+ {
+ "case_id": "skill-package-contract",
+ "winner_variant": "",
+ "confidence": null,
+ "reason": ""
+ },
+ {
+ "case_id": "output-eval-expectation",
+ "winner_variant": "",
+ "confidence": null,
+ "reason": ""
+ },
+ {
+ "case_id": "ir-before-packaging",
+ "winner_variant": "",
+ "confidence": null,
+ "reason": ""
+ },
+ {
+ "case_id": "near-neighbor-boundary",
+ "winner_variant": "",
+ "confidence": null,
+ "reason": ""
+ },
+ {
+ "case_id": "file-backed-governed-package",
+ "winner_variant": "",
+ "confidence": null,
+ "reason": ""
+ }
+ ]
+}
diff --git a/reports/output_review_kit.html b/reports/output_review_kit.html
new file mode 100644
index 00000000..ae9348f3
--- /dev/null
+++ b/reports/output_review_kit.html
@@ -0,0 +1,220 @@
+
+
+
+
+
+ Output Review Kit
+
+
+
+
+
+
+ Blind A/B Human Review
+
Reviewer cockpit for output quality decisions
+
Compare visible Variant A and Variant B outputs, fill the decision file, then run adjudication. The answer key is intentionally hidden from this page.
+
Cases5
Ready0
Pending5
Invalid0
+
+
+
Review Flow
Open reports/output_blind_review_pack.md or this kit and compare Variant A vs Variant B for each case.
Record choices in reports/output_review_decisions.json without opening the answer key.
Use winner_variant A or B, confidence from 0 to 1, and a short reason for every case.
Set reviewer_attestation truthfully after choices are recorded and before adjudication.
Run python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication after choices are recorded.
Refresh python3 scripts/yao.py review-studio . before asking for release approval.
+
+
+
+
+
+
+ Case 01
+
skill-package-contract
+
+ Awaiting
+
+
Turn this repeated workflow into a reusable team skill package.
+
Rubric
has-entrypoint
Output names the routeable Skill entrypoint.
1.0
has-interface
Output includes neutral interface metadata.
1.0
has-report
Output points reviewer to generated overview report.
1.0
has-resource-boundary
Output preserves entrypoint/resource separation.
1.0
+
+
Variant A
Create a routeable SKILL.md, agents/interface.yaml, reports/skill-overview.html, and a focused eval plan. Keep the root entrypoint lean, move durable guidance into references/, scripts into scripts/, and evidence into reports/.
+
Variant B
I can write a prompt for that workflow and include a short checklist.
+
+
+
+
+
+
+
+ Case 02
+
output-eval-expectation
+
+ Awaiting
+
+
Upgrade this production skill so we know whether the generated output is better than baseline.
+
Rubric
has-baseline-comparison
Output explicitly compares with-skill and baseline outputs.
1.0
has-assertion-grading
Output includes objective assertion grading.
1.0
has-scorecard
Output produces a scorecard report path.
1.0
has-failure-taxonomy
Output records failure taxonomy.
1.0
+
+
Variant A
Add Output Eval Lab cases with baseline_output and with_skill_output, run assertion grading, report with-skill vs baseline pass-rate delta, and record failure taxonomy plus next fixes in reports/output_quality_scorecard.md.
+
Variant B
Add more examples and run the trigger eval again.
+
+
+
+
+
+
+
+ Case 03
+
ir-before-packaging
+
+ Awaiting
+
+
Package this skill for OpenAI, Claude, Agent Skills, and generic targets.
+
Rubric
has-ir-first
Output requires Skill IR before packaging.
1.0
has-core-ir-fields
Output lists core IR contract fields.
1.0
has-targets
Output names requested runtime targets.
1.0
has-semantic-preservation
Output says targets must preserve the capability contract.
1.0
+
+
Variant A
Export folders for each platform and adjust files as needed.
+
Variant B
Export Skill IR first with job_to_be_done, trigger_surface, workflow, resources, eval_plan, risk, and governance. Then compile or package targets from the IR so OpenAI, Claude, Agent Skills, and generic packages preserve the same capability contract.
+
+
+
+
+
+
+
+ Case 04
+
near-neighbor-boundary
+
+ Awaiting
+
+
I only need a one-off summary of these notes, not a reusable process.
+
Rubric
declines-skill
Output refuses unnecessary skill creation for one-off work.
1.0
names-near-neighbor
Output labels the request as near-neighbor instead of owned work.
1.0
requires-reuse-signal
Output asks for repeat-use evidence before packaging.
1.0
+
+
Variant A
Create a SKILL.md and a reusable workflow anyway so future notes can use it.
+
Variant B
Do not create a skill for this one-off request. Treat it as a near-neighbor: answer the summary directly unless the user confirms repeated use, shared ownership, or a reusable output contract.
+
+
+
+
+
+
+
+ Case 05
+
file-backed-governed-package
+
+ Awaiting
+
+
Turn the attached release brief source into a governed skill package.
+
Rubric
uses-file-backed-evidence
Output names file-backed source evidence.
1.0
has-governance
Output preserves governed ownership metadata.
1.0
has-output-contract
Output preserves output and rollback boundaries.
1.0
has-trust-and-scorecard
Output requires trust and output scorecard artifacts.
1.0
does-not-invent-evidence
Output forbids invented launch evidence.
1.0
+
+
Variant A
Draft a release announcement with the changelog, support notes, and owner name.
+
Variant B
Use the file-backed fixture as source evidence, then create a governed skill package with SKILL.md, agents/interface.yaml, owner, review cadence, input_files, output contract, rollback boundary, trust report, and reports/output_quality_scorecard.md. Mark missing launch metrics as missing evidence instead of inventing them.
+
+
+
+
Decision Template
Use this shape in reports/output_review_decisions.json; leave a case blank when the reviewer is not ready.
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
{"id":"skill-package-contract","prompt":"Turn this repeated workflow into a reusable team skill package.","baseline_output":"I can write a prompt for that workflow and include a s…
Run provider-backed holdout cases with real credentials and commit only aggregate evidence.
阻断检查
暂无阻断检查。
阶段队列
#20 · unblock-accessUnblock access1 / 1 blockedowners: operator with provider credentialsevidence: provider-holdoutSet one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
修复清单
#20 · unblock-access · precheckprovider-api-keyoperator with provider credentialsRequired env_any precheck is missing.Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false
submission-ref1 / 1 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
Do not commit provider credentials or environment dumps.
The output execution report records output hashes and aggregate run metadata, not raw provider prompts.
Record real A/B choices, reviewer metadata, and blind-review attestation, then regenerate adjudication.
阻断检查
No pending decisionspending_count: 5 / ==0Record a reviewer choice and reason for every pair.
Judgments completejudgment_count: 0 / ==pair_countEvery pair needs one valid human judgment.
Reviewer metadatareviewer_metadata_present: False / trueRecord reviewer and reviewed_at before adjudication can count.
Blind review attestedblind_review_attested: False / trueSet reviewer_attestation only after choices are completed before opening the answer key.
Human evidence readyready_for_human_evidence: False / trueComplete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.
阶段队列
#20 · unblock-accessUnblock access1 / 1 blockedowners: human reviewerevidence: human-adjudicationAssign a real reviewer identity before claiming human adjudication.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-sourceCollect source5 / 5 blockedowners: human reviewerevidence: human-adjudicationSet reviewer_attestation only after choices are completed before opening the answer key.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
修复清单
#20 · unblock-access · precheckhuman-reviewerhuman reviewerRequired human precheck is human-required.Assign a real reviewer identity before claiming human adjudication.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkblind_review_attestedhuman reviewerCurrent value False does not satisfy 'true'.Set reviewer_attestation only after choices are completed before opening the answer key.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkjudgment_counthuman reviewerCurrent value 0 does not satisfy '==pair_count'.Every pair needs one valid human judgment.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkpending_counthuman reviewerCurrent value 5 does not satisfy '==0'.Record a reviewer choice and reason for every pair.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkready_for_human_evidencehuman reviewerCurrent value False does not satisfy 'true'.Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkreviewer_metadata_presenthuman reviewerCurrent value False does not satisfy 'true'.Record reviewer and reviewed_at before adjudication can count.python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
reports/output_review_adjudication.json summary.reviewer_metadata_present is true
reports/output_review_adjudication.json summary.blind_review_attested is true
reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json
reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text
reports/output_review_adjudication.json summary.ready_for_human_evidence is true
reports/skill_os2_audit.json item human-adjudication status becomes pass
证据资产
reports/output_blind_review_pack.md
reports/output_review_kit.md
reports/output_review_decisions.json
reports/output_review_adjudication.json
reports/output_review_adjudication.md
资产角色
source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false
submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
supporting-evidence8 / 8 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
Reviewer decisions should not include raw user data or private customer detail.
Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.
The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.
The adjudication evidence stores prompt_sha256 instead of raw prompt text.
#20 · unblock-access · prechecknative-guardtarget client or installer integratorRequired external precheck is external-required.Attach a real target-client or external installer runtime guard; metadata fallback is not enough.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checknative_enforcement_counttarget client or installer integratorCurrent value 0 does not satisfy '>0'.Collect real target-client or external runtime guard proof.python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
证据资产
dist/targets/*/adapter.json
reports/runtime_permission_probes.json
reports/runtime_permission_probes.md
reports/install_simulation.json
reports/install_simulation.md
资产角色
source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false
submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
supporting-evidence11 / 11 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
Do not mark native_enforcement true for metadata-only fallbacks.
Keep residual risks visible for targets that still rely on operator enforcement.
#20 · unblock-access · precheckexternal-clientBrowser/Chrome/IDE/provider client integratorRequired external precheck is external-required.Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkadoption_sample_countBrowser/Chrome/IDE/provider client integratorCurrent value 0 does not satisfy '>0'.Telemetry must include adoption outcome evidence.python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
#40 · collect-source · source-checkexternal_source_eventsBrowser/Chrome/IDE/provider client integratorCurrent value 0 does not satisfy '>0'.Import at least one metadata-only event from a real client.python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
reports/skill_os2_audit.json item native-client-telemetry status becomes pass
证据资产
reports/adoption_drift_report.json
reports/adoption_drift_report.md
reports/telemetry_hook_recipes.json
scripts/telemetry_native_host.py
evidence/world_class/intake.schema.json
资产角色
source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false
submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
Telemetry must remain metadata-only and local-first.
Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
No pending decisionspending_count: 5 / ==0Record a reviewer choice and reason for every pair.
Judgments completejudgment_count: 0 / ==pair_countEvery pair needs one valid human judgment.
No invalid decisionsinvalid_decision_count: 0 / ==0Fix malformed winner/confidence entries.
Reviewer metadatareviewer_metadata_present: False / trueRecord reviewer and reviewed_at before adjudication can count.
Reason requiredreason_required: True / trueKeep reason mandatory for every imported or direct reviewer decision.
Blind review attestedblind_review_attested: False / trueSet reviewer_attestation only after choices are completed before opening the answer key.
Raw content attestedraw_content_excluded_attested: True / trueAttest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content.
Raw content blockedraw_content_allowed: False / falseAdjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages.
Human evidence readyready_for_human_evidence: False / trueComplete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.
待补证 · external
Native Permission Enforcement
Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
隐私边界
Do not mark native_enforcement true for metadata-only fallbacks.
Keep residual risks visible for targets that still rely on operator enforcement.
awaiting-submission · external
Native Client Telemetry
No real evidence submission has been provided yet.
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
Logic
Decide whether the request should become a skill and choose the lightest fit.; Capture job, output, exclusions, constraints, and standards.; Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.
Usage
Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.; Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
Next
Review the top iteration directions before growing the package.
+
Inputs
workflow, prompt, transcript, docs, or notes
Boundary
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
Logic
For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.; Capture job, output, exclusions, constraints, standards, and the lightest fit.; Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
Usage
Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.; Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
Next
Review the top iteration directions before growing the package.
Core logic
-
Decide whether the request should become a skill and choose the lightest fit.
Capture job, output, exclusions, constraints, and standards.
Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.
Write the `description` early and test route quality before expanding the package.
Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.
+
For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.
Capture job, output, exclusions, constraints, standards, and the lightest fit.
Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
Write `description` early, test route quality, then add only earned folders and gates.
Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful.
How to use it
-
Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.
Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.
Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
@@ -221,7 +221,7 @@
Why this package is strong
-
触发面保持精简，并锚定在 frontmatter description。
已打包 agents/interface.yaml，便于后续做跨平台适配。
长指导被拆到 references 中，入口文件可以保持轻量。
确定性辅助逻辑放在 scripts 中，而不是藏在提示词里。
包内包含可随 Skill 迁移的质量门禁或触发检查。
+
触发面保持精简，并锚定在 frontmatter description。
已生成 Skill IR，核心语义可先于平台打包被审查和迁移。
已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
tokens 65 (+57)dev 0 (0)holdout 0 (0)
Adds relative to baseline
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.
Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
Drops from baseline
Create and improve agent skills.
Winner — Current
current
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
tokens 65 (+57)dev 0 (0)holdout 0 (0)
Adds relative to baseline
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.
Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
Drops from baseline
Create and improve agent skills.
+
Baseline
existing
Create and improve agent skills.
tokens 8 (0)dev 0 (0)holdout 0 (0)
Adds relative to baseline
Create and improve agent skills.
Drops from baseline
No dropped cues.
Current
current
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
tokens 53 (+45)dev 0 (0)holdout 0 (0)
Adds relative to baseline
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.
Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
Drops from baseline
Create and improve agent skills.
Winner — Current
current
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
tokens 53 (+45)dev 0 (0)holdout 0 (0)
Adds relative to baseline
Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.
Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
Conflict handling · decision needed The stated preference leans lightweight or speed-first, while the benchmark mix leans toward governance, review, or heavier evaluation structure.
Artifact design profile · ready Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
Report or brief High-trust editorial report with a clear first-screen thesis, compact evidence blocks, and decisions separated from supporting detail.
Code, CLI, or implementation guide Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.
+
Design system: metric editorial
+
Dashboard or metrics page Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.
Review viewer Side-by-side reviewer studio with explicit tradeoffs, evidence readiness, and fast paths for approving, blocking, or requesting one focused fix.
Code, CLI, or implementation guide Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.
Visual quality gates
-
Keep the first screen useful without requiring the reader to parse every detail.
Use tables only for comparisons; move explanations below the table.
Keep source notes readable without flooding the body with markers.
Name the working directory and required inputs for commands.
Mark destructive, networked, or external side-effect operations.
+
Avoid paragraph-heavy table cells.
Keep charts tied to one analytical question each.
Preserve stable color meaning across metrics and entities.
Make differences visible instead of hiding them in prose.
Separate author-facing recommendations from reviewer-only evidence.
Completeness · 100 / 100 Name missing inputs, outputs, constraints, or success standards before deepening the package.
Clarity · 85 / 100 Replace broad verbs with observable actions and define what done means.
Consistency · 90 / 100 Check that role, task, format, exclusions, and examples do not contradict each other.
Practicality · 90 / 100 Add runnable steps, examples, or verification cues instead of abstract advice.
Specificity · 80 / 100 Anchor wording in the user's audience, domain nouns, and target outcome.
RTF to skill mapping
-
Role Use a conversational role that asks only high-leverage questions and remembers the user's goal.
Task Clarify intent, resolve uncertainty, and converge toward a recommendation instead of a long option list.
Format Return concise prompts, decision points, and reviewer-visible assumptions.
+
Role Use an operator role with explicit boundaries, inputs, outputs, and failure handling.
Task Convert the job into ordered steps with validation checks and stop conditions.
Format Return a runbook-like handoff with commands, checks, owners, and next actions when relevant.
@@ -310,11 +310,11 @@
Reference synthesis
-
Official workflow product ergonomics
Borrow now
Borrow a first-time operator flow that explains itself before it asks for more structure.
Avoid
Do not mimic product polish that adds UI bulk without improving clarity.
Hypothesis-test-learn loop
Borrow now
Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.
Avoid
Do not create experimental overhead that exceeds the skill's real risk tier.
Boundary-first design
Borrow now
Borrow the discipline of defining what the skill should not own before growing the package.
Avoid
Do not expand execution assets until route boundaries stay clean.
+
Official skill anatomy and context discipline
Borrow now
Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
Avoid
Do not let packaging or platform concerns swallow the core job boundary.
Human-in-the-loop verification
Borrow now
Borrow a review checkpoint wherever trust matters more than raw speed.
Avoid
Do not force every skill through heavyweight review when the risk is low.
Boundary-first design
Borrow now
Borrow the discipline of defining what the skill should not own before growing the package.
Avoid
Do not expand execution assets until route boundaries stay clean.
Borrow now
-
Borrow a first-time operator flow that explains itself before it asks for more structure.
Borrow a small hypothesis-test-learn loop so the first revision is evidence-backed.
Borrow the discipline of defining what the skill should not own before growing the package.
Borrow the way it turns a messy workflow into a repeatable operating path.
+
Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.
Borrow a review checkpoint wherever trust matters more than raw speed.
Borrow the discipline of defining what the skill should not own before growing the package.
Borrow the way it turns a messy workflow into a repeatable operating path.
Use the recommendation by default. Only surface the underlying benchmark tradeoffs when intent is uncertain or a real design conflict needs a deliberate call.
这份报告默认使用中文简体，把新 Skill 的定位、指标、原理、契约、质量、风险、资产和迭代路线整理为一份可审计的 HTML 报告。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
完整度Completeness100SKILL.md 已存在，是 Skill 的入口。SKILL.md exists and acts as the skill entrypoint.触发清晰Trigger clarity100frontmatter description 已存在，具备基础路由面。The frontmatter description exists, giving the skill a basic routing surface.证据充分Evidence depth100已生成 20 / 20 类报告证据。Generated 20 / 20 evidence report types.上下文成本Context cost42入口约 339 个词/字，references 约 16934 个词/字。Entrypoint is about 339 words/characters; references are about 16934.
+
+
+
+
+
+
技能概述Overview
+
先用一屏说明这个 Skill 是什么、给谁用、交付什么。A first-screen explanation of what this skill is, who it serves, and what it delivers.
+
+
+
+
作用定位Role
+
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。Turn one-off experience into a reusable, evaluable, and portable skill package.
+
+ 交付流程把用户输入、生成的包体和可复用能力放在一条线上。The delivery flow places user input, generated package, and reusable capability on one path.
+
+
+
+
+
+
+
+
总览指标Metrics
+
分数来自本地文件和 reports 证据，缺失时明确标为证据不足。Scores are derived from package files and reports; missing inputs are shown as evidence gaps.
+
+
+
指标判读Reading
+
先看雷达图判断能力短板，再看每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.
触发强度：手动触发 + description 路由Trigger strength: Manual activation plus description-based routing
复用范围：跨平台Reuse scope: Cross-platform
+
+
+
+
+
+
+
+
+
原理结构Principle
+
说明入口、参考、脚本、评估和报告如何组成一个稳定闭环。Explains how entrypoint, references, scripts, evals, and reports form a stable loop.
+
+
+
分层结构展示入口、参考、脚本、评估和报告如何各司其职。The layered structure shows how entrypoint, references, scripts, evals, and reports each carry a distinct role.
+
+
+
执行流程Execution Flow
+
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Capture job, output, exclusions, constraints, standards, and the lightest fit.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Write `description` early, test route quality, then add only earned folders and gates.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful.
+
+
+
调用方式How To Use
+
当你需要把工作流或笔记沉淀成结构精简、触发清晰且带必要 eval 的可复用 Skill 时使用 $yao-meta-skill。Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.
当用户请求与该 Skill 的触发描述匹配时使用。Use this skill when the request matches the frontmatter description.
+
+
+
+
+
+
+
+
+
+
契约边界Contract
+
把触发、输入、输出和排除场景放在同一屏。Keeps trigger, inputs, outputs, and exclusions on the same screen.
+
+
+
+
触发描述Trigger
+
该 Skill 的触发描述来自 SKILL.md frontmatter；默认中文报告先呈现能力边界，原始英文描述可切换到英文查看。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。an existing skill directory that needs refactor, evaluation, packaging, or governance hardening
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。benchmark references, local constraints, desired maturity tier, and review standards
+
+
+
输出结果Outputs
+
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.
只需要一次性回答、没有复用价值的临时请求。One-off requests that do not need reusable skill behavior.
要求直接执行相邻任务，而不是沉淀或使用这个 Skill。Requests to perform an adjacent task directly rather than create or use this skill.
缺少必要事实且用户不允许澄清的场景。Cases that lack required facts and do not allow clarification.
+
+
+
+
+
+
+
+
+
质量评估Quality
+
展示强项、缺口和建议，避免只给分不解释。Shows strengths, gaps, and recommendations instead of scores without explanation.
+
+
+
+
类型Type
证据Evidence
建议Action
+
强项Strength
触发面保持精简，并锚定在 frontmatter description。The trigger surface stays lean and anchored in the frontmatter description.
保留并复用Keep
强项Strength
已生成 Skill IR，核心语义可先于平台打包被审查和迁移。Skill IR is generated so core semantics can be reviewed and migrated before platform packaging.
保留并复用Keep
强项Strength
已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。Target compilation evidence is generated to review how IR maps to OpenAI, Claude, generic, and other target contracts.
保留并复用Keep
缺口Gap
上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。Context cost needs improvement: Entrypoint is about 339 words/characters; references are about 16934.
纳入下一轮修复Fix next
+
+
+
+
执行证据Execution Evidence
+
变体运行：10Variant runs: 10
模型执行：10Model executions: 10
记录样本：0Recorded fixtures: 0
Token 估算：0Token estimates: 0
+
+
+
盲评审定Blind Adjudication
+
评审进度：0 / 5Review progress: 0 / 5
待评审：5Pending review: 5
一致率：暂无Agreement rate: not available yet
非法决策：0Invalid decisions: 0
+
+
+
+
+
评审原则Review Rule
+
先记录 reviewer 对 A/B 的选择，再打开答案 key 计算一致率。Record the reviewer's A/B choice before opening the answer key and calculating agreement.
缺少真实 reviewer 决策时只显示待评审，不伪造人工结论。When real reviewer decisions are missing, show pending status instead of fabricating human conclusions.
+
+
+
运行原则Run Rule
+
recorded fixture 只能证明可复现样本，不等同于模型执行。A recorded fixture proves reproducible samples only; it is not model execution.
只有 provider runner 返回 model metadata 时才计入 model-executed。Only provider runners that return model metadata count as model-executed.
+
+
+
+
+
+
+
+
+
+
风险治理Risk
+
提前暴露误触发、漂移、证据不足和迁移风险。Surfaces trigger, drift, evidence, and portability risks before the package grows.
+
+
+ 风险热力图用影响程度和发生概率标出当前治理重点。The risk heatmap marks governance priorities by impact and probability.
+
+
+
风险Risk
信号Signal
应对Response
+
误触发风险Trigger risk
frontmatter description 已存在，具备基础路由面。The frontmatter description exists, giving the skill a basic routing surface.
先补证据和边界，再增加包体复杂度。Improve evidence and boundaries before adding package complexity.
先补证据和边界，再增加包体复杂度。Improve evidence and boundaries before adding package complexity.
+
+
+
+
世界证据World Evidence
世界级证据尚未完成：4 项待补，0 项已接受。World-class evidence is not complete: 4 pending, 0 accepted.
证据待补Evidence pending
待补证据Pending4仍需外部或人工证据接受。External or human evidence still needs acceptance.已接受Accepted0已通过 source check 与提交契约。Passed source checks and submission contract.源检查Source Checks11 / 19通过数 / 总检查数。Passed checks / total checks.
外部证据External evidence
提供商留出Provider Holdout
缺少真实 provider 模型运行和 token metadata。Missing a real provider model run and token metadata.
阻塞检查Blocked Checks
当前没有阻塞检查。No blocked checks right now.
人工证据Human evidence
人工盲评Human Adjudication
盲评 pair 仍待真实 reviewer 决策。Blind-review pairs still need real reviewer decisions.
阻塞检查Blocked Checks
无待判定No pending decisions
盲评完成Judgments complete
Reviewer metadataReviewer metadata
外部证据External evidence
原生权限Native Permission
原生 runtime enforcement 仍待目标客户端或外部安装器证明。Native runtime enforcement still needs target-client or external-installer proof.
阻塞检查Blocked Checks
原生执行Native enforcement
外部证据External evidence
原生遥测Native Telemetry
真实外部客户端 metadata-only 事件仍未导入。Real external-client metadata-only events have not been imported yet.
阻塞检查Blocked Checks
外部事件External events
采用样本Adoption sample
+
+
+
+
+
+
+
包体资产Assets
+
让 reviewer 快速确认关键文件、目录和资产分布。Lets reviewers confirm key files, directories, and asset distribution quickly.
+
+
+ 资产分布图展示当前包体的文件和目录重心。The asset distribution chart shows where files and directories are concentrated.
+
+
路径Path
作用Role
类型Type
+
SKILL.md
Skill 入口文件Skill entrypoint
文件file
README.md
人类可读使用说明Human-readable usage guide
文件file
agents/interface.yaml
跨平台接口元数据Neutral interface metadata
文件file
manifest.json
生命周期与打包元数据Lifecycle and portability metadata
文件file
references
扩展指导与复用资料Extended guidance and reusable notes
目录folder
scripts
确定性脚本或本地工具Deterministic helpers or local tooling
目录folder
evals
触发与质量检查Trigger and quality checks
目录folder
reports
生成的证据与总结报告Generated evidence and overview artifacts
目录folder
+
+
+
+
+
+
+
+
+
迭代路线Roadmap
+
把下一步升级收束为少数高价值动作。Keeps next iteration moves focused and actionable.
+
+
+
迭代时间线把下一步升级收束成少数可执行动作。The iteration timeline narrows the next upgrade into a few executable moves.
+
下一步 1Next 1
补齐世界证据Close world-class evidence
世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。World-class evidence still has 4 pending item(s); public completion claims must stay blocked.
补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。Close provider holdout evidence: Missing a real provider model run and token metadata.
补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。Close human adjudication evidence: Blind-review pairs still need real reviewer decisions.
按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。Close all 4 pending evidence item(s) in the ledger (external 3, human 1; 2 not shown here) and keep the claim guard pending.
全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。Only after the ledger accepts all external and human evidence should the public world-class claim move to review.
下一步 2Next 2
有选择地借鉴一个成熟模式Borrow one proven pattern on purpose
已经有公开 benchmark 对象，下一步应主动选择一个模式借鉴，而不是松散吸收所有做法。You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.
阅读 `obra/superpowers` 中最值得借鉴的模式。Read the strongest pattern from obra/superpowers.
先判断要借鉴的是方法、结构、执行方式还是可迁移性，并且第一轮只借鉴其中一个。Decide whether to borrow method, structure, execution, or portability, but only one of them first.
记录本轮不借鉴的内容，避免包体过重。Record what you will not borrow so the package stays light.
包体形态更清晰，也减少偶然过度设计。A cleaner package shape with less accidental over-design.
下一步 3Next 3
加固跨环境语义Harden portability semantics
这个 Skill 已经面向跨环境复用，因此早期就需要把契约语义说清楚。The skill already signals reuse across environments, so contract clarity matters early.
确认激活模式、执行上下文和信任假设。Confirm activation mode, execution context, and trust assumptions.
补充或复核非原生目标端的降级策略。Add or review degradation strategy for non-native targets.
至少打包一次 Skill，用来验证 adapter 预期。Package the skill once to verify adapter expectations.
跨环境复用更安全，目标漂移更少。Safer cross-environment reuse with less target drift.
+
+
+
+
+
+
+
diff --git a/reports/skill-interpretation.json b/reports/skill-interpretation.json
new file mode 100644
index 00000000..a2392d37
--- /dev/null
+++ b/reports/skill-interpretation.json
@@ -0,0 +1,2295 @@
+{
+ "skill_summary": {
+ "name": "yao-meta-skill",
+ "title": "Yao Meta Skill",
+ "display_name": "Yao Meta Skill",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "maturity": "governed",
+ "updated_at": "2026-03-31",
+ "core_value": "把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。",
+ "audience": "Skill 作者、复用团队和后续 reviewer。",
+ "deliverables": [
+ "SKILL.md",
+ "agents/interface.yaml",
+ "skill-ir/examples/yao-meta-skill.json",
+ "reports/compiled_targets.md",
+ "reports/output_quality_scorecard.md",
+ "reports/output_execution_runs.md",
+ "reports/output_blind_review_pack.md",
+ "reports/output_review_kit.md",
+ "reports/output_review_adjudication.md",
+ "reports/benchmark_reproducibility.md",
+ "reports/output_blind_answer_key.json",
+ "reports/conformance_matrix.md",
+ "reports/security_trust_report.md",
+ "reports/runtime_permission_probes.md",
+ "reports/skill_atlas.html",
+ "reports/registry_audit.md",
+ "reports/package_verification.md",
+ "reports/install_simulation.md",
+ "reports/upgrade_check.md",
+ "reports/adoption_drift_report.md",
+ "reports/review_waivers.md",
+ "reports/world_class_evidence_plan.md",
+ "reports/world_class_evidence_ledger.md",
+ "reports/review_annotations.md",
+ "reports/review-studio.html",
+ "reports/skill-interpretation.html",
+ "reports/skill-overview.html",
+ "reports/skill-interpretation.json"
+ ],
+ "flow": [
+ "输入材料",
+ "Skill 包体",
+ "可复用能力"
+ ]
+ },
+ "scorecard": {
+ "completeness_score": {
+ "label": "完整度",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 已存在，是 Skill 的入口。",
+ "README.md 已存在，便于人工阅读。",
+ "agents/interface.yaml 已存在，便于跨平台适配。",
+ "manifest.json 已存在，生命周期信息可追踪。",
+ "reports/ 已存在，生成证据可以随包体迁移。"
+ ]
+ },
+ "trigger_score": {
+ "label": "触发清晰",
+ "score": 100,
+ "reasons": [
+ "frontmatter description 已存在，具备基础路由面。",
+ "description 有足够长度说明任务边界。",
+ "description 已包含使用场景或排除边界信号。",
+ "evals/ 已存在，可承载触发样例或质量检查。",
+ "intent-confidence 报告已生成，可辅助判断触发稳定性。"
+ ]
+ },
+ "evidence_score": {
+ "label": "证据充分",
+ "score": 100,
+ "reasons": [
+ "已生成 20 / 20 类报告证据。",
+ "skill-ir.json 已存在。",
+ "compiled_targets.json 已存在。",
+ "intent-dialogue.json 已存在。"
+ ]
+ },
+ "maintainability_score": {
+ "label": "可维护性",
+ "score": 100,
+ "reasons": [
+ "SKILL.md 约 339 个词/字。",
+ "入口文件保持克制，可维护性较好。",
+ "references/ 已承载扩展指导。",
+ "scripts/ 已承载确定性逻辑。",
+ "evals/ 已承载可迁移检查。"
+ ]
+ },
+ "portability_score": {
+ "label": "可迁移性",
+ "score": 100,
+ "reasons": [
+ "agents/interface.yaml 已存在。",
+ "manifest.json 已存在。",
+ "目标平台或 adapter target 已声明。",
+ "入口文件未发现明显私有绝对路径。"
+ ]
+ },
+ "context_cost": {
+ "label": "上下文成本",
+ "score": 42,
+ "reasons": [
+ "入口约 339 个词/字，references 约 16934 个词/字。",
+ "分数越高代表上下文成本越低。",
+ "上下文成本偏高，建议压缩入口或拆分 references。"
+ ]
+ }
+ },
+ "capability_profile": {
+ "archetype": "governed",
+ "task_family": "Execution operation",
+ "maturity": "governed",
+ "trigger_strength": "手动触发 + description 路由",
+ "reuse_scope": "跨平台",
+ "matrix": {
+ "execution_certainty": 72,
+ "knowledge_density": 80
+ }
+ },
+ "principle_model": {
+ "nodes": [
+ {
+ "title": "意图澄清",
+ "body": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse."
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ }
+ ],
+ "layers": [
+ "入口层",
+ "参考层",
+ "脚本层",
+ "评估层",
+ "报告层"
+ ]
+ },
+ "contract_boundary": {
+ "trigger": {
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "inputs": [
+ "rough workflow notes, SOPs, runbooks, prompts, transcripts, documents, or repeated task descriptions",
+ "an existing skill directory that needs refactor, evaluation, packaging, or governance hardening",
+ "target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution",
+ "benchmark references, local constraints, desired maturity tier, and review standards"
+ ],
+ "outputs": [
+ "A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ],
+ "should_trigger": [
+ "把重复流程沉淀为可复用的 agent skill。",
+ "把分散提示词、对话记录或操作规范整理为稳定能力。",
+ "团队复用前，需要明确触发边界、质量证据和维护责任。"
+ ],
+ "should_not_trigger": [
+ "只需要一次性回答、没有复用价值的临时请求。",
+ "要求直接执行相邻任务，而不是沉淀或使用这个 Skill。",
+ "缺少必要事实且用户不允许澄清的场景。"
+ ],
+ "boundary_cards": [
+ {
+ "label": "Owned",
+ "body": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
+ },
+ {
+ "label": "Adjacent",
+ "body": "相邻任务需要先确认是否应转为独立 Skill。"
+ },
+ {
+ "label": "Excluded",
+ "body": "不替代人工事实核查，也不静默扩大职责。"
+ }
+ ]
+ },
+ "quality_review": {
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Output Eval Lab scorecard，可比较 with-skill 与 baseline 输出质量。",
+ "已生成 Output Execution Runs，可区分记录样本、命令执行和模型执行证据。",
+ "已生成 Output Review Adjudication，可记录盲评决策、一致率和待评审项。"
+ ],
+ "gaps": [
+ "上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。"
+ ],
+ "recommendations": [
+ "先改触发边界，再扩展工作流。",
+ "只把重复且稳定的步骤沉淀为脚本。",
+ "每次升级后重新生成报告并检查分数原因。"
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 89.0,
+ "highlights": [
+ "Primary prompt task family: Execution operation.",
+ "Complexity: expert — multiple task families plus governance, evaluation, or expert-level constraints",
+ "Completeness: 100/100.",
+ "Clarity: 85/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 100,
+ "band": "system-ready"
+ },
+ "highlights": [
+ "Stability: system-ready (100/100).",
+ "Owned job: Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language.",
+ "Leverage: Install output self-repair checks — Add only the checks that prevent recurring output mistakes."
+ ]
+ }
+ },
+ "risk_governance": {
+ "risks": [
+ {
+ "name": "误触发风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "frontmatter description 已存在，具备基础路由面。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "输出漂移风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "证据不足风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "已生成 20 / 20 类报告证据。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "包体膨胀风险",
+ "impact": 2,
+ "probability": 1,
+ "signal": "SKILL.md 约 339 个词/字。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ },
+ {
+ "name": "跨平台迁移风险",
+ "impact": 3,
+ "probability": 1,
+ "signal": "agents/interface.yaml 已存在。",
+ "response": "先补证据和边界，再增加包体复杂度。"
+ }
+ ],
+ "risk_families": [
+ {
+ "key": "markdown_readability",
+ "label": "Markdown readability",
+ "matched_keywords": [
+ "md",
+ "report",
+ "doc"
+ ],
+ "score": 3,
+ "risks": [
+ "Tables can render as dense grids with weak hierarchy or poor mobile readability.",
+ "Long bullets can make the output look complete while hiding the actual decision logic.",
+ "Mixed heading levels can reduce scanability."
+ ],
+ "constraints": [
+ "Use tables only when comparison is the main job; otherwise prefer compact cards or grouped bullets.",
+ "Keep table cells short and move explanations below the table.",
+ "Use heading levels consistently and keep each section anchored to a user-facing outcome."
+ ],
+ "self_repair": [
+ "Preview whether each table still reads well when columns are narrow.",
+ "Convert any table with paragraph-length cells into bullets or cards."
+ ]
+ },
+ {
+ "key": "citation_clutter",
+ "label": "Citation and footnote clutter",
+ "matched_keywords": [
+ "source",
+ "reference"
+ ],
+ "score": 2,
+ "risks": [
+ "Footnote markers or dense citation notes can interrupt the reading flow.",
+ "Evidence can be over-attached to obvious statements and under-attached to risky claims.",
+ "Source notes may become more prominent than the tutorial itself."
+ ],
+ "constraints": [
+ "Attach citations only to claims that need evidence, not to every sentence.",
+ "Group source notes at the end of a section when inline markers would hurt readability.",
+ "Keep the main sentence readable without requiring the reader to chase a footnote."
+ ],
+ "self_repair": [
+ "Remove decorative citations that do not support a material claim.",
+ "Move repeated source explanations into one compact source note."
+ ]
+ },
+ {
+ "key": "visual_capture",
+ "label": "Screenshot and visual capture",
+ "matched_keywords": [
+ "capture"
+ ],
+ "score": 1,
+ "risks": [
+ "Screenshots can be captured from the wrong state, wrong viewport, or wrong crop.",
+ "Missing screenshots can cause the skill to invent visual references instead of declaring the gap.",
+ "Image descriptions can omit the action-relevant region."
+ ],
+ "constraints": [
+ "Never invent a screenshot; state when visual evidence is missing.",
+ "Record the source, viewport, and crop intent for any screenshot-dependent output.",
+ "Describe what the reader should inspect in the image, not just that an image exists."
+ ],
+ "self_repair": [
+ "Check that every screenshot reference points to a real provided or generated asset.",
+ "Reword any visual instruction that depends on an unseen screen state."
+ ]
+ },
+ {
+ "key": "code_or_command_safety",
+ "label": "Code and command safety",
+ "matched_keywords": [
+ "script"
+ ],
+ "score": 1,
+ "risks": [
+ "Commands can omit environment assumptions, working directory, or rollback notes.",
+ "Code snippets can look runnable while missing required inputs.",
+ "Error handling can be either absent or over-engineered."
+ ],
+ "constraints": [
+ "Name the working directory, required inputs, and expected output for each command.",
+ "Mark destructive or external side-effect operations explicitly.",
+ "Prefer the smallest runnable snippet over broad framework code."
+ ],
+ "self_repair": [
+ "Scan each command for cwd, input, output, and side-effect assumptions.",
+ "Remove speculative error handling that is not tied to a real failure mode."
+ ]
+ },
+ {
+ "key": "tone_and_specificity",
+ "label": "Tone and specificity",
+ "matched_keywords": [
+ "summary"
+ ],
+ "score": 1,
+ "risks": [
+ "Headings and summaries can drift into generic, interchangeable language.",
+ "The output can sound polished but lose the user's actual taste, audience, or scenario.",
+ "Strong claims can appear without examples or constraints."
+ ],
+ "constraints": [
+ "Anchor titles and summaries in the user's audience, object, and concrete outcome.",
+ "Avoid placeholder phrases such as comprehensive guide, ultimate solution, or key insights unless the source demands them.",
+ "Preserve one distinctive phrase, constraint, or standard from the user's input."
+ ],
+ "self_repair": [
+ "Replace generic title candidates with scenario-specific alternatives.",
+ "Delete any polished sentence that could fit almost any project unchanged."
+ ]
+ }
+ ],
+ "human_judgment_boundary": [
+ "Ask one focused clarification when the real job, output, or exclusion boundary is unclear.",
+ "Escalate visible tradeoffs when benchmark patterns conflict with local privacy, naming, or governance constraints.",
+ "Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
+ ]
+ },
+ "world_class_readiness": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "entry_count": 4,
+ "pending_count": 4,
+ "accepted_count": 0,
+ "external_pending_count": 3,
+ "human_pending_count": 1,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "conclusion_zh": "世界级证据尚未完成：4 项待补，0 项已接受。",
+ "conclusion_en": "World-class evidence is not complete: 4 pending, 0 accepted.",
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label_zh": "提供商留出",
+ "label_en": "Provider Holdout",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "缺少真实 provider 模型运行和 token metadata。",
+ "summary_en": "Missing a real provider model run and token metadata.",
+ "blocked_checks": []
+ },
+ {
+ "key": "human-adjudication",
+ "label_zh": "人工盲评",
+ "label_en": "Human Adjudication",
+ "category": "human",
+ "category_zh": "人工证据",
+ "category_en": "Human evidence",
+ "status": "pending",
+ "summary_zh": "盲评 pair 仍待真实 reviewer 决策。",
+ "summary_en": "Blind-review pairs still need real reviewer decisions.",
+ "blocked_checks": [
+ "No pending decisions",
+ "Judgments complete",
+ "Reviewer metadata"
+ ]
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label_zh": "原生权限",
+ "label_en": "Native Permission",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "原生 runtime enforcement 仍待目标客户端或外部安装器证明。",
+ "summary_en": "Native runtime enforcement still needs target-client or external-installer proof.",
+ "blocked_checks": [
+ "Native enforcement"
+ ]
+ },
+ {
+ "key": "native-client-telemetry",
+ "label_zh": "原生遥测",
+ "label_en": "Native Telemetry",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "真实外部客户端 metadata-only 事件仍未导入。",
+ "summary_en": "Real external-client metadata-only events have not been imported yet.",
+ "blocked_checks": [
+ "External events",
+ "Adoption sample"
+ ]
+ }
+ ]
+ },
+ "package_assets": {
+ "entries": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 35
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 153
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 29
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 235
+ }
+ ],
+ "file_count": 456,
+ "folder_count": 4,
+ "distribution": [
+ {
+ "label": "SKILL.md",
+ "value": 1
+ },
+ {
+ "label": "README.md",
+ "value": 1
+ },
+ {
+ "label": "agents/interface.yaml",
+ "value": 1
+ },
+ {
+ "label": "manifest.json",
+ "value": 1
+ },
+ {
+ "label": "references",
+ "value": 35
+ },
+ {
+ "label": "scripts",
+ "value": 153
+ },
+ {
+ "label": "evals",
+ "value": 29
+ },
+ {
+ "label": "reports",
+ "value": 235
+ }
+ ]
+ },
+ "iteration_roadmap": {
+ "items": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
+ {
+ "title": "Borrow one proven pattern on purpose",
+ "why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
+ "actions": [
+ "Read the strongest pattern from obra/superpowers.",
+ "Decide whether to borrow method, structure, execution, or portability, but only one of them first.",
+ "Record what you will not borrow so the package stays light."
+ ],
+ "unlocks": "A cleaner package shape with less accidental over-design."
+ },
+ {
+ "title": "Harden portability semantics",
+ "why": "The skill already signals reuse across environments, so contract clarity matters early.",
+ "actions": [
+ "Confirm activation mode, execution context, and trust assumptions.",
+ "Add or review degradation strategy for non-native targets.",
+ "Package the skill once to verify adapter expectations."
+ ],
+ "unlocks": "Safer cross-environment reuse with less target drift."
+ }
+ ]
+ },
+ "report_contract": {
+ "schema_version": "2.0",
+ "html_report": "reports/skill-interpretation.html",
+ "language": "zh-CN",
+ "default_language": "zh-CN",
+ "languages": [
+ "zh-CN",
+ "en"
+ ],
+ "layout": "kami-white-audit-v2",
+ "nav_labels": [
+ "技能概述",
+ "总览指标",
+ "能力画像",
+ "原理结构",
+ "契约边界",
+ "质量评估",
+ "风险治理",
+ "包体资产",
+ "迭代路线"
+ ],
+ "nav_labels_en": [
+ "Overview",
+ "Metrics",
+ "Profile",
+ "Principle",
+ "Contract",
+ "Quality",
+ "Risk",
+ "Assets",
+ "Roadmap"
+ ],
+ "report_kind": "skill-interpretation",
+ "canonical_overview_report": "reports/skill-overview.html",
+ "json_report": "reports/skill-interpretation.json",
+ "purpose": "Explain the generated skill's role, principles, usage scenarios, trigger contract, inputs, outputs, quality evidence, risks, assets, highlights, and next upgrade directions."
+ },
+ "name": "yao-meta-skill",
+ "title": "Yao Meta Skill",
+ "display_name": "Yao Meta Skill",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "logic_steps": [
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
+ ],
+ "usage_steps": [
+ "Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.",
+ "Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
+ ],
+ "package_map": [
+ {
+ "path": "SKILL.md",
+ "label": "Skill entrypoint",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "README.md",
+ "label": "Human-readable usage guide",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "agents/interface.yaml",
+ "label": "Neutral interface metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "manifest.json",
+ "label": "Lifecycle and portability metadata",
+ "kind": "file",
+ "file_count": 1
+ },
+ {
+ "path": "references",
+ "label": "Extended guidance and reusable notes",
+ "kind": "folder",
+ "file_count": 35
+ },
+ {
+ "path": "scripts",
+ "label": "Deterministic helpers or local tooling",
+ "kind": "folder",
+ "file_count": 153
+ },
+ {
+ "path": "evals",
+ "label": "Trigger and quality checks",
+ "kind": "folder",
+ "file_count": 29
+ },
+ {
+ "path": "reports",
+ "label": "Generated evidence and overview artifacts",
+ "kind": "folder",
+ "file_count": 235
+ }
+ ],
+ "strengths": [
+ "触发面保持精简，并锚定在 frontmatter description。",
+ "已生成 Skill IR，核心语义可先于平台打包被审查和迁移。",
+ "已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。",
+ "已生成 Output Eval Lab scorecard，可比较 with-skill 与 baseline 输出质量。",
+ "已生成 Output Execution Runs，可区分记录样本、命令执行和模型执行证据。",
+ "已生成 Output Review Adjudication，可记录盲评决策、一致率和待评审项。"
+ ],
+ "scenarios": [
+ "把重复流程沉淀为可复用的 agent skill。",
+ "把分散提示词、对话记录或操作规范整理为稳定能力。",
+ "团队复用前，需要明确触发边界、质量证据和维护责任。",
+ "用户说出类似需求时：Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals."
+ ],
+ "trigger_contract": {
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "activation": "manual",
+ "execution": "inline",
+ "shell": "bash"
+ },
+ "io_contract": {
+ "inputs": [
+ "rough workflow notes, SOPs, runbooks, prompts, transcripts, documents, or repeated task descriptions",
+ "an existing skill directory that needs refactor, evaluation, packaging, or governance hardening",
+ "target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution",
+ "benchmark references, local constraints, desired maturity tier, and review standards"
+ ],
+ "outputs": [
+ "A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.",
+ "结构化 Skill 目录，共 8 类关键资产。"
+ ]
+ },
+ "principles": [
+ {
+ "title": "意图澄清",
+ "body": "Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse."
+ },
+ {
+ "title": "边界路由",
+ "body": "用 frontmatter description 决定是否触发，并写明相邻非目标。"
+ },
+ {
+ "title": "资产分层",
+ "body": "把入口、参考、脚本、评估和报告拆到各自目录，避免 SKILL.md 膨胀。"
+ },
+ {
+ "title": "证据回路",
+ "body": "Ask only the highest-leverage clarification before adding package weight."
+ },
+ {
+ "title": "漂移观察",
+ "body": "Add near-neighbor exclusions and route evals before expanding workflow steps."
+ },
+ {
+ "title": "杠杆升级",
+ "body": "Name the recurring job, expected input, output, and strongest non-goal in compact language."
+ }
+ ],
+ "roadmap": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
+ {
+ "title": "Borrow one proven pattern on purpose",
+ "why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
+ "actions": [
+ "Read the strongest pattern from obra/superpowers.",
+ "Decide whether to borrow method, structure, execution, or portability, but only one of them first.",
+ "Record what you will not borrow so the package stays light."
+ ],
+ "unlocks": "A cleaner package shape with less accidental over-design."
+ },
+ {
+ "title": "Harden portability semantics",
+ "why": "The skill already signals reuse across environments, so contract clarity matters early.",
+ "actions": [
+ "Confirm activation mode, execution context, and trust assumptions.",
+ "Add or review degradation strategy for non-native targets.",
+ "Package the skill once to verify adapter expectations."
+ ],
+ "unlocks": "Safer cross-environment reuse with less target drift."
+ }
+ ],
+ "cards": [],
+ "introduction": [
+ "这份报告用于快速理解新生成 Skill 的定位、原理、触发边界和交付内容。",
+ "先确认重复任务、真实输入形态和可交付输出，再决定是否继续加 references、scripts 或 evals。",
+ "如果需求仍然模糊，优先回到 intent dialogue 收紧边界，再扩展包体结构。"
+ ],
+ "benchmark_highlights": [],
+ "skill_ir": {
+ "schema_version": "2.0.0",
+ "source_path": "skill-ir/examples/yao-meta-skill.json",
+ "target_count": 5,
+ "trigger_samples": 8,
+ "output_eval_cases": 3
+ },
+ "compiled_targets": {
+ "ok": true,
+ "schema_version": "1.0",
+ "summary": {
+ "target_count": 5,
+ "pass_count": 5,
+ "warn_count": 0,
+ "block_count": 0,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "targets": [
+ {
+ "target": "openai",
+ "status": "pass",
+ "adapter_mode": "metadata-adapter",
+ "degradation_strategy": "metadata-adapter",
+ "native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
+ "permission_enforcement": "metadata-only",
+ "generated_files": [
+ "targets/openai/adapter.json",
+ "targets/openai/agents/openai.yaml"
+ ],
+ "unsupported_features": [
+ "client-native script permission prompts are represented as permission contract metadata"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "claude",
+ "status": "pass",
+ "adapter_mode": "neutral-source-plus-adapter",
+ "degradation_strategy": "neutral-source-plus-adapter",
+ "native_surface": "Claude-compatible neutral source folder with adapter notes",
+ "permission_enforcement": "metadata-fallback",
+ "generated_files": [
+ "targets/claude/adapter.json",
+ "targets/claude/README.md"
+ ],
+ "unsupported_features": [
+ "vendor-native metadata fields are carried as adapter JSON and README notes"
+ ],
+ "warnings": []
+ },
+ {
+ "target": "generic",
+ "status": "pass",
+ "adapter_mode": "agent-skills-compatible",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills compatible neutral package",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "targets/generic/adapter.json"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "agent-skills-compatible",
+ "status": "pass",
+ "adapter_mode": "neutral-agent-skills-source",
+ "degradation_strategy": "neutral-source",
+ "native_surface": "Agent Skills standard source tree",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "generated_files": [
+ "SKILL.md",
+ "agents/interface.yaml"
+ ],
+ "unsupported_features": [],
+ "warnings": []
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "permission_enforcement": "client-or-workspace-trust",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "unsupported_features": [
+ "VS Code installation scope is documented but not installed by this compiler"
+ ],
+ "warnings": []
+ }
+ ],
+ "failures": [],
+ "warnings": []
+ },
+ "output_quality": {
+ "case_count": 5,
+ "file_backed_case_count": 1,
+ "near_neighbor_case_count": 1,
+ "boundary_case_count": 1,
+ "baseline_pass_rate": 0.0,
+ "with_skill_pass_rate": 100.0,
+ "delta": 100.0,
+ "regression_count": 0,
+ "gate_pass": true,
+ "failure_taxonomy": [],
+ "blind_pair_count": 5
+ },
+ "output_execution": {
+ "ok": true,
+ "summary": {
+ "case_count": 5,
+ "variant_run_count": 10,
+ "command_executed_count": 10,
+ "model_executed_count": 10,
+ "recorded_fixture_count": 0,
+ "timing_observed_count": 10,
+ "token_observed_count": 10,
+ "token_estimated_count": 0,
+ "baseline_pass_rate": 23.67,
+ "with_skill_pass_rate": 43.67,
+ "delta": 20.0,
+ "regression_count": 0,
+ "failure_count": 0,
+ "gate_pass": true
+ },
+ "runner": {
+ "mode": "command",
+ "command": [
+ "python3",
+ "scripts/provider_output_eval_runner.py",
+ "--provider",
+ "deepseek",
+ "--model",
+ "deepseek-v4-flash",
+ "--api-format",
+ "chat-completions",
+ "--thinking",
+ "disabled",
+ "--api-key-env",
+ "DEEPSEEK_API_KEY"
+ ],
+ "timeout_seconds": 180.0
+ },
+ "failures": []
+ },
+ "output_blind_review": {
+ "summary": {
+ "pair_count": 5,
+ "answer_key_separate": true,
+ "with_skill_hidden_count": 5
+ },
+ "seed": "yao-output-eval-blind-v1",
+ "pair_count": 5,
+ "answer_key_separate": true
+ },
+ "output_review_kit": {
+ "ok": true,
+ "summary": {
+ "case_count": 5,
+ "ready_for_adjudication_count": 0,
+ "pending_decision_count": 5,
+ "invalid_decision_count": 0,
+ "reviewer_metadata_present": false,
+ "answer_key_hidden": true,
+ "answer_key_path_exposed": false,
+ "ready_to_run_adjudication": false,
+ "failure_count": 0
+ },
+ "artifacts": {
+ "reviewer_kit_json": "reports/output_review_kit.json",
+ "reviewer_kit_markdown": "reports/output_review_kit.md",
+ "reviewer_kit_html": "reports/output_review_kit.html",
+ "blind_pack_json": "reports/output_blind_review_pack.json",
+ "blind_pack_markdown": "reports/output_blind_review_pack.md",
+ "decisions": "reports/output_review_decisions.json",
+ "adjudication_json": "reports/output_review_adjudication.json",
+ "adjudication_markdown": "reports/output_review_adjudication.md"
+ },
+ "failures": []
+ },
+ "output_review_adjudication": {
+ "ok": true,
+ "summary": {
+ "pair_count": 5,
+ "judgment_count": 0,
+ "pending_count": 5,
+ "agreement_count": 0,
+ "disagreement_count": 0,
+ "invalid_decision_count": 0,
+ "answer_revealed_count": 0,
+ "pending_answer_hidden_count": 5,
+ "agreement_rate": null,
+ "needs_review": true,
+ "reviewer_metadata_present": false,
+ "reason_required": true,
+ "blind_review_attested": false,
+ "blind_review_completed_before_answer_key": false,
+ "answer_key_not_opened_before_decisions": false,
+ "raw_content_excluded_attested": true,
+ "reviewer_reason_required_attested": true,
+ "ready_for_human_evidence": false,
+ "failure_count": 0,
+ "reviewer_checklist_count": 5,
+ "reviewer_checklist_pending_count": 5,
+ "reviewer_checklist_invalid_count": 0,
+ "reviewer_checklist_ready_count": 0
+ },
+ "reviewer": "",
+ "reviewed_at": "",
+ "failures": []
+ },
+ "benchmark_reproducibility": {
+ "ok": true,
+ "summary": {
+ "reproducibility_ready": true,
+ "release_lock_ready": true,
+ "methodology_complete": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "evidence_bundle_sha256": "55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0",
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "output_case_count": 5,
+ "failure_disclosure_count": 3,
+ "command_count": 23,
+ "command_executed_count": 10,
+ "timing_observed_count": 10,
+ "model_executed_count": 10,
+ "token_observed_count": 10,
+ "human_review_complete": false,
+ "provider_evidence_complete": true,
+ "world_class_ready": false,
+ "world_class_open_gap_count": 3,
+ "world_class_task_count": 4,
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3,
+ "working_tree_dirty": true,
+ "changed_file_count": 29,
+ "source_tree_dirty": false,
+ "source_changed_file_count": 0,
+ "generated_tree_dirty": true,
+ "generated_changed_file_count": 29
+ },
+ "commit": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "missing_artifacts": [],
+ "limitations": [
+ "The git commit and dirty flags are generation-time context; release lock is blocked by source changes, while generated evidence artifacts are tracked separately.",
+ "Provider-backed model holdout source evidence is complete, but ledger acceptance still requires a valid independently reviewed submission packet.",
+ "Pending blind-review decisions are visible but do not count as human adjudication.",
+ "World-class readiness remains false until external and human evidence gaps close.",
+ "Beta/public testing may proceed without human blind-review only when wording avoids superiority, fully-reviewed, or world-class claims."
+ ]
+ },
+ "runtime_conformance": {
+ "target_count": 5,
+ "pass_count": 5,
+ "fail_count": 0
+ },
+ "runtime_permissions": {
+ "ok": true,
+ "summary": {
+ "target_count": 4,
+ "pass_count": 4,
+ "fail_count": 0,
+ "native_enforcement_count": 0,
+ "metadata_fallback_count": 4,
+ "residual_risk_count": 4,
+ "required_capability_count": 3,
+ "failure_count": 0,
+ "installer_enforcement_source_status": "present",
+ "installer_enforcement_target_count": 4,
+ "installer_enforcement_pass_count": 4,
+ "installer_permission_enforced_count": 12,
+ "installer_permission_failure_count": 0,
+ "installer_permission_capability_count": 3,
+ "world_class_native_evidence_ready": false,
+ "installer_enforcement_ready": true
+ },
+ "expected_capabilities": [
+ "file_write",
+ "network",
+ "subprocess"
+ ],
+ "targets": [
+ {
+ "target": "openai",
+ "status": "pass",
+ "assurance": "metadata-fallback-explicit",
+ "native_enforcement": false,
+ "metadata_fallback_explicit": true,
+ "residual_risks": [
+ "Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
+ ]
+ },
+ {
+ "target": "claude",
+ "status": "pass",
+ "assurance": "metadata-fallback-explicit",
+ "native_enforcement": false,
+ "metadata_fallback_explicit": true,
+ "residual_risks": [
+ "Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
+ ]
+ },
+ {
+ "target": "generic",
+ "status": "pass",
+ "assurance": "metadata-fallback-explicit",
+ "native_enforcement": false,
+ "metadata_fallback_explicit": true,
+ "residual_risks": [
+ "Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
+ ]
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "assurance": "metadata-fallback-explicit",
+ "native_enforcement": false,
+ "metadata_fallback_explicit": true,
+ "residual_risks": [
+ "Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
+ ]
+ }
+ ],
+ "failures": []
+ },
+ "trust_security": {
+ "scanned_files": 245,
+ "script_count": 153,
+ "internal_module_count": 67,
+ "secret_findings": 0,
+ "dependency_files": [
+ "requirements-ci.txt"
+ ],
+ "network_script_count": 3,
+ "network_policy_covered_count": 3,
+ "network_policy_missing_count": 0,
+ "file_write_script_count": 73,
+ "permission_required_count": 3,
+ "permission_approved_count": 3,
+ "permission_missing_count": 0,
+ "permission_invalid_count": 0,
+ "permission_expired_count": 0,
+ "help_smoke_checked_count": 86,
+ "help_smoke_failed_count": 0,
+ "interactive_script_count": 0,
+ "package_hash_scope": "source-contract-without-generated-reports",
+ "package_hash_file_count": 245,
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257"
+ },
+ "skill_atlas": {
+ "skill_count": 12,
+ "actionable_skill_count": 1,
+ "route_collision_count": 5,
+ "actionable_route_collision_count": 0,
+ "owner_gap_count": 9,
+ "actionable_owner_gap_count": 0,
+ "stale_count": 10,
+ "actionable_stale_count": 0,
+ "shared_resource_count": 0,
+ "no_route_opportunity_count": 0,
+ "telemetry_report_count": 1,
+ "drift_signal_count": 0,
+ "actionable_drift_signal_count": 0,
+ "non_actionable_issue_count": 24
+ },
+ "registry_distribution": {
+ "ok": true,
+ "package": {
+ "schema_version": "2.0",
+ "name": "yao-meta-skill",
+ "version": "1.1.0",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
+ "targets": [
+ "openai",
+ "claude",
+ "generic",
+ "agent-skills-compatible",
+ "vscode"
+ ],
+ "maturity": "governed",
+ "owner": "Yao Team",
+ "review_cadence": "quarterly",
+ "trust_level": "local",
+ "license": "MIT",
+ "checksums": {
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2"
+ },
+ "compatibility": {
+ "openai": "pass",
+ "claude": "pass",
+ "agent-skills": "pass",
+ "vscode": "pass",
+ "generic": "pass",
+ "agent-skills-compatible": "pass"
+ },
+ "source": {
+ "skill_dir": ".",
+ "skill_ir": "skill-ir/examples/yao-meta-skill.json",
+ "ir_schema_version": "2.0.0",
+ "canonical_metadata": "agents/interface.yaml"
+ },
+ "artifacts": {
+ "overview_html": "reports/skill-overview.html",
+ "review_studio_html": "reports/review-studio.html",
+ "trust_report": "reports/security_trust_report.md",
+ "conformance_matrix": "reports/conformance_matrix.md",
+ "compiled_targets": "reports/compiled_targets.md",
+ "package_verification": "reports/package_verification.md",
+ "install_simulation": "reports/install_simulation.md",
+ "adoption_drift": "reports/adoption_drift_report.md",
+ "review_waivers": "reports/review_waivers.md",
+ "review_annotations": "reports/review_annotations.md",
+ "package_metadata": "registry/packages/yao-meta-skill.json"
+ },
+ "distribution": {
+ "archive_verified": true,
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "package_verification": "reports/package_verification.json",
+ "install_simulated": true,
+ "install_simulation": "reports/install_simulation.json"
+ },
+ "generated_at": "2026-06-13"
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "package_verification": {
+ "ok": true,
+ "summary": {
+ "target_count": 4,
+ "adapter_count": 4,
+ "archive_present": true,
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "archive_entry_count": 689,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "install_simulation": {
+ "ok": true,
+ "summary": {
+ "archive_present": true,
+ "archive_entry_count": 696,
+ "archive_extracted": true,
+ "entrypoint_loaded": true,
+ "manifest_loaded": true,
+ "interface_loaded": true,
+ "adapter_count": 4,
+ "installer_permission_enforced_count": 12,
+ "installer_permission_failure_count": 0,
+ "permission_target_count": 4,
+ "permission_capability_count": 3,
+ "install_root_is_temp": false,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "upgrade_check": {
+ "ok": true,
+ "summary": {
+ "declared_bump": "minor",
+ "recommended_bump": "minor",
+ "breaking_change_count": 0,
+ "added_target_count": 2,
+ "metadata_change_count": 1,
+ "checksum_change_count": 2,
+ "failure_count": 0,
+ "warning_count": 0
+ },
+ "upgrade_diff": {
+ "name_changed": false,
+ "added_targets": [
+ "agent-skills-compatible",
+ "vscode"
+ ],
+ "removed_targets": [],
+ "compatibility_changes": [
+ {
+ "target": "agent-skills",
+ "from": "missing",
+ "to": "pass",
+ "direction": "improved"
+ },
+ {
+ "target": "agent-skills-compatible",
+ "from": "missing",
+ "to": "pass",
+ "direction": "improved"
+ },
+ {
+ "target": "vscode",
+ "from": "missing",
+ "to": "pass",
+ "direction": "improved"
+ }
+ ],
+ "metadata_changes": [
+ {
+ "field": "description",
+ "from": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.",
+ "to": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
+ }
+ ],
+ "checksum_changes": [
+ {
+ "field": "archive_sha256",
+ "from": "",
+ "to": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2"
+ },
+ {
+ "field": "package_sha256",
+ "from": "0000000000000000000000000000000000000000000000000000000000000000",
+ "to": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257"
+ }
+ ]
+ },
+ "release_notes": [
+ "Recommended version bump: minor.",
+ "Added targets: agent-skills-compatible, vscode.",
+ "Package or archive checksum changed; reviewers should verify package artifacts before release.",
+ "Package verification evidence: reports/package_verification.md."
+ ],
+ "failures": [],
+ "warnings": []
+ },
+ "adoption_drift": {
+ "ok": true,
+ "summary": {
+ "event_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
+ "edited_count": 0,
+ "rejected_count": 0,
+ "missed_count": 0,
+ "failed_count": 0,
+ "adoption_rate": 0,
+ "missed_trigger_count": 0,
+ "wrong_trigger_count": 0,
+ "bad_output_count": 0,
+ "script_error_count": 0,
+ "missing_resource_count": 0,
+ "review_overdue_count": 0,
+ "risk_band": "low",
+ "event_types": {
+ "review_event": 1
+ },
+ "failure_types": {},
+ "source_types": {
+ "manual": 1
+ },
+ "command_counts": {}
+ },
+ "next_iteration_candidates": [],
+ "privacy_contract": {
+ "storage": "local-first",
+ "event_scope": "metadata-only",
+ "raw_content_allowed": false,
+ "raw_event_log_packaged": false,
+ "blocked_fields": [
+ "content",
+ "input",
+ "inputs",
+ "message",
+ "messages",
+ "note",
+ "output",
+ "outputs",
+ "prompt",
+ "raw",
+ "text",
+ "transcript"
+ ]
+ },
+ "failures": []
+ },
+ "review_waivers": {
+ "ok": true,
+ "summary": {
+ "waiver_count": 0,
+ "active_count": 0,
+ "expired_count": 0,
+ "invalid_count": 0,
+ "covered_gate_count": 0,
+ "covered_gate_keys": [],
+ "waiver_candidate_count": 2,
+ "waiverable_open_count": 1,
+ "non_waivable_count": 1
+ },
+ "policy": {
+ "blocker_waivers_allowed": false,
+ "minimum_reason_chars": 20,
+ "expires_required": true,
+ "review_studio_gate_keys": [
+ "architecture-maintainability",
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "python-compat",
+ "registry-audit",
+ "release-notes",
+ "review-waivers",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report",
+ "world-class-evidence"
+ ],
+ "known_gate_keys": [
+ "architecture-maintainability",
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "python-compat",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ],
+ "waiverable_gate_keys": [
+ "architecture-maintainability",
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "python-compat",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ],
+ "non_waivable_gate_keys": [
+ "review-waivers",
+ "world-class-evidence"
+ ]
+ },
+ "failures": [],
+ "warnings": []
+ },
+ "review_annotations": {
+ "ok": true,
+ "summary": {
+ "annotation_count": 0,
+ "open_count": 0,
+ "resolved_count": 0,
+ "deferred_count": 0,
+ "open_blocker_count": 0,
+ "open_warning_count": 0,
+ "linked_gate_count": 0,
+ "target_missing_count": 0,
+ "failure_count": 0
+ },
+ "annotations": [],
+ "failures": []
+ },
+ "world_class_evidence_plan": {
+ "ok": true,
+ "summary": {
+ "audit_decision": "continue-iteration",
+ "world_class_ready": false,
+ "audit_world_class_ready": false,
+ "ready_to_claim_world_class": false,
+ "ledger_completion_required": true,
+ "evidence_requirement_count": 4,
+ "task_count": 4,
+ "human_task_count": 1,
+ "external_task_count": 3,
+ "review_task_count": 0,
+ "decision": "collect-external-evidence"
+ },
+ "tasks": [
+ {
+ "key": "provider-holdout",
+ "label": "Provider Holdout",
+ "status": "pass",
+ "category": "external",
+ "owner": "operator with provider credentials",
+ "current": "model-executed 10; token-observed 10",
+ "objective": "Collect at least one provider-backed output-eval holdout run with model, timing, and token metadata.",
+ "runbook": [
+ "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60",
+ "For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/output_execution_runs.json summary.model_executed_count > 0",
+ "reports/output_execution_runs.json summary.timing_observed_count > 0",
+ "reports/output_execution_runs.json summary.token_observed_count > 0",
+ "reports/skill_os2_audit.json item provider-holdout status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_execution_runs.json",
+ "reports/output_execution_runs.md",
+ "reports/skill_os2_audit.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/provider-holdout.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not commit provider credentials or environment dumps.",
+ "The output execution report records output hashes and aggregate run metadata, not raw provider prompts."
+ ],
+ "audit_next_action": "Run provider-backed holdout cases with real credentials and commit only aggregate evidence."
+ },
+ {
+ "key": "human-adjudication",
+ "label": "Human Adjudication",
+ "status": "human_required",
+ "category": "human",
+ "owner": "human reviewer",
+ "current": "0/5 decisions; pending 5",
+ "objective": "Record real blind A/B reviewer decisions before claiming human output review completion.",
+ "runbook": [
+ "python3 scripts/yao.py output-review-kit --write-template",
+ "Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.",
+ "python3 scripts/adjudicate_output_review.py --write-template",
+ "Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.",
+ "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "python3 scripts/yao.py output-review",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/output_review_adjudication.json summary.pending_count == 0",
+ "reports/output_review_adjudication.json summary.judgment_count == summary.pair_count",
+ "reports/output_review_adjudication.json summary.invalid_decision_count == 0",
+ "reports/output_review_adjudication.json summary.reviewer_metadata_present is true",
+ "reports/output_review_adjudication.json summary.blind_review_attested is true",
+ "reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text",
+ "reports/output_review_adjudication.json summary.ready_for_human_evidence is true",
+ "reports/skill_os2_audit.json item human-adjudication status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_blind_review_pack.md",
+ "reports/output_review_kit.md",
+ "reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json",
+ "reports/output_review_adjudication.md",
+ "scripts/import_output_review_decisions.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/human-adjudication.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Reviewer decisions should not include raw user data or private customer detail.",
+ "Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.",
+ "The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.",
+ "The adjudication evidence stores prompt_sha256 instead of raw prompt text.",
+ "The decision and adjudication artifacts preserve blind_pack_sha256 so reviewers can audit exactly which pack was judged.",
+ "Keep the answer key separate until after decisions are recorded."
+ ],
+ "audit_next_action": "Record real A/B choices, reviewer metadata, and blind-review attestation, then regenerate adjudication."
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "status": "external_required",
+ "category": "external",
+ "owner": "target client or installer integrator",
+ "current": "native-enforced targets 0; installer-enforced targets 4",
+ "objective": "Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.",
+ "runbook": [
+ "Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.",
+ "Update the generated target adapter only when the guard is actually enforced by that target.",
+ "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip",
+ "python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation",
+ "python3 scripts/yao.py runtime-permissions . --package-dir dist",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/runtime_permission_probes.json summary.native_enforcement_count > 0",
+ "reports/runtime_permission_probes.json summary.failure_count == 0",
+ "reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence",
+ "reports/skill_os2_audit.json item native-permission-enforcement status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "dist/targets/*/adapter.json",
+ "reports/runtime_permission_probes.json",
+ "reports/runtime_permission_probes.md",
+ "reports/install_simulation.json",
+ "reports/install_simulation.md",
+ "security/permission_policy.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not mark native_enforcement true for metadata-only fallbacks.",
+ "Keep residual risks visible for targets that still rely on operator enforcement."
+ ],
+ "audit_next_action": "Integrate a real target-client or external installer runtime guard before claiming native permission enforcement."
+ },
+ {
+ "key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "status": "external_required",
+ "category": "external",
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "current": "external source events 0; adoption samples 0",
+ "objective": "Import production metadata-only events from a real external client into the local drift loop.",
+ "runbook": [
+ "python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///",
+ "Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
+ "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl",
+ "python3 scripts/yao.py skill-atlas --workspace-root .",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/adoption_drift_report.json summary.source_types.external > 0",
+ "reports/adoption_drift_report.json summary.adoption_sample_count > 0",
+ "reports/skill_os2_audit.json item native-client-telemetry status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/adoption_drift_report.json",
+ "reports/adoption_drift_report.md",
+ "reports/telemetry_hook_recipes.json",
+ "scripts/telemetry_native_host.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Telemetry must remain metadata-only and local-first.",
+ "Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field."
+ ],
+ "audit_next_action": "Install a real client against the native host and import production metadata-only events."
+ }
+ ],
+ "source_audit": {
+ "json": "reports/skill_os2_audit.json",
+ "markdown": "reports/skill_os2_audit.md",
+ "open_gap_count": 3
+ }
+ },
+ "world_class_evidence_ledger": {
+ "ok": true,
+ "summary": {
+ "ledger_entry_count": 4,
+ "source_accepted_count": 1,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "submitted_entry_count": 0,
+ "reviewer_approved_submission_count": 0,
+ "missing_submission_count": 4,
+ "invalid_submission_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "submitted_but_pending_count": 0,
+ "source_accepted_without_valid_submission_count": 1,
+ "overclaim_guard_active": true,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label": "Provider Holdout",
+ "category": "external",
+ "owner": "operator with provider credentials",
+ "status": "pending",
+ "source_status": "pass",
+ "source_accepted": true,
+ "current": "model-executed 10; token-observed 10",
+ "objective": "Collect at least one provider-backed output-eval holdout run with model, timing, and token metadata.",
+ "runbook": [
+ "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60",
+ "For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "provider-backed model run",
+ "observed timing",
+ "observed token metadata"
+ ],
+ "success_checks": [
+ "reports/output_execution_runs.json summary.model_executed_count > 0",
+ "reports/output_execution_runs.json summary.timing_observed_count > 0",
+ "reports/output_execution_runs.json summary.token_observed_count > 0",
+ "reports/skill_os2_audit.json item provider-holdout status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_execution_runs.json",
+ "reports/output_execution_runs.md",
+ "reports/skill_os2_audit.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/provider-holdout.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not commit provider credentials or environment dumps.",
+ "The output execution report records output hashes and aggregate run metadata, not raw provider prompts."
+ ],
+ "observed_state": {
+ "model_executed_count": 10,
+ "timing_observed_count": 10,
+ "token_observed_count": 10,
+ "accepted": true
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Provider model run",
+ "field": "model_executed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Run provider-backed output-exec with real credentials."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Timing observed",
+ "field": "timing_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should record timing metadata."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Token usage observed",
+ "field": "token_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should return non-estimated token usage."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 3,
+ "source_blocked_count": 0,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/provider-holdout.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Run provider-backed holdout cases with real credentials and commit only aggregate evidence."
+ },
+ {
+ "key": "human-adjudication",
+ "label": "Human Adjudication",
+ "category": "human",
+ "owner": "human reviewer",
+ "status": "pending",
+ "source_status": "human_required",
+ "source_accepted": false,
+ "current": "0/5 decisions; pending 5",
+ "objective": "Record real blind A/B reviewer decisions before claiming human output review completion.",
+ "runbook": [
+ "python3 scripts/yao.py output-review-kit --write-template",
+ "Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.",
+ "python3 scripts/adjudicate_output_review.py --write-template",
+ "Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.",
+ "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "python3 scripts/yao.py output-review",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real reviewer identity",
+ "blind A/B decisions",
+ "answer key unopened until decisions exist"
+ ],
+ "success_checks": [
+ "reports/output_review_adjudication.json summary.pending_count == 0",
+ "reports/output_review_adjudication.json summary.judgment_count == summary.pair_count",
+ "reports/output_review_adjudication.json summary.invalid_decision_count == 0",
+ "reports/output_review_adjudication.json summary.reviewer_metadata_present is true",
+ "reports/output_review_adjudication.json summary.blind_review_attested is true",
+ "reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text",
+ "reports/output_review_adjudication.json summary.ready_for_human_evidence is true",
+ "reports/skill_os2_audit.json item human-adjudication status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_blind_review_pack.md",
+ "reports/output_review_kit.md",
+ "reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json",
+ "reports/output_review_adjudication.md",
+ "scripts/import_output_review_decisions.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/human-adjudication.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Reviewer decisions should not include raw user data or private customer detail.",
+ "Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.",
+ "The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.",
+ "The adjudication evidence stores prompt_sha256 instead of raw prompt text.",
+ "The decision and adjudication artifacts preserve blind_pack_sha256 so reviewers can audit exactly which pack was judged.",
+ "Keep the answer key separate until after decisions are recorded."
+ ],
+ "observed_state": {
+ "pair_count": 5,
+ "judgment_count": 0,
+ "pending_count": 5,
+ "invalid_decision_count": 0,
+ "answer_revealed_count": 0,
+ "reviewer_metadata_present": false,
+ "reason_required": true,
+ "blind_review_attested": false,
+ "raw_content_excluded_attested": true,
+ "reviewer_reason_required_attested": true,
+ "ready_for_human_evidence": false,
+ "raw_content_allowed": false,
+ "raw_content_path_count": 0,
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Review pairs exist",
+ "field": "pair_count",
+ "expected": ">0",
+ "actual": 5,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Generate the blind A/B review pack."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No pending decisions",
+ "field": "pending_count",
+ "expected": "==0",
+ "actual": 5,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record a reviewer choice and reason for every pair."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Judgments complete",
+ "field": "judgment_count",
+ "expected": "==pair_count",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Every pair needs one valid human judgment."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No invalid decisions",
+ "field": "invalid_decision_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Fix malformed winner/confidence entries."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reviewer metadata",
+ "field": "reviewer_metadata_present",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record reviewer and reviewed_at before adjudication can count."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reason required",
+ "field": "reason_required",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Keep reason mandatory for every imported or direct reviewer decision."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Blind review attested",
+ "field": "blind_review_attested",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content attested",
+ "field": "raw_content_excluded_attested",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Human evidence ready",
+ "field": "ready_for_human_evidence",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."
+ }
+ ],
+ "source_check_count": 10,
+ "source_pass_count": 5,
+ "source_blocked_count": 5,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/human-adjudication.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Record real A/B choices, reviewer metadata, and blind-review attestation, then regenerate adjudication."
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "category": "external",
+ "owner": "target client or installer integrator",
+ "status": "pending",
+ "source_status": "external_required",
+ "source_accepted": false,
+ "current": "native-enforced targets 0; installer-enforced targets 4",
+ "objective": "Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.",
+ "runbook": [
+ "Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.",
+ "Update the generated target adapter only when the guard is actually enforced by that target.",
+ "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip",
+ "python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation",
+ "python3 scripts/yao.py runtime-permissions . --package-dir dist",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real target client or external installer runtime guard",
+ "native enforcement flag or externally accepted guard proof",
+ "residual risk retained for fallback targets"
+ ],
+ "success_checks": [
+ "reports/runtime_permission_probes.json summary.native_enforcement_count > 0",
+ "reports/runtime_permission_probes.json summary.failure_count == 0",
+ "reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence",
+ "reports/skill_os2_audit.json item native-permission-enforcement status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "dist/targets/*/adapter.json",
+ "reports/runtime_permission_probes.json",
+ "reports/runtime_permission_probes.md",
+ "reports/install_simulation.json",
+ "reports/install_simulation.md",
+ "security/permission_policy.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not mark native_enforcement true for metadata-only fallbacks.",
+ "Keep residual risks visible for targets that still rely on operator enforcement."
+ ],
+ "observed_state": {
+ "native_enforcement_count": 0,
+ "metadata_fallback_count": 4,
+ "installer_enforcement_pass_count": 4,
+ "installer_permission_failure_count": 0,
+ "installer_enforcement_ready": true,
+ "residual_risk_count": 4,
+ "failure_count": 0,
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Native enforcement",
+ "field": "native_enforcement_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Collect real target-client or external runtime guard proof."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Probe failures",
+ "field": "failure_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Runtime permission probes must stay clean."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Installer support",
+ "field": "installer_enforcement_ready",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Installer enforcement is supporting evidence, not native proof."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 2,
+ "source_blocked_count": 1,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/native-permission-enforcement.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Integrate a real target-client or external installer runtime guard before claiming native permission enforcement."
+ },
+ {
+ "key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "category": "external",
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "pending",
+ "source_status": "external_required",
+ "source_accepted": false,
+ "current": "external source events 0; adoption samples 0",
+ "objective": "Import production metadata-only events from a real external client into the local drift loop.",
+ "runbook": [
+ "python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///",
+ "Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
+ "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl",
+ "python3 scripts/yao.py skill-atlas --workspace-root .",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real external client source",
+ "metadata-only event",
+ "local-first import path"
+ ],
+ "success_checks": [
+ "reports/adoption_drift_report.json summary.source_types.external > 0",
+ "reports/adoption_drift_report.json summary.adoption_sample_count > 0",
+ "reports/skill_os2_audit.json item native-client-telemetry status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/adoption_drift_report.json",
+ "reports/adoption_drift_report.md",
+ "reports/telemetry_hook_recipes.json",
+ "scripts/telemetry_native_host.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Telemetry must remain metadata-only and local-first.",
+ "Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field."
+ ],
+ "observed_state": {
+ "external_source_events": 0,
+ "adoption_sample_count": 0,
+ "raw_content_allowed": false,
+ "risk_band": "low",
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "External events",
+ "field": "external_source_events",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Import at least one metadata-only event from a real client."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Adoption sample",
+ "field": "adoption_sample_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Telemetry must include adoption outcome evidence."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Telemetry must stay metadata-only."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 1,
+ "source_blocked_count": 2,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/native-client-telemetry.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Install a real client against the native host and import production metadata-only events."
+ }
+ ],
+ "source_plan": {
+ "json": "reports/world_class_evidence_plan.json",
+ "markdown": "reports/world_class_evidence_plan.md",
+ "task_count": 4,
+ "evidence_requirement_count": 4
+ }
+ },
+ "synthesis_highlights": [
+ "Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
+ "Borrow a review checkpoint wherever trust matters more than raw speed.",
+ "Borrow the discipline of defining what the skill should not own before growing the package."
+ ],
+ "artifact_design": {
+ "design_system": "metric editorial",
+ "highlights": [
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
+ ]
+ },
+ "prompt_quality": {
+ "overall_quality_score": 89.0,
+ "highlights": [
+ "Primary prompt task family: Execution operation.",
+ "Complexity: expert — multiple task families plus governance, evaluation, or expert-level constraints",
+ "Completeness: 100/100.",
+ "Clarity: 85/100."
+ ]
+ },
+ "system_model": {
+ "stability": {
+ "score": 100,
+ "band": "system-ready"
+ },
+ "highlights": [
+ "Stability: system-ready (100/100).",
+ "Owned job: Turn repeated workflows, prompts, transcripts, runbooks, documents, or existing skill packages into routeable, evaluable, packageable, and governable agent skills for personal, team, library, or governed reuse.",
+ "Leverage: Tune the frontmatter description — Name the recurring job, expected input, output, and strongest non-goal in compact language.",
+ "Leverage: Install output self-repair checks — Add only the checks that prevent recurring output mistakes."
+ ]
+ },
+ "metadata": {
+ "canonical_format": "agent-skills",
+ "targets": [
+ "openai",
+ "claude",
+ "generic",
+ "vscode"
+ ],
+ "maturity_tier": "governed",
+ "skill_archetype": "governed",
+ "updated_at": "2026-03-31"
+ },
+ "interpretation_contract": {
+ "schema_version": "2.0",
+ "source_model": "skill-overview-v2",
+ "source_model_reused": true,
+ "overview_report": "reports/skill-overview.html",
+ "html_report": "reports/skill-interpretation.html",
+ "json_report": "reports/skill-interpretation.json",
+ "default_language": "zh-CN",
+ "languages": [
+ "zh-CN",
+ "en"
+ ],
+ "includes": [
+ "skill role",
+ "principles",
+ "usage scenarios",
+ "trigger contract",
+ "inputs and outputs",
+ "quality evidence",
+ "risk governance",
+ "package assets",
+ "highlights",
+ "upgrade roadmap"
+ ]
+ }
+}
\ No newline at end of file
diff --git a/reports/skill-os-2-review.md b/reports/skill-os-2-review.md
index ae003a98..3223a857 100644
--- a/reports/skill-os-2-review.md
+++ b/reports/skill-os-2-review.md
@@ -8,84 +8,123 @@ Scope: Yao Meta Skill against the user-provided Skill OS 2.0 upgrade plan.
Yao Meta Skill is no longer only a Meta Skill factory. The current working tree now has the first verifiable Skill OS foundation:
- Skill IR v0 for platform-neutral contract capture.
-- Output Eval Lab v0 for with-skill vs baseline assertion grading plus local command-runner execution evidence, blind A/B review pack, separate answer key evidence, and reviewer adjudication reporting.
+- Output Eval Lab v0 for with-skill vs baseline assertion grading plus local command-runner execution evidence, blind A/B review pack, separate answer key evidence, reviewer decision import, and adjudication reporting.
- Runtime Conformance v0 for target-consumption checks.
- Trust/Security v0 for secret, dependency, script, bounded network host policy, execution-level CLI help smoke checks, trust metadata, and stable source-contract integrity checks.
-- Skill Atlas v0 for portfolio catalog, route overlap, stale ownership, dependency signals, and no-route opportunities.
+- Skill Atlas v0 for portfolio catalog, route overlap, stale ownership, dependency signals, aggregate drift signals, and no-route opportunities.
- Bilingual Skill Overview v2 that includes these evidence surfaces.
+- Skill Overview Asset Split v0 so long report CSS and JavaScript live in `assets/skill-overview.css` and `assets/skill-overview.js`, while the Python layout module keeps the generated report single-file by inlining those assets at render time.
+- Review Studio Source Anchor Excerpts v0 so every non-pass action source ref carries a matched pattern and short line excerpt, letting reviewers understand the anchor before opening the full artifact.
+- Review Studio World-Class Collection Contract v0 so each pending world-class evidence action card shows provenance requirements, success checks, evidence artifacts, and privacy boundaries alongside commands and blocked checks.
+- Review Studio Asset Split v0 so long Review Studio CSS lives in `assets/review-studio.css`, while the Python layout module still inlines it into the generated static HTML report.
+- Skill OS 2.0 Audit v0 for requirement-by-requirement evidence mapping, with explicit separation between local evidence, human-required review, and external-required provider/client integrations.
+- World-Class Evidence Plan v0 that turns the remaining provider holdout, human adjudication, native permission enforcement, and native client telemetry gaps into executable runbooks with success checks and privacy contracts.
+- World-Class Evidence Ledger v0 that records current acceptance state, provenance requirements, privacy contracts, and anti-overclaim guards for those remaining gaps.
+- World-Class Intake Contract Hardening v0 so real evidence submissions must use the ledger's canonical `.json` filename and are recursively rejected when they include raw prompt, output, transcript, message, credential, secret, token, or API-key fields.
+- World-Class Human Evidence Guard v0 so human-adjudication decision artifacts recursively reject raw content, credential, secret, token, and answer-key fields before blind A/B review evidence can be accepted.
+- Output Review Privacy Guard v0 so the blind-review importer and world-class human evidence validator share one recursive blocked-field contract for raw content, credential, secret, token, and answer-key leakage.
+- World-Class Submission Privacy Guard v0 so real evidence submission packets reuse the same blocked-field contract and reject nested answer-key leakage before ledger review.
+- World-Class Provider Evidence Guard v0 so provider holdout submissions must reconcile `summary` counts with `runs` rows and include at least one passing model run whose provider, model, timing, non-estimated usage, and output hash match the submitted provenance.
+- World-Class Native Permission Evidence Guard v0 so native-permission submissions must reconcile runtime probe summary counts with target rows, require at least one native-enforced target row, and keep installer permission checks failure-free.
+- World-Class Native Telemetry Evidence Guard v0 so native-client-telemetry submissions must reconcile adoption summary counts with external metadata event rows and keep hook recipes metadata-only without claiming native auto-capture.
+- World-Class Submission Matrix v0 so the submission kit exposes a single operator-facing matrix for draft status, artifact readiness, source-check blockers, and the next action without counting matrix rows as completion evidence.
+- World-Class Repair Checklist v0 so the submission kit turns every draft, artifact, and source-check blocker into machine-readable repair rows without counting repair guidance as completion evidence.
+- World-Class Preflight Repair Rows v0 so the collection preflight exposes required precheck and source-check blockers as machine-readable operator actions before evidence submission.
+- World-Class Submission Artifact Roles v0 so submission kits distinguish `submission-ref` rows that belong in `artifact_refs` from supporting audit assets, reducing operator ambiguity without accepting evidence.
+- World-Class Submission Kit Rendering v0 so Markdown and HTML kit presentation lives in a dedicated internal renderer, keeping the CLI focused on evidence assembly and file emission.
+- Benchmark Reproducibility v0 that turns public benchmark methodology, required artifacts, failure disclosure, and reproduction commands into machine-checkable release evidence.
- Review Studio 2.0 v0 for one-page blocker, warning, evidence-path, review-action, and release-gate review.
- Review Studio Source Refs v0 so every non-pass review action can expose structured relative source/report links with best-effort line numbers.
-- Target Compiler v0 so Skill IR compiles into OpenAI, Claude, generic, and Agent Skills compatible target contracts before packaging, including target-specific permission contracts and native behavior contracts.
-- IR-first packaging v0 so adapters carry the platform-neutral semantic contract, compiler contract, parity checks, and IR provenance.
+- Target Compiler v0 so Skill IR compiles into OpenAI, Claude, generic, Agent Skills compatible, and VS Code / Copilot target contracts before packaging, including target-specific permission contracts and native behavior contracts.
+- IR-first packaging v0 so adapters carry the platform-neutral semantic contract, compiler contract, parity checks, IR provenance, and VS Code / Copilot install-scope notes where applicable.
- Registry & Distribution v0 for package metadata, checksum, owner, license, compatibility matrix, and audit reporting.
- Package Verification v0 for generated manifest, target adapters, zip archive safety, archive checksum, and registry parity.
-- Install Simulation v0 for local extraction, entrypoint loading, interface loading, report presence, and adapter readability.
+- Install Simulation v0 for local extraction, entrypoint loading, interface loading, report presence, adapter readability, and installer-level permission approval/enforcement checks.
- Upgrade Check v0 for registry package diffs, semver bump recommendations, breaking-change notes, and release evidence.
-- Adoption Drift v0 for local-first metadata-only telemetry, privacy-field blocking, usage-signal aggregation, and iteration candidate generation.
+- Adoption Drift v0 for local-first metadata-only telemetry, privacy-field blocking, usage-signal aggregation, iteration candidate generation, and Skill Atlas drift-signal input.
+- CLI Telemetry Capture v0 for opt-in `yao.py` command-run metadata that records command name, source, outcome, failure class, and timestamp without command arguments or raw user/model content.
+- External Telemetry Import v0 for whole-file validated metadata-only JSONL ingestion from non-CLI clients, with dry-run support and all-or-nothing rejection on raw content fields or schema violations.
+- Telemetry Emit Hook v0 for Browser/Chrome/IDE/wrapper integrations to append one validated metadata-only external event into a local spool before import, with dry-run validation and raw-content blocking.
+- Telemetry Hook Recipes v0 for Browser, Chrome, VS Code, CLI wrapper, and provider-adapter recipe generation with dry-run commands, metadata-only privacy checks, and explicit non-claiming of native host integration.
+- Telemetry Native Host v0 for Browser/Chrome Native Messaging length-prefixed stdio, metadata-only event validation, local launcher generation, and Chrome host manifest generation.
- Review Waivers v0 for human warning acceptance with reviewer, reason, scope, expiry, and blocker-safe policy.
- Governed Permission Gates v0 for reviewer-approved network, file-write, and subprocess capabilities with scope, reason, expiry, evidence, and target-enforcement mapping.
- Runtime Permission Probes v0 for packaged target adapter checks, explicit native-enforcement flags, metadata fallback evidence, and residual permission risks.
+- Local Install Sync Preflight v0 so source-to-local and source-to-active install syncs rebuild the package first, run install simulation, enforce installer permission coverage, and refuse to copy files before any destructive sync when the package is not install-ready.
- Atlas Scope Policy v0 so examples, evolution snapshots, embedded generated skills, and validator fixtures remain visible in the full portfolio report without polluting release-actionable gates.
- Review Annotations v0 for reviewer comments tied to Review Studio gates, source/report paths, and optional line numbers; open blocker annotations now block the Review Studio decision.
-- Review Studio now reaches a clean `ready` decision for the root Meta Skill: `13 / 13` gates pass, world-class score `100`, no blockers, no warnings, and no review actions required.
+- Review Studio now avoids over-claiming release readiness when blind A/B adjudication, waiver handling, and world-class evidence are still pending: the root Meta Skill is in `review` with score `91`, no blockers, `3` warnings, and explicit actions for Output Lab reviewer adjudication, waiver handling, and world-class evidence completion.
+- Review Studio Output Lab actions now link directly to `reports/output_review_decisions.json`, so pending blind A/B reviewer decisions have a concrete template instead of only a general adjudication warning.
+- Output Review Adjudication now preserves blind-review integrity by hiding expected winners for pending or invalid reviewer decisions; answer keys are revealed only after a valid A/B decision exists for that case.
+- Output Review Import v0 now accepts reviewer JSON, JSONL, or CSV decision sources, recursively rejects raw prompt/output/transcript/message, credential, secret, token, and answer-key fields case-insensitively, writes canonical `reports/output_review_decisions.json`, and can run adjudication immediately without opening the answer key before valid decisions exist.
+- Provider Output Eval Runner v0 so `python3 scripts/yao.py output-exec --provider-runner openai` can collect real provider-backed model evidence through a reviewed OpenAI Responses API compatible runner instead of ad hoc shell glue.
+- Weekly SkillOps Curator v0 so daily SkillOps opportunities, Skill Atlas portfolio signals, release-lock state, and world-class evidence gaps roll up into a proposal-only weekly maintenance queue without source-file writes.
-This is still not the final world-class state. Target-native behavior contracts are now explicit and local output-eval command execution is wired, but deeper provider-native execution transforms, real client telemetry capture, installer integration, provider-backed model-executed output eval, real human adjudication decisions, and native runtime permission enforcement remain open.
+This is still not the final world-class state. Target-native behavior contracts are now explicit, VS Code / Copilot package metadata is auditable, provider-backed output-eval execution is wired through DeepSeek evidence, blind-review answers remain hidden until valid decisions exist, a provider-backed output runner exists, installer-level permission coverage is now locally enforced during install simulation and local install sync, opt-in `yao.py` CLI telemetry can capture metadata-only real run signals, external clients can now emit one validated metadata-only event into a local spool, hook recipes now make Browser/Chrome/VS Code/wrapper/provider-adapter integration commands auditable, a Browser/Chrome Native Messaging host can receive length-prefixed metadata events and generate a local launcher plus manifest, validated external JSONL imports can bring those non-CLI client signals into the same drift loop, and Skill Atlas now consumes aggregate drift reports. World-class intake now rejects non-canonical submission filenames and nested raw content or credential-like fields before ledger review. The preflight and submission kit now give operators repair rows, matrix state, artifact readiness, source blockers, and next actions for each pending world-class evidence key. Review Studio keeps pending human adjudication, waiver handling, and pending world-class evidence visible as warnings instead of treating them as a clean pass. Accepted external ledger evidence, real human adjudication decisions, installed platform-native client telemetry implementations, provider-native installer integration, and native runtime permission enforcement remain open.
## Coverage Matrix
| 2.0 Area | Current Evidence | Status |
| --- | --- | --- |
| Skill IR | `skill-ir/schema.json`, `skill-ir/examples/yao-meta-skill.json`, `scripts/export_skill_ir.py` | v0 landed |
-| Target Compiler | `scripts/compile_skill.py`, `reports/compiled_targets.md`, `tests/verify_compile_skill.py` | v0 landed |
-| Output Eval Lab | `evals/output/cases.jsonl`, `scripts/run_output_eval.py`, `scripts/run_output_execution.py`, `scripts/local_output_eval_runner.py`, `scripts/adjudicate_output_review.py`, `reports/output_quality_scorecard.md`, `reports/output_execution_runs.md`, `reports/output_blind_review_pack.md`, `reports/output_blind_answer_key.json`, `reports/output_review_adjudication.md` | v0 landed |
+| Target Compiler | `scripts/compile_skill.py`, `scripts/compile_skill_targets.py`, `reports/compiled_targets.md`, `tests/verify_compile_skill.py` | v0 landed |
+| Output Eval Lab | `evals/output/cases.jsonl`, `scripts/run_output_eval.py`, `scripts/run_output_execution.py`, `scripts/local_output_eval_runner.py`, `scripts/provider_output_eval_runner.py`, `scripts/import_output_review_decisions.py`, `scripts/output_review_privacy.py`, `scripts/adjudicate_output_review.py`, `reports/output_quality_scorecard.md`, `reports/output_execution_runs.md`, `reports/output_blind_review_pack.md`, `reports/output_blind_answer_key.json`, `reports/output_review_decisions.json`, `reports/output_review_adjudication.md` | v0 landed |
| Benchmark methodology | `reports/benchmark_methodology.md` | v0 landed |
+| Benchmark Reproducibility | `scripts/render_benchmark_reproducibility.py`, `reports/benchmark_reproducibility.md`, `tests/verify_benchmark_reproducibility.py` | v0 landed |
+| Skill OS 2.0 Audit | `scripts/render_skill_os2_audit.py`, `reports/skill_os2_audit.md`, `tests/verify_skill_os2_audit.py` | v0 landed |
+| World-Class Evidence Plan | `scripts/render_world_class_evidence_plan.py`, `reports/world_class_evidence_plan.md`, `tests/verify_world_class_evidence_plan.py` | v0 landed |
+| World-Class Evidence Ledger | `scripts/render_world_class_evidence_ledger.py`, `reports/world_class_evidence_ledger.md`, `tests/verify_world_class_evidence_ledger.py` | v0 landed |
+| World-Class Evidence Intake | `scripts/world_class_evidence_contract.py`, `scripts/world_class_human_evidence.py`, `scripts/output_review_privacy.py`, `scripts/world_class_provider_evidence.py`, `scripts/world_class_native_permission_evidence.py`, `scripts/world_class_native_telemetry_evidence.py`, `scripts/render_world_class_evidence_intake.py`, `evidence/world_class/intake.schema.json`, `tests/verify_world_class_evidence_intake.py` with canonical filename, source-artifact validation, recursive human decision privacy and answer-key validation, provider run-row validation, native permission target-row validation, native telemetry event-row validation, nested raw-field rejection, and real-submission answer-key leakage rejection | v0 landed |
+| World-Class Submission Kit | `scripts/prepare_world_class_submission_kit.py`, `scripts/world_class_submission_matrix.py`, `scripts/world_class_repair_checklist.py`, `scripts/world_class_submission_kit_rendering.py`, `tests/verify_world_class_submission_kit.py` with draft, artifact, source-check, next-action matrix evidence, machine-readable repair rows, and separated Markdown/HTML rendering | v0 landed |
| Runtime Conformance | `scripts/run_conformance_suite.py`, `reports/conformance_matrix.md` | v0 landed |
| Trust & Security | `scripts/trust_check.py`, `reports/security_trust_report.md`, `security/*.md` | v0 landed |
| Review Studio 2.0 | `scripts/render_review_studio.py`, `reports/review-studio.html`, `reports/review-studio.json` with per-warning `review_actions` | v0 landed |
| Review Studio Source Refs | `scripts/render_review_studio.py`, `tests/verify_review_studio.py`, `references/review-studio-method.md` | v0 landed |
| Review Annotations | `scripts/render_review_annotations.py`, `reports/review_annotations.md`, `tests/verify_review_annotations.py`, Review Studio annotation panel and blocker decision hook | v0 landed |
-| Skill Atlas | `scripts/build_skill_atlas.py`, `skill_atlas/catalog.json`, `skill_atlas/route_overlap_matrix.csv`, `reports/skill_atlas.html` | v0 landed |
+| Weekly SkillOps Curator | `scripts/render_weekly_curator_report.py`, `reports/skillops/weekly/2026-W25.md`, `tests/verify_weekly_curator.py`, proposal-only maintenance queue with no source writes | v0 landed |
+| Skill Atlas | `scripts/build_skill_atlas.py`, `skill_atlas/catalog.json`, `skill_atlas/route_overlap_matrix.csv`, `skill_atlas/drift_signals.json`, `reports/skill_atlas.html` | v0 landed |
| Registry & Distribution | `registry/*.schema.json`, `scripts/registry_audit.py`, `reports/registry_audit.md`, `registry/packages/yao-meta-skill.json` | v0 landed |
| Package Verification | `scripts/verify_package.py`, `reports/package_verification.md`, `tests/verify_package_verification.py` | v0 landed |
-| Install Simulation | `scripts/simulate_install.py`, `reports/install_simulation.md`, `tests/verify_install_simulation.py` | v0 landed |
+| Install Simulation | `scripts/simulate_install.py`, `reports/install_simulation.md`, `tests/verify_install_simulation.py` with installer permission coverage checks | v0 landed |
+| Local Install Sync Preflight | `scripts/sync_local_install.py`, `tests/verify_local_install_sync.py`, `Makefile` package-check prerequisites | v0 landed |
| Upgrade Check | `scripts/upgrade_check.py`, `reports/upgrade_check.md`, `tests/verify_upgrade_check.py` | v0 landed |
-| Telemetry & Drift | `scripts/render_adoption_drift_report.py`, `reports/adoption_drift_report.md`, `tests/verify_adoption_drift.py`, `references/telemetry-drift-method.md` | v0 landed |
+| Telemetry & Drift | `scripts/render_adoption_drift_report.py`, `scripts/yao_cli_telemetry.py`, `scripts/emit_telemetry_event.py`, `scripts/render_telemetry_hook_recipes.py`, `scripts/telemetry_native_host.py`, `scripts/import_telemetry_events.py`, `reports/adoption_drift_report.md`, `reports/telemetry_hook_recipes.md`, `tests/verify_adoption_drift.py`, `tests/verify_telemetry_emit.py`, `tests/verify_telemetry_hooks.py`, `tests/verify_telemetry_native_host.py`, `tests/verify_telemetry_import.py`, `tests/verify_yao_cli.py`, `references/telemetry-drift-method.md` | v0 landed |
| Review Waivers | `scripts/render_review_waivers.py`, `reports/review_waivers.md`, `tests/verify_review_waivers.py`, `references/review-waiver-method.md` | v0 landed |
| Governed Permission Gates | `security/permission_policy.json`, `scripts/trust_check.py`, `scripts/render_review_studio.py`, `tests/verify_trust_check.py`, `tests/verify_review_studio.py` | v0 landed |
| Runtime Permission Probes | `scripts/probe_runtime_permissions.py`, `reports/runtime_permission_probes.md`, `tests/verify_runtime_permission_probes.py`, `tests/verify_review_studio.py` | v0 landed |
| Atlas Scope Policy | `skill_atlas/policy.json`, `scripts/build_skill_atlas.py`, `tests/verify_skill_atlas.py` | v0 landed |
| Compiler from IR | Packager consumes compiled target contracts for compiler provenance, generated files, adapter modes, permissions, preserved semantics, warnings, and unsupported features | v0 landed |
-| Target Native Contracts | `scripts/compile_skill.py`, `scripts/cross_packager.py`, `reports/compiled_targets.md`, adapter snapshots, and generated target outputs carry native surface, activation policy, resource/script strategy, permission enforcement, install scope, review artifacts, fallback behavior, and unsupported native features | v0 landed |
+| Target Native Contracts | `scripts/compile_skill.py`, `scripts/compile_skill_targets.py`, `scripts/cross_packager.py`, `reports/compiled_targets.md`, adapter snapshots, and generated target outputs carry native surface, activation policy, resource/script strategy, permission enforcement, install scope, review artifacts, fallback behavior, and unsupported native features | v0 landed |
## Top Findings
### 1. Target-native behavior contracts are landed, but provider execution is still shallow
-`scripts/compile_skill.py` now reads Skill IR and emits target-specific contracts for OpenAI, Claude, generic, and Agent Skills compatible outputs. The packager embeds `compiler`, `compiled_contract`, `permission_contract`, `target_permission_contract`, `target_native_contract`, `target_transform`, warnings, and unsupported-feature notes in each adapter. The native contract makes the target surface, activation policy, resource strategy, script strategy, permission enforcement mode, install scope, review artifacts, fallback behavior, and unsupported native features reviewable instead of implicit.
+`scripts/compile_skill.py` now reads Skill IR and emits target-specific contracts for OpenAI, Claude, generic, Agent Skills compatible, and VS Code / Copilot outputs, with target platform models and contract builders isolated in `scripts/compile_skill_targets.py`. The packager embeds `compiler`, `compiled_contract`, `permission_contract`, `target_permission_contract`, `target_native_contract`, `target_transform`, warnings, and unsupported-feature notes in each adapter. The native contract makes the target surface, activation policy, resource strategy, script strategy, permission enforcement mode, install scope, review artifacts, fallback behavior, and unsupported native features reviewable instead of implicit.
Next move: deepen provider-native execution transforms so OpenAI, Claude, VS Code/Copilot, and generic packages can express and verify behavior through real client or installer capabilities, not only auditable contracts and fallback notes.
-### 2. Output eval now has command-runner evidence, but provider model execution is still open
+### 2. Output eval now has provider-backed model evidence, but human review is still open
-The v0 cases now cover five scenarios, including near-neighbor and file-backed governed package cases. Each run emits assertion grading, execution provenance, a blind A/B review pack that hides baseline vs with-skill labels, a separate answer key, and an adjudication report from reviewer choices. The current root execution report now runs all `10` variants through `scripts/local_output_eval_runner.py`, so timing and command-runner behavior are observed instead of relying only on static recorded fixtures. It still intentionally reports `0` provider-backed model-executed runs and `10` estimated token counts, because the local runner verifies the harness contract without pretending to be a model provider. The current root adjudication report intentionally records `0 / 5` human judgments pending instead of pretending human review happened. The next gap is provider-backed model-executed holdout runs with real provider timing/tokens and real reviewer decision records.
+The v0 cases now cover five scenarios, including near-neighbor and file-backed governed package cases. Each run emits assertion grading, execution provenance, a blind A/B review pack that hides baseline vs with-skill labels, a separate answer key, an importer for reviewer JSON/JSONL/CSV decision sources, and an adjudication report from reviewer choices. The current root execution report runs all `10` variants through `scripts/provider_output_eval_runner.py` against DeepSeek `deepseek-v4-flash`, so model execution, timing, and token usage are observed rather than estimated. The provider source checks now pass with `10` model-executed runs, `10` observed timing rows, `10` observed token rows, `0` estimated token rows, and no regressions, while the world-class ledger still requires an independent reviewer-approved submission before it counts as accepted evidence. The current root adjudication report intentionally records `0 / 5` human judgments pending instead of pretending human review happened.
-Next move: add provider-backed model execution for holdout cases and one real multi-file fixture with actual reviewer decisions.
+Next move: complete independent provider evidence ledger review, then record actual blind A/B reviewer decisions.
### 3. Review Studio is unified and now has reviewer actions plus annotations
-The Review Studio page aggregates intent, trigger, output, context, runtime, trust, permission approvals, runtime permission probes, atlas, operations-loop, reviewer waiver, reviewer annotations, registry, and release gates. It exposes current warnings directly and emits `review_actions` for each non-pass gate with a source-fix location, structured `source_refs`, reason, evidence path, and verification command. It now also loads `reports/review_annotations.json`, renders reviewer comments tied to gates and source/report paths, and blocks the page decision when any open blocker annotation exists. After adding root intent context, Atlas scope policy, metadata-only telemetry evidence, runtime permission probe evidence, action source refs, and an empty annotation ledger, the current root report is clean: decision `ready`, score `100`, `13` gates, `0` gate blockers, `0` gate warnings, `0` actions, and `0` open annotation blockers.
+The Review Studio page aggregates intent, trigger, output, context, runtime, trust, permission approvals, runtime permission probes, atlas, operations-loop, reviewer waiver, reviewer annotations, registry, world-class evidence, and release gates. It exposes current warnings directly and emits `review_actions` for each non-pass gate with a source-fix location, structured `source_refs`, reason, evidence path, and verification command. It now also loads `reports/review_annotations.json`, renders reviewer comments tied to gates and source/report paths, and blocks the page decision when any open blocker annotation exists. The current root report is intentionally not a clean pass: decision `review`, score `91`, `16` gates, `0` blockers, `3` warnings, `3` actions, and `0` open annotation blockers. The warnings are useful evidence: the automated Output Eval Lab has provider-backed execution but blind A/B adjudication still has `5` pending reviewer decisions, warning waivers still need reviewer handling, and the world-class ledger still has `4` pending external or human evidence entries, so the package should not claim fully reviewed or world-class status.
-Next move: add richer source-line anchors inside generated reports and record real reviewer annotations during the next human review pass.
+Next move: add richer source-line anchors inside generated reports, record real reviewer annotations during the next human review pass, and close the provider ledger-review, native-permission, real-client telemetry, and human review evidence entries in the world-class ledger.
-### 4. Multi-skill operation now has v0 coverage, but needs real telemetry
+### 4. Multi-skill operation now links Atlas with drift, but needs platform-native capture
-The new Skill Atlas can scan a workspace and report catalog, route overlap, dependency graph, stale skill, missing owner/review metadata, and no-route opportunities. It now also supports `skill_atlas/policy.json` so release gates distinguish actionable library skills from examples and fixtures while keeping full visibility. Adoption Drift v0 can record metadata-only local events, block raw prompt/output fields, summarize missed-trigger, bad-output, script-error, and review-overdue signals, and feed next iteration candidates into Review Studio. It still needs real client-side capture instead of manual CLI event recording.
+The new Skill Atlas can scan a workspace and report catalog, route overlap, dependency graph, stale skill, missing owner/review metadata, aggregate drift signals, and no-route opportunities. It now also supports `skill_atlas/policy.json` so release gates distinguish actionable library skills from examples and fixtures while keeping full visibility. Adoption Drift v0 can record metadata-only local events, block raw prompt/output fields, summarize missed-trigger, bad-output, script-error, and review-overdue signals, feed next iteration candidates into Review Studio, and publish aggregate drift input for Atlas. `yao.py` now adds opt-in automatic CLI run capture through `YAO_CLI_TELEMETRY=1` or `--record-cli-telemetry`, recording only `source=yao_cli`, normalized subcommand name, outcome, failure class, and timestamp. `telemetry-emit` gives Browser/Chrome/IDE/wrapper integrations a stable local hook for appending one normalized external event into `.yao/telemetry_spool/external_events.jsonl`, with dry-run validation and no raw content fields. `telemetry-hooks` renders Browser, Chrome, VS Code, CLI wrapper, and provider-adapter recipes with dry-run commands, import commands, trigger points, and explicit `native_auto_capture=false` caveats. `telemetry_native_host.py` now provides a tested Browser/Chrome Native Messaging host: it reads length-prefixed stdio JSON, rejects raw content fields, writes metadata-only events, and generates an operator-installable launcher plus Chrome host manifest. `telemetry-import` accepts already-sanitized external client JSONL or the emitted spool, validates the whole file before appending, supports dry-run, defaults `source=external`, and rejects any raw content or unknown metadata fields without mutating the local event stream. It still needs an installed real extension or provider client that sends production events to the native host.
-Next move: connect client activations and failure history so Atlas can rank stale or conflicting skills by real usage impact.
+Next move: install a real Browser/Chrome extension or provider client against the native host and record production metadata events, then let Atlas rank stale, drifting, or conflicting skills by real usage impact.
### 5. Trust report is structural, not full security review
-Trust v0 blocks obvious secrets and remote inline execution, and now makes checksum scope explicit: `package_sha256` is a stable source-contract digest that excludes generated reports, packages, and raw local telemetry; archive integrity is carried by Package Verification and Registry as `archive_sha256`. Registry v0 carries both checksums into distributable metadata. Package Verification v0 now checks generated manifests, target adapters, archive path safety, required source entries, and registry parity. Install Simulation v0 now extracts the generated archive into a temporary local skill root and verifies entrypoint, manifest, interface, reports, and adapters. Upgrade Check v0 now compares package baselines, recommends semver bumps, and blocks breaking changes without an adequate declared version. Trust Report now distinguishes CLI scripts from declared internal modules, so help-surface warnings focus on executable entrypoints. Network-capable scripts are now covered by `security/network_policy.json`, with `allowed_hosts` checked against HTTPS URL literals. Trust Report also executes `python3 scripts/name.py --help` for CLI scripts with `argparse`, recording pass/fail smoke evidence without executing scripts that lack a help surface. Target adapters now carry `permission_contract` and `target_permission_contract` fields, including network, file-write, subprocess, and interactive capability counts. Governed permission gates now require reviewer-approved scope, reason, expiry, evidence, and target-enforcement mapping in `security/permission_policy.json`. Runtime Permission Probes now verify that packaged adapters expose the target permission contract and explicitly report native-enforcement limits. The remaining trust gap is native runtime permission enforcement in real clients/installers.
+Trust v0 blocks obvious secrets and remote inline execution, and now makes checksum scope explicit: `package_sha256` is a stable source-contract digest that excludes generated reports, packages, and raw local telemetry; archive integrity is carried by Package Verification and Registry as `archive_sha256`. Registry v0 carries both checksums into distributable metadata. Package Verification v0 now checks generated manifests, target adapters, archive path safety, required source entries, and registry parity. Install Simulation v0 now extracts the generated archive into a temporary local skill root, verifies entrypoint, manifest, interface, reports, and adapters, then checks every packaged target/capability pair against the installed `security/permission_policy.json` approval and `target_enforcement` mapping. Local Install Sync Preflight now runs the same install simulation before syncing source into the disabled or active local install path, so a broken package or permission enforcement gap fails before stale files are removed or new files are copied. Upgrade Check v0 now compares package baselines, recommends semver bumps, and blocks breaking changes without an adequate declared version. Trust Report now distinguishes CLI scripts from declared internal modules, so help-surface warnings focus on executable entrypoints. Network-capable scripts are now covered by `security/network_policy.json`, with `allowed_hosts` checked against HTTPS URL literals. Trust Report also executes `python3 scripts/name.py --help` for CLI scripts with `argparse`, recording pass/fail smoke evidence without executing scripts that lack a help surface. Target adapters now carry `permission_contract` and `target_permission_contract` fields, including network, file-write, subprocess, and interactive capability counts. Governed permission gates now require reviewer-approved scope, reason, expiry, evidence, and target-enforcement mapping in `security/permission_policy.json`. Runtime Permission Probes now verify that packaged adapters expose the target permission contract and explicitly report native-enforcement limits. The remaining trust gap is native runtime permission enforcement in real clients/installers.
Next move: add real client or installer permission enforcement integration.
@@ -93,29 +132,34 @@ Next move: add real client or installer permission enforcement integration.
| Gate | Current Result |
| --- | --- |
-| Output Eval | `5` cases, with-skill pass rate `100`, baseline pass rate `0`, with file-backed, near-neighbor, boundary coverage, `10` local command-runner execution runs, `0` recorded fixture runs, `0` provider model-executed runs, `10` estimated token counts, `5` blind A/B review pairs, and `0 / 5` reviewer decisions pending |
+| Output Eval | `5` cases, DeepSeek provider-backed execution for `10 / 10` variants, with-skill pass rate `43.67`, baseline pass rate `23.67`, delta `20.0`, with file-backed, near-neighbor, boundary coverage, `10` command execution runs, `0` recorded fixture runs, `10` provider model-executed runs in root release evidence, `10` observed token rows, `0` estimated token rows, provider runner v0 available, `5` blind A/B review pairs, a generated `reports/output_review_decisions.json` template, `0 / 5` reviewer decisions pending, `0` answer keys revealed, and `5` pending answers hidden |
| Runtime Conformance | `5 / 5` targets passing |
-| Target Compiler | compiled target contracts generated for OpenAI, Claude, generic, and Agent Skills compatible outputs, including target permission contracts and target-native behavior contracts |
-| Trust | `0` secret findings, `1` pinned dependency file, `3` declared internal modules, `2 / 2` network-capable scripts covered by bounded host policy, `59 / 59` CLI help smoke checks passing across `62` scripts, source-contract hash scope explicit |
+| Target Compiler | `5 / 5` compiled target contracts generated for OpenAI, Claude, generic, Agent Skills compatible, and VS Code / Copilot outputs, including target permission contracts and target-native behavior contracts |
+| Trust | `0` secret findings, `1` pinned dependency file, `67` declared internal modules, `3 / 3` network-capable scripts covered by bounded host policy, `86 / 86` CLI help smoke checks passing across `153` scripts, source-contract hash scope explicit |
| Permission Governance | `3 / 3` required high-permission capabilities approved, `0` missing, `0` invalid, `0` expired |
-| Runtime Permission Probes | `3 / 3` target adapters probed, `0` native-enforcement adapters, `3` explicit metadata fallbacks, `3` residual risks retained for reviewer visibility |
-| Skill Atlas | `12` scanned skills, `1` actionable root skill, `0` actionable route collisions, `0` actionable owner gaps, `0` actionable stale skills, `24` scoped non-actionable issue signals retained for visibility |
+| Runtime Permission Probes | `4 / 4` target adapters probed, `0` native-enforcement adapters, `4` explicit metadata fallbacks, `4` residual risks retained for reviewer visibility |
+| Skill Atlas | `12` scanned skills, `1` actionable root skill, `1` telemetry report, `0` actionable route collisions, `0` actionable owner gaps, `0` actionable stale skills, `0` actionable drift signals, `24` scoped non-actionable issue signals retained for visibility |
| Registry Audit | package metadata generated with version, owner, license, source checksum, archive checksum, Skill IR provenance, and compatibility matrix |
-| Package Verification | `3 / 3` target adapters present, archive verified, `483` zip entries, `0` failures, `0` warnings |
-| Install Simulation | archive with `483` entries extracted into a local verification root, entrypoint/manifest/interface loaded, reports present, `3` adapters readable, `0` failures, `0` warnings |
+| Package Verification | `4 / 4` target adapters present, archive verified, `689` zip entries, `0` failures, `0` warnings |
+| Install Simulation | archive with `689` entries extracted into a local verification root, entrypoint/manifest/interface loaded, reports present, `4` adapters readable, `12` installer permission checks enforced, `0` permission failures, `0` failures, `0` warnings |
+| Local Install Sync Preflight | `make sync-local-install` and `make sync-active-install` rebuild the package first, then sync only after install simulation passes with `12` enforced installer permission checks and `0` permission failures |
| Upgrade Check | current package declares `minor` over the 1.0.0 baseline, recommended bump is `minor`, and release notes include added targets plus checksum changes |
-| Adoption Drift | `1` metadata-only activation event, adoption `100`, risk band `low`; raw `reports/telemetry_events.jsonl` is gitignored and blocked from zip packages |
-| Review Waivers | ledger generated; current release has no warning gates to waive, blockers remain non-waivable in v0 |
+| Adoption Drift | `1` metadata-only review event, `1` adoption sample, adoption `100`, risk band `low`; optional `yao.py` CLI capture, external client `telemetry-emit`, `5` `telemetry-hooks` recipes, Browser/Chrome native messaging host, and validated external JSONL import are available but off by default for reproducible release evidence; raw `reports/telemetry_events.jsonl` is gitignored and blocked from zip packages |
+| Review Waivers | ledger generated; current release has `2` warning gates that still need reviewer decision or a time-bounded waiver; blockers remain non-waivable in v0, and world-class evidence gaps must be closed with accepted evidence rather than waived as completed |
| Review Annotations | ledger generated; current release has `0` reviewer annotations and `0` open annotation blockers |
-| Review Studio | decision `ready`, world-class score `100`, `13` gates, `0` blockers, `0` warnings, `0` review actions, `0` open annotation blockers |
-| IR-first Packaging | `openai`, `claude`, and `generic` adapters include compiler contracts, permission contracts, target-native behavior contracts, IR provenance, and semantic parity checks |
-| Context Budget | initial load `987/1000`, under the production budget |
-| CI | `make ci-test` target count is now `56` after adding Review Annotations |
+| Review Studio | decision `review`, world-class score `91`, `16` gates, `0` blockers, `3` warnings, `3` review actions, `0` open annotation blockers |
+| Skill OS 2.0 Audit | `15` audited areas, with local foundation evidence separated from human-required and external-required gaps |
+| World-Class Evidence Plan | `4` remaining evidence tasks: `1` human-required and `3` external-required, all with runbooks, success checks, and privacy contracts |
+| World-Class Evidence Ledger | `4` evidence entries remain pending: `1` human-required and `3` external-required; anti-overclaim guards block planned work, metadata fallbacks, pending review, and local command runners from counting as completion |
+| Benchmark Reproducibility | local reproducibility ready with `25` required artifacts, `0` missing artifacts, `23` reproduction commands, and `3` disclosed failure cases; human review, native permission enforcement, native telemetry, and provider ledger approval remain explicit limitations |
+| IR-first Packaging | `openai`, `claude`, `generic`, and `vscode` adapters include compiler contracts, permission contracts, target-native behavior contracts, IR provenance, semantic parity checks, and install-scope notes where applicable |
+| Context Budget | initial load `998/1000`; deferred `512782/120000`; resource governance governed |
+| CI | `make ci-test` target count is `82` after the dedicated CLI world-class verifier split |
## Next Highest-Leverage Moves
-1. Deepen target-native behavior contracts into provider-native execution and installer integrations.
-2. Add native client or installer enforcement for approved high-permission capabilities.
-3. Expand Output Eval Lab from local command-runner evidence to provider-backed model holdout and real reviewer-adjudicated cases.
+1. Deepen target-native behavior contracts into provider-native execution and provider-native installer integrations.
+2. Add native client or installer runtime enforcement for approved high-permission capabilities.
+3. Complete independent provider ledger review, then record the current blind A/B decisions before claiming fully ready status.
4. Add real reviewer annotation records during the next human review pass.
-5. Connect Skill Atlas with real client telemetry and drift history.
+5. Install a real Browser/Chrome extension or provider client against `telemetry_native_host.py` and record production metadata events.
diff --git a/reports/skill-overview.html b/reports/skill-overview.html
index 6e3e2091..7448c1c8 100644
--- a/reports/skill-overview.html
+++ b/reports/skill-overview.html
@@ -307,8 +307,8 @@
}
.section-head {
display: grid;
- grid-template-columns: minmax(0, 246px) minmax(0, 1fr);
- gap: 36px;
+ grid-template-columns: minmax(0, 1fr);
+ gap: 22px;
align-items: start;
}
h2 {
@@ -330,12 +330,15 @@
line-height: 1.3;
margin-bottom: 10px;
}
- .section-head p {
+ .section-head > div:first-child p {
margin: 12px 0 0;
color: var(--muted);
- max-width: 43ch;
+ max-width: 68ch;
text-wrap: pretty;
}
+ .section-body {
+ margin-top: 24px;
+ }
.two-col, .metric-grid, .chart-grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
@@ -345,20 +348,28 @@
.quality-panels {
margin-top: 16px;
}
- .metrics-stack {
+ .metrics-flow {
display: grid;
- gap: 18px;
+ gap: 30px;
}
- .metrics-lead {
+ .metrics-primary {
display: grid;
- grid-template-columns: minmax(420px, 1.15fr) minmax(300px, 0.85fr);
- gap: 18px;
- align-items: stretch;
+ grid-template-columns: minmax(360px, 0.92fr) minmax(0, 1.08fr);
+ gap: 22px;
+ align-items: start;
+ }
+ .metrics-primary .chart-figure {
+ min-height: 0;
+ display: grid;
+ align-content: center;
+ }
+ .metrics-primary .chart-figure svg {
+ max-height: 520px;
}
.metrics-note {
display: grid;
align-content: start;
- gap: 16px;
+ gap: 14px;
}
.metrics-note p {
margin: 0;
@@ -395,6 +406,22 @@
grid-column: 2 / 4;
color: var(--muted);
line-height: 1.5;
+ overflow-wrap: break-word;
+ }
+ .metric-detail-section {
+ display: grid;
+ gap: 14px;
+ }
+ .detail-section-kicker {
+ display: inline-flex;
+ width: fit-content;
+ min-height: 28px;
+ align-items: center;
+ padding: 3px 8px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 12px;
}
.metric-status {
display: inline-flex;
@@ -406,9 +433,9 @@
color: var(--brand);
font-size: 12px;
}
- .metric-grid {
- grid-template-columns: repeat(2, minmax(0, 1fr));
- margin-top: 2px;
+ .metric-detail-grid {
+ grid-template-columns: repeat(2, minmax(min(100%, 420px), 1fr));
+ align-items: stretch;
}
.list, .compact-list, .step-list {
margin: 0;
@@ -419,8 +446,13 @@
.list li::marker, .compact-list li::marker, .step-list li::marker { color: var(--brand); }
.compact-list {
gap: 6px;
- font-size: 13px;
+ font-size: 13.5px;
color: var(--muted);
+ line-height: 1.55;
+ }
+ .compact-list li, .list li, .step-list li {
+ min-width: 0;
+ overflow-wrap: break-word;
}
.panel, .metric-card, .roadmap-item {
background: #ffffff;
@@ -428,21 +460,23 @@
border-radius: 8px;
padding: 22px;
box-shadow: 0 1px 2px rgba(27, 54, 93, 0.04);
+ min-width: 0;
}
.metric-card {
display: grid;
- grid-template-columns: 92px minmax(0, 1fr);
+ grid-template-columns: minmax(112px, 0.28fr) minmax(0, 1fr);
gap: 18px;
align-content: start;
min-height: 0;
}
.metric-card-head {
display: grid;
+ gap: 10px;
align-content: start;
}
.metric-card strong {
display: block;
- margin: 8px 0 10px;
+ margin: 0;
color: var(--brand);
font-family: var(--mono);
font-size: 2rem;
@@ -453,10 +487,20 @@
.metric-label {
color: var(--muted);
font-size: 13px;
+ line-height: 1.45;
+ overflow-wrap: break-word;
}
.metric-card-body {
min-width: 0;
}
+ .metric-card-body .compact-list {
+ gap: 7px;
+ max-width: 76ch;
+ }
+ .metric-card-body .compact-list li {
+ overflow-wrap: break-word;
+ word-break: normal;
+ }
.chart-figure {
margin: 0;
padding: 18px;
@@ -512,6 +556,120 @@
font-size: 12px;
}
td:first-child, th:first-child { width: 96px; }
+ .world-readiness {
+ margin-top: 16px;
+ display: grid;
+ gap: 16px;
+ }
+ .section-head > .world-readiness { grid-column: 1; }
+ .world-readiness-head {
+ display: grid;
+ grid-template-columns: minmax(0, 1fr) auto;
+ gap: 16px;
+ align-items: start;
+ }
+ .world-readiness-head p {
+ margin-top: 8px;
+ max-width: 72ch;
+ }
+ .world-status {
+ display: inline-flex;
+ align-items: center;
+ min-height: 30px;
+ padding: 4px 10px;
+ border-radius: 6px;
+ background: var(--brand-soft);
+ color: var(--brand);
+ font-size: 13px;
+ white-space: nowrap;
+ }
+ .evidence-kpis {
+ display: grid;
+ grid-template-columns: repeat(3, minmax(0, 1fr));
+ gap: 10px;
+ }
+ .evidence-kpi {
+ padding: 14px;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ background: var(--wash);
+ min-width: 0;
+ }
+ .evidence-kpi span {
+ display: block;
+ color: var(--muted);
+ font-size: 12px;
+ }
+ .evidence-kpi strong {
+ display: block;
+ margin-top: 4px;
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 1.5rem;
+ font-weight: 500;
+ line-height: 1;
+ font-variant-numeric: tabular-nums;
+ }
+ .evidence-kpi small {
+ display: block;
+ margin-top: 8px;
+ color: var(--muted);
+ line-height: 1.45;
+ overflow-wrap: break-word;
+ }
+ .evidence-list {
+ display: grid;
+ grid-template-columns: repeat(2, minmax(0, 1fr));
+ gap: 10px;
+ }
+ .evidence-item {
+ padding: 14px;
+ border: 1px solid var(--line-soft);
+ border-radius: 8px;
+ background: #ffffff;
+ min-width: 0;
+ }
+ .evidence-item span {
+ color: var(--brand);
+ font-family: var(--mono);
+ font-size: 11px;
+ text-transform: uppercase;
+ }
+ .evidence-item h4 {
+ margin: 4px 0 8px;
+ color: var(--ink);
+ font-size: 1rem;
+ font-weight: 500;
+ line-height: 1.3;
+ }
+ .evidence-item h5 {
+ margin: 12px 0 6px;
+ color: var(--brand);
+ font-size: 12px;
+ font-weight: 500;
+ }
+ .evidence-item p {
+ margin: 0;
+ max-width: none;
+ color: var(--muted);
+ overflow-wrap: break-word;
+ }
+ .blocked-checks {
+ margin: 0;
+ padding-left: 1.05em;
+ color: var(--muted);
+ font-size: 13px;
+ line-height: 1.45;
+ }
+ .blocked-checks li {
+ overflow-wrap: break-word;
+ }
+ .blocked-checks li::marker {
+ color: var(--brand);
+ }
+ .blocked-checks-empty {
+ font-size: 13px;
+ }
.roadmap {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
@@ -542,10 +700,12 @@
.report-mark { display: none; }
.report-nav a { min-width: 72px; }
.hero-grid, .section-head { grid-template-columns: 1fr; }
+ .section-head > .world-readiness { grid-column: auto; }
.hero-grid { gap: 24px; }
.score-strip { grid-template-columns: repeat(2, minmax(0, 1fr)); }
- .metrics-lead, .two-col, .metric-grid, .chart-grid, .roadmap { grid-template-columns: 1fr; }
- .metric-card { grid-template-columns: 86px minmax(0, 1fr); }
+ .metrics-primary, .metric-detail-grid, .two-col, .metric-grid, .chart-grid, .roadmap { grid-template-columns: 1fr; }
+ .evidence-list { grid-template-columns: 1fr; }
+ .metric-card { grid-template-columns: 1fr; }
h1 { font-size: 2.75rem; }
}
@media (max-width: 540px) {
@@ -569,6 +729,7 @@
.score-strip { grid-template-columns: 1fr; }
section { margin-top: 34px; padding-top: 34px; }
.panel, .metric-card, .roadmap-item, .chart-figure, .hero-card { padding: 18px; }
+ .world-readiness-head, .evidence-kpis { grid-template-columns: 1fr; }
.metric-card { grid-template-columns: 1fr; gap: 10px; }
.metric-card strong { margin-bottom: 2px; }
.metric-summary-list li { grid-template-columns: auto minmax(0, 1fr) auto; }
@@ -600,16 +761,16 @@
YAO Skill 生成审计报告YAO Skill Generation Audit
yao-meta-skill
技能审计报告Yao Meta Skill Audit Report
-
这份报告默认使用中文简体，把新 Skill 的定位、指标、原理、契约、质量、风险、资产和迭代路线整理为一份可审计的 HTML 报告。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
这份报告默认使用中文简体，把新 Skill 的定位、指标、原理、契约、质量、风险、资产和迭代路线整理为一份可审计的 HTML 报告。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
完整度Completeness100SKILL.md 已存在，是 Skill 的入口。SKILL.md exists and acts as the skill entrypoint.触发清晰Trigger clarity100frontmatter description 已存在，具备基础路由面。The frontmatter description exists, giving the skill a basic routing surface.证据充分Evidence depth100已生成 20 / 20 类报告证据。Generated 20 / 20 evidence report types.上下文成本Context cost42入口约 339 个词/字，references 约 16934 个词/字。Entrypoint is about 339 words/characters; references are about 16934.
@@ -621,9 +782,9 @@
作用定位Role
-
从工作流、提示词、对话记录、文档或笔记中创建、重构、评估和打包 agent skill；适用于新建 Skill、沉淀重复流程、改进现有 Skill、补充 eval 或团队复用打包。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。Turn one-off experience into a reusable, evaluable, and portable skill package.
- 交付流程把用户输入、生成的包体和可复用能力放在一条线上。
+ 交付流程把用户输入、生成的包体和可复用能力放在一条线上。The delivery flow places user input, generated package, and reusable capability on one path.
@@ -634,17 +795,24 @@
总览指标Metrics
分数来自本地文件和 reports 证据，缺失时明确标为证据不足。Scores are derived from package files and reports; missing inputs are shown as evidence gaps.
-
-
-
- 评分雷达展示结构完整度、触发边界、证据、维护和迁移的相对强弱。
-
-
指标判读Reading
-
先看雷达图判断能力短板，再看下方每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.
先看雷达图判断能力短板，再看每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.
触发强度：手动触发 + description 路由Trigger strength: Manual activation plus description-based routing
复用范围：跨平台Reuse scope: Cross-platform
@@ -673,15 +841,15 @@
说明入口、参考、脚本、评估和报告如何组成一个稳定闭环。Explains how entrypoint, references, scripts, evals, and reports form a stable loop.
-
分层结构展示入口、参考、脚本、评估和报告如何各司其职。
+
分层结构展示入口、参考、脚本、评估和报告如何各司其职。The layered structure shows how entrypoint, references, scripts, evals, and reports each carry a distinct role.
执行流程Execution Flow
-
判断请求是否应该沉淀为 Skill，并选择最轻量可靠的模式。Decide whether the request should become a skill and choose the lightest fit.
捕捉任务、输出、排除项、约束和质量标准。Capture job, output, exclusions, constraints, and standards.
运行参考扫描：先看外部 benchmark，再看用户材料，最后校验本地适配；只暴露不确定性或冲突。Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.
尽早写出 `description`，先测试路由质量，再扩展包体。Write the `description` early and test route quality before expanding the package.
只在确有价值时添加 output-risk、artifact-design、prompt-quality 和 system-model 报告。Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter.
+
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Capture job, output, exclusions, constraints, standards, and the lightest fit.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Write `description` early, test route quality, then add only earned folders and gates.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful.
调用方式How To Use
-
当你需要把工作流或笔记沉淀成结构精简、触发清晰且带必要 eval 的可复用 Skill 时使用 $yao-meta-skill。Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.
当用户请求与该 Skill 的触发描述匹配时使用。Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
当你需要把工作流或笔记沉淀成结构精简、触发清晰且带必要 eval 的可复用 Skill 时使用 $yao-meta-skill。Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.
当用户请求与该 Skill 的触发描述匹配时使用。Use this skill when the request matches the frontmatter description.
@@ -697,15 +865,15 @@
触发描述Trigger
-
该 Skill 的触发描述来自 SKILL.md frontmatter；默认中文报告先呈现能力边界，原始英文描述可切换到英文查看。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
+
该 Skill 的触发描述来自 SKILL.md frontmatter；默认中文报告先呈现能力边界，原始英文描述可切换到英文查看。Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。an existing skill directory that needs refactor, evaluation, packaging, or governance hardening
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。target platform requirements such as OpenAI, Claude, generic Agent Skills, or team distribution
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。benchmark references, local constraints, desired maturity tier, and review standards
输出结果Outputs
-
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.
结构化 Skill 目录，共 8 类关键资产。结构化 Skill 目录，共 8 类关键资产。
+
原始说明可切换到英文查看；默认中文报告保留结论与结构说明。A working skill package with lean SKILL.md, aligned agents/interface.yaml, justified references, scripts only when useful, eval evidence, reports, packaging metadata, and clear next iteration recommendations.
触发面保持精简，并锚定在 frontmatter description。The trigger surface stays lean and anchored in the frontmatter description.
保留并复用Keep
强项Strength
已生成 Skill IR，核心语义可先于平台打包被审查和迁移。Skill IR is generated so core semantics can be reviewed and migrated before platform packaging.
保留并复用Keep
强项Strength
已生成目标编译报告，可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。Target compilation evidence is generated to review how IR maps to OpenAI, Claude, generic, and other target contracts.
保留并复用Keep
缺口Gap
上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。Context cost needs improvement: Entrypoint is about 339 words/characters; references are about 16934.
纳入下一轮修复Fix next
执行证据Execution Evidence
-
变体运行：10变体运行：10
模型执行：0模型执行：0
记录样本：0记录样本：0
Token 估算：10Token 估算：10
+
变体运行：10Variant runs: 10
模型执行：10Model executions: 10
记录样本：0Recorded fixtures: 0
Token 估算：0Token estimates: 0
盲评审定Blind Adjudication
-
评审进度：0 / 5评审进度：0 / 5
待评审：5待评审：5
一致率：暂无一致率：暂无
非法决策：0非法决策：0
+
评审进度：0 / 5Review progress: 0 / 5
待评审：5Pending review: 5
一致率：暂无Agreement rate: not available yet
非法决策：0Invalid decisions: 0
@@ -753,14 +921,15 @@
提前暴露误触发、漂移、证据不足和迁移风险。Surfaces trigger, drift, evidence, and portability risks before the package grows.
- 风险热力图用影响程度和发生概率标出当前治理重点。
+ 风险热力图用影响程度和发生概率标出当前治理重点。The risk heatmap marks governance priorities by impact and probability.
先补证据和边界，再增加包体复杂度。Improve evidence and boundaries before adding package complexity.
+
世界证据World Evidence
世界级证据尚未完成：4 项待补，0 项已接受。World-class evidence is not complete: 4 pending, 0 accepted.
证据待补Evidence pending
待补证据Pending4仍需外部或人工证据接受。External or human evidence still needs acceptance.已接受Accepted0已通过 source check 与提交契约。Passed source checks and submission contract.源检查Source Checks11 / 19通过数 / 总检查数。Passed checks / total checks.
外部证据External evidence
提供商留出Provider Holdout
缺少真实 provider 模型运行和 token metadata。Missing a real provider model run and token metadata.
阻塞检查Blocked Checks
当前没有阻塞检查。No blocked checks right now.
人工证据Human evidence
人工盲评Human Adjudication
盲评 pair 仍待真实 reviewer 决策。Blind-review pairs still need real reviewer decisions.
阻塞检查Blocked Checks
无待判定No pending decisions
盲评完成Judgments complete
Reviewer metadataReviewer metadata
外部证据External evidence
原生权限Native Permission
原生 runtime enforcement 仍待目标客户端或外部安装器证明。Native runtime enforcement still needs target-client or external-installer proof.
阻塞检查Blocked Checks
原生执行Native enforcement
外部证据External evidence
原生遥测Native Telemetry
真实外部客户端 metadata-only 事件仍未导入。Real external-client metadata-only events have not been imported yet.
阻塞检查Blocked Checks
外部事件External events
采用样本Adoption sample
@@ -771,7 +940,7 @@
让 reviewer 快速确认关键文件、目录和资产分布。Lets reviewers confirm key files, directories, and asset distribution quickly.
- 资产分布图展示当前包体的文件和目录重心。
+ 资产分布图展示当前包体的文件和目录重心。The asset distribution chart shows where files and directories are concentrated.
路径Path
作用Role
类型Type
SKILL.md
Skill 入口文件Skill entrypoint
文件file
README.md
人类可读使用说明Human-readable usage guide
文件file
agents/interface.yaml
跨平台接口元数据Neutral interface metadata
文件file
manifest.json
生命周期与打包元数据Lifecycle and portability metadata
文件file
references
扩展指导与复用资料Extended guidance and reusable notes
目录folder
scripts
确定性脚本或本地工具Deterministic helpers or local tooling
目录folder
evals
触发与质量检查Trigger and quality checks
目录folder
reports
生成的证据与总结报告Generated evidence and overview artifacts
目录folder
@@ -787,8 +956,8 @@
把下一步升级收束为少数高价值动作。Keeps next iteration moves focused and actionable.
-
迭代时间线把下一步升级收束成少数可执行动作。
-
下一步 1Next 1
有选择地借鉴一个成熟模式Borrow one proven pattern on purpose
已经有公开 benchmark 对象，下一步应主动选择一个模式借鉴，而不是松散吸收所有做法。You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.
阅读 `obra/superpowers` 中最值得借鉴的模式。Read the strongest pattern from obra/superpowers.
先判断要借鉴的是方法、结构、执行方式还是可迁移性，并且第一轮只借鉴其中一个。Decide whether to borrow method, structure, execution, or portability, but only one of them first.
记录本轮不借鉴的内容，避免包体过重。Record what you will not borrow so the package stays light.
包体形态更清晰，也减少偶然过度设计。A cleaner package shape with less accidental over-design.
下一步 2Next 2
加固跨环境语义Harden portability semantics
这个 Skill 已经面向跨环境复用，因此早期就需要把契约语义说清楚。The skill already signals reuse across environments, so contract clarity matters early.
确认激活模式、执行上下文和信任假设。Confirm activation mode, execution context, and trust assumptions.
补充或复核非原生目标端的降级策略。Add or review degradation strategy for non-native targets.
至少打包一次 Skill，用来验证 adapter 预期。Package the skill once to verify adapter expectations.
跨环境复用更安全，目标漂移更少。Safer cross-environment reuse with less target drift.
下一步 3Next 3
建立迭代证据回路Create an iteration evidence loop
第一版之后，包体应该能说明改了什么以及为什么改。The package should show what changed and why after the first draft.
生成 HTML Skill 报告，并保持它与包体内容一致。Generate the HTML skill report and keep it aligned with the package.
记录参考扫描的取舍和非目标。Record reference scan choices and non-goals.
在继续增加文件前，明确记录下一轮迭代选择。Capture the next iteration choice explicitly before adding more files.
让下一位作者或评审者更容易接手。A clearer path for the next author or reviewer.
+
迭代时间线把下一步升级收束成少数可执行动作。The iteration timeline narrows the next upgrade into a few executable moves.
+
下一步 1Next 1
补齐世界证据Close world-class evidence
世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。World-class evidence still has 4 pending item(s); public completion claims must stay blocked.
补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。Close provider holdout evidence: Missing a real provider model run and token metadata.
补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。Close human adjudication evidence: Blind-review pairs still need real reviewer decisions.
按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。Close all 4 pending evidence item(s) in the ledger (external 3, human 1; 2 not shown here) and keep the claim guard pending.
全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。Only after the ledger accepts all external and human evidence should the public world-class claim move to review.
下一步 2Next 2
有选择地借鉴一个成熟模式Borrow one proven pattern on purpose
已经有公开 benchmark 对象，下一步应主动选择一个模式借鉴，而不是松散吸收所有做法。You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.
阅读 `obra/superpowers` 中最值得借鉴的模式。Read the strongest pattern from obra/superpowers.
先判断要借鉴的是方法、结构、执行方式还是可迁移性，并且第一轮只借鉴其中一个。Decide whether to borrow method, structure, execution, or portability, but only one of them first.
记录本轮不借鉴的内容，避免包体过重。Record what you will not borrow so the package stays light.
包体形态更清晰，也减少偶然过度设计。A cleaner package shape with less accidental over-design.
下一步 3Next 3
加固跨环境语义Harden portability semantics
这个 Skill 已经面向跨环境复用，因此早期就需要把契约语义说清楚。The skill already signals reuse across environments, so contract clarity matters early.
确认激活模式、执行上下文和信任假设。Confirm activation mode, execution context, and trust assumptions.
补充或复核非原生目标端的降级策略。Add or review degradation strategy for non-native targets.
至少打包一次 Skill，用来验证 adapter 预期。Package the skill once to verify adapter expectations.
跨环境复用更安全，目标漂移更少。Safer cross-environment reuse with less target drift.
diff --git a/reports/skill-overview.json b/reports/skill-overview.json
index 41de1c7a..a6034e28 100644
--- a/reports/skill-overview.json
+++ b/reports/skill-overview.json
@@ -3,7 +3,7 @@
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"display_name": "Yao Meta Skill",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"maturity": "governed",
"updated_at": "2026-03-31",
"core_value": "把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。",
@@ -11,13 +11,15 @@
"deliverables": [
"SKILL.md",
"agents/interface.yaml",
- "reports/skill-ir.json",
+ "skill-ir/examples/yao-meta-skill.json",
"reports/compiled_targets.md",
"reports/output_quality_scorecard.md",
"reports/output_execution_runs.md",
"reports/output_blind_review_pack.md",
- "reports/output_blind_answer_key.json",
+ "reports/output_review_kit.md",
"reports/output_review_adjudication.md",
+ "reports/benchmark_reproducibility.md",
+ "reports/output_blind_answer_key.json",
"reports/conformance_matrix.md",
"reports/security_trust_report.md",
"reports/runtime_permission_probes.md",
@@ -28,8 +30,11 @@
"reports/upgrade_check.md",
"reports/adoption_drift_report.md",
"reports/review_waivers.md",
+ "reports/world_class_evidence_plan.md",
+ "reports/world_class_evidence_ledger.md",
"reports/review_annotations.md",
"reports/review-studio.html",
+ "reports/skill-interpretation.html",
"reports/skill-overview.html"
],
"flow": [
@@ -75,7 +80,7 @@
"label": "可维护性",
"score": 100,
"reasons": [
- "SKILL.md 约 365 个词/字。",
+ "SKILL.md 约 339 个词/字。",
"入口文件保持克制，可维护性较好。",
"references/ 已承载扩展指导。",
"scripts/ 已承载确定性逻辑。",
@@ -96,7 +101,7 @@
"label": "上下文成本",
"score": 42,
"reasons": [
- "入口约 365 个词/字，references 约 14247 个词/字。",
+ "入口约 339 个词/字，references 约 16934 个词/字。",
"分数越高代表上下文成本越低。",
"上下文成本偏高，建议压缩入口或拆分 references。"
]
@@ -150,7 +155,7 @@
},
"contract_boundary": {
"trigger": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"activation": "manual",
"execution": "inline",
"shell": "bash"
@@ -178,7 +183,7 @@
"boundary_cards": [
{
"label": "Owned",
- "body": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse."
+ "body": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
},
{
"label": "Adjacent",
@@ -200,7 +205,7 @@
"已生成 Output Review Adjudication，可记录盲评决策、一致率和待评审项。"
],
"gaps": [
- "上下文成本需要补强：入口约 365 个词/字，references 约 14247 个词/字。"
+ "上下文成本需要补强：入口约 339 个词/字，references 约 16934 个词/字。"
],
"recommendations": [
"先改触发边界，再扩展工作流。",
@@ -210,10 +215,10 @@
"artifact_design": {
"design_system": "metric editorial",
"highlights": [
- "Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.",
- "Name the working directory and required inputs for commands.",
- "Mark destructive, networked, or external side-effect operations.",
- "Prefer the smallest runnable snippet over broad framework scaffolding."
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
]
},
"prompt_quality": {
@@ -265,7 +270,7 @@
"name": "包体膨胀风险",
"impact": 2,
"probability": 1,
- "signal": "SKILL.md 约 365 个词/字。",
+ "signal": "SKILL.md 约 339 个词/字。",
"response": "先补证据和边界，再增加包体复杂度。"
},
{
@@ -397,6 +402,78 @@
"Do not silently broaden the skill into adjacent jobs just because the examples are nearby."
]
},
+ "world_class_readiness": {
+ "ready": false,
+ "decision": "evidence-pending",
+ "entry_count": 4,
+ "pending_count": 4,
+ "accepted_count": 0,
+ "external_pending_count": 3,
+ "human_pending_count": 1,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "conclusion_zh": "世界级证据尚未完成：4 项待补，0 项已接受。",
+ "conclusion_en": "World-class evidence is not complete: 4 pending, 0 accepted.",
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label_zh": "提供商留出",
+ "label_en": "Provider Holdout",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "缺少真实 provider 模型运行和 token metadata。",
+ "summary_en": "Missing a real provider model run and token metadata.",
+ "blocked_checks": []
+ },
+ {
+ "key": "human-adjudication",
+ "label_zh": "人工盲评",
+ "label_en": "Human Adjudication",
+ "category": "human",
+ "category_zh": "人工证据",
+ "category_en": "Human evidence",
+ "status": "pending",
+ "summary_zh": "盲评 pair 仍待真实 reviewer 决策。",
+ "summary_en": "Blind-review pairs still need real reviewer decisions.",
+ "blocked_checks": [
+ "No pending decisions",
+ "Judgments complete",
+ "Reviewer metadata"
+ ]
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label_zh": "原生权限",
+ "label_en": "Native Permission",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "原生 runtime enforcement 仍待目标客户端或外部安装器证明。",
+ "summary_en": "Native runtime enforcement still needs target-client or external-installer proof.",
+ "blocked_checks": [
+ "Native enforcement"
+ ]
+ },
+ {
+ "key": "native-client-telemetry",
+ "label_zh": "原生遥测",
+ "label_en": "Native Telemetry",
+ "category": "external",
+ "category_zh": "外部证据",
+ "category_en": "External evidence",
+ "status": "pending",
+ "summary_zh": "真实外部客户端 metadata-only 事件仍未导入。",
+ "summary_en": "Real external-client metadata-only events have not been imported yet.",
+ "blocked_checks": [
+ "External events",
+ "Adoption sample"
+ ]
+ }
+ ]
+ },
"package_assets": {
"entries": [
{
@@ -427,28 +504,28 @@
"path": "references",
"label": "Extended guidance and reusable notes",
"kind": "folder",
- "file_count": 32
+ "file_count": 35
},
{
"path": "scripts",
"label": "Deterministic helpers or local tooling",
"kind": "folder",
- "file_count": 66
+ "file_count": 153
},
{
"path": "evals",
"label": "Trigger and quality checks",
"kind": "folder",
- "file_count": 28
+ "file_count": 29
},
{
"path": "reports",
"label": "Generated evidence and overview artifacts",
"kind": "folder",
- "file_count": 170
+ "file_count": 235
}
],
- "file_count": 300,
+ "file_count": 456,
"folder_count": 4,
"distribution": [
{
@@ -469,24 +546,35 @@
},
{
"label": "references",
- "value": 32
+ "value": 35
},
{
"label": "scripts",
- "value": 66
+ "value": 153
},
{
"label": "evals",
- "value": 28
+ "value": 29
},
{
"label": "reports",
- "value": 170
+ "value": 235
}
]
},
"iteration_roadmap": {
"items": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
{
"title": "Borrow one proven pattern on purpose",
"why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
@@ -506,16 +594,6 @@
"Package the skill once to verify adapter expectations."
],
"unlocks": "Safer cross-environment reuse with less target drift."
- },
- {
- "title": "Create an iteration evidence loop",
- "why": "The package should show what changed and why after the first draft.",
- "actions": [
- "Generate the HTML skill report and keep it aligned with the package.",
- "Record reference scan choices and non-goals.",
- "Capture the next iteration choice explicitly before adding more files."
- ],
- "unlocks": "A clearer path for the next author or reviewer."
}
]
},
@@ -555,17 +633,17 @@
"name": "yao-meta-skill",
"title": "Yao Meta Skill",
"display_name": "Yao Meta Skill",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"logic_steps": [
- "Decide whether the request should become a skill and choose the lightest fit.",
- "Capture job, output, exclusions, constraints, and standards.",
- "Run reference scan: external benchmarks first, user references second, local fit third; surface only uncertainty or conflict.",
- "Write the `description` early and test route quality before expanding the package.",
- "Add output-risk, artifact-design, prompt-quality, and system-model reports only when they matter."
+ "For one-off/no reusable process: `Do not create a skill`; `near-neighbor`; require `repeated use` + `reusable output contract`.",
+ "Capture job, output, exclusions, constraints, standards, and the lightest fit.",
+ "Scan references in order: external benchmark, user source, local fit; surface only uncertainty or conflict.",
+ "Write `description` early, test route quality, then add only earned folders and gates.",
+ "Add output-risk, artifact-design, prompt-quality, system-model, and next directions only when useful."
],
"usage_steps": [
"Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals.",
- "Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse."
+ "Use this skill when the request matches: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
],
"package_map": [
{
@@ -596,25 +674,25 @@
"path": "references",
"label": "Extended guidance and reusable notes",
"kind": "folder",
- "file_count": 32
+ "file_count": 35
},
{
"path": "scripts",
"label": "Deterministic helpers or local tooling",
"kind": "folder",
- "file_count": 66
+ "file_count": 153
},
{
"path": "evals",
"label": "Trigger and quality checks",
"kind": "folder",
- "file_count": 28
+ "file_count": 29
},
{
"path": "reports",
"label": "Generated evidence and overview artifacts",
"kind": "folder",
- "file_count": 170
+ "file_count": 235
}
],
"strengths": [
@@ -632,7 +710,7 @@
"用户说出类似需求时：Use $yao-meta-skill to turn my workflow or notes into a reusable skill with lean structure, clear triggering, and the right evals."
],
"trigger_contract": {
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"activation": "manual",
"execution": "inline",
"shell": "bash"
@@ -676,6 +754,17 @@
}
],
"roadmap": [
+ {
+ "title": "补齐世界证据",
+ "why": "世界级证据仍有 4 项待补；公开完成态 claim 必须继续保持阻塞。",
+ "actions": [
+ "补齐提供商留出证据：缺少真实 provider 模型运行和 token metadata。",
+ "补齐人工盲评证据：盲评 pair 仍待真实 reviewer 决策。",
+ "按 ledger 总量继续补齐 4 项待补证据（外部 3 项、人工 1 项；未展开 2 项），并保持 claim guard 为 pending 状态。"
+ ],
+ "unlocks": "全部外部/人工证据被 ledger 接受后，才能进入公开 world-class claim 复核。",
+ "source": "world_class_evidence_ledger"
+ },
{
"title": "Borrow one proven pattern on purpose",
"why": "You already have public benchmark objects. The next gain is to choose one pattern intentionally instead of absorbing everything loosely.",
@@ -695,16 +784,6 @@
"Package the skill once to verify adapter expectations."
],
"unlocks": "Safer cross-environment reuse with less target drift."
- },
- {
- "title": "Create an iteration evidence loop",
- "why": "The package should show what changed and why after the first draft.",
- "actions": [
- "Generate the HTML skill report and keep it aligned with the package.",
- "Record reference scan choices and non-goals.",
- "Capture the next iteration choice explicitly before adding more files."
- ],
- "unlocks": "A clearer path for the next author or reviewer."
}
],
"cards": [],
@@ -716,7 +795,8 @@
"benchmark_highlights": [],
"skill_ir": {
"schema_version": "2.0.0",
- "target_count": 4,
+ "source_path": "skill-ir/examples/yao-meta-skill.json",
+ "target_count": 5,
"trigger_samples": 8,
"output_eval_cases": 3
},
@@ -724,8 +804,8 @@
"ok": true,
"schema_version": "1.0",
"summary": {
- "target_count": 4,
- "pass_count": 4,
+ "target_count": 5,
+ "pass_count": 5,
"warn_count": 0,
"block_count": 0,
"failure_count": 0,
@@ -790,6 +870,22 @@
],
"unsupported_features": [],
"warnings": []
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "degradation_strategy": "agent-skills-source-with-vscode-notes",
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "permission_enforcement": "client-or-workspace-trust",
+ "generated_files": [
+ "targets/vscode/adapter.json",
+ "targets/vscode/README.md"
+ ],
+ "unsupported_features": [
+ "VS Code installation scope is documented but not installed by this compiler"
+ ],
+ "warnings": []
}
],
"failures": [],
@@ -814,14 +910,14 @@
"case_count": 5,
"variant_run_count": 10,
"command_executed_count": 10,
- "model_executed_count": 0,
+ "model_executed_count": 10,
"recorded_fixture_count": 0,
"timing_observed_count": 10,
- "token_observed_count": 0,
- "token_estimated_count": 10,
- "baseline_pass_rate": 0.0,
- "with_skill_pass_rate": 100.0,
- "delta": 100.0,
+ "token_observed_count": 10,
+ "token_estimated_count": 0,
+ "baseline_pass_rate": 23.67,
+ "with_skill_pass_rate": 43.67,
+ "delta": 20.0,
"regression_count": 0,
"failure_count": 0,
"gate_pass": true
@@ -830,9 +926,19 @@
"mode": "command",
"command": [
"python3",
- "scripts/local_output_eval_runner.py"
+ "scripts/provider_output_eval_runner.py",
+ "--provider",
+ "deepseek",
+ "--model",
+ "deepseek-v4-flash",
+ "--api-format",
+ "chat-completions",
+ "--thinking",
+ "disabled",
+ "--api-key-env",
+ "DEEPSEEK_API_KEY"
],
- "timeout_seconds": 30.0
+ "timeout_seconds": 180.0
},
"failures": []
},
@@ -846,6 +952,31 @@
"pair_count": 5,
"answer_key_separate": true
},
+ "output_review_kit": {
+ "ok": true,
+ "summary": {
+ "case_count": 5,
+ "ready_for_adjudication_count": 0,
+ "pending_decision_count": 5,
+ "invalid_decision_count": 0,
+ "reviewer_metadata_present": false,
+ "answer_key_hidden": true,
+ "answer_key_path_exposed": false,
+ "ready_to_run_adjudication": false,
+ "failure_count": 0
+ },
+ "artifacts": {
+ "reviewer_kit_json": "reports/output_review_kit.json",
+ "reviewer_kit_markdown": "reports/output_review_kit.md",
+ "reviewer_kit_html": "reports/output_review_kit.html",
+ "blind_pack_json": "reports/output_blind_review_pack.json",
+ "blind_pack_markdown": "reports/output_blind_review_pack.md",
+ "decisions": "reports/output_review_decisions.json",
+ "adjudication_json": "reports/output_review_adjudication.json",
+ "adjudication_markdown": "reports/output_review_adjudication.md"
+ },
+ "failures": []
+ },
"output_review_adjudication": {
"ok": true,
"summary": {
@@ -855,14 +986,77 @@
"agreement_count": 0,
"disagreement_count": 0,
"invalid_decision_count": 0,
+ "answer_revealed_count": 0,
+ "pending_answer_hidden_count": 5,
"agreement_rate": null,
"needs_review": true,
- "failure_count": 0
+ "reviewer_metadata_present": false,
+ "reason_required": true,
+ "blind_review_attested": false,
+ "blind_review_completed_before_answer_key": false,
+ "answer_key_not_opened_before_decisions": false,
+ "raw_content_excluded_attested": true,
+ "reviewer_reason_required_attested": true,
+ "ready_for_human_evidence": false,
+ "failure_count": 0,
+ "reviewer_checklist_count": 5,
+ "reviewer_checklist_pending_count": 5,
+ "reviewer_checklist_invalid_count": 0,
+ "reviewer_checklist_ready_count": 0
},
"reviewer": "",
"reviewed_at": "",
"failures": []
},
+ "benchmark_reproducibility": {
+ "ok": true,
+ "summary": {
+ "reproducibility_ready": true,
+ "release_lock_ready": true,
+ "methodology_complete": true,
+ "required_artifact_count": 25,
+ "missing_artifact_count": 0,
+ "evidence_bundle_sha256": "55a6c33c12e14d6507d4a5d4d4cf1268db35fb323d523e8b23ce9c73aedf8db0",
+ "source_contract_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "output_case_count": 5,
+ "failure_disclosure_count": 3,
+ "command_count": 23,
+ "command_executed_count": 10,
+ "timing_observed_count": 10,
+ "model_executed_count": 10,
+ "token_observed_count": 10,
+ "human_review_complete": false,
+ "provider_evidence_complete": true,
+ "world_class_ready": false,
+ "world_class_open_gap_count": 3,
+ "world_class_task_count": 4,
+ "world_class_ledger_pending_count": 4,
+ "world_class_source_check_count": 19,
+ "world_class_source_pass_count": 11,
+ "world_class_source_blocked_count": 8,
+ "beta_test_ready": true,
+ "beta_test_blocker_count": 0,
+ "beta_test_deferred_evidence_count": 4,
+ "public_claim_ready": false,
+ "public_claim_blocker_count": 3,
+ "working_tree_dirty": true,
+ "changed_file_count": 29,
+ "source_tree_dirty": false,
+ "source_changed_file_count": 0,
+ "generated_tree_dirty": true,
+ "generated_changed_file_count": 29
+ },
+ "commit": "e79050ed12d057811d7d13ab7d8bcc5a06af76d4",
+ "missing_artifacts": [],
+ "limitations": [
+ "The git commit and dirty flags are generation-time context; release lock is blocked by source changes, while generated evidence artifacts are tracked separately.",
+ "Provider-backed model holdout source evidence is complete, but ledger acceptance still requires a valid independently reviewed submission packet.",
+ "Pending blind-review decisions are visible but do not count as human adjudication.",
+ "World-class readiness remains false until external and human evidence gaps close.",
+ "Beta/public testing may proceed without human blind-review only when wording avoids superiority, fully-reviewed, or world-class claims."
+ ]
+ },
"runtime_conformance": {
"target_count": 5,
"pass_count": 5,
@@ -871,14 +1065,22 @@
"runtime_permissions": {
"ok": true,
"summary": {
- "target_count": 3,
- "pass_count": 3,
+ "target_count": 4,
+ "pass_count": 4,
"fail_count": 0,
"native_enforcement_count": 0,
- "metadata_fallback_count": 3,
- "residual_risk_count": 3,
+ "metadata_fallback_count": 4,
+ "residual_risk_count": 4,
"required_capability_count": 3,
- "failure_count": 0
+ "failure_count": 0,
+ "installer_enforcement_source_status": "present",
+ "installer_enforcement_target_count": 4,
+ "installer_enforcement_pass_count": 4,
+ "installer_permission_enforced_count": 12,
+ "installer_permission_failure_count": 0,
+ "installer_permission_capability_count": 3,
+ "world_class_native_evidence_ready": false,
+ "installer_enforcement_ready": true
},
"expected_capabilities": [
"file_write",
@@ -915,33 +1117,43 @@
"residual_risks": [
"Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
]
+ },
+ {
+ "target": "vscode",
+ "status": "pass",
+ "assurance": "metadata-fallback-explicit",
+ "native_enforcement": false,
+ "metadata_fallback_explicit": true,
+ "residual_risks": [
+ "Client-native permission enforcement is not provided by this target; installer or operator must honor metadata."
+ ]
}
],
"failures": []
},
"trust_security": {
- "scanned_files": 150,
- "script_count": 66,
- "internal_module_count": 7,
+ "scanned_files": 245,
+ "script_count": 153,
+ "internal_module_count": 67,
"secret_findings": 0,
"dependency_files": [
"requirements-ci.txt"
],
- "network_script_count": 2,
- "network_policy_covered_count": 2,
+ "network_script_count": 3,
+ "network_policy_covered_count": 3,
"network_policy_missing_count": 0,
- "file_write_script_count": 47,
+ "file_write_script_count": 73,
"permission_required_count": 3,
"permission_approved_count": 3,
"permission_missing_count": 0,
"permission_invalid_count": 0,
"permission_expired_count": 0,
- "help_smoke_checked_count": 59,
+ "help_smoke_checked_count": 86,
"help_smoke_failed_count": 0,
"interactive_script_count": 0,
"package_hash_scope": "source-contract-without-generated-reports",
- "package_hash_file_count": 150,
- "package_sha256": "742dc905b0d221c4f336e741d168861edaf7b75e97fe1e89f7d454c1cabf928f"
+ "package_hash_file_count": 245,
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257"
},
"skill_atlas": {
"skill_count": 12,
@@ -954,6 +1166,9 @@
"actionable_stale_count": 0,
"shared_resource_count": 0,
"no_route_opportunity_count": 0,
+ "telemetry_report_count": 1,
+ "drift_signal_count": 0,
+ "actionable_drift_signal_count": 0,
"non_actionable_issue_count": 24
},
"registry_distribution": {
@@ -962,12 +1177,13 @@
"schema_version": "2.0",
"name": "yao-meta-skill",
"version": "1.1.0",
- "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
+ "description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution.",
"targets": [
"openai",
"claude",
"generic",
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
],
"maturity": "governed",
"owner": "Yao Team",
@@ -975,8 +1191,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
- "package_sha256": "742dc905b0d221c4f336e741d168861edaf7b75e97fe1e89f7d454c1cabf928f",
- "archive_sha256": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9"
+ "package_sha256": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2"
},
"compatibility": {
"openai": "pass",
@@ -1007,7 +1223,7 @@
},
"distribution": {
"archive_verified": true,
- "archive_sha256": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9",
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
@@ -1020,11 +1236,11 @@
"package_verification": {
"ok": true,
"summary": {
- "target_count": 3,
- "adapter_count": 3,
+ "target_count": 4,
+ "adapter_count": 4,
"archive_present": true,
- "archive_sha256": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9",
- "archive_entry_count": 488,
+ "archive_sha256": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2",
+ "archive_entry_count": 689,
"failure_count": 0,
"warning_count": 0
},
@@ -1035,12 +1251,16 @@
"ok": true,
"summary": {
"archive_present": true,
- "archive_entry_count": 488,
+ "archive_entry_count": 696,
"archive_extracted": true,
"entrypoint_loaded": true,
"manifest_loaded": true,
"interface_loaded": true,
- "adapter_count": 3,
+ "adapter_count": 4,
+ "installer_permission_enforced_count": 12,
+ "installer_permission_failure_count": 0,
+ "permission_target_count": 4,
+ "permission_capability_count": 3,
"install_root_is_temp": false,
"failure_count": 0,
"warning_count": 0
@@ -1054,7 +1274,7 @@
"declared_bump": "minor",
"recommended_bump": "minor",
"breaking_change_count": 0,
- "added_target_count": 1,
+ "added_target_count": 2,
"metadata_change_count": 1,
"checksum_change_count": 2,
"failure_count": 0,
@@ -1063,7 +1283,8 @@
"upgrade_diff": {
"name_changed": false,
"added_targets": [
- "agent-skills-compatible"
+ "agent-skills-compatible",
+ "vscode"
],
"removed_targets": [],
"compatibility_changes": [
@@ -1090,25 +1311,25 @@
{
"field": "description",
"from": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes.",
- "to": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse."
+ "to": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use for skill creation, reusable workflow packaging, skill improvement, evals, and team-ready distribution."
}
],
"checksum_changes": [
{
"field": "archive_sha256",
"from": "",
- "to": "7bc83805501d24cf8e5932ef76e0454100398d5a73652e7440be47984d0cf4a9"
+ "to": "7db44f059844b71930ea8194a72ba1a316116ca970620de5550d60616e8f23c2"
},
{
"field": "package_sha256",
"from": "0000000000000000000000000000000000000000000000000000000000000000",
- "to": "742dc905b0d221c4f336e741d168861edaf7b75e97fe1e89f7d454c1cabf928f"
+ "to": "5a99ef3133b0148cd5bbc1cd85fad4830bf34ca2bbac709f1e1fea2f810b1257"
}
]
},
"release_notes": [
"Recommended version bump: minor.",
- "Added targets: agent-skills-compatible.",
+ "Added targets: agent-skills-compatible, vscode.",
"Package or archive checksum changed; reviewers should verify package artifacts before release.",
"Package verification evidence: reports/package_verification.md."
],
@@ -1119,13 +1340,14 @@
"ok": true,
"summary": {
"event_count": 1,
- "activation_count": 1,
- "accepted_count": 1,
+ "adoption_sample_count": 0,
+ "activation_count": 0,
+ "accepted_count": 0,
"edited_count": 0,
"rejected_count": 0,
"missed_count": 0,
"failed_count": 0,
- "adoption_rate": 100.0,
+ "adoption_rate": 0,
"missed_trigger_count": 0,
"wrong_trigger_count": 0,
"bad_output_count": 0,
@@ -1134,9 +1356,13 @@
"review_overdue_count": 0,
"risk_band": "low",
"event_types": {
- "skill_activation": 1
+ "review_event": 1
},
- "failure_types": {}
+ "failure_types": {},
+ "source_types": {
+ "manual": 1
+ },
+ "command_counts": {}
},
"next_iteration_candidates": [],
"privacy_contract": {
@@ -1169,25 +1395,68 @@
"expired_count": 0,
"invalid_count": 0,
"covered_gate_count": 0,
- "covered_gate_keys": []
+ "covered_gate_keys": [],
+ "waiver_candidate_count": 2,
+ "waiverable_open_count": 1,
+ "non_waivable_count": 1
},
"policy": {
"blocker_waivers_allowed": false,
"minimum_reason_chars": 20,
"expires_required": true,
- "known_gate_keys": [
+ "review_studio_gate_keys": [
+ "architecture-maintainability",
"context-budget",
"intent-canvas",
"operations-loop",
"output-lab",
"permission-gates",
"permission-runtime",
+ "python-compat",
+ "registry-audit",
+ "release-notes",
+ "review-waivers",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report",
+ "world-class-evidence"
+ ],
+ "known_gate_keys": [
+ "architecture-maintainability",
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "python-compat",
"registry-audit",
"release-notes",
"runtime-matrix",
"skill-atlas",
"trigger-lab",
"trust-report"
+ ],
+ "waiverable_gate_keys": [
+ "architecture-maintainability",
+ "context-budget",
+ "intent-canvas",
+ "operations-loop",
+ "output-lab",
+ "permission-gates",
+ "permission-runtime",
+ "python-compat",
+ "registry-audit",
+ "release-notes",
+ "runtime-matrix",
+ "skill-atlas",
+ "trigger-lab",
+ "trust-report"
+ ],
+ "non_waivable_gate_keys": [
+ "review-waivers",
+ "world-class-evidence"
]
},
"failures": [],
@@ -1209,6 +1478,743 @@
"annotations": [],
"failures": []
},
+ "world_class_evidence_plan": {
+ "ok": true,
+ "summary": {
+ "audit_decision": "continue-iteration",
+ "world_class_ready": false,
+ "audit_world_class_ready": false,
+ "ready_to_claim_world_class": false,
+ "ledger_completion_required": true,
+ "evidence_requirement_count": 4,
+ "task_count": 4,
+ "human_task_count": 1,
+ "external_task_count": 3,
+ "review_task_count": 0,
+ "decision": "collect-external-evidence"
+ },
+ "tasks": [
+ {
+ "key": "provider-holdout",
+ "label": "Provider Holdout",
+ "status": "pass",
+ "category": "external",
+ "owner": "operator with provider credentials",
+ "current": "model-executed 10; token-observed 10",
+ "objective": "Collect at least one provider-backed output-eval holdout run with model, timing, and token metadata.",
+ "runbook": [
+ "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60",
+ "For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/output_execution_runs.json summary.model_executed_count > 0",
+ "reports/output_execution_runs.json summary.timing_observed_count > 0",
+ "reports/output_execution_runs.json summary.token_observed_count > 0",
+ "reports/skill_os2_audit.json item provider-holdout status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_execution_runs.json",
+ "reports/output_execution_runs.md",
+ "reports/skill_os2_audit.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/provider-holdout.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not commit provider credentials or environment dumps.",
+ "The output execution report records output hashes and aggregate run metadata, not raw provider prompts."
+ ],
+ "audit_next_action": "Run provider-backed holdout cases with real credentials and commit only aggregate evidence."
+ },
+ {
+ "key": "human-adjudication",
+ "label": "Human Adjudication",
+ "status": "human_required",
+ "category": "human",
+ "owner": "human reviewer",
+ "current": "0/5 decisions; pending 5",
+ "objective": "Record real blind A/B reviewer decisions before claiming human output review completion.",
+ "runbook": [
+ "python3 scripts/yao.py output-review-kit --write-template",
+ "Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.",
+ "python3 scripts/adjudicate_output_review.py --write-template",
+ "Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.",
+ "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "python3 scripts/yao.py output-review",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/output_review_adjudication.json summary.pending_count == 0",
+ "reports/output_review_adjudication.json summary.judgment_count == summary.pair_count",
+ "reports/output_review_adjudication.json summary.invalid_decision_count == 0",
+ "reports/output_review_adjudication.json summary.reviewer_metadata_present is true",
+ "reports/output_review_adjudication.json summary.blind_review_attested is true",
+ "reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text",
+ "reports/output_review_adjudication.json summary.ready_for_human_evidence is true",
+ "reports/skill_os2_audit.json item human-adjudication status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_blind_review_pack.md",
+ "reports/output_review_kit.md",
+ "reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json",
+ "reports/output_review_adjudication.md",
+ "scripts/import_output_review_decisions.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/human-adjudication.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Reviewer decisions should not include raw user data or private customer detail.",
+ "Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.",
+ "The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.",
+ "The adjudication evidence stores prompt_sha256 instead of raw prompt text.",
+ "The decision and adjudication artifacts preserve blind_pack_sha256 so reviewers can audit exactly which pack was judged.",
+ "Keep the answer key separate until after decisions are recorded."
+ ],
+ "audit_next_action": "Record real A/B choices, reviewer metadata, and blind-review attestation, then regenerate adjudication."
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "status": "external_required",
+ "category": "external",
+ "owner": "target client or installer integrator",
+ "current": "native-enforced targets 0; installer-enforced targets 4",
+ "objective": "Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.",
+ "runbook": [
+ "Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.",
+ "Update the generated target adapter only when the guard is actually enforced by that target.",
+ "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip",
+ "python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation",
+ "python3 scripts/yao.py runtime-permissions . --package-dir dist",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/runtime_permission_probes.json summary.native_enforcement_count > 0",
+ "reports/runtime_permission_probes.json summary.failure_count == 0",
+ "reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence",
+ "reports/skill_os2_audit.json item native-permission-enforcement status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "dist/targets/*/adapter.json",
+ "reports/runtime_permission_probes.json",
+ "reports/runtime_permission_probes.md",
+ "reports/install_simulation.json",
+ "reports/install_simulation.md",
+ "security/permission_policy.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not mark native_enforcement true for metadata-only fallbacks.",
+ "Keep residual risks visible for targets that still rely on operator enforcement."
+ ],
+ "audit_next_action": "Integrate a real target-client or external installer runtime guard before claiming native permission enforcement."
+ },
+ {
+ "key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "status": "external_required",
+ "category": "external",
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "current": "external source events 0; adoption samples 0",
+ "objective": "Import production metadata-only events from a real external client into the local drift loop.",
+ "runbook": [
+ "python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///",
+ "Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
+ "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl",
+ "python3 scripts/yao.py skill-atlas --workspace-root .",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "success_checks": [
+ "reports/adoption_drift_report.json summary.source_types.external > 0",
+ "reports/adoption_drift_report.json summary.adoption_sample_count > 0",
+ "reports/skill_os2_audit.json item native-client-telemetry status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/adoption_drift_report.json",
+ "reports/adoption_drift_report.md",
+ "reports/telemetry_hook_recipes.json",
+ "scripts/telemetry_native_host.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Telemetry must remain metadata-only and local-first.",
+ "Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field."
+ ],
+ "audit_next_action": "Install a real client against the native host and import production metadata-only events."
+ }
+ ],
+ "source_audit": {
+ "json": "reports/skill_os2_audit.json",
+ "markdown": "reports/skill_os2_audit.md",
+ "open_gap_count": 3
+ }
+ },
+ "world_class_evidence_ledger": {
+ "ok": true,
+ "summary": {
+ "ledger_entry_count": 4,
+ "source_accepted_count": 1,
+ "accepted_count": 0,
+ "pending_count": 4,
+ "human_pending_count": 1,
+ "external_pending_count": 3,
+ "submitted_entry_count": 0,
+ "reviewer_approved_submission_count": 0,
+ "missing_submission_count": 4,
+ "invalid_submission_count": 0,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "submitted_but_pending_count": 0,
+ "source_accepted_without_valid_submission_count": 1,
+ "overclaim_guard_active": true,
+ "ready_to_claim_world_class": false,
+ "decision": "evidence-pending"
+ },
+ "entries": [
+ {
+ "key": "provider-holdout",
+ "label": "Provider Holdout",
+ "category": "external",
+ "owner": "operator with provider credentials",
+ "status": "pending",
+ "source_status": "pass",
+ "source_accepted": true,
+ "current": "model-executed 10; token-observed 10",
+ "objective": "Collect at least one provider-backed output-eval holdout run with model, timing, and token metadata.",
+ "runbook": [
+ "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60",
+ "For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "provider-backed model run",
+ "observed timing",
+ "observed token metadata"
+ ],
+ "success_checks": [
+ "reports/output_execution_runs.json summary.model_executed_count > 0",
+ "reports/output_execution_runs.json summary.timing_observed_count > 0",
+ "reports/output_execution_runs.json summary.token_observed_count > 0",
+ "reports/skill_os2_audit.json item provider-holdout status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_execution_runs.json",
+ "reports/output_execution_runs.md",
+ "reports/skill_os2_audit.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/provider-holdout.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not commit provider credentials or environment dumps.",
+ "The output execution report records output hashes and aggregate run metadata, not raw provider prompts."
+ ],
+ "observed_state": {
+ "model_executed_count": 10,
+ "timing_observed_count": 10,
+ "token_observed_count": 10,
+ "accepted": true
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Provider model run",
+ "field": "model_executed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Run provider-backed output-exec with real credentials."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Timing observed",
+ "field": "timing_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should record timing metadata."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Token usage observed",
+ "field": "token_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should return non-estimated token usage."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 3,
+ "source_blocked_count": 0,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/provider-holdout.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Run provider-backed holdout cases with real credentials and commit only aggregate evidence."
+ },
+ {
+ "key": "human-adjudication",
+ "label": "Human Adjudication",
+ "category": "human",
+ "owner": "human reviewer",
+ "status": "pending",
+ "source_status": "human_required",
+ "source_accepted": false,
+ "current": "0/5 decisions; pending 5",
+ "objective": "Record real blind A/B reviewer decisions before claiming human output review completion.",
+ "runbook": [
+ "python3 scripts/yao.py output-review-kit --write-template",
+ "Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.",
+ "python3 scripts/adjudicate_output_review.py --write-template",
+ "Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.",
+ "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "python3 scripts/yao.py output-review",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real reviewer identity",
+ "blind A/B decisions",
+ "answer key unopened until decisions exist"
+ ],
+ "success_checks": [
+ "reports/output_review_adjudication.json summary.pending_count == 0",
+ "reports/output_review_adjudication.json summary.judgment_count == summary.pair_count",
+ "reports/output_review_adjudication.json summary.invalid_decision_count == 0",
+ "reports/output_review_adjudication.json summary.reviewer_metadata_present is true",
+ "reports/output_review_adjudication.json summary.blind_review_attested is true",
+ "reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text",
+ "reports/output_review_adjudication.json summary.ready_for_human_evidence is true",
+ "reports/skill_os2_audit.json item human-adjudication status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/output_blind_review_pack.md",
+ "reports/output_review_kit.md",
+ "reports/output_review_decisions.json",
+ "reports/output_review_adjudication.json",
+ "reports/output_review_adjudication.md",
+ "scripts/import_output_review_decisions.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/human-adjudication.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Reviewer decisions should not include raw user data or private customer detail.",
+ "Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.",
+ "The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.",
+ "The adjudication evidence stores prompt_sha256 instead of raw prompt text.",
+ "The decision and adjudication artifacts preserve blind_pack_sha256 so reviewers can audit exactly which pack was judged.",
+ "Keep the answer key separate until after decisions are recorded."
+ ],
+ "observed_state": {
+ "pair_count": 5,
+ "judgment_count": 0,
+ "pending_count": 5,
+ "invalid_decision_count": 0,
+ "answer_revealed_count": 0,
+ "reviewer_metadata_present": false,
+ "reason_required": true,
+ "blind_review_attested": false,
+ "raw_content_excluded_attested": true,
+ "reviewer_reason_required_attested": true,
+ "ready_for_human_evidence": false,
+ "raw_content_allowed": false,
+ "raw_content_path_count": 0,
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Review pairs exist",
+ "field": "pair_count",
+ "expected": ">0",
+ "actual": 5,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Generate the blind A/B review pack."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No pending decisions",
+ "field": "pending_count",
+ "expected": "==0",
+ "actual": 5,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record a reviewer choice and reason for every pair."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Judgments complete",
+ "field": "judgment_count",
+ "expected": "==pair_count",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Every pair needs one valid human judgment."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No invalid decisions",
+ "field": "invalid_decision_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Fix malformed winner/confidence entries."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reviewer metadata",
+ "field": "reviewer_metadata_present",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record reviewer and reviewed_at before adjudication can count."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reason required",
+ "field": "reason_required",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Keep reason mandatory for every imported or direct reviewer decision."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Blind review attested",
+ "field": "blind_review_attested",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content attested",
+ "field": "raw_content_excluded_attested",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Human evidence ready",
+ "field": "ready_for_human_evidence",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."
+ }
+ ],
+ "source_check_count": 10,
+ "source_pass_count": 5,
+ "source_blocked_count": 5,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/human-adjudication.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Record real A/B choices, reviewer metadata, and blind-review attestation, then regenerate adjudication."
+ },
+ {
+ "key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "category": "external",
+ "owner": "target client or installer integrator",
+ "status": "pending",
+ "source_status": "external_required",
+ "source_accepted": false,
+ "current": "native-enforced targets 0; installer-enforced targets 4",
+ "objective": "Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.",
+ "runbook": [
+ "Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.",
+ "Update the generated target adapter only when the guard is actually enforced by that target.",
+ "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip",
+ "python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation",
+ "python3 scripts/yao.py runtime-permissions . --package-dir dist",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real target client or external installer runtime guard",
+ "native enforcement flag or externally accepted guard proof",
+ "residual risk retained for fallback targets"
+ ],
+ "success_checks": [
+ "reports/runtime_permission_probes.json summary.native_enforcement_count > 0",
+ "reports/runtime_permission_probes.json summary.failure_count == 0",
+ "reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence",
+ "reports/skill_os2_audit.json item native-permission-enforcement status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "dist/targets/*/adapter.json",
+ "reports/runtime_permission_probes.json",
+ "reports/runtime_permission_probes.md",
+ "reports/install_simulation.json",
+ "reports/install_simulation.md",
+ "security/permission_policy.json",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Do not mark native_enforcement true for metadata-only fallbacks.",
+ "Keep residual risks visible for targets that still rely on operator enforcement."
+ ],
+ "observed_state": {
+ "native_enforcement_count": 0,
+ "metadata_fallback_count": 4,
+ "installer_enforcement_pass_count": 4,
+ "installer_permission_failure_count": 0,
+ "installer_enforcement_ready": true,
+ "residual_risk_count": 4,
+ "failure_count": 0,
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Native enforcement",
+ "field": "native_enforcement_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Collect real target-client or external runtime guard proof."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Probe failures",
+ "field": "failure_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Runtime permission probes must stay clean."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Installer support",
+ "field": "installer_enforcement_ready",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Installer enforcement is supporting evidence, not native proof."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 2,
+ "source_blocked_count": 1,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/native-permission-enforcement.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Integrate a real target-client or external installer runtime guard before claiming native permission enforcement."
+ },
+ {
+ "key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "category": "external",
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "pending",
+ "source_status": "external_required",
+ "source_accepted": false,
+ "current": "external source events 0; adoption samples 0",
+ "objective": "Import production metadata-only events from a real external client into the local drift loop.",
+ "runbook": [
+ "python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///",
+ "Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
+ "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl",
+ "python3 scripts/yao.py skill-atlas --workspace-root .",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "provenance_requirements": [
+ "real external client source",
+ "metadata-only event",
+ "local-first import path"
+ ],
+ "success_checks": [
+ "reports/adoption_drift_report.json summary.source_types.external > 0",
+ "reports/adoption_drift_report.json summary.adoption_sample_count > 0",
+ "reports/skill_os2_audit.json item native-client-telemetry status becomes pass"
+ ],
+ "evidence_artifacts": [
+ "reports/adoption_drift_report.json",
+ "reports/adoption_drift_report.md",
+ "reports/telemetry_hook_recipes.json",
+ "scripts/telemetry_native_host.py",
+ "evidence/world_class/intake.schema.json",
+ "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "reports/world_class_evidence_intake.json",
+ "reports/world_class_evidence_intake.md"
+ ],
+ "privacy_contract": [
+ "Telemetry must remain metadata-only and local-first.",
+ "Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field."
+ ],
+ "observed_state": {
+ "external_source_events": 0,
+ "adoption_sample_count": 0,
+ "raw_content_allowed": false,
+ "risk_band": "low",
+ "accepted": false
+ },
+ "source_checklist": [
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "External events",
+ "field": "external_source_events",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Import at least one metadata-only event from a real client."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Adoption sample",
+ "field": "adoption_sample_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Telemetry must include adoption outcome evidence."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Telemetry must stay metadata-only."
+ }
+ ],
+ "source_check_count": 3,
+ "source_pass_count": 1,
+ "source_blocked_count": 2,
+ "submission_state": {
+ "status": "missing",
+ "path": "evidence/world_class/submissions/native-client-telemetry.json",
+ "artifact_ref_count": 0,
+ "attested_real_evidence": false,
+ "privacy_contract_satisfied": false,
+ "ledger_reviewer_approved": false,
+ "ledger_reviewer": "",
+ "ledger_reviewed_at": "",
+ "ledger_counts_as_completion": false
+ },
+ "anti_overclaim": {
+ "planned_work_counts_as_evidence": false,
+ "metadata_fallback_counts_as_native_enforcement": false,
+ "pending_review_counts_as_human_decision": false,
+ "local_command_runner_counts_as_provider_model": false
+ },
+ "next_action": "Install a real client against the native host and import production metadata-only events."
+ }
+ ],
+ "source_plan": {
+ "json": "reports/world_class_evidence_plan.json",
+ "markdown": "reports/world_class_evidence_plan.md",
+ "task_count": 4,
+ "evidence_requirement_count": 4
+ }
+ },
"synthesis_highlights": [
"Borrow progressive disclosure: keep the entrypoint lean and move depth into references or scripts.",
"Borrow a review checkpoint wherever trust matters more than raw speed.",
@@ -1217,10 +2223,10 @@
"artifact_design": {
"design_system": "metric editorial",
"highlights": [
- "Execution-focused technical artifact with environment assumptions, copyable commands, expected outputs, and side effects made explicit.",
- "Name the working directory and required inputs for commands.",
- "Mark destructive, networked, or external side-effect operations.",
- "Prefer the smallest runnable snippet over broad framework scaffolding."
+ "Metric-first dashboard with stable dimensions, short labels, visible deltas, and narrative callouts only where they change interpretation.",
+ "Avoid paragraph-heavy table cells.",
+ "Keep charts tied to one analytical question each.",
+ "Preserve stable color meaning across metrics and entities."
]
},
"prompt_quality": {
@@ -1249,7 +2255,8 @@
"targets": [
"openai",
"claude",
- "generic"
+ "generic",
+ "vscode"
],
"maturity_tier": "governed",
"skill_archetype": "governed",
diff --git a/reports/skill_atlas.html b/reports/skill_atlas.html
index 0c14b4a4..196b1c40 100644
--- a/reports/skill_atlas.html
+++ b/reports/skill_atlas.html
@@ -22,22 +22,28 @@
Skill Atlas
-
Portfolio-level review for route overlap, stale ownership, shared resources, and no-route opportunities.
+
Portfolio-level review for route overlap, stale ownership, shared resources, telemetry drift, and no-route opportunities.
Skills12
Actionable1
Route Collisions0
Owner Gaps0
Stale Skills0
+
Drift Signals0
No-Route Opportunities0
Actionable Issues
No blocking portfolio issues detected.
+
+
No-Route Opportunities
+
Missed-trigger telemetry and explicit failure cases become candidate routing work without storing raw prompts or outputs.
+
No no-route opportunities detected.
+
Full Portfolio Counts
-
All scanned skills remain visible: 5 total route collisions, 9 total owner gaps, and 10 total stale signals.
+
All scanned skills remain visible: 5 total route collisions, 9 total owner gaps, 10 total stale signals, and 0 telemetry drift signals.
This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.
Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.
Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
+ copy to artifact_refs: false
+
+
+
submission-ref rows are the paths expected in artifact_refs; supporting-evidence rows stay available for audit context.
+
+
+
+
+
Phase Queue
+
Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later source collection. Phase queue rows remain procedural and never count as accepted evidence.
+
+
+
+ #20 · blocked
+
Unblock access
+
+
+
Phase
unblock-access
+
Rows
4 / 4 blocked
+
Owners
Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
Environment variables are displayed only as set or not-set; secret values are never printed.
Human-required and external-required states are operator work, not accepted evidence.
The world-class ledger remains the only source of truth for ready_to_claim_world_class.
+
+
+
+
diff --git a/reports/world_class_evidence_preflight.json b/reports/world_class_evidence_preflight.json
new file mode 100644
index 00000000..9b3b2372
--- /dev/null
+++ b/reports/world_class_evidence_preflight.json
@@ -0,0 +1,2038 @@
+{
+ "schema_version": "1.0",
+ "ok": true,
+ "generated_at": "2026-06-17",
+ "skill_dir": ".",
+ "summary": {
+ "evidence_item_count": 4,
+ "precheck_count": 14,
+ "precheck_pass_count": 9,
+ "precheck_missing_count": 1,
+ "precheck_optional_count": 1,
+ "precheck_human_required_count": 1,
+ "precheck_external_required_count": 2,
+ "collection_ready_count": 1,
+ "collection_blocked_count": 3,
+ "source_check_count": 19,
+ "source_pass_count": 11,
+ "source_blocked_count": 8,
+ "repair_checklist_count": 12,
+ "repair_blocked_count": 12,
+ "repair_ready_count": 0,
+ "repair_phase_counts": {
+ "unblock-access": 4,
+ "collect-source": 8
+ },
+ "next_repair_action_id": "human-adjudication-precheck-human-reviewer",
+ "next_repair_phase": "unblock-access",
+ "next_repair_owner": "human reviewer",
+ "next_repair_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "repair_counts_as_completion": false,
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 12,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false,
+ "pending_count": 4,
+ "ready_to_claim_world_class": false,
+ "credential_value_exposed": false,
+ "preflight_counts_as_evidence": false,
+ "decision": "collection-preflight-blocked"
+ },
+ "status_counts": {
+ "blocked": 3,
+ "ready-for-human-review": 1
+ },
+ "precheck_status_counts": {
+ "pass": 9,
+ "missing": 1,
+ "optional": 1,
+ "human-required": 1,
+ "external-required": 2
+ },
+ "items": [
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Provider Holdout",
+ "category": "external",
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "ledger_status": "pending",
+ "intake_readiness": "awaiting-submission",
+ "review_state": "awaiting-submission",
+ "source_accepted": true,
+ "prechecks": [
+ {
+ "evidence_key": "provider-holdout",
+ "key": "output-cases",
+ "label": "Output eval cases",
+ "kind": "file",
+ "required": true,
+ "next_action": "Keep output holdout cases available before provider execution.",
+ "secret_value_redacted": false,
+ "path": "evals/output/cases.jsonl",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-runner",
+ "label": "Provider runner",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the provider runner instead of the local command runner for model-backed evidence.",
+ "secret_value_redacted": false,
+ "path": "scripts/provider_output_eval_runner.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-api-key",
+ "label": "Provider credential",
+ "kind": "env_any",
+ "required": true,
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "secret_value_redacted": true,
+ "env_any": [
+ "OPENAI_API_KEY",
+ "DEEPSEEK_API_KEY"
+ ],
+ "status": "missing",
+ "actual": "not-set",
+ "set_count": 0,
+ "set_envs": []
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-model",
+ "label": "Provider model",
+ "kind": "env",
+ "required": false,
+ "next_action": "Optionally set YAO_OUTPUT_EVAL_MODEL, or pass --provider-model for the selected provider.",
+ "secret_value_redacted": false,
+ "env": "YAO_OUTPUT_EVAL_MODEL",
+ "status": "optional",
+ "actual": "not-set",
+ "default": "provider-specific model, or pass --provider-model"
+ }
+ ],
+ "source_checklist": [
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Provider model run",
+ "field": "model_executed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Run provider-backed output-exec with real credentials."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Timing observed",
+ "field": "timing_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should record timing metadata."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Token usage observed",
+ "field": "token_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should return non-estimated token usage."
+ }
+ ],
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "submission_path": "evidence/world_class/submissions/provider-holdout.json",
+ "template_path": "evidence/world_class/templates/provider-holdout.intake.json",
+ "commands": {
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard ."
+ },
+ "submission_kit": {
+ "prepare_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions",
+ "prefill_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "output_dir": "evidence/world_class/submissions",
+ "draft_path": "evidence/world_class/submissions/provider-holdout.json",
+ "template_path": "evidence/world_class/templates/provider-holdout.intake.json",
+ "artifact_role_contract": {
+ "schema_version": "1.0",
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 1,
+ "submission_ref_ready_count": 1,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "roles": [
+ {
+ "role": "submission-ref",
+ "label": "Submission refs",
+ "copy_to_artifact_refs": true,
+ "description": "Rows marked submission-ref are the aggregate paths expected in artifact_refs."
+ },
+ {
+ "role": "supporting-evidence",
+ "label": "Supporting evidence",
+ "copy_to_artifact_refs": false,
+ "description": "Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs."
+ }
+ ]
+ },
+ "drafts_count_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false
+ },
+ "runbook": [
+ "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60",
+ "For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "repair_checklist": [
+ {
+ "action_id": "provider-holdout-precheck-provider-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "provider-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env_any precheck is missing.",
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-provider-api-key",
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "provider-holdout-precheck-provider-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "provider-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env_any precheck is missing.",
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Human Adjudication",
+ "category": "human",
+ "owner": "human reviewer",
+ "status": "ready-for-human-review",
+ "ledger_status": "pending",
+ "intake_readiness": "awaiting-submission",
+ "review_state": "awaiting-submission",
+ "source_accepted": false,
+ "prechecks": [
+ {
+ "evidence_key": "human-adjudication",
+ "key": "review-kit",
+ "label": "Blind review kit",
+ "kind": "file",
+ "required": true,
+ "next_action": "Open the blind review kit and record real reviewer choices with required rationale.",
+ "secret_value_redacted": false,
+ "path": "reports/output_review_kit.html",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "decision-template",
+ "label": "Decision template",
+ "kind": "file",
+ "required": true,
+ "next_action": "Import real A/B decisions with reviewer, reviewed_at, winner_variant, confidence, reason, and truthful blind-review attestation via `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`.",
+ "secret_value_redacted": false,
+ "path": "reports/output_review_decisions.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "decision-importer",
+ "label": "Decision importer",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the importer to reject raw content fields and normalize reviewer decisions before adjudication.",
+ "secret_value_redacted": false,
+ "path": "scripts/import_output_review_decisions.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "human-reviewer",
+ "label": "Human reviewer",
+ "kind": "human",
+ "required": true,
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "secret_value_redacted": false,
+ "status": "human-required",
+ "actual": "external-human-action"
+ }
+ ],
+ "source_checklist": [
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Review pairs exist",
+ "field": "pair_count",
+ "expected": ">0",
+ "actual": 5,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Generate the blind A/B review pack."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No pending decisions",
+ "field": "pending_count",
+ "expected": "==0",
+ "actual": 5,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record a reviewer choice and reason for every pair."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Judgments complete",
+ "field": "judgment_count",
+ "expected": "==pair_count",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Every pair needs one valid human judgment."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No invalid decisions",
+ "field": "invalid_decision_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Fix malformed winner/confidence entries."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reviewer metadata",
+ "field": "reviewer_metadata_present",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record reviewer and reviewed_at before adjudication can count."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reason required",
+ "field": "reason_required",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Keep reason mandatory for every imported or direct reviewer decision."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Blind review attested",
+ "field": "blind_review_attested",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content attested",
+ "field": "raw_content_excluded_attested",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Human evidence ready",
+ "field": "ready_for_human_evidence",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."
+ }
+ ],
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "submission_path": "evidence/world_class/submissions/human-adjudication.json",
+ "template_path": "evidence/world_class/templates/human-adjudication.intake.json",
+ "commands": {
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard ."
+ },
+ "submission_kit": {
+ "prepare_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions",
+ "prefill_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "output_dir": "evidence/world_class/submissions",
+ "draft_path": "evidence/world_class/submissions/human-adjudication.json",
+ "template_path": "evidence/world_class/templates/human-adjudication.intake.json",
+ "artifact_role_contract": {
+ "schema_version": "1.0",
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 8,
+ "supporting_evidence_ready_count": 8,
+ "roles": [
+ {
+ "role": "submission-ref",
+ "label": "Submission refs",
+ "copy_to_artifact_refs": true,
+ "description": "Rows marked submission-ref are the aggregate paths expected in artifact_refs."
+ },
+ {
+ "role": "supporting-evidence",
+ "label": "Supporting evidence",
+ "copy_to_artifact_refs": false,
+ "description": "Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs."
+ }
+ ]
+ },
+ "drafts_count_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false
+ },
+ "runbook": [
+ "python3 scripts/yao.py output-review-kit --write-template",
+ "Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.",
+ "python3 scripts/adjudicate_output_review.py --write-template",
+ "Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.",
+ "python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication",
+ "python3 scripts/yao.py output-review",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "repair_checklist": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Native Permission Enforcement",
+ "category": "external",
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "ledger_status": "pending",
+ "intake_readiness": "awaiting-submission",
+ "review_state": "awaiting-submission",
+ "source_accepted": false,
+ "prechecks": [
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "permission-policy",
+ "label": "Permission policy",
+ "kind": "file",
+ "required": true,
+ "next_action": "Keep approved high-permission capabilities explicit.",
+ "secret_value_redacted": false,
+ "path": "security/permission_policy.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "permission-probes",
+ "label": "Runtime probes",
+ "kind": "file",
+ "required": true,
+ "next_action": "Refresh runtime permission probes after packaging changes.",
+ "secret_value_redacted": false,
+ "path": "reports/runtime_permission_probes.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "native-guard",
+ "label": "Native guard",
+ "kind": "external",
+ "required": true,
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "secret_value_redacted": false,
+ "status": "external-required",
+ "actual": "external-integration-required"
+ }
+ ],
+ "source_checklist": [
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Native enforcement",
+ "field": "native_enforcement_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Collect real target-client or external runtime guard proof."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Probe failures",
+ "field": "failure_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Runtime permission probes must stay clean."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Installer support",
+ "field": "installer_enforcement_ready",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Installer enforcement is supporting evidence, not native proof."
+ }
+ ],
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "submission_path": "evidence/world_class/submissions/native-permission-enforcement.json",
+ "template_path": "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "commands": {
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard ."
+ },
+ "submission_kit": {
+ "prepare_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions",
+ "prefill_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "output_dir": "evidence/world_class/submissions",
+ "draft_path": "evidence/world_class/submissions/native-permission-enforcement.json",
+ "template_path": "evidence/world_class/templates/native-permission-enforcement.intake.json",
+ "artifact_role_contract": {
+ "schema_version": "1.0",
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 11,
+ "supporting_evidence_ready_count": 11,
+ "roles": [
+ {
+ "role": "submission-ref",
+ "label": "Submission refs",
+ "copy_to_artifact_refs": true,
+ "description": "Rows marked submission-ref are the aggregate paths expected in artifact_refs."
+ },
+ {
+ "role": "supporting-evidence",
+ "label": "Supporting evidence",
+ "copy_to_artifact_refs": false,
+ "description": "Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs."
+ }
+ ]
+ },
+ "drafts_count_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false
+ },
+ "runbook": [
+ "Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.",
+ "Update the generated target adapter only when the guard is actually enforced by that target.",
+ "python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip",
+ "python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation",
+ "python3 scripts/yao.py runtime-permissions . --package-dir dist",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "repair_checklist": [
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Native Client Telemetry",
+ "category": "external",
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "ledger_status": "pending",
+ "intake_readiness": "awaiting-submission",
+ "review_state": "awaiting-submission",
+ "source_accepted": false,
+ "prechecks": [
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "native-host",
+ "label": "Native telemetry host",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the native host to receive metadata-only client events.",
+ "secret_value_redacted": false,
+ "path": "scripts/telemetry_native_host.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "hook-recipes",
+ "label": "Hook recipes",
+ "kind": "file",
+ "required": true,
+ "next_action": "Refresh telemetry hook recipes before external client installation.",
+ "secret_value_redacted": false,
+ "path": "reports/telemetry_hook_recipes.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "external-client",
+ "label": "External client",
+ "kind": "external",
+ "required": true,
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "secret_value_redacted": false,
+ "status": "external-required",
+ "actual": "external-integration-required"
+ }
+ ],
+ "source_checklist": [
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "External events",
+ "field": "external_source_events",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Import at least one metadata-only event from a real client."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Adoption sample",
+ "field": "adoption_sample_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Telemetry must include adoption outcome evidence."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Telemetry must stay metadata-only."
+ }
+ ],
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "submission_path": "evidence/world_class/submissions/native-client-telemetry.json",
+ "template_path": "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "commands": {
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard ."
+ },
+ "submission_kit": {
+ "prepare_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions",
+ "prefill_command": "python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "output_dir": "evidence/world_class/submissions",
+ "draft_path": "evidence/world_class/submissions/native-client-telemetry.json",
+ "template_path": "evidence/world_class/templates/native-client-telemetry.intake.json",
+ "artifact_role_contract": {
+ "schema_version": "1.0",
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 2,
+ "submission_ref_ready_count": 2,
+ "supporting_evidence_total_count": 6,
+ "supporting_evidence_ready_count": 6,
+ "roles": [
+ {
+ "role": "submission-ref",
+ "label": "Submission refs",
+ "copy_to_artifact_refs": true,
+ "description": "Rows marked submission-ref are the aggregate paths expected in artifact_refs."
+ },
+ {
+ "role": "supporting-evidence",
+ "label": "Supporting evidence",
+ "copy_to_artifact_refs": false,
+ "description": "Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs."
+ }
+ ]
+ },
+ "drafts_count_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false
+ },
+ "runbook": [
+ "python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///",
+ "Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
+ "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl",
+ "python3 scripts/yao.py skill-atlas --workspace-root .",
+ "python3 scripts/yao.py skill-os2-audit . --generated-at ",
+ "Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.",
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ],
+ "repair_checklist": [
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "adoption_sample_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Telemetry must include adoption outcome evidence.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "next_action": "Telemetry must include adoption outcome evidence.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "adoption_sample_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Telemetry must include adoption outcome evidence.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "prechecks": [
+ {
+ "evidence_key": "provider-holdout",
+ "key": "output-cases",
+ "label": "Output eval cases",
+ "kind": "file",
+ "required": true,
+ "next_action": "Keep output holdout cases available before provider execution.",
+ "secret_value_redacted": false,
+ "path": "evals/output/cases.jsonl",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-runner",
+ "label": "Provider runner",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the provider runner instead of the local command runner for model-backed evidence.",
+ "secret_value_redacted": false,
+ "path": "scripts/provider_output_eval_runner.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-api-key",
+ "label": "Provider credential",
+ "kind": "env_any",
+ "required": true,
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "secret_value_redacted": true,
+ "env_any": [
+ "OPENAI_API_KEY",
+ "DEEPSEEK_API_KEY"
+ ],
+ "status": "missing",
+ "actual": "not-set",
+ "set_count": 0,
+ "set_envs": []
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "key": "provider-model",
+ "label": "Provider model",
+ "kind": "env",
+ "required": false,
+ "next_action": "Optionally set YAO_OUTPUT_EVAL_MODEL, or pass --provider-model for the selected provider.",
+ "secret_value_redacted": false,
+ "env": "YAO_OUTPUT_EVAL_MODEL",
+ "status": "optional",
+ "actual": "not-set",
+ "default": "provider-specific model, or pass --provider-model"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "review-kit",
+ "label": "Blind review kit",
+ "kind": "file",
+ "required": true,
+ "next_action": "Open the blind review kit and record real reviewer choices with required rationale.",
+ "secret_value_redacted": false,
+ "path": "reports/output_review_kit.html",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "decision-template",
+ "label": "Decision template",
+ "kind": "file",
+ "required": true,
+ "next_action": "Import real A/B decisions with reviewer, reviewed_at, winner_variant, confidence, reason, and truthful blind-review attestation via `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`.",
+ "secret_value_redacted": false,
+ "path": "reports/output_review_decisions.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "decision-importer",
+ "label": "Decision importer",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the importer to reject raw content fields and normalize reviewer decisions before adjudication.",
+ "secret_value_redacted": false,
+ "path": "scripts/import_output_review_decisions.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "key": "human-reviewer",
+ "label": "Human reviewer",
+ "kind": "human",
+ "required": true,
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "secret_value_redacted": false,
+ "status": "human-required",
+ "actual": "external-human-action"
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "permission-policy",
+ "label": "Permission policy",
+ "kind": "file",
+ "required": true,
+ "next_action": "Keep approved high-permission capabilities explicit.",
+ "secret_value_redacted": false,
+ "path": "security/permission_policy.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "permission-probes",
+ "label": "Runtime probes",
+ "kind": "file",
+ "required": true,
+ "next_action": "Refresh runtime permission probes after packaging changes.",
+ "secret_value_redacted": false,
+ "path": "reports/runtime_permission_probes.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "key": "native-guard",
+ "label": "Native guard",
+ "kind": "external",
+ "required": true,
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "secret_value_redacted": false,
+ "status": "external-required",
+ "actual": "external-integration-required"
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "native-host",
+ "label": "Native telemetry host",
+ "kind": "file",
+ "required": true,
+ "next_action": "Use the native host to receive metadata-only client events.",
+ "secret_value_redacted": false,
+ "path": "scripts/telemetry_native_host.py",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "hook-recipes",
+ "label": "Hook recipes",
+ "kind": "file",
+ "required": true,
+ "next_action": "Refresh telemetry hook recipes before external client installation.",
+ "secret_value_redacted": false,
+ "path": "reports/telemetry_hook_recipes.json",
+ "status": "pass",
+ "actual": "present"
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "key": "external-client",
+ "label": "External client",
+ "kind": "external",
+ "required": true,
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "secret_value_redacted": false,
+ "status": "external-required",
+ "actual": "external-integration-required"
+ }
+ ],
+ "source_checklist": [
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Provider model run",
+ "field": "model_executed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Run provider-backed output-exec with real credentials."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Timing observed",
+ "field": "timing_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should record timing metadata."
+ },
+ {
+ "evidence_key": "provider-holdout",
+ "label": "Token usage observed",
+ "field": "token_observed_count",
+ "expected": ">0",
+ "actual": 10,
+ "status": "pass",
+ "source_accepted": true,
+ "next_action": "Provider execution should return non-estimated token usage."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Review pairs exist",
+ "field": "pair_count",
+ "expected": ">0",
+ "actual": 5,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Generate the blind A/B review pack."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No pending decisions",
+ "field": "pending_count",
+ "expected": "==0",
+ "actual": 5,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record a reviewer choice and reason for every pair."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Judgments complete",
+ "field": "judgment_count",
+ "expected": "==pair_count",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Every pair needs one valid human judgment."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "No invalid decisions",
+ "field": "invalid_decision_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Fix malformed winner/confidence entries."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reviewer metadata",
+ "field": "reviewer_metadata_present",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Record reviewer and reviewed_at before adjudication can count."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Reason required",
+ "field": "reason_required",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Keep reason mandatory for every imported or direct reviewer decision."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Blind review attested",
+ "field": "blind_review_attested",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content attested",
+ "field": "raw_content_excluded_attested",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages."
+ },
+ {
+ "evidence_key": "human-adjudication",
+ "label": "Human evidence ready",
+ "field": "ready_for_human_evidence",
+ "expected": "true",
+ "actual": false,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Native enforcement",
+ "field": "native_enforcement_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Collect real target-client or external runtime guard proof."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Probe failures",
+ "field": "failure_count",
+ "expected": "==0",
+ "actual": 0,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Runtime permission probes must stay clean."
+ },
+ {
+ "evidence_key": "native-permission-enforcement",
+ "label": "Installer support",
+ "field": "installer_enforcement_ready",
+ "expected": "true",
+ "actual": true,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Installer enforcement is supporting evidence, not native proof."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "External events",
+ "field": "external_source_events",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Import at least one metadata-only event from a real client."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Adoption sample",
+ "field": "adoption_sample_count",
+ "expected": ">0",
+ "actual": 0,
+ "status": "blocked",
+ "source_accepted": false,
+ "next_action": "Telemetry must include adoption outcome evidence."
+ },
+ {
+ "evidence_key": "native-client-telemetry",
+ "label": "Raw content blocked",
+ "field": "raw_content_allowed",
+ "expected": "false",
+ "actual": false,
+ "status": "pass",
+ "source_accepted": false,
+ "next_action": "Telemetry must stay metadata-only."
+ }
+ ],
+ "repair_checklist": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-precheck-provider-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "provider-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env_any precheck is missing.",
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "adoption_sample_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Telemetry must include adoption outcome evidence.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-precheck-provider-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "provider-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env_any precheck is missing.",
+ "next_action": "Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 8,
+ "row_count": 8,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-adoption_sample_count",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "adoption_sample_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Telemetry must include adoption outcome evidence.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
+ "submissions": {
+ "directory": "evidence/world_class/submissions",
+ "commands": {
+ "prepare_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions",
+ "prepare_prefilled_submission": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "validate_intake": "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions",
+ "submission_review": "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions",
+ "refresh_ledger": "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard ."
+ },
+ "submission_kit_command": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions",
+ "submission_kit_prefill_command": "python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts",
+ "preflight_counts_submission_as_completion": false,
+ "drafts_count_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "artifact_role_contract": {
+ "schema_version": "1.0",
+ "role_source": "world-class-submission-kit",
+ "counts_as_evidence": false,
+ "artifact_prefill_counts_as_evidence": false,
+ "submission_ref_total_count": 7,
+ "submission_ref_ready_count": 7,
+ "supporting_evidence_total_count": 31,
+ "supporting_evidence_ready_count": 31,
+ "roles": [
+ {
+ "role": "submission-ref",
+ "label": "Submission refs",
+ "copy_to_artifact_refs": true,
+ "description": "Rows marked submission-ref are the aggregate paths expected in artifact_refs."
+ },
+ {
+ "role": "supporting-evidence",
+ "label": "Supporting evidence",
+ "copy_to_artifact_refs": false,
+ "description": "Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs."
+ }
+ ]
+ }
+ },
+ "source_reports": {
+ "ledger": "reports/world_class_evidence_ledger.json",
+ "intake": "reports/world_class_evidence_intake.json",
+ "submission_review": "reports/world_class_submission_review.json",
+ "operator_runbook": "reports/world_class_operator_runbook.json"
+ },
+ "artifacts": {
+ "json": "reports/world_class_evidence_preflight.json",
+ "markdown": "reports/world_class_evidence_preflight.md",
+ "html": "reports/world_class_evidence_preflight.html"
+ }
+}
diff --git a/reports/world_class_evidence_preflight.md b/reports/world_class_evidence_preflight.md
new file mode 100644
index 00000000..048f9548
--- /dev/null
+++ b/reports/world_class_evidence_preflight.md
@@ -0,0 +1,201 @@
+# World-Class Evidence Preflight
+
+Generated at: `2026-06-17`
+
+## Summary
+
+- decision: `collection-preflight-blocked`
+- ready to claim world-class: `false`
+- preflight counts as evidence: `false`
+- credential value exposed: `false`
+- collection ready: `1`
+- collection blocked: `3`
+- source checks: `11` pass / `19` total
+- repair rows: `12` blocked / `12` total
+- phase queue: `2` blocked / `2` phases
+- phase queue rows: `12`
+- next repair action: `human-adjudication-precheck-human-reviewer`
+- next repair owner: `human reviewer`
+- next phase: `unblock-access`
+- next phase action: `human-adjudication-precheck-human-reviewer`
+
+This preflight report checks whether an operator can start collecting the remaining external or human evidence. It never accepts evidence, prints secret values, or changes the world-class ledger.
+
+## Submission Kit Handoff
+
+- submissions directory: `evidence/world_class/submissions`
+- prepare drafts: `python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions`
+- prepare drafts with artifact SHA prefill: `python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts`
+- validate intake: `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
+- review queue: `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
+- refresh ledger: `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
+- guard claims: `python3 scripts/yao.py world-class-claim-guard .`
+- drafts count as evidence: `false`
+- artifact prefill counts as evidence: `false`
+- submission refs ready: `7` / `7`
+- supporting evidence ready: `31` / `31`
+
+Generate the submission kit after the real provider, human, native-permission, or native-client work exists. The generated JSON drafts remain `template_only: true` until an operator edits them with real aggregate artifact references and matching SHA-256 digests. The prefill command only inserts local artifact SHA-256 digests; it does not make a draft count as evidence.
+
+| Role | Copy to artifact_refs | Ready | Meaning |
+| --- | --- | --- | --- |
+| `submission-ref` | `true` | `7 / 7` | Rows marked submission-ref are the aggregate paths expected in artifact_refs. |
+| `supporting-evidence` | `false` | `31 / 31` | Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs. |
+
+`submission-ref` rows are the only checklist rows expected in `artifact_refs`; `supporting-evidence` rows stay available for audit context and reviewer traceability.
+
+## Phase Queue
+
+Phase queue rows group the same repair checklist into operator execution phases. They are procedural guidance only and do not count as completion evidence.
+
+| Priority | Phase | Status | Rows | Owners | Evidence | Verify | Next action |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| `20` | `unblock-access` | `blocked` | 4 / 4 blocked | Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator | human-adjudication, native-client-telemetry, native-permission-enforcement, provider-holdout | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Assign a real reviewer identity before claiming human adjudication. |
+| `40` | `collect-source` | `blocked` | 8 / 8 blocked | Browser/Chrome/IDE/provider client integrator, human reviewer, target client or installer integrator | human-adjudication, native-client-telemetry, native-permission-enforcement | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Set reviewer_attestation only after choices are completed before opening the answer key. |
+
+## Evidence Items
+
+| Evidence | Status | Intake | Review | Next action |
+| --- | --- | --- | --- | --- |
+| `provider-holdout` | `blocked` | `awaiting-submission` | `awaiting-submission` | Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value. |
+| `human-adjudication` | `ready-for-human-review` | `awaiting-submission` | `awaiting-submission` | Assign a real reviewer identity before claiming human adjudication. |
+| `native-permission-enforcement` | `blocked` | `awaiting-submission` | `awaiting-submission` | Attach a real target-client or external installer runtime guard; metadata fallback is not enough. |
+| `native-client-telemetry` | `blocked` | `awaiting-submission` | `awaiting-submission` | Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. |
+
+## Repair Checklist
+
+Repair rows convert preflight and source blockers into a prioritized operator queue. They are guidance only and do not count as completion evidence.
+
+| Priority | Phase | Owner | Evidence | Type | Target | Status | Verify | Next action |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| `20` | `unblock-access` | human reviewer | `human-adjudication` | `precheck` | `human-reviewer` | `blocked` | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Assign a real reviewer identity before claiming human adjudication. |
+| `20` | `unblock-access` | Browser/Chrome/IDE/provider client integrator | `native-client-telemetry` | `precheck` | `external-client` | `blocked` | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. |
+| `20` | `unblock-access` | target client or installer integrator | `native-permission-enforcement` | `precheck` | `native-guard` | `blocked` | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Attach a real target-client or external installer runtime guard; metadata fallback is not enough. |
+| `20` | `unblock-access` | operator with provider credentials | `provider-holdout` | `precheck` | `provider-api-key` | `blocked` | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value. |
+| `40` | `collect-source` | human reviewer | `human-adjudication` | `source-check` | `blind_review_attested` | `blocked` | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Set reviewer_attestation only after choices are completed before opening the answer key. |
+| `40` | `collect-source` | human reviewer | `human-adjudication` | `source-check` | `judgment_count` | `blocked` | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Every pair needs one valid human judgment. |
+| `40` | `collect-source` | human reviewer | `human-adjudication` | `source-check` | `pending_count` | `blocked` | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Record a reviewer choice and reason for every pair. |
+| `40` | `collect-source` | human reviewer | `human-adjudication` | `source-check` | `ready_for_human_evidence` | `blocked` | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints. |
+| `40` | `collect-source` | human reviewer | `human-adjudication` | `source-check` | `reviewer_metadata_present` | `blocked` | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Record reviewer and reviewed_at before adjudication can count. |
+| `40` | `collect-source` | Browser/Chrome/IDE/provider client integrator | `native-client-telemetry` | `source-check` | `adoption_sample_count` | `blocked` | `python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Telemetry must include adoption outcome evidence. |
+| `40` | `collect-source` | Browser/Chrome/IDE/provider client integrator | `native-client-telemetry` | `source-check` | `external_source_events` | `blocked` | `python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Import at least one metadata-only event from a real client. |
+| `40` | `collect-source` | target client or installer integrator | `native-permission-enforcement` | `source-check` | `native_enforcement_count` | `blocked` | `python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` | Collect real target-client or external runtime guard proof. |
+
+## Provider Holdout
+
+- status: `blocked`
+- ledger: `pending`
+- submission: `evidence/world_class/submissions/provider-holdout.json`
+- prepare draft: `python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions`
+- prepare draft with artifact SHA prefill: `python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts`
+- submission refs ready: `1` / `1`
+- supporting evidence ready: `6` / `6`
+
+### Prechecks
+
+| Check | Kind | Current | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Output eval cases | `file` | `present` | `pass` | Keep output holdout cases available before provider execution. |
+| Provider runner | `file` | `present` | `pass` | Use the provider runner instead of the local command runner for model-backed evidence. |
+| Provider credential | `env_any` | `not-set` | `missing` | Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value. |
+| Provider model | `env` | `not-set` | `optional` | Optionally set YAO_OUTPUT_EVAL_MODEL, or pass --provider-model for the selected provider. |
+
+### Source Checks
+
+| Check | Current | Expected | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Provider model run | `10` | `>0` | `pass` | Run provider-backed output-exec with real credentials. |
+| Timing observed | `10` | `>0` | `pass` | Provider execution should record timing metadata. |
+| Token usage observed | `10` | `>0` | `pass` | Provider execution should return non-estimated token usage. |
+
+## Human Adjudication
+
+- status: `ready-for-human-review`
+- ledger: `pending`
+- submission: `evidence/world_class/submissions/human-adjudication.json`
+- prepare draft: `python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions`
+- prepare draft with artifact SHA prefill: `python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts`
+- submission refs ready: `2` / `2`
+- supporting evidence ready: `8` / `8`
+
+### Prechecks
+
+| Check | Kind | Current | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Blind review kit | `file` | `present` | `pass` | Open the blind review kit and record real reviewer choices with required rationale. |
+| Decision template | `file` | `present` | `pass` | Import real A/B decisions with reviewer, reviewed_at, winner_variant, confidence, reason, and truthful blind-review attestation via `python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication`. |
+| Decision importer | `file` | `present` | `pass` | Use the importer to reject raw content fields and normalize reviewer decisions before adjudication. |
+| Human reviewer | `human` | `external-human-action` | `human-required` | Assign a real reviewer identity before claiming human adjudication. |
+
+### Source Checks
+
+| Check | Current | Expected | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Review pairs exist | `5` | `>0` | `pass` | Generate the blind A/B review pack. |
+| No pending decisions | `5` | `==0` | `blocked` | Record a reviewer choice and reason for every pair. |
+| Judgments complete | `0` | `==pair_count` | `blocked` | Every pair needs one valid human judgment. |
+| No invalid decisions | `0` | `==0` | `pass` | Fix malformed winner/confidence entries. |
+| Reviewer metadata | `False` | `true` | `blocked` | Record reviewer and reviewed_at before adjudication can count. |
+| Reason required | `True` | `true` | `pass` | Keep reason mandatory for every imported or direct reviewer decision. |
+| Blind review attested | `False` | `true` | `blocked` | Set reviewer_attestation only after choices are completed before opening the answer key. |
+| Raw content attested | `True` | `true` | `pass` | Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content. |
+| Raw content blocked | `False` | `false` | `pass` | Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages. |
+| Human evidence ready | `False` | `true` | `blocked` | Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints. |
+
+## Native Permission Enforcement
+
+- status: `blocked`
+- ledger: `pending`
+- submission: `evidence/world_class/submissions/native-permission-enforcement.json`
+- prepare draft: `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions`
+- prepare draft with artifact SHA prefill: `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts`
+- submission refs ready: `2` / `2`
+- supporting evidence ready: `11` / `11`
+
+### Prechecks
+
+| Check | Kind | Current | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Permission policy | `file` | `present` | `pass` | Keep approved high-permission capabilities explicit. |
+| Runtime probes | `file` | `present` | `pass` | Refresh runtime permission probes after packaging changes. |
+| Native guard | `external` | `external-integration-required` | `external-required` | Attach a real target-client or external installer runtime guard; metadata fallback is not enough. |
+
+### Source Checks
+
+| Check | Current | Expected | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Native enforcement | `0` | `>0` | `blocked` | Collect real target-client or external runtime guard proof. |
+| Probe failures | `0` | `==0` | `pass` | Runtime permission probes must stay clean. |
+| Installer support | `True` | `true` | `pass` | Installer enforcement is supporting evidence, not native proof. |
+
+## Native Client Telemetry
+
+- status: `blocked`
+- ledger: `pending`
+- submission: `evidence/world_class/submissions/native-client-telemetry.json`
+- prepare draft: `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions`
+- prepare draft with artifact SHA prefill: `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts`
+- submission refs ready: `2` / `2`
+- supporting evidence ready: `6` / `6`
+
+### Prechecks
+
+| Check | Kind | Current | Status | Next action |
+| --- | --- | --- | --- | --- |
+| Native telemetry host | `file` | `present` | `pass` | Use the native host to receive metadata-only client events. |
+| Hook recipes | `file` | `present` | `pass` | Refresh telemetry hook recipes before external client installation. |
+| External client | `external` | `external-integration-required` | `external-required` | Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. |
+
+### Source Checks
+
+| Check | Current | Expected | Status | Next action |
+| --- | --- | --- | --- | --- |
+| External events | `0` | `>0` | `blocked` | Import at least one metadata-only event from a real client. |
+| Adoption sample | `0` | `>0` | `blocked` | Telemetry must include adoption outcome evidence. |
+| Raw content blocked | `False` | `false` | `pass` | Telemetry must stay metadata-only. |
+
+## Boundary
+
+- Environment variables are reported only as `set` or `not-set`; values are never printed.
+- Human-required and external-required states are operator actions, not accepted evidence.
+- The world-class ledger remains the source of truth for `ready_to_claim_world_class`.
diff --git a/reports/world_class_operator_runbook.html b/reports/world_class_operator_runbook.html
new file mode 100644
index 00000000..eab34933
--- /dev/null
+++ b/reports/world_class_operator_runbook.html
@@ -0,0 +1,168 @@
+
+
+
+
+
+ World-Class Operator Runbook
+
+
+
+
+
+
+ Evidence Operations
+
World-Class Operator Runbook
+
A single operating page for collecting the remaining human and external evidence. It coordinates action, but does not accept evidence or change the ledger.
Run the real external or human work for one evidence item.
Generate and fill the matching submission draft.
Validate intake and inspect the submission review queue.
Refresh the ledger and run the claim guard before making any completion claim.
+
Coordination Plan
These steps tell the assistant what to run and where the user must supply real provider, reviewer, client, or telemetry input. Every row is operational guidance only and keeps counts_as_completion false.
Step
Evidence
Owner
User
User action
Assistant action
Command
Pass condition
prepare-evidence-session
all
assistant + user
true
Confirm provider access, reviewer availability, target client path, and telemetry client path before collection starts.
Run preflight and prepare submission drafts without accepting them as evidence.
Unblock accessunblock-access: 1 / 1Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
+
Source Runbook
Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
No pending decisionspending_count: 5 / ==0Record a reviewer choice and reason for every pair.
Judgments completejudgment_count: 0 / ==pair_countEvery pair needs one valid human judgment.
No invalid decisionsinvalid_decision_count: 0 / ==0Fix malformed winner/confidence entries.
Reviewer metadatareviewer_metadata_present: False / trueRecord reviewer and reviewed_at before adjudication can count.
Reason requiredreason_required: True / trueKeep reason mandatory for every imported or direct reviewer decision.
Blind review attestedblind_review_attested: False / trueSet reviewer_attestation only after choices are completed before opening the answer key.
Raw content attestedraw_content_excluded_attested: True / trueAttest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content.
Raw content blockedraw_content_allowed: False / falseAdjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages.
Human evidence readyready_for_human_evidence: False / trueComplete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
+
Privacy
Do not mark native_enforcement true for metadata-only fallbacks.
Keep residual risks visible for targets that still rely on operator enforcement.
+
+
Next Source Actions
Collect real target-client or external runtime guard proof.
+
Source Evidence Snapshot
Native enforcementnative_enforcement_count: 0 / >0Collect real target-client or external runtime guard proof.
Probe failuresfailure_count: 0 / ==0Runtime permission probes must stay clean.
Installer supportinstaller_enforcement_ready: True / trueInstaller enforcement is supporting evidence, not native proof.
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
All scanned skills remain visible: {summary['route_collision_count']} total route collisions, {summary['owner_gap_count']} total owner gaps, and {summary['stale_count']} total stale signals.
Missed-trigger telemetry and explicit failure cases become candidate routing work without storing raw prompts or outputs.
+
{opportunity_items or '
No no-route opportunities detected.
'}
+
+
+
Full Portfolio Counts
+
All scanned skills remain visible: {summary['route_collision_count']} total route collisions, {summary['owner_gap_count']} total owner gaps, {summary['stale_count']} total stale signals, and {summary['drift_signal_count']} telemetry drift signals.
+
+
+
Catalog
+
+
Name
Path
Owner
Maturity
Review
Scope
+ {''.join(rows)}
+
+
+
+
+
+"""
diff --git a/scripts/build_skill_atlas_opportunities.py b/scripts/build_skill_atlas_opportunities.py
new file mode 100644
index 00000000..d0d2b2ee
--- /dev/null
+++ b/scripts/build_skill_atlas_opportunities.py
@@ -0,0 +1,82 @@
+from pathlib import Path
+from typing import Any, Callable
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by build_skill_atlas.py to keep no-route opportunity detection out of the atlas CLI."
+
+
+SkipPredicate = Callable[[Path, Path], bool]
+RelFormatter = Callable[[Path, Path], str]
+
+
+def failure_case_no_route_opportunities(
+ workspace_root: Path,
+ *,
+ should_skip: SkipPredicate,
+ safe_rel: RelFormatter,
+) -> list[dict[str, Any]]:
+ opportunities = []
+ for path in sorted(workspace_root.rglob("failure-cases.md")):
+ if should_skip(path, workspace_root):
+ continue
+ text = path.read_text(encoding="utf-8", errors="replace")
+ for line in text.splitlines():
+ stripped = line.strip()
+ if not stripped.startswith("-"):
+ continue
+ lowered = stripped.casefold()
+ if "no_route" in lowered or "no route" in lowered or "missed" in lowered or "under-trigger" in lowered:
+ opportunities.append(
+ {
+ "source_type": "failure-case",
+ "source": safe_rel(workspace_root, path),
+ "note": stripped.lstrip("- ").strip(),
+ "actionable": True,
+ "privacy_contract": "source note only; raw prompts are not required",
+ }
+ )
+ return opportunities[:50]
+
+
+def telemetry_no_route_opportunities(drift_signals: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ opportunities = []
+ for signal in drift_signals:
+ signal_types = {str(item) for item in signal.get("signal_types", [])}
+ if not {"missed trigger", "under trigger"} & signal_types:
+ continue
+ summary = signal.get("summary", {}) if isinstance(signal.get("summary"), dict) else {}
+ opportunities.append(
+ {
+ "source_type": "telemetry",
+ "source": str(signal.get("source", "")),
+ "skill": str(signal.get("name", "")),
+ "path": str(signal.get("path", "")),
+ "signal": "missed trigger",
+ "missed_trigger_count": int(summary.get("missed_trigger_count") or 0),
+ "recommendation": str(
+ signal.get("recommendation")
+ or "Add missed prompts to trigger eval and evaluate whether a new skill route is needed."
+ ),
+ "actionable": bool(signal.get("actionable")),
+ "scope": str(signal.get("scope", "")),
+ "privacy_contract": "metadata-only telemetry; no raw prompt, output, transcript, or note is stored",
+ }
+ )
+ return opportunities
+
+
+def no_route_opportunities(
+ workspace_root: Path,
+ drift_signals: list[dict[str, Any]],
+ *,
+ should_skip: SkipPredicate,
+ safe_rel: RelFormatter,
+) -> list[dict[str, Any]]:
+ opportunities = failure_case_no_route_opportunities(
+ workspace_root,
+ should_skip=should_skip,
+ safe_rel=safe_rel,
+ )
+ opportunities.extend(telemetry_no_route_opportunities(drift_signals))
+ return opportunities[:80]
diff --git a/scripts/ci_test.py b/scripts/ci_test.py
index fda84fe1..9ec03633 100644
--- a/scripts/ci_test.py
+++ b/scripts/ci_test.py
@@ -20,6 +20,7 @@ SAFE_ENV_KEYS = (
"TERM",
"TMP",
"TMPDIR",
+ "TZ",
)
DEFAULT_TARGETS = [
"eval",
@@ -29,14 +30,20 @@ DEFAULT_TARGETS = [
"description-optimization",
"description-optimization-check",
"promotion-check",
+ "python-compat-check",
+ "architecture-maintainability-check",
"yao-cli-check",
+ "yao-cli-world-class-check",
"skill-overview-check",
+ "skill-interpretation-check",
"skill-report-metrics-check",
"skill-report-charts-check",
+ "html-rendering-check",
"skill-ir-check",
"compiler-check",
"output-eval-check",
"output-execution-check",
+ "output-review-kit-check",
"output-review-adjudication-check",
"runtime-conformance-check",
"runtime-permission-check",
@@ -48,8 +55,28 @@ DEFAULT_TARGETS = [
"upgrade-check",
"review-viewer-check",
"review-studio-check",
+ "skill-os2-audit-check",
+ "skill-os2-coverage-check",
+ "world-class-evidence-check",
+ "world-class-ledger-check",
+ "world-class-intake-check",
+ "world-class-submission-kit-check",
+ "world-class-preflight-check",
+ "world-class-submission-review-check",
+ "world-class-runbook-check",
+ "world-class-claim-guard-check",
+ "benchmark-reproducibility-check",
+ "evidence-consistency-check",
"feedback-check",
+ "adaptation-safety-check",
+ "skillops-opportunity-check",
+ "daily-skillops-check",
+ "weekly-curator-check",
"adoption-drift-check",
+ "telemetry-import-check",
+ "telemetry-emit-check",
+ "telemetry-hooks-check",
+ "telemetry-native-host-check",
"review-waivers-check",
"review-annotations-check",
"baseline-compare-check",
diff --git a/scripts/compile_skill.py b/scripts/compile_skill.py
index 62c59938..997e7223 100644
--- a/scripts/compile_skill.py
+++ b/scripts/compile_skill.py
@@ -10,96 +10,15 @@ try:
except ImportError: # pragma: no cover
yaml = None
+from skill_ir_paths import find_skill_ir as find_skill_ir_document
+from compile_skill_targets import TARGET_TRANSFORMS, target_native_contract, target_permission_contract
+
ROOT = Path(__file__).resolve().parent.parent
COMPILER_SCHEMA_VERSION = "1.0"
COMPILER_NAME = "yao-skill-ir-compiler"
-TARGET_TRANSFORMS: dict[str, dict[str, Any]] = {
- "openai": {
- "adapter_mode": "metadata-adapter",
- "generated_files": ["targets/openai/adapter.json", "targets/openai/agents/openai.yaml"],
- "metadata_mapping": {
- "display_name": "targets/openai/agents/openai.yaml::interface.display_name",
- "default_prompt": "targets/openai/agents/openai.yaml::interface.default_prompt",
- "activation": "targets/openai/agents/openai.yaml::compatibility.activation_mode",
- "execution": "targets/openai/agents/openai.yaml::compatibility.execution_context",
- "trust": "targets/openai/agents/openai.yaml::compatibility.trust_level",
- "permissions": "targets/openai/agents/openai.yaml::compatibility.permission_contract",
- "degradation": "targets/openai/agents/openai.yaml::compatibility.degradation_strategy",
- },
- "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": ["client-native script permission prompts are represented as permission contract metadata"],
- },
- "claude": {
- "adapter_mode": "neutral-source-plus-adapter",
- "generated_files": ["targets/claude/adapter.json", "targets/claude/README.md"],
- "metadata_mapping": {
- "display_name": "targets/claude/adapter.json::display_name",
- "default_prompt": "targets/claude/adapter.json::default_prompt",
- "activation": "targets/claude/adapter.json::activation_mode",
- "execution": "targets/claude/adapter.json::execution_context",
- "trust": "targets/claude/adapter.json::trust_level",
- "permissions": "targets/claude/adapter.json::target_permission_contract",
- "degradation": "targets/claude/adapter.json::degradation_strategy",
- },
- "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": ["vendor-native metadata fields are carried as adapter JSON and README notes"],
- },
- "generic": {
- "adapter_mode": "agent-skills-compatible",
- "generated_files": ["targets/generic/adapter.json"],
- "metadata_mapping": {
- "display_name": "targets/generic/adapter.json::display_name",
- "default_prompt": "targets/generic/adapter.json::default_prompt",
- "activation": "targets/generic/adapter.json::activation_mode",
- "execution": "targets/generic/adapter.json::execution_context",
- "trust": "targets/generic/adapter.json::trust_level",
- "permissions": "targets/generic/adapter.json::target_permission_contract",
- "degradation": "targets/generic/adapter.json::degradation_strategy",
- },
- "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": [],
- },
- "agent-skills-compatible": {
- "adapter_mode": "neutral-agent-skills-source",
- "generated_files": ["SKILL.md", "agents/interface.yaml"],
- "metadata_mapping": {
- "name": "SKILL.md::frontmatter.name",
- "description": "SKILL.md::frontmatter.description",
- "interface": "agents/interface.yaml",
- "manifest": "manifest.json",
- },
- "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": [],
- },
- "agent-skills": {
- "adapter_mode": "neutral-agent-skills-source",
- "generated_files": ["SKILL.md", "agents/interface.yaml"],
- "metadata_mapping": {
- "name": "SKILL.md::frontmatter.name",
- "description": "SKILL.md::frontmatter.description",
- "interface": "agents/interface.yaml",
- "manifest": "manifest.json",
- },
- "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": [],
- },
- "vscode": {
- "adapter_mode": "agent-skills-compatible-project-or-user-scope",
- "generated_files": ["SKILL.md", "agents/interface.yaml"],
- "metadata_mapping": {
- "name": "folder-name-and-SKILL.md::frontmatter.name",
- "description": "SKILL.md::frontmatter.description",
- "interface": "agents/interface.yaml",
- },
- "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
- "unsupported_features": ["VS Code installation scope is documented but not installed by this compiler"],
- },
-}
-
-
def display_path(path: Path, root: Path = ROOT) -> str:
try:
return str(path.resolve().relative_to(root.resolve()))
@@ -146,20 +65,7 @@ def read_frontmatter(path: Path) -> dict[str, Any]:
def find_skill_ir(skill_dir: Path, name: str) -> tuple[dict[str, Any], str]:
- candidates = [
- skill_dir / "reports" / "skill-ir.json",
- skill_dir / "skill-ir" / "examples" / f"{name}.json",
- skill_dir / "skill-ir" / "examples" / f"{skill_dir.name}.json",
- ]
- seen: set[Path] = set()
- for path in candidates:
- if path in seen:
- continue
- seen.add(path)
- payload = load_json(path)
- if payload:
- return payload, display_path(path, skill_dir)
- return {}, "frontmatter-fallback"
+ return find_skill_ir_document(skill_dir, name, fallback_source="frontmatter-fallback")
def count_list(payload: dict[str, Any], key: str) -> int:
@@ -246,191 +152,6 @@ def permission_contract(skill_dir: Path) -> dict[str, Any]:
}
-TARGET_PERMISSION_MODELS = {
- "openai": {
- "model": "metadata-only",
- "native_enforcement": False,
- "representation": "targets/openai/agents/openai.yaml::compatibility.permission_contract plus adapter.json",
- "operator_note": "OpenAI target carries permission metadata for reviewer visibility; host enforcement remains outside the package.",
- },
- "claude": {
- "model": "neutral-source-plus-adapter",
- "native_enforcement": False,
- "representation": "targets/claude/adapter.json::target_permission_contract and README notes",
- "operator_note": "Claude-compatible package keeps permission intent in adapter metadata for install review.",
- },
- "generic": {
- "model": "agent-skills-compatible-metadata",
- "native_enforcement": False,
- "representation": "targets/generic/adapter.json::target_permission_contract",
- "operator_note": "Generic target exposes permission metadata for downstream clients to enforce or review.",
- },
-}
-
-
-TARGET_NATIVE_MODELS = {
- "openai": {
- "native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
- "activation_policy": "Use frontmatter description for catalog routing and targets/openai/agents/openai.yaml for display name, default prompt, and compatibility metadata.",
- "resource_strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.",
- "script_strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.",
- "permission_enforcement": "metadata-only",
- "install_scope": "plugin or skill package consumer",
- "review_artifacts": ["targets/openai/agents/openai.yaml", "targets/openai/adapter.json", "reports/review-studio.html"],
- "fallback_behavior": "If OpenAI-native metadata is ignored, the package remains readable as neutral Agent Skills source.",
- "unsupported_native_features": [
- "client-native permission prompts",
- "provider-executed scripts",
- ],
- },
- "claude": {
- "native_surface": "Claude-compatible neutral source folder with adapter notes",
- "activation_policy": "Use SKILL.md frontmatter description as the primary activation contract and adapter.json for review metadata.",
- "resource_strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.",
- "script_strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.",
- "permission_enforcement": "metadata-fallback",
- "install_scope": "user or project skill directory",
- "review_artifacts": ["targets/claude/README.md", "targets/claude/adapter.json", "reports/review-studio.html"],
- "fallback_behavior": "If Claude-specific metadata is not consumed, SKILL.md and references remain the source of truth.",
- "unsupported_native_features": [
- "vendor-native permission enforcement",
- "provider-specific execution transforms",
- ],
- },
- "generic": {
- "native_surface": "Agent Skills compatible neutral package",
- "activation_policy": "Use SKILL.md name and description; consumers decide automatic or manual activation.",
- "resource_strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.",
- "script_strategy": "Expose script and permission metadata for downstream clients or installers to enforce.",
- "permission_enforcement": "consumer-enforced-or-metadata-only",
- "install_scope": "generic Agent Skills compatible root",
- "review_artifacts": ["targets/generic/adapter.json", "reports/review-studio.html"],
- "fallback_behavior": "Neutral source is the runtime fallback.",
- "unsupported_native_features": [],
- },
- "agent-skills-compatible": {
- "native_surface": "Agent Skills standard source tree",
- "activation_policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
- "resource_strategy": "Keep optional directories as relative resources next to SKILL.md.",
- "script_strategy": "Scripts remain local optional resources and should advertise --help when executable.",
- "permission_enforcement": "consumer-enforced-or-metadata-only",
- "install_scope": "Agent Skills source root",
- "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
- "fallback_behavior": "The source tree itself is the target artifact.",
- "unsupported_native_features": [],
- },
- "agent-skills": {
- "native_surface": "Agent Skills standard source tree",
- "activation_policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
- "resource_strategy": "Keep optional directories as relative resources next to SKILL.md.",
- "script_strategy": "Scripts remain local optional resources and should advertise --help when executable.",
- "permission_enforcement": "consumer-enforced-or-metadata-only",
- "install_scope": "Agent Skills source root",
- "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
- "fallback_behavior": "The source tree itself is the target artifact.",
- "unsupported_native_features": [],
- },
- "vscode": {
- "native_surface": "VS Code/Copilot Agent Skills project or user scope",
- "activation_policy": "Use folder name plus SKILL.md name/description; keep description under platform limits.",
- "resource_strategy": "Install as project or user scoped skill source, preserving relative references and scripts.",
- "script_strategy": "Scripts require workspace trust and operator/client approval outside this compiler.",
- "permission_enforcement": "client-or-workspace-trust",
- "install_scope": "VS Code user or project skills directory",
- "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
- "fallback_behavior": "If VS Code scope is not installed, use the neutral Agent Skills source.",
- "unsupported_native_features": [
- "automatic VS Code installation",
- ],
- },
-}
-
-
-def target_permission_contract(target: str, permissions: dict[str, Any]) -> dict[str, Any]:
- model = TARGET_PERMISSION_MODELS.get(
- target,
- {
- "model": "metadata-only",
- "native_enforcement": False,
- "representation": "adapter metadata",
- "operator_note": "Permission semantics are preserved as metadata for reviewer visibility.",
- },
- )
- return {
- "schema_version": "1.0",
- "target": target,
- "permission_model": model["model"],
- "native_enforcement": model["native_enforcement"],
- "representation": model["representation"],
- "review_required": bool(permissions.get("review_required")),
- "declared_capabilities": permissions.get("declared_capabilities", []),
- "capability_counts": {
- name: item.get("script_count", 0)
- for name, item in permissions.get("capabilities", {}).items()
- },
- "evidence": permissions.get("source", ""),
- "operator_note": model["operator_note"],
- }
-
-
-def target_native_contract(
- target: str,
- profile: dict[str, Any],
- contract: dict[str, Any],
- target_permissions: dict[str, Any],
-) -> dict[str, Any]:
- model = TARGET_NATIVE_MODELS.get(
- target,
- {
- "native_surface": "adapter metadata",
- "activation_policy": "Carry activation semantics as metadata for the target consumer.",
- "resource_strategy": "Preserve neutral package resources.",
- "script_strategy": "Expose script metadata for reviewer visibility.",
- "permission_enforcement": "metadata-only",
- "install_scope": "target consumer",
- "review_artifacts": ["adapter.json", "reports/review-studio.html"],
- "fallback_behavior": "Use neutral source package.",
- "unsupported_native_features": [],
- },
- )
- return {
- "schema_version": "1.0",
- "target": target,
- "native_surface": model["native_surface"],
- "activation": {
- "policy": model["activation_policy"],
- "trigger_description": contract.get("trigger", {}).get("description", ""),
- "manual_activation_supported": True,
- "automatic_activation_note": "Depends on the target client route/catalog implementation.",
- },
- "resources": {
- "strategy": model["resource_strategy"],
- "counts": contract.get("resources", {}).get("counts", {}),
- "generated_files": profile.get("generated_files", []),
- },
- "scripts": {
- "strategy": model["script_strategy"],
- "script_count": contract.get("resources", {}).get("counts", {}).get("scripts", 0),
- "help_smoke_failed_count": contract.get("permissions", {}).get("help_smoke", {}).get("failed_count", 0),
- },
- "permissions": {
- "enforcement": model["permission_enforcement"],
- "native_enforcement": bool(target_permissions.get("native_enforcement")),
- "declared_capabilities": target_permissions.get("declared_capabilities", []),
- "review_required": bool(target_permissions.get("review_required")),
- },
- "review": {
- "artifacts": model["review_artifacts"],
- "fallback_behavior": model["fallback_behavior"],
- "unsupported_native_features": [
- *model.get("unsupported_native_features", []),
- *profile.get("unsupported_features", []),
- ],
- },
- "install_scope": model["install_scope"],
- }
-
-
def load_sources(skill_dir: Path) -> dict[str, Any]:
skill_dir = skill_dir.resolve()
frontmatter = read_frontmatter(skill_dir / "SKILL.md")
@@ -545,7 +266,7 @@ def compile_target_contract(skill_dir: Path, target: str) -> dict[str, Any]:
"preserved_semantics": [],
"unsupported_features": [],
}
- if target not in declared and not (target == "agent-skills" and "agent-skills-compatible" in declared):
+ if target not in declared and not (target in {"agent-skills", "vscode"} and "agent-skills-compatible" in declared):
warnings.append(f"Target is not declared in Skill IR or interface metadata: {target}")
if not sources["ir"]:
warnings.append("Skill IR is missing; compiler used frontmatter fallback")
diff --git a/scripts/compile_skill_targets.py b/scripts/compile_skill_targets.py
new file mode 100644
index 00000000..414740f5
--- /dev/null
+++ b/scripts/compile_skill_targets.py
@@ -0,0 +1,282 @@
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by compile_skill for target platform model and contract builders."
+
+
+TARGET_TRANSFORMS: dict[str, dict[str, Any]] = {
+ "openai": {
+ "adapter_mode": "metadata-adapter",
+ "generated_files": ["targets/openai/adapter.json", "targets/openai/agents/openai.yaml"],
+ "metadata_mapping": {
+ "display_name": "targets/openai/agents/openai.yaml::interface.display_name",
+ "default_prompt": "targets/openai/agents/openai.yaml::interface.default_prompt",
+ "activation": "targets/openai/agents/openai.yaml::compatibility.activation_mode",
+ "execution": "targets/openai/agents/openai.yaml::compatibility.execution_context",
+ "trust": "targets/openai/agents/openai.yaml::compatibility.trust_level",
+ "permissions": "targets/openai/agents/openai.yaml::compatibility.permission_contract",
+ "degradation": "targets/openai/agents/openai.yaml::compatibility.degradation_strategy",
+ },
+ "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": ["client-native script permission prompts are represented as permission contract metadata"],
+ },
+ "claude": {
+ "adapter_mode": "neutral-source-plus-adapter",
+ "generated_files": ["targets/claude/adapter.json", "targets/claude/README.md"],
+ "metadata_mapping": {
+ "display_name": "targets/claude/adapter.json::display_name",
+ "default_prompt": "targets/claude/adapter.json::default_prompt",
+ "activation": "targets/claude/adapter.json::activation_mode",
+ "execution": "targets/claude/adapter.json::execution_context",
+ "trust": "targets/claude/adapter.json::trust_level",
+ "permissions": "targets/claude/adapter.json::target_permission_contract",
+ "degradation": "targets/claude/adapter.json::degradation_strategy",
+ },
+ "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": ["vendor-native metadata fields are carried as adapter JSON and README notes"],
+ },
+ "generic": {
+ "adapter_mode": "agent-skills-compatible",
+ "generated_files": ["targets/generic/adapter.json"],
+ "metadata_mapping": {
+ "display_name": "targets/generic/adapter.json::display_name",
+ "default_prompt": "targets/generic/adapter.json::default_prompt",
+ "activation": "targets/generic/adapter.json::activation_mode",
+ "execution": "targets/generic/adapter.json::execution_context",
+ "trust": "targets/generic/adapter.json::trust_level",
+ "permissions": "targets/generic/adapter.json::target_permission_contract",
+ "degradation": "targets/generic/adapter.json::degradation_strategy",
+ },
+ "preserved_semantics": ["trigger", "workflow-counts", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": [],
+ },
+ "agent-skills-compatible": {
+ "adapter_mode": "neutral-agent-skills-source",
+ "generated_files": ["SKILL.md", "agents/interface.yaml"],
+ "metadata_mapping": {
+ "name": "SKILL.md::frontmatter.name",
+ "description": "SKILL.md::frontmatter.description",
+ "interface": "agents/interface.yaml",
+ "manifest": "manifest.json",
+ },
+ "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": [],
+ },
+ "agent-skills": {
+ "adapter_mode": "neutral-agent-skills-source",
+ "generated_files": ["SKILL.md", "agents/interface.yaml"],
+ "metadata_mapping": {
+ "name": "SKILL.md::frontmatter.name",
+ "description": "SKILL.md::frontmatter.description",
+ "interface": "agents/interface.yaml",
+ "manifest": "manifest.json",
+ },
+ "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": [],
+ },
+ "vscode": {
+ "adapter_mode": "vscode-agent-skills-adapter",
+ "generated_files": ["targets/vscode/adapter.json", "targets/vscode/README.md"],
+ "metadata_mapping": {
+ "name": "folder-name-and-SKILL.md::frontmatter.name",
+ "description": "SKILL.md::frontmatter.description",
+ "interface": "agents/interface.yaml",
+ "permissions": "targets/vscode/adapter.json::target_permission_contract",
+ "install_scope": "targets/vscode/README.md",
+ },
+ "preserved_semantics": ["trigger", "workflow", "resources", "eval-plan", "risk", "governance", "runtime", "trust", "permissions"],
+ "unsupported_features": ["VS Code installation scope is documented but not installed by this compiler"],
+ },
+}
+
+
+TARGET_PERMISSION_MODELS = {
+ "openai": {
+ "model": "metadata-only",
+ "native_enforcement": False,
+ "representation": "targets/openai/agents/openai.yaml::compatibility.permission_contract plus adapter.json",
+ "operator_note": "OpenAI target carries permission metadata for reviewer visibility; host enforcement remains outside the package.",
+ },
+ "claude": {
+ "model": "neutral-source-plus-adapter",
+ "native_enforcement": False,
+ "representation": "targets/claude/adapter.json::target_permission_contract and README notes",
+ "operator_note": "Claude-compatible package keeps permission intent in adapter metadata for install review.",
+ },
+ "generic": {
+ "model": "agent-skills-compatible-metadata",
+ "native_enforcement": False,
+ "representation": "targets/generic/adapter.json::target_permission_contract",
+ "operator_note": "Generic target exposes permission metadata for downstream clients to enforce or review.",
+ },
+ "vscode": {
+ "model": "vscode-workspace-trust-plus-metadata",
+ "native_enforcement": False,
+ "representation": "targets/vscode/adapter.json::target_permission_contract and targets/vscode/README.md install notes",
+ "operator_note": "VS Code target relies on project or user skill installation plus VS Code workspace trust; Yao preserves permission metadata for reviewer and installer checks.",
+ },
+}
+
+
+TARGET_NATIVE_MODELS = {
+ "openai": {
+ "native_surface": "OpenAI-style interface metadata plus neutral Agent Skills source",
+ "activation_policy": "Use frontmatter description for catalog routing and targets/openai/agents/openai.yaml for display name, default prompt, and compatibility metadata.",
+ "resource_strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.",
+ "script_strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.",
+ "permission_enforcement": "metadata-only",
+ "install_scope": "plugin or skill package consumer",
+ "review_artifacts": ["targets/openai/agents/openai.yaml", "targets/openai/adapter.json", "reports/review-studio.html"],
+ "fallback_behavior": "If OpenAI-native metadata is ignored, the package remains readable as neutral Agent Skills source.",
+ "unsupported_native_features": [
+ "client-native permission prompts",
+ "provider-executed scripts",
+ ],
+ },
+ "claude": {
+ "native_surface": "Claude-compatible neutral source folder with adapter notes",
+ "activation_policy": "Use SKILL.md frontmatter description as the primary activation contract and adapter.json for review metadata.",
+ "resource_strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.",
+ "script_strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.",
+ "permission_enforcement": "metadata-fallback",
+ "install_scope": "user or project skill directory",
+ "review_artifacts": ["targets/claude/README.md", "targets/claude/adapter.json", "reports/review-studio.html"],
+ "fallback_behavior": "If Claude-specific metadata is not consumed, SKILL.md and references remain the source of truth.",
+ "unsupported_native_features": [
+ "vendor-native permission enforcement",
+ "provider-specific execution transforms",
+ ],
+ },
+ "generic": {
+ "native_surface": "Agent Skills compatible neutral package",
+ "activation_policy": "Use SKILL.md name and description; consumers decide automatic or manual activation.",
+ "resource_strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.",
+ "script_strategy": "Expose script and permission metadata for downstream clients or installers to enforce.",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "install_scope": "generic Agent Skills compatible root",
+ "review_artifacts": ["targets/generic/adapter.json", "reports/review-studio.html"],
+ "fallback_behavior": "Neutral source is the runtime fallback.",
+ "unsupported_native_features": [],
+ },
+ "agent-skills-compatible": {
+ "native_surface": "Agent Skills standard source tree",
+ "activation_policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
+ "resource_strategy": "Keep optional directories as relative resources next to SKILL.md.",
+ "script_strategy": "Scripts remain local optional resources and should advertise --help when executable.",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "install_scope": "Agent Skills source root",
+ "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
+ "fallback_behavior": "The source tree itself is the target artifact.",
+ "unsupported_native_features": [],
+ },
+ "agent-skills": {
+ "native_surface": "Agent Skills standard source tree",
+ "activation_policy": "Use SKILL.md frontmatter name and description for progressive disclosure.",
+ "resource_strategy": "Keep optional directories as relative resources next to SKILL.md.",
+ "script_strategy": "Scripts remain local optional resources and should advertise --help when executable.",
+ "permission_enforcement": "consumer-enforced-or-metadata-only",
+ "install_scope": "Agent Skills source root",
+ "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
+ "fallback_behavior": "The source tree itself is the target artifact.",
+ "unsupported_native_features": [],
+ },
+ "vscode": {
+ "native_surface": "VS Code/Copilot Agent Skills project or user scope",
+ "activation_policy": "Use folder name plus SKILL.md name/description; keep description under platform limits.",
+ "resource_strategy": "Install as project or user scoped skill source, preserving relative references and scripts.",
+ "script_strategy": "Scripts require workspace trust and operator/client approval outside this compiler.",
+ "permission_enforcement": "client-or-workspace-trust",
+ "install_scope": "VS Code user or project skills directory",
+ "review_artifacts": ["SKILL.md", "agents/interface.yaml", "reports/review-studio.html"],
+ "fallback_behavior": "If VS Code scope is not installed, use the neutral Agent Skills source.",
+ "unsupported_native_features": [
+ "automatic VS Code installation",
+ ],
+ },
+}
+
+
+def target_permission_contract(target: str, permissions: dict[str, Any]) -> dict[str, Any]:
+ model = TARGET_PERMISSION_MODELS.get(
+ target,
+ {
+ "model": "metadata-only",
+ "native_enforcement": False,
+ "representation": "adapter metadata",
+ "operator_note": "Permission semantics are preserved as metadata for reviewer visibility.",
+ },
+ )
+ return {
+ "schema_version": "1.0",
+ "target": target,
+ "permission_model": model["model"],
+ "native_enforcement": model["native_enforcement"],
+ "representation": model["representation"],
+ "review_required": bool(permissions.get("review_required")),
+ "declared_capabilities": permissions.get("declared_capabilities", []),
+ "capability_counts": {
+ name: item.get("script_count", 0)
+ for name, item in permissions.get("capabilities", {}).items()
+ },
+ "evidence": permissions.get("source", ""),
+ "operator_note": model["operator_note"],
+ }
+
+
+def target_native_contract(
+ target: str,
+ profile: dict[str, Any],
+ contract: dict[str, Any],
+ target_permissions: dict[str, Any],
+) -> dict[str, Any]:
+ model = TARGET_NATIVE_MODELS.get(
+ target,
+ {
+ "native_surface": "adapter metadata",
+ "activation_policy": "Carry activation semantics as metadata for the target consumer.",
+ "resource_strategy": "Preserve neutral package resources.",
+ "script_strategy": "Expose script metadata for reviewer visibility.",
+ "permission_enforcement": "metadata-only",
+ "install_scope": "target consumer",
+ "review_artifacts": ["adapter.json", "reports/review-studio.html"],
+ "fallback_behavior": "Use neutral source package.",
+ "unsupported_native_features": [],
+ },
+ )
+ return {
+ "schema_version": "1.0",
+ "target": target,
+ "native_surface": model["native_surface"],
+ "activation": {
+ "policy": model["activation_policy"],
+ "trigger_description": contract.get("trigger", {}).get("description", ""),
+ "manual_activation_supported": True,
+ "automatic_activation_note": "Depends on the target client route/catalog implementation.",
+ },
+ "resources": {
+ "strategy": model["resource_strategy"],
+ "counts": contract.get("resources", {}).get("counts", {}),
+ "generated_files": profile.get("generated_files", []),
+ },
+ "scripts": {
+ "strategy": model["script_strategy"],
+ "script_count": contract.get("resources", {}).get("counts", {}).get("scripts", 0),
+ "help_smoke_failed_count": contract.get("permissions", {}).get("help_smoke", {}).get("failed_count", 0),
+ },
+ "permissions": {
+ "enforcement": model["permission_enforcement"],
+ "native_enforcement": bool(target_permissions.get("native_enforcement")),
+ "declared_capabilities": target_permissions.get("declared_capabilities", []),
+ "review_required": bool(target_permissions.get("review_required")),
+ },
+ "review": {
+ "artifacts": model["review_artifacts"],
+ "fallback_behavior": model["fallback_behavior"],
+ "unsupported_native_features": [
+ *model.get("unsupported_native_features", []),
+ *profile.get("unsupported_features", []),
+ ],
+ },
+ "install_scope": model["install_scope"],
+ }
diff --git a/scripts/context_sizer.py b/scripts/context_sizer.py
index a6e4f8d8..5abc5631 100644
--- a/scripts/context_sizer.py
+++ b/scripts/context_sizer.py
@@ -4,7 +4,7 @@ import json
from pathlib import Path
-TEXT_EXTS = {".md", ".txt", ".yaml", ".yml", ".json", ".py", ".sh", ".js", ".ts"}
+TEXT_EXTS = {".css", ".md", ".txt", ".yaml", ".yml", ".json", ".py", ".sh", ".js", ".ts"}
IGNORED_RELATIVE_DIRS = {
Path("reports") / "release_snapshots",
Path("tests") / "tmp",
@@ -26,6 +26,27 @@ PACKAGE_PATHS = (
"input",
"outputs",
)
+IGNORED_FILE_PATTERNS = {
+ "reports/benchmark_reproducibility*.json",
+ "reports/benchmark_reproducibility*.md",
+ "reports/context_budget*.json",
+ "reports/context_budget*.md",
+ "reports/evidence_consistency*.json",
+ "reports/evidence_consistency*.md",
+ "reports/review-studio*.html",
+ "reports/review-studio*.json",
+ "reports/review-viewer*.html",
+ "reports/review-viewer*.json",
+ "reports/skill-interpretation*.html",
+ "reports/skill-interpretation*.json",
+ "reports/skill-overview*.html",
+ "reports/skill-overview*.json",
+ "reports/world_class_evidence_preflight*.json",
+ "reports/world_class_evidence_preflight*.md",
+ "reports/world_class_evidence_preflight*.html",
+ "reports/*pattern-analysis*.md",
+ "reports/*research-plan*.md",
+}
def estimate_tokens(text: str) -> int:
@@ -61,6 +82,8 @@ def should_ignore(path: Path, skill_dir: Path) -> bool:
rel = path.relative_to(skill_dir)
if any(rel == ignored or ignored in rel.parents for ignored in IGNORED_RELATIVE_DIRS):
return True
+ if any(rel.match(pattern) for pattern in IGNORED_FILE_PATTERNS):
+ return True
return len(rel.parts) >= 2 and rel.parts[0] == "tests" and rel.parts[1].startswith("tmp_")
diff --git a/scripts/cross_packager.py b/scripts/cross_packager.py
index 6799afee..f71b7e7d 100644
--- a/scripts/cross_packager.py
+++ b/scripts/cross_packager.py
@@ -3,10 +3,12 @@ import argparse
import json
import shutil
import zipfile
-from pathlib import Path
+from pathlib import Path, PurePosixPath
import yaml
from compile_skill import compile_target_contract
+from cross_packager_contracts import PLATFORM_CONTRACTS
+from skill_ir_paths import find_skill_ir as find_skill_ir_document
def display_path(path: Path, root: Path) -> str:
@@ -48,20 +50,15 @@ def read_interface(skill_dir: Path) -> dict:
def find_skill_ir(skill_dir: Path, name: str) -> tuple[dict, str]:
- candidates = [
- skill_dir / "reports" / "skill-ir.json",
- skill_dir / "skill-ir" / "examples" / f"{name}.json",
- skill_dir / "skill-ir" / "examples" / f"{skill_dir.name}.json",
- ]
- seen = set()
- for path in candidates:
- if path in seen:
- continue
- seen.add(path)
- payload = read_json(path)
- if payload:
- return payload, display_path(path, skill_dir)
- return {}, "frontmatter-fallback"
+ return find_skill_ir_document(skill_dir, name, fallback_source="frontmatter-fallback")
+
+
+def package_name_from_manifest(manifest: dict, skill_dir: Path) -> str:
+ name = str(manifest.get("name") or "").strip()
+ if name:
+ return name
+ frontmatter = read_frontmatter(skill_dir / "SKILL.md")
+ return str(frontmatter.get("name") or skill_dir.name)
def require_fields(payload: dict, fields: list[str], label: str) -> None:
@@ -147,13 +144,17 @@ def build_semantic_contract(
"targets": target_values,
"source_files_count": count_list(ir, "source_files") if ir else 0,
}
+ alias_declared = (
+ platform in {"agent-skills", "vscode"} and "agent-skills-compatible" in target_values
+ )
semantic_parity = {
"source": "skill-ir" if ir else "frontmatter-fallback",
"ir_source": ir_source,
"name_matches_ir": bool(ir) and frontmatter_name == name,
"description_matches_ir": bool(ir) and frontmatter_description == description,
"platform_declared_in_ir": platform in target_values
- or (platform == "generic" and "agent-skills-compatible" in target_values),
+ or (platform == "generic" and "agent-skills-compatible" in target_values)
+ or alias_declared,
"platform_declared_in_interface": platform in adapter_targets,
"display_name_present": bool(interface.get("display_name")),
"default_prompt_present": bool(interface.get("default_prompt")),
@@ -209,7 +210,7 @@ def build_manifest(skill_dir: Path, platform: str) -> dict:
"description": semantic["description"],
"version": manifest.get("version") or frontmatter.get("version", "1.0.0"),
"platform": platform,
- "skill_root": skill_dir.name,
+ "skill_root": semantic["name"],
"job_to_be_done": semantic["job_to_be_done"],
"ir_source": semantic["ir_source"],
"ir_schema_version": semantic["ir_schema_version"],
@@ -250,115 +251,7 @@ def build_manifest(skill_dir: Path, platform: str) -> dict:
}
-PLATFORM_CONTRACTS = {
- "openai": {
- "required_fields": [
- "name",
- "description",
- "version",
- "display_name",
- "short_description",
- "default_prompt",
- "job_to_be_done",
- "ir_source",
- "ir_schema_version",
- "semantic_contract",
- "semantic_parity",
- "canonical_metadata",
- "canonical_format",
- "activation_mode",
- "execution_context",
- "shell",
- "trust_level",
- "remote_inline_execution",
- "degradation_strategy",
- "portability_profile",
- "permission_contract",
- "target_permission_contract",
- "target_native_contract",
- ],
- "required_files": ["targets/openai/adapter.json", "targets/openai/agents/openai.yaml"],
- "field_mapping": {
- "display_name": "interface.display_name",
- "short_description": "interface.short_description",
- "default_prompt": "interface.default_prompt",
- "execution_context": "compatibility.execution.context",
- "shell": "compatibility.execution.shell",
- },
- },
- "claude": {
- "required_fields": [
- "name",
- "description",
- "version",
- "display_name",
- "short_description",
- "default_prompt",
- "job_to_be_done",
- "ir_source",
- "ir_schema_version",
- "semantic_contract",
- "semantic_parity",
- "canonical_metadata",
- "canonical_format",
- "activation_mode",
- "execution_context",
- "shell",
- "trust_level",
- "remote_inline_execution",
- "degradation_strategy",
- "portability_profile",
- "permission_contract",
- "target_permission_contract",
- "target_native_contract",
- ],
- "required_files": ["targets/claude/adapter.json", "targets/claude/README.md"],
- "field_mapping": {
- "display_name": "adapter.display_name",
- "short_description": "adapter.short_description",
- "default_prompt": "adapter.default_prompt",
- "execution_context": "compatibility.execution.context",
- "shell": "compatibility.execution.shell",
- },
- },
- "generic": {
- "required_fields": [
- "name",
- "description",
- "version",
- "display_name",
- "short_description",
- "default_prompt",
- "job_to_be_done",
- "ir_source",
- "ir_schema_version",
- "semantic_contract",
- "semantic_parity",
- "canonical_metadata",
- "canonical_format",
- "activation_mode",
- "execution_context",
- "shell",
- "trust_level",
- "remote_inline_execution",
- "degradation_strategy",
- "portability_profile",
- "permission_contract",
- "target_permission_contract",
- "target_native_contract",
- ],
- "required_files": ["targets/generic/adapter.json"],
- "field_mapping": {
- "display_name": "adapter.display_name",
- "short_description": "adapter.short_description",
- "default_prompt": "adapter.default_prompt",
- "execution_context": "compatibility.execution.context",
- "shell": "compatibility.execution.shell",
- },
- },
-}
-
-EXCLUDED_ARCHIVE_PARTS = {".git", "__pycache__", ".venv", "venv", "node_modules", "dist"}
+EXCLUDED_ARCHIVE_PARTS = {".git", ".previews", "__pycache__", ".venv", "venv", "node_modules", "dist"}
def should_skip_archive_path(rel_path: Path) -> bool:
@@ -454,7 +347,7 @@ def write_adapter(skill_dir: Path, out_dir: Path, platform: str) -> Path:
notes = target_dir / "README.md"
native = payload["target_native_contract"]
notes.write_text(
- f"# Claude-Compatible Package\n\nUse `{skill_dir.name}` with its neutral source files. This target does not require vendor metadata by default.\n\n"
+ f"# Claude-Compatible Package\n\nUse `{payload['name']}` with its neutral source files. This target does not require vendor metadata by default.\n\n"
f"Native surface: {native['native_surface']}.\n\n"
f"Activation: {native['activation']['policy']}\n\n"
f"Resources: {native['resources']['strategy']}\n\n"
@@ -463,16 +356,34 @@ def write_adapter(skill_dir: Path, out_dir: Path, platform: str) -> Path:
encoding="utf-8",
)
payload["install_hint"] = f"Use the packaged skill directly; this target relies on SKILL.md and optional neutral metadata."
+ elif platform == "vscode":
+ notes = target_dir / "README.md"
+ native = payload["target_native_contract"]
+ notes.write_text(
+ f"# VS Code / Copilot Agent Skills Package\n\n"
+ f"Install `{payload['name']}` as a VS Code user or project scoped Agent Skill. Keep the folder name aligned with `SKILL.md` frontmatter name.\n\n"
+ f"Native surface: {native['native_surface']}.\n\n"
+ f"Activation: {native['activation']['policy']}\n\n"
+ f"Resources: {native['resources']['strategy']}\n\n"
+ f"Scripts: {native['scripts']['strategy']}\n\n"
+ f"Permission model: {payload['target_permission_contract']['permission_model']}. "
+ "Review `target_permission_contract`, workspace trust, and `reports/security_trust_report.md` before running scripts.\n\n"
+ "This adapter does not perform automatic VS Code installation; it preserves the reviewed source package plus install notes.\n",
+ encoding="utf-8",
+ )
+ payload["install_hint"] = (
+ "Install the package as a VS Code user or project scoped Agent Skill; use targets/vscode/README.md for scope and trust notes."
+ )
else:
- payload["install_hint"] = f"Use {skill_dir.name} as an Agent Skills compatible package."
+ payload["install_hint"] = f"Use {payload['name']} as an Agent Skills compatible package."
path = target_dir / "adapter.json"
payload["contract"] = PLATFORM_CONTRACTS.get(platform, PLATFORM_CONTRACTS["generic"])
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
return path
-def make_zip(skill_dir: Path, out_dir: Path) -> Path:
- zip_path = out_dir / f"{skill_dir.name}.zip"
+def make_zip(skill_dir: Path, out_dir: Path, package_name: str) -> Path:
+ zip_path = out_dir / f"{package_name}.zip"
skill_root = skill_dir.resolve()
out_root = out_dir.resolve()
with zipfile.ZipFile(zip_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
@@ -487,17 +398,18 @@ def make_zip(skill_dir: Path, out_dir: Path) -> Path:
rel_path = path.relative_to(skill_dir)
if should_skip_archive_path(rel_path):
continue
- zf.write(path, arcname=str(path.relative_to(skill_dir.parent)))
+ zf.write(path, arcname=str(PurePosixPath(package_name, *rel_path.parts)))
return zip_path
-def copy_manifest(skill_dir: Path, out_dir: Path) -> Path:
+def copy_manifest(skill_dir: Path, out_dir: Path) -> tuple[Path, str]:
manifest_path = out_dir / "manifest.json"
+ manifest = build_manifest(skill_dir, "generic")
manifest_path.write_text(
- json.dumps(build_manifest(skill_dir, "generic"), ensure_ascii=False, indent=2),
+ json.dumps(manifest, ensure_ascii=False, indent=2),
encoding="utf-8",
)
- return manifest_path
+ return manifest_path, package_name_from_manifest(manifest, skill_dir)
def load_expectations(path: Path | None) -> dict:
@@ -511,9 +423,9 @@ def validate_exports(out_dir: Path, expectations: dict) -> dict:
required_targets = expectations.get("required_targets", [])
required_fields = expectations.get("required_fields", [])
required_by_target = {
- "openai": expectations.get("openai_required_files", []),
- "claude": expectations.get("claude_required_files", []),
- "generic": expectations.get("generic_required_files", []),
+ key[: -len("_required_files")]: value
+ for key, value in expectations.items()
+ if key.endswith("_required_files")
}
for target in required_targets:
@@ -534,7 +446,7 @@ def validate_exports(out_dir: Path, expectations: dict) -> dict:
def main() -> None:
parser = argparse.ArgumentParser(description="Generate lightweight cross-platform packaging artifacts.")
parser.add_argument("skill_dir", help="Path to the skill directory")
- parser.add_argument("--platform", action="append", default=[], help="Target platform: openai, claude, generic")
+ parser.add_argument("--platform", action="append", default=[], help="Target platform: openai, claude, generic, vscode")
parser.add_argument("--output-dir", default="dist", help="Output directory")
parser.add_argument("--expectations", help="JSON file describing packaging expectations")
parser.add_argument("--zip", action="store_true", help="Create a zip package")
@@ -548,12 +460,12 @@ def main() -> None:
if out_dir.exists():
shutil.rmtree(out_dir)
out_dir.mkdir(parents=True)
- manifest = copy_manifest(skill_dir, out_dir)
+ manifest, package_name = copy_manifest(skill_dir, out_dir)
generated.append(str(manifest))
for platform in (args.platform or ["generic"]):
generated.append(str(write_adapter(skill_dir, out_dir, platform)))
if args.zip:
- generated.append(str(make_zip(skill_dir, out_dir)))
+ generated.append(str(make_zip(skill_dir, out_dir, package_name)))
except (FileNotFoundError, ValueError, yaml.YAMLError) as exc:
failures.append(str(exc))
diff --git a/scripts/cross_packager_contracts.py b/scripts/cross_packager_contracts.py
new file mode 100644
index 00000000..f5c2b8bf
--- /dev/null
+++ b/scripts/cross_packager_contracts.py
@@ -0,0 +1,107 @@
+"""Platform contract definitions for cross-platform skill packages."""
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by cross_packager.py to keep platform contract data separate from packaging flow."
+
+
+COMMON_REQUIRED_FIELDS = [
+ "name",
+ "description",
+ "version",
+ "display_name",
+ "short_description",
+ "default_prompt",
+ "job_to_be_done",
+ "ir_source",
+ "ir_schema_version",
+ "semantic_contract",
+ "semantic_parity",
+ "canonical_metadata",
+ "canonical_format",
+ "activation_mode",
+ "execution_context",
+ "shell",
+ "trust_level",
+ "remote_inline_execution",
+ "degradation_strategy",
+ "portability_profile",
+ "permission_contract",
+ "target_permission_contract",
+ "target_native_contract",
+]
+
+STANDARD_FIELD_MAPPING = {
+ "display_name": "adapter.display_name",
+ "short_description": "adapter.short_description",
+ "default_prompt": "adapter.default_prompt",
+ "execution_context": "compatibility.execution.context",
+ "shell": "compatibility.execution.shell",
+}
+
+
+def interface_field_mapping() -> dict[str, str]:
+ return {
+ "display_name": "interface.display_name",
+ "short_description": "interface.short_description",
+ "default_prompt": "interface.default_prompt",
+ "execution_context": "compatibility.execution.context",
+ "shell": "compatibility.execution.shell",
+ }
+
+
+PLATFORM_CONTRACTS = {
+ "openai": {
+ "required_fields": list(COMMON_REQUIRED_FIELDS),
+ "required_files": ["targets/openai/adapter.json", "targets/openai/agents/openai.yaml"],
+ "field_mapping": interface_field_mapping(),
+ },
+ "claude": {
+ "required_fields": list(COMMON_REQUIRED_FIELDS),
+ "required_files": ["targets/claude/adapter.json", "targets/claude/README.md"],
+ "field_mapping": dict(STANDARD_FIELD_MAPPING),
+ },
+ "generic": {
+ "required_fields": list(COMMON_REQUIRED_FIELDS),
+ "required_files": ["targets/generic/adapter.json"],
+ "field_mapping": dict(STANDARD_FIELD_MAPPING),
+ },
+ "vscode": {
+ "required_fields": [
+ "name",
+ "description",
+ "version",
+ "display_name",
+ "short_description",
+ "default_prompt",
+ "job_to_be_done",
+ "ir_source",
+ "ir_schema_version",
+ "semantic_contract",
+ "semantic_parity",
+ "compiler",
+ "compiled_contract",
+ "permission_contract",
+ "target_permission_contract",
+ "target_native_contract",
+ "target_transform",
+ "canonical_metadata",
+ "canonical_format",
+ "activation_mode",
+ "execution_context",
+ "shell",
+ "trust_level",
+ "remote_inline_execution",
+ "degradation_strategy",
+ "portability_profile",
+ ],
+ "required_files": ["targets/vscode/adapter.json", "targets/vscode/README.md"],
+ "field_mapping": {
+ "name": "SKILL.md::frontmatter.name and folder name",
+ "description": "SKILL.md::frontmatter.description",
+ "display_name": "agents/interface.yaml::interface.display_name",
+ "execution_context": "compatibility.execution.context",
+ "permissions": "adapter.target_permission_contract",
+ },
+ },
+}
diff --git a/scripts/description_optimizer_reporting.py b/scripts/description_optimizer_reporting.py
new file mode 100644
index 00000000..5889c0b1
--- /dev/null
+++ b/scripts/description_optimizer_reporting.py
@@ -0,0 +1,141 @@
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by optimize_description.py to render description optimization reports."
+
+
+def render_markdown(report: dict, title: str) -> str:
+ lines = [
+ f"# {title}",
+ "",
+ f"Winner: `{report['winner']['label']}`",
+ "",
+ f"- current tokens: `{report['current_candidate']['estimated_tokens']}`",
+ f"- winner tokens: `{report['winner']['estimated_tokens']}`",
+ ]
+ if report["baseline"]:
+ lines.append(f"- baseline tokens: `{report['baseline']['estimated_tokens']}`")
+ lines.extend(
+ [
+ "",
+ "## Winner",
+ "",
+ report["winner"]["description"],
+ "",
+ "## Candidate Ranking",
+ "",
+ "| Candidate | Tokens | Dev FP | Dev FN | Dev Near | Holdout FP | Holdout FN |",
+ "| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
+ ]
+ )
+ for candidate in report["candidates"]:
+ holdout = candidate.get("holdout", {})
+ lines.append(
+ f"| `{candidate['label']}` | {candidate['estimated_tokens']} | {candidate['dev']['false_positives']} | {candidate['dev']['false_negatives']} | {candidate['dev']['near_neighbor_pass_rate']} | {holdout.get('false_positives', '-')} | {holdout.get('false_negatives', '-')} |"
+ )
+
+ lines.extend(
+ [
+ "",
+ "## Acceptance Gates",
+ "",
+ "| Gate | Winner FP | Winner FN | Current FP | Current FN | Baseline FP | Baseline FN |",
+ "| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
+ ]
+ )
+ for gate_name, gate in (
+ ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
+ ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
+ ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
+ ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
+ ):
+ winner_gate = gate.get("winner") or {}
+ current_gate = gate.get("current") or {}
+ baseline_gate = gate.get("baseline") or {}
+ if not winner_gate and not current_gate and not baseline_gate:
+ continue
+ lines.append(
+ f"| {gate_name} | {winner_gate.get('false_positives', '-')} | {winner_gate.get('false_negatives', '-')} | {current_gate.get('false_positives', '-')} | {current_gate.get('false_negatives', '-')} | {baseline_gate.get('false_positives', '-')} | {baseline_gate.get('false_negatives', '-')} |"
+ )
+
+ lines.extend(
+ [
+ "",
+ "## Calibration",
+ "",
+ "| Gate | Winner Gap | Winner Risk | Winner Boundary Rate | Current Gap | Baseline Gap |",
+ "| --- | ---: | --- | ---: | ---: | ---: |",
+ ]
+ )
+ for gate_name, gate in (
+ ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
+ ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
+ ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
+ ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
+ ):
+ winner_calibration = gate.get("winner_calibration") or {}
+ current_calibration = gate.get("current_calibration") or {}
+ baseline_calibration = gate.get("baseline_calibration") or {}
+ if not winner_calibration and not current_calibration and not baseline_calibration:
+ continue
+ lines.append(
+ f"| {gate_name} | {winner_calibration.get('score_gap', '-')} | {winner_calibration.get('risk_band', '-')} | {winner_calibration.get('boundary_case_rate', '-')} | {current_calibration.get('score_gap', '-')} | {baseline_calibration.get('score_gap', '-')} |"
+ )
+
+ lines.extend(
+ [
+ "",
+ "## Judge Blind Summary",
+ "",
+ "| Gate | Winner Agreement | Winner Mean Confidence | Current Agreement | Baseline Agreement |",
+ "| --- | ---: | ---: | ---: | ---: |",
+ ]
+ )
+ judge_gate = report["acceptance_gates"]["judge_blind_holdout_non_regression"]
+ judge_winner = (judge_gate.get("winner") or {}).get("judge_summary") or {}
+ judge_current = (judge_gate.get("current") or {}).get("judge_summary") or {}
+ judge_baseline = (judge_gate.get("baseline") or {}).get("judge_summary") or {}
+ if judge_winner or judge_current or judge_baseline:
+ lines.append(
+ f"| Judge Blind Holdout | {judge_winner.get('agreement_rate', '-')} | {judge_winner.get('mean_confidence', '-')} | {judge_current.get('agreement_rate', '-')} | {judge_baseline.get('agreement_rate', '-')} |"
+ )
+
+ lines.extend(
+ [
+ "",
+ "## Family Health",
+ "",
+ "| Gate | Winner Clean Families | Winner Weakest Family | Current Clean Families | Baseline Clean Families |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for gate_name, gate in (
+ ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
+ ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
+ ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
+ ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
+ ):
+ winner_health = gate.get("winner_family_health") or {}
+ current_health = gate.get("current_family_health") or {}
+ baseline_health = gate.get("baseline_family_health") or {}
+ if not winner_health and not current_health and not baseline_health:
+ continue
+ weakest = winner_health.get("weakest_family") or {}
+ weakest_label = (
+ f"{weakest.get('family')} ({weakest.get('errors')} errors)"
+ if weakest.get("family")
+ else "-"
+ )
+ lines.append(
+ f"| {gate_name} | {winner_health.get('clean_family_count', '-')}/{winner_health.get('family_count', '-')} | {weakest_label} | {current_health.get('clean_family_count', '-')}/{current_health.get('family_count', '-')} | {baseline_health.get('clean_family_count', '-')}/{baseline_health.get('family_count', '-')} |"
+ )
+
+ lines.extend(
+ [
+ "",
+ "## Selection Logic",
+ "",
+ "Ordered by:",
+ ]
+ )
+ for item in report["selection_logic"]["priority"]:
+ lines.append(f"- {item}")
+ return "\n".join(lines) + "\n"
diff --git a/scripts/emit_telemetry_event.py b/scripts/emit_telemetry_event.py
new file mode 100644
index 00000000..d737703c
--- /dev/null
+++ b/scripts/emit_telemetry_event.py
@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import shlex
+from pathlib import Path
+from typing import Any
+
+from render_adoption_drift_report import (
+ ALLOWED_ACTIVATION_TYPES,
+ ALLOWED_EVENTS,
+ ALLOWED_FAILURE_TYPES,
+ ALLOWED_OUTCOMES,
+ ALLOWED_SOURCES,
+ display_path,
+ normalize_event,
+ skill_defaults,
+)
+
+
+def default_spool_path(skill_dir: Path) -> Path:
+ return skill_dir / ".yao" / "telemetry_spool" / "external_events.jsonl"
+
+
+def append_event(path: Path, event: dict[str, str]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("a", encoding="utf-8") as handle:
+ handle.write(json.dumps(event, ensure_ascii=False, sort_keys=True) + "\n")
+
+
+def import_command(skill_dir: Path, output_jsonl: Path) -> str:
+ return shlex.join(
+ [
+ "python3",
+ "scripts/yao.py",
+ "telemetry-import",
+ display_path(skill_dir),
+ "--input-jsonl",
+ display_path(output_jsonl),
+ ]
+ )
+
+
+def emit_event(
+ skill_dir: Path,
+ output_jsonl: Path | None = None,
+ event_name: str = "script_run",
+ activation_type: str = "manual",
+ outcome: str = "unknown",
+ failure_type: str = "none",
+ source: str = "external",
+ command: str = "external-client",
+ timestamp: str | None = None,
+ skill_name: str | None = None,
+ version: str | None = None,
+ dry_run: bool = False,
+) -> dict[str, Any]:
+ skill_dir = skill_dir.resolve()
+ output_jsonl = (output_jsonl or default_spool_path(skill_dir)).resolve()
+ raw_event: dict[str, Any] = {
+ "event": event_name,
+ "activation_type": activation_type,
+ "outcome": outcome,
+ "failure_type": failure_type,
+ "source": source,
+ "command": command,
+ }
+ if timestamp:
+ raw_event["timestamp"] = timestamp
+ if skill_name:
+ raw_event["skill"] = skill_name
+ if version:
+ raw_event["version"] = version
+ event, failures = normalize_event(raw_event, skill_defaults(skill_dir), "emit")
+ if event and not failures and not dry_run:
+ append_event(output_jsonl, event)
+ return {
+ "ok": not failures,
+ "schema_version": "1.0",
+ "skill_dir": display_path(skill_dir),
+ "output_jsonl": display_path(output_jsonl),
+ "dry_run": dry_run,
+ "emitted": bool(event and not failures and not dry_run),
+ "event": event or {},
+ "failures": failures,
+ "artifacts": {
+ "spool_jsonl": display_path(output_jsonl),
+ "import_command": import_command(skill_dir, output_jsonl),
+ },
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Emit one metadata-only telemetry event for later import.")
+ parser.add_argument("skill_dir", nargs="?", default=".")
+ parser.add_argument("--output-jsonl")
+ parser.add_argument("--event", choices=sorted(ALLOWED_EVENTS), default="script_run")
+ parser.add_argument("--activation-type", choices=sorted(ALLOWED_ACTIVATION_TYPES), default="manual")
+ parser.add_argument("--outcome", choices=sorted(ALLOWED_OUTCOMES), default="unknown")
+ parser.add_argument("--failure-type", choices=sorted(ALLOWED_FAILURE_TYPES), default="none")
+ parser.add_argument("--source", choices=sorted(ALLOWED_SOURCES), default="external")
+ parser.add_argument("--command", default="external-client")
+ parser.add_argument("--timestamp")
+ parser.add_argument("--skill-name")
+ parser.add_argument("--version")
+ parser.add_argument("--dry-run", action="store_true")
+ args = parser.parse_args()
+
+ report = emit_event(
+ Path(args.skill_dir),
+ output_jsonl=Path(args.output_jsonl) if args.output_jsonl else None,
+ event_name=args.event,
+ activation_type=args.activation_type,
+ outcome=args.outcome,
+ failure_type=args.failure_type,
+ source=args.source,
+ command=args.command,
+ timestamp=args.timestamp,
+ skill_name=args.skill_name,
+ version=args.version,
+ dry_run=args.dry_run,
+ )
+ print(json.dumps(report, ensure_ascii=False, indent=2))
+ if not report["ok"]:
+ raise SystemExit(2)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/evidence_consistency_artifact_roles.py b/scripts/evidence_consistency_artifact_roles.py
new file mode 100644
index 00000000..12e82fe3
--- /dev/null
+++ b/scripts/evidence_consistency_artifact_roles.py
@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+"""Compare artifact-role handoffs across world-class evidence reports."""
+
+from pathlib import Path
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_evidence_consistency.py to compare preflight and Review Studio artifact-role contracts."
+
+
+def _compare_check(
+ *,
+ key: str,
+ label: str,
+ expected: Any,
+ actual: Any,
+ paths: list[str],
+ detail: str,
+) -> dict[str, Any]:
+ return {
+ "key": key,
+ "label": label,
+ "status": "pass" if expected == actual else "fail",
+ "expected": expected,
+ "actual": actual,
+ "paths": paths,
+ "detail": detail,
+ }
+
+
+def role_contract_signature(contract: dict[str, Any]) -> dict[str, Any]:
+ roles = {
+ str(item.get("role", "")): item
+ for item in contract.get("roles", [])
+ if isinstance(item, dict) and str(item.get("role", "")).strip()
+ }
+ return {
+ "role_source": contract.get("role_source"),
+ "counts_as_evidence": contract.get("counts_as_evidence"),
+ "artifact_prefill_counts_as_evidence": contract.get("artifact_prefill_counts_as_evidence"),
+ "submission_ref_total_count": contract.get("submission_ref_total_count"),
+ "submission_ref_ready_count": contract.get("submission_ref_ready_count"),
+ "supporting_evidence_total_count": contract.get("supporting_evidence_total_count"),
+ "supporting_evidence_ready_count": contract.get("supporting_evidence_ready_count"),
+ "submission_ref_copy_to_artifact_refs": roles.get("submission-ref", {}).get("copy_to_artifact_refs"),
+ "supporting_evidence_copy_to_artifact_refs": roles.get("supporting-evidence", {}).get(
+ "copy_to_artifact_refs"
+ ),
+ }
+
+
+def preflight_role_signatures(world_class_preflight: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ signatures: dict[str, dict[str, Any]] = {}
+ items = world_class_preflight.get("items", []) if isinstance(world_class_preflight, dict) else []
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ key = str(item.get("evidence_key", "")).strip()
+ submission_kit = item.get("submission_kit", {}) if isinstance(item.get("submission_kit", {}), dict) else {}
+ contract = (
+ submission_kit.get("artifact_role_contract", {})
+ if isinstance(submission_kit.get("artifact_role_contract", {}), dict)
+ else {}
+ )
+ if key and contract:
+ signatures[key] = role_contract_signature(contract)
+ return signatures
+
+
+def review_studio_role_signatures(review_studio: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ actions = review_studio.get("review_actions", []) if isinstance(review_studio, dict) else []
+ for action in actions:
+ if not isinstance(action, dict) or action.get("gate_key") != "world-class-evidence":
+ continue
+ signatures: dict[str, dict[str, Any]] = {}
+ for step in action.get("evidence_steps", []):
+ if not isinstance(step, dict):
+ continue
+ key = str(step.get("key", "")).strip()
+ contract = (
+ step.get("artifact_role_contract", {})
+ if isinstance(step.get("artifact_role_contract", {}), dict)
+ else {}
+ )
+ if key and contract:
+ signatures[key] = role_contract_signature(contract)
+ return signatures
+ return {}
+
+
+def build_preflight_artifact_role_handoff_checks(
+ *,
+ skill_dir: Path,
+ world_class_preflight: dict[str, Any],
+ review_studio: dict[str, Any],
+ report_paths: dict[str, str],
+) -> list[dict[str, Any]]:
+ preflight_submissions = (
+ world_class_preflight.get("submissions", {})
+ if isinstance(world_class_preflight.get("submissions", {}), dict)
+ else {}
+ )
+ preflight_commands = (
+ preflight_submissions.get("commands", {})
+ if isinstance(preflight_submissions.get("commands", {}), dict)
+ else {}
+ )
+ preflight_role_contract = (
+ preflight_submissions.get("artifact_role_contract", {})
+ if isinstance(preflight_submissions.get("artifact_role_contract", {}), dict)
+ else {}
+ )
+ preflight_roles = {
+ str(item.get("role", "")): item for item in preflight_role_contract.get("roles", []) if isinstance(item, dict)
+ }
+ default_submissions_dir = "evidence/world_class/submissions"
+ expected_preflight_handoff = {
+ "directory": default_submissions_dir,
+ "drafts_count_as_evidence": False,
+ "preflight_counts_submission_as_completion": False,
+ "html_report": "reports/world_class_evidence_preflight.html",
+ "html_exists": True,
+ "prepare_submission": f"python3 scripts/yao.py world-class-submission-kit . --output-dir {default_submissions_dir}",
+ "prepare_prefilled_submission": (
+ f"python3 scripts/yao.py world-class-submission-kit . --output-dir {default_submissions_dir} "
+ "--prefill-artifacts"
+ ),
+ "validate_intake": f"python3 scripts/yao.py world-class-intake . --submissions-dir {default_submissions_dir}",
+ "submission_review": f"python3 scripts/yao.py world-class-submission-review . --submissions-dir {default_submissions_dir}",
+ "refresh_ledger": f"python3 scripts/yao.py world-class-ledger . --submissions-dir {default_submissions_dir}",
+ "guard_claim": "python3 scripts/yao.py world-class-claim-guard .",
+ "artifact_prefill_counts_as_evidence": False,
+ "artifact_role_source": "world-class-submission-kit",
+ "artifact_role_counts_as_evidence": False,
+ "artifact_role_prefill_counts_as_evidence": False,
+ "submission_ref_role_present": True,
+ "supporting_evidence_role_present": True,
+ "submission_ref_copy_to_artifact_refs": True,
+ "supporting_evidence_copy_to_artifact_refs": False,
+ "submission_ref_total_present": True,
+ "supporting_evidence_total_present": True,
+ }
+ actual_preflight_handoff = {
+ "directory": preflight_submissions.get("directory"),
+ "drafts_count_as_evidence": preflight_submissions.get("drafts_count_as_evidence"),
+ "preflight_counts_submission_as_completion": preflight_submissions.get(
+ "preflight_counts_submission_as_completion"
+ ),
+ "html_report": world_class_preflight.get("artifacts", {}).get("html")
+ if isinstance(world_class_preflight.get("artifacts", {}), dict)
+ else None,
+ "html_exists": (skill_dir / "reports" / "world_class_evidence_preflight.html").exists(),
+ "prepare_submission": preflight_commands.get("prepare_submission"),
+ "prepare_prefilled_submission": preflight_commands.get("prepare_prefilled_submission"),
+ "validate_intake": preflight_commands.get("validate_intake"),
+ "submission_review": preflight_commands.get("submission_review"),
+ "refresh_ledger": preflight_commands.get("refresh_ledger"),
+ "guard_claim": preflight_commands.get("guard_claim"),
+ "artifact_prefill_counts_as_evidence": preflight_submissions.get("artifact_prefill_counts_as_evidence"),
+ "artifact_role_source": preflight_role_contract.get("role_source"),
+ "artifact_role_counts_as_evidence": preflight_role_contract.get("counts_as_evidence"),
+ "artifact_role_prefill_counts_as_evidence": preflight_role_contract.get(
+ "artifact_prefill_counts_as_evidence"
+ ),
+ "submission_ref_role_present": "submission-ref" in preflight_roles,
+ "supporting_evidence_role_present": "supporting-evidence" in preflight_roles,
+ "submission_ref_copy_to_artifact_refs": preflight_roles.get("submission-ref", {}).get(
+ "copy_to_artifact_refs"
+ ),
+ "supporting_evidence_copy_to_artifact_refs": preflight_roles.get("supporting-evidence", {}).get(
+ "copy_to_artifact_refs"
+ ),
+ "submission_ref_total_present": int(preflight_role_contract.get("submission_ref_total_count", 0)) > 0,
+ "supporting_evidence_total_present": int(preflight_role_contract.get("supporting_evidence_total_count", 0))
+ > 0,
+ }
+ return [
+ _compare_check(
+ key="preflight-submission-kit-handoff",
+ label="Preflight exposes a safe submission-kit handoff",
+ expected=expected_preflight_handoff,
+ actual=actual_preflight_handoff,
+ paths=[report_paths["world_class_preflight"], "reports/world_class_evidence_preflight.html"],
+ detail=(
+ "Preflight must give operators the exact draft, SHA-prefill, intake, review, ledger, "
+ "and claim-guard commands without letting drafts, prefill, or submissions count as accepted evidence."
+ ),
+ ),
+ _compare_check(
+ key="review-studio-preflight-artifact-role-handoff",
+ label="Review Studio mirrors preflight artifact roles",
+ expected=preflight_role_signatures(world_class_preflight),
+ actual=review_studio_role_signatures(review_studio),
+ paths=[report_paths["world_class_preflight"], report_paths["review_studio"]],
+ detail=(
+ "The Review Studio world-class action card must carry the same submission-ref versus "
+ "supporting-evidence contract as the preflight handoff."
+ ),
+ ),
+ ]
diff --git a/scripts/evidence_consistency_core.py b/scripts/evidence_consistency_core.py
new file mode 100644
index 00000000..9a0dc618
--- /dev/null
+++ b/scripts/evidence_consistency_core.py
@@ -0,0 +1,328 @@
+#!/usr/bin/env python3
+"""Shared helpers for cross-report evidence consistency checks."""
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_evidence_consistency.py for shared report loading, comparison, and Markdown rendering helpers."
+
+REQUIRED_REPORTS = {
+ "benchmark": "reports/benchmark_reproducibility.json",
+ "overview": "reports/skill-overview.json",
+ "interpretation": "reports/skill-interpretation.json",
+ "adoption": "reports/adoption_drift_report.json",
+ "world_class_ledger": "reports/world_class_evidence_ledger.json",
+ "world_class_plan": "reports/world_class_evidence_plan.json",
+ "world_class_intake": "reports/world_class_evidence_intake.json",
+ "world_class_preflight": "reports/world_class_evidence_preflight.json",
+ "world_class_submission_review": "reports/world_class_submission_review.json",
+ "world_class_operator_runbook": "reports/world_class_operator_runbook.json",
+ "skill_os2_coverage": "reports/skill_os2_coverage.json",
+ "review_studio": "reports/review-studio.json",
+ "package_verification": "reports/package_verification.json",
+ "install_simulation": "reports/install_simulation.json",
+ "trust": "reports/security_trust_report.json",
+ "context_budget": "reports/context_budget.json",
+ "world_class_claim_guard": "reports/world_class_claim_guard.json",
+}
+REQUIRED_TEXT_REPORTS = {
+ "skill_os2_review": "reports/skill-os-2-review.md",
+}
+BENCHMARK_SUMMARY_KEYS = [
+ "release_lock_ready",
+ "required_artifact_count",
+ "missing_artifact_count",
+ "source_contract_sha256",
+ "archive_sha256",
+ "world_class_ledger_pending_count",
+ "world_class_source_check_count",
+ "world_class_source_pass_count",
+ "world_class_source_blocked_count",
+ "beta_test_ready",
+ "beta_test_blocker_count",
+ "beta_test_deferred_evidence_count",
+ "public_claim_ready",
+ "public_claim_blocker_count",
+]
+ADOPTION_SUMMARY_KEYS = [
+ "event_count",
+ "adoption_sample_count",
+ "activation_count",
+ "accepted_count",
+ "adoption_rate",
+ "risk_band",
+ "event_types",
+ "source_types",
+]
+LEDGER_SUMMARY_KEYS = [
+ "ledger_entry_count",
+ "accepted_count",
+ "pending_count",
+ "human_pending_count",
+ "external_pending_count",
+ "source_check_count",
+ "source_pass_count",
+ "source_blocked_count",
+ "ready_to_claim_world_class",
+ "decision",
+]
+LOCKSTEP_SECTIONS = [
+ "scorecard",
+ "capability_profile",
+ "principle_model",
+ "contract_boundary",
+ "quality_review",
+ "risk_governance",
+ "world_class_readiness",
+ "package_assets",
+ "iteration_roadmap",
+]
+
+
+def load_json(path: Path) -> tuple[dict[str, Any], str | None]:
+ if not path.exists():
+ return {}, "missing"
+ try:
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ except json.JSONDecodeError as exc:
+ return {}, f"invalid-json: {exc}"
+ if not isinstance(payload, dict):
+ return {}, "json-root-not-object"
+ return payload, None
+
+
+def load_text(path: Path) -> tuple[str, str | None]:
+ if not path.exists():
+ return "", "missing"
+ return path.read_text(encoding="utf-8"), None
+
+
+def rel_path(path: Path, root: Path) -> str:
+ try:
+ return str(path.resolve().relative_to(root.resolve()))
+ except ValueError:
+ return str(path.resolve())
+
+
+def nested(payload: dict[str, Any], path: list[str], default: Any = None) -> Any:
+ current: Any = payload
+ for key in path:
+ if not isinstance(current, dict) or key not in current:
+ return default
+ current = current[key]
+ return current
+
+
+def scanned_surface_paths(payload: dict[str, Any]) -> set[str]:
+ surfaces = payload.get("scanned_surfaces")
+ if not isinstance(surfaces, list):
+ return set()
+ paths: set[str] = set()
+ for item in surfaces:
+ if isinstance(item, dict) and isinstance(item.get("path"), str):
+ paths.add(item["path"])
+ elif isinstance(item, str):
+ paths.add(item)
+ return paths
+
+
+def as_int(value: Any) -> int | None:
+ if isinstance(value, bool):
+ return None
+ try:
+ return int(value)
+ except (TypeError, ValueError):
+ return None
+
+
+def add_check(
+ checks: list[dict[str, Any]],
+ *,
+ key: str,
+ label: str,
+ status: str,
+ expected: Any,
+ actual: Any,
+ paths: list[str],
+ detail: str,
+) -> None:
+ checks.append(
+ {
+ "key": key,
+ "label": label,
+ "status": status,
+ "expected": expected,
+ "actual": actual,
+ "paths": paths,
+ "detail": detail,
+ }
+ )
+
+
+def compare_values(
+ checks: list[dict[str, Any]],
+ *,
+ key: str,
+ label: str,
+ expected: Any,
+ actual: Any,
+ paths: list[str],
+ detail: str,
+) -> None:
+ add_check(
+ checks,
+ key=key,
+ label=label,
+ status="pass" if expected == actual else "fail",
+ expected=expected,
+ actual=actual,
+ paths=paths,
+ detail=detail,
+ )
+
+
+def compare_summary_keys(
+ checks: list[dict[str, Any]],
+ *,
+ key_prefix: str,
+ label: str,
+ source_summary: dict[str, Any],
+ embedded_summary: dict[str, Any],
+ keys: list[str],
+ paths: list[str],
+) -> None:
+ expected = {key: source_summary.get(key) for key in keys}
+ actual = {key: embedded_summary.get(key) for key in keys}
+ compare_values(
+ checks,
+ key=key_prefix,
+ label=label,
+ expected=expected,
+ actual=actual,
+ paths=paths,
+ detail="Selected summary fields must match exactly across generated reports.",
+ )
+
+
+def gate_by_key(review_studio: dict[str, Any], key: str) -> dict[str, Any]:
+ gates = review_studio.get("gates")
+ if not isinstance(gates, list):
+ return {}
+ for item in gates:
+ if isinstance(item, dict) and item.get("key") == key:
+ return item
+ return {}
+
+
+def beta_public_claim_split_values(
+ benchmark: dict[str, Any],
+ ledger: dict[str, Any],
+ review_studio: dict[str, Any],
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ benchmark_summary = nested(benchmark, ["summary"], {})
+ ledger_summary = nested(ledger, ["summary"], {})
+ expected_beta_ready = (
+ bool(benchmark_summary.get("reproducibility_ready"))
+ and bool(benchmark_summary.get("release_lock_ready"))
+ and bool(benchmark_summary.get("provider_evidence_complete"))
+ )
+ output_review_summary = nested(review_studio, ["data", "output_review_adjudication", "summary"], {})
+ review_pair_count = as_int(output_review_summary.get("pair_count")) if isinstance(output_review_summary, dict) else None
+ review_pending_count = as_int(output_review_summary.get("pending_count")) if isinstance(output_review_summary, dict) else None
+ expected_human_review_complete = (
+ review_pair_count is not None and review_pair_count > 0 and review_pending_count == 0
+ )
+ ledger_entries = ledger.get("entries", []) if isinstance(ledger, dict) else []
+ pending_ledger_keys = sorted(
+ str(entry.get("key", ""))
+ for entry in ledger_entries
+ if isinstance(entry, dict) and entry.get("status") == "pending" and str(entry.get("key", ""))
+ )
+ beta_boundary = {
+ "beta_test_ready": expected_beta_ready,
+ "public_claim_ready": ledger_summary.get("ready_to_claim_world_class"),
+ "human_review_complete": expected_human_review_complete,
+ "beta_release_ready": expected_beta_ready,
+ "beta_release_scope": "beta/public test release without superiority, fully-reviewed, or world-class claims",
+ "deferred_evidence_keys": pending_ledger_keys,
+ "deferred_human_review": "human-adjudication" in pending_ledger_keys,
+ }
+ beta_release = benchmark.get("beta_test_release", {}) if isinstance(benchmark, dict) else {}
+ deferred_keys = sorted(
+ str(item.get("key", ""))
+ for item in beta_release.get("allowed_deferred_evidence", [])
+ if isinstance(item, dict) and str(item.get("key", ""))
+ )
+ actual_beta_boundary = {
+ "beta_test_ready": benchmark_summary.get("beta_test_ready") if isinstance(benchmark_summary, dict) else None,
+ "public_claim_ready": benchmark_summary.get("public_claim_ready") if isinstance(benchmark_summary, dict) else None,
+ "human_review_complete": benchmark_summary.get("human_review_complete")
+ if isinstance(benchmark_summary, dict)
+ else None,
+ "beta_release_ready": beta_release.get("ready") if isinstance(beta_release, dict) else None,
+ "beta_release_scope": beta_release.get("scope") if isinstance(beta_release, dict) else None,
+ "deferred_evidence_keys": deferred_keys,
+ "deferred_human_review": "human-adjudication" in deferred_keys,
+ }
+ return beta_boundary, actual_beta_boundary
+
+
+def report_contract(payload: dict[str, Any]) -> dict[str, Any]:
+ contract = payload.get("report_contract")
+ return contract if isinstance(contract, dict) else {}
+
+
+def render_markdown(report: dict[str, Any]) -> str:
+ summary = report["summary"]
+ lines = [
+ "# Evidence Consistency",
+ "",
+ f"Generated at: `{report['generated_at']}`",
+ "",
+ "## Summary",
+ "",
+ f"- decision: `{summary['decision']}`",
+ f"- checks: `{summary['check_count']}`",
+ f"- pass: `{summary['pass_count']}`",
+ f"- warn: `{summary['warn_count']}`",
+ f"- fail: `{summary['fail_count']}`",
+ "",
+ "This gate compares generated evidence reports against each other. It does not create provider, human, native-client, or permission-enforcement evidence; it only catches drift between reports that already exist.",
+ "",
+ "## Checks",
+ "",
+ "| Check | Status | Detail | Paths |",
+ "| --- | --- | --- | --- |",
+ ]
+ for check in report["checks"]:
+ paths = ", ".join(f"`{path}`" for path in check["paths"])
+ lines.append(
+ "| "
+ + " | ".join(
+ [
+ check["label"].replace("|", "\\|"),
+ f"`{check['status']}`",
+ check["detail"].replace("|", "\\|"),
+ paths.replace("|", "\\|"),
+ ]
+ )
+ + " |"
+ )
+ failures = [check for check in report["checks"] if check["status"] == "fail"]
+ if failures:
+ lines.extend(["", "## Failures", ""])
+ for check in failures:
+ lines.extend(
+ [
+ f"### {check['label']}",
+ "",
+ f"- key: `{check['key']}`",
+ f"- expected: `{json.dumps(check['expected'], ensure_ascii=False, sort_keys=True)}`",
+ f"- actual: `{json.dumps(check['actual'], ensure_ascii=False, sort_keys=True)}`",
+ "",
+ ]
+ )
+ return "\n".join(lines).rstrip() + "\n"
diff --git a/scripts/evidence_consistency_phase_queue.py b/scripts/evidence_consistency_phase_queue.py
new file mode 100644
index 00000000..007cc454
--- /dev/null
+++ b/scripts/evidence_consistency_phase_queue.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+"""Compare world-class phase queues across generated reports."""
+
+from typing import Any
+
+from evidence_consistency_world_class import world_class_review_action_steps
+from world_class_phase_queue import build_phase_queue, summarize_phase_queue
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_evidence_consistency.py to prevent preflight and Review Studio phase-queue drift."
+
+
+def phase_queue_signature(queue: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ signature: list[dict[str, Any]] = []
+ for item in queue:
+ if not isinstance(item, dict):
+ continue
+ signature.append(
+ {
+ "phase": item.get("phase"),
+ "priority": item.get("priority"),
+ "status": item.get("status"),
+ "blocked_count": item.get("blocked_count"),
+ "row_count": item.get("row_count"),
+ "owners": sorted(str(owner) for owner in item.get("owners", [])),
+ "evidence_keys": sorted(str(key) for key in item.get("evidence_keys", [])),
+ "next_action_id": item.get("next_action_id"),
+ "verification_command": item.get("verification_command"),
+ "counts_as_completion": item.get("counts_as_completion"),
+ }
+ )
+ return signature
+
+
+def summary_signature(summary: dict[str, Any]) -> dict[str, Any]:
+ keys = [
+ "phase_queue_count",
+ "phase_queue_blocked_count",
+ "phase_queue_row_count",
+ "phase_queue_next_phase",
+ "phase_queue_next_action_id",
+ "phase_queue_next_command",
+ "phase_queue_counts_as_completion",
+ ]
+ return {key: summary.get(key) for key in keys}
+
+
+def keyed_preflight_items(world_class_preflight: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ items = world_class_preflight.get("items", []) if isinstance(world_class_preflight, dict) else []
+ keyed: dict[str, dict[str, Any]] = {}
+ for item in items:
+ if not isinstance(item, dict):
+ continue
+ key = str(item.get("evidence_key", "")).strip()
+ if key:
+ keyed[key] = item
+ return keyed
+
+
+def keyed_phase_queue_signatures_from_repair_rows(items: dict[str, dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
+ signatures: dict[str, list[dict[str, Any]]] = {}
+ for key, item in items.items():
+ rows = item.get("repair_checklist", []) if isinstance(item.get("repair_checklist", []), list) else []
+ signatures[key] = phase_queue_signature(build_phase_queue(rows))
+ return signatures
+
+
+def keyed_phase_queue_signatures_from_items(items: dict[str, dict[str, Any]], queue_key: str) -> dict[str, list[dict[str, Any]]]:
+ signatures: dict[str, list[dict[str, Any]]] = {}
+ for key, item in items.items():
+ queue = item.get(queue_key, []) if isinstance(item.get(queue_key, []), list) else []
+ signatures[key] = phase_queue_signature(queue)
+ return signatures
+
+
+def any_phase_queue_counts_as_completion(
+ *queues: list[dict[str, Any]],
+ item_maps: dict[str, dict[str, Any]] | None = None,
+) -> bool:
+ item_maps = item_maps or {}
+ for queue in queues:
+ for item in queue:
+ if isinstance(item, dict) and item.get("counts_as_completion") is True:
+ return True
+ for item in item_maps.values():
+ queue = item.get("phase_queue", []) if isinstance(item.get("phase_queue", []), list) else []
+ for row in queue:
+ if isinstance(row, dict) and row.get("counts_as_completion") is True:
+ return True
+ return False
+
+
+def build_phase_queue_consistency_check(
+ *,
+ world_class_preflight: dict[str, Any],
+ world_class_operator_runbook: dict[str, Any],
+ review_studio: dict[str, Any],
+ report_paths: dict[str, str],
+) -> dict[str, Any]:
+ repair_rows = (
+ world_class_preflight.get("repair_checklist", [])
+ if isinstance(world_class_preflight.get("repair_checklist", []), list)
+ else []
+ )
+ expected_queue = build_phase_queue(repair_rows)
+ expected_summary = summarize_phase_queue(expected_queue)
+ preflight_items = keyed_preflight_items(world_class_preflight)
+ operator_runbook_items = keyed_preflight_items(world_class_operator_runbook)
+ review_steps = world_class_review_action_steps(review_studio)
+ expected = {
+ "summary": summary_signature(expected_summary),
+ "top_level_phase_queue": phase_queue_signature(expected_queue),
+ "item_phase_queues": keyed_phase_queue_signatures_from_repair_rows(preflight_items),
+ "phase_queue_counts_as_completion": False,
+ }
+ actual = {
+ "summary": summary_signature(
+ world_class_preflight.get("summary", {})
+ if isinstance(world_class_preflight.get("summary", {}), dict)
+ else {}
+ ),
+ "top_level_phase_queue": phase_queue_signature(
+ world_class_preflight.get("phase_queue", [])
+ if isinstance(world_class_preflight.get("phase_queue", []), list)
+ else []
+ ),
+ "operator_runbook_summary": summary_signature(
+ world_class_operator_runbook.get("summary", {})
+ if isinstance(world_class_operator_runbook.get("summary", {}), dict)
+ else {}
+ ),
+ "operator_runbook_top_level_phase_queue": phase_queue_signature(
+ world_class_operator_runbook.get("phase_queue", [])
+ if isinstance(world_class_operator_runbook.get("phase_queue", []), list)
+ else []
+ ),
+ "item_phase_queues": keyed_phase_queue_signatures_from_items(preflight_items, "phase_queue"),
+ "operator_runbook_phase_queues": keyed_phase_queue_signatures_from_items(
+ operator_runbook_items,
+ "phase_queue",
+ ),
+ "review_studio_phase_queues": keyed_phase_queue_signatures_from_items(review_steps, "phase_queue"),
+ "phase_queue_counts_as_completion": any_phase_queue_counts_as_completion(
+ world_class_preflight.get("phase_queue", [])
+ if isinstance(world_class_preflight.get("phase_queue", []), list)
+ else [],
+ world_class_operator_runbook.get("phase_queue", [])
+ if isinstance(world_class_operator_runbook.get("phase_queue", []), list)
+ else [],
+ item_maps={**preflight_items, **operator_runbook_items, **review_steps},
+ ),
+ }
+ expected["operator_runbook_summary"] = expected["summary"]
+ expected["operator_runbook_top_level_phase_queue"] = expected["top_level_phase_queue"]
+ expected["operator_runbook_phase_queues"] = expected["item_phase_queues"]
+ expected["review_studio_phase_queues"] = expected["item_phase_queues"]
+ return {
+ "key": "world-class-phase-queue-consistency",
+ "label": "World-class phase queues mirror repair rows",
+ "status": "pass" if expected == actual else "fail",
+ "expected": expected,
+ "actual": actual,
+ "paths": [
+ report_paths["world_class_preflight"],
+ report_paths["world_class_operator_runbook"],
+ report_paths["review_studio"],
+ ],
+ "detail": (
+ "Phase queues must be derived from repair rows in preflight and mirrored into the operator runbook "
+ "and Review Studio without counting queue guidance as completion evidence."
+ ),
+ }
diff --git a/scripts/evidence_consistency_release.py b/scripts/evidence_consistency_release.py
new file mode 100644
index 00000000..72e16c42
--- /dev/null
+++ b/scripts/evidence_consistency_release.py
@@ -0,0 +1,132 @@
+from pathlib import Path
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_evidence_consistency.py to verify release evidence refresh instructions."
+
+SOURCE_REFRESH_HEADER = "After source changes that affect scripts"
+CLEAN_LOCK_HEADER = "For final release evidence"
+CLEAN_LOCK_END = "If `reports/benchmark_reproducibility.json`"
+
+SOURCE_REFRESH_REPORT_COMMANDS = [
+ 'python3 scripts/run_output_execution.py --runner-command \'["python3","scripts/local_output_eval_runner.py"]\'',
+ 'python3 scripts/compile_skill.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/cross_packager.py . --platform openai --platform claude --platform generic --platform vscode --expectations evals/packaging_expectations.json --output-dir dist --zip",
+ 'python3 scripts/simulate_install.py . --package-dir dist --install-root dist/install-simulation --output-json reports/install_simulation.json --output-md reports/install_simulation.md --generated-at "$GENERATED_AT"',
+ "python3 scripts/trust_check.py . --output-json reports/security_trust_report.json --output-md reports/security_trust_report.md",
+ 'python3 scripts/registry_audit.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/verify_package.py . --package-dir dist --expectations evals/packaging_expectations.json --registry-json reports/registry_audit.json --output-json reports/package_verification.json --output-md reports/package_verification.md --require-zip --generated-at "$GENERATED_AT"',
+ 'python3 scripts/upgrade_check.py . --previous-package-json registry/examples/yao-meta-skill-1.0.0.json --current-package-json reports/registry_audit.json --output-json reports/upgrade_check.json --output-md reports/upgrade_check.md --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_adoption_drift_report.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_architecture_maintainability.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/python_compat_check.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/probe_runtime_permissions.py . --package-dir dist",
+ 'python3 scripts/render_review_waivers.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/render_review_annotations.py .",
+ 'python3 scripts/build_skill_atlas.py --workspace-root . --output-dir skill_atlas --report-html reports/skill_atlas.html --report-json reports/skill_atlas.json --today "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_evidence_plan.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_evidence_ledger.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_evidence_intake.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_submission_review.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_operator_runbook.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_world_class_claim_guard.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_daily_skillops_report.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_weekly_curator_report.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_skill_os2_audit.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_skill_os2_coverage.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_context_reports.py --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_benchmark_reproducibility.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/render_skill_overview.py .",
+ "python3 scripts/render_skill_interpretation.py .",
+ "python3 scripts/render_review_viewer.py .",
+ 'python3 scripts/render_world_class_preflight.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json",
+ 'python3 scripts/render_evidence_consistency.py . --generated-at "$GENERATED_AT"',
+]
+CLEAN_LOCK_REPORT_COMMANDS = [
+ 'python3 scripts/render_context_reports.py --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_benchmark_reproducibility.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_daily_skillops_report.py . --generated-at "$GENERATED_AT"',
+ 'python3 scripts/render_weekly_curator_report.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/render_skill_overview.py .",
+ "python3 scripts/render_skill_interpretation.py .",
+ "python3 scripts/render_review_viewer.py .",
+ 'python3 scripts/render_world_class_preflight.py . --generated-at "$GENERATED_AT"',
+ "python3 scripts/render_review_studio.py . --output-html reports/review-studio.html --output-json reports/review-studio.json",
+ 'python3 scripts/render_evidence_consistency.py . --generated-at "$GENERATED_AT"',
+]
+
+
+def section_between(text: str, start: str, end: str) -> str:
+ if start not in text:
+ return ""
+ section = text.split(start, 1)[1]
+ if end in section:
+ section = section.split(end, 1)[0]
+ return section
+
+
+def command_presence(section: str, commands: list[str]) -> dict[str, bool]:
+ return {command: command in section for command in commands}
+
+
+def build_release_evidence_flow_check(skill_dir: Path) -> dict[str, Any]:
+ agents_path = skill_dir / "AGENTS.md"
+ agents_text = agents_path.read_text(encoding="utf-8") if agents_path.exists() else ""
+ source_refresh = section_between(agents_text, SOURCE_REFRESH_HEADER, CLEAN_LOCK_HEADER)
+ clean_lock = section_between(agents_text, CLEAN_LOCK_HEADER, CLEAN_LOCK_END)
+ expected = {
+ "AGENTS.md": True,
+ "source_refresh_section": True,
+ "clean_lock_section": True,
+ "source_refresh_commands": {command: True for command in SOURCE_REFRESH_REPORT_COMMANDS},
+ "clean_lock_commands": {command: True for command in CLEAN_LOCK_REPORT_COMMANDS},
+ }
+ actual = {
+ "AGENTS.md": agents_path.exists(),
+ "source_refresh_section": bool(source_refresh),
+ "clean_lock_section": bool(clean_lock),
+ "source_refresh_commands": command_presence(source_refresh, SOURCE_REFRESH_REPORT_COMMANDS),
+ "clean_lock_commands": command_presence(clean_lock, CLEAN_LOCK_REPORT_COMMANDS),
+ }
+ return {
+ "key": "release-evidence-flow-covers-first-class-reports",
+ "label": "Release evidence flow covers first-class reports",
+ "status": "pass" if expected == actual else "fail",
+ "expected": expected,
+ "actual": actual,
+ "paths": [
+ "AGENTS.md",
+ "reports/output_execution_runs.json",
+ "reports/install_simulation.json",
+ "reports/security_trust_report.json",
+ "reports/registry_audit.json",
+ "reports/package_verification.json",
+ "reports/upgrade_check.json",
+ "reports/adoption_drift_report.json",
+ "reports/architecture_maintainability.json",
+ "reports/python_compatibility.json",
+ "reports/runtime_permission_probes.json",
+ "reports/review_waivers.json",
+ "reports/review_annotations.json",
+ "reports/skill_atlas.json",
+ "reports/skill_os2_audit.json",
+ "reports/skill_os2_coverage.json",
+ "reports/context_budget.json",
+ "reports/context_budget_summary.json",
+ "reports/benchmark_reproducibility.json",
+ "reports/skill-overview.json",
+ "reports/skill-interpretation.json",
+ "reports/review-viewer.json",
+ "reports/world_class_evidence_preflight.json",
+ "reports/skillops/daily",
+ "reports/skillops/weekly",
+ "reports/review-studio.json",
+ "reports/evidence_consistency.json",
+ ],
+ "detail": (
+ "Release refresh and clean-lock instructions must regenerate every first-class report "
+ "before evidence consistency can be trusted."
+ ),
+ }
diff --git a/scripts/evidence_consistency_skill_os2_review.py b/scripts/evidence_consistency_skill_os2_review.py
new file mode 100644
index 00000000..d483a217
--- /dev/null
+++ b/scripts/evidence_consistency_skill_os2_review.py
@@ -0,0 +1,83 @@
+import ast
+from pathlib import Path
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = (
+ "Imported by render_evidence_consistency.py to keep Skill OS 2.0 review summary drift checks "
+ "out of the main consistency renderer."
+)
+
+
+def ci_default_target_count(path: Path) -> int | None:
+ if not path.exists():
+ return None
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ for node in tree.body:
+ if not isinstance(node, ast.Assign):
+ continue
+ if not any(isinstance(target, ast.Name) and target.id == "DEFAULT_TARGETS" for target in node.targets):
+ continue
+ if isinstance(node.value, (ast.List, ast.Tuple)):
+ return len(node.value.elts)
+ return None
+
+
+def build_skill_os2_review_current_evidence_check(
+ *,
+ skill_dir: Path,
+ skill_os2_review: str,
+ studio_summary: dict[str, Any],
+ trust_summary: dict[str, Any],
+ package_summary: dict[str, Any],
+ install_summary: dict[str, Any],
+ benchmark_summary: dict[str, Any],
+ context_stats: dict[str, Any],
+ required_text_reports: dict[str, str],
+ required_reports: dict[str, str],
+) -> dict[str, Any]:
+ ci_target_count = ci_default_target_count(skill_dir / "scripts" / "ci_test.py")
+ expected_review_snippets = [
+ f"score `{studio_summary.get('world_class_score')}`",
+ f"`{studio_summary.get('gate_count')}` gates",
+ f"`{studio_summary.get('warning_count')}` warnings",
+ f"`{trust_summary.get('internal_module_count')}` declared internal modules",
+ (
+ f"`{trust_summary.get('help_smoke_checked_count')} / {trust_summary.get('help_smoke_checked_count')}` "
+ f"CLI help smoke checks passing across `{trust_summary.get('script_count')}` scripts"
+ ),
+ f"`{package_summary.get('archive_entry_count')}` zip entries",
+ f"archive with `{package_summary.get('archive_entry_count')}` entries",
+ f"`{install_summary.get('installer_permission_enforced_count')}` installer permission checks enforced",
+ f"`{install_summary.get('installer_permission_failure_count')}` permission failures",
+ f"`{benchmark_summary.get('required_artifact_count')}` required artifacts",
+ f"`{benchmark_summary.get('command_count')}` reproduction commands",
+ (
+ f"initial load `{context_stats.get('estimated_initial_load_tokens')}/"
+ f"{context_stats.get('context_budget_limit')}`"
+ ),
+ f"target count is `{ci_target_count}`",
+ ]
+ missing_review_snippets = [snippet for snippet in expected_review_snippets if snippet not in skill_os2_review]
+ return {
+ "key": "skill-os-2-review-current-evidence",
+ "label": "Skill OS 2.0 review summary mirrors current evidence",
+ "status": "pass" if not missing_review_snippets else "fail",
+ "expected": expected_review_snippets,
+ "actual": "all present" if not missing_review_snippets else {"missing": missing_review_snippets},
+ "paths": [
+ required_text_reports["skill_os2_review"],
+ required_reports["review_studio"],
+ required_reports["package_verification"],
+ required_reports["install_simulation"],
+ required_reports["trust"],
+ required_reports["context_budget"],
+ required_reports["benchmark"],
+ "scripts/ci_test.py",
+ ],
+ "detail": (
+ "Manual 2.0 review summaries must not drift from generated gate, package, trust, "
+ "context, benchmark, or CI evidence."
+ ),
+ }
diff --git a/scripts/evidence_consistency_world_class.py b/scripts/evidence_consistency_world_class.py
new file mode 100644
index 00000000..10a6f969
--- /dev/null
+++ b/scripts/evidence_consistency_world_class.py
@@ -0,0 +1,222 @@
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_evidence_consistency.py to isolate world-class evidence workflow consistency checks."
+
+
+def keyed_items(payload: dict[str, Any], collection_key: str) -> dict[str, dict[str, Any]]:
+ collection = payload.get(collection_key)
+ if not isinstance(collection, list):
+ return {}
+ keyed: dict[str, dict[str, Any]] = {}
+ for item in collection:
+ if not isinstance(item, dict):
+ continue
+ key = item.get("key") or item.get("evidence_key")
+ if isinstance(key, str):
+ keyed[key] = item
+ return keyed
+
+
+def command_key_set(item: dict[str, Any]) -> set[str]:
+ commands = item.get("commands")
+ if isinstance(commands, dict):
+ return {key for key, value in commands.items() if isinstance(key, str) and value}
+ if isinstance(commands, list):
+ keys: set[str] = set()
+ for command in commands:
+ if isinstance(command, dict) and isinstance(command.get("key"), str) and command.get("command"):
+ keys.add(command["key"])
+ return keys
+ return set()
+
+
+def world_class_review_action_steps(review_studio: dict[str, Any]) -> dict[str, dict[str, Any]]:
+ actions = review_studio.get("review_actions")
+ if not isinstance(actions, list):
+ return {}
+ for action in actions:
+ if not isinstance(action, dict) or action.get("gate_key") != "world-class-evidence":
+ continue
+ steps = action.get("evidence_steps")
+ if not isinstance(steps, list):
+ return {}
+ keyed: dict[str, dict[str, Any]] = {}
+ for step in steps:
+ if isinstance(step, dict) and isinstance(step.get("key"), str):
+ keyed[step["key"]] = step
+ return keyed
+ return {}
+
+
+def command_groups_present(command_keys: set[str]) -> dict[str, bool]:
+ return {
+ "prepare_submission": "prepare_submission" in command_keys,
+ "validate_intake": "validate_intake" in command_keys,
+ "submission_review": bool({"submission_review", "review_queue"} & command_keys),
+ "refresh_ledger": "refresh_ledger" in command_keys,
+ "guard_claim": "guard_claim" in command_keys,
+ }
+
+
+def has_next_action(item: dict[str, Any]) -> bool:
+ for key in ["next_action", "audit_next_action"]:
+ value = item.get(key)
+ if isinstance(value, str) and value.strip():
+ return True
+ return False
+
+
+def command_expectations(keys: list[str]) -> dict[str, dict[str, dict[str, bool]]]:
+ expected_group = {
+ "prepare_submission": True,
+ "validate_intake": True,
+ "submission_review": True,
+ "refresh_ledger": True,
+ "guard_claim": True,
+ }
+ return {
+ key: {
+ "intake": dict(expected_group),
+ "operator_runbook": dict(expected_group),
+ "review_studio": dict(expected_group),
+ }
+ for key in keys
+ }
+
+
+def build_world_class_workflow_check(
+ *,
+ ledger: dict[str, Any],
+ world_class_plan: dict[str, Any],
+ world_class_intake: dict[str, Any],
+ world_class_submission_review: dict[str, Any],
+ world_class_operator_runbook: dict[str, Any],
+ review_studio: dict[str, Any],
+ report_paths: dict[str, str],
+) -> dict[str, Any]:
+ ledger_summary = ledger.get("summary") if isinstance(ledger.get("summary"), dict) else {}
+ plan_summary = world_class_plan.get("summary") if isinstance(world_class_plan.get("summary"), dict) else {}
+ intake_summary = world_class_intake.get("summary") if isinstance(world_class_intake.get("summary"), dict) else {}
+ submission_review_summary = (
+ world_class_submission_review.get("summary")
+ if isinstance(world_class_submission_review.get("summary"), dict)
+ else {}
+ )
+ operator_runbook_summary = (
+ world_class_operator_runbook.get("summary")
+ if isinstance(world_class_operator_runbook.get("summary"), dict)
+ else {}
+ )
+
+ ledger_items = keyed_items(ledger, "entries")
+ plan_tasks = keyed_items(world_class_plan, "tasks")
+ intake_checklist = keyed_items(world_class_intake, "operator_checklist")
+ submission_review_items = keyed_items(world_class_submission_review, "items")
+ operator_runbook_items = keyed_items(world_class_operator_runbook, "items")
+ coordination_plan = (
+ world_class_operator_runbook.get("coordination_plan")
+ if isinstance(world_class_operator_runbook.get("coordination_plan"), list)
+ else []
+ )
+ release_gate = (
+ world_class_operator_runbook.get("release_gate")
+ if isinstance(world_class_operator_runbook.get("release_gate"), dict)
+ else {}
+ )
+ review_action_steps = world_class_review_action_steps(review_studio)
+ pending_keys = sorted(key for key, item in ledger_items.items() if item.get("status") != "accepted")
+ operator_coordination_keys = sorted(
+ key
+ for key in {str(step.get("evidence_key", "")) for step in coordination_plan if isinstance(step, dict)}
+ if key
+ )
+
+ actual_command_groups = {
+ key: {
+ "intake": command_groups_present(command_key_set(intake_checklist.get(key, {}))),
+ "operator_runbook": command_groups_present(command_key_set(operator_runbook_items.get(key, {}))),
+ "review_studio": command_groups_present(command_key_set(review_action_steps.get(key, {}))),
+ }
+ for key in pending_keys
+ }
+ expected = {
+ "keys": pending_keys,
+ "pending_count": ledger_summary.get("pending_count"),
+ "human_pending_count": ledger_summary.get("human_pending_count"),
+ "external_pending_count": ledger_summary.get("external_pending_count"),
+ "source_check_count": ledger_summary.get("source_check_count"),
+ "source_pass_count": ledger_summary.get("source_pass_count"),
+ "source_blocked_count": ledger_summary.get("source_blocked_count"),
+ "plan_keys": pending_keys,
+ "intake_keys": pending_keys,
+ "submission_review_keys": pending_keys,
+ "operator_runbook_keys": pending_keys,
+ "operator_coordination_keys": pending_keys,
+ "operator_coordination_counts_as_completion": False,
+ "operator_release_gate_ready": ledger_summary.get("ready_to_claim_world_class") is True,
+ "operator_release_gate_counts_as_completion": False,
+ "review_studio_keys": pending_keys,
+ "intake_ready_to_claim_world_class": False,
+ "submission_review_ready_to_claim_world_class": False,
+ "submission_review_counts_as_completion": False,
+ "operator_runbook_ready_to_claim_world_class": False,
+ "operator_runbook_counts_as_completion": False,
+ "next_actions_present": {key: True for key in pending_keys},
+ "commands": command_expectations(pending_keys),
+ }
+ actual = {
+ "keys": pending_keys,
+ "pending_count": plan_summary.get("task_count"),
+ "human_pending_count": plan_summary.get("human_task_count"),
+ "external_pending_count": plan_summary.get("external_task_count"),
+ "source_check_count": submission_review_summary.get("source_check_count"),
+ "source_pass_count": submission_review_summary.get("source_pass_count"),
+ "source_blocked_count": submission_review_summary.get("source_blocked_count"),
+ "plan_keys": sorted(plan_tasks),
+ "intake_keys": sorted(intake_checklist),
+ "submission_review_keys": sorted(submission_review_items),
+ "operator_runbook_keys": sorted(operator_runbook_items),
+ "operator_coordination_keys": operator_coordination_keys,
+ "operator_coordination_counts_as_completion": operator_runbook_summary.get(
+ "coordination_counts_as_completion"
+ ),
+ "operator_release_gate_ready": release_gate.get("ready"),
+ "operator_release_gate_counts_as_completion": release_gate.get("counts_as_completion"),
+ "review_studio_keys": sorted(review_action_steps),
+ "intake_ready_to_claim_world_class": intake_summary.get("ready_to_claim_world_class"),
+ "submission_review_ready_to_claim_world_class": submission_review_summary.get("ready_to_claim_world_class"),
+ "submission_review_counts_as_completion": submission_review_summary.get(
+ "review_counts_submission_as_completion"
+ ),
+ "operator_runbook_ready_to_claim_world_class": operator_runbook_summary.get("ready_to_claim_world_class"),
+ "operator_runbook_counts_as_completion": operator_runbook_summary.get("runbook_counts_as_completion"),
+ "next_actions_present": {
+ key: all(
+ has_next_action(collection.get(key, {}))
+ for collection in [plan_tasks, intake_checklist, submission_review_items, review_action_steps]
+ )
+ for key in pending_keys
+ },
+ "commands": actual_command_groups,
+ }
+ return {
+ "key": "world-class-evidence-workflow-coverage",
+ "label": "World-class evidence workflows cover every pending ledger entry",
+ "status": "pass" if expected == actual else "fail",
+ "expected": expected,
+ "actual": actual,
+ "paths": [
+ report_paths["world_class_ledger"],
+ report_paths["world_class_plan"],
+ report_paths["world_class_intake"],
+ report_paths["world_class_submission_review"],
+ report_paths["world_class_operator_runbook"],
+ report_paths["review_studio"],
+ ],
+ "detail": (
+ "Every pending world-class evidence key must have matching plan, intake, submission review, "
+ "operator runbook, and Review Studio actions without counting planned work as completion."
+ ),
+ }
diff --git a/scripts/export_skill_ir.py b/scripts/export_skill_ir.py
index 08386a68..f02b913c 100644
--- a/scripts/export_skill_ir.py
+++ b/scripts/export_skill_ir.py
@@ -51,6 +51,8 @@ KEY_REPORTS = [
"reports/security_trust_report.md",
"reports/runtime_permission_probes.json",
"reports/runtime_permission_probes.md",
+ "reports/telemetry_hook_recipes.json",
+ "reports/telemetry_hook_recipes.md",
"reports/skill_atlas.json",
"reports/skill_atlas.html",
"reports/skill-os-2-review.md",
@@ -157,7 +159,7 @@ def report_list(skill_dir: Path) -> list[str]:
return [rel for rel in KEY_REPORTS if (skill_dir / rel).exists()]
-def file_list(skill_dir: Path, folder: str, suffixes: set[str] | None = None, limit: int = 80) -> list[str]:
+def file_list(skill_dir: Path, folder: str, suffixes: set[str] | None = None, limit: int | None = None) -> list[str]:
target = skill_dir / folder
if not target.exists():
return []
@@ -168,7 +170,7 @@ def file_list(skill_dir: Path, folder: str, suffixes: set[str] | None = None, li
if suffixes is not None and path.suffix not in suffixes:
continue
paths.append(str(path.relative_to(skill_dir)))
- if len(paths) >= limit:
+ if limit is not None and len(paths) >= limit:
break
return paths
diff --git a/scripts/html_rendering.py b/scripts/html_rendering.py
new file mode 100644
index 00000000..afb7f750
--- /dev/null
+++ b/scripts/html_rendering.py
@@ -0,0 +1,14 @@
+#!/usr/bin/env python3
+"""Shared HTML rendering helpers for static report generators."""
+
+import html
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Used by report renderers to escape HTML while preserving meaningful falsey values."
+
+
+def html_text(value: Any) -> str:
+ """Escape text for HTML without dropping 0 or False values."""
+ return html.escape("" if value is None else str(value), quote=True)
diff --git a/scripts/import_output_review_decisions.py b/scripts/import_output_review_decisions.py
new file mode 100644
index 00000000..2a4a8336
--- /dev/null
+++ b/scripts/import_output_review_decisions.py
@@ -0,0 +1,329 @@
+#!/usr/bin/env python3
+import argparse
+import csv
+import json
+from datetime import date
+from pathlib import Path
+from typing import Any
+
+from adjudicate_output_review import (
+ DEFAULT_ANSWER_KEY,
+ DEFAULT_BLIND_PACK,
+ DEFAULT_DECISIONS,
+ DEFAULT_OUTPUT_JSON,
+ DEFAULT_OUTPUT_MD,
+ adjudicate_output_review,
+ build_decision_template,
+ default_reviewer_attestation,
+ normalize_variant,
+ pair_index,
+ review_integrity,
+)
+from output_review_privacy import forbidden_decision_field_paths
+
+
+ROOT = Path(__file__).resolve().parent.parent
+SCRIPT_INTERFACE = "cli"
+SCRIPT_INTERFACE_REASON = "Imports human blind A/B reviewer decisions into the canonical output-review decision file."
+
+
+def display_path(path: Path) -> str:
+ try:
+ return str(path.resolve().relative_to(ROOT.resolve()))
+ except ValueError:
+ return str(path.resolve())
+
+
+def load_json(path: Path) -> tuple[Any, list[str]]:
+ try:
+ return json.loads(path.read_text(encoding="utf-8")), []
+ except json.JSONDecodeError as exc:
+ return None, [f"Invalid JSON file {display_path(path)}: {exc}"]
+
+
+def detect_format(path: Path, requested: str) -> str:
+ if requested != "auto":
+ return requested
+ suffix = path.suffix.lower()
+ if suffix == ".jsonl":
+ return "jsonl"
+ if suffix == ".csv":
+ return "csv"
+ return "json"
+
+
+def read_decision_source(path: Path, source_format: str) -> tuple[dict[str, Any], list[dict[str, Any]], list[str]]:
+ if not path.exists():
+ return {}, [], [f"Missing decision source: {display_path(path)}"]
+ if source_format == "json":
+ payload, failures = load_json(path)
+ if failures:
+ return {}, [], failures
+ if isinstance(payload, list):
+ return {}, payload, []
+ if not isinstance(payload, dict):
+ return {}, [], ["JSON decision source must be an object or list"]
+ decisions = payload.get("decisions", [])
+ if not isinstance(decisions, list):
+ return payload, [], ["decisions must be a list"]
+ return payload, decisions, []
+ if source_format == "jsonl":
+ decisions: list[dict[str, Any]] = []
+ failures: list[str] = []
+ for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):
+ if not line.strip():
+ continue
+ try:
+ item = json.loads(line)
+ except json.JSONDecodeError as exc:
+ failures.append(f"JSONL line {line_number} is invalid: {exc}")
+ continue
+ if not isinstance(item, dict):
+ failures.append(f"JSONL line {line_number} must be an object")
+ continue
+ decisions.append(item)
+ return {}, decisions, failures
+ if source_format == "csv":
+ with path.open("r", encoding="utf-8", newline="") as handle:
+ return {}, [dict(row) for row in csv.DictReader(handle)], []
+ return {}, [], [f"Unsupported decision source format: {source_format}"]
+
+
+def known_case_ids(blind_pack_path: Path) -> tuple[set[str], list[str]]:
+ payload, failures = load_json(blind_pack_path)
+ if failures:
+ return set(), failures
+ if not isinstance(payload, dict):
+ return set(), [f"Blind pack root must be an object: {display_path(blind_pack_path)}"]
+ pairs = pair_index(payload)
+ if not pairs:
+ return set(), [f"Blind pack has no pairs: {display_path(blind_pack_path)}"]
+ return set(pairs), []
+
+
+def parse_confidence(value: Any, case_id: str) -> tuple[float | None, str | None]:
+ if value is None or str(value).strip() == "":
+ return None, None
+ try:
+ parsed = float(value)
+ except (TypeError, ValueError):
+ return None, f"{case_id}: confidence must be numeric"
+ if parsed < 0 or parsed > 1:
+ return None, f"{case_id}: confidence must be between 0 and 1"
+ return round(parsed, 3), None
+
+
+def normalize_decisions(
+ source_items: list[dict[str, Any]],
+ case_ids: set[str],
+) -> tuple[list[dict[str, Any]], list[str]]:
+ failures: list[str] = []
+ normalized: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for index, item in enumerate(source_items, start=1):
+ if not isinstance(item, dict):
+ failures.append(f"decision #{index} must be an object")
+ continue
+ blocked_fields = forbidden_decision_field_paths(item, f"decision #{index}")
+ if blocked_fields:
+ failures.append(f"decision #{index} contains forbidden raw or answer-key fields: {', '.join(blocked_fields)}")
+ case_id = str(item.get("case_id", "")).strip()
+ if not case_id:
+ failures.append(f"decision #{index} is missing case_id")
+ continue
+ if case_id not in case_ids:
+ failures.append(f"decision #{index} references unknown case_id: {case_id}")
+ if case_id in seen:
+ failures.append(f"duplicate decision for case_id: {case_id}")
+ seen.add(case_id)
+ winner = normalize_variant(item.get("winner_variant", item.get("winner", item.get("reviewer_winner_variant", ""))))
+ if winner and winner not in {"A", "B"}:
+ failures.append(f"{case_id}: winner_variant must be A or B")
+ confidence, confidence_failure = parse_confidence(item.get("confidence"), case_id)
+ if confidence_failure:
+ failures.append(confidence_failure)
+ reason = str(item.get("reason", "")).strip()
+ if winner in {"A", "B"} and not reason:
+ failures.append(f"{case_id}: reason is required for imported human decisions")
+ normalized.append(
+ {
+ "case_id": case_id,
+ "winner_variant": winner,
+ "confidence": confidence,
+ "reason": reason,
+ }
+ )
+ return normalized, failures
+
+
+def canonical_payload(
+ blind_pack_path: Path,
+ source_path: Path,
+ source_format: str,
+ reviewer: str,
+ reviewed_at: str,
+ decisions: list[dict[str, Any]],
+ reviewer_attestation: dict[str, Any],
+) -> dict[str, Any]:
+ blind_pack = json.loads(blind_pack_path.read_text(encoding="utf-8"))
+ template = build_decision_template(blind_pack)
+ completed = sum(1 for item in decisions if item.get("winner_variant") in {"A", "B"})
+ return {
+ "schema_version": "1.0",
+ "reviewer": reviewer,
+ "reviewed_at": reviewed_at,
+ "review_integrity": review_integrity(blind_pack),
+ "reviewer_attestation": reviewer_attestation,
+ "decision_contract": template["decision_contract"],
+ "import_contract": {
+ "schema_version": "1.0",
+ "source_path": display_path(source_path),
+ "source_format": source_format,
+ "raw_content_allowed": False,
+ "answer_key_fields_allowed": False,
+ "answer_key_opened_by_importer": False,
+ "all_or_nothing_write": True,
+ "decision_count": len(decisions),
+ "completed_decision_count": completed,
+ "pending_decision_count": len(decisions) - completed,
+ "blind_review_attestation_supported": True,
+ "blind_review_attested": reviewer_attestation.get("blind_review_completed_before_answer_key") is True
+ and reviewer_attestation.get("answer_key_not_opened_before_decisions") is True,
+ },
+ "decisions": decisions,
+ }
+
+
+def normalize_reviewer_attestation(source_meta: dict[str, Any], blind_review_attested: bool) -> dict[str, Any]:
+ attestation = default_reviewer_attestation()
+ source_attestation = source_meta.get("reviewer_attestation", {})
+ if isinstance(source_attestation, dict):
+ for key in attestation:
+ if source_attestation.get(key) is True:
+ attestation[key] = True
+ elif source_attestation.get(key) is False:
+ attestation[key] = False
+ if blind_review_attested:
+ attestation["blind_review_completed_before_answer_key"] = True
+ attestation["answer_key_not_opened_before_decisions"] = True
+ return attestation
+
+
+def import_output_review_decisions(
+ source_path: Path,
+ blind_pack_path: Path,
+ output_json: Path,
+ source_format: str = "auto",
+ reviewer: str = "",
+ reviewed_at: str = "",
+ run_adjudication: bool = False,
+ answer_key_path: Path = DEFAULT_ANSWER_KEY,
+ adjudication_json: Path = DEFAULT_OUTPUT_JSON,
+ adjudication_md: Path = DEFAULT_OUTPUT_MD,
+ blind_review_attested: bool = False,
+) -> dict[str, Any]:
+ detected_format = detect_format(source_path, source_format)
+ source_meta, source_items, failures = read_decision_source(source_path, detected_format)
+ reviewer = reviewer or str(source_meta.get("reviewer", "")).strip()
+ reviewed_at = reviewed_at or str(source_meta.get("reviewed_at", "")).strip()
+ if not reviewer:
+ failures.append("reviewer is required for imported human decisions")
+ if not reviewed_at:
+ failures.append("reviewed_at is required for imported human decisions")
+ reviewer_attestation = normalize_reviewer_attestation(source_meta, blind_review_attested)
+ case_ids, case_failures = known_case_ids(blind_pack_path)
+ failures.extend(case_failures)
+ decisions, decision_failures = normalize_decisions(source_items, case_ids)
+ failures.extend(decision_failures)
+ payload: dict[str, Any] = {
+ "schema_version": "1.0",
+ "ok": not failures,
+ "summary": {
+ "decision_count": len(decisions),
+ "completed_decision_count": sum(1 for item in decisions if item.get("winner_variant") in {"A", "B"}),
+ "pending_decision_count": sum(1 for item in decisions if not item.get("winner_variant")),
+ "failure_count": len(failures),
+ "canonical_written": False,
+ "adjudication_run": False,
+ },
+ "artifacts": {
+ "source": display_path(source_path),
+ "blind_pack": display_path(blind_pack_path),
+ "decisions": display_path(output_json),
+ "adjudication_json": display_path(adjudication_json),
+ "adjudication_markdown": display_path(adjudication_md),
+ },
+ "failures": failures,
+ }
+ if failures:
+ return payload
+ canonical = canonical_payload(
+ blind_pack_path,
+ source_path,
+ detected_format,
+ reviewer,
+ reviewed_at,
+ decisions,
+ reviewer_attestation,
+ )
+ output_json.parent.mkdir(parents=True, exist_ok=True)
+ output_json.write_text(json.dumps(canonical, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ payload["summary"]["canonical_written"] = True
+ payload["canonical"] = canonical
+ if run_adjudication:
+ adjudication = adjudicate_output_review(
+ blind_pack_path=blind_pack_path,
+ answer_key_path=answer_key_path,
+ decisions_path=output_json,
+ output_json=adjudication_json,
+ output_md=adjudication_md,
+ )
+ payload["summary"]["adjudication_run"] = True
+ payload["summary"]["adjudication_ok"] = adjudication["ok"]
+ payload["summary"]["adjudication_pending_count"] = adjudication["summary"]["pending_count"]
+ payload["summary"]["adjudication_invalid_decision_count"] = adjudication["summary"]["invalid_decision_count"]
+ payload["adjudication_summary"] = adjudication["summary"]
+ if not adjudication["ok"]:
+ payload["ok"] = False
+ payload["failures"] = list(adjudication.get("failures", []))
+ payload["summary"]["failure_count"] = len(payload["failures"])
+ return payload
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Import blind A/B reviewer decisions into the canonical output-review decision file.")
+ parser.add_argument("--input", required=True, help="Reviewer decision source in JSON, JSONL, or CSV format.")
+ parser.add_argument("--format", choices=["auto", "json", "jsonl", "csv"], default="auto")
+ parser.add_argument("--blind-pack", default=str(DEFAULT_BLIND_PACK))
+ parser.add_argument("--output-json", default=str(DEFAULT_DECISIONS))
+ parser.add_argument("--reviewer")
+ parser.add_argument("--reviewed-at", default=date.today().isoformat())
+ parser.add_argument(
+ "--blind-review-attested",
+ action="store_true",
+ help="Attest that reviewer choices were completed before the answer key was opened.",
+ )
+ parser.add_argument("--run-adjudication", action="store_true")
+ parser.add_argument("--answer-key", default=str(DEFAULT_ANSWER_KEY))
+ parser.add_argument("--adjudication-json", default=str(DEFAULT_OUTPUT_JSON))
+ parser.add_argument("--adjudication-md", default=str(DEFAULT_OUTPUT_MD))
+ args = parser.parse_args()
+ payload = import_output_review_decisions(
+ source_path=Path(args.input).resolve(),
+ blind_pack_path=Path(args.blind_pack).resolve(),
+ output_json=Path(args.output_json).resolve(),
+ source_format=args.format,
+ reviewer=args.reviewer or "",
+ reviewed_at=args.reviewed_at or "",
+ run_adjudication=args.run_adjudication,
+ answer_key_path=Path(args.answer_key).resolve(),
+ adjudication_json=Path(args.adjudication_json).resolve(),
+ adjudication_md=Path(args.adjudication_md).resolve(),
+ blind_review_attested=args.blind_review_attested,
+ )
+ print(json.dumps(payload, ensure_ascii=False, indent=2))
+ raise SystemExit(0 if payload["ok"] else 2)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/import_telemetry_events.py b/scripts/import_telemetry_events.py
new file mode 100644
index 00000000..d2802fe8
--- /dev/null
+++ b/scripts/import_telemetry_events.py
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from render_adoption_drift_report import display_path, normalize_event, render_report, skill_defaults
+
+
+def load_candidate_events(
+ input_jsonl: Path,
+ defaults: dict[str, str],
+ default_source: str,
+ default_command: str,
+) -> tuple[list[dict[str, str]], list[str]]:
+ events: list[dict[str, str]] = []
+ failures: list[str] = []
+ for index, line in enumerate(input_jsonl.read_text(encoding="utf-8").splitlines(), start=1):
+ if not line.strip():
+ continue
+ try:
+ raw = json.loads(line)
+ except json.JSONDecodeError as exc:
+ failures.append(f"line {index}: invalid JSONL event: {exc.msg}")
+ continue
+ if not isinstance(raw, dict):
+ failures.append(f"line {index}: telemetry event must be a JSON object")
+ continue
+ raw.setdefault("source", default_source)
+ raw.setdefault("command", default_command)
+ event, event_failures = normalize_event(raw, defaults, f"line {index}")
+ failures.extend(event_failures)
+ if event:
+ events.append(event)
+ return events, failures
+
+
+def append_events(events_jsonl: Path, events: list[dict[str, str]]) -> None:
+ events_jsonl.parent.mkdir(parents=True, exist_ok=True)
+ with events_jsonl.open("a", encoding="utf-8") as handle:
+ for event in events:
+ handle.write(json.dumps(event, ensure_ascii=False, sort_keys=True) + "\n")
+
+
+def import_events(
+ skill_dir: Path,
+ input_jsonl: Path,
+ events_jsonl: Path | None = None,
+ output_json: Path | None = None,
+ output_md: Path | None = None,
+ generated_at: str | None = None,
+ default_source: str = "external",
+ default_command: str = "external-client",
+ dry_run: bool = False,
+) -> dict[str, Any]:
+ skill_dir = skill_dir.resolve()
+ input_jsonl = input_jsonl.resolve()
+ reports_dir = skill_dir / "reports"
+ events_jsonl = (events_jsonl or reports_dir / "telemetry_events.jsonl").resolve()
+ output_json = (output_json or reports_dir / "adoption_drift_report.json").resolve()
+ output_md = (output_md or reports_dir / "adoption_drift_report.md").resolve()
+ failures: list[str] = []
+ if not input_jsonl.exists():
+ failures.append(f"Input telemetry JSONL does not exist: {display_path(input_jsonl)}")
+ candidate_events: list[dict[str, str]] = []
+ else:
+ candidate_events, load_failures = load_candidate_events(
+ input_jsonl,
+ skill_defaults(skill_dir),
+ default_source,
+ default_command,
+ )
+ failures.extend(load_failures)
+
+ adoption_report: dict[str, Any] | None = None
+ if not failures and not dry_run:
+ append_events(events_jsonl, candidate_events)
+ adoption_report = render_report(
+ skill_dir,
+ events_jsonl=events_jsonl,
+ output_json=output_json,
+ output_md=output_md,
+ generated_at=generated_at,
+ )
+
+ return {
+ "ok": not failures,
+ "schema_version": "1.0",
+ "skill_dir": display_path(skill_dir),
+ "input_jsonl": display_path(input_jsonl),
+ "events_jsonl": display_path(events_jsonl),
+ "dry_run": dry_run,
+ "imported_count": 0 if failures or dry_run else len(candidate_events),
+ "candidate_count": len(candidate_events),
+ "failures": failures,
+ "imported_preview": candidate_events[:5],
+ "artifacts": {
+ "events_jsonl": display_path(events_jsonl),
+ "adoption_drift_json": display_path(output_json),
+ "adoption_drift_md": display_path(output_md),
+ },
+ "adoption_drift": {
+ "summary": adoption_report.get("summary", {}) if adoption_report else {},
+ "artifacts": adoption_report.get("artifacts", {}) if adoption_report else {},
+ },
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="Import external metadata-only telemetry into a local skill event stream.")
+ parser.add_argument("skill_dir", nargs="?", default=".")
+ parser.add_argument("--input-jsonl", required=True)
+ parser.add_argument("--events-jsonl")
+ parser.add_argument("--output-json")
+ parser.add_argument("--output-md")
+ parser.add_argument("--generated-at")
+ parser.add_argument("--source", choices=["external", "manual", "unknown", "yao_cli"], default="external")
+ parser.add_argument("--command", default="external-client")
+ parser.add_argument("--dry-run", action="store_true")
+ args = parser.parse_args()
+
+ report = import_events(
+ Path(args.skill_dir),
+ Path(args.input_jsonl),
+ events_jsonl=Path(args.events_jsonl) if args.events_jsonl else None,
+ output_json=Path(args.output_json) if args.output_json else None,
+ output_md=Path(args.output_md) if args.output_md else None,
+ generated_at=args.generated_at,
+ default_source=args.source,
+ default_command=args.command,
+ dry_run=args.dry_run,
+ )
+ print(json.dumps(report, ensure_ascii=False, indent=2))
+ if not report["ok"]:
+ raise SystemExit(2)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/init_skill.py b/scripts/init_skill.py
index be9b33e8..ff67e8fc 100644
--- a/scripts/init_skill.py
+++ b/scripts/init_skill.py
@@ -21,6 +21,7 @@ from render_reference_scan import parse_reference, render_reference_scan
from render_reference_synthesis import render_reference_synthesis
from render_review_studio import render_review_studio
from render_review_viewer import render_review_viewer
+from render_skill_interpretation import render_skill_interpretation
from render_skill_overview import render_skill_overview
from render_system_model import render_system_model
@@ -73,19 +74,20 @@ README_TEMPLATE = """# {title}
4. Review `reports/intent-confidence.md` to see whether the real job, inputs, outputs, and exclusions are clear enough yet.
5. Open `reports/reference-synthesis.md` to see the GitHub benchmarks plus curated official, research, and principle tracks in one place.
6. Follow the workflow steps in `SKILL.md`.
-7. Check `reports/skill-overview.html` for the generated bilingual HTML skill audit report: overview, metrics, capability profile, principle, contract, quality, risk, assets, and iteration roadmap. It defaults to Simplified Chinese and includes an English switch in the top right.
-8. Open `reports/review-studio.html` for the one-page Review Studio 2.0 gate view.
-9. Record source-line reviewer comments in `reports/review_annotations.md` when review needs follow-up.
-10. Open `reports/review-viewer.html` for a compact visual review of the package.
-11. Check `reports/output-risk-profile.md` to see likely output mistakes and self-repair checks.
-12. Check `reports/artifact-design-profile.md` to see the intended artifact direction, layout patterns, visual quality gates, and anti-patterns.
-13. Check `reports/prompt-quality-profile.md` to see the need model, RTF-to-skill mapping, complexity, and prompt-facing quality matrix.
-14. Review `reports/skill-ir.json` for the platform-neutral Skill IR contract before platform-specific packaging.
-15. Review `reports/compiled_targets.md` to see how Skill IR compiles into OpenAI, Claude, generic, and Agent Skills compatible target contracts.
-16. Review `reports/iteration-directions.md` for the three most valuable next moves.
-17. Review `reports/system-model.md` to understand the boundary, feedback loops, drift watch, failure map, and highest-leverage next changes.
-18. Review `reports/adoption_drift_report.md` to see local-first metadata-only adoption and drift signals.
-19. Review `reports/review_waivers.md` to see human reviewer risk approvals and expiry dates.
+7. Open `reports/skill-interpretation.html` first for the first-class bilingual interpretation report: role, principle, scenarios, trigger, inputs, outputs, highlights, risks, assets, and upgrade directions. It defaults to Simplified Chinese and includes an English switch in the top right.
+8. Check `reports/skill-overview.html` for the generated bilingual HTML skill audit report: overview, metrics, capability profile, principle, contract, quality, risk, assets, and iteration roadmap.
+9. Open `reports/review-studio.html` for the one-page Review Studio 2.0 gate view.
+10. Record source-line reviewer comments in `reports/review_annotations.md` when review needs follow-up.
+11. Open `reports/review-viewer.html` for a compact visual review of the package.
+12. Check `reports/output-risk-profile.md` to see likely output mistakes and self-repair checks.
+13. Check `reports/artifact-design-profile.md` to see the intended artifact direction, layout patterns, visual quality gates, and anti-patterns.
+14. Check `reports/prompt-quality-profile.md` to see the need model, RTF-to-skill mapping, complexity, and prompt-facing quality matrix.
+15. Review `reports/skill-ir.json` for the platform-neutral Skill IR contract before platform-specific packaging.
+16. Review `reports/compiled_targets.md` to see how Skill IR compiles into OpenAI, Claude, generic, and Agent Skills compatible target contracts.
+17. Review `reports/iteration-directions.md` for the three most valuable next moves.
+18. Review `reports/system-model.md` to understand the boundary, feedback loops, drift watch, failure map, and highest-leverage next changes.
+19. Review `reports/adoption_drift_report.md` to see local-first metadata-only adoption and drift signals.
+20. Review `reports/review_waivers.md` to see human reviewer risk approvals and expiry dates.
## Honest Boundaries
@@ -109,6 +111,7 @@ README_TEMPLATE = """# {title}
- `reports/system-model.md`: systems-thinking model for boundary, feedback loops, drift, failure patterns, and leverage points
- `reports/skill-ir.json`: platform-neutral 2.0 Skill IR contract for trigger, workflow, resources, evals, risk, and governance
- `reports/compiled_targets.md`: target compiler report showing generated contracts, adapter modes, preserved semantics, warnings, and unsupported features
+- `reports/skill-interpretation.html`: first-class bilingual interpretation report for role, principle, scenarios, trigger, inputs, outputs, quality evidence, risks, assets, highlights, and upgrade directions
- `reports/skill-overview.html`: white-background bilingual HTML skill audit report with sticky four-character Chinese navigation, a top-right language switch, metrics, SVG charts, contract boundary, quality review, risk governance, assets, and iteration roadmap
- `reports/review-studio.html`: Review Studio 2.0 gate page for intent, trigger, output eval, context, runtime conformance, trust, atlas, and release readiness
- `reports/review-viewer.html`: compact review page for architecture, usage, feedback, and next steps
@@ -129,6 +132,7 @@ compatibility:
- "openai"
- "claude"
- "generic"
+ - "vscode"
activation:
mode: "manual"
paths: []
@@ -143,6 +147,7 @@ compatibility:
openai: "metadata-adapter"
claude: "neutral-source-plus-adapter"
generic: "neutral-source"
+ vscode: "agent-skills-source-with-vscode-notes"
"""
@@ -213,6 +218,7 @@ def build_manifest(name: str, mode: str, archetype: str) -> dict:
"claude",
"generic",
"agent-skills-compatible",
+ "vscode",
],
"factory_components": [
"references",
@@ -225,23 +231,26 @@ def build_manifest(name: str, mode: str, archetype: str) -> dict:
def build_report_view(artifacts: dict) -> dict:
html_report = artifacts.get("skill_overview_html", "")
json_report = artifacts.get("skill_overview_json", "")
+ interpretation_report = artifacts.get("skill_interpretation_html", "")
system_model = artifacts.get("system_model_md", "")
review_studio = artifacts.get("review_studio_html", "")
return {
"title": "Skill 总结报告",
"html_report": html_report,
"json_report": json_report,
+ "interpretation_report": interpretation_report,
"system_model": system_model,
"review_studio": review_studio,
"message": (
- f"Skill 已创建完成。建议先打开总结报告：{html_report}。"
- "它会展示这个 Skill 的概述、指标、原理、触发边界、输入输出、目标编译、质量评估、风险治理、包体资产和升级路线；"
+ f"Skill 已创建完成。建议先打开解读报告：{interpretation_report}；再查看总结报告：{html_report}。"
+ "解读报告会用中文简体默认展示这个 Skill 的作用、原理、使用场景、触发方式、输入输出、亮点和后续升级方向；"
+ "总结报告会展示概述、指标、原理、触发边界、输入输出、目标编译、质量评估、风险治理、包体资产和升级路线；"
f"然后打开 Review Studio 2.0：{review_studio}，检查意图、触发、输出评测、运行一致性、信任和发布闸门。"
"后续 reviewer 的文件级或行级意见可以记录到 reports/review_annotations.md。"
"如需审查平台适配细节，请打开 reports/compiled_targets.md。"
"报告默认使用中文简体，右上角可以切换英文版。"
),
- "next_action": "Open reports/skill-overview.html before editing more files.",
+ "next_action": "Open reports/skill-interpretation.html before editing more files.",
}
@@ -356,6 +365,7 @@ def initialize_skill(
review_waivers = render_review_waivers(root)
review_annotations = render_review_annotations(root)
overview = render_skill_overview(root)
+ interpretation = render_skill_interpretation(root)
review_viewer = render_review_viewer(root)
review_studio = render_review_studio(root)
artifacts = {
@@ -366,6 +376,8 @@ def initialize_skill(
"intent_context_json": intent_confidence["artifacts"]["context_json"],
"skill_overview_html": overview["artifacts"]["html"],
"skill_overview_json": overview["artifacts"]["json"],
+ "skill_interpretation_html": interpretation["artifacts"]["html"],
+ "skill_interpretation_json": interpretation["artifacts"]["json"],
"intent_dialogue_md": intent_dialogue["artifacts"]["markdown"],
"intent_dialogue_json": intent_dialogue["artifacts"]["json"],
"reference_scan_md": reference_scan["artifacts"]["markdown"],
diff --git a/scripts/optimize_description.py b/scripts/optimize_description.py
index 560626e5..9bfee6d5 100644
--- a/scripts/optimize_description.py
+++ b/scripts/optimize_description.py
@@ -4,6 +4,7 @@ import json
from pathlib import Path
from context_sizer import estimate_tokens
+from description_optimizer_reporting import render_markdown
from judge_blind_eval import evaluate_judge
from trigger_eval import (
compare_reports,
@@ -539,145 +540,6 @@ def optimize(
return report
-def render_markdown(report: dict, title: str) -> str:
- lines = [
- f"# {title}",
- "",
- f"Winner: `{report['winner']['label']}`",
- "",
- f"- current tokens: `{report['current_candidate']['estimated_tokens']}`",
- f"- winner tokens: `{report['winner']['estimated_tokens']}`",
- ]
- if report["baseline"]:
- lines.append(f"- baseline tokens: `{report['baseline']['estimated_tokens']}`")
- lines.extend(
- [
- "",
- "## Winner",
- "",
- report["winner"]["description"],
- "",
- "## Candidate Ranking",
- "",
- "| Candidate | Tokens | Dev FP | Dev FN | Dev Near | Holdout FP | Holdout FN |",
- "| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
- ]
- )
- for candidate in report["candidates"]:
- holdout = candidate.get("holdout", {})
- lines.append(
- f"| `{candidate['label']}` | {candidate['estimated_tokens']} | {candidate['dev']['false_positives']} | {candidate['dev']['false_negatives']} | {candidate['dev']['near_neighbor_pass_rate']} | {holdout.get('false_positives', '-')} | {holdout.get('false_negatives', '-')} |"
- )
-
- lines.extend(
- [
- "",
- "## Acceptance Gates",
- "",
- "| Gate | Winner FP | Winner FN | Current FP | Current FN | Baseline FP | Baseline FN |",
- "| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
- ]
- )
- for gate_name, gate in (
- ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
- ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
- ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
- ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
- ):
- winner_gate = gate.get("winner") or {}
- current_gate = gate.get("current") or {}
- baseline_gate = gate.get("baseline") or {}
- if not winner_gate and not current_gate and not baseline_gate:
- continue
- lines.append(
- f"| {gate_name} | {winner_gate.get('false_positives', '-')} | {winner_gate.get('false_negatives', '-')} | {current_gate.get('false_positives', '-')} | {current_gate.get('false_negatives', '-')} | {baseline_gate.get('false_positives', '-')} | {baseline_gate.get('false_negatives', '-')} |"
- )
-
- lines.extend(
- [
- "",
- "## Calibration",
- "",
- "| Gate | Winner Gap | Winner Risk | Winner Boundary Rate | Current Gap | Baseline Gap |",
- "| --- | ---: | --- | ---: | ---: | ---: |",
- ]
- )
- for gate_name, gate in (
- ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
- ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
- ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
- ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
- ):
- winner_calibration = gate.get("winner_calibration") or {}
- current_calibration = gate.get("current_calibration") or {}
- baseline_calibration = gate.get("baseline_calibration") or {}
- if not winner_calibration and not current_calibration and not baseline_calibration:
- continue
- lines.append(
- f"| {gate_name} | {winner_calibration.get('score_gap', '-')} | {winner_calibration.get('risk_band', '-')} | {winner_calibration.get('boundary_case_rate', '-')} | {current_calibration.get('score_gap', '-')} | {baseline_calibration.get('score_gap', '-')} |"
- )
-
- lines.extend(
- [
- "",
- "## Judge Blind Summary",
- "",
- "| Gate | Winner Agreement | Winner Mean Confidence | Current Agreement | Baseline Agreement |",
- "| --- | ---: | ---: | ---: | ---: |",
- ]
- )
- judge_gate = report["acceptance_gates"]["judge_blind_holdout_non_regression"]
- judge_winner = (judge_gate.get("winner") or {}).get("judge_summary") or {}
- judge_current = (judge_gate.get("current") or {}).get("judge_summary") or {}
- judge_baseline = (judge_gate.get("baseline") or {}).get("judge_summary") or {}
- if judge_winner or judge_current or judge_baseline:
- lines.append(
- f"| Judge Blind Holdout | {judge_winner.get('agreement_rate', '-')} | {judge_winner.get('mean_confidence', '-')} | {judge_current.get('agreement_rate', '-')} | {judge_baseline.get('agreement_rate', '-')} |"
- )
-
- lines.extend(
- [
- "",
- "## Family Health",
- "",
- "| Gate | Winner Clean Families | Winner Weakest Family | Current Clean Families | Baseline Clean Families |",
- "| --- | --- | --- | --- | --- |",
- ]
- )
- for gate_name, gate in (
- ("Holdout", report["acceptance_gates"]["holdout_non_regression"]),
- ("Blind Holdout", report["acceptance_gates"]["blind_holdout_non_regression"]),
- ("Judge Blind Holdout", report["acceptance_gates"]["judge_blind_holdout_non_regression"]),
- ("Adversarial Holdout", report["acceptance_gates"]["adversarial_holdout_non_regression"]),
- ):
- winner_health = gate.get("winner_family_health") or {}
- current_health = gate.get("current_family_health") or {}
- baseline_health = gate.get("baseline_family_health") or {}
- if not winner_health and not current_health and not baseline_health:
- continue
- weakest = winner_health.get("weakest_family") or {}
- weakest_label = (
- f"{weakest.get('family')} ({weakest.get('errors')} errors)"
- if weakest.get("family")
- else "-"
- )
- lines.append(
- f"| {gate_name} | {winner_health.get('clean_family_count', '-')}/{winner_health.get('family_count', '-')} | {weakest_label} | {current_health.get('clean_family_count', '-')}/{current_health.get('family_count', '-')} | {baseline_health.get('clean_family_count', '-')}/{baseline_health.get('family_count', '-')} |"
- )
-
- lines.extend(
- [
- "",
- "## Selection Logic",
- "",
- "Ordered by:",
- ]
- )
- for item in report["selection_logic"]["priority"]:
- lines.append(f"- {item}")
- return "\n".join(lines) + "\n"
-
-
def main() -> None:
parser = argparse.ArgumentParser(
description="Generate and score description candidates on dev, holdout, blind, and adversarial suites."
diff --git a/scripts/output_review_privacy.py b/scripts/output_review_privacy.py
new file mode 100644
index 00000000..a19011cc
--- /dev/null
+++ b/scripts/output_review_privacy.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by output-review import and world-class human evidence validators to reject raw content, credential, secret, token, and answer-key fields recursively."
+
+RAW_CONTENT_FIELDS = {
+ "api_key",
+ "assistant_message",
+ "assistant_messages",
+ "baseline_output",
+ "credential",
+ "credentials",
+ "input",
+ "inputs",
+ "message",
+ "messages",
+ "model_output",
+ "output",
+ "outputs",
+ "prompt",
+ "prompts",
+ "raw_content",
+ "raw_output",
+ "raw_prompt",
+ "raw_provider_prompt",
+ "raw_user_content",
+ "secret",
+ "secrets",
+ "token",
+ "transcript",
+ "transcripts",
+ "user_message",
+ "user_messages",
+ "with_skill_output",
+}
+
+ANSWER_KEY_FIELDS = {
+ "answer_key",
+ "baseline_label",
+ "expected",
+ "expected_winner",
+ "expected_winner_role",
+ "expected_winner_variant",
+ "label",
+ "variant_label",
+ "with_skill_label",
+}
+
+BLOCKED_DECISION_FIELDS = RAW_CONTENT_FIELDS | ANSWER_KEY_FIELDS
+
+
+def forbidden_decision_field_paths(value: Any, prefix: str) -> list[str]:
+ found: list[str] = []
+ if isinstance(value, dict):
+ for key, child in value.items():
+ key_text = str(key).strip()
+ child_path = f"{prefix}.{key_text}"
+ if key_text.casefold() in BLOCKED_DECISION_FIELDS:
+ found.append(child_path)
+ found.extend(forbidden_decision_field_paths(child, child_path))
+ elif isinstance(value, list):
+ for index, child in enumerate(value):
+ found.extend(forbidden_decision_field_paths(child, f"{prefix}[{index}]"))
+ return found
+
+
+def forbidden_raw_content_field_paths(value: Any, prefix: str) -> list[str]:
+ found: list[str] = []
+ if isinstance(value, dict):
+ for key, child in value.items():
+ key_text = str(key).strip()
+ child_path = f"{prefix}.{key_text}"
+ if key_text.casefold() in RAW_CONTENT_FIELDS:
+ found.append(child_path)
+ found.extend(forbidden_raw_content_field_paths(child, child_path))
+ elif isinstance(value, list):
+ for index, child in enumerate(value):
+ found.extend(forbidden_raw_content_field_paths(child, f"{prefix}[{index}]"))
+ return found
diff --git a/scripts/prepare_output_review_kit.py b/scripts/prepare_output_review_kit.py
new file mode 100644
index 00000000..26c972ad
--- /dev/null
+++ b/scripts/prepare_output_review_kit.py
@@ -0,0 +1,485 @@
+#!/usr/bin/env python3
+"""Prepare a reviewer-facing blind A/B output review kit."""
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+from adjudicate_output_review import (
+ build_decision_template,
+ confidence_value,
+ decision_index,
+ display_path,
+ load_json,
+ normalize_variant,
+)
+from html_rendering import html_text
+
+
+ROOT = Path(__file__).resolve().parent.parent
+DEFAULT_BLIND_PACK_JSON = ROOT / "reports" / "output_blind_review_pack.json"
+DEFAULT_BLIND_PACK_MD = ROOT / "reports" / "output_blind_review_pack.md"
+DEFAULT_DECISIONS = ROOT / "reports" / "output_review_decisions.json"
+DEFAULT_OUTPUT_JSON = ROOT / "reports" / "output_review_kit.json"
+DEFAULT_OUTPUT_MD = ROOT / "reports" / "output_review_kit.md"
+DEFAULT_OUTPUT_HTML = ROOT / "reports" / "output_review_kit.html"
+
+
+def load_optional_decisions(path: Path) -> tuple[dict[str, Any], list[str]]:
+ if not path.exists():
+ return {"schema_version": "1.0", "decisions": []}, []
+ return load_json(path)
+
+
+def pairs_from_pack(blind_pack: dict[str, Any]) -> list[dict[str, Any]]:
+ pairs = blind_pack.get("pairs", [])
+ if not isinstance(pairs, list):
+ return []
+ return [item for item in pairs if isinstance(item, dict) and item.get("case_id")]
+
+
+def decision_state(decision: dict[str, Any] | None) -> dict[str, Any]:
+ if not decision:
+ return {
+ "status": "awaiting-decision",
+ "winner_variant_recorded": False,
+ "confidence_recorded": False,
+ "reason_recorded": False,
+ "blocking_reason": "No reviewer choice has been recorded yet.",
+ }
+ winner = normalize_variant(decision.get("winner_variant", decision.get("reviewer_winner_variant", decision.get("winner", ""))))
+ confidence, confidence_failure = confidence_value(decision.get("confidence"))
+ reason = str(decision.get("reason", "")).strip()
+ if not winner and confidence is None and not reason:
+ return {
+ "status": "awaiting-decision",
+ "winner_variant_recorded": False,
+ "confidence_recorded": False,
+ "reason_recorded": False,
+ "blocking_reason": "Decision template exists but this row is still blank.",
+ }
+ failures = []
+ if winner not in {"A", "B"}:
+ failures.append("winner_variant must be A or B")
+ if confidence_failure:
+ failures.append(confidence_failure)
+ if not reason:
+ failures.append("reason should explain the reviewer rationale")
+ if failures:
+ return {
+ "status": "needs-fix",
+ "winner_variant_recorded": winner in {"A", "B"},
+ "confidence_recorded": confidence is not None,
+ "reason_recorded": bool(reason),
+ "blocking_reason": "; ".join(failures),
+ }
+ return {
+ "status": "ready-for-adjudication",
+ "winner_variant_recorded": True,
+ "confidence_recorded": confidence is not None,
+ "reason_recorded": True,
+ "blocking_reason": "Reviewer choice is complete enough for adjudication.",
+ }
+
+
+def build_cases(blind_pack: dict[str, Any], decisions: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:
+ cases = []
+ for pair in pairs_from_pack(blind_pack):
+ case_id = str(pair.get("case_id", ""))
+ rubric = []
+ for item in pair.get("rubric", []) if isinstance(pair.get("rubric", []), list) else []:
+ if isinstance(item, dict):
+ rubric.append(
+ {
+ "id": str(item.get("id", "")),
+ "description": str(item.get("description", "")),
+ "weight": float(item.get("weight", 1) or 0),
+ }
+ )
+ cases.append(
+ {
+ "case_id": case_id,
+ "prompt": str(pair.get("prompt", "")),
+ "input_files": pair.get("input_files", []) if isinstance(pair.get("input_files", []), list) else [],
+ "review_instruction": str(pair.get("review_instruction", "")),
+ "rubric": rubric,
+ "variant_a": {
+ "blind_id": str(pair.get("variant_a", {}).get("blind_id", "")) if isinstance(pair.get("variant_a"), dict) else "",
+ "output": str(pair.get("variant_a", {}).get("output", "")) if isinstance(pair.get("variant_a"), dict) else "",
+ },
+ "variant_b": {
+ "blind_id": str(pair.get("variant_b", {}).get("blind_id", "")) if isinstance(pair.get("variant_b"), dict) else "",
+ "output": str(pair.get("variant_b", {}).get("output", "")) if isinstance(pair.get("variant_b"), dict) else "",
+ },
+ "decision_state": decision_state(decisions.get(case_id)),
+ }
+ )
+ return cases
+
+
+def build_summary(cases: list[dict[str, Any]], failures: list[str], reviewer: str, reviewed_at: str) -> dict[str, Any]:
+ status_counts: dict[str, int] = {}
+ for case in cases:
+ status = str(case.get("decision_state", {}).get("status", "awaiting-decision"))
+ status_counts[status] = status_counts.get(status, 0) + 1
+ ready_count = status_counts.get("ready-for-adjudication", 0)
+ pending_count = status_counts.get("awaiting-decision", 0)
+ invalid_count = status_counts.get("needs-fix", 0)
+ return {
+ "case_count": len(cases),
+ "ready_for_adjudication_count": ready_count,
+ "pending_decision_count": pending_count,
+ "invalid_decision_count": invalid_count,
+ "reviewer_metadata_present": bool(str(reviewer).strip() and str(reviewed_at).strip()),
+ "answer_key_hidden": True,
+ "answer_key_path_exposed": False,
+ "ready_to_run_adjudication": bool(cases) and ready_count == len(cases) and invalid_count == 0,
+ "failure_count": len(failures),
+ }
+
+
+def build_contract(blind_pack_md: Path, decisions_path: Path) -> dict[str, Any]:
+ return {
+ "reviewer_steps": [
+ f"Open {display_path(blind_pack_md)} or this kit and compare Variant A vs Variant B for each case.",
+ f"Record choices in {display_path(decisions_path)} without opening the answer key.",
+ "Use winner_variant A or B, confidence from 0 to 1, and a short reason for every case.",
+ "Set reviewer_attestation truthfully after choices are recorded and before adjudication.",
+ "Run python3 scripts/yao.py output-review-import --input --blind-review-attested --run-adjudication after choices are recorded.",
+ "Refresh python3 scripts/yao.py review-studio . before asking for release approval.",
+ ],
+ "required_fields": {
+ "reviewer": "Human reviewer name or review group.",
+ "reviewed_at": "Review date or timestamp.",
+ "winner_variant": "A or B for every case.",
+ "confidence": "Optional numeric confidence from 0 to 1.",
+ "reason": "Required rationale based on the rubric, not on hidden labels.",
+ "reviewer_attestation": "Blind-review and answer-key-order attestation after decisions are complete.",
+ "review_integrity": "Blind pack and prompt fingerprints generated by the template.",
+ },
+ "privacy_contract": [
+ "The answer key is intentionally withheld from this kit.",
+ "Do not inspect hidden labels or expected winners before decisions are recorded.",
+ "Do not paste private user data into decision reasons.",
+ "Pending decisions must stay pending instead of being counted as human agreement.",
+ ],
+ }
+
+
+def render_rubric(rubric: list[dict[str, Any]]) -> list[str]:
+ if not rubric:
+ return ["- No rubric items found."]
+ return [f"- `{item['id']}` ({item['weight']}): {item['description']}" for item in rubric]
+
+
+def render_markdown(payload: dict[str, Any]) -> str:
+ summary = payload["summary"]
+ contract = payload["review_contract"]
+ lines = [
+ "# Output Review Kit",
+ "",
+ "This reviewer-facing packet contains the blind A/B cases, decision fields, and command flow. It intentionally does not expose the answer key.",
+ "",
+ "## Summary",
+ "",
+ f"- cases: `{summary['case_count']}`",
+ f"- ready for adjudication: `{summary['ready_for_adjudication_count']}`",
+ f"- pending decisions: `{summary['pending_decision_count']}`",
+ f"- invalid decisions: `{summary['invalid_decision_count']}`",
+ f"- reviewer metadata present: `{str(summary['reviewer_metadata_present']).lower()}`",
+ f"- answer key hidden: `{str(summary['answer_key_hidden']).lower()}`",
+ f"- answer key path exposed: `{str(summary['answer_key_path_exposed']).lower()}`",
+ "",
+ "## Review Flow",
+ "",
+ ]
+ lines.extend(f"{index}. {step}" for index, step in enumerate(contract["reviewer_steps"], start=1))
+ lines.extend(["", "## Required Fields", ""])
+ lines.extend(f"- `{key}`: {value}" for key, value in contract["required_fields"].items())
+ lines.extend(["", "## Privacy Contract", ""])
+ lines.extend(f"- {item}" for item in contract["privacy_contract"])
+ lines.extend(
+ [
+ "",
+ "## Decision States",
+ "",
+ "| Case | State | Winner | Confidence | Reason | Blocking Reason |",
+ "| --- | --- | --- | --- | --- | --- |",
+ ]
+ )
+ for case in payload["cases"]:
+ state = case["decision_state"]
+ lines.append(
+ f"| `{case['case_id']}` | `{state['status']}` | `{str(state['winner_variant_recorded']).lower()}` | "
+ f"`{str(state['confidence_recorded']).lower()}` | `{str(state['reason_recorded']).lower()}` | {state['blocking_reason']} |"
+ )
+ lines.extend(["", "## Blind Cases", ""])
+ for case in payload["cases"]:
+ lines.extend(
+ [
+ f"### {case['case_id']}",
+ "",
+ f"Prompt: {case['prompt']}",
+ "",
+ "Rubric:",
+ *render_rubric(case["rubric"]),
+ "",
+ "#### Variant A",
+ "",
+ case["variant_a"]["output"] or "_No output found._",
+ "",
+ "#### Variant B",
+ "",
+ case["variant_b"]["output"] or "_No output found._",
+ "",
+ ]
+ )
+ if payload.get("failures"):
+ lines.extend(["## Failures", ""])
+ lines.extend(f"- {failure}" for failure in payload["failures"])
+ return "\n".join(lines).strip() + "\n"
+
+
+def status_label(status: str) -> str:
+ return {
+ "awaiting-decision": "Awaiting",
+ "needs-fix": "Needs fix",
+ "ready-for-adjudication": "Ready",
+ }.get(status, status)
+
+
+def render_html_rubric(rubric: list[dict[str, Any]]) -> str:
+ if not rubric:
+ return "
No rubric items found.
"
+ return "".join(
+ "
{id}
{description}
{weight}
".format(
+ id=html_text(item.get("id", "")),
+ description=html_text(item.get("description", "")),
+ weight=html_text(item.get("weight", "")),
+ )
+ for item in rubric
+ )
+
+
+def render_html_cases(cases: list[dict[str, Any]]) -> str:
+ cards = []
+ for index, case in enumerate(cases, start=1):
+ state = case.get("decision_state", {})
+ status = str(state.get("status", "awaiting-decision"))
+ cards.append(
+ f"""
+
+
+
{bi_span("分数来自本地文件和 reports 证据，缺失时明确标为证据不足。", "Scores are derived from package files and reports; missing inputs are shown as evidence gaps.")}
-
-
-
- {charts["radar"]}
-
-
{bi_span("指标判读", "Reading")}
-
{bi_span("先看雷达图判断能力短板，再看下方每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。", "Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.")}
- {render_metric_summary(scorecard)}
-
-
-
{render_metric_cards(scorecard)}
+
+
{bi_span("指标判读", "Reading")}
+
{bi_span("先看雷达图判断能力短板，再看每项分数的证据原因。分数不是装饰数字，必须和本地文件、reports 证据或证据不足提示对应。", "Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.")}
+
+
+
+
+
+ {charts["radar"]}
+
+
{bi_span("成熟度条", "Maturity Bar")}
+
{bi_span("这里把每个指标压缩为状态、分数和第一条证据，先给出整体判断，再进入下方明细。", "This compresses each metric into status, score, and the first evidence item before the detailed cards below.")}
A single operating page for collecting the remaining human and external evidence. It coordinates action, but does not accept evidence or change the ledger.
+
{stat_html}
+
+
Fast Path
Run the real external or human work for one evidence item.
Generate and fill the matching submission draft.
Validate intake and inspect the submission review queue.
Refresh the ledger and run the claim guard before making any completion claim.
+
Coordination Plan
These steps tell the assistant what to run and where the user must supply real provider, reviewer, client, or telemetry input. Every row is operational guidance only and keeps counts_as_completion false.
{html.escape(str(step.get('repair_blocked_count', 0)))} repair rows; counts as completion: {html.escape(str(step.get('repair_counts_as_completion') is True).lower())}
"
+ f"
阶段
{html.escape(str(step.get('phase_queue_blocked_count', 0)))} blocked phases; counts as completion: {html.escape(str(step.get('phase_queue_counts_as_completion') is True).lower())}
This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.
+
{stat_html}
+
+
+
+
Submission Kit
+
Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.
submission-ref rows are the paths expected in artifact_refs; supporting-evidence rows stay available for audit context.
+
+
+
+
+
Phase Queue
+
Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later source collection. Phase queue rows remain procedural and never count as accepted evidence.
+
{phase_queue_html}
+
Evidence Queue
+
{item_cards}
+
+
+
Safety Boundary
+
Environment variables are displayed only as set or not-set; secret values are never printed.
Human-required and external-required states are operator work, not accepted evidence.
The world-class ledger remains the only source of truth for ready_to_claim_world_class.
+
+
+
+
+"""
+ return "\n".join(line.rstrip() for line in html.splitlines()) + "\n"
diff --git a/scripts/world_class_preflight_markdown.py b/scripts/world_class_preflight_markdown.py
new file mode 100644
index 00000000..da074e01
--- /dev/null
+++ b/scripts/world_class_preflight_markdown.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Markdown renderer for the world-class evidence preflight report."""
+
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by render_world_class_preflight.py to keep Markdown layout separate from preflight data assembly."
+
+
+def md_cell(value: Any) -> str:
+ return str(value).replace("|", "\\|")
+
+
+def render_markdown(report: dict[str, Any]) -> str:
+ summary = report["summary"]
+ role_contract = report["submissions"]["artifact_role_contract"]
+ lines = [
+ "# World-Class Evidence Preflight",
+ "",
+ f"Generated at: `{report['generated_at']}`",
+ "",
+ "## Summary",
+ "",
+ f"- decision: `{summary['decision']}`",
+ f"- ready to claim world-class: `{str(summary['ready_to_claim_world_class']).lower()}`",
+ f"- preflight counts as evidence: `{str(summary['preflight_counts_as_evidence']).lower()}`",
+ f"- credential value exposed: `{str(summary['credential_value_exposed']).lower()}`",
+ f"- collection ready: `{summary['collection_ready_count']}`",
+ f"- collection blocked: `{summary['collection_blocked_count']}`",
+ f"- source checks: `{summary['source_pass_count']}` pass / `{summary['source_check_count']}` total",
+ f"- repair rows: `{summary['repair_blocked_count']}` blocked / `{summary['repair_checklist_count']}` total",
+ f"- phase queue: `{summary['phase_queue_blocked_count']}` blocked / `{summary['phase_queue_count']}` phases",
+ f"- phase queue rows: `{summary['phase_queue_row_count']}`",
+ f"- next repair action: `{summary.get('next_repair_action_id', '')}`",
+ f"- next repair owner: `{summary.get('next_repair_owner', '')}`",
+ f"- next phase: `{summary.get('phase_queue_next_phase', '')}`",
+ f"- next phase action: `{summary.get('phase_queue_next_action_id', '')}`",
+ "",
+ "This preflight report checks whether an operator can start collecting the remaining external or human evidence. It never accepts evidence, prints secret values, or changes the world-class ledger.",
+ "",
+ "## Submission Kit Handoff",
+ "",
+ f"- submissions directory: `{report['submissions']['directory']}`",
+ f"- prepare drafts: `{report['submissions']['commands']['prepare_submission']}`",
+ f"- prepare drafts with artifact SHA prefill: `{report['submissions']['commands']['prepare_prefilled_submission']}`",
+ f"- validate intake: `{report['submissions']['commands']['validate_intake']}`",
+ f"- review queue: `{report['submissions']['commands']['submission_review']}`",
+ f"- refresh ledger: `{report['submissions']['commands']['refresh_ledger']}`",
+ f"- guard claims: `{report['submissions']['commands']['guard_claim']}`",
+ f"- drafts count as evidence: `{str(report['submissions']['drafts_count_as_evidence']).lower()}`",
+ f"- artifact prefill counts as evidence: `{str(report['submissions']['artifact_prefill_counts_as_evidence']).lower()}`",
+ f"- submission refs ready: `{role_contract['submission_ref_ready_count']}` / `{role_contract['submission_ref_total_count']}`",
+ f"- supporting evidence ready: `{role_contract['supporting_evidence_ready_count']}` / `{role_contract['supporting_evidence_total_count']}`",
+ "",
+ "Generate the submission kit after the real provider, human, native-permission, or native-client work exists. The generated JSON drafts remain `template_only: true` until an operator edits them with real aggregate artifact references and matching SHA-256 digests. The prefill command only inserts local artifact SHA-256 digests; it does not make a draft count as evidence.",
+ "",
+ "| Role | Copy to artifact_refs | Ready | Meaning |",
+ "| --- | --- | --- | --- |",
+ ]
+ for role in role_contract.get("roles", []):
+ if role.get("role") == "submission-ref":
+ ready = f"{role_contract['submission_ref_ready_count']} / {role_contract['submission_ref_total_count']}"
+ else:
+ ready = (
+ f"{role_contract['supporting_evidence_ready_count']} / "
+ f"{role_contract['supporting_evidence_total_count']}"
+ )
+ lines.append(
+ f"| `{role['role']}` | `{str(role['copy_to_artifact_refs']).lower()}` | `{ready}` | "
+ f"{md_cell(role['description'])} |"
+ )
+ lines.extend(
+ [
+ "",
+ "`submission-ref` rows are the only checklist rows expected in `artifact_refs`; `supporting-evidence` rows stay available for audit context and reviewer traceability.",
+ "",
+ "## Phase Queue",
+ "",
+ "Phase queue rows group the same repair checklist into operator execution phases. They are procedural guidance only and do not count as completion evidence.",
+ "",
+ "| Priority | Phase | Status | Rows | Owners | Evidence | Verify | Next action |",
+ "| --- | --- | --- | --- | --- | --- | --- | --- |",
+ ]
+ )
+ for row in report.get("phase_queue", []):
+ owners = ", ".join(str(owner) for owner in row.get("owners", []))
+ evidence_keys = ", ".join(str(key) for key in row.get("evidence_keys", []))
+ rows = f"{row.get('blocked_count', 0)} / {row.get('row_count', 0)} blocked"
+ lines.append(
+ f"| `{row.get('priority', '')}` | `{row.get('phase', '')}` | `{row.get('status', '')}` | "
+ f"{md_cell(rows)} | {md_cell(owners)} | {md_cell(evidence_keys)} | "
+ f"`{md_cell(row.get('verification_command', ''))}` | {md_cell(row.get('next_action', ''))} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Evidence Items",
+ "",
+ "| Evidence | Status | Intake | Review | Next action |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for item in report["items"]:
+ next_action = str(item.get("next_action", "")).replace("|", "\\|")
+ lines.append(
+ f"| `{item['evidence_key']}` | `{item['status']}` | `{item['intake_readiness']}` | `{item['review_state']}` | {next_action} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Repair Checklist",
+ "",
+ "Repair rows convert preflight and source blockers into a prioritized operator queue. They are guidance only and do not count as completion evidence.",
+ "",
+ "| Priority | Phase | Owner | Evidence | Type | Target | Status | Verify | Next action |",
+ "| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
+ ]
+ )
+ for row in report.get("repair_checklist", []):
+ lines.append(
+ f"| `{row.get('priority', '')}` | `{row.get('phase', '')}` | {md_cell(row.get('owner', ''))} | "
+ f"`{row['evidence_key']}` | `{row['repair_type']}` | `{row['target']}` | "
+ f"`{row['status']}` | `{md_cell(row.get('verification_command', ''))}` | {md_cell(row['next_action'])} |"
+ )
+ for item in report["items"]:
+ lines.extend(
+ [
+ "",
+ f"## {item['label']}",
+ "",
+ f"- status: `{item['status']}`",
+ f"- ledger: `{item['ledger_status']}`",
+ f"- submission: `{item.get('submission_path') or 'missing'}`",
+ f"- prepare draft: `{item['commands']['prepare_submission']}`",
+ f"- prepare draft with artifact SHA prefill: `{item['commands']['prepare_prefilled_submission']}`",
+ f"- submission refs ready: `{item['submission_kit']['artifact_role_contract']['submission_ref_ready_count']}` / `{item['submission_kit']['artifact_role_contract']['submission_ref_total_count']}`",
+ f"- supporting evidence ready: `{item['submission_kit']['artifact_role_contract']['supporting_evidence_ready_count']}` / `{item['submission_kit']['artifact_role_contract']['supporting_evidence_total_count']}`",
+ "",
+ "### Prechecks",
+ "",
+ "| Check | Kind | Current | Status | Next action |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for row in item.get("prechecks", []):
+ action = md_cell(row.get("next_action", ""))
+ lines.append(
+ f"| {row['label']} | `{row['kind']}` | `{row['actual']}` | `{row['status']}` | {action} |"
+ )
+ lines.extend(
+ [
+ "",
+ "### Source Checks",
+ "",
+ "| Check | Current | Expected | Status | Next action |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for row in item.get("source_checklist", []):
+ action = md_cell(row.get("next_action", ""))
+ lines.append(
+ f"| {row['label']} | `{row['actual']}` | `{row['expected']}` | `{row['status']}` | {action} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Boundary",
+ "",
+ "- Environment variables are reported only as `set` or `not-set`; values are never printed.",
+ "- Human-required and external-required states are operator actions, not accepted evidence.",
+ "- The world-class ledger remains the source of truth for `ready_to_claim_world_class`.",
+ ]
+ )
+ return "\n".join(lines).rstrip() + "\n"
diff --git a/scripts/world_class_provider_evidence.py b/scripts/world_class_provider_evidence.py
new file mode 100644
index 00000000..271e35ed
--- /dev/null
+++ b/scripts/world_class_provider_evidence.py
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+import re
+from typing import Any
+
+from output_review_privacy import forbidden_decision_field_paths
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by world_class_evidence_contract.py to validate provider-backed holdout execution evidence from run rows."
+
+SHA256_RE = re.compile(r"^[0-9a-fA-F]{64}$")
+
+
+def add_error(errors: list[str], condition: bool, message: str) -> None:
+ if not condition:
+ errors.append(message)
+
+
+def real_int(value: Any) -> int | None:
+ return value if isinstance(value, int) and not isinstance(value, bool) else None
+
+
+def summary(payload: dict[str, Any]) -> dict[str, Any]:
+ value = payload.get("summary", {})
+ return value if isinstance(value, dict) else {}
+
+
+def normalized(value: Any) -> str:
+ return str(value or "").strip().casefold()
+
+
+def run_rows(execution: dict[str, Any]) -> list[dict[str, Any]]:
+ runs = execution.get("runs", [])
+ return [item for item in runs if isinstance(item, dict)] if isinstance(runs, list) else []
+
+
+def validate_no_raw_fields(execution: dict[str, Any], errors: list[str]) -> None:
+ blocked = forbidden_decision_field_paths(execution, "output_execution_runs")
+ add_error(
+ errors,
+ not blocked,
+ "provider-holdout output execution report must not include raw content, credential, secret, token, prompt, output, transcript, message, or answer-key fields: "
+ + ", ".join(blocked[:8]),
+ )
+
+
+def positive_duration(value: Any) -> bool:
+ return isinstance(value, (int, float)) and not isinstance(value, bool) and value > 0
+
+
+def observed_usage(run: dict[str, Any]) -> bool:
+ usage = run.get("usage", {})
+ if not isinstance(usage, dict) or usage.get("estimated") is not False:
+ return False
+ token_values = [
+ real_int(usage.get("input_tokens")),
+ real_int(usage.get("output_tokens")),
+ real_int(usage.get("total_tokens")),
+ ]
+ return all(value is not None and value > 0 for value in token_values)
+
+
+def provider_model_runs(execution: dict[str, Any], provenance: dict[str, Any]) -> list[dict[str, Any]]:
+ expected_provider = normalized(provenance.get("provider"))
+ expected_model = normalized(provenance.get("model"))
+ matches: list[dict[str, Any]] = []
+ for run in run_rows(execution):
+ if run.get("status") != "pass":
+ continue
+ if run.get("model_executed") is not True or run.get("execution_mode") != "model":
+ continue
+ if normalized(run.get("provider")) != expected_provider:
+ continue
+ if normalized(run.get("model")) != expected_model:
+ continue
+ if not observed_usage(run):
+ continue
+ if not positive_duration(run.get("duration_ms")):
+ continue
+ if not SHA256_RE.match(str(run.get("output_sha256", "")).strip()):
+ continue
+ matches.append(run)
+ return matches
+
+
+def validate_provider_execution_report(
+ execution: dict[str, Any],
+ provenance: dict[str, Any],
+ errors: list[str],
+) -> None:
+ execution_summary = summary(execution)
+ runs = run_rows(execution)
+ validate_no_raw_fields(execution, errors)
+ add_error(errors, execution.get("ok") is True, "provider-holdout output execution report ok must be true")
+ add_error(errors, bool(runs), "provider-holdout output execution report must include run rows")
+
+ computed_model_runs = sum(1 for run in runs if run.get("model_executed") is True)
+ computed_timing = sum(1 for run in runs if run.get("duration_ms") is not None)
+ computed_token_observed = sum(1 for run in runs if isinstance(run.get("usage"), dict) and run["usage"].get("estimated") is False)
+ computed_failures = sum(1 for run in runs if run.get("status") != "pass")
+ add_error(
+ errors,
+ real_int(execution_summary.get("model_executed_count")) == computed_model_runs,
+ "provider-holdout output execution summary.model_executed_count must match model-executed run rows",
+ )
+ add_error(
+ errors,
+ real_int(execution_summary.get("timing_observed_count")) == computed_timing,
+ "provider-holdout output execution summary.timing_observed_count must match timed run rows",
+ )
+ add_error(
+ errors,
+ real_int(execution_summary.get("token_observed_count")) == computed_token_observed,
+ "provider-holdout output execution summary.token_observed_count must match non-estimated usage rows",
+ )
+ add_error(
+ errors,
+ real_int(execution_summary.get("failure_count")) == computed_failures,
+ "provider-holdout output execution summary.failure_count must match failed run rows",
+ )
+ add_error(
+ errors,
+ bool(real_int(execution_summary.get("model_executed_count")) and execution_summary["model_executed_count"] > 0),
+ "provider-holdout output execution summary.model_executed_count must be >0",
+ )
+ add_error(
+ errors,
+ bool(real_int(execution_summary.get("timing_observed_count")) and execution_summary["timing_observed_count"] > 0),
+ "provider-holdout output execution summary.timing_observed_count must be >0",
+ )
+ add_error(
+ errors,
+ bool(real_int(execution_summary.get("token_observed_count")) and execution_summary["token_observed_count"] > 0),
+ "provider-holdout output execution summary.token_observed_count must be >0",
+ )
+ add_error(
+ errors,
+ execution_summary.get("failure_count") == 0,
+ "provider-holdout output execution summary.failure_count must be 0",
+ )
+ add_error(
+ errors,
+ bool(provider_model_runs(execution, provenance)),
+ "provider-holdout output execution runs must include a passing model run with matching provider, model, timing, non-estimated usage, and output hash",
+ )
diff --git a/scripts/world_class_repair_checklist.py b/scripts/world_class_repair_checklist.py
new file mode 100644
index 00000000..9e8d9134
--- /dev/null
+++ b/scripts/world_class_repair_checklist.py
@@ -0,0 +1,286 @@
+#!/usr/bin/env python3
+"""Build machine-readable repair checklists for world-class submission kits."""
+
+from collections import defaultdict
+import re
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Shared by submission kit generation to turn readiness blockers into actionable repair rows."
+
+
+OWNER_BY_EVIDENCE = {
+ "provider-holdout": "operator with provider credentials",
+ "human-adjudication": "human reviewer",
+ "native-permission-enforcement": "target client or installer integrator",
+ "native-client-telemetry": "Browser/Chrome/IDE/provider client integrator",
+}
+
+REPAIR_PHASES = {
+ "unknown-key": ("select-evidence", 5),
+ "draft": ("prepare-draft", 10),
+ "precheck": ("unblock-access", 20),
+ "artifact": ("attach-artifacts", 30),
+ "source-check": ("collect-source", 40),
+}
+
+
+def _by_key(rows: list[dict[str, Any]], key_name: str = "evidence_key") -> dict[str, list[dict[str, Any]]]:
+ grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+ for row in rows:
+ key = str(row.get(key_name, "")).strip()
+ if key:
+ grouped[key].append(row)
+ return dict(grouped)
+
+
+def _slug(value: str) -> str:
+ slug = re.sub(r"[^a-zA-Z0-9_.-]+", "-", value.strip()).strip("-")
+ return slug or "repair"
+
+
+def repair_verification_command(evidence_key: str, repair_type: str) -> str:
+ preflight = "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions"
+ if repair_type == "draft":
+ return (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ f"--evidence-key {evidence_key} --output-dir evidence/world_class/submissions"
+ )
+ if repair_type == "artifact":
+ return (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ f"--evidence-key {evidence_key} --output-dir evidence/world_class/submissions --prefill-artifacts"
+ )
+ if repair_type == "unknown-key":
+ return "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ if evidence_key == "provider-holdout" and repair_type == "source-check":
+ return (
+ "python3 scripts/yao.py output-exec --provider-runner "
+ "--provider-model --timeout-seconds 60 && "
+ + preflight
+ )
+ if evidence_key == "human-adjudication" and repair_type == "source-check":
+ return "python3 scripts/yao.py output-review && " + preflight
+ if evidence_key == "native-permission-enforcement" and repair_type == "source-check":
+ return "python3 scripts/yao.py runtime-permissions . --package-dir dist && " + preflight
+ if evidence_key == "native-client-telemetry" and repair_type == "source-check":
+ return (
+ "python3 scripts/yao.py telemetry-import . "
+ "--input-jsonl .yao/telemetry_spool/external_events.jsonl && "
+ + preflight
+ )
+ return preflight
+
+
+def _repair_row(
+ *,
+ evidence_key: str,
+ repair_type: str,
+ target: str,
+ status: str,
+ blocking_reason: str,
+ next_action: str,
+ source: str,
+) -> dict[str, Any]:
+ phase, priority = REPAIR_PHASES.get(repair_type, ("repair", 90))
+ return {
+ "action_id": _slug(f"{evidence_key}:{repair_type}:{target}"),
+ "evidence_key": evidence_key,
+ "repair_type": repair_type,
+ "target": target,
+ "phase": phase,
+ "priority": priority,
+ "owner": OWNER_BY_EVIDENCE.get(evidence_key, "release reviewer"),
+ "status": status,
+ "blocking_reason": blocking_reason,
+ "next_action": next_action,
+ "verification_command": repair_verification_command(evidence_key, repair_type),
+ "source": source,
+ "counts_as_completion": False,
+ }
+
+
+def sort_repair_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return sorted(
+ rows,
+ key=lambda row: (
+ int(row.get("priority", 90) or 90),
+ str(row.get("evidence_key", "")),
+ str(row.get("target", "")),
+ ),
+ )
+
+
+def build_repair_checklist(
+ evidence_items: list[dict[str, Any]],
+ files: list[dict[str, Any]],
+ artifact_checklist: list[dict[str, Any]],
+ source_checklist: list[dict[str, Any]],
+ unknown_keys: list[str],
+) -> list[dict[str, Any]]:
+ """Return concrete repair rows for every draft, artifact, and source blocker."""
+ files_by_key = _by_key(files)
+ artifacts_by_key = _by_key(artifact_checklist)
+ sources_by_key = _by_key(source_checklist)
+ rows: list[dict[str, Any]] = []
+ known_keys = [str(item.get("evidence_key", "")).strip() for item in evidence_items]
+
+ for key in known_keys:
+ if not key:
+ continue
+ draft_rows = files_by_key.get(key, [])
+ if not draft_rows:
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="draft",
+ target=key,
+ status="blocked",
+ blocking_reason="Submission draft was not generated.",
+ next_action="Run the submission kit command again for this evidence key.",
+ source="files",
+ )
+ )
+ for draft in draft_rows:
+ draft_status = str(draft.get("status", "")).strip()
+ if draft_status not in {"written", "exists"}:
+ target = str(draft.get("output_path") or draft.get("template_path") or key)
+ errors = draft.get("errors", [])
+ error_text = "; ".join(str(error) for error in errors) if isinstance(errors, list) else ""
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="draft",
+ target=target,
+ status="blocked",
+ blocking_reason=error_text or f"Draft status is {draft_status or 'unknown'}.",
+ next_action="Fix template generation before asking for ledger review.",
+ source="files",
+ )
+ )
+
+ for artifact in artifacts_by_key.get(key, []):
+ if artifact.get("artifact_ref_ready"):
+ continue
+ target = str(artifact.get("path") or artifact.get("source_pattern") or key)
+ artifact_status = str(artifact.get("status", "missing"))
+ role = str(artifact.get("artifact_role", "supporting-evidence"))
+ if artifact.get("concrete_reference_required"):
+ action = "Replace the glob with concrete files, then reference the generated SHA-256 digests."
+ elif artifact.get("submission_ref_required"):
+ action = "Create the required submission artifact or update artifact_refs to a concrete existing aggregate file."
+ else:
+ action = "Add the supporting artifact if it is needed for reviewer audit."
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="artifact",
+ target=target,
+ status="blocked",
+ blocking_reason=f"{role} artifact is {artifact_status}.",
+ next_action=action,
+ source="artifact_checklist",
+ )
+ )
+
+ for source in sources_by_key.get(key, []):
+ if source.get("status") == "pass":
+ continue
+ field = str(source.get("field") or source.get("label") or key)
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="source-check",
+ target=field,
+ status="blocked",
+ blocking_reason=f"Current value {source.get('actual')!r} does not satisfy {source.get('expected')!r}.",
+ next_action=str(source.get("next_action") or "Collect the required source evidence."),
+ source="source_checklist",
+ )
+ )
+
+ for key in unknown_keys:
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="unknown-key",
+ target=key,
+ status="blocked",
+ blocking_reason="Requested evidence key is not present in the operator checklist.",
+ next_action="Use one of the evidence keys listed by world-class-intake.",
+ source="unknown_evidence_keys",
+ )
+ )
+
+ return sort_repair_rows(rows)
+
+
+def build_preflight_repair_checklist(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Return repair rows from preflight precheck and source-check blockers."""
+ rows: list[dict[str, Any]] = []
+ for item in items:
+ key = str(item.get("evidence_key", "")).strip()
+ if not key:
+ continue
+ for precheck in item.get("prechecks", []):
+ if not isinstance(precheck, dict):
+ continue
+ if precheck.get("required") is not True or precheck.get("status") == "pass":
+ continue
+ target = str(precheck.get("key") or precheck.get("label") or key)
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="precheck",
+ target=target,
+ status="blocked",
+ blocking_reason=(
+ f"Required {precheck.get('kind', 'precheck')} precheck is "
+ f"{precheck.get('status', 'unknown')}."
+ ),
+ next_action=str(precheck.get("next_action") or "Complete the required preflight action."),
+ source="prechecks",
+ )
+ )
+ for source in item.get("source_checklist", []):
+ if not isinstance(source, dict):
+ continue
+ if source.get("status") == "pass":
+ continue
+ target = str(source.get("field") or source.get("label") or key)
+ rows.append(
+ _repair_row(
+ evidence_key=key,
+ repair_type="source-check",
+ target=target,
+ status="blocked",
+ blocking_reason=(
+ f"Current value {source.get('actual')!r} does not satisfy "
+ f"{source.get('expected')!r}."
+ ),
+ next_action=str(source.get("next_action") or "Collect the required source evidence."),
+ source="source_checklist",
+ )
+ )
+ return sort_repair_rows(rows)
+
+
+def summarize_repair_checklist(rows: list[dict[str, Any]]) -> dict[str, Any]:
+ blocked_count = sum(1 for row in rows if row.get("status") != "ready")
+ phase_counts: dict[str, int] = {}
+ for row in rows:
+ phase = str(row.get("phase", "repair"))
+ phase_counts[phase] = phase_counts.get(phase, 0) + 1
+ next_row = sort_repair_rows(rows)[0] if rows else {}
+ return {
+ "repair_checklist_count": len(rows),
+ "repair_blocked_count": blocked_count,
+ "repair_ready_count": len(rows) - blocked_count,
+ "repair_phase_counts": phase_counts,
+ "next_repair_action_id": next_row.get("action_id", ""),
+ "next_repair_phase": next_row.get("phase", ""),
+ "next_repair_owner": next_row.get("owner", ""),
+ "next_repair_command": next_row.get("verification_command", ""),
+ "repair_counts_as_completion": False,
+ }
diff --git a/scripts/world_class_source_checks.py b/scripts/world_class_source_checks.py
new file mode 100644
index 00000000..a05580b2
--- /dev/null
+++ b/scripts/world_class_source_checks.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""Shared source-evidence checks for world-class evidence workflows."""
+
+from typing import Any
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Imported by world-class evidence reports to keep source-evidence readiness checks consistent."
+
+
+SOURCE_CHECK_SPECS = {
+ "provider-holdout": [
+ ("Provider model run", "model_executed_count", ">0", "Run provider-backed output-exec with real credentials."),
+ ("Timing observed", "timing_observed_count", ">0", "Provider execution should record timing metadata."),
+ ("Token usage observed", "token_observed_count", ">0", "Provider execution should return non-estimated token usage."),
+ ],
+ "human-adjudication": [
+ ("Review pairs exist", "pair_count", ">0", "Generate the blind A/B review pack."),
+ ("No pending decisions", "pending_count", "==0", "Record a reviewer choice and reason for every pair."),
+ ("Judgments complete", "judgment_count", "==pair_count", "Every pair needs one valid human judgment."),
+ ("No invalid decisions", "invalid_decision_count", "==0", "Fix malformed winner/confidence entries."),
+ ("Reviewer metadata", "reviewer_metadata_present", "true", "Record reviewer and reviewed_at before adjudication can count."),
+ ("Reason required", "reason_required", "true", "Keep reason mandatory for every imported or direct reviewer decision."),
+ ("Blind review attested", "blind_review_attested", "true", "Set reviewer_attestation only after choices are completed before opening the answer key."),
+ ("Raw content attested", "raw_content_excluded_attested", "true", "Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content."),
+ ("Raw content blocked", "raw_content_allowed", "false", "Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages."),
+ ("Human evidence ready", "ready_for_human_evidence", "true", "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."),
+ ],
+ "native-permission-enforcement": [
+ ("Native enforcement", "native_enforcement_count", ">0", "Collect real target-client or external runtime guard proof."),
+ ("Probe failures", "failure_count", "==0", "Runtime permission probes must stay clean."),
+ ("Installer support", "installer_enforcement_ready", "true", "Installer enforcement is supporting evidence, not native proof."),
+ ],
+ "native-client-telemetry": [
+ ("External events", "external_source_events", ">0", "Import at least one metadata-only event from a real client."),
+ ("Adoption sample", "adoption_sample_count", ">0", "Telemetry must include adoption outcome evidence."),
+ ("Raw content blocked", "raw_content_allowed", "false", "Telemetry must stay metadata-only."),
+ ],
+}
+
+
+def source_check_passed(actual: Any, expected: str, observed_state: dict[str, Any]) -> bool:
+ if expected == ">0":
+ return isinstance(actual, (int, float)) and actual > 0
+ if expected == "==0":
+ return actual == 0
+ if expected == "true":
+ return actual is True
+ if expected == "false":
+ return actual is False
+ if expected == "==pair_count":
+ return actual == observed_state.get("pair_count") and isinstance(actual, (int, float)) and actual > 0
+ return False
+
+
+def item_evidence_key(item: dict[str, Any]) -> str:
+ return str(item.get("evidence_key") or item.get("key") or "")
+
+
+def item_observed_state(item: dict[str, Any]) -> dict[str, Any]:
+ observed = item.get("observed_state", {})
+ return observed if isinstance(observed, dict) else {}
+
+
+def item_source_accepted(item: dict[str, Any], observed_state: dict[str, Any]) -> bool:
+ return item.get("source_accepted") is True or observed_state.get("accepted") is True
+
+
+def build_source_checklist(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ rows: list[dict[str, Any]] = []
+ for item in items:
+ key = item_evidence_key(item)
+ observed_state = item_observed_state(item)
+ for label, field, expected, next_action in SOURCE_CHECK_SPECS.get(key, []):
+ actual = observed_state.get(field)
+ passed = source_check_passed(actual, expected, observed_state)
+ rows.append(
+ {
+ "evidence_key": key,
+ "label": label,
+ "field": field,
+ "expected": expected,
+ "actual": actual,
+ "status": "pass" if passed else "blocked",
+ "source_accepted": item_source_accepted(item, observed_state),
+ "next_action": next_action,
+ }
+ )
+ return rows
+
+
+def summarize_source_checklist(rows: list[dict[str, Any]]) -> dict[str, int]:
+ pass_count = sum(1 for item in rows if item.get("status") == "pass")
+ blocked_count = sum(1 for item in rows if item.get("status") != "pass")
+ return {
+ "source_check_count": len(rows),
+ "source_pass_count": pass_count,
+ "source_blocked_count": blocked_count,
+ }
diff --git a/scripts/world_class_submission_kit_rendering.py b/scripts/world_class_submission_kit_rendering.py
new file mode 100644
index 00000000..ae0da581
--- /dev/null
+++ b/scripts/world_class_submission_kit_rendering.py
@@ -0,0 +1,576 @@
+#!/usr/bin/env python3
+"""Render Markdown and HTML artifacts for world-class submission kits."""
+
+from typing import Any
+
+from html_rendering import html_text
+
+
+SCRIPT_INTERFACE = "internal-module"
+SCRIPT_INTERFACE_REASON = "Shared renderer for world-class submission kit Markdown and HTML artifacts."
+
+
+def render_readme(report: dict[str, Any]) -> str:
+ commands = report["commands"]
+ lines = [
+ "# World-Class Evidence Submission Kit",
+ "",
+ f"Generated at: `{report['generated_at']}`",
+ "",
+ "This kit contains editable drafts for human and external evidence packets. Drafts are not accepted evidence.",
+ "",
+ "## Workflow",
+ "",
+ "1. Run the real provider, human review, native permission, or native client telemetry work first.",
+ "2. Edit the matching JSON draft with only aggregate artifact references and provenance metadata.",
+ "3. Set `template_only` to `false` only after real evidence exists.",
+ "4. Set attestation booleans truthfully; do not include credentials, raw prompts, raw outputs, transcripts, notes, or private user content.",
+ "5. Validate the packet before asking the ledger reviewer to set `attestation.ledger_reviewer`, `attestation.ledger_reviewed_at`, and `attestation.ledger_reviewer_approved` truthfully.",
+ "6. Optional artifact prefill only inserts SHA-256 digests for current local aggregate artifacts; it does not mark a draft as real evidence.",
+ "",
+ "## Commands",
+ "",
+ f"- validate intake: `{commands['validate_intake']}`",
+ f"- review submission: `{commands['review_submission']}`",
+ f"- refresh ledger: `{commands['refresh_ledger']}`",
+ f"- guard public claims: `{commands['guard_claim']}`",
+ "",
+ "## Operator Handoff",
+ "",
+ "Follow these steps in order when handing the kit from operator to reviewer. Handoff rows are procedural and never count as completion evidence.",
+ "",
+ "| Step | Status | Command | Completion signal |",
+ "| --- | --- | --- | --- |",
+ ]
+ for item in report.get("operator_handoff", []):
+ command = item.get("command") or "manual"
+ lines.append(
+ f"| `{item['step_id']}` | `{item['status']}` | `{command}` | {item['completion_signal']} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Phase Queue",
+ "",
+ "This queue groups repair rows by execution phase so operators can clear access, artifact, and source blockers in order. Queue rows are procedural guidance only.",
+ "",
+ "| Phase | Status | Rows | Owners | Next action | Verify |",
+ "| --- | --- | ---: | --- | --- | --- |",
+ ]
+ )
+ phase_queue = report.get("phase_queue", [])
+ if phase_queue:
+ for item in phase_queue:
+ owners = ", ".join(str(owner) for owner in item.get("owners", [])) or "n/a"
+ lines.append(
+ f"| `{item['phase']}` | `{item['status']}` | `{item['row_count']}` | {owners} | "
+ f"{item['next_action']} | `{item['verification_command']}` |"
+ )
+ else:
+ lines.append("| `all` | `ready` | `0` | n/a | No phase blockers were generated. | `n/a` |")
+ lines.extend(
+ [
+ "",
+ "## Drafts",
+ "",
+ "| Evidence | Draft | Status | Prefilled refs |",
+ "| --- | --- | --- | ---: |",
+ ]
+ )
+ for item in report["files"]:
+ lines.append(
+ f"| `{item['evidence_key']}` | `{item['output_path']}` | `{item['status']}` | `{item.get('prefilled_artifact_ref_count', 0)}` |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Evidence Matrix",
+ "",
+ "This matrix combines draft, artifact, and source-check readiness into one operator action list. Matrix rows are guidance only; they do not count as completion evidence.",
+ "",
+ "| Evidence | Stage | Draft | Submission refs | Supporting assets | Source checks | Next action |",
+ "| --- | --- | --- | ---: | ---: | ---: | --- |",
+ ]
+ )
+ for item in report.get("evidence_matrix", []):
+ lines.append(
+ f"| `{item['evidence_key']}` | `{item['stage']}` | `{item['draft_status']}` | "
+ f"`{item.get('submission_ref_ready_count', 0)}/{item.get('submission_ref_total_count', 0)}` | "
+ f"`{item.get('supporting_artifact_ready_count', 0)}/{item.get('supporting_artifact_total_count', 0)}` | "
+ f"`{item['source_pass_count']}/{item['source_check_count']}` | {item['next_action']} |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Repair Checklist",
+ "",
+ "This checklist turns every draft, artifact, and source blocker into a concrete repair row. Repair rows are procedural guidance and do not count as completion evidence.",
+ "",
+ "| Evidence | Type | Target | Status | Next action |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ repair_rows = report.get("repair_checklist", [])
+ if repair_rows:
+ for item in repair_rows:
+ lines.append(
+ f"| `{item['evidence_key']}` | `{item['repair_type']}` | `{item['target']}` | "
+ f"`{item['status']}` | {item['next_action']} |"
+ )
+ else:
+ lines.append("| `all` | `none` | `n/a` | `ready` | No repair rows were generated. |")
+ lines.extend(["", "## Execution Runbook", ""])
+ for item in report.get("evidence_items", []):
+ must_collect = item.get("must_collect", {}) if isinstance(item.get("must_collect", {}), dict) else {}
+ runbook = must_collect.get("runbook", [])
+ lines.extend(["", f"### {item.get('label', item.get('evidence_key', 'Evidence'))}", ""])
+ if runbook:
+ for step in runbook:
+ lines.append(f"- `{step}`" if str(step).startswith("python3 ") or "=" in str(step) else f"- {step}")
+ else:
+ lines.append("- No source runbook listed.")
+ lines.extend(
+ [
+ "",
+ "## Artifact Checklist",
+ "",
+ "Use these paths and SHA-256 digests when filling `artifact_refs`. Glob patterns are expanded into concrete files; submissions must reference concrete paths, not globs.",
+ "",
+ "| Evidence | Role | Path | Status | SHA-256 |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for item in report.get("artifact_checklist", []):
+ digest = item.get("sha256") or "n/a"
+ lines.append(
+ f"| `{item['evidence_key']}` | `{item.get('artifact_role', 'supporting-evidence')}` | "
+ f"`{item['path']}` | `{item['status']}` | `{digest}` |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Source Evidence Snapshot",
+ "",
+ "These checks explain why a draft is not ready for ledger acceptance yet. They mirror current aggregate reports and do not accept evidence by themselves.",
+ "",
+ "| Evidence | Check | Current | Expected | Status |",
+ "| --- | --- | --- | --- | --- |",
+ ]
+ )
+ for item in report.get("source_checklist", []):
+ lines.append(
+ f"| `{item['evidence_key']}` | {item['label']} | `{item['actual']}` | `{item['expected']}` | `{item['status']}` |"
+ )
+ lines.extend(
+ [
+ "",
+ "## Anti-Overclaim",
+ "",
+ "- This kit never marks ledger evidence as accepted.",
+ "- Planned work, metadata fallback, pending review, and local command-runner output remain non-evidence.",
+ "- A valid intake packet means ready for ledger review, not world-class completion.",
+ ]
+ )
+ return "\n".join(lines).rstrip() + "\n"
+
+
+def render_html_list(values: list[Any], empty: str) -> str:
+ if not values:
+ return f"
{html_text(empty)}
"
+ return "".join(f"
{html_text(value)}
" for value in values)
+
+
+def render_html_commands(commands: dict[str, str]) -> str:
+ return "".join(
+ f"
Use this cockpit to prepare human and external evidence packets. Drafts are not accepted evidence, artifact prefill only inserts local SHA-256 digests, and this page never changes the ledger result.
+
{stat_html}
+
+
+
Workflow
Run the real provider, human review, native permission, or native client telemetry work first.
Edit the matching JSON draft with aggregate artifact references and provenance metadata.
Set template_only to false only after real evidence exists.
Use prefilled SHA-256 values as convenience data, not evidence acceptance.
Validate intake, ask the ledger reviewer to approve explicitly, refresh the ledger, then guard public claims.
+
+
+
Operator Handoff
These ordered steps make the operator-to-reviewer handoff auditable. They are procedural guardrails and never count as world-class evidence.
{handoff_html}
+
Phase Queue
Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later evidence work. Queue rows remain procedural and never count as accepted evidence.
{phase_queue_html}
+
Evidence Matrix
The matrix separates submission artifact_refs from supporting assets, then combines draft status, source checks, and the next operator action. It is guidance only and never counts as accepted evidence.
{matrix_html}
+
Repair Checklist
Each row names one concrete blocker and the next action required before ledger review. This checklist does not count as completion evidence.
{repair_html}
+
Drafts
{render_html_files(report['files'])}
+
Artifact Checklist
Rows marked submission-ref are the paths expected in artifact_refs. Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into the submission. Glob patterns are expanded for operator convenience only.
{artifact_html}
+
Source Evidence Snapshot
This section shows current aggregate source checks. It explains remaining blockers without changing the ledger.
{source_html}
+
Evidence Requirements
{evidence_html}
+
Safety Boundary
Drafts never count as accepted ledger evidence.
Valid intake means ready for ledger review, not world-class completion.
Do not include credentials, raw prompts, raw outputs, transcripts, notes, or private user content.
" in synthetic_html, synthetic_html
+ assert "pattern: Output Review Kit" in synthetic_html, synthetic_html
html = output_html.read_text(encoding="utf-8")
assert "Review Studio 2.0" in html, html[:400]
+ assert "
" in html, html
+ assert "pattern: # World-Class Evidence Ledger" in html, html
assert "审查闸门" in html, html[:1200]
assert "修复动作" in html, html[:3000]
- assert "当前没有 blocker 或 warning" in html, html[:9000]
+ assert "补足 output eval 覆盖、execution evidence、blind A/B 和 reviewer adjudication。" in html, html[:9000]
+ assert "resource governance governed" in html, html[:9000]
+ assert "reports/output_review_kit.html" in html, html[:9000]
+ assert "python3 scripts/adjudicate_output_review.py --write-template" in html, html[:9000]
+ assert "对保留的 warning 写入 reviewer、理由、范围和到期时间，或修掉 warning。" in html, html[:9000]
+ assert "补齐 provider、真人盲评、原生权限执行和真实客户端遥测证据" in html, html
+ assert "action-card warn with-evidence" in html, html
+ assert "action-evidence-panel" in html, html
+ assert "证据采集" in html, html
+ assert "action-evidence-grid" in html, html
+ assert "action-evidence-checks" in html, html
+ assert "阶段队列" in html, html
+ assert "action-phase-list" in html, html
+ assert "action-phase-row blocked" in html, html
+ assert "#20 · unblock-access" in html, html
+ assert "#40 · collect-source" in html, html
+ assert "action-repair-list" in html, html
+ assert "action-repair-row blocked" in html, html
+ assert "provider-api-key" in html, html
+ assert "model_executed_count" in html, html
+ assert "#20 · unblock-access · precheck" in html, html
+ assert "#40 · collect-source · source-check" in html, html
+ assert "operator with provider credentials" in html, html
+ assert "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions" in html, html
+ assert "action-command-list" in html, html
+ assert "action-runbook-list" in html, html
+ assert "采集契约" in html, html
+ assert "action-collection-grid" in html, html
+ assert "来源要求" in html, html
+ assert "通过条件" in html, html
+ assert "证据资产" in html, html
+ assert "隐私边界" in html, html
+ assert "世界证据" in html, html
+ assert "证据台账" in html, html
+ assert "证据入口" in html, html
+ assert "入口边界" in html, html
+ assert "声明守卫" in html, html
+ assert "声明边界" in html, html
+ assert "提交清单" in html, html
+ assert "world-intake-grid" in html, html
+ assert "操作命令" in html, html
+ assert "收集要求" in html, html
+ assert "通过条件" in html, html
+ assert "evidence/world_class/submissions/provider-holdout.json" in html, html
+ assert "evidence/world_class/templates/provider-holdout.intake.json" in html, html
+ assert "Native Client Telemetry" in html, html
+ assert "native-client-telemetry.json" in html, html
+ assert "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions" in html, html
+ assert "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions" in html, html
+ assert "资产角色" in html, html
+ assert "submission-ref" in html, html
+ assert "supporting-evidence" in html, html
+ assert "artifact_refs: true" in html, html
+ assert "intake 只校验证据包格式、来源、隐私和反过度声明" in html, html
+ assert "reports/world_class_evidence_intake.md" in html, html
+ assert "reports/world_class_operator_runbook.html" in html, html
+ assert "reports/world_class_claim_guard.md" in html, html
+ assert "日常运维" in html, html
+ assert "周度队列" in html, html
+ assert "Weekly Queue" in html, html
+ assert "英文完成断言、true 状态声明或中文完成态" in html, html
+ assert "world-evidence-grid" in html, html
+ assert "Provider Holdout" in html, html
+ assert "Native Permission Enforcement" in html, html
+ assert "提交态" in html, html
+ assert "status: missing" in html, html
+ assert "完成定义" in html, html
+ assert "证据来源" in html, html
+ assert "隐私约束" in html, html
+ assert "reports/output_execution_runs.json summary.model_executed_count > 0" in html, html
+ assert "计划、metadata fallback、待评审和本地命令不会被当成完成证据" in html, html
+ assert "执行步骤" in html, html
+ assert "output-exec --provider-runner openai" in html, html
+ assert "<redacted>" not in html and "" not in html, html
+ assert "world-runbook-panel" in html, html
+ assert "源证据检查" in html, html
+ assert "world-source-checks" in html, html
+ assert "Provider model run" in html, html
+ assert "model_executed_count: 10 / >0" in html, html
+ assert "Token usage observed" in html, html
+ assert "蓝图覆盖" in html, html
+ assert "Extension Track Count" in html, html
+ assert "Adaptive Extension Ready" in html, html
+ assert "
" in html, html
+ assert "Phase Queue" in html, html
+ assert "Coordination Plan" in html, html
+ assert "Release Gate" in html, html
+ assert "review-and-release-gate" in html, html
+ assert "blocked-until-evidence-accepted" in html, html
+ assert "Next Source Actions" in html, html
+ assert "Source Runbook" in html, html
+ assert "--provider-runner openai" in html, html
+ assert "--provider-runner deepseek" in html, html
+ assert "<redacted>" not in html and "" not in html, html
+ assert "OPENAI_API_KEY=<redacted>" not in html, html
+ assert "Source Evidence Snapshot" in html, html
+ assert "model_executed_count" in html, html
+ assert "model_executed_count: 10 / >0" in html, html
+ assert "raw_content_allowed: False / false" in html, html
+ assert "= 6, submitted_summary
+ assert submitted_summary["phase_queue_count"] == 2, submitted_summary
+ assert submitted_summary["phase_queue_blocked_count"] == 2, submitted_summary
+ assert submitted_summary["phase_queue_counts_as_completion"] is False, submitted_summary
+ assert submitted_summary["ready_to_claim_world_class"] is False, submitted_summary
+ submitted_provider = {item["evidence_key"]: item for item in submitted["items"]}["provider-holdout"]
+ assert submitted_provider["ledger_status"] == "accepted", submitted_provider
+ assert submitted_provider["intake_readiness"] == "ready-for-ledger-review", submitted_provider
+ assert submitted_provider["review_state"] == "accepted", submitted_provider
+ assert submitted_provider["source_accepted"] is True, submitted_provider
+ assert submitted_provider["blocked_source_check_count"] == 0, submitted_provider
+ assert submitted_provider["phase_queue_blocked_count"] == 0, submitted_provider
+ assert submitted_provider["next_source_actions"] == [], submitted_provider
+ assert "tests/tmp_world_class_operator_runbook/valid_submissions" in submitted_provider["commands"]["validate_intake"], submitted_provider
+ assert "tests/tmp_world_class_operator_runbook/valid_submissions" in submitted_provider["commands"]["review_queue"], submitted_provider
+ assert "tests/tmp_world_class_operator_runbook/valid_submissions" in submitted_provider["commands"]["refresh_ledger"], submitted_provider
+ print(json.dumps({"ok": True}, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/verify_world_class_preflight.py b/tests/verify_world_class_preflight.py
new file mode 100644
index 00000000..ebfd7daf
--- /dev/null
+++ b/tests/verify_world_class_preflight.py
@@ -0,0 +1,355 @@
+#!/usr/bin/env python3
+import json
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent.parent
+SCRIPT = ROOT / "scripts" / "render_world_class_preflight.py"
+CLI = ROOT / "scripts" / "yao.py"
+TMP = ROOT / "tests" / "tmp_world_class_preflight"
+
+
+def run_preflight(extra_env: dict[str, str] | None = None, *extra: str) -> subprocess.CompletedProcess[str]:
+ env = dict(os.environ)
+ env.pop("OPENAI_API_KEY", None)
+ env.pop("DEEPSEEK_API_KEY", None)
+ env.pop("YAO_OUTPUT_EVAL_MODEL", None)
+ if extra_env:
+ env.update(extra_env)
+ return subprocess.run(
+ [
+ sys.executable,
+ str(SCRIPT),
+ str(ROOT),
+ "--generated-at",
+ "2026-06-16",
+ *extra,
+ ],
+ cwd=ROOT,
+ capture_output=True,
+ text=True,
+ env=env,
+ check=True,
+ )
+
+
+def run_cli(*extra: str) -> subprocess.CompletedProcess[str]:
+ env = dict(os.environ)
+ env["YAO_CLI_TELEMETRY"] = "0"
+ env.pop("OPENAI_API_KEY", None)
+ env.pop("DEEPSEEK_API_KEY", None)
+ return subprocess.run(
+ [sys.executable, str(CLI), "world-class-preflight", str(ROOT), *extra],
+ cwd=ROOT,
+ capture_output=True,
+ text=True,
+ env=env,
+ check=True,
+ )
+
+
+def by_key(items: list[dict], key: str) -> dict:
+ return next(item for item in items if item.get("evidence_key") == key)
+
+
+def main() -> None:
+ shutil.rmtree(TMP, ignore_errors=True)
+ TMP.mkdir(parents=True, exist_ok=True)
+
+ output_json = TMP / "world_class_evidence_preflight.json"
+ output_md = TMP / "world_class_evidence_preflight.md"
+ output_html = TMP / "world_class_evidence_preflight.html"
+ proc = run_preflight(None, "--output-json", str(output_json), "--output-md", str(output_md), "--output-html", str(output_html))
+ payload = json.loads(proc.stdout)
+ assert payload["schema_version"] == "1.0", payload
+ assert payload["ok"] is True, payload
+ summary = payload["summary"]
+ assert summary["decision"] == "collection-preflight-blocked", summary
+ assert summary["ready_to_claim_world_class"] is False, summary
+ assert summary["preflight_counts_as_evidence"] is False, summary
+ assert summary["credential_value_exposed"] is False, summary
+ assert summary["evidence_item_count"] == 4, summary
+ assert summary["pending_count"] == 4, summary
+ assert summary["precheck_count"] >= 12, summary
+ assert summary["precheck_missing_count"] >= 1, summary
+ assert summary["precheck_external_required_count"] == 2, summary
+ assert summary["precheck_human_required_count"] == 1, summary
+ assert summary["source_check_count"] >= 13, summary
+ assert summary["source_pass_count"] + summary["source_blocked_count"] == summary["source_check_count"], summary
+ assert summary["source_blocked_count"] >= 6, summary
+ assert summary["repair_checklist_count"] == (
+ summary["source_blocked_count"]
+ + summary["precheck_missing_count"]
+ + summary["precheck_human_required_count"]
+ + summary["precheck_external_required_count"]
+ ), summary
+ assert summary["repair_blocked_count"] == summary["repair_checklist_count"], summary
+ assert summary["repair_ready_count"] == 0, summary
+ assert summary["repair_phase_counts"]["unblock-access"] >= 1, summary
+ assert summary["repair_phase_counts"]["collect-source"] >= 1, summary
+ assert summary["phase_queue_count"] == 2, summary
+ assert summary["phase_queue_blocked_count"] == 2, summary
+ assert summary["phase_queue_row_count"] == summary["repair_checklist_count"], summary
+ assert summary["phase_queue_next_phase"] == "unblock-access", summary
+ assert summary["phase_queue_next_action_id"] == "human-adjudication-precheck-human-reviewer", summary
+ assert summary["phase_queue_next_command"] == (
+ "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions"
+ ), summary
+ assert summary["phase_queue_counts_as_completion"] is False, summary
+ assert summary["next_repair_action_id"] == "human-adjudication-precheck-human-reviewer", summary
+ assert summary["next_repair_phase"] == "unblock-access", summary
+ assert summary["next_repair_owner"] == "human reviewer", summary
+ assert summary["next_repair_command"] == (
+ "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions"
+ ), summary
+ assert summary["repair_counts_as_completion"] is False, summary
+ assert payload["submissions"]["preflight_counts_submission_as_completion"] is False, payload
+ assert payload["submissions"]["drafts_count_as_evidence"] is False, payload
+ assert payload["submissions"]["submission_kit_command"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--output-dir evidence/world_class/submissions"
+ ), payload["submissions"]
+ assert payload["submissions"]["submission_kit_prefill_command"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--output-dir evidence/world_class/submissions --prefill-artifacts"
+ ), payload["submissions"]
+ assert payload["submissions"]["artifact_prefill_counts_as_evidence"] is False, payload
+ phase_queue = payload["phase_queue"]
+ assert [item["phase"] for item in phase_queue] == ["unblock-access", "collect-source"], phase_queue
+ assert all(item["counts_as_completion"] is False for item in phase_queue), phase_queue
+ assert phase_queue[0]["status"] == "blocked", phase_queue
+ assert phase_queue[0]["blocked_count"] >= 1, phase_queue
+ assert phase_queue[0]["next_action_id"] == "human-adjudication-precheck-human-reviewer", phase_queue
+ assert phase_queue[0]["verification_command"] == (
+ "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions"
+ ), phase_queue
+ assert "human reviewer" in phase_queue[0]["owners"], phase_queue
+ assert "human-adjudication" in phase_queue[0]["evidence_keys"], phase_queue
+ assert phase_queue[1]["phase"] == "collect-source", phase_queue
+ assert phase_queue[1]["row_count"] == summary["source_blocked_count"], phase_queue
+ role_contract = payload["submissions"]["artifact_role_contract"]
+ assert role_contract["role_source"] == "world-class-submission-kit", role_contract
+ assert role_contract["counts_as_evidence"] is False, role_contract
+ assert role_contract["artifact_prefill_counts_as_evidence"] is False, role_contract
+ assert role_contract["submission_ref_total_count"] >= 4, role_contract
+ assert role_contract["submission_ref_ready_count"] <= role_contract["submission_ref_total_count"], role_contract
+ assert role_contract["supporting_evidence_total_count"] >= 4, role_contract
+ roles = {item["role"]: item for item in role_contract["roles"]}
+ assert roles["submission-ref"]["copy_to_artifact_refs"] is True, roles
+ assert roles["supporting-evidence"]["copy_to_artifact_refs"] is False, roles
+ assert payload["artifacts"]["html"] == "reports/world_class_evidence_preflight.html", payload["artifacts"]
+ submission_commands = payload["submissions"]["commands"]
+ assert submission_commands["prepare_submission"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--output-dir evidence/world_class/submissions"
+ ), submission_commands
+ assert submission_commands["prepare_prefilled_submission"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--output-dir evidence/world_class/submissions --prefill-artifacts"
+ ), submission_commands
+ assert submission_commands["validate_intake"] == (
+ "python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions"
+ ), submission_commands
+ assert submission_commands["submission_review"] == (
+ "python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions"
+ ), submission_commands
+ assert submission_commands["refresh_ledger"] == (
+ "python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions"
+ ), submission_commands
+ assert submission_commands["guard_claim"] == "python3 scripts/yao.py world-class-claim-guard .", submission_commands
+
+ provider = by_key(payload["items"], "provider-holdout")
+ assert provider["status"] == "blocked", provider
+ assert provider["commands"]["prepare_submission"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--evidence-key provider-holdout --output-dir evidence/world_class/submissions"
+ ), provider
+ assert provider["commands"]["prepare_prefilled_submission"] == (
+ "python3 scripts/yao.py world-class-submission-kit . "
+ "--evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts"
+ ), provider
+ assert provider["submission_kit"]["drafts_count_as_evidence"] is False, provider
+ assert provider["submission_kit"]["artifact_prefill_counts_as_evidence"] is False, provider
+ assert provider["submission_kit"]["prefill_command"] == provider["commands"]["prepare_prefilled_submission"], provider
+ assert provider["submission_kit"]["output_dir"] == "evidence/world_class/submissions", provider
+ assert provider["submission_kit"]["draft_path"] == "evidence/world_class/submissions/provider-holdout.json", provider
+ provider_roles = provider["submission_kit"]["artifact_role_contract"]
+ assert provider_roles["submission_ref_total_count"] == 1, provider_roles
+ assert provider_roles["submission_ref_ready_count"] == 1, provider_roles
+ assert provider_roles["supporting_evidence_total_count"] >= 1, provider_roles
+ provider_checks = {item["key"]: item for item in provider["prechecks"]}
+ assert provider_checks["provider-api-key"]["status"] == "missing", provider_checks
+ assert provider_checks["provider-api-key"]["actual"] == "not-set", provider_checks
+ assert provider_checks["provider-api-key"]["secret_value_redacted"] is True, provider_checks
+ assert provider_checks["provider-api-key"]["env_any"] == ["OPENAI_API_KEY", "DEEPSEEK_API_KEY"], provider_checks
+ assert "sk-test-secret" not in proc.stdout, proc.stdout
+ assert "OPENAI_API_KEY" in proc.stdout, proc.stdout
+ assert "DEEPSEEK_API_KEY" in proc.stdout, proc.stdout
+ provider_repairs = {item["target"]: item for item in provider["repair_checklist"]}
+ provider_phases = {item["phase"]: item for item in provider["phase_queue"]}
+ assert set(provider_phases) == {"unblock-access"}, provider_phases
+ assert provider_phases["unblock-access"]["next_action_id"] == "provider-holdout-precheck-provider-api-key", provider_phases
+ assert provider_repairs["provider-api-key"]["repair_type"] == "precheck", provider_repairs
+ assert provider_repairs["provider-api-key"]["action_id"] == "provider-holdout-precheck-provider-api-key", provider_repairs
+ assert provider_repairs["provider-api-key"]["phase"] == "unblock-access", provider_repairs
+ assert provider_repairs["provider-api-key"]["priority"] == 20, provider_repairs
+ assert provider_repairs["provider-api-key"]["owner"] == "operator with provider credentials", provider_repairs
+ assert provider_repairs["provider-api-key"]["verification_command"].endswith(
+ "world-class-preflight . --submissions-dir evidence/world_class/submissions"
+ ), provider_repairs
+ assert provider_repairs["provider-api-key"]["counts_as_completion"] is False, provider_repairs
+ provider_source = {item["field"]: item for item in provider["source_checklist"]}
+ assert provider_source["model_executed_count"]["status"] == "pass", provider_source
+ assert provider_source["timing_observed_count"]["status"] == "pass", provider_source
+ assert provider_source["token_observed_count"]["status"] == "pass", provider_source
+
+ human = by_key(payload["items"], "human-adjudication")
+ assert human["status"] == "ready-for-human-review", human
+ human_checks = {item["key"]: item for item in human["prechecks"]}
+ assert human_checks["human-reviewer"]["status"] == "human-required", human_checks
+ assert "reviewer identity" in human["next_action"], human
+ assert any("Record a reviewer choice and reason" in row["next_action"] for row in human["source_checklist"]), human
+ assert any("required rationale" in item["next_action"] for item in human["prechecks"]), human
+ assert any("reviewed_at" in item["next_action"] for item in human["prechecks"]), human
+ human_repairs = {item["target"]: item for item in human["repair_checklist"]}
+ assert human_repairs["human-reviewer"]["repair_type"] == "precheck", human_repairs
+ assert human_repairs["human-reviewer"]["owner"] == "human reviewer", human_repairs
+ assert human_repairs["pending_count"]["repair_type"] == "source-check", human_repairs
+ assert "output-review" in human_repairs["pending_count"]["verification_command"], human_repairs
+
+ native = by_key(payload["items"], "native-permission-enforcement")
+ assert native["status"] == "blocked", native
+ assert any(item["status"] == "external-required" for item in native["prechecks"]), native
+
+ markdown = output_md.read_text(encoding="utf-8")
+ assert "World-Class Evidence Preflight" in markdown, markdown
+ assert "ready to claim world-class: `false`" in markdown, markdown
+ assert "preflight counts as evidence: `false`" in markdown, markdown
+ assert "credential value exposed: `false`" in markdown, markdown
+ assert "Submission Kit Handoff" in markdown, markdown
+ assert "Phase Queue" in markdown, markdown
+ assert "Phase queue rows group the same repair checklist into operator execution phases" in markdown, markdown
+ assert "phase queue: `2` blocked / `2` phases" in markdown, markdown
+ assert "next phase action: `human-adjudication-precheck-human-reviewer`" in markdown, markdown
+ assert "| Priority | Phase | Status | Rows | Owners | Evidence | Verify | Next action |" in markdown, markdown
+ assert "Repair Checklist" in markdown, markdown
+ assert "Repair rows convert preflight and source blockers into a prioritized operator queue" in markdown, markdown
+ assert "repair rows:" in markdown, markdown
+ assert "next repair action: `human-adjudication-precheck-human-reviewer`" in markdown, markdown
+ assert "| Priority | Phase | Owner | Evidence | Type | Target | Status | Verify | Next action |" in markdown, markdown
+ assert "`unblock-access`" in markdown, markdown
+ assert "operator with provider credentials" in markdown, markdown
+ assert "`provider-api-key`" in markdown, markdown
+ assert "Provider model run" in markdown, markdown
+ assert "world-class-submission-kit . --output-dir evidence/world_class/submissions" in markdown, markdown
+ assert "world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts" in markdown, markdown
+ assert "world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions" in markdown, markdown
+ assert "world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts" in markdown, markdown
+ assert "drafts count as evidence: `false`" in markdown, markdown
+ assert "artifact prefill counts as evidence: `false`" in markdown, markdown
+ assert "does not make a draft count as evidence" in markdown, markdown
+ assert "submission refs ready:" in markdown, markdown
+ assert "`submission-ref` rows are the only checklist rows expected in `artifact_refs`" in markdown, markdown
+ assert "| `supporting-evidence` | `false` |" in markdown, markdown
+ assert "values are never printed" in markdown, markdown
+ html = output_html.read_text(encoding="utf-8")
+ assert "World-Class Evidence Preflight" in html, html
+ assert "World-Class Evidence Preflight" in html, html
+ assert "Phase Queue" in html, html
+ assert "phase-row blocked" in html, html
+ assert "
Rows
" in html, html
+ assert "
Counts
" in html, html
+ assert "does not count as completion" in html, html
+ assert "Evidence Queue" in html, html
+ assert "Submission Kit" in html, html
+ assert "Repair Rows" in html, html
+ assert "Repairs" in html, html
+ assert "repair-row blocked" in html, html
+ assert "
Priority
" in html, html
+ assert "
Phase
" in html, html
+ assert "
Owner
" in html, html
+ assert "operator with provider credentials" in html, html
+ assert "world-class-preflight . --submissions-dir evidence/world_class/submissions" in html, html
+ assert "provider-api-key" in html, html
+ assert "Provider model run" in html, html
+ assert "world-class-submission-kit . --output-dir evidence/world_class/submissions" in html, html
+ assert "world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts" in html, html
+ assert "provider-holdout" in html, html
+ assert "Artifact prefill is convenience data only." in html, html
+ assert "artifact prefill counts as evidence" in html, html
+ assert "Artifact Roles" in html, html
+ assert "copy to artifact_refs: true" in html, html
+ assert "supporting-evidence" in html, html
+ assert "Source Checks" in html, html
+ assert "ready_to_claim_world_class" in html, html
+ assert "Environment variables are displayed only as set or not-set" in html, html
+
+ env_json = TMP / "preflight_with_env.json"
+ env_proc = run_preflight(
+ {"DEEPSEEK_API_KEY": "sk-test-secret", "YAO_OUTPUT_EVAL_MODEL": "deepseek-test"},
+ "--output-json",
+ str(env_json),
+ "--output-md",
+ str(TMP / "preflight_with_env.md"),
+ "--output-html",
+ str(TMP / "preflight_with_env.html"),
+ )
+ env_payload = json.loads(env_proc.stdout)
+ env_provider = by_key(env_payload["items"], "provider-holdout")
+ env_provider_checks = {item["key"]: item for item in env_provider["prechecks"]}
+ assert env_provider_checks["provider-api-key"]["status"] == "pass", env_provider_checks
+ assert env_provider_checks["provider-api-key"]["actual"] == "set", env_provider_checks
+ assert env_provider_checks["provider-api-key"]["set_envs"] == ["DEEPSEEK_API_KEY"], env_provider_checks
+ assert env_provider_checks["provider-model"]["status"] == "pass", env_provider_checks
+ assert "sk-test-secret" not in env_proc.stdout, env_proc.stdout
+ assert "sk-test-secret" not in (TMP / "preflight_with_env.html").read_text(encoding="utf-8"), env_payload
+ assert env_payload["summary"]["credential_value_exposed"] is False, env_payload
+ assert env_payload["summary"]["ready_to_claim_world_class"] is False, env_payload
+ assert env_payload["summary"]["repair_checklist_count"] == payload["summary"]["repair_checklist_count"] - 1, env_payload
+ assert env_payload["summary"]["phase_queue_row_count"] == payload["summary"]["phase_queue_row_count"] - 1, env_payload
+ assert env_payload["summary"]["phase_queue_count"] == 2, env_payload
+
+ spaced_dir = TMP / "submission kit spaced"
+ spaced_proc = run_preflight(
+ None,
+ "--submissions-dir",
+ str(spaced_dir),
+ "--output-json",
+ str(TMP / "preflight_spaced.json"),
+ "--output-md",
+ str(TMP / "preflight_spaced.md"),
+ "--output-html",
+ str(TMP / "preflight_spaced.html"),
+ )
+ spaced_payload = json.loads(spaced_proc.stdout)
+ quoted_spaced = "'tests/tmp_world_class_preflight/submission kit spaced'"
+ assert quoted_spaced in spaced_payload["submissions"]["commands"]["prepare_submission"], spaced_payload["submissions"]
+ assert quoted_spaced in spaced_payload["submissions"]["commands"]["prepare_prefilled_submission"], spaced_payload["submissions"]
+ assert quoted_spaced in by_key(spaced_payload["items"], "provider-holdout")["commands"]["prepare_submission"], spaced_payload["items"]
+ assert quoted_spaced in by_key(spaced_payload["items"], "provider-holdout")["commands"]["prepare_prefilled_submission"], spaced_payload["items"]
+
+ cli_proc = run_cli(
+ "--output-json",
+ str(TMP / "cli_preflight.json"),
+ "--output-md",
+ str(TMP / "cli_preflight.md"),
+ "--output-html",
+ str(TMP / "cli_preflight.html"),
+ "--generated-at",
+ "2026-06-16",
+ )
+ cli_payload = json.loads(cli_proc.stdout)
+ assert cli_payload["summary"]["decision"] == "collection-preflight-blocked", cli_payload
+ assert cli_payload["summary"]["preflight_counts_as_evidence"] is False, cli_payload
+ assert (TMP / "cli_preflight.md").exists(), cli_payload
+ assert (TMP / "cli_preflight.html").exists(), cli_payload
+
+ print(json.dumps({"ok": True}, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/verify_world_class_submission_kit.py b/tests/verify_world_class_submission_kit.py
new file mode 100644
index 00000000..50bffc1a
--- /dev/null
+++ b/tests/verify_world_class_submission_kit.py
@@ -0,0 +1,351 @@
+#!/usr/bin/env python3
+import json
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parent.parent
+INTAKE_SCRIPT = ROOT / "scripts" / "render_world_class_evidence_intake.py"
+KIT_SCRIPT = ROOT / "scripts" / "prepare_world_class_submission_kit.py"
+TMP = ROOT / "tests" / "tmp_world_class_submission_kit"
+
+
+def run_intake(*extra: str) -> dict:
+ extra_args = list(extra)
+ if "--output-json" not in extra_args:
+ extra_args.extend(["--output-json", str(TMP / "last_intake.json")])
+ if "--output-md" not in extra_args:
+ extra_args.extend(["--output-md", str(TMP / "last_intake.md")])
+ proc = subprocess.run(
+ [
+ sys.executable,
+ str(INTAKE_SCRIPT),
+ str(ROOT),
+ "--generated-at",
+ "2026-06-14",
+ *extra_args,
+ ],
+ cwd=ROOT,
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ return json.loads(proc.stdout)
+
+
+def run_kit(*extra: str) -> subprocess.CompletedProcess[str]:
+ return subprocess.run(
+ [
+ sys.executable,
+ str(KIT_SCRIPT),
+ str(ROOT),
+ "--generated-at",
+ "2026-06-14",
+ *extra,
+ ],
+ cwd=ROOT,
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+
+
+def assert_documented_submission_commands() -> None:
+ expected_fragments = [
+ 'SUBMISSIONS_DIR="${SUBMISSIONS_DIR:-evidence/world_class/submissions}"',
+ 'world-class-preflight . --submissions-dir "$SUBMISSIONS_DIR"',
+ 'world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR"',
+ 'world-class-submission-kit . --output-dir "$SUBMISSIONS_DIR" --prefill-artifacts',
+ 'world-class-intake . --submissions-dir "$SUBMISSIONS_DIR"',
+ 'world-class-submission-review . --submissions-dir "$SUBMISSIONS_DIR"',
+ 'world-class-ledger . --submissions-dir "$SUBMISSIONS_DIR"',
+ ]
+ for relative_path in ("README.md", "evidence/world_class/README.md"):
+ text = (ROOT / relative_path).read_text(encoding="utf-8")
+ for fragment in expected_fragments:
+ assert fragment in text, (relative_path, fragment)
+ assert "/tmp/yao-world-class-submission-kit" not in text, relative_path
+
+
+def main() -> None:
+ shutil.rmtree(TMP, ignore_errors=True)
+ TMP.mkdir(parents=True, exist_ok=True)
+ assert_documented_submission_commands()
+
+ kit_dir = TMP / "submission_kit"
+ kit_proc = run_kit("--output-dir", str(kit_dir), "--evidence-key", "provider-holdout")
+ kit_payload = json.loads(kit_proc.stdout)
+ assert kit_payload["ok"] is True, kit_payload
+ assert kit_payload["summary"]["decision"] == "submission-kit-ready", kit_payload["summary"]
+ assert kit_payload["summary"]["requested_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["written_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["artifact_checklist_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["artifact_ready_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["submission_ref_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["submission_ref_ready_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["supporting_artifact_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["supporting_artifact_ready_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["source_check_count"] == 3, kit_payload["summary"]
+ assert kit_payload["summary"]["source_pass_count"] == 3, kit_payload["summary"]
+ assert kit_payload["summary"]["source_blocked_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_collect_source_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_submission_ref_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_submission_ref_ready_count"] == 1, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_supporting_artifact_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_supporting_artifact_ready_count"] >= 1, kit_payload["summary"]
+ assert kit_payload["summary"]["evidence_matrix_counts_as_completion"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["repair_checklist_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["repair_blocked_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["repair_ready_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["repair_phase_counts"] == {}, kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_blocked_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_row_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_next_phase"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_next_action_id"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_next_command"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["phase_queue_counts_as_completion"] is False, kit_payload["summary"]
+ assert kit_payload["summary"]["next_repair_action_id"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["next_repair_phase"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["next_repair_owner"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["next_repair_command"] == "", kit_payload["summary"]
+ assert kit_payload["summary"]["repair_counts_as_completion"] is False, kit_payload["summary"]
+ assert kit_payload["summary"]["handoff_step_count"] == 7, kit_payload["summary"]
+ assert kit_payload["summary"]["handoff_blocked_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["handoff_fix_required_count"] == 0, kit_payload["summary"]
+ assert kit_payload["summary"]["handoff_counts_as_completion"] is False, kit_payload["summary"]
+ assert kit_payload["summary"]["drafts_count_as_evidence"] is False, kit_payload["summary"]
+ assert kit_payload["safety"]["template_only_drafts"] is True, kit_payload["safety"]
+ assert kit_payload["safety"]["raw_content_allowed"] is False, kit_payload["safety"]
+ assert kit_payload["safety"]["operator_handoff_counts_as_evidence"] is False, kit_payload["safety"]
+ assert "review_submission" in kit_payload["commands"], kit_payload["commands"]
+ assert kit_payload["files"][0]["output_path"].endswith(
+ f"tests/{TMP.name}/submission_kit/provider-holdout.json"
+ ), kit_payload["files"]
+
+ assert len(kit_payload["evidence_matrix"]) == 1, kit_payload["evidence_matrix"]
+ matrix_row = kit_payload["evidence_matrix"][0]
+ assert matrix_row["evidence_key"] == "provider-holdout", matrix_row
+ assert matrix_row["stage"] == "validate-packet", matrix_row
+ assert matrix_row["draft_status"] == "written", matrix_row
+ assert matrix_row["artifact_ready_count"] >= 1, matrix_row
+ assert matrix_row["artifact_total_count"] >= matrix_row["artifact_ready_count"], matrix_row
+ assert matrix_row["submission_ref_ready_count"] == 1, matrix_row
+ assert matrix_row["submission_ref_total_count"] == 1, matrix_row
+ assert matrix_row["supporting_artifact_ready_count"] >= 1, matrix_row
+ assert matrix_row["supporting_artifact_total_count"] >= matrix_row["supporting_artifact_ready_count"], matrix_row
+ assert matrix_row["source_pass_count"] == 3, matrix_row
+ assert matrix_row["source_check_count"] == 3, matrix_row
+ assert matrix_row["source_blocked_count"] == 0, matrix_row
+ assert matrix_row["counts_as_completion"] is False, matrix_row
+ assert "validate intake" in matrix_row["next_action"].lower() or "packet" in matrix_row["next_action"].lower(), matrix_row
+
+ repair_rows = {item["target"]: item for item in kit_payload["repair_checklist"]}
+ assert repair_rows == {}, repair_rows
+
+ assert kit_payload["phase_queue"] == [], kit_payload["phase_queue"]
+
+ handoff_steps = {item["step_id"]: item for item in kit_payload["operator_handoff"]}
+ assert list(handoff_steps) == [
+ "prepare-drafts",
+ "collect-source",
+ "edit-submission",
+ "validate-intake",
+ "review-submission",
+ "refresh-ledger",
+ "guard-claim",
+ ], handoff_steps
+ assert handoff_steps["prepare-drafts"]["status"] == "ready", handoff_steps
+ assert handoff_steps["collect-source"]["status"] == "ready", handoff_steps
+ assert handoff_steps["collect-source"]["counts_as_completion"] is False, handoff_steps
+ assert handoff_steps["collect-source"]["blocking_condition"] == "", handoff_steps
+ assert "world-class-submission-review" in handoff_steps["review-submission"]["command"], handoff_steps
+ assert all(item["counts_as_completion"] is False for item in kit_payload["operator_handoff"]), handoff_steps
+
+ artifact_rows = {item["path"]: item for item in kit_payload["artifact_checklist"]}
+ assert "reports/output_execution_runs.json" in artifact_rows, artifact_rows
+ assert artifact_rows["reports/output_execution_runs.json"]["artifact_ref_ready"] is True, artifact_rows
+ assert artifact_rows["reports/output_execution_runs.json"]["artifact_role"] == "submission-ref", artifact_rows
+ assert artifact_rows["reports/output_execution_runs.json"]["submission_ref_required"] is True, artifact_rows
+ assert len(artifact_rows["reports/output_execution_runs.json"]["sha256"]) == 64, artifact_rows
+ assert artifact_rows["reports/output_execution_runs.json"]["contains_raw_content"] is False, artifact_rows
+ assert artifact_rows["reports/output_execution_runs.md"]["artifact_role"] == "supporting-evidence", artifact_rows
+ assert artifact_rows["reports/output_execution_runs.md"]["submission_ref_required"] is False, artifact_rows
+
+ source_rows = {item["field"]: item for item in kit_payload["source_checklist"]}
+ assert source_rows["model_executed_count"]["status"] == "pass", source_rows
+ assert source_rows["model_executed_count"]["actual"] > 0, source_rows
+ assert source_rows["timing_observed_count"]["status"] == "pass", source_rows
+ assert source_rows["token_observed_count"]["status"] == "pass", source_rows
+
+ kit_draft = json.loads((kit_dir / "provider-holdout.json").read_text(encoding="utf-8"))
+ assert kit_draft["template_only"] is True, kit_draft
+ assert kit_draft["attestation"]["real_external_or_human_evidence"] is False, kit_draft
+ assert kit_draft["attestation"]["ledger_reviewer_approved"] is False, kit_draft
+ assert kit_draft["attestation"]["ledger_reviewer"] == "", kit_draft
+ assert kit_draft["attestation"]["ledger_reviewed_at"] == "", kit_draft
+ assert "sha256" not in kit_draft["artifact_refs"][0], kit_draft
+ assert kit_draft["provenance"]["run_command"] == (
+ "python3 scripts/yao.py output-exec --provider-runner --provider-model --timeout-seconds 60"
+ ), kit_draft
+ assert kit_draft["provenance"]["credential_env"] == "", kit_draft
+ assert "" not in json.dumps(kit_draft), kit_draft
+
+ kit_manifest = json.loads((kit_dir / "submission_manifest.json").read_text(encoding="utf-8"))
+ assert kit_manifest["summary"]["ledger_counts_submission_as_completion"] is False, kit_manifest["summary"]
+ assert kit_manifest["summary"]["artifact_prefill_enabled"] is False, kit_manifest["summary"]
+ assert kit_manifest["summary"]["artifact_ref_prefill_count"] == 0, kit_manifest["summary"]
+ assert kit_manifest["artifact_checklist"] == kit_payload["artifact_checklist"], kit_manifest["artifact_checklist"]
+ assert kit_manifest["source_checklist"] == kit_payload["source_checklist"], kit_manifest["source_checklist"]
+ assert kit_manifest["repair_checklist"] == kit_payload["repair_checklist"], kit_manifest["repair_checklist"]
+ assert kit_manifest["phase_queue"] == kit_payload["phase_queue"], kit_manifest["phase_queue"]
+ assert kit_manifest["artifacts"]["html"].endswith(
+ f"tests/{TMP.name}/submission_kit/index.html"
+ ), kit_manifest["artifacts"]
+
+ kit_readme = (kit_dir / "README.md").read_text(encoding="utf-8")
+ assert "Drafts are not accepted evidence." in kit_readme, kit_readme
+ assert "Execution Runbook" in kit_readme, kit_readme
+ assert "--provider-runner openai" in kit_readme, kit_readme
+ assert "--provider-runner deepseek" in kit_readme, kit_readme
+ assert "" not in kit_readme, kit_readme
+ assert "validate intake" in kit_readme, kit_readme
+ assert "Artifact Checklist" in kit_readme, kit_readme
+ assert "Evidence Matrix" in kit_readme, kit_readme
+ assert "Phase Queue" in kit_readme, kit_readme
+ assert "This queue groups repair rows by execution phase" in kit_readme, kit_readme
+ assert "Repair Checklist" in kit_readme, kit_readme
+ assert "Operator Handoff" in kit_readme, kit_readme
+ assert "Handoff rows are procedural" in kit_readme, kit_readme
+ assert "ledger_reviewed_at" in kit_readme, kit_readme
+ assert "`review-submission`" in kit_readme, kit_readme
+ assert "Submission refs" in kit_readme, kit_readme
+ assert "Supporting assets" in kit_readme, kit_readme
+ assert "`submission-ref`" in kit_readme, kit_readme
+ assert "`supporting-evidence`" in kit_readme, kit_readme
+ assert "`collect-source`" in kit_readme, kit_readme
+ assert "Queue rows are procedural guidance only" in kit_readme, kit_readme
+ assert "Matrix rows are guidance only" in kit_readme, kit_readme
+ assert "Repair rows are procedural guidance and do not count as completion evidence." in kit_readme, kit_readme
+ assert "Source Evidence Snapshot" in kit_readme, kit_readme
+ assert "Provider model run" in kit_readme, kit_readme
+ assert "reports/output_execution_runs.json" in kit_readme, kit_readme
+ assert "Provider model run" in kit_readme, kit_readme
+
+ kit_html = (kit_dir / "index.html").read_text(encoding="utf-8")
+ assert "World-Class Evidence Submission Kit" in kit_html, kit_html
+ assert "Drafts are not accepted evidence" in kit_html, kit_html
+ assert "provider-holdout" in kit_html, kit_html
+ assert "Artifact Checklist" in kit_html, kit_html
+ assert "Evidence Matrix" in kit_html, kit_html
+ assert "Phase Queue" in kit_html, kit_html
+ assert "No phase queue rows were generated" in kit_html, kit_html
+ assert "Repair Checklist" in kit_html, kit_html
+ assert "No repair rows were generated" in kit_html, kit_html
+ assert "collect-source" in kit_html, kit_html
+ assert "operator with provider credentials" in kit_html, kit_html
+ assert "--provider-runner openai" in kit_html, kit_html
+ assert "--provider-runner deepseek" in kit_html, kit_html
+ assert "Operator Handoff" in kit_html, kit_html
+ assert "handoff-card ready" in kit_html, kit_html
+ assert "does not count as completion" in kit_html, kit_html
+ assert "matrix-card validate-packet" in kit_html, kit_html
+ assert (
+ f"