diff --git a/reports/review-studio.html b/reports/review-studio.html index 34c67e3..f4adf05 100644 --- a/reports/review-studio.html +++ b/reports/review-studio.html @@ -760,7 +760,7 @@

修复动作

-
关注

输出实验

补足 output eval 覆盖、execution evidence、blind A/B 和 reviewer adjudication。

没有输出质量和人工盲评证据时，Skill 只能证明会触发，不能证明输出真的更好且经得起审查。
修复位置
evals/output/cases.jsonl + reports/output_quality_scorecard.md + reports/output_review_kit.html + reports/output_review_adjudication.md
验证命令
python3 scripts/adjudicate_output_review.py --write-template && python3 scripts/yao.py output-review
关注

人工批准

对保留的 warning 写入 reviewer、理由、范围和到期时间，或修掉 warning。

warning 可以被接受，但必须可审计、会过期，并且不能掩盖 blocker。
修复位置
reports/review_waivers.md
验证命令
python3 scripts/render_review_waivers.py .
关注

世界证据

补齐 provider、真人盲评、原生权限执行和真实客户端遥测证据，或明确本次发布不声明 world-class 完成。

世界级结论必须来自已接受的外部/人工证据；计划、metadata fallback、待评审和本地命令都不能替代完成证据。
修复位置
reports/world_class_operator_runbook.html + reports/world_class_evidence_ledger.md + reports/world_class_evidence_intake.md + reports/world_class_submission_review.md
验证命令
python3 scripts/yao.py world-class-runbook . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py review-studio .

证据采集

以下条目仍需真实外部或人工证据；提交文件、校验命令和阻断检查必须同时闭环。

pending · external

Provider Holdout

model-executed 0; token-observed 0

提交
evidence/world_class/submissions/provider-holdout.json
模板
evidence/world_class/templates/provider-holdout.intake.json
阻断
2 blocked / 1 pass
下一步
Run provider-backed holdout cases with real credentials and commit only aggregate evidence.
阻断检查
  • Provider model runmodel_executed_count: 0 / >0Run provider-backed output-exec with real credentials.
  • Token usage observedtoken_observed_count: 0 / >0Provider execution should return non-estimated token usage.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. Set OPENAI_API_KEY in the operator shell before running provider evidence; never commit or print the value.
  2. export YAO_OUTPUT_EVAL_MODEL=${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini}
  3. python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60
采集契约
来源要求
  • provider-backed model run
  • observed timing
  • observed token metadata
通过条件
  • reports/output_execution_runs.json summary.model_executed_count > 0
  • reports/output_execution_runs.json summary.timing_observed_count > 0
  • reports/output_execution_runs.json summary.token_observed_count > 0
  • reports/skill_os2_audit.json item provider-holdout status becomes pass
证据资产
  • reports/output_execution_runs.json
  • reports/output_execution_runs.md
  • reports/skill_os2_audit.json
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/provider-holdout.intake.json
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref1 / 1 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Do not commit provider credentials or environment dumps.
  • The output execution report records output hashes and aggregate run metadata, not raw provider prompts.
pending · human

Human Adjudication

0/5 decisions; pending 5

提交
evidence/world_class/submissions/human-adjudication.json
模板
evidence/world_class/templates/human-adjudication.intake.json
阻断
4 blocked / 4 pass
下一步
Record real A/B choices in the decision template, then regenerate adjudication.
阻断检查
  • No pending decisionspending_count: 5 / ==0Record a reviewer choice and reason for every pair.
  • Judgments completejudgment_count: 0 / ==pair_countEvery pair needs one valid human judgment.
  • Reviewer metadatareviewer_metadata_present: False / trueRecord reviewer and reviewed_at before adjudication can count.
  • Human evidence readyready_for_human_evidence: False / trueComplete all reviewer decisions with reviewer metadata and rationale.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. python3 scripts/yao.py output-review-kit --write-template
  2. Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
  3. python3 scripts/adjudicate_output_review.py --write-template
采集契约
来源要求
  • real reviewer identity
  • blind A/B decisions
  • answer key unopened until decisions exist
通过条件
  • reports/output_review_adjudication.json summary.pending_count == 0
  • reports/output_review_adjudication.json summary.judgment_count == summary.pair_count
  • reports/output_review_adjudication.json summary.invalid_decision_count == 0
  • reports/output_review_adjudication.json summary.reviewer_metadata_present is true
  • reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text
证据资产
  • reports/output_blind_review_pack.md
  • reports/output_review_kit.md
  • reports/output_review_decisions.json
  • reports/output_review_adjudication.json
  • reports/output_review_adjudication.md
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence8 / 8 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Reviewer decisions should not include raw user data or private customer detail.
  • Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.
  • The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.
  • The adjudication evidence stores prompt_sha256 instead of raw prompt text.
pending · external

Native Permission Enforcement

native-enforced targets 0; installer-enforced targets 4

提交
evidence/world_class/submissions/native-permission-enforcement.json
模板
evidence/world_class/templates/native-permission-enforcement.intake.json
阻断
1 blocked / 2 pass
下一步
Integrate a real target-client or external installer runtime guard before claiming native permission enforcement.
阻断检查
  • Native enforcementnative_enforcement_count: 0 / >0Collect real target-client or external runtime guard proof.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
  2. Update the generated target adapter only when the guard is actually enforced by that target.
  3. python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
采集契约
来源要求
  • real target client or external installer runtime guard
  • native enforcement flag or externally accepted guard proof
  • residual risk retained for fallback targets
通过条件
  • reports/runtime_permission_probes.json summary.native_enforcement_count > 0
  • reports/runtime_permission_probes.json summary.failure_count == 0
  • reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
  • reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
证据资产
  • dist/targets/*/adapter.json
  • reports/runtime_permission_probes.json
  • reports/runtime_permission_probes.md
  • reports/install_simulation.json
  • reports/install_simulation.md
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence11 / 11 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Do not mark native_enforcement true for metadata-only fallbacks.
  • Keep residual risks visible for targets that still rely on operator enforcement.
pending · external

Native Client Telemetry

external source events 0; adoption samples 1

提交
evidence/world_class/submissions/native-client-telemetry.json
模板
evidence/world_class/templates/native-client-telemetry.intake.json
阻断
1 blocked / 2 pass
下一步
Install a real client against the native host and import production metadata-only events.
阻断检查
  • External eventsexternal_source_events: 0 / >0Import at least one metadata-only event from a real client.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
  2. Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
  3. python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
采集契约
来源要求
  • real external client source
  • metadata-only event
  • local-first import path
通过条件
  • reports/adoption_drift_report.json summary.source_types.external > 0
  • reports/adoption_drift_report.json summary.adoption_sample_count > 0
  • reports/skill_os2_audit.json item native-client-telemetry status becomes pass
证据资产
  • reports/adoption_drift_report.json
  • reports/adoption_drift_report.md
  • reports/telemetry_hook_recipes.json
  • scripts/telemetry_native_host.py
  • evidence/world_class/intake.schema.json
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Telemetry must remain metadata-only and local-first.
  • Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
+
关注

输出实验

补足 output eval 覆盖、execution evidence、blind A/B 和 reviewer adjudication。

没有输出质量和人工盲评证据时，Skill 只能证明会触发，不能证明输出真的更好且经得起审查。
修复位置
evals/output/cases.jsonl + reports/output_quality_scorecard.md + reports/output_review_kit.html + reports/output_review_adjudication.md
验证命令
python3 scripts/adjudicate_output_review.py --write-template && python3 scripts/yao.py output-review
关注

人工批准

对保留的 warning 写入 reviewer、理由、范围和到期时间，或修掉 warning。

warning 可以被接受，但必须可审计、会过期，并且不能掩盖 blocker。
修复位置
reports/review_waivers.md
验证命令
python3 scripts/render_review_waivers.py .
关注

世界证据

补齐 provider、真人盲评、原生权限执行和真实客户端遥测证据，或明确本次发布不声明 world-class 完成。

世界级结论必须来自已接受的外部/人工证据；计划、metadata fallback、待评审和本地命令都不能替代完成证据。
修复位置
reports/world_class_operator_runbook.html + reports/world_class_evidence_ledger.md + reports/world_class_evidence_intake.md + reports/world_class_submission_review.md
验证命令
python3 scripts/yao.py world-class-runbook . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py review-studio .

证据采集

以下条目仍需真实外部或人工证据；提交文件、校验命令和阻断检查必须同时闭环。

pending · external

Provider Holdout

model-executed 0; token-observed 0

提交
evidence/world_class/submissions/provider-holdout.json
模板
evidence/world_class/templates/provider-holdout.intake.json
阻断
2 blocked / 1 pass
下一步
Run provider-backed holdout cases with real credentials and commit only aggregate evidence.
阻断检查
  • Provider model runmodel_executed_count: 0 / >0Run provider-backed output-exec with real credentials.
  • Token usage observedtoken_observed_count: 0 / >0Provider execution should return non-estimated token usage.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. Set OPENAI_API_KEY in the operator shell before running provider evidence; never commit or print the value.
  2. export YAO_OUTPUT_EVAL_MODEL=${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini}
  3. python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60
采集契约
来源要求
  • provider-backed model run
  • observed timing
  • observed token metadata
通过条件
  • reports/output_execution_runs.json summary.model_executed_count > 0
  • reports/output_execution_runs.json summary.timing_observed_count > 0
  • reports/output_execution_runs.json summary.token_observed_count > 0
  • reports/skill_os2_audit.json item provider-holdout status becomes pass
证据资产
  • reports/output_execution_runs.json
  • reports/output_execution_runs.md
  • reports/skill_os2_audit.json
  • evidence/world_class/intake.schema.json
  • evidence/world_class/templates/provider-holdout.intake.json
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref1 / 1 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Do not commit provider credentials or environment dumps.
  • The output execution report records output hashes and aggregate run metadata, not raw provider prompts.
pending · human

Human Adjudication

0/5 decisions; pending 5

提交
evidence/world_class/submissions/human-adjudication.json
模板
evidence/world_class/templates/human-adjudication.intake.json
阻断
4 blocked / 4 pass
下一步
Record real A/B choices in the decision template, then regenerate adjudication.
阻断检查
  • No pending decisionspending_count: 5 / ==0Record a reviewer choice and reason for every pair.
  • Judgments completejudgment_count: 0 / ==pair_countEvery pair needs one valid human judgment.
  • Reviewer metadatareviewer_metadata_present: False / trueRecord reviewer and reviewed_at before adjudication can count.
  • Human evidence readyready_for_human_evidence: False / trueComplete all reviewer decisions with reviewer metadata and rationale.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. python3 scripts/yao.py output-review-kit --write-template
  2. Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
  3. python3 scripts/adjudicate_output_review.py --write-template
采集契约
来源要求
  • real reviewer identity
  • blind A/B decisions
  • answer key unopened until decisions exist
通过条件
  • reports/output_review_adjudication.json summary.pending_count == 0
  • reports/output_review_adjudication.json summary.judgment_count == summary.pair_count
  • reports/output_review_adjudication.json summary.invalid_decision_count == 0
  • reports/output_review_adjudication.json summary.reviewer_metadata_present is true
  • reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text
证据资产
  • reports/output_blind_review_pack.md
  • reports/output_review_kit.md
  • reports/output_review_decisions.json
  • reports/output_review_adjudication.json
  • reports/output_review_adjudication.md
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence8 / 8 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Reviewer decisions should not include raw user data or private customer detail.
  • Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.
  • The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.
  • The adjudication evidence stores prompt_sha256 instead of raw prompt text.
pending · external

Native Permission Enforcement

native-enforced targets 0; installer-enforced targets 4

提交
evidence/world_class/submissions/native-permission-enforcement.json
模板
evidence/world_class/templates/native-permission-enforcement.intake.json
阻断
1 blocked / 2 pass
下一步
Integrate a real target-client or external installer runtime guard before claiming native permission enforcement.
阻断检查
  • Native enforcementnative_enforcement_count: 0 / >0Collect real target-client or external runtime guard proof.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
  2. Update the generated target adapter only when the guard is actually enforced by that target.
  3. python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
采集契约
来源要求
  • real target client or external installer runtime guard
  • native enforcement flag or externally accepted guard proof
  • residual risk retained for fallback targets
通过条件
  • reports/runtime_permission_probes.json summary.native_enforcement_count > 0
  • reports/runtime_permission_probes.json summary.failure_count == 0
  • reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
  • reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
证据资产
  • dist/targets/*/adapter.json
  • reports/runtime_permission_probes.json
  • reports/runtime_permission_probes.md
  • reports/install_simulation.json
  • reports/install_simulation.md
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence11 / 11 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Do not mark native_enforcement true for metadata-only fallbacks.
  • Keep residual risks visible for targets that still rely on operator enforcement.
pending · external

Native Client Telemetry

external source events 0; adoption samples 1

提交
evidence/world_class/submissions/native-client-telemetry.json
模板
evidence/world_class/templates/native-client-telemetry.intake.json
阻断
1 blocked / 2 pass
下一步
Install a real client against the native host and import production metadata-only events.
阻断检查
  • External eventsexternal_source_events: 0 / >0Import at least one metadata-only event from a real client.
操作命令
  • 准备提交python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions
  • 校验入口python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
  • 审查提交python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions
  • 刷新台账python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions
首要步骤
  1. python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
  2. Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
  3. python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
采集契约
来源要求
  • real external client source
  • metadata-only event
  • local-first import path
通过条件
  • reports/adoption_drift_report.json summary.source_types.external > 0
  • reports/adoption_drift_report.json summary.adoption_sample_count > 0
  • reports/skill_os2_audit.json item native-client-telemetry status becomes pass
证据资产
  • reports/adoption_drift_report.json
  • reports/adoption_drift_report.md
  • reports/telemetry_hook_recipes.json
  • scripts/telemetry_native_host.py
  • evidence/world_class/intake.schema.json
资产角色

source: world-class-submission-kit; counts as evidence: false; prefill counts as evidence: false

  • submission-ref2 / 2 readyartifact_refs: trueRows marked submission-ref are the aggregate paths expected in artifact_refs.
  • supporting-evidence6 / 6 readyartifact_refs: falseSupporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.
隐私边界
  • Telemetry must remain metadata-only and local-first.
  • Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
@@ -817,7 +817,7 @@
-

信任报告

Secret
0
脚本数
147
网络脚本
3
Help 失败
0
包体哈希
83ea6e60852fae9271eb72658d8b3c0268e6adb21d80b2fd0fd9438805481ca7
+

信任报告

Secret
0
脚本数
147
网络脚本
3
Help 失败
0
包体哈希
e9d5b26e20c81f565271a1ecd793981ee1e56826313b160aebe776d2f89ca995

安全边界

高风险 secret、远程 inline execution、缺失依赖策略或无法解释的脚本接口应阻断 governed release。

diff --git a/reports/review-studio.json b/reports/review-studio.json index 0c0ec01..0f6efe0 100644 --- a/reports/review-studio.json +++ b/reports/review-studio.json @@ -232,7 +232,7 @@ "label": "output eval cases", "kind": "eval", "line": 1, - "matched_pattern": "", + "matched_pattern": "\"id\"", "excerpt": "{\"id\":\"skill-package-contract\",\"prompt\":\"Turn this repeated workflow into a reusable team skill package.\",\"baseline_output\":\"I can write a prompt for that workflow and include a s…", "exists": true, "link": "../evals/output/cases.jsonl" @@ -4769,7 +4769,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.8, + "duration_ms": 26.9, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4797,7 +4797,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.3, + "duration_ms": 25.49, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4820,7 +4820,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 27.02, + "duration_ms": 25.48, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4848,7 +4848,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.2, + "duration_ms": 28.61, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4871,7 +4871,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 30.27, + "duration_ms": 28.59, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4899,7 +4899,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 31.59, + "duration_ms": 26.16, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4922,7 +4922,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 30.08, + "duration_ms": 26.0, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4949,7 +4949,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 28.0, + "duration_ms": 26.3, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4972,7 +4972,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 28.31, + "duration_ms": 25.82, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -5001,7 +5001,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 28.12, + "duration_ms": 25.81, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -6973,7 +6973,7 @@ "compiled_targets": { "schema_version": "1.0", "ok": true, - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "skill_dir": ".", "summary": { "target_count": 5, @@ -12344,7 +12344,7 @@ "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", "package_hash_file_count": 239, - "package_sha256": "83ea6e60852fae9271eb72658d8b3c0268e6adb21d80b2fd0fd9438805481ca7" + "package_sha256": "e9d5b26e20c81f565271a1ecd793981ee1e56826313b160aebe776d2f89ca995" }, "failures": [], "warnings": [], @@ -15761,7 +15761,7 @@ "python_compatibility": { "schema_version": "1.0", "ok": true, - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "root": ".", "summary": { "target_python": "3.11", @@ -17142,7 +17142,7 @@ "architecture_maintainability": { "schema_version": "1.0", "ok": true, - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "skill_dir": ".", "summary": { "python_file_count": 222, @@ -17175,28 +17175,12 @@ }, { "path": "tests/verify_world_class_evidence_intake.py", - "lines": 690, + "lines": 628, "kind": "test", "severity": "pass", "early_watch": true, "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands." }, - { - "path": "tests/verify_output_review_adjudication.py", - "lines": 598, - "kind": "test", - "severity": "pass", - "early_watch": false, - "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands." - }, - { - "path": "tests/verify_review_studio.py", - "lines": 596, - "kind": "test", - "severity": "pass", - "early_watch": false, - "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands." - }, { "path": "scripts/render_benchmark_reproducibility.py", "lines": 595, @@ -17205,14 +17189,6 @@ "early_watch": false, "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." }, - { - "path": "scripts/render_skill_overview.py", - "lines": 588, - "kind": "cli-script", - "severity": "pass", - "early_watch": false, - "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." - }, { "path": "scripts/build_skill_atlas.py", "lines": 586, @@ -17230,21 +17206,29 @@ "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." }, { - "path": "scripts/trust_check.py", - "lines": 582, + "path": "scripts/render_skill_overview.py", + "lines": 584, "kind": "cli-script", "severity": "pass", "early_watch": false, "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." }, { - "path": "tests/verify_world_class_evidence_ledger.py", - "lines": 579, + "path": "tests/verify_review_studio.py", + "lines": 584, "kind": "test", "severity": "pass", "early_watch": false, "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands." }, + { + "path": "scripts/trust_check.py", + "lines": 582, + "kind": "cli-script", + "severity": "pass", + "early_watch": false, + "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." + }, { "path": "scripts/render_evidence_consistency.py", "lines": 568, @@ -17260,6 +17244,22 @@ "severity": "pass", "early_watch": false, "recommendation": "Move data loading and large section renderers into focused review_studio_* modules." + }, + { + "path": "tests/verify_world_class_evidence_ledger.py", + "lines": 567, + "kind": "test", + "severity": "pass", + "early_watch": false, + "recommendation": "Break broad integration assertions into focused verifier helpers when the next behavior change lands." + }, + { + "path": "scripts/render_world_class_preflight.py", + "lines": 562, + "kind": "cli-script", + "severity": "pass", + "early_watch": false, + "recommendation": "Watch this file before adding new responsibilities; extract a helper module when one concern dominates." } ], "watchlist": [], @@ -17274,7 +17274,7 @@ }, { "path": "tests/verify_world_class_evidence_intake.py", - "lines": 690, + "lines": 628, "kind": "test", "severity": "pass", "early_watch": true, @@ -17296,9 +17296,9 @@ "context_budget_tier": "production", "context_budget_limit": 1000, "skill_body_tokens": 797, - "other_text_tokens": 1094768, + "other_text_tokens": 1094713, "estimated_initial_load_tokens": 990, - "estimated_total_text_tokens": 1095565, + "estimated_total_text_tokens": 1095510, "deferred_resource_tokens": 498644, "deferred_resource_warn_threshold": 120000, "deferred_resource_dirs": [ @@ -18200,7 +18200,7 @@ }, "catalog": { "workspace_root": ".", - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "skills": [ { "name": "yao-meta-skill", @@ -19101,7 +19101,7 @@ "name": "incident-command-governor", "path": "examples/governed-incident-command/generated-skill", "reason": "review overdue by cadence monthly", - "age_days": 74, + "age_days": 78, "allowed_days": 31, "actionable": false, "scope": "example" @@ -19269,7 +19269,7 @@ "adoption_drift": { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-16T19:04:13Z", + "generated_at": "2026-06-16T17:53:31Z", "skill_dir": ".", "privacy_contract": { "storage": "local-first", @@ -24577,11 +24577,11 @@ "install_simulated": true, "install_simulation": "reports/install_simulation.json" }, - "generated_at": "2026-06-13" + "generated_at": "2026-06-17" }, "index": { "schema_version": "2.0", - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "package_count": 1, "packages": [ { @@ -25279,11 +25279,11 @@ "install_simulation": { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "skill_dir": ".", - "package_dir": "tests/tmp_review_studio/dist", - "install_root": "tests/tmp_review_studio/install-root/simulate-yao-meta-skill", - "installed_skill_dir": "tests/tmp_review_studio/install-root/simulate-yao-meta-skill/yao-meta-skill", + "package_dir": "dist", + "install_root": "dist/install-simulation/simulate-yao-meta-skill", + "installed_skill_dir": "dist/install-simulation/simulate-yao-meta-skill/yao-meta-skill", "summary": { "archive_present": true, "archive_entry_count": 685, @@ -25304,7 +25304,7 @@ { "id": "archive-present", "status": "pass", - "detail": "Package archive exists: tests/tmp_review_studio/dist/yao-meta-skill.zip" + "detail": "Package archive exists: dist/yao-meta-skill.zip" }, { "id": "archive-safe-paths", @@ -25550,16 +25550,16 @@ "failures": [], "warnings": [], "artifacts": { - "archive": "tests/tmp_review_studio/dist/yao-meta-skill.zip", - "package_manifest": "tests/tmp_review_studio/dist/manifest.json", - "json": "tests/tmp_review_studio/install_simulation.json", - "markdown": "tests/tmp_review_studio/install_simulation.md" + "archive": "dist/yao-meta-skill.zip", + "package_manifest": "dist/manifest.json", + "json": "reports/install_simulation.json", + "markdown": "reports/install_simulation.md" } }, "upgrade_check": { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-13", + "generated_at": "2026-06-17", "previous": { "name": "yao-meta-skill", "version": "1.0.0", @@ -25648,8 +25648,8 @@ "artifacts": { "previous_package": "registry/examples/yao-meta-skill-1.0.0.json", "current_package": "reports/registry_audit.json", - "json": "tests/tmp_review_studio/upgrade_check.json", - "markdown": "tests/tmp_review_studio/upgrade_check.md" + "json": "reports/upgrade_check.json", + "markdown": "reports/upgrade_check.md" } }, "manifest": { diff --git a/scripts/review_studio_actions.py b/scripts/review_studio_actions.py index 6e7bb1b..ba9788c 100644 --- a/scripts/review_studio_actions.py +++ b/scripts/review_studio_actions.py @@ -102,7 +102,7 @@ ACTION_GUIDANCE: dict[str, dict[str, Any]] = { "why": "没有输出质量和人工盲评证据时，Skill 只能证明会触发，不能证明输出真的更好且经得起审查。", "source_fix": "evals/output/cases.jsonl + reports/output_quality_scorecard.md + reports/output_review_kit.html + reports/output_review_adjudication.md", "source_paths": [ - {"path": "evals/output/cases.jsonl", "label": "output eval cases", "kind": "eval", "patterns": ["case_id"]}, + {"path": "evals/output/cases.jsonl", "label": "output eval cases", "kind": "eval", "patterns": ['"case_id"', '"id"']}, {"path": "reports/output_quality_scorecard.md", "label": "output scorecard", "kind": "report", "patterns": ["# Output"]}, {"path": "reports/output_execution_runs.md", "label": "output execution runs", "kind": "report", "patterns": ["# Output Execution"]}, {"path": "reports/output_blind_review_pack.md", "label": "blind A/B review pack", "kind": "report", "patterns": ["# Output Blind"]}, diff --git a/tests/verify_review_studio.py b/tests/verify_review_studio.py index b64e80d..2942fc4 100644 --- a/tests/verify_review_studio.py +++ b/tests/verify_review_studio.py @@ -261,6 +261,7 @@ def main() -> None: assert all(item["exists"] for item in world_class_action["source_refs"]), world_class_action assert all("matched_pattern" in item for item in world_class_action["source_refs"]), world_class_action assert all("excerpt" in item for item in world_class_action["source_refs"]), world_class_action + assert all(isinstance(item["line"], int) and item["line"] >= 1 for item in world_class_action["source_refs"]), world_class_action assert any( item["path"] == "reports/world_class_evidence_ledger.md" and item["matched_pattern"] == "# World-Class Evidence Ledger" @@ -313,11 +314,8 @@ def main() -> None: assert full_payload["data"]["world_class_submission_review"]["summary"]["awaiting_submission_count"] == 4, full_payload["data"]["world_class_submission_review"] assert full_payload["data"]["world_class_submission_review"]["summary"]["source_check_count"] >= 13, full_payload["data"]["world_class_submission_review"] assert full_payload["data"]["world_class_submission_review"]["summary"]["source_blocked_count"] >= 6, full_payload["data"]["world_class_submission_review"] - human_review_item = next( - item - for item in full_payload["data"]["world_class_submission_review"]["items"] - if item["evidence_key"] == "human-adjudication" - ) + human_review_items = full_payload["data"]["world_class_submission_review"]["items"] + human_review_item = next(item for item in human_review_items if item["evidence_key"] == "human-adjudication") human_review_source = {item["field"]: item for item in human_review_item["source_checklist"]} assert human_review_item["observed_state"]["raw_content_allowed"] is False, human_review_item assert human_review_item["observed_state"]["raw_content_path_count"] == 0, human_review_item @@ -405,12 +403,8 @@ def main() -> None: assert all(item["exists"] for item in synthetic_actions[0]["source_refs"]), synthetic_actions assert all(isinstance(item["line"], int) and item["line"] >= 1 for item in synthetic_actions[0]["source_refs"]), synthetic_actions assert all("excerpt" in item for item in synthetic_actions[0]["source_refs"]), synthetic_actions - assert any( - item["path"] == "reports/output_review_kit.html" - and item["matched_pattern"] == "Output Review Kit" - and "Output Review Kit" in item["excerpt"] - for item in synthetic_actions[0]["source_refs"] - ), synthetic_actions + assert any(item["path"] == "reports/output_review_kit.html" and item["matched_pattern"] == "Output Review Kit" and "Output Review Kit" in item["excerpt"] for item in synthetic_actions[0]["source_refs"]), synthetic_actions + assert any(item["path"] == "evals/output/cases.jsonl" and item["matched_pattern"] == '"id"' and '"id"' in item["excerpt"] for item in synthetic_actions[0]["source_refs"]), synthetic_actions synthetic_json = json.dumps(synthetic_actions, ensure_ascii=False) assert str(ROOT) not in synthetic_json, synthetic_json synthetic_html = review_studio.render_review_actions(synthetic_actions)