diff --git a/README.md b/README.md index 50bda2c..a9978de 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,8 @@ It turns rough workflows, transcripts, prompts, notes, and runbooks into reusabl - a world-class evidence plan that turns remaining provider, human, native-permission, and real-client telemetry gaps into executable evidence tasks - a world-class evidence ledger that records which external and human evidence is accepted or still pending without treating planned work as proof - a world-class evidence intake contract that validates external and human evidence packets for provenance, privacy, artifact refs, and anti-overclaim rules before ledger review +- a world-class submission review queue that compares evidence packets, intake validation, source artifacts, and ledger state without accepting evidence +- a world-class operator runbook that gives reviewers the exact commands, artifacts, and collection checklist needed to close remaining evidence gaps - a world-class claim guard that scans public claim surfaces and blocks premature completed/true claims while the evidence ledger still has pending external or human evidence - a benchmark reproducibility manifest that checks methodology sections, required artifacts, failure disclosure, and reproduction commands - Output Eval Lab evidence with assertion grading, execution/timing/token evidence, a blind A/B review pack, a separate answer key, and reviewer adjudication reports @@ -147,6 +149,7 @@ python3 scripts/yao.py world-class-submission-kit . --output-dir /tmp/yao-world- python3 scripts/yao.py world-class-ledger . python3 scripts/yao.py world-class-intake . python3 scripts/yao.py world-class-submission-review . +python3 scripts/yao.py world-class-runbook . python3 scripts/yao.py world-class-claim-guard . python3 scripts/yao.py benchmark-reproducibility . ``` @@ -298,6 +301,7 @@ Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/fa - current world-class evidence acceptance state is published in [reports/world_class_evidence_ledger.md](reports/world_class_evidence_ledger.md) - world-class evidence intake readiness is published in [reports/world_class_evidence_intake.md](reports/world_class_evidence_intake.md) - world-class submission review queue is published in [reports/world_class_submission_review.md](reports/world_class_submission_review.md) +- world-class operator runbook is published in [reports/world_class_operator_runbook.md](reports/world_class_operator_runbook.md) and [reports/world_class_operator_runbook.html](reports/world_class_operator_runbook.html) - world-class public claim guard status is published in [reports/world_class_claim_guard.md](reports/world_class_claim_guard.md) - benchmark reproducibility evidence is published in [reports/benchmark_reproducibility.md](reports/benchmark_reproducibility.md) - target compiler evidence is published in [reports/compiled_targets.md](reports/compiled_targets.md) @@ -416,7 +420,7 @@ Utility scripts that make the meta-skill operational: - `run_description_optimization_suite.py`: runs description optimization across the root skill and governed examples, then writes reusable reports and optional drift snapshots with calibration and family summaries - `promotion_checker.py`: applies promotion policy to current description candidates, writes promotion decisions, builds candidate registries, and emits iteration bundles with review stubs - `create_iteration_snapshot.py`: freezes the current promotion decision into a versioned release snapshot with review, route, and context evidence -- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, python-compat, review, release-snapshot, workspace-flow, report, skill-ir, compile-skill, output-exec, output-review, skill-os2-audit, skill-os2-coverage, world-class-evidence, world-class-ledger, world-class-intake, world-class-submission-kit, world-class-submission-review, world-class-claim-guard, benchmark-reproducibility, telemetry-emit, telemetry-hooks, telemetry-import, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint +- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, python-compat, review, release-snapshot, workspace-flow, report, skill-ir, compile-skill, output-exec, output-review, skill-os2-audit, skill-os2-coverage, world-class-evidence, world-class-ledger, world-class-intake, world-class-submission-kit, world-class-submission-review, world-class-runbook, world-class-claim-guard, benchmark-reproducibility, telemetry-emit, telemetry-hooks, telemetry-import, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint - `render_description_drift_history.py`: turns description-optimization snapshots into a readable drift-history report - `build_confusion_matrix.py`: scores route confusion across tracked sibling skills and `no_route` cases, then writes a route scorecard and optional milestone snapshot - `render_iteration_ledger.py`: compresses regression milestones, description optimization drift, and route scorecards into one iteration-facing ledger @@ -431,6 +435,7 @@ Utility scripts that make the meta-skill operational: - `render_world_class_evidence_ledger.py`: renders a machine-checkable ledger for current world-class evidence acceptance, anti-overclaim guards, provenance requirements, and privacy contracts - `render_world_class_evidence_intake.py`: validates world-class external and human evidence packets against provenance, privacy, artifact, and anti-overclaim requirements before ledger review - `render_world_class_submission_review.py`: renders a read-only queue that compares submissions, intake validation, source evidence, and ledger state without accepting evidence +- `render_world_class_operator_runbook.py`: renders an operator-facing checklist and command map for collecting pending world-class evidence without accepting evidence - `render_world_class_claim_guard.py`: scans README, docs, and reports for premature world-class completion claims while accepted evidence is still pending - `render_benchmark_reproducibility.py`: renders methodology, artifact, failure-disclosure, and reproduction-command evidence for public benchmark claims - `python_compat_check.py`: checks Python source for supported-runtime compatibility hazards such as Python 3.11 f-string expression backslashes diff --git a/registry/index.json b/registry/index.json index b9c0d75..3a39eca 100644 --- a/registry/index.json +++ b/registry/index.json @@ -16,7 +16,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] } diff --git a/registry/packages/yao-meta-skill.json b/registry/packages/yao-meta-skill.json index 7837ed7..7c8afd3 100644 --- a/registry/packages/yao-meta-skill.json +++ b/registry/packages/yao-meta-skill.json @@ -16,8 +16,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179", - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, "compatibility": { "openai": "pass", @@ -48,7 +48,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" diff --git a/reports/adoption_drift_report.json b/reports/adoption_drift_report.json index 8ed61f0..3cbe51f 100644 --- a/reports/adoption_drift_report.json +++ b/reports/adoption_drift_report.json @@ -1,7 +1,7 @@ { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-14T00:05:43Z", + "generated_at": "2026-06-14T00:16:48Z", "skill_dir": ".", "privacy_contract": { "storage": "local-first", diff --git a/reports/benchmark_reproducibility.json b/reports/benchmark_reproducibility.json index 6d0537d..4e1875b 100644 --- a/reports/benchmark_reproducibility.json +++ b/reports/benchmark_reproducibility.json @@ -3,34 +3,34 @@ "ok": true, "generated_at": "2026-06-14", "skill_dir": ".", - "commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1", + "commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03", "git_status": { "available": true, "dirty": true, - "changed_file_count": 61, + "changed_file_count": 25, "sample": [ - "M AGENTS.md", - "M Makefile", - "M registry/index.json", - "M registry/packages/yao-meta-skill.json", - "MM reports/adoption_drift_report.json", - " M reports/adoption_drift_report.md", - "MM reports/architecture_maintainability.json", - "MM reports/architecture_maintainability.md", - "MM reports/benchmark_reproducibility.json", - "MM reports/benchmark_reproducibility.md", - "MM reports/compiled_targets.json", - "MM reports/context_budget.json" + " M README.md", + " M registry/index.json", + " M registry/packages/yao-meta-skill.json", + " M reports/adoption_drift_report.json", + " M reports/benchmark_reproducibility.json", + " M reports/benchmark_reproducibility.md", + " M reports/context_budget.json", + " M reports/output_execution_runs.json", + " M reports/output_execution_runs.md", + " M reports/package_verification.json", + " M reports/package_verification.md", + " M reports/registry_audit.json" ] }, "summary": { "reproducibility_ready": true, "methodology_complete": true, - "required_artifact_count": 20, + "required_artifact_count": 24, "missing_artifact_count": 0, "output_case_count": 5, "failure_disclosure_count": 3, - "command_count": 19, + "command_count": 21, "command_executed_count": 10, "timing_observed_count": 10, "model_executed_count": 0, @@ -42,7 +42,7 @@ "world_class_task_count": 4, "world_class_ledger_pending_count": 4, "working_tree_dirty": true, - "changed_file_count": 61 + "changed_file_count": 25 }, "methodology": { "path": "reports/benchmark_methodology.md", @@ -116,7 +116,7 @@ "path": "reports/output_execution_runs.json", "exists": true, "bytes": 7965, - "sha256": "99b2e3f03710fdcee8fe66a476bc589f4e22a379bfcc1cd5561de49c184286e8" + "sha256": "e4b3b5bf45a34ad8a616015289e968fdb13ec9a9ea1b15ff2ea94ef6cec40d7b" }, { "label": "blind_review", @@ -151,56 +151,56 @@ "path": "reports/security_trust_report.json", "exists": true, "bytes": 97005, - "sha256": "252c580028df6ac2404b53cc10bb607fa82b98ef17965cb7354c4731f6c6b08e" + "sha256": "f518d733e79e26ea36136c1e38e22e36922481724bb6c004a7475c73bfa9fbb6" }, { "label": "python_compatibility", "path": "reports/python_compatibility.json", "exists": true, "bytes": 20367, - "sha256": "a8d14087863459534fadac760a64c213c9474413e92046a9ac2e772d4174d347" + "sha256": "3698aad5388cad659840d1d31dbc9c694e0067b1de1e84423336a34e22df9625" }, { "label": "registry_audit", "path": "reports/registry_audit.json", "exists": true, "bytes": 3183, - "sha256": "6527b723a4bc367477e1090afbf7f1653f20d33fed0e0691725b7ee30896848b" + "sha256": "90137332142ac7ff036f1efd5eb1970c31baf60e8a5f8c541dc12aadc91492b6" }, { "label": "package_verification", "path": "reports/package_verification.json", "exists": true, "bytes": 19325, - "sha256": "6bf19c9e29803ad8668914489d22b9e306ad0ad092c91c179c3aaffe81eed2c6" + "sha256": "6c62c89994790f7034e115e349dd4627939e5c73361c36d5a56c887346405f90" }, { "label": "install_simulation", "path": "reports/install_simulation.json", "exists": true, - "bytes": 8758, - "sha256": "270a28b82dc49d80ba9c5a82f912cc960e50a5ac42f022c6a2c345e4e84fbb36" + "bytes": 8604, + "sha256": "97055f63c699b2812bab1a0f05460f1065a6211164b4c22c9bebc8aab6ef012f" }, { "label": "skill_os2_audit", "path": "reports/skill_os2_audit.json", "exists": true, "bytes": 14309, - "sha256": "cad5296cd500a40cb0d83897d327b3542f31c1ce6bfe7eaa387ea2e88d229aa1" + "sha256": "9c7b86742ed4dd5c930fbc3572d943937a4817bbd465d89da12f0929911607d0" }, { "label": "world_class_evidence_plan", "path": "reports/world_class_evidence_plan.json", "exists": true, "bytes": 10083, - "sha256": "02e49b66159d239309bd31be33f2119db23604b5506c4445ba6c6c84b6cdd1aa" + "sha256": "194d5a681920c3aca5e454d8839cdec551a076d54e360424831c5f312d1367b1" }, { "label": "world_class_evidence_ledger", "path": "reports/world_class_evidence_ledger.json", "exists": true, "bytes": 11308, - "sha256": "0b52d0d1528ba95a0fa7bead97cb8224b5b189d35497a05d90956fc90512b651" + "sha256": "dc4cf8cf18899bdf10d32fe72bb8f306ee297106e6cc25fd041d56e36631af63" }, { "label": "world_class_evidence_intake", @@ -209,6 +209,34 @@ "bytes": 12740, "sha256": "5fbfcd35ac6afd494c64845a7130520fd1cffbdb86e4efed2eaf2fa3d381f8b3" }, + { + "label": "world_class_submission_review", + "path": "reports/world_class_submission_review.json", + "exists": true, + "bytes": 7263, + "sha256": "185a8cab25e7155ebd80cb66049c0965e4dc2d7cc38991f17464946f87fd26f2" + }, + { + "label": "world_class_operator_runbook", + "path": "reports/world_class_operator_runbook.json", + "exists": true, + "bytes": 15002, + "sha256": "3d643dc8170b50a37f91ffb190f69ad1e0d32f64914ebe2d731226728ac28cdc" + }, + { + "label": "world_class_operator_runbook_markdown", + "path": "reports/world_class_operator_runbook.md", + "exists": true, + "bytes": 9206, + "sha256": "f67ca3d0aa9c02953e967d9657ca4cfe41ee883c73950bce16d6c25a1e206e10" + }, + { + "label": "world_class_operator_runbook_html", + "path": "reports/world_class_operator_runbook.html", + "exists": true, + "bytes": 13191, + "sha256": "963026f551dc687ae53e7ba1df5003dd59292c259080543dcaf4b4343437fed6" + }, { "label": "world_class_claim_guard", "path": "reports/world_class_claim_guard.json", @@ -304,6 +332,16 @@ "command": "python3 scripts/yao.py world-class-intake .", "evidence": "reports/world_class_evidence_intake.json" }, + { + "label": "world-class submission review", + "command": "python3 scripts/yao.py world-class-submission-review .", + "evidence": "reports/world_class_submission_review.json" + }, + { + "label": "world-class operator runbook", + "command": "python3 scripts/yao.py world-class-runbook .", + "evidence": "reports/world_class_operator_runbook.json" + }, { "label": "world-class claim guard", "command": "python3 scripts/yao.py world-class-claim-guard .", diff --git a/reports/benchmark_reproducibility.md b/reports/benchmark_reproducibility.md index 0d4fa69..3a46287 100644 --- a/reports/benchmark_reproducibility.md +++ b/reports/benchmark_reproducibility.md @@ -1,22 +1,22 @@ # Benchmark Reproducibility Generated at: `2026-06-14` -Commit: `08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1` +Commit: `5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03` Working tree dirty at generation: `true` ## Summary - reproducibility ready: `true` - methodology complete: `true` -- required artifacts: `20` +- required artifacts: `24` - missing artifacts: `0` - output cases: `5` - disclosed failure cases: `3` -- reproduction commands: `19` +- reproduction commands: `21` - provider evidence complete: `false` - human review complete: `false` - world-class ready: `false` -- changed files at generation: `61` +- changed files at generation: `25` This report proves local benchmark reproducibility only. It keeps external provider and human-review gaps visible instead of counting them as complete. @@ -40,20 +40,24 @@ This report proves local benchmark reproducibility only. It keeps external provi | output_cases | `evals/output/cases.jsonl` | present | `a6ae96857116` | | output_schema | `evals/output/schema.json` | present | `8ee340c95064` | | output_scorecard | `reports/output_quality_scorecard.json` | present | `0806258a8e08` | -| output_execution | `reports/output_execution_runs.json` | present | `99b2e3f03710` | +| output_execution | `reports/output_execution_runs.json` | present | `e4b3b5bf45a3` | | blind_review | `reports/output_blind_review_pack.json` | present | `bbe2db8ec277` | | review_adjudication | `reports/output_review_adjudication.json` | present | `240485a721af` | | trigger_scorecard | `reports/route_scorecard.json` | present | `c164e83e36d0` | | runtime_conformance | `reports/conformance_matrix.json` | present | `8251329e663d` | -| trust_report | `reports/security_trust_report.json` | present | `252c580028df` | -| python_compatibility | `reports/python_compatibility.json` | present | `a8d140878634` | -| registry_audit | `reports/registry_audit.json` | present | `6527b723a4bc` | -| package_verification | `reports/package_verification.json` | present | `6bf19c9e2980` | -| install_simulation | `reports/install_simulation.json` | present | `270a28b82dc4` | -| skill_os2_audit | `reports/skill_os2_audit.json` | present | `cad5296cd500` | -| world_class_evidence_plan | `reports/world_class_evidence_plan.json` | present | `02e49b66159d` | -| world_class_evidence_ledger | `reports/world_class_evidence_ledger.json` | present | `0b52d0d1528b` | +| trust_report | `reports/security_trust_report.json` | present | `f518d733e79e` | +| python_compatibility | `reports/python_compatibility.json` | present | `3698aad5388c` | +| registry_audit | `reports/registry_audit.json` | present | `90137332142a` | +| package_verification | `reports/package_verification.json` | present | `6c62c8999479` | +| install_simulation | `reports/install_simulation.json` | present | `97055f63c699` | +| skill_os2_audit | `reports/skill_os2_audit.json` | present | `9c7b86742ed4` | +| world_class_evidence_plan | `reports/world_class_evidence_plan.json` | present | `194d5a681920` | +| world_class_evidence_ledger | `reports/world_class_evidence_ledger.json` | present | `dc4cf8cf1889` | | world_class_evidence_intake | `reports/world_class_evidence_intake.json` | present | `5fbfcd35ac6a` | +| world_class_submission_review | `reports/world_class_submission_review.json` | present | `185a8cab25e7` | +| world_class_operator_runbook | `reports/world_class_operator_runbook.json` | present | `3d643dc8170b` | +| world_class_operator_runbook_markdown | `reports/world_class_operator_runbook.md` | present | `f67ca3d0aa9c` | +| world_class_operator_runbook_html | `reports/world_class_operator_runbook.html` | present | `963026f551dc` | | world_class_claim_guard | `reports/world_class_claim_guard.json` | present | `250d616b028c` | ## Reproduction Commands @@ -92,6 +96,10 @@ This report proves local benchmark reproducibility only. It keeps external provi - evidence: `reports/world_class_evidence_ledger.json` - `python3 scripts/yao.py world-class-intake .` - evidence: `reports/world_class_evidence_intake.json` +- `python3 scripts/yao.py world-class-submission-review .` + - evidence: `reports/world_class_submission_review.json` +- `python3 scripts/yao.py world-class-runbook .` + - evidence: `reports/world_class_operator_runbook.json` - `python3 scripts/yao.py world-class-claim-guard .` - evidence: `reports/world_class_claim_guard.json` - `make ci-test` diff --git a/reports/context_budget.json b/reports/context_budget.json index e6228ca..65fb2c5 100644 --- a/reports/context_budget.json +++ b/reports/context_budget.json @@ -6,9 +6,9 @@ "context_budget_tier": "production", "context_budget_limit": 1000, "skill_body_tokens": 751, - "other_text_tokens": 1175718, + "other_text_tokens": 1176737, "estimated_initial_load_tokens": 944, - "estimated_total_text_tokens": 1176469, + "estimated_total_text_tokens": 1177488, "relevant_file_count": 498, "unused_resource_dirs": [], "quality_signal_points": 130, diff --git a/reports/output_execution_runs.json b/reports/output_execution_runs.json index 8a2522a..ec618b6 100644 --- a/reports/output_execution_runs.json +++ b/reports/output_execution_runs.json @@ -34,7 +34,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.47, + "duration_ms": 28.47, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -62,7 +62,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.78, + "duration_ms": 28.4, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -85,7 +85,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.73, + "duration_ms": 26.37, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -136,7 +136,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 29.2, + "duration_ms": 28.04, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -164,7 +164,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 28.81, + "duration_ms": 29.89, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -187,7 +187,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.65, + "duration_ms": 26.27, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -214,7 +214,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.22, + "duration_ms": 25.7, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -237,7 +237,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.43, + "duration_ms": 25.72, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -266,7 +266,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.88, + "duration_ms": 25.67, "provider": "local-output-eval-runner", "model": "", "usage": { diff --git a/reports/output_execution_runs.md b/reports/output_execution_runs.md index 57ab3e4..4581fa9 100644 --- a/reports/output_execution_runs.md +++ b/reports/output_execution_runs.md @@ -23,16 +23,16 @@ Command runner evidence is present. This proves the eval harness executed an ext | Case | Variant | Mode | Model | Duration ms | Tokens | Score | Status | | --- | --- | --- | --- | ---: | ---: | ---: | --- | -| skill-package-contract | baseline | command | local-output-eval-runner | 26.47 | 33 | 0.0 | pass | -| skill-package-contract | with_skill | command | local-output-eval-runner | 25.78 | 73 | 100.0 | pass | -| output-eval-expectation | baseline | command | local-output-eval-runner | 25.73 | 36 | 0.0 | pass | +| skill-package-contract | baseline | command | local-output-eval-runner | 28.47 | 33 | 0.0 | pass | +| skill-package-contract | with_skill | command | local-output-eval-runner | 28.4 | 73 | 100.0 | pass | +| output-eval-expectation | baseline | command | local-output-eval-runner | 26.37 | 36 | 0.0 | pass | | output-eval-expectation | with_skill | command | local-output-eval-runner | 29.11 | 80 | 100.0 | pass | -| ir-before-packaging | baseline | command | local-output-eval-runner | 29.2 | 33 | 0.0 | pass | -| ir-before-packaging | with_skill | command | local-output-eval-runner | 28.81 | 80 | 100.0 | pass | -| near-neighbor-boundary | baseline | command | local-output-eval-runner | 25.65 | 36 | 0.0 | pass | -| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 26.22 | 65 | 100.0 | pass | -| file-backed-governed-package | baseline | command | local-output-eval-runner | 25.43 | 37 | 0.0 | pass | -| file-backed-governed-package | with_skill | command | local-output-eval-runner | 25.88 | 98 | 100.0 | pass | +| ir-before-packaging | baseline | command | local-output-eval-runner | 28.04 | 33 | 0.0 | pass | +| ir-before-packaging | with_skill | command | local-output-eval-runner | 29.89 | 80 | 100.0 | pass | +| near-neighbor-boundary | baseline | command | local-output-eval-runner | 26.27 | 36 | 0.0 | pass | +| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 25.7 | 65 | 100.0 | pass | +| file-backed-governed-package | baseline | command | local-output-eval-runner | 25.72 | 37 | 0.0 | pass | +| file-backed-governed-package | with_skill | command | local-output-eval-runner | 25.67 | 98 | 100.0 | pass | ## Next Fixes diff --git a/reports/package_verification.json b/reports/package_verification.json index 9e8105c..f0edc48 100644 --- a/reports/package_verification.json +++ b/reports/package_verification.json @@ -8,7 +8,7 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "archive_entry_count": 573, "failure_count": 0, "warning_count": 0 diff --git a/reports/package_verification.md b/reports/package_verification.md index 0013353..593f466 100644 --- a/reports/package_verification.md +++ b/reports/package_verification.md @@ -4,7 +4,7 @@ - Package directory: `dist` - Targets: `4 / 4` adapters present - Archive present: `True` -- Archive SHA256: `de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e` +- Archive SHA256: `5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27` - Failures: `0` - Warnings: `0` diff --git a/reports/registry_audit.json b/reports/registry_audit.json index ee90b8e..855e96c 100644 --- a/reports/registry_audit.json +++ b/reports/registry_audit.json @@ -21,8 +21,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179", - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, "compatibility": { "openai": "pass", @@ -53,7 +53,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -78,7 +78,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, diff --git a/reports/registry_audit.md b/reports/registry_audit.md index 040a4b4..28037f0 100644 --- a/reports/registry_audit.md +++ b/reports/registry_audit.md @@ -6,8 +6,8 @@ - Maturity: `governed` - Owner: `Yao Team` - License: `MIT` -- Package SHA256: `97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179` -- Archive SHA256: `de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e` +- Package SHA256: `2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d` +- Archive SHA256: `5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27` - Install simulated: `True` ## Compatibility diff --git a/reports/review-studio.html b/reports/review-studio.html index 5b809fb..ff7f2cf 100644 --- a/reports/review-studio.html +++ b/reports/review-studio.html @@ -506,7 +506,7 @@
-

信任报告

Secret
0
脚本数
94
网络脚本
3
Help 失败
0
包体哈希
97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179
+

信任报告

Secret
0
脚本数
94
网络脚本
3
Help 失败
0
包体哈希
2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d

安全边界

高风险 secret、远程 inline execution、缺失依赖策略或无法解释的脚本接口应阻断 governed release。

@@ -574,12 +574,12 @@

注册审计

yao-meta-skill 1.1.0; 6/6 compatibility entries pass; install pass with 4 adapters; installer permissions 12 enforced / 0 failures

-

包体元数据

名称
yao-meta-skill
版本
1.1.0
Maturity
governed
Owner
Yao Team
License
MIT
信任级别
local
目标平台
openai, claude, generic, agent-skills-compatible, vscode
兼容通过
6/6
归档哈希
de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e
+

包体元数据

名称
yao-meta-skill
版本
1.1.0
Maturity
governed
Owner
Yao Team
License
MIT
信任级别
local
目标平台
openai, claude, generic, agent-skills-compatible, vscode
兼容通过
6/6
归档哈希
5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27

发布路线

0 promote; 3 keep current; 0 blocked; upgrade minor declared / minor recommended

-

包体验证

目标数
4
Adapter
4
归档存在
是
Zip 条目
573
失败数
0
警告数
0
归档哈希
de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e
+

包体验证

目标数
4
Adapter
4
归档存在
是
Zip 条目
573
失败数
0
警告数
0
归档哈希
5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27
diff --git a/reports/review-studio.json b/reports/review-studio.json index 1e66ced..5a69012 100644 --- a/reports/review-studio.json +++ b/reports/review-studio.json @@ -1333,11 +1333,11 @@ "summary": { "reproducibility_ready": true, "methodology_complete": true, - "required_artifact_count": 20, + "required_artifact_count": 24, "missing_artifact_count": 0, "output_case_count": 5, "failure_disclosure_count": 3, - "command_count": 19, + "command_count": 21, "command_executed_count": 10, "timing_observed_count": 10, "model_executed_count": 0, @@ -1349,9 +1349,9 @@ "world_class_task_count": 4, "world_class_ledger_pending_count": 4, "working_tree_dirty": true, - "changed_file_count": 61 + "changed_file_count": 41 }, - "commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1", + "commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03", "missing_artifacts": [], "limitations": [ "Local command-runner evidence is reproducible but does not replace provider-backed model holdout evidence.", @@ -1455,7 +1455,7 @@ "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", "package_hash_file_count": 178, - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" }, "skill_atlas": { "skill_count": 12, @@ -1493,8 +1493,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179", - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, "compatibility": { "openai": "pass", @@ -1525,7 +1525,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -1541,7 +1541,7 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "archive_entry_count": 573, "failure_count": 0, "warning_count": 0 @@ -1620,12 +1620,12 @@ { "field": "archive_sha256", "from": "", - "to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, @@ -3794,7 +3794,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.47, + "duration_ms": 28.47, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3822,7 +3822,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.78, + "duration_ms": 28.4, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3845,7 +3845,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.73, + "duration_ms": 26.37, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3896,7 +3896,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 29.2, + "duration_ms": 28.04, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3924,7 +3924,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 28.81, + "duration_ms": 29.89, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3947,7 +3947,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.65, + "duration_ms": 26.27, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3974,7 +3974,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 26.22, + "duration_ms": 25.7, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3997,7 +3997,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.43, + "duration_ms": 25.72, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -4026,7 +4026,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 25.88, + "duration_ms": 25.67, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -5049,7 +5049,7 @@ "label": "Benchmark Methodology", "status": "pass", "objective": "Recommended Skill OS 2.0 implementation PR from the upgrade plan.", - "current": "20 required artifacts checked", + "current": "24 required artifacts checked", "command": "make ci-test", "test": "tests/verify_benchmark_reproducibility.py", "evidence": [ @@ -10242,7 +10242,7 @@ "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", "package_hash_file_count": 178, - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" }, "failures": [], "warnings": [], @@ -13724,9 +13724,9 @@ "context_budget_tier": "production", "context_budget_limit": 1000, "skill_body_tokens": 751, - "other_text_tokens": 1175718, + "other_text_tokens": 1176737, "estimated_initial_load_tokens": 944, - "estimated_total_text_tokens": 1176469, + "estimated_total_text_tokens": 1177488, "relevant_file_count": 498, "unused_resource_dirs": [], "quality_signal_points": 130, @@ -15580,7 +15580,7 @@ "adoption_drift": { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-14T00:05:43Z", + "generated_at": "2026-06-14T00:16:48Z", "skill_dir": ".", "privacy_contract": { "storage": "local-first", @@ -17169,8 +17169,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179", - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, "compatibility": { "openai": "pass", @@ -17201,7 +17201,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -17226,7 +17226,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, @@ -17249,7 +17249,7 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "archive_entry_count": 573, "failure_count": 0, "warning_count": 0 @@ -18257,12 +18257,12 @@ { "field": "archive_sha256", "from": "", - "to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, diff --git a/reports/security_trust_report.json b/reports/security_trust_report.json index c6f4bbd..5a2376b 100644 --- a/reports/security_trust_report.json +++ b/reports/security_trust_report.json @@ -23,7 +23,7 @@ "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", "package_hash_file_count": 178, - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" }, "failures": [], "warnings": [], diff --git a/reports/security_trust_report.md b/reports/security_trust_report.md index 71e1e1d..5e7c786 100644 --- a/reports/security_trust_report.md +++ b/reports/security_trust_report.md @@ -16,7 +16,7 @@ - Interactive scripts: `0` - Package hash scope: `source-contract-without-generated-reports` - Package hash files: `178` -- Package SHA256: `97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179` +- Package SHA256: `2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d` ## Failures diff --git a/reports/skill-overview.json b/reports/skill-overview.json index dc51038..9b619a7 100644 --- a/reports/skill-overview.json +++ b/reports/skill-overview.json @@ -919,11 +919,11 @@ "summary": { "reproducibility_ready": true, "methodology_complete": true, - "required_artifact_count": 20, + "required_artifact_count": 24, "missing_artifact_count": 0, "output_case_count": 5, "failure_disclosure_count": 3, - "command_count": 19, + "command_count": 21, "command_executed_count": 10, "timing_observed_count": 10, "model_executed_count": 0, @@ -935,9 +935,9 @@ "world_class_task_count": 4, "world_class_ledger_pending_count": 4, "working_tree_dirty": true, - "changed_file_count": 61 + "changed_file_count": 41 }, - "commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1", + "commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03", "missing_artifacts": [], "limitations": [ "Local command-runner evidence is reproducible but does not replace provider-backed model holdout evidence.", @@ -1041,7 +1041,7 @@ "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", "package_hash_file_count": 178, - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" }, "skill_atlas": { "skill_count": 12, @@ -1079,8 +1079,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179", - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, "compatibility": { "openai": "pass", @@ -1111,7 +1111,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -1127,7 +1127,7 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e", + "archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27", "archive_entry_count": 573, "failure_count": 0, "warning_count": 0 @@ -1206,12 +1206,12 @@ { "field": "archive_sha256", "from": "", - "to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, diff --git a/reports/skill_os2_audit.json b/reports/skill_os2_audit.json index e7d4abd..d085c79 100644 --- a/reports/skill_os2_audit.json +++ b/reports/skill_os2_audit.json @@ -145,7 +145,7 @@ "key": "benchmark-reproducibility", "label": "Benchmark Reproducibility", "status": "pass", - "current": "artifacts 20; missing 0; failures 3", + "current": "artifacts 24; missing 0; failures 3", "target": "Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable", "evidence": [ { diff --git a/reports/skill_os2_audit.md b/reports/skill_os2_audit.md index ab20a7e..1bed703 100644 --- a/reports/skill_os2_audit.md +++ b/reports/skill_os2_audit.md @@ -21,7 +21,7 @@ Generated at: `2026-06-13` | Output Eval Lab | pass | 5 cases; delta 100.0; exec 10; blind 5 | with-skill/baseline, assertions, execution evidence, blind A/B, failure taxonomy | Add more real-file and adversarial holdout cases as usage grows. | | Provider Holdout | external-required | model-executed 0; token-observed 0 | At least one real provider-backed holdout run with observed model/timing/token metadata | Run provider-backed holdout cases with real credentials and commit only aggregate evidence. | | Human Adjudication | human-required | 0/5 decisions; pending 5 | Real reviewer decisions recorded before claiming output review completion | Record real A/B choices in the decision template, then regenerate adjudication. | -| Benchmark Reproducibility | pass | artifacts 20; missing 0; failures 3 | Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable | Keep the manifest current with every benchmark, package, and release evidence change. | +| Benchmark Reproducibility | pass | artifacts 24; missing 0; failures 3 | Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable | Keep the manifest current with every benchmark, package, and release evidence change. | | Runtime Conformance | pass | 5/5 targets pass | Target package structure, metadata, relative paths, and degradation notes pass | Keep target conformance fixtures updated as platform contracts change. | | Trust Security | pass | secrets 0; scripts 94; help failures 0 | Secrets, scripts, dependencies, permissions, and package hash are reviewable | Keep high-permission approvals scoped, expiring, and target-mapped. | | Permission Metadata | pass | 4/4 target probes pass; metadata fallback 4; installer enforcement 4 | Packaged adapters expose explicit permission metadata, residual risks, and installer enforcement evidence when available | Preserve residual-risk notes until real native enforcement exists. | diff --git a/reports/skill_os2_coverage.json b/reports/skill_os2_coverage.json index 654e158..7f768f2 100644 --- a/reports/skill_os2_coverage.json +++ b/reports/skill_os2_coverage.json @@ -305,7 +305,7 @@ "label": "Benchmark Methodology", "status": "pass", "objective": "Recommended Skill OS 2.0 implementation PR from the upgrade plan.", - "current": "20 required artifacts checked", + "current": "24 required artifacts checked", "command": "make ci-test", "test": "tests/verify_benchmark_reproducibility.py", "evidence": [ diff --git a/reports/skill_os2_coverage.md b/reports/skill_os2_coverage.md index 33af700..d60139b 100644 --- a/reports/skill_os2_coverage.md +++ b/reports/skill_os2_coverage.md @@ -31,7 +31,7 @@ This report maps the Skill OS 2.0 upgrade blueprint to concrete local artifacts, | Item | Status | Current | Command | Test | | --- | --- | --- | --- | --- | -| Benchmark Methodology | `pass` | 20 required artifacts checked | `make ci-test` | `tests/verify_benchmark_reproducibility.py` | +| Benchmark Methodology | `pass` | 24 required artifacts checked | `make ci-test` | `tests/verify_benchmark_reproducibility.py` | | Output Eval Schema | `pass` | 5 output cases | `make ci-test` | `tests/verify_output_eval_lab.py` | | Output Eval Runner | `pass` | delta 100.0 | `make ci-test` | `tests/verify_output_eval_lab.py` | | Output Quality Scorecard | `pass` | gate pass True | `make ci-test` | `tests/verify_output_eval_lab.py` | diff --git a/reports/upgrade_check.json b/reports/upgrade_check.json index 43cf110..d2ad6b1 100644 --- a/reports/upgrade_check.json +++ b/reports/upgrade_check.json @@ -70,12 +70,12 @@ { "field": "archive_sha256", "from": "", - "to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e" + "to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179" + "to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d" } ] }, diff --git a/scripts/render_benchmark_reproducibility.py b/scripts/render_benchmark_reproducibility.py index 4ee704d..17dc93f 100644 --- a/scripts/render_benchmark_reproducibility.py +++ b/scripts/render_benchmark_reproducibility.py @@ -41,6 +41,10 @@ REQUIRED_ARTIFACTS = [ ("world_class_evidence_plan", "reports/world_class_evidence_plan.json"), ("world_class_evidence_ledger", "reports/world_class_evidence_ledger.json"), ("world_class_evidence_intake", "reports/world_class_evidence_intake.json"), + ("world_class_submission_review", "reports/world_class_submission_review.json"), + ("world_class_operator_runbook", "reports/world_class_operator_runbook.json"), + ("world_class_operator_runbook_markdown", "reports/world_class_operator_runbook.md"), + ("world_class_operator_runbook_html", "reports/world_class_operator_runbook.html"), ("world_class_claim_guard", "reports/world_class_claim_guard.json"), ] @@ -130,6 +134,16 @@ REPRODUCTION_COMMANDS = [ "command": "python3 scripts/yao.py world-class-intake .", "evidence": "reports/world_class_evidence_intake.json", }, + { + "label": "world-class submission review", + "command": "python3 scripts/yao.py world-class-submission-review .", + "evidence": "reports/world_class_submission_review.json", + }, + { + "label": "world-class operator runbook", + "command": "python3 scripts/yao.py world-class-runbook .", + "evidence": "reports/world_class_operator_runbook.json", + }, { "label": "world-class claim guard", "command": "python3 scripts/yao.py world-class-claim-guard .", diff --git a/tests/verify_benchmark_reproducibility.py b/tests/verify_benchmark_reproducibility.py index 315069b..4c39196 100644 --- a/tests/verify_benchmark_reproducibility.py +++ b/tests/verify_benchmark_reproducibility.py @@ -40,6 +40,32 @@ def main() -> None: text=True, check=True, ) + subprocess.run( + [ + sys.executable, + str(ROOT / "scripts" / "render_world_class_submission_review.py"), + str(ROOT), + "--generated-at", + "2026-06-13", + ], + cwd=ROOT, + capture_output=True, + text=True, + check=True, + ) + subprocess.run( + [ + sys.executable, + str(ROOT / "scripts" / "render_world_class_operator_runbook.py"), + str(ROOT), + "--generated-at", + "2026-06-13", + ], + cwd=ROOT, + capture_output=True, + text=True, + check=True, + ) subprocess.run( [ sys.executable, @@ -108,17 +134,25 @@ def main() -> None: assert artifacts["reports/world_class_evidence_plan.json"]["exists"], artifacts assert artifacts["reports/world_class_evidence_ledger.json"]["exists"], artifacts assert artifacts["reports/world_class_evidence_intake.json"]["exists"], artifacts + assert artifacts["reports/world_class_submission_review.json"]["exists"], artifacts + assert artifacts["reports/world_class_operator_runbook.json"]["exists"], artifacts + assert artifacts["reports/world_class_operator_runbook.md"]["exists"], artifacts + assert artifacts["reports/world_class_operator_runbook.html"]["exists"], artifacts assert artifacts["reports/world_class_claim_guard.json"]["exists"], artifacts assert artifacts["reports/python_compatibility.json"]["exists"], artifacts assert any(command["command"] == "make ci-test" for command in payload["reproduction_commands"]), payload assert any(command["command"] == "python3 scripts/yao.py world-class-ledger ." for command in payload["reproduction_commands"]), payload assert any(command["command"] == "python3 scripts/yao.py world-class-intake ." for command in payload["reproduction_commands"]), payload + assert any(command["command"] == "python3 scripts/yao.py world-class-submission-review ." for command in payload["reproduction_commands"]), payload + assert any(command["command"] == "python3 scripts/yao.py world-class-runbook ." for command in payload["reproduction_commands"]), payload assert any(command["command"] == "python3 scripts/yao.py world-class-claim-guard ." for command in payload["reproduction_commands"]), payload assert any(command["command"] == "python3 scripts/yao.py python-compat ." for command in payload["reproduction_commands"]), payload assert any("provider-backed" in item for item in payload["limitations"]), payload["limitations"] markdown = output_md.read_text(encoding="utf-8") assert "Benchmark Reproducibility" in markdown, markdown assert "reports/benchmark_methodology.md" in markdown, markdown + assert "reports/world_class_operator_runbook.html" in markdown, markdown + assert "python3 scripts/yao.py world-class-runbook ." in markdown, markdown assert "make ci-test" in markdown, markdown print(json.dumps({"ok": True}, ensure_ascii=False, indent=2))