frostbyte_neo/yao-meta-skill
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

feat: include world-class runbook in benchmark evidence

Browse Source
This commit is contained in:
yaojingang
2026-06-14 08:19:16 +08:00
parent 5d1d3bbd30
commit bd7f1d8ccc
25 changed files with 227 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+6 -1
View File
@@ -28,6 +28,8 @@ It turns rough workflows, transcripts, prompts, notes, and runbooks into reusabl
- a world-class evidence plan that turns remaining provider, human, native-permission, and real-client telemetry gaps into executable evidence tasks
- a world-class evidence ledger that records which external and human evidence is accepted or still pending without treating planned work as proof
- a world-class evidence intake contract that validates external and human evidence packets for provenance, privacy, artifact refs, and anti-overclaim rules before ledger review
- a world-class submission review queue that compares evidence packets, intake validation, source artifacts, and ledger state without accepting evidence
- a world-class operator runbook that gives reviewers the exact commands, artifacts, and collection checklist needed to close remaining evidence gaps
- a world-class claim guard that scans public claim surfaces and blocks premature completed/true claims while the evidence ledger still has pending external or human evidence
- a benchmark reproducibility manifest that checks methodology sections, required artifacts, failure disclosure, and reproduction commands
- Output Eval Lab evidence with assertion grading, execution/timing/token evidence, a blind A/B review pack, a separate answer key, and reviewer adjudication reports
@@ -147,6 +149,7 @@ python3 scripts/yao.py world-class-submission-kit . --output-dir /tmp/yao-world-
python3 scripts/yao.py world-class-ledger .
python3 scripts/yao.py world-class-intake .
python3 scripts/yao.py world-class-submission-review .
python3 scripts/yao.py world-class-runbook .
python3 scripts/yao.py world-class-claim-guard .
python3 scripts/yao.py benchmark-reproducibility .
```
@@ -298,6 +301,7 @@ Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/fa
- current world-class evidence acceptance state is published in [reports/world_class_evidence_ledger.md](reports/world_class_evidence_ledger.md)
- world-class evidence intake readiness is published in [reports/world_class_evidence_intake.md](reports/world_class_evidence_intake.md)
- world-class submission review queue is published in [reports/world_class_submission_review.md](reports/world_class_submission_review.md)
- world-class operator runbook is published in [reports/world_class_operator_runbook.md](reports/world_class_operator_runbook.md) and [reports/world_class_operator_runbook.html](reports/world_class_operator_runbook.html)
- world-class public claim guard status is published in [reports/world_class_claim_guard.md](reports/world_class_claim_guard.md)
- benchmark reproducibility evidence is published in [reports/benchmark_reproducibility.md](reports/benchmark_reproducibility.md)
- target compiler evidence is published in [reports/compiled_targets.md](reports/compiled_targets.md)
@@ -416,7 +420,7 @@ Utility scripts that make the meta-skill operational:
- `run_description_optimization_suite.py`: runs description optimization across the root skill and governed examples, then writes reusable reports and optional drift snapshots with calibration and family summaries
- `promotion_checker.py`: applies promotion policy to current description candidates, writes promotion decisions, builds candidate registries, and emits iteration bundles with review stubs
- `create_iteration_snapshot.py`: freezes the current promotion decision into a versioned release snapshot with review, route, and context evidence
- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, python-compat, review, release-snapshot, workspace-flow, report, skill-ir, compile-skill, output-exec, output-review, skill-os2-audit, skill-os2-coverage, world-class-evidence, world-class-ledger, world-class-intake, world-class-submission-kit, world-class-submission-review, world-class-claim-guard, benchmark-reproducibility, telemetry-emit, telemetry-hooks, telemetry-import, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint
- `yao.py`: unified authoring CLI that exposes init, validate, optimize-description, promote-check, python-compat, review, release-snapshot, workspace-flow, report, skill-ir, compile-skill, output-exec, output-review, skill-os2-audit, skill-os2-coverage, world-class-evidence, world-class-ledger, world-class-intake, world-class-submission-kit, world-class-submission-review, world-class-runbook, world-class-claim-guard, benchmark-reproducibility, telemetry-emit, telemetry-hooks, telemetry-import, package, registry-audit, package-verify, install-simulate, upgrade-check, review-waivers, and test as one entrypoint
- `render_description_drift_history.py`: turns description-optimization snapshots into a readable drift-history report
- `build_confusion_matrix.py`: scores route confusion across tracked sibling skills and `no_route` cases, then writes a route scorecard and optional milestone snapshot
- `render_iteration_ledger.py`: compresses regression milestones, description optimization drift, and route scorecards into one iteration-facing ledger
@@ -431,6 +435,7 @@ Utility scripts that make the meta-skill operational:
- `render_world_class_evidence_ledger.py`: renders a machine-checkable ledger for current world-class evidence acceptance, anti-overclaim guards, provenance requirements, and privacy contracts
- `render_world_class_evidence_intake.py`: validates world-class external and human evidence packets against provenance, privacy, artifact, and anti-overclaim requirements before ledger review
- `render_world_class_submission_review.py`: renders a read-only queue that compares submissions, intake validation, source evidence, and ledger state without accepting evidence
- `render_world_class_operator_runbook.py`: renders an operator-facing checklist and command map for collecting pending world-class evidence without accepting evidence
- `render_world_class_claim_guard.py`: scans README, docs, and reports for premature world-class completion claims while accepted evidence is still pending
- `render_benchmark_reproducibility.py`: renders methodology, artifact, failure-disclosure, and reproduction-command evidence for public benchmark claims
- `python_compat_check.py`: checks Python source for supported-runtime compatibility hazards such as Python 3.11 f-string expression backslashes
registry/index.json
+1 -1
View File
@@ -16,7 +16,7 @@
"vscode"
],
"package_metadata": "registry/packages/yao-meta-skill.json",
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
}
registry/packages/yao-meta-skill.json
+3 -3
View File
@@ -16,8 +16,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179",
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
"compatibility": {
"openai": "pass",
@@ -48,7 +48,7 @@
},
"distribution": {
"archive_verified": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
reports/adoption_drift_report.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"ok": true,
"schema_version": "2.0",
"generated_at": "2026-06-14T00:05:43Z",
"generated_at": "2026-06-14T00:16:48Z",
"skill_dir": ".",
"privacy_contract": {
"storage": "local-first",
reports/benchmark_reproducibility.json
+65 -27
View File
@@ -3,34 +3,34 @@
"ok": true,
"generated_at": "2026-06-14",
"skill_dir": ".",
"commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1",
"commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03",
"git_status": {
"available": true,
"dirty": true,
"changed_file_count": 61,
"changed_file_count": 25,
"sample": [
"M AGENTS.md",
"M Makefile",
"M registry/index.json",
"M registry/packages/yao-meta-skill.json",
"MM reports/adoption_drift_report.json",
" M reports/adoption_drift_report.md",
"MM reports/architecture_maintainability.json",
"MM reports/architecture_maintainability.md",
"MM reports/benchmark_reproducibility.json",
"MM reports/benchmark_reproducibility.md",
"MM reports/compiled_targets.json",
"MM reports/context_budget.json"
" M README.md",
" M registry/index.json",
" M registry/packages/yao-meta-skill.json",
" M reports/adoption_drift_report.json",
" M reports/benchmark_reproducibility.json",
" M reports/benchmark_reproducibility.md",
" M reports/context_budget.json",
" M reports/output_execution_runs.json",
" M reports/output_execution_runs.md",
" M reports/package_verification.json",
" M reports/package_verification.md",
" M reports/registry_audit.json"
]
},
"summary": {
"reproducibility_ready": true,
"methodology_complete": true,
"required_artifact_count": 20,
"required_artifact_count": 24,
"missing_artifact_count": 0,
"output_case_count": 5,
"failure_disclosure_count": 3,
"command_count": 19,
"command_count": 21,
"command_executed_count": 10,
"timing_observed_count": 10,
"model_executed_count": 0,
@@ -42,7 +42,7 @@
"world_class_task_count": 4,
"world_class_ledger_pending_count": 4,
"working_tree_dirty": true,
"changed_file_count": 61
"changed_file_count": 25
},
"methodology": {
"path": "reports/benchmark_methodology.md",
@@ -116,7 +116,7 @@
"path": "reports/output_execution_runs.json",
"exists": true,
"bytes": 7965,
"sha256": "99b2e3f03710fdcee8fe66a476bc589f4e22a379bfcc1cd5561de49c184286e8"
"sha256": "e4b3b5bf45a34ad8a616015289e968fdb13ec9a9ea1b15ff2ea94ef6cec40d7b"
},
{
"label": "blind_review",
@@ -151,56 +151,56 @@
"path": "reports/security_trust_report.json",
"exists": true,
"bytes": 97005,
"sha256": "252c580028df6ac2404b53cc10bb607fa82b98ef17965cb7354c4731f6c6b08e"
"sha256": "f518d733e79e26ea36136c1e38e22e36922481724bb6c004a7475c73bfa9fbb6"
},
{
"label": "python_compatibility",
"path": "reports/python_compatibility.json",
"exists": true,
"bytes": 20367,
"sha256": "a8d14087863459534fadac760a64c213c9474413e92046a9ac2e772d4174d347"
"sha256": "3698aad5388cad659840d1d31dbc9c694e0067b1de1e84423336a34e22df9625"
},
{
"label": "registry_audit",
"path": "reports/registry_audit.json",
"exists": true,
"bytes": 3183,
"sha256": "6527b723a4bc367477e1090afbf7f1653f20d33fed0e0691725b7ee30896848b"
"sha256": "90137332142ac7ff036f1efd5eb1970c31baf60e8a5f8c541dc12aadc91492b6"
},
{
"label": "package_verification",
"path": "reports/package_verification.json",
"exists": true,
"bytes": 19325,
"sha256": "6bf19c9e29803ad8668914489d22b9e306ad0ad092c91c179c3aaffe81eed2c6"
"sha256": "6c62c89994790f7034e115e349dd4627939e5c73361c36d5a56c887346405f90"
},
{
"label": "install_simulation",
"path": "reports/install_simulation.json",
"exists": true,
"bytes": 8758,
"sha256": "270a28b82dc49d80ba9c5a82f912cc960e50a5ac42f022c6a2c345e4e84fbb36"
"bytes": 8604,
"sha256": "97055f63c699b2812bab1a0f05460f1065a6211164b4c22c9bebc8aab6ef012f"
},
{
"label": "skill_os2_audit",
"path": "reports/skill_os2_audit.json",
"exists": true,
"bytes": 14309,
"sha256": "cad5296cd500a40cb0d83897d327b3542f31c1ce6bfe7eaa387ea2e88d229aa1"
"sha256": "9c7b86742ed4dd5c930fbc3572d943937a4817bbd465d89da12f0929911607d0"
},
{
"label": "world_class_evidence_plan",
"path": "reports/world_class_evidence_plan.json",
"exists": true,
"bytes": 10083,
"sha256": "02e49b66159d239309bd31be33f2119db23604b5506c4445ba6c6c84b6cdd1aa"
"sha256": "194d5a681920c3aca5e454d8839cdec551a076d54e360424831c5f312d1367b1"
},
{
"label": "world_class_evidence_ledger",
"path": "reports/world_class_evidence_ledger.json",
"exists": true,
"bytes": 11308,
"sha256": "0b52d0d1528ba95a0fa7bead97cb8224b5b189d35497a05d90956fc90512b651"
"sha256": "dc4cf8cf18899bdf10d32fe72bb8f306ee297106e6cc25fd041d56e36631af63"
},
{
"label": "world_class_evidence_intake",
@@ -209,6 +209,34 @@
"bytes": 12740,
"sha256": "5fbfcd35ac6afd494c64845a7130520fd1cffbdb86e4efed2eaf2fa3d381f8b3"
},
{
"label": "world_class_submission_review",
"path": "reports/world_class_submission_review.json",
"exists": true,
"bytes": 7263,
"sha256": "185a8cab25e7155ebd80cb66049c0965e4dc2d7cc38991f17464946f87fd26f2"
},
{
"label": "world_class_operator_runbook",
"path": "reports/world_class_operator_runbook.json",
"exists": true,
"bytes": 15002,
"sha256": "3d643dc8170b50a37f91ffb190f69ad1e0d32f64914ebe2d731226728ac28cdc"
},
{
"label": "world_class_operator_runbook_markdown",
"path": "reports/world_class_operator_runbook.md",
"exists": true,
"bytes": 9206,
"sha256": "f67ca3d0aa9c02953e967d9657ca4cfe41ee883c73950bce16d6c25a1e206e10"
},
{
"label": "world_class_operator_runbook_html",
"path": "reports/world_class_operator_runbook.html",
"exists": true,
"bytes": 13191,
"sha256": "963026f551dc687ae53e7ba1df5003dd59292c259080543dcaf4b4343437fed6"
},
{
"label": "world_class_claim_guard",
"path": "reports/world_class_claim_guard.json",
@@ -304,6 +332,16 @@
"command": "python3 scripts/yao.py world-class-intake .",
"evidence": "reports/world_class_evidence_intake.json"
},
{
"label": "world-class submission review",
"command": "python3 scripts/yao.py world-class-submission-review .",
"evidence": "reports/world_class_submission_review.json"
},
{
"label": "world-class operator runbook",
"command": "python3 scripts/yao.py world-class-runbook .",
"evidence": "reports/world_class_operator_runbook.json"
},
{
"label": "world-class claim guard",
"command": "python3 scripts/yao.py world-class-claim-guard .",
reports/benchmark_reproducibility.md
+21 -13
View File
@@ -1,22 +1,22 @@
# Benchmark Reproducibility
Generated at: `2026-06-14`
Commit: `08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1`
Commit: `5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03`
Working tree dirty at generation: `true`
## Summary
- reproducibility ready: `true`
- methodology complete: `true`
- required artifacts: `20`
- required artifacts: `24`
- missing artifacts: `0`
- output cases: `5`
- disclosed failure cases: `3`
- reproduction commands: `19`
- reproduction commands: `21`
- provider evidence complete: `false`
- human review complete: `false`
- world-class ready: `false`
- changed files at generation: `61`
- changed files at generation: `25`
This report proves local benchmark reproducibility only. It keeps external provider and human-review gaps visible instead of counting them as complete.
@@ -40,20 +40,24 @@ This report proves local benchmark reproducibility only. It keeps external provi
| output_cases | `evals/output/cases.jsonl` | present | `a6ae96857116` |
| output_schema | `evals/output/schema.json` | present | `8ee340c95064` |
| output_scorecard | `reports/output_quality_scorecard.json` | present | `0806258a8e08` |
| output_execution | `reports/output_execution_runs.json` | present | `99b2e3f03710` |
| output_execution | `reports/output_execution_runs.json` | present | `e4b3b5bf45a3` |
| blind_review | `reports/output_blind_review_pack.json` | present | `bbe2db8ec277` |
| review_adjudication | `reports/output_review_adjudication.json` | present | `240485a721af` |
| trigger_scorecard | `reports/route_scorecard.json` | present | `c164e83e36d0` |
| runtime_conformance | `reports/conformance_matrix.json` | present | `8251329e663d` |
| trust_report | `reports/security_trust_report.json` | present | `252c580028df` |
| python_compatibility | `reports/python_compatibility.json` | present | `a8d140878634` |
| registry_audit | `reports/registry_audit.json` | present | `6527b723a4bc` |
| package_verification | `reports/package_verification.json` | present | `6bf19c9e2980` |
| install_simulation | `reports/install_simulation.json` | present | `270a28b82dc4` |
| skill_os2_audit | `reports/skill_os2_audit.json` | present | `cad5296cd500` |
| world_class_evidence_plan | `reports/world_class_evidence_plan.json` | present | `02e49b66159d` |
| world_class_evidence_ledger | `reports/world_class_evidence_ledger.json` | present | `0b52d0d1528b` |
| trust_report | `reports/security_trust_report.json` | present | `f518d733e79e` |
| python_compatibility | `reports/python_compatibility.json` | present | `3698aad5388c` |
| registry_audit | `reports/registry_audit.json` | present | `90137332142a` |
| package_verification | `reports/package_verification.json` | present | `6c62c8999479` |
| install_simulation | `reports/install_simulation.json` | present | `97055f63c699` |
| skill_os2_audit | `reports/skill_os2_audit.json` | present | `9c7b86742ed4` |
| world_class_evidence_plan | `reports/world_class_evidence_plan.json` | present | `194d5a681920` |
| world_class_evidence_ledger | `reports/world_class_evidence_ledger.json` | present | `dc4cf8cf1889` |
| world_class_evidence_intake | `reports/world_class_evidence_intake.json` | present | `5fbfcd35ac6a` |
| world_class_submission_review | `reports/world_class_submission_review.json` | present | `185a8cab25e7` |
| world_class_operator_runbook | `reports/world_class_operator_runbook.json` | present | `3d643dc8170b` |
| world_class_operator_runbook_markdown | `reports/world_class_operator_runbook.md` | present | `f67ca3d0aa9c` |
| world_class_operator_runbook_html | `reports/world_class_operator_runbook.html` | present | `963026f551dc` |
| world_class_claim_guard | `reports/world_class_claim_guard.json` | present | `250d616b028c` |
## Reproduction Commands
@@ -92,6 +96,10 @@ This report proves local benchmark reproducibility only. It keeps external provi
- evidence: `reports/world_class_evidence_ledger.json`
- `python3 scripts/yao.py world-class-intake .`
- evidence: `reports/world_class_evidence_intake.json`
- `python3 scripts/yao.py world-class-submission-review .`
- evidence: `reports/world_class_submission_review.json`
- `python3 scripts/yao.py world-class-runbook .`
- evidence: `reports/world_class_operator_runbook.json`
- `python3 scripts/yao.py world-class-claim-guard .`
- evidence: `reports/world_class_claim_guard.json`
- `make ci-test`
reports/context_budget.json
+2 -2
View File
@@ -6,9 +6,9 @@
"context_budget_tier": "production",
"context_budget_limit": 1000,
"skill_body_tokens": 751,
"other_text_tokens": 1175718,
"other_text_tokens": 1176737,
"estimated_initial_load_tokens": 944,
"estimated_total_text_tokens": 1176469,
"estimated_total_text_tokens": 1177488,
"relevant_file_count": 498,
"unused_resource_dirs": [],
"quality_signal_points": 130,
reports/output_execution_runs.json
+9 -9
View File
@@ -34,7 +34,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 26.47,
"duration_ms": 28.47,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -62,7 +62,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.78,
"duration_ms": 28.4,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -85,7 +85,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.73,
"duration_ms": 26.37,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -136,7 +136,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 29.2,
"duration_ms": 28.04,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -164,7 +164,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 28.81,
"duration_ms": 29.89,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -187,7 +187,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.65,
"duration_ms": 26.27,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -214,7 +214,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 26.22,
"duration_ms": 25.7,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -237,7 +237,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.43,
"duration_ms": 25.72,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -266,7 +266,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.88,
"duration_ms": 25.67,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
reports/output_execution_runs.md
+9 -9
View File
@@ -23,16 +23,16 @@ Command runner evidence is present. This proves the eval harness executed an ext
| Case | Variant | Mode | Model | Duration ms | Tokens | Score | Status |
| --- | --- | --- | --- | ---: | ---: | ---: | --- |
| skill-package-contract | baseline | command | local-output-eval-runner | 26.47 | 33 | 0.0 | pass |
| skill-package-contract | with_skill | command | local-output-eval-runner | 25.78 | 73 | 100.0 | pass |
| output-eval-expectation | baseline | command | local-output-eval-runner | 25.73 | 36 | 0.0 | pass |
| skill-package-contract | baseline | command | local-output-eval-runner | 28.47 | 33 | 0.0 | pass |
| skill-package-contract | with_skill | command | local-output-eval-runner | 28.4 | 73 | 100.0 | pass |
| output-eval-expectation | baseline | command | local-output-eval-runner | 26.37 | 36 | 0.0 | pass |
| output-eval-expectation | with_skill | command | local-output-eval-runner | 29.11 | 80 | 100.0 | pass |
| ir-before-packaging | baseline | command | local-output-eval-runner | 29.2 | 33 | 0.0 | pass |
| ir-before-packaging | with_skill | command | local-output-eval-runner | 28.81 | 80 | 100.0 | pass |
| near-neighbor-boundary | baseline | command | local-output-eval-runner | 25.65 | 36 | 0.0 | pass |
| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 26.22 | 65 | 100.0 | pass |
| file-backed-governed-package | baseline | command | local-output-eval-runner | 25.43 | 37 | 0.0 | pass |
| file-backed-governed-package | with_skill | command | local-output-eval-runner | 25.88 | 98 | 100.0 | pass |
| ir-before-packaging | baseline | command | local-output-eval-runner | 28.04 | 33 | 0.0 | pass |
| ir-before-packaging | with_skill | command | local-output-eval-runner | 29.89 | 80 | 100.0 | pass |
| near-neighbor-boundary | baseline | command | local-output-eval-runner | 26.27 | 36 | 0.0 | pass |
| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 25.7 | 65 | 100.0 | pass |
| file-backed-governed-package | baseline | command | local-output-eval-runner | 25.72 | 37 | 0.0 | pass |
| file-backed-governed-package | with_skill | command | local-output-eval-runner | 25.67 | 98 | 100.0 | pass |
## Next Fixes
reports/package_verification.json
+1 -1
View File
@@ -8,7 +8,7 @@
"target_count": 4,
"adapter_count": 4,
"archive_present": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"archive_entry_count": 573,
"failure_count": 0,
"warning_count": 0
reports/package_verification.md
+1 -1
View File
@@ -4,7 +4,7 @@
- Package directory: `dist`
- Targets: `4 / 4` adapters present
- Archive present: `True`
- Archive SHA256: `de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e`
- Archive SHA256: `5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27`
- Failures: `0`
- Warnings: `0`
reports/registry_audit.json
+4 -4
View File
@@ -21,8 +21,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179",
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
"compatibility": {
"openai": "pass",
@@ -53,7 +53,7 @@
},
"distribution": {
"archive_verified": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
@@ -78,7 +78,7 @@
"vscode"
],
"package_metadata": "registry/packages/yao-meta-skill.json",
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
reports/registry_audit.md
+2 -2
View File
@@ -6,8 +6,8 @@
- Maturity: `governed`
- Owner: `Yao Team`
- License: `MIT`
- Package SHA256: `97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179`
- Archive SHA256: `de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e`
- Package SHA256: `2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d`
- Archive SHA256: `5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27`
- Install simulated: `True`
## Compatibility
reports/review-studio.html
+3 -3
View File
@@ -506,7 +506,7 @@
</section>
<section id="trust" class="twocol">
<div class="panel"><h2>信任报告</h2><dl class='kv-grid'><div><dt>Secret</dt><dd>0</dd></div><div><dt>脚本数</dt><dd>94</dd></div><div><dt>网络脚本</dt><dd>3</dd></div><div><dt>Help 失败</dt><dd>0</dd></div><div><dt>包体哈希</dt><dd><code>97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179</code></dd></div></dl></div>
<div class="panel"><h2>信任报告</h2><dl class='kv-grid'><div><dt>Secret</dt><dd>0</dd></div><div><dt>脚本数</dt><dd>94</dd></div><div><dt>网络脚本</dt><dd>3</dd></div><div><dt>Help 失败</dt><dd>0</dd></div><div><dt>包体哈希</dt><dd><code>2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d</code></dd></div></dl></div>
<div class="panel"><h2>安全边界</h2><p>高风险 secret、远程 inline execution、缺失依赖策略或无法解释的脚本接口应阻断 governed release。</p></div>
</section>
@@ -574,12 +574,12 @@
<section id="registry" class="twocol">
<div class="panel"><h2>注册审计</h2><p>yao-meta-skill 1.1.0; 6/6 compatibility entries pass; install pass with 4 adapters; installer permissions 12 enforced / 0 failures</p></div>
<div class="panel"><h2>包体元数据</h2><dl class='kv-grid'><div><dt>名称</dt><dd>yao-meta-skill</dd></div><div><dt>版本</dt><dd>1.1.0</dd></div><div><dt>Maturity</dt><dd>governed</dd></div><div><dt>Owner</dt><dd>Yao Team</dd></div><div><dt>License</dt><dd>MIT</dd></div><div><dt>信任级别</dt><dd>local</dd></div><div><dt>目标平台</dt><dd>openai, claude, generic, agent-skills-compatible, vscode</dd></div><div><dt>兼容通过</dt><dd>6/6</dd></div><div><dt>归档哈希</dt><dd><code>de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e</code></dd></div></dl></div>
<div class="panel"><h2>包体元数据</h2><dl class='kv-grid'><div><dt>名称</dt><dd>yao-meta-skill</dd></div><div><dt>版本</dt><dd>1.1.0</dd></div><div><dt>Maturity</dt><dd>governed</dd></div><div><dt>Owner</dt><dd>Yao Team</dd></div><div><dt>License</dt><dd>MIT</dd></div><div><dt>信任级别</dt><dd>local</dd></div><div><dt>目标平台</dt><dd>openai, claude, generic, agent-skills-compatible, vscode</dd></div><div><dt>兼容通过</dt><dd>6/6</dd></div><div><dt>归档哈希</dt><dd><code>5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27</code></dd></div></dl></div>
</section>
<section id="release" class="twocol">
<div class="panel"><h2>发布路线</h2><p>0 promote; 3 keep current; 0 blocked; upgrade minor declared / minor recommended</p></div>
<div class="panel"><h2>包体验证</h2><dl class='kv-grid'><div><dt>目标数</dt><dd>4</dd></div><div><dt>Adapter</dt><dd>4</dd></div><div><dt>归档存在</dt><dd>是</dd></div><div><dt>Zip 条目</dt><dd>573</dd></div><div><dt>失败数</dt><dd>0</dd></div><div><dt>警告数</dt><dd>0</dd></div><div><dt>归档哈希</dt><dd><code>de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e</code></dd></div></dl></div>
<div class="panel"><h2>包体验证</h2><dl class='kv-grid'><div><dt>目标数</dt><dd>4</dd></div><div><dt>Adapter</dt><dd>4</dd></div><div><dt>归档存在</dt><dd>是</dd></div><div><dt>Zip 条目</dt><dd>573</dd></div><div><dt>失败数</dt><dd>0</dd></div><div><dt>警告数</dt><dd>0</dd></div><div><dt>归档哈希</dt><dd><code>5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27</code></dd></div></dl></div>
</section>
<section class="twocol">
reports/review-studio.json
+32 -32
View File
@@ -1333,11 +1333,11 @@
"summary": {
"reproducibility_ready": true,
"methodology_complete": true,
"required_artifact_count": 20,
"required_artifact_count": 24,
"missing_artifact_count": 0,
"output_case_count": 5,
"failure_disclosure_count": 3,
"command_count": 19,
"command_count": 21,
"command_executed_count": 10,
"timing_observed_count": 10,
"model_executed_count": 0,
@@ -1349,9 +1349,9 @@
"world_class_task_count": 4,
"world_class_ledger_pending_count": 4,
"working_tree_dirty": true,
"changed_file_count": 61
"changed_file_count": 41
},
"commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1",
"commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03",
"missing_artifacts": [],
"limitations": [
"Local command-runner evidence is reproducible but does not replace provider-backed model holdout evidence.",
@@ -1455,7 +1455,7 @@
"interactive_script_count": 0,
"package_hash_scope": "source-contract-without-generated-reports",
"package_hash_file_count": 178,
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
},
"skill_atlas": {
"skill_count": 12,
@@ -1493,8 +1493,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179",
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
"compatibility": {
"openai": "pass",
@@ -1525,7 +1525,7 @@
},
"distribution": {
"archive_verified": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
@@ -1541,7 +1541,7 @@
"target_count": 4,
"adapter_count": 4,
"archive_present": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"archive_entry_count": 573,
"failure_count": 0,
"warning_count": 0
@@ -1620,12 +1620,12 @@
{
"field": "archive_sha256",
"from": "",
"to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
{
"field": "package_sha256",
"from": "0000000000000000000000000000000000000000000000000000000000000000",
"to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
@@ -3794,7 +3794,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 26.47,
"duration_ms": 28.47,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3822,7 +3822,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.78,
"duration_ms": 28.4,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3845,7 +3845,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.73,
"duration_ms": 26.37,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3896,7 +3896,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 29.2,
"duration_ms": 28.04,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3924,7 +3924,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 28.81,
"duration_ms": 29.89,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3947,7 +3947,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.65,
"duration_ms": 26.27,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3974,7 +3974,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 26.22,
"duration_ms": 25.7,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -3997,7 +3997,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.43,
"duration_ms": 25.72,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -4026,7 +4026,7 @@
"execution_mode": "command",
"model_executed": false,
"command_executed": true,
"duration_ms": 25.88,
"duration_ms": 25.67,
"provider": "local-output-eval-runner",
"model": "",
"usage": {
@@ -5049,7 +5049,7 @@
"label": "Benchmark Methodology",
"status": "pass",
"objective": "Recommended Skill OS 2.0 implementation PR from the upgrade plan.",
"current": "20 required artifacts checked",
"current": "24 required artifacts checked",
"command": "make ci-test",
"test": "tests/verify_benchmark_reproducibility.py",
"evidence": [
@@ -10242,7 +10242,7 @@
"interactive_script_count": 0,
"package_hash_scope": "source-contract-without-generated-reports",
"package_hash_file_count": 178,
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
},
"failures": [],
"warnings": [],
@@ -13724,9 +13724,9 @@
"context_budget_tier": "production",
"context_budget_limit": 1000,
"skill_body_tokens": 751,
"other_text_tokens": 1175718,
"other_text_tokens": 1176737,
"estimated_initial_load_tokens": 944,
"estimated_total_text_tokens": 1176469,
"estimated_total_text_tokens": 1177488,
"relevant_file_count": 498,
"unused_resource_dirs": [],
"quality_signal_points": 130,
@@ -15580,7 +15580,7 @@
"adoption_drift": {
"ok": true,
"schema_version": "2.0",
"generated_at": "2026-06-14T00:05:43Z",
"generated_at": "2026-06-14T00:16:48Z",
"skill_dir": ".",
"privacy_contract": {
"storage": "local-first",
@@ -17169,8 +17169,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179",
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
"compatibility": {
"openai": "pass",
@@ -17201,7 +17201,7 @@
},
"distribution": {
"archive_verified": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
@@ -17226,7 +17226,7 @@
"vscode"
],
"package_metadata": "registry/packages/yao-meta-skill.json",
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
@@ -17249,7 +17249,7 @@
"target_count": 4,
"adapter_count": 4,
"archive_present": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"archive_entry_count": 573,
"failure_count": 0,
"warning_count": 0
@@ -18257,12 +18257,12 @@
{
"field": "archive_sha256",
"from": "",
"to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
{
"field": "package_sha256",
"from": "0000000000000000000000000000000000000000000000000000000000000000",
"to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
reports/security_trust_report.json
+1 -1
View File
@@ -23,7 +23,7 @@
"interactive_script_count": 0,
"package_hash_scope": "source-contract-without-generated-reports",
"package_hash_file_count": 178,
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
},
"failures": [],
"warnings": [],
reports/security_trust_report.md
+1 -1
View File
@@ -16,7 +16,7 @@
- Interactive scripts: `0`
- Package hash scope: `source-contract-without-generated-reports`
- Package hash files: `178`
- Package SHA256: `97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179`
- Package SHA256: `2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d`
## Failures
reports/skill-overview.json
+11 -11
View File
@@ -919,11 +919,11 @@
"summary": {
"reproducibility_ready": true,
"methodology_complete": true,
"required_artifact_count": 20,
"required_artifact_count": 24,
"missing_artifact_count": 0,
"output_case_count": 5,
"failure_disclosure_count": 3,
"command_count": 19,
"command_count": 21,
"command_executed_count": 10,
"timing_observed_count": 10,
"model_executed_count": 0,
@@ -935,9 +935,9 @@
"world_class_task_count": 4,
"world_class_ledger_pending_count": 4,
"working_tree_dirty": true,
"changed_file_count": 61
"changed_file_count": 41
},
"commit": "08347d5c2cae9528ae1fe3c50ad39bb3c773a6e1",
"commit": "5d1d3bbd302aa9c6d0e5f5767c5bdc776b517a03",
"missing_artifacts": [],
"limitations": [
"Local command-runner evidence is reproducible but does not replace provider-backed model holdout evidence.",
@@ -1041,7 +1041,7 @@
"interactive_script_count": 0,
"package_hash_scope": "source-contract-without-generated-reports",
"package_hash_file_count": 178,
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
},
"skill_atlas": {
"skill_count": 12,
@@ -1079,8 +1079,8 @@
"trust_level": "local",
"license": "MIT",
"checksums": {
"package_sha256": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179",
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"package_sha256": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
"compatibility": {
"openai": "pass",
@@ -1111,7 +1111,7 @@
},
"distribution": {
"archive_verified": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"package_verification": "reports/package_verification.json",
"install_simulated": true,
"install_simulation": "reports/install_simulation.json"
@@ -1127,7 +1127,7 @@
"target_count": 4,
"adapter_count": 4,
"archive_present": true,
"archive_sha256": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e",
"archive_sha256": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27",
"archive_entry_count": 573,
"failure_count": 0,
"warning_count": 0
@@ -1206,12 +1206,12 @@
{
"field": "archive_sha256",
"from": "",
"to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
{
"field": "package_sha256",
"from": "0000000000000000000000000000000000000000000000000000000000000000",
"to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
reports/skill_os2_audit.json
+1 -1
View File
@@ -145,7 +145,7 @@
"key": "benchmark-reproducibility",
"label": "Benchmark Reproducibility",
"status": "pass",
"current": "artifacts 20; missing 0; failures 3",
"current": "artifacts 24; missing 0; failures 3",
"target": "Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable",
"evidence": [
{
reports/skill_os2_audit.md
+1 -1
View File
@@ -21,7 +21,7 @@ Generated at: `2026-06-13`
| Output Eval Lab | pass | 5 cases; delta 100.0; exec 10; blind 5 | with-skill/baseline, assertions, execution evidence, blind A/B, failure taxonomy | Add more real-file and adversarial holdout cases as usage grows. |
| Provider Holdout | external-required | model-executed 0; token-observed 0 | At least one real provider-backed holdout run with observed model/timing/token metadata | Run provider-backed holdout cases with real credentials and commit only aggregate evidence. |
| Human Adjudication | human-required | 0/5 decisions; pending 5 | Real reviewer decisions recorded before claiming output review completion | Record real A/B choices in the decision template, then regenerate adjudication. |
| Benchmark Reproducibility | pass | artifacts 20; missing 0; failures 3 | Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable | Keep the manifest current with every benchmark, package, and release evidence change. |
| Benchmark Reproducibility | pass | artifacts 24; missing 0; failures 3 | Public methodology, reproducible commands, required artifacts, and failure disclosure are machine-checkable | Keep the manifest current with every benchmark, package, and release evidence change. |
| Runtime Conformance | pass | 5/5 targets pass | Target package structure, metadata, relative paths, and degradation notes pass | Keep target conformance fixtures updated as platform contracts change. |
| Trust Security | pass | secrets 0; scripts 94; help failures 0 | Secrets, scripts, dependencies, permissions, and package hash are reviewable | Keep high-permission approvals scoped, expiring, and target-mapped. |
| Permission Metadata | pass | 4/4 target probes pass; metadata fallback 4; installer enforcement 4 | Packaged adapters expose explicit permission metadata, residual risks, and installer enforcement evidence when available | Preserve residual-risk notes until real native enforcement exists. |
reports/skill_os2_coverage.json
+1 -1
View File
@@ -305,7 +305,7 @@
"label": "Benchmark Methodology",
"status": "pass",
"objective": "Recommended Skill OS 2.0 implementation PR from the upgrade plan.",
"current": "20 required artifacts checked",
"current": "24 required artifacts checked",
"command": "make ci-test",
"test": "tests/verify_benchmark_reproducibility.py",
"evidence": [
reports/skill_os2_coverage.md
+1 -1
View File
@@ -31,7 +31,7 @@ This report maps the Skill OS 2.0 upgrade blueprint to concrete local artifacts,
| Item | Status | Current | Command | Test |
| --- | --- | --- | --- | --- |
| Benchmark Methodology | `pass` | 20 required artifacts checked | `make ci-test` | `tests/verify_benchmark_reproducibility.py` |
| Benchmark Methodology | `pass` | 24 required artifacts checked | `make ci-test` | `tests/verify_benchmark_reproducibility.py` |
| Output Eval Schema | `pass` | 5 output cases | `make ci-test` | `tests/verify_output_eval_lab.py` |
| Output Eval Runner | `pass` | delta 100.0 | `make ci-test` | `tests/verify_output_eval_lab.py` |
| Output Quality Scorecard | `pass` | gate pass True | `make ci-test` | `tests/verify_output_eval_lab.py` |
reports/upgrade_check.json
+2 -2
View File
@@ -70,12 +70,12 @@
{
"field": "archive_sha256",
"from": "",
"to": "de2056721d0fddfbae7fbde46493213c0a5a2879ed9dd6b3875ae4625ae2072e"
"to": "5b472f78172a45c1f7b95de9d133d9d295e4d2dc82ac5fe50e0834f2c8158f27"
},
{
"field": "package_sha256",
"from": "0000000000000000000000000000000000000000000000000000000000000000",
"to": "97ae1c187b764f2504ead4bf0c3063b466f8e8daef0de0ed8001d2722ea2d179"
"to": "2ce3f2625183c9fad67ea0e48c927f49d5f5051222f478986e03df2d22752a3d"
}
]
},
scripts/render_benchmark_reproducibility.py
+14
View File
@@ -41,6 +41,10 @@ REQUIRED_ARTIFACTS = [
("world_class_evidence_plan", "reports/world_class_evidence_plan.json"),
("world_class_evidence_ledger", "reports/world_class_evidence_ledger.json"),
("world_class_evidence_intake", "reports/world_class_evidence_intake.json"),
("world_class_submission_review", "reports/world_class_submission_review.json"),
("world_class_operator_runbook", "reports/world_class_operator_runbook.json"),
("world_class_operator_runbook_markdown", "reports/world_class_operator_runbook.md"),
("world_class_operator_runbook_html", "reports/world_class_operator_runbook.html"),
("world_class_claim_guard", "reports/world_class_claim_guard.json"),
]
@@ -130,6 +134,16 @@ REPRODUCTION_COMMANDS = [
"command": "python3 scripts/yao.py world-class-intake .",
"evidence": "reports/world_class_evidence_intake.json",
},
{
"label": "world-class submission review",
"command": "python3 scripts/yao.py world-class-submission-review .",
"evidence": "reports/world_class_submission_review.json",
},
{
"label": "world-class operator runbook",
"command": "python3 scripts/yao.py world-class-runbook .",
"evidence": "reports/world_class_operator_runbook.json",
},
{
"label": "world-class claim guard",
"command": "python3 scripts/yao.py world-class-claim-guard .",
tests/verify_benchmark_reproducibility.py
+34
View File
@@ -40,6 +40,32 @@ def main() -> None:
text=True,
check=True,
)
subprocess.run(
[
sys.executable,
str(ROOT / "scripts" / "render_world_class_submission_review.py"),
str(ROOT),
"--generated-at",
"2026-06-13",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
subprocess.run(
[
sys.executable,
str(ROOT / "scripts" / "render_world_class_operator_runbook.py"),
str(ROOT),
"--generated-at",
"2026-06-13",
],
cwd=ROOT,
capture_output=True,
text=True,
check=True,
)
subprocess.run(
[
sys.executable,
@@ -108,17 +134,25 @@ def main() -> None:
assert artifacts["reports/world_class_evidence_plan.json"]["exists"], artifacts
assert artifacts["reports/world_class_evidence_ledger.json"]["exists"], artifacts
assert artifacts["reports/world_class_evidence_intake.json"]["exists"], artifacts
assert artifacts["reports/world_class_submission_review.json"]["exists"], artifacts
assert artifacts["reports/world_class_operator_runbook.json"]["exists"], artifacts
assert artifacts["reports/world_class_operator_runbook.md"]["exists"], artifacts
assert artifacts["reports/world_class_operator_runbook.html"]["exists"], artifacts
assert artifacts["reports/world_class_claim_guard.json"]["exists"], artifacts
assert artifacts["reports/python_compatibility.json"]["exists"], artifacts
assert any(command["command"] == "make ci-test" for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py world-class-ledger ." for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py world-class-intake ." for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py world-class-submission-review ." for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py world-class-runbook ." for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py world-class-claim-guard ." for command in payload["reproduction_commands"]), payload
assert any(command["command"] == "python3 scripts/yao.py python-compat ." for command in payload["reproduction_commands"]), payload
assert any("provider-backed" in item for item in payload["limitations"]), payload["limitations"]
markdown = output_md.read_text(encoding="utf-8")
assert "Benchmark Reproducibility" in markdown, markdown
assert "reports/benchmark_methodology.md" in markdown, markdown
assert "reports/world_class_operator_runbook.html" in markdown, markdown
assert "python3 scripts/yao.py world-class-runbook ." in markdown, markdown
assert "make ci-test" in markdown, markdown
print(json.dumps({"ok": True}, ensure_ascii=False, indent=2))