diff --git a/AGENTS.md b/AGENTS.md index 793f3b2..37ec717 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -69,6 +69,7 @@ Clean test-only scratch directories after verification with `find tests -maxdept - `scripts/skill_report_model.py`, `scripts/skill_report_metrics.py`, `scripts/skill_report_charts.py`: skill overview data model, scoring, and inline SVG chart generation. - `scripts/yao_cli_config.py`: CLI target maps, archetype heuristics, diagnosis copy, and side-effect-free shaping helpers. - `scripts/yao_cli_parser.py`: CLI argparse command surface, flags, choices, and command handler binding. +- `scripts/yao_cli_telemetry.py`: opt-in metadata-only CLI run telemetry. Keep it free of prompt, argument, output, transcript, note, or message capture. New helper modules that are imported by CLI/report scripts but are not standalone commands must declare: diff --git a/README.md b/README.md index df8c17a..426cf05 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ It turns rough workflows, transcripts, prompts, notes, and runbooks into reusabl - a systems-thinking model that maps boundaries, feedback loops, drift risks, recurring failure patterns, and highest-leverage quality moves - three high-value next iteration directions after the first package is created - a lightweight feedback log that does not require a full promotion cycle -- a local-first metadata-only adoption and drift report that turns real usage signals into next iteration candidates +- a local-first metadata-only adoption and drift report that turns real usage signals into next iteration candidates, with optional `yao.py` CLI run capture that records command names and outcomes without arguments or raw content - a Skill Atlas drift layer that reads aggregate adoption reports and surfaces portfolio-level drift signals without packaging raw telemetry logs - a baseline compare report for with-skill vs baseline review - a conversation-style, archetype-aware quickstart that steers new packages toward scaffold, production, library, or governed fits @@ -108,6 +108,7 @@ python3 scripts/yao.py prompt-quality-profile my-skill python3 scripts/yao.py system-model my-skill python3 scripts/yao.py feedback my-skill --note "Tighten exclusions before adding scripts." --rating 4 --category boundary python3 scripts/yao.py adoption-drift my-skill --record-event skill_activation --activation-type explicit --outcome accepted +YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate my-skill python3 scripts/yao.py review-waivers my-skill --add-waiver --gate-key trust-report --reviewer "Yao Team" --reason "Known warning accepted for this release with bounded follow-up." --expires-at 2026-09-30 python3 scripts/yao.py review-waivers my-skill --add-waiver --gate-key permission-gates --reviewer "Yao Team" --reason "Permission warning accepted only for this non-governed release window." --expires-at 2026-09-30 python3 scripts/yao.py review-annotations my-skill --add-annotation --gate-key output-lab --target-path reports/output_quality_scorecard.md --line 1 --body "Clarify recorded fixture vs model-executed evidence before release." @@ -413,6 +414,7 @@ Utility scripts that make the meta-skill operational: - `simulate_install.py`: extracts a generated zip into a temporary skill root and verifies entrypoint, manifest, interface, reports, and adapters can be loaded - `upgrade_check.py`: compares current and previous registry package metadata, recommends a version bump, and blocks incompatible upgrade claims - `render_adoption_drift_report.py`: records metadata-only local telemetry and renders adoption, missed-trigger, bad-output, script-error, and review-drift signals without packaging raw event logs +- `yao_cli_telemetry.py`: opt-in metadata-only `yao.py` run capture for command name, source, outcome, and failure class without command arguments or raw content - `render_review_waivers.py`: validates human reviewer risk approvals with gate keys, reasons, expiry dates, and blocker-safe waiver policy - `init_skill.py`, `lint_skill.py`, `validate_skill.py`, `diff_eval.py`: minimal authoring toolchain - `check_update.py`: checks GitHub for a newer `VERSION` or remote manifest version and reports a reinstall hint without modifying local files diff --git a/references/telemetry-drift-method.md b/references/telemetry-drift-method.md index 071bbb2..14caaff 100644 --- a/references/telemetry-drift-method.md +++ b/references/telemetry-drift-method.md @@ -17,6 +17,8 @@ The local event stream is `reports/telemetry_events.jsonl`. It is intentionally "event": "skill_activation", "skill": "example-skill", "version": "2.0.0", + "source": "yao_cli", + "command": "quickstart", "activation_type": "implicit", "outcome": "accepted", "failure_type": "none", @@ -26,10 +28,39 @@ The local event stream is `reports/telemetry_events.jsonl`. It is intentionally Allowed events: `skill_activation`, `skill_output`, `script_run`, `review_event`. +Allowed sources: `manual`, `yao_cli`, `external`, `unknown`. + Allowed outcomes: `accepted`, `edited`, `rejected`, `missed`, `failed`, `reviewed`, `unknown`. Allowed failure types: `wrong_trigger`, `under_trigger`, `bad_output`, `missing_resource`, `script_error`, `review_overdue`, `none`. +`source` and `command` are metadata fields. They may identify that `yao.py` ran `quickstart`, `validate`, `output-exec`, or another subcommand, but they must not include arguments, prompt text, file content, model output, transcripts, or reviewer notes. + +## CLI Capture + +`scripts/yao.py` can record metadata-only `script_run` events automatically. It is opt-in to keep release evidence reproducible and avoid surprising local writes: + +```bash +YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate . +``` + +Optional destination override: + +```bash +YAO_CLI_TELEMETRY=1 \ +YAO_CLI_TELEMETRY_EVENTS=/tmp/yao-telemetry.jsonl \ +python3 scripts/yao.py output-exec +``` + +Equivalent global flags are available before the subcommand: + +```bash +python3 scripts/yao.py --record-cli-telemetry validate . +python3 scripts/yao.py --no-cli-telemetry validate . +``` + +Successful CLI runs record `event=script_run`, `source=yao_cli`, `outcome=accepted`, and `failure_type=none`. Failed CLI runs record `outcome=failed` and `failure_type=script_error`. The command name is normalized to the subcommand only; command arguments are never recorded. + ## Privacy Rule The raw JSONL event log is local evidence and should not be distributed in skill packages. The distributable artifact is the aggregate report: @@ -48,7 +79,7 @@ Package builders should exclude `reports/telemetry_events.jsonl`. The root repos ## Iteration Loop -1. Capture metadata-only events locally. +1. Capture metadata-only events locally, either manually with `adoption-drift --record-event` or automatically with opt-in `yao.py` CLI capture. 2. Render `reports/adoption_drift_report.md`. 3. Convert missed triggers into trigger eval cases. 4. Convert bad outputs into Output Eval assertions and failure taxonomy entries. diff --git a/registry/index.json b/registry/index.json index eeeea0c..653004b 100644 --- a/registry/index.json +++ b/registry/index.json @@ -16,7 +16,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] } diff --git a/registry/packages/yao-meta-skill.json b/registry/packages/yao-meta-skill.json index 31f6411..8537a80 100644 --- a/registry/packages/yao-meta-skill.json +++ b/registry/packages/yao-meta-skill.json @@ -16,8 +16,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd", - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, "compatibility": { "openai": "pass", @@ -48,7 +48,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" diff --git a/reports/adoption_drift_report.json b/reports/adoption_drift_report.json index 50f198d..269794a 100644 --- a/reports/adoption_drift_report.json +++ b/reports/adoption_drift_report.json @@ -1,7 +1,7 @@ { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-13T13:13:24Z", + "generated_at": "2026-06-13T13:30:00Z", "skill_dir": ".", "privacy_contract": { "storage": "local-first", @@ -25,14 +25,14 @@ }, "summary": { "event_count": 1, - "adoption_sample_count": 1, - "activation_count": 1, - "accepted_count": 1, + "adoption_sample_count": 0, + "activation_count": 0, + "accepted_count": 0, "edited_count": 0, "rejected_count": 0, "missed_count": 0, "failed_count": 0, - "adoption_rate": 100.0, + "adoption_rate": 0, "missed_trigger_count": 0, "wrong_trigger_count": 0, "bad_output_count": 0, @@ -41,37 +41,43 @@ "review_overdue_count": 0, "risk_band": "low", "event_types": { - "skill_activation": 1 + "review_event": 1 }, - "failure_types": {} + "failure_types": {}, + "source_types": { + "manual": 1 + }, + "command_counts": {} }, "adoption_by_skill": [ { "skill": "yao-meta-skill", "events": 1, - "adoption_events": 1, - "accepted": 1, + "adoption_events": 0, + "accepted": 0, "edited": 0, "rejected": 0, "missed": 0, - "adoption_rate": 100.0 + "adoption_rate": 0 } ], "next_iteration_candidates": [], "recent_events": [ { - "event": "skill_activation", + "command": "unknown", + "event": "review_event", "skill": "yao-meta-skill", + "source": "manual", "version": "1.1.0", - "activation_type": "explicit", - "outcome": "accepted", + "activation_type": "manual", + "outcome": "reviewed", "failure_type": "none", - "timestamp": "2026-06-13T10:00:00Z" + "timestamp": "2026-06-13T12:00:00Z" } ], "failures": [], "artifacts": { - "events_jsonl": "tests/tmp_review_studio/telemetry_events.jsonl", + "events_jsonl": "reports/telemetry_events.jsonl", "json": "reports/adoption_drift_report.json", "markdown": "reports/adoption_drift_report.md" } diff --git a/reports/adoption_drift_report.md b/reports/adoption_drift_report.md index 6945d35..91b3198 100644 --- a/reports/adoption_drift_report.md +++ b/reports/adoption_drift_report.md @@ -5,9 +5,9 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs, ## Summary - Events: `1` -- Adoption samples: `1` -- Activation events: `1` -- Adoption rate: `100.0` +- Adoption samples: `0` +- Activation events: `0` +- Adoption rate: `0` - Missed trigger signals: `0` - Bad output signals: `0` - Script error signals: `0` @@ -25,7 +25,7 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs, | Skill | Events | Adoption Samples | Accepted | Edited | Rejected | Missed | Adoption Rate | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| `yao-meta-skill` | 1 | 1 | 1 | 0 | 0 | 0 | 100.0 | +| `yao-meta-skill` | 1 | 0 | 0 | 0 | 0 | 0 | 0 | ## Next Iteration Candidates @@ -33,4 +33,4 @@ Local-first, metadata-only telemetry for skill operations. Raw prompts, outputs, ## Recent Metadata Events -- `2026-06-13T10:00:00Z` `yao-meta-skill` event=`skill_activation` activation=`explicit` outcome=`accepted` failure=`none` +- `2026-06-13T12:00:00Z` `yao-meta-skill` event=`review_event` source=`manual` command=`unknown` activation=`manual` outcome=`reviewed` failure=`none` diff --git a/reports/compiled_targets.json b/reports/compiled_targets.json index 2108741..e917ddc 100644 --- a/reports/compiled_targets.json +++ b/reports/compiled_targets.json @@ -213,7 +213,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -264,7 +265,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -505,7 +506,7 @@ "strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -516,7 +517,7 @@ }, "scripts": { "strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -730,7 +731,7 @@ "strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -741,7 +742,7 @@ }, "scripts": { "strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -1011,7 +1012,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -1062,7 +1064,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -1303,7 +1305,7 @@ "strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -1314,7 +1316,7 @@ }, "scripts": { "strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -1528,7 +1530,7 @@ "strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -1539,7 +1541,7 @@ }, "scripts": { "strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -1809,7 +1811,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -1860,7 +1863,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -2101,7 +2104,7 @@ "strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -2111,7 +2114,7 @@ }, "scripts": { "strategy": "Expose script and permission metadata for downstream clients or installers to enforce.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -2319,7 +2322,7 @@ "strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -2329,7 +2332,7 @@ }, "scripts": { "strategy": "Expose script and permission metadata for downstream clients or installers to enforce.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -2591,7 +2594,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -2642,7 +2646,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -2883,7 +2887,7 @@ "strategy": "Keep optional directories as relative resources next to SKILL.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -2894,7 +2898,7 @@ }, "scripts": { "strategy": "Scripts remain local optional resources and should advertise --help when executable.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -3101,7 +3105,7 @@ "strategy": "Keep optional directories as relative resources next to SKILL.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -3112,7 +3116,7 @@ }, "scripts": { "strategy": "Scripts remain local optional resources and should advertise --help when executable.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -3373,7 +3377,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -3424,7 +3429,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -3665,7 +3670,7 @@ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -3676,7 +3681,7 @@ }, "scripts": { "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -3887,7 +3892,7 @@ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -3898,7 +3903,7 @@ }, "scripts": { "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { diff --git a/reports/context_budget.json b/reports/context_budget.json index cb22e04..b068fe2 100644 --- a/reports/context_budget.json +++ b/reports/context_budget.json @@ -6,10 +6,10 @@ "context_budget_tier": "production", "context_budget_limit": 1000, "skill_body_tokens": 751, - "other_text_tokens": 915033, + "other_text_tokens": 917960, "estimated_initial_load_tokens": 944, - "estimated_total_text_tokens": 915784, - "relevant_file_count": 392, + "estimated_total_text_tokens": 918711, + "relevant_file_count": 394, "unused_resource_dirs": [], "quality_signal_points": 130, "quality_density": 137.7 diff --git a/reports/install_simulation.json b/reports/install_simulation.json index 92a0067..6973bf8 100644 --- a/reports/install_simulation.json +++ b/reports/install_simulation.json @@ -8,7 +8,7 @@ "installed_skill_dir": "dist/install-simulation/simulate-yao-meta-skill/yao-meta-skill", "summary": { "archive_present": true, - "archive_entry_count": 494, + "archive_entry_count": 495, "archive_extracted": true, "entrypoint_loaded": true, "manifest_loaded": true, diff --git a/reports/output_execution_runs.json b/reports/output_execution_runs.json index 0b6541a..78b3d8c 100644 --- a/reports/output_execution_runs.json +++ b/reports/output_execution_runs.json @@ -34,7 +34,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.6, + "duration_ms": 31.15, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -62,7 +62,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.9, + "duration_ms": 31.4, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -85,7 +85,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 37.17, + "duration_ms": 33.61, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -113,7 +113,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 49.26, + "duration_ms": 33.86, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -136,7 +136,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 32.41, + "duration_ms": 33.02, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -164,7 +164,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.26, + "duration_ms": 31.39, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -187,7 +187,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.85, + "duration_ms": 30.77, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -214,7 +214,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.38, + "duration_ms": 32.92, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -237,7 +237,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.12, + "duration_ms": 29.09, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -266,7 +266,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.95, + "duration_ms": 27.95, "provider": "local-output-eval-runner", "model": "", "usage": { diff --git a/reports/output_execution_runs.md b/reports/output_execution_runs.md index dd06dd4..a4d9cd1 100644 --- a/reports/output_execution_runs.md +++ b/reports/output_execution_runs.md @@ -23,16 +23,16 @@ Command runner evidence is present. This proves the eval harness executed an ext | Case | Variant | Mode | Model | Duration ms | Tokens | Score | Status | | --- | --- | --- | --- | ---: | ---: | ---: | --- | -| skill-package-contract | baseline | command | local-output-eval-runner | 33.6 | 33 | 0.0 | pass | -| skill-package-contract | with_skill | command | local-output-eval-runner | 33.9 | 73 | 100.0 | pass | -| output-eval-expectation | baseline | command | local-output-eval-runner | 37.17 | 36 | 0.0 | pass | -| output-eval-expectation | with_skill | command | local-output-eval-runner | 49.26 | 80 | 100.0 | pass | -| ir-before-packaging | baseline | command | local-output-eval-runner | 32.41 | 33 | 0.0 | pass | -| ir-before-packaging | with_skill | command | local-output-eval-runner | 33.26 | 80 | 100.0 | pass | -| near-neighbor-boundary | baseline | command | local-output-eval-runner | 33.85 | 36 | 0.0 | pass | -| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 34.38 | 65 | 100.0 | pass | -| file-backed-governed-package | baseline | command | local-output-eval-runner | 34.12 | 37 | 0.0 | pass | -| file-backed-governed-package | with_skill | command | local-output-eval-runner | 34.95 | 98 | 100.0 | pass | +| skill-package-contract | baseline | command | local-output-eval-runner | 31.15 | 33 | 0.0 | pass | +| skill-package-contract | with_skill | command | local-output-eval-runner | 31.4 | 73 | 100.0 | pass | +| output-eval-expectation | baseline | command | local-output-eval-runner | 33.61 | 36 | 0.0 | pass | +| output-eval-expectation | with_skill | command | local-output-eval-runner | 33.86 | 80 | 100.0 | pass | +| ir-before-packaging | baseline | command | local-output-eval-runner | 33.02 | 33 | 0.0 | pass | +| ir-before-packaging | with_skill | command | local-output-eval-runner | 31.39 | 80 | 100.0 | pass | +| near-neighbor-boundary | baseline | command | local-output-eval-runner | 30.77 | 36 | 0.0 | pass | +| near-neighbor-boundary | with_skill | command | local-output-eval-runner | 32.92 | 65 | 100.0 | pass | +| file-backed-governed-package | baseline | command | local-output-eval-runner | 29.09 | 37 | 0.0 | pass | +| file-backed-governed-package | with_skill | command | local-output-eval-runner | 27.95 | 98 | 100.0 | pass | ## Next Fixes diff --git a/reports/package_verification.json b/reports/package_verification.json index 797ea0b..b390b49 100644 --- a/reports/package_verification.json +++ b/reports/package_verification.json @@ -8,8 +8,8 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", - "archive_entry_count": 494, + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", + "archive_entry_count": 495, "failure_count": 0, "warning_count": 0 }, diff --git a/reports/package_verification.md b/reports/package_verification.md index a783613..7178423 100644 --- a/reports/package_verification.md +++ b/reports/package_verification.md @@ -4,7 +4,7 @@ - Package directory: `dist` - Targets: `4 / 4` adapters present - Archive present: `True` -- Archive SHA256: `cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933` +- Archive SHA256: `0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474` - Failures: `0` - Warnings: `0` diff --git a/reports/registry_audit.json b/reports/registry_audit.json index 0a30ff5..001d867 100644 --- a/reports/registry_audit.json +++ b/reports/registry_audit.json @@ -21,8 +21,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd", - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, "compatibility": { "openai": "pass", @@ -53,7 +53,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -78,7 +78,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, diff --git a/reports/registry_audit.md b/reports/registry_audit.md index 003be7c..98065b2 100644 --- a/reports/registry_audit.md +++ b/reports/registry_audit.md @@ -6,8 +6,8 @@ - Maturity: `governed` - Owner: `Yao Team` - License: `MIT` -- Package SHA256: `44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd` -- Archive SHA256: `cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933` +- Package SHA256: `495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2` +- Archive SHA256: `0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474` - Install simulated: `True` ## Compatibility diff --git a/reports/review-studio.html b/reports/review-studio.html index adc4126..2a86ed1 100644 --- a/reports/review-studio.html +++ b/reports/review-studio.html @@ -242,12 +242,12 @@

核心指标

-
Skill IR2.0.0

5 targets in platform-neutral contract

Compiler5/5

target contracts compiled from Skill IR

Output Delta100.0

5 cases; 1 file-backed

Exec Runs10

command 10; model 0; recorded 0

Blind A/B5

review pairs hide baseline vs with-skill labels

Review A/B0/5

adjudication decisions; pending 5

Runtime5/5

target conformance pass rate

Perm Probe4/4

0 native enforcement targets

Trust0

69 scripts scanned; secrets found

Atlas5

12 scanned skills; route collisions

Driftlow

1 metadata events; 0 missed triggers

Waivers0

0 gates covered; human risk decisions

Notes0/0

0 open blocker annotations

Registry1.1.0

5 targets; MIT license

Archivepass

494 zip entries; package verification

Installpass

4 adapters; 12 permissions enforced; 0 permission failures

Upgrademinor

declared minor; 0 breaking changes

+
Skill IR2.0.0

5 targets in platform-neutral contract

Compiler5/5

target contracts compiled from Skill IR

Output Delta100.0

5 cases; 1 file-backed

Exec Runs10

command 10; model 0; recorded 0

Blind A/B5

review pairs hide baseline vs with-skill labels

Review A/B0/5

adjudication decisions; pending 5

Runtime5/5

target conformance pass rate

Perm Probe4/4

0 native enforcement targets

Trust0

70 scripts scanned; secrets found

Atlas5

12 scanned skills; route collisions

Driftlow

1 metadata events; 0 missed triggers

Waivers0

0 gates covered; human risk decisions

Notes0/0

0 open blocker annotations

Registry1.1.0

5 targets; MIT license

Archivepass

495 zip entries; package verification

Installpass

4 adapters; 12 permissions enforced; 0 permission failures

Upgrademinor

declared minor; 0 breaking changes

审查闸门

-
通过

意图画布

intent confidence 100/100; Intent is clear enough to package the first routeable version.

reports/intent-confidence.json 证据
通过

触发实验

13 trigger cases; 0 misroutes; 0 ambiguous

reports/route_scorecard.json 证据
关注

输出实验

5/5 cases; with-skill 100.0; baseline 0.0; file-backed 1; near-neighbor 1; blind A/B 5; exec 10; command 10; model 0; recorded 0; reviewed 0/5; review pending 5

reports/output_quality_scorecard.json 证据
通过

上下文

initial load 944/1000; quality density 137.7

reports/context_budget.json 证据
通过

运行矩阵

5 / 5 targets pass

reports/conformance_matrix.json 证据
通过

信任报告

0 secrets; 69 scripts; 3 network-capable scripts; 0 help smoke failures

reports/security_trust_report.json 证据
通过

权限批准

3/3 permissions approved; gaps 0; required file_write, network, subprocess

reports/security_trust_report.json + security/permission_policy.json 证据
通过

权限探针

4/4 targets probed; native 0; metadata fallback 4; residual risks 4

reports/runtime_permission_probes.json 证据
通过

组合治理

12 skills, 1 actionable; 0 actionable route collisions; 0 actionable owner gaps; 0 actionable stale; 0 actionable drift; 24 scoped non-actionable issues

reports/skill_atlas.json 证据
通过

运营回路

1 metadata events; adoption 100.0; missed 0; bad-output 0; risk low

reports/adoption_drift_report.json 证据
关注

人工批准

0 active waivers; 1 warning gates still need reviewer decision

reports/review_waivers.json 证据
通过

注册审计

yao-meta-skill 1.1.0; 6/6 compatibility entries pass; install pass with 4 adapters; installer permissions 12 enforced / 0 failures

reports/registry_audit.json + reports/install_simulation.json 证据
通过

发布路线

0 promote; 3 keep current; 0 blocked; upgrade minor declared / minor recommended

reports/promotion_decisions.json + reports/upgrade_check.json + docs/migration-v2.md 证据
+
通过

意图画布

intent confidence 100/100; Intent is clear enough to package the first routeable version.

reports/intent-confidence.json 证据
通过

触发实验

13 trigger cases; 0 misroutes; 0 ambiguous

reports/route_scorecard.json 证据
关注

输出实验

5/5 cases; with-skill 100.0; baseline 0.0; file-backed 1; near-neighbor 1; blind A/B 5; exec 10; command 10; model 0; recorded 0; reviewed 0/5; review pending 5

reports/output_quality_scorecard.json 证据
通过

上下文

initial load 944/1000; quality density 137.7

reports/context_budget.json 证据
通过

运行矩阵

5 / 5 targets pass

reports/conformance_matrix.json 证据
通过

信任报告

0 secrets; 70 scripts; 3 network-capable scripts; 0 help smoke failures

reports/security_trust_report.json 证据
通过

权限批准

3/3 permissions approved; gaps 0; required file_write, network, subprocess

reports/security_trust_report.json + security/permission_policy.json 证据
通过

权限探针

4/4 targets probed; native 0; metadata fallback 4; residual risks 4

reports/runtime_permission_probes.json 证据
通过

组合治理

12 skills, 1 actionable; 0 actionable route collisions; 0 actionable owner gaps; 0 actionable stale; 0 actionable drift; 24 scoped non-actionable issues

reports/skill_atlas.json 证据
通过

运营回路

1 metadata events; adoption 0; missed 0; bad-output 0; risk low

reports/adoption_drift_report.json 证据
关注

人工批准

0 active waivers; 1 warning gates still need reviewer decision

reports/review_waivers.json 证据
通过

注册审计

yao-meta-skill 1.1.0; 6/6 compatibility entries pass; install pass with 4 adapters; installer permissions 12 enforced / 0 failures

reports/registry_audit.json + reports/install_simulation.json 证据
通过

发布路线

0 promote; 3 keep current; 0 blocked; upgrade minor declared / minor recommended

@@ -313,7 +313,7 @@
-

信任报告

Secret
0
脚本数
69
网络脚本
3
Help 失败
0
包体哈希
44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd
+

信任报告

Secret
0
脚本数
70
网络脚本
3
Help 失败
0
包体哈希
495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2

安全边界

高风险 secret、远程 inline execution、缺失依赖策略或无法解释的脚本接口应阻断 governed release。

@@ -333,8 +333,8 @@
-

运营回路

1 metadata events; adoption 100.0; missed 0; bad-output 0; risk low

-

漂移信号

事件数
1
采用率
100
漏触发
0
Bad Output Count
0
风险带
low
+

运营回路

1 metadata events; adoption 0; missed 0; bad-output 0; risk low

+

漂移信号

事件数
1
采用率
0
漏触发
0
Bad Output Count
0
风险带
low
@@ -344,12 +344,12 @@

注册审计

yao-meta-skill 1.1.0; 6/6 compatibility entries pass; install pass with 4 adapters; installer permissions 12 enforced / 0 failures

-

包体元数据

名称
yao-meta-skill
版本
1.1.0
Maturity
governed
Owner
Yao Team
License
MIT
信任级别
local
目标平台
openai, claude, generic, agent-skills-compatible, vscode
兼容通过
6/6
归档哈希
cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933
+

包体元数据

名称
yao-meta-skill
版本
1.1.0
Maturity
governed
Owner
Yao Team
License
MIT
信任级别
local
目标平台
openai, claude, generic, agent-skills-compatible, vscode
兼容通过
6/6
归档哈希
0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474

发布路线

0 promote; 3 keep current; 0 blocked; upgrade minor declared / minor recommended

-

包体验证

目标数
4
Adapter
4
归档存在
Zip 条目
494
失败数
0
警告数
0
归档哈希
cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933
+

包体验证

目标数
4
Adapter
4
归档存在
Zip 条目
495
失败数
0
警告数
0
归档哈希
0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474
diff --git a/reports/review-studio.json b/reports/review-studio.json index c849859..0e0b726 100644 --- a/reports/review-studio.json +++ b/reports/review-studio.json @@ -59,7 +59,7 @@ "key": "trust-report", "label": "信任报告", "status": "pass", - "detail": "0 secrets; 69 scripts; 3 network-capable scripts; 0 help smoke failures", + "detail": "0 secrets; 70 scripts; 3 network-capable scripts; 0 help smoke failures", "evidence": "reports/security_trust_report.json", "link": "security_trust_report.md" }, @@ -91,7 +91,7 @@ "key": "operations-loop", "label": "运营回路", "status": "pass", - "detail": "1 metadata events; adoption 100.0; missed 0; bad-output 0; risk low", + "detail": "1 metadata events; adoption 0; missed 0; bad-output 0; risk low", "evidence": "reports/adoption_drift_report.json", "link": "adoption_drift_report.md" }, @@ -355,7 +355,7 @@ "label": "上下文成本", "score": 42, "reasons": [ - "入口约 350 个词/字,references 约 14711 个词/字。", + "入口约 350 个词/字,references 约 14859 个词/字。", "分数越高代表上下文成本越低。", "上下文成本偏高,建议压缩入口或拆分 references。" ] @@ -459,7 +459,7 @@ "已生成 Output Review Adjudication,可记录盲评决策、一致率和待评审项。" ], "gaps": [ - "上下文成本需要补强:入口约 350 个词/字,references 约 14711 个词/字。" + "上下文成本需要补强:入口约 350 个词/字,references 约 14859 个词/字。" ], "recommendations": [ "先改触发边界,再扩展工作流。", @@ -692,7 +692,7 @@ "path": "scripts", "label": "Deterministic helpers or local tooling", "kind": "folder", - "file_count": 69 + "file_count": 70 }, { "path": "evals", @@ -707,7 +707,7 @@ "file_count": 171 } ], - "file_count": 304, + "file_count": 305, "folder_count": 4, "distribution": [ { @@ -732,7 +732,7 @@ }, { "label": "scripts", - "value": 69 + "value": 70 }, { "label": "evals", @@ -861,7 +861,7 @@ "path": "scripts", "label": "Deterministic helpers or local tooling", "kind": "folder", - "file_count": 69 + "file_count": 70 }, { "path": "evals", @@ -1207,9 +1207,9 @@ "failures": [] }, "trust_security": { - "scanned_files": 153, - "script_count": 69, - "internal_module_count": 9, + "scanned_files": 154, + "script_count": 70, + "internal_module_count": 10, "secret_findings": 0, "dependency_files": [ "requirements-ci.txt" @@ -1227,8 +1227,8 @@ "help_smoke_failed_count": 0, "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", - "package_hash_file_count": 153, - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_hash_file_count": 154, + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" }, "skill_atlas": { "skill_count": 12, @@ -1266,8 +1266,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd", - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, "compatibility": { "openai": "pass", @@ -1298,7 +1298,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -1314,8 +1314,8 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", - "archive_entry_count": 494, + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", + "archive_entry_count": 495, "failure_count": 0, "warning_count": 0 }, @@ -1326,7 +1326,7 @@ "ok": true, "summary": { "archive_present": true, - "archive_entry_count": 494, + "archive_entry_count": 495, "archive_extracted": true, "entrypoint_loaded": true, "manifest_loaded": true, @@ -1393,12 +1393,12 @@ { "field": "archive_sha256", "from": "", - "to": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "to": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "to": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, @@ -1415,14 +1415,14 @@ "ok": true, "summary": { "event_count": 1, - "adoption_sample_count": 1, - "activation_count": 1, - "accepted_count": 1, + "adoption_sample_count": 0, + "activation_count": 0, + "accepted_count": 0, "edited_count": 0, "rejected_count": 0, "missed_count": 0, "failed_count": 0, - "adoption_rate": 100.0, + "adoption_rate": 0, "missed_trigger_count": 0, "wrong_trigger_count": 0, "bad_output_count": 0, @@ -1431,9 +1431,13 @@ "review_overdue_count": 0, "risk_band": "low", "event_types": { - "skill_activation": 1 + "review_event": 1 }, - "failure_types": {} + "failure_types": {}, + "source_types": { + "manual": 1 + }, + "command_counts": {} }, "next_iteration_candidates": [], "privacy_contract": { @@ -3128,7 +3132,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.6, + "duration_ms": 31.15, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3156,7 +3160,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.9, + "duration_ms": 31.4, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3179,7 +3183,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 37.17, + "duration_ms": 33.61, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3207,7 +3211,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 49.26, + "duration_ms": 33.86, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3230,7 +3234,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 32.41, + "duration_ms": 33.02, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3258,7 +3262,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.26, + "duration_ms": 31.39, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3281,7 +3285,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 33.85, + "duration_ms": 30.77, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3308,7 +3312,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.38, + "duration_ms": 32.92, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3331,7 +3335,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.12, + "duration_ms": 29.09, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3360,7 +3364,7 @@ "execution_mode": "command", "model_executed": false, "command_executed": true, - "duration_ms": 34.95, + "duration_ms": 27.95, "provider": "local-output-eval-runner", "model": "", "usage": { @@ -3882,7 +3886,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -3933,7 +3938,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -4174,7 +4179,7 @@ "strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -4185,7 +4190,7 @@ }, "scripts": { "strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -4399,7 +4404,7 @@ "strategy": "Ship the neutral source tree and expose OpenAI-facing interface metadata as a generated companion file.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -4410,7 +4415,7 @@ }, "scripts": { "strategy": "Keep scripts as local package resources; expose help-smoke and permission metadata for reviewer approval before execution.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -4680,7 +4685,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -4731,7 +4737,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -4972,7 +4978,7 @@ "strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -4983,7 +4989,7 @@ }, "scripts": { "strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -5197,7 +5203,7 @@ "strategy": "Preserve the source tree directly; write target notes in targets/claude/README.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -5208,7 +5214,7 @@ }, "scripts": { "strategy": "Scripts remain local package resources and must be reviewed through trust and permission reports before use.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -5478,7 +5484,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -5529,7 +5536,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -5770,7 +5777,7 @@ "strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -5780,7 +5787,7 @@ }, "scripts": { "strategy": "Expose script and permission metadata for downstream clients or installers to enforce.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -5988,7 +5995,7 @@ "strategy": "Preserve references, scripts, assets, evals, reports, and adapter metadata as relative package resources.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -5998,7 +6005,7 @@ }, "scripts": { "strategy": "Expose script and permission metadata for downstream clients or installers to enforce.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -6260,7 +6267,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -6311,7 +6319,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -6552,7 +6560,7 @@ "strategy": "Keep optional directories as relative resources next to SKILL.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -6563,7 +6571,7 @@ }, "scripts": { "strategy": "Scripts remain local optional resources and should advertise --help when executable.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -6770,7 +6778,7 @@ "strategy": "Keep optional directories as relative resources next to SKILL.md.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -6781,7 +6789,7 @@ }, "scripts": { "strategy": "Scripts remain local optional resources and should advertise --help when executable.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -7042,7 +7050,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", @@ -7093,7 +7102,7 @@ ], "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 } @@ -7334,7 +7343,7 @@ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -7345,7 +7354,7 @@ }, "scripts": { "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -7556,7 +7565,7 @@ "strategy": "Install as project or user scoped skill source, preserving relative references and scripts.", "counts": { "references": 32, - "scripts": 69, + "scripts": 70, "assets": 2, "reports": 41 }, @@ -7567,7 +7576,7 @@ }, "scripts": { "strategy": "Scripts require workspace trust and operator/client approval outside this compiler.", - "script_count": 69, + "script_count": 70, "help_smoke_failed_count": 0 }, "permissions": { @@ -8207,9 +8216,9 @@ "ok": true, "skill_dir": ".", "summary": { - "scanned_files": 153, - "script_count": 69, - "internal_module_count": 9, + "scanned_files": 154, + "script_count": 70, + "internal_module_count": 10, "secret_findings": 0, "dependency_files": [ "requirements-ci.txt" @@ -8227,8 +8236,8 @@ "help_smoke_failed_count": 0, "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", - "package_hash_file_count": 153, - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_hash_file_count": 154, + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" }, "failures": [], "warnings": [], @@ -9216,6 +9225,20 @@ "uses_subprocess": false, "network_urls": [], "network_hosts": [] + }, + { + "path": "scripts/yao_cli_telemetry.py", + "interface": "internal-module", + "interface_declared": true, + "interface_reason": "Imported by yao.py to record opt-in metadata-only CLI run telemetry.", + "has_argparse": true, + "has_main_guard": false, + "uses_input": false, + "uses_network": false, + "uses_file_write": false, + "uses_subprocess": false, + "network_urls": [], + "network_hosts": [] } ], "dependencies": { @@ -9252,7 +9275,7 @@ "checked_count": 60, "passed_count": 60, "failed_count": 0, - "skipped_count": 9, + "skipped_count": 10, "failed_scripts": [], "results": [ { @@ -9852,7 +9875,7 @@ "timed_out": false, "passed": true, "has_help_text": true, - "stdout_excerpt": "usage: yao.py [-h]\n {init,quickstart,validate,optimize-description,promote-check,review,release-snapshot,workspace-flow,report,skill-report,review-viewer,review-studio,reference-scan,github-benchmark-scan,intent-confidence,inte", + "stdout_excerpt": "usage: yao.py [-h] [--record-cli-telemetry] [--no-cli-telemetry]\n [--telemetry-events-jsonl TELEMETRY_EVENTS_JSONL]\n {init,quickstart,validate,optimize-description,promote-check,review,release-snapshot,workspace-fl", "stderr_excerpt": "" } ], @@ -9892,6 +9915,10 @@ { "path": "scripts/yao_cli_parser.py", "reason": "internal module" + }, + { + "path": "scripts/yao_cli_telemetry.py", + "reason": "internal module" } ] }, @@ -10060,10 +10087,10 @@ "context_budget_tier": "production", "context_budget_limit": 1000, "skill_body_tokens": 751, - "other_text_tokens": 915033, + "other_text_tokens": 917960, "estimated_initial_load_tokens": 944, - "estimated_total_text_tokens": 915784, - "relevant_file_count": 392, + "estimated_total_text_tokens": 918711, + "relevant_file_count": 394, "unused_resource_dirs": [], "quality_signal_points": 130, "quality_density": 137.7 @@ -10996,6 +11023,7 @@ "scripts/yao.py", "scripts/yao_cli_config.py", "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py", "references/artifact-design-doctrine.md", "references/authoring-discipline.md", "references/distribution-registry-method.md", @@ -11891,7 +11919,7 @@ "adoption_drift": { "ok": true, "schema_version": "2.0", - "generated_at": "2026-06-13T13:13:24Z", + "generated_at": "2026-06-13T13:30:00Z", "skill_dir": ".", "privacy_contract": { "storage": "local-first", @@ -11915,14 +11943,14 @@ }, "summary": { "event_count": 1, - "adoption_sample_count": 1, - "activation_count": 1, - "accepted_count": 1, + "adoption_sample_count": 0, + "activation_count": 0, + "accepted_count": 0, "edited_count": 0, "rejected_count": 0, "missed_count": 0, "failed_count": 0, - "adoption_rate": 100.0, + "adoption_rate": 0, "missed_trigger_count": 0, "wrong_trigger_count": 0, "bad_output_count": 0, @@ -11931,37 +11959,43 @@ "review_overdue_count": 0, "risk_band": "low", "event_types": { - "skill_activation": 1 + "review_event": 1 }, - "failure_types": {} + "failure_types": {}, + "source_types": { + "manual": 1 + }, + "command_counts": {} }, "adoption_by_skill": [ { "skill": "yao-meta-skill", "events": 1, - "adoption_events": 1, - "accepted": 1, + "adoption_events": 0, + "accepted": 0, "edited": 0, "rejected": 0, "missed": 0, - "adoption_rate": 100.0 + "adoption_rate": 0 } ], "next_iteration_candidates": [], "recent_events": [ { - "event": "skill_activation", + "command": "unknown", + "event": "review_event", "skill": "yao-meta-skill", + "source": "manual", "version": "1.1.0", - "activation_type": "explicit", - "outcome": "accepted", + "activation_type": "manual", + "outcome": "reviewed", "failure_type": "none", - "timestamp": "2026-06-13T10:00:00Z" + "timestamp": "2026-06-13T12:00:00Z" } ], "failures": [], "artifacts": { - "events_jsonl": "tests/tmp_review_studio/telemetry_events.jsonl", + "events_jsonl": "reports/telemetry_events.jsonl", "json": "reports/adoption_drift_report.json", "markdown": "reports/adoption_drift_report.md" } @@ -12010,7 +12044,7 @@ "schema_version": "1.0", "ok": true, "skill_dir": ".", - "source": "tests/tmp_review_studio/empty_review_annotations_input.json", + "source": "reports/review_annotations_input.json", "summary": { "annotation_count": 0, "open_count": 0, @@ -12053,8 +12087,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd", - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, "compatibility": { "openai": "pass", @@ -12085,7 +12119,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -12110,7 +12144,7 @@ "vscode" ], "package_metadata": "registry/packages/yao-meta-skill.json", - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, @@ -12133,8 +12167,8 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", - "archive_entry_count": 494, + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", + "archive_entry_count": 495, "failure_count": 0, "warning_count": 0 }, @@ -12799,7 +12833,7 @@ "installed_skill_dir": "dist/install-simulation/simulate-yao-meta-skill/yao-meta-skill", "summary": { "archive_present": true, - "archive_entry_count": 494, + "archive_entry_count": 495, "archive_extracted": true, "entrypoint_loaded": true, "manifest_loaded": true, @@ -13141,12 +13175,12 @@ { "field": "archive_sha256", "from": "", - "to": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "to": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "to": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, diff --git a/reports/review_annotations.json b/reports/review_annotations.json index 66f2b1d..76fadc1 100644 --- a/reports/review_annotations.json +++ b/reports/review_annotations.json @@ -2,7 +2,7 @@ "schema_version": "1.0", "ok": true, "skill_dir": ".", - "source": "tests/tmp_review_studio/empty_review_annotations_input.json", + "source": "reports/review_annotations_input.json", "summary": { "annotation_count": 0, "open_count": 0, diff --git a/reports/security_trust_report.json b/reports/security_trust_report.json index 10219c2..3d7d373 100644 --- a/reports/security_trust_report.json +++ b/reports/security_trust_report.json @@ -2,9 +2,9 @@ "ok": true, "skill_dir": ".", "summary": { - "scanned_files": 153, - "script_count": 69, - "internal_module_count": 9, + "scanned_files": 154, + "script_count": 70, + "internal_module_count": 10, "secret_findings": 0, "dependency_files": [ "requirements-ci.txt" @@ -22,8 +22,8 @@ "help_smoke_failed_count": 0, "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", - "package_hash_file_count": 153, - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_hash_file_count": 154, + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" }, "failures": [], "warnings": [], @@ -1011,6 +1011,20 @@ "uses_subprocess": false, "network_urls": [], "network_hosts": [] + }, + { + "path": "scripts/yao_cli_telemetry.py", + "interface": "internal-module", + "interface_declared": true, + "interface_reason": "Imported by yao.py to record opt-in metadata-only CLI run telemetry.", + "has_argparse": true, + "has_main_guard": false, + "uses_input": false, + "uses_network": false, + "uses_file_write": false, + "uses_subprocess": false, + "network_urls": [], + "network_hosts": [] } ], "dependencies": { @@ -1047,7 +1061,7 @@ "checked_count": 60, "passed_count": 60, "failed_count": 0, - "skipped_count": 9, + "skipped_count": 10, "failed_scripts": [], "results": [ { @@ -1647,7 +1661,7 @@ "timed_out": false, "passed": true, "has_help_text": true, - "stdout_excerpt": "usage: yao.py [-h]\n {init,quickstart,validate,optimize-description,promote-check,review,release-snapshot,workspace-flow,report,skill-report,review-viewer,review-studio,reference-scan,github-benchmark-scan,intent-confidence,inte", + "stdout_excerpt": "usage: yao.py [-h] [--record-cli-telemetry] [--no-cli-telemetry]\n [--telemetry-events-jsonl TELEMETRY_EVENTS_JSONL]\n {init,quickstart,validate,optimize-description,promote-check,review,release-snapshot,workspace-fl", "stderr_excerpt": "" } ], @@ -1687,6 +1701,10 @@ { "path": "scripts/yao_cli_parser.py", "reason": "internal module" + }, + { + "path": "scripts/yao_cli_telemetry.py", + "reason": "internal module" } ] }, diff --git a/reports/security_trust_report.md b/reports/security_trust_report.md index 1919ce0..094edde 100644 --- a/reports/security_trust_report.md +++ b/reports/security_trust_report.md @@ -1,9 +1,9 @@ # Security Trust Report - OK: `True` -- Scanned files: `153` -- Scripts: `69` -- Internal script modules: `9` +- Scanned files: `154` +- Scripts: `70` +- Internal script modules: `10` - Secret findings: `0` - Network-capable scripts: `3` - Network policy covered scripts: `3` @@ -15,8 +15,8 @@ - CLI help smoke failures: `0` - Interactive scripts: `0` - Package hash scope: `source-contract-without-generated-reports` -- Package hash files: `153` -- Package SHA256: `44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd` +- Package hash files: `154` +- Package SHA256: `495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2` ## Failures @@ -131,3 +131,4 @@ | scripts/yao.py | cli | False | True | True | False | False | False | True | Default CLI classification; add SCRIPT_INTERFACE for internal modules. | | scripts/yao_cli_config.py | internal-module | True | False | False | False | False | False | False | Imported by yao.py for CLI target maps and side-effect-free shaping helpers. | | scripts/yao_cli_parser.py | internal-module | True | True | False | False | False | False | False | Imported by yao.py to keep CLI parser declarations separate from command orchestration. | +| scripts/yao_cli_telemetry.py | internal-module | True | True | False | False | False | False | False | Imported by yao.py to record opt-in metadata-only CLI run telemetry. | diff --git a/reports/skill-os-2-review.md b/reports/skill-os-2-review.md index 7fee917..0e47bc9 100644 --- a/reports/skill-os-2-review.md +++ b/reports/skill-os-2-review.md @@ -22,6 +22,7 @@ Yao Meta Skill is no longer only a Meta Skill factory. The current working tree - Install Simulation v0 for local extraction, entrypoint loading, interface loading, report presence, adapter readability, and installer-level permission approval/enforcement checks. - Upgrade Check v0 for registry package diffs, semver bump recommendations, breaking-change notes, and release evidence. - Adoption Drift v0 for local-first metadata-only telemetry, privacy-field blocking, usage-signal aggregation, iteration candidate generation, and Skill Atlas drift-signal input. +- CLI Telemetry Capture v0 for opt-in `yao.py` command-run metadata that records command name, source, outcome, failure class, and timestamp without command arguments or raw user/model content. - Review Waivers v0 for human warning acceptance with reviewer, reason, scope, expiry, and blocker-safe policy. - Governed Permission Gates v0 for reviewer-approved network, file-write, and subprocess capabilities with scope, reason, expiry, evidence, and target-enforcement mapping. - Runtime Permission Probes v0 for packaged target adapter checks, explicit native-enforcement flags, metadata fallback evidence, and residual permission risks. @@ -33,7 +34,7 @@ Yao Meta Skill is no longer only a Meta Skill factory. The current working tree - Output Review Adjudication now preserves blind-review integrity by hiding expected winners for pending or invalid reviewer decisions; answer keys are revealed only after a valid A/B decision exists for that case. - Provider Output Eval Runner v0 so `python3 scripts/yao.py output-exec --provider-runner openai` can collect real provider-backed model evidence through a reviewed OpenAI Responses API compatible runner instead of ad hoc shell glue. -This is still not the final world-class state. Target-native behavior contracts are now explicit, VS Code / Copilot package metadata is auditable, local output-eval command execution is wired, blind-review answers remain hidden until valid decisions exist, a provider-backed output runner exists, installer-level permission coverage is now locally enforced during install simulation and local install sync, and Skill Atlas now consumes aggregate drift reports. Review Studio keeps pending human adjudication visible as a warning instead of treating it as a clean pass. Deeper provider-native execution transforms, real client telemetry capture, provider-native installer integration, real provider holdout runs, real human adjudication decisions, and native runtime permission enforcement remain open. +This is still not the final world-class state. Target-native behavior contracts are now explicit, VS Code / Copilot package metadata is auditable, local output-eval command execution is wired, blind-review answers remain hidden until valid decisions exist, a provider-backed output runner exists, installer-level permission coverage is now locally enforced during install simulation and local install sync, opt-in `yao.py` CLI telemetry can capture metadata-only real run signals, and Skill Atlas now consumes aggregate drift reports. Review Studio keeps pending human adjudication visible as a warning instead of treating it as a clean pass. Deeper provider-native execution transforms, external client telemetry capture, provider-native installer integration, real provider holdout runs, real human adjudication decisions, and native runtime permission enforcement remain open. ## Coverage Matrix @@ -54,7 +55,7 @@ This is still not the final world-class state. Target-native behavior contracts | Install Simulation | `scripts/simulate_install.py`, `reports/install_simulation.md`, `tests/verify_install_simulation.py` with installer permission coverage checks | v0 landed | | Local Install Sync Preflight | `scripts/sync_local_install.py`, `tests/verify_local_install_sync.py`, `Makefile` package-check prerequisites | v0 landed | | Upgrade Check | `scripts/upgrade_check.py`, `reports/upgrade_check.md`, `tests/verify_upgrade_check.py` | v0 landed | -| Telemetry & Drift | `scripts/render_adoption_drift_report.py`, `reports/adoption_drift_report.md`, `tests/verify_adoption_drift.py`, `references/telemetry-drift-method.md` | v0 landed | +| Telemetry & Drift | `scripts/render_adoption_drift_report.py`, `scripts/yao_cli_telemetry.py`, `reports/adoption_drift_report.md`, `tests/verify_adoption_drift.py`, `tests/verify_yao_cli.py`, `references/telemetry-drift-method.md` | v0 landed | | Review Waivers | `scripts/render_review_waivers.py`, `reports/review_waivers.md`, `tests/verify_review_waivers.py`, `references/review-waiver-method.md` | v0 landed | | Governed Permission Gates | `security/permission_policy.json`, `scripts/trust_check.py`, `scripts/render_review_studio.py`, `tests/verify_trust_check.py`, `tests/verify_review_studio.py` | v0 landed | | Runtime Permission Probes | `scripts/probe_runtime_permissions.py`, `reports/runtime_permission_probes.md`, `tests/verify_runtime_permission_probes.py`, `tests/verify_review_studio.py` | v0 landed | @@ -84,9 +85,9 @@ Next move: add richer source-line anchors inside generated reports and record re ### 4. Multi-skill operation now links Atlas with drift, but needs real client capture -The new Skill Atlas can scan a workspace and report catalog, route overlap, dependency graph, stale skill, missing owner/review metadata, aggregate drift signals, and no-route opportunities. It now also supports `skill_atlas/policy.json` so release gates distinguish actionable library skills from examples and fixtures while keeping full visibility. Adoption Drift v0 can record metadata-only local events, block raw prompt/output fields, summarize missed-trigger, bad-output, script-error, and review-overdue signals, feed next iteration candidates into Review Studio, and publish aggregate drift input for Atlas. It still needs real client-side capture instead of manual CLI event recording. +The new Skill Atlas can scan a workspace and report catalog, route overlap, dependency graph, stale skill, missing owner/review metadata, aggregate drift signals, and no-route opportunities. It now also supports `skill_atlas/policy.json` so release gates distinguish actionable library skills from examples and fixtures while keeping full visibility. Adoption Drift v0 can record metadata-only local events, block raw prompt/output fields, summarize missed-trigger, bad-output, script-error, and review-overdue signals, feed next iteration candidates into Review Studio, and publish aggregate drift input for Atlas. `yao.py` now adds opt-in automatic CLI run capture through `YAO_CLI_TELEMETRY=1` or `--record-cli-telemetry`, recording only `source=yao_cli`, normalized subcommand name, outcome, failure class, and timestamp. It still needs external client-side capture beyond the local CLI. -Next move: connect client activations and failure history so Atlas can rank stale, drifting, or conflicting skills by real usage impact. +Next move: connect non-CLI client activations and failure history so Atlas can rank stale, drifting, or conflicting skills by real usage impact. ### 5. Trust report is structural, not full security review @@ -106,11 +107,11 @@ Next move: add real client or installer permission enforcement integration. | Runtime Permission Probes | `4 / 4` target adapters probed, `0` native-enforcement adapters, `4` explicit metadata fallbacks, `4` residual risks retained for reviewer visibility | | Skill Atlas | `12` scanned skills, `1` actionable root skill, `1` telemetry report, `0` actionable route collisions, `0` actionable owner gaps, `0` actionable stale skills, `0` actionable drift signals, `24` scoped non-actionable issue signals retained for visibility | | Registry Audit | package metadata generated with version, owner, license, source checksum, archive checksum, Skill IR provenance, and compatibility matrix | -| Package Verification | `4 / 4` target adapters present, archive verified, `494` zip entries, `0` failures, `0` warnings | -| Install Simulation | archive with `494` entries extracted into a local verification root, entrypoint/manifest/interface loaded, reports present, `4` adapters readable, `12` installer permission checks enforced, `0` permission failures, `0` failures, `0` warnings | +| Package Verification | `4 / 4` target adapters present, archive verified, `495` zip entries, `0` failures, `0` warnings | +| Install Simulation | archive with `495` entries extracted into a local verification root, entrypoint/manifest/interface loaded, reports present, `4` adapters readable, `12` installer permission checks enforced, `0` permission failures, `0` failures, `0` warnings | | Local Install Sync Preflight | `make sync-local-install` and `make sync-active-install` rebuild the package first, then sync only after install simulation passes with `12` enforced installer permission checks and `0` permission failures | | Upgrade Check | current package declares `minor` over the 1.0.0 baseline, recommended bump is `minor`, and release notes include added targets plus checksum changes | -| Adoption Drift | `1` metadata-only review event, `0` adoption samples, adoption `0`, risk band `low`; raw `reports/telemetry_events.jsonl` is gitignored and blocked from zip packages | +| Adoption Drift | `1` metadata-only review event, `0` adoption samples, adoption `0`, risk band `low`; optional `yao.py` CLI capture is available but off by default for reproducible release evidence; raw `reports/telemetry_events.jsonl` is gitignored and blocked from zip packages | | Review Waivers | ledger generated; current release has `1` warning gate that still needs reviewer decision or a time-bounded waiver; blockers remain non-waivable in v0 | | Review Annotations | ledger generated; current release has `0` reviewer annotations and `0` open annotation blockers | | Review Studio | decision `review`, world-class score `92`, `13` gates, `0` blockers, `2` warnings, `2` review actions, `0` open annotation blockers | @@ -124,4 +125,4 @@ Next move: add real client or installer permission enforcement integration. 2. Add native client or installer runtime enforcement for approved high-permission capabilities. 3. Run the new provider-backed output runner against holdout cases with real credentials, then record the current blind A/B decisions before claiming fully ready status. 4. Add real reviewer annotation records during the next human review pass. -5. Connect real client telemetry capture and failure history so Atlas drift signals are based on live usage, not manual samples. +5. Connect external client telemetry capture and failure history so Atlas drift signals include non-CLI live usage, not only manual samples or opt-in CLI runs. diff --git a/reports/skill-overview.html b/reports/skill-overview.html index 3225f1f..6197154 100644 --- a/reports/skill-overview.html +++ b/reports/skill-overview.html @@ -609,7 +609,7 @@
  • 把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。把一次性经验沉淀为可复用、可评估、可迁移的 Skill 包体。
  • Skill 作者、复用团队和后续 reviewer。Skill 作者、复用团队和后续 reviewer。
  • 创建完成后建议先打开 reports/skill-overview.html,再继续扩展包体。创建完成后建议先打开 reports/skill-overview.html,再继续扩展包体。
-
完整度100SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
触发清晰100frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
证据充分100已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
上下文成本42入口约 350 个词/字,references 约 14711 个词/字。入口约 350 个词/字,references 约 14711 个词/字。
+
完整度100SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
触发清晰100frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
证据充分100已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
上下文成本42入口约 350 个词/字,references 约 14859 个词/字。入口约 350 个词/字,references 约 14859 个词/字。
@@ -641,10 +641,10 @@

指标判读Reading

先看雷达图判断能力短板,再看下方每项分数的证据原因。分数不是装饰数字,必须和本地文件、reports 证据或证据不足提示对应。Read the radar first for weak spots, then inspect each score with its evidence. Scores must map to local files, reports, or explicit evidence gaps.

-
  1. 稳定Stable完整度100SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
  2. 稳定Stable触发清晰100frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
  3. 稳定Stable证据充分100已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
  4. 稳定Stable可维护性100SKILL.md 约 350 个词/字。SKILL.md 约 350 个词/字。
  5. 稳定Stable可迁移性100agents/interface.yaml 已存在。agents/interface.yaml 已存在。
  6. 关注Watch上下文成本42入口约 350 个词/字,references 约 14711 个词/字。入口约 350 个词/字,references 约 14711 个词/字。
+
  1. 稳定Stable完整度100SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
  2. 稳定Stable触发清晰100frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
  3. 稳定Stable证据充分100已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
  4. 稳定Stable可维护性100SKILL.md 约 350 个词/字。SKILL.md 约 350 个词/字。
  5. 稳定Stable可迁移性100agents/interface.yaml 已存在。agents/interface.yaml 已存在。
  6. 关注Watch上下文成本42入口约 350 个词/字,references 约 14859 个词/字。入口约 350 个词/字,references 约 14859 个词/字。
-
完整度100
  • SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
  • README.md 已存在,便于人工阅读。README.md 已存在,便于人工阅读。
  • agents/interface.yaml 已存在,便于跨平台适配。agents/interface.yaml 已存在,便于跨平台适配。
触发清晰100
  • frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
  • description 有足够长度说明任务边界。description 有足够长度说明任务边界。
  • description 已包含使用场景或排除边界信号。description 已包含使用场景或排除边界信号。
证据充分100
  • 已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
  • skill-ir.json 已存在。skill-ir.json 已存在。
  • compiled_targets.json 已存在。compiled_targets.json 已存在。
可维护性100
  • SKILL.md 约 350 个词/字。SKILL.md 约 350 个词/字。
  • 入口文件保持克制,可维护性较好。入口文件保持克制,可维护性较好。
  • references/ 已承载扩展指导。references/ 已承载扩展指导。
可迁移性100
  • agents/interface.yaml 已存在。agents/interface.yaml 已存在。
  • manifest.json 已存在。manifest.json 已存在。
  • 目标平台或 adapter target 已声明。目标平台或 adapter target 已声明。
上下文成本42
  • 入口约 350 个词/字,references 约 14711 个词/字。入口约 350 个词/字,references 约 14711 个词/字。
  • 分数越高代表上下文成本越低。分数越高代表上下文成本越低。
  • 上下文成本偏高,建议压缩入口或拆分 references。上下文成本偏高,建议压缩入口或拆分 references。
+
完整度100
  • SKILL.md 已存在,是 Skill 的入口。SKILL.md 已存在,是 Skill 的入口。
  • README.md 已存在,便于人工阅读。README.md 已存在,便于人工阅读。
  • agents/interface.yaml 已存在,便于跨平台适配。agents/interface.yaml 已存在,便于跨平台适配。
触发清晰100
  • frontmatter description 已存在,具备基础路由面。frontmatter description 已存在,具备基础路由面。
  • description 有足够长度说明任务边界。description 有足够长度说明任务边界。
  • description 已包含使用场景或排除边界信号。description 已包含使用场景或排除边界信号。
证据充分100
  • 已生成 20 / 20 类报告证据。已生成 20 / 20 类报告证据。
  • skill-ir.json 已存在。skill-ir.json 已存在。
  • compiled_targets.json 已存在。compiled_targets.json 已存在。
可维护性100
  • SKILL.md 约 350 个词/字。SKILL.md 约 350 个词/字。
  • 入口文件保持克制,可维护性较好。入口文件保持克制,可维护性较好。
  • references/ 已承载扩展指导。references/ 已承载扩展指导。
可迁移性100
  • agents/interface.yaml 已存在。agents/interface.yaml 已存在。
  • manifest.json 已存在。manifest.json 已存在。
  • 目标平台或 adapter target 已声明。目标平台或 adapter target 已声明。
上下文成本42
  • 入口约 350 个词/字,references 约 14859 个词/字。入口约 350 个词/字,references 约 14859 个词/字。
  • 分数越高代表上下文成本越低。分数越高代表上下文成本越低。
  • 上下文成本偏高,建议压缩入口或拆分 references。上下文成本偏高,建议压缩入口或拆分 references。
@@ -720,7 +720,7 @@
- +
类型Type证据Evidence建议Action
强项Strength触发面保持精简,并锚定在 frontmatter description。The trigger surface stays lean and anchored in the frontmatter description.保留并复用Keep
强项Strength已生成 Skill IR,核心语义可先于平台打包被审查和迁移。已生成 Skill IR,核心语义可先于平台打包被审查和迁移。保留并复用Keep
强项Strength已生成目标编译报告,可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。已生成目标编译报告,可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。保留并复用Keep
缺口Gap上下文成本需要补强:入口约 350 个词/字,references 约 14711 个词/字。上下文成本需要补强:入口约 350 个词/字,references 约 14711 个词/字。纳入下一轮修复Fix next
强项Strength触发面保持精简,并锚定在 frontmatter description。The trigger surface stays lean and anchored in the frontmatter description.保留并复用Keep
强项Strength已生成 Skill IR,核心语义可先于平台打包被审查和迁移。已生成 Skill IR,核心语义可先于平台打包被审查和迁移。保留并复用Keep
强项Strength已生成目标编译报告,可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。已生成目标编译报告,可审查 IR 到 OpenAI、Claude、generic 等目标契约的映射。保留并复用Keep
缺口Gap上下文成本需要补强:入口约 350 个词/字,references 约 14859 个词/字。上下文成本需要补强:入口约 350 个词/字,references 约 14859 个词/字。纳入下一轮修复Fix next
@@ -771,7 +771,7 @@

让 reviewer 快速确认关键文件、目录和资产分布。Lets reviewers confirm key files, directories, and asset distribution quickly.

-
资产分布304项SKILL.mdREADME.mdagents/interface.yamlmanifest.jsonreferencesscripts
资产分布图展示当前包体的文件和目录重心。
+
资产分布305项SKILL.mdREADME.mdagents/interface.yamlmanifest.jsonreferencesscripts
资产分布图展示当前包体的文件和目录重心。
diff --git a/reports/skill-overview.json b/reports/skill-overview.json index 7f8672a..941260b 100644 --- a/reports/skill-overview.json +++ b/reports/skill-overview.json @@ -96,7 +96,7 @@ "label": "上下文成本", "score": 42, "reasons": [ - "入口约 350 个词/字,references 约 14711 个词/字。", + "入口约 350 个词/字,references 约 14859 个词/字。", "分数越高代表上下文成本越低。", "上下文成本偏高,建议压缩入口或拆分 references。" ] @@ -200,7 +200,7 @@ "已生成 Output Review Adjudication,可记录盲评决策、一致率和待评审项。" ], "gaps": [ - "上下文成本需要补强:入口约 350 个词/字,references 约 14711 个词/字。" + "上下文成本需要补强:入口约 350 个词/字,references 约 14859 个词/字。" ], "recommendations": [ "先改触发边界,再扩展工作流。", @@ -433,7 +433,7 @@ "path": "scripts", "label": "Deterministic helpers or local tooling", "kind": "folder", - "file_count": 69 + "file_count": 70 }, { "path": "evals", @@ -448,7 +448,7 @@ "file_count": 171 } ], - "file_count": 304, + "file_count": 305, "folder_count": 4, "distribution": [ { @@ -473,7 +473,7 @@ }, { "label": "scripts", - "value": 69 + "value": 70 }, { "label": "evals", @@ -602,7 +602,7 @@ "path": "scripts", "label": "Deterministic helpers or local tooling", "kind": "folder", - "file_count": 69 + "file_count": 70 }, { "path": "evals", @@ -948,9 +948,9 @@ "failures": [] }, "trust_security": { - "scanned_files": 153, - "script_count": 69, - "internal_module_count": 9, + "scanned_files": 154, + "script_count": 70, + "internal_module_count": 10, "secret_findings": 0, "dependency_files": [ "requirements-ci.txt" @@ -968,8 +968,8 @@ "help_smoke_failed_count": 0, "interactive_script_count": 0, "package_hash_scope": "source-contract-without-generated-reports", - "package_hash_file_count": 153, - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "package_hash_file_count": 154, + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" }, "skill_atlas": { "skill_count": 12, @@ -1007,8 +1007,8 @@ "trust_level": "local", "license": "MIT", "checksums": { - "package_sha256": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd", - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "package_sha256": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, "compatibility": { "openai": "pass", @@ -1039,7 +1039,7 @@ }, "distribution": { "archive_verified": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", "package_verification": "reports/package_verification.json", "install_simulated": true, "install_simulation": "reports/install_simulation.json" @@ -1055,8 +1055,8 @@ "target_count": 4, "adapter_count": 4, "archive_present": true, - "archive_sha256": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933", - "archive_entry_count": 494, + "archive_sha256": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474", + "archive_entry_count": 495, "failure_count": 0, "warning_count": 0 }, @@ -1067,7 +1067,7 @@ "ok": true, "summary": { "archive_present": true, - "archive_entry_count": 494, + "archive_entry_count": 495, "archive_extracted": true, "entrypoint_loaded": true, "manifest_loaded": true, @@ -1134,12 +1134,12 @@ { "field": "archive_sha256", "from": "", - "to": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "to": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "to": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, @@ -1156,14 +1156,14 @@ "ok": true, "summary": { "event_count": 1, - "adoption_sample_count": 1, - "activation_count": 1, - "accepted_count": 1, + "adoption_sample_count": 0, + "activation_count": 0, + "accepted_count": 0, "edited_count": 0, "rejected_count": 0, "missed_count": 0, "failed_count": 0, - "adoption_rate": 100.0, + "adoption_rate": 0, "missed_trigger_count": 0, "wrong_trigger_count": 0, "bad_output_count": 0, @@ -1172,9 +1172,13 @@ "review_overdue_count": 0, "risk_band": "low", "event_types": { - "skill_activation": 1 + "review_event": 1 }, - "failure_types": {} + "failure_types": {}, + "source_types": { + "manual": 1 + }, + "command_counts": {} }, "next_iteration_candidates": [], "privacy_contract": { diff --git a/reports/skill_atlas.json b/reports/skill_atlas.json index b59a793..492bb5d 100644 --- a/reports/skill_atlas.json +++ b/reports/skill_atlas.json @@ -133,6 +133,7 @@ "scripts/yao.py", "scripts/yao_cli_config.py", "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py", "references/artifact-design-doctrine.md", "references/authoring-discipline.md", "references/distribution-registry-method.md", diff --git a/reports/upgrade_check.json b/reports/upgrade_check.json index e228f00..897b5ee 100644 --- a/reports/upgrade_check.json +++ b/reports/upgrade_check.json @@ -70,12 +70,12 @@ { "field": "archive_sha256", "from": "", - "to": "cdadcbc9a0c92eba0c80a9f4fbd840530d32d19bf3cbb508a24bebb8e8551933" + "to": "0f80e2e194d003a99da34f73923b9459f260b2d177c5f1f6ec7104fdd5713474" }, { "field": "package_sha256", "from": "0000000000000000000000000000000000000000000000000000000000000000", - "to": "44d9cb1aa096fff1b2c8844a609574c05c8ed9ae12005dc74a5662d3d94013fd" + "to": "495c5824cf84b56ef77bebc75e415191425e2f1131f0a4027d17037631b52fa2" } ] }, diff --git a/scripts/render_adoption_drift_report.py b/scripts/render_adoption_drift_report.py index 518cc2f..c1bf6d4 100644 --- a/scripts/render_adoption_drift_report.py +++ b/scripts/render_adoption_drift_report.py @@ -23,14 +23,17 @@ ALLOWED_FAILURE_TYPES = { "review_overdue", } ALLOWED_FIELDS = { + "command", "event", "skill", + "source", "version", "activation_type", "outcome", "failure_type", "timestamp", } +ALLOWED_SOURCES = {"manual", "yao_cli", "external", "unknown"} SENSITIVE_FIELDS = { "prompt", "content", @@ -112,6 +115,8 @@ def normalize_event(raw: dict[str, Any], defaults: dict[str, str], line_label: s timestamp = str(raw.get("timestamp") or utc_now()) skill = str(raw.get("skill") or defaults["skill"]) version = str(raw.get("version") or defaults["version"]) + source = str(raw.get("source") or "manual") + command = str(raw.get("command") or "unknown") if event not in ALLOWED_EVENTS: failures.append(f"{line_label}: unsupported event `{event}`") @@ -121,12 +126,18 @@ def normalize_event(raw: dict[str, Any], defaults: dict[str, str], line_label: s failures.append(f"{line_label}: unsupported outcome `{outcome}`") if failure_type not in ALLOWED_FAILURE_TYPES: failures.append(f"{line_label}: unsupported failure_type `{failure_type}`") + if source not in ALLOWED_SOURCES: + failures.append(f"{line_label}: unsupported source `{source}`") + if not command.replace("-", "").replace("_", "").isalnum() or len(command) > 64: + failures.append(f"{line_label}: command must use only letters, numbers, hyphens, or underscores and stay under 64 chars") if failures: return None, failures return { + "command": command, "event": event, "skill": skill, + "source": source, "version": version, "activation_type": activation_type, "outcome": outcome, @@ -254,6 +265,8 @@ def summarize(events: list[dict[str, str]], review_overdue_count: int) -> dict[s outcomes = Counter(event["outcome"] for event in adoption_events) failures = Counter(event["failure_type"] for event in events if event["failure_type"] != "none") event_types = Counter(event["event"] for event in events) + source_types = Counter(event.get("source", "manual") for event in events) + command_counts = Counter(event.get("command", "unknown") for event in events if event.get("command", "unknown") != "unknown") adopted = outcomes["accepted"] + outcomes["edited"] event_count = len(events) adoption_sample_count = len(adoption_events) @@ -289,6 +302,8 @@ def summarize(events: list[dict[str, str]], review_overdue_count: int) -> dict[s "risk_band": risk_band, "event_types": dict(sorted(event_types.items())), "failure_types": dict(sorted(failures.items())), + "source_types": dict(sorted(source_types.items())), + "command_counts": dict(sorted(command_counts.items())), } @@ -339,6 +354,7 @@ def render_markdown(report: dict[str, Any]) -> str: for event in report["recent_events"]: lines.append( f"- `{event['timestamp']}` `{event['skill']}` event=`{event['event']}` " + f"source=`{event.get('source', 'manual')}` command=`{event.get('command', 'unknown')}` " f"activation=`{event['activation_type']}` outcome=`{event['outcome']}` failure=`{event['failure_type']}`" ) if not report["recent_events"]: @@ -415,6 +431,8 @@ def main() -> None: parser.add_argument("--activation-type", choices=sorted(ALLOWED_ACTIVATION_TYPES), default="unknown") parser.add_argument("--outcome", choices=sorted(ALLOWED_OUTCOMES), default="unknown") parser.add_argument("--failure-type", choices=sorted(ALLOWED_FAILURE_TYPES), default="none") + parser.add_argument("--source", choices=sorted(ALLOWED_SOURCES), default="manual") + parser.add_argument("--command", default="unknown") parser.add_argument("--timestamp") parser.add_argument("--skill-name") parser.add_argument("--version") @@ -427,6 +445,8 @@ def main() -> None: "activation_type": args.activation_type, "outcome": args.outcome, "failure_type": args.failure_type, + "source": args.source, + "command": args.command, } if args.timestamp: record_event["timestamp"] = args.timestamp diff --git a/scripts/yao.py b/scripts/yao.py index e8d74fc..4de4f77 100644 --- a/scripts/yao.py +++ b/scripts/yao.py @@ -23,6 +23,7 @@ from yao_cli_config import ( resolve_target, ) from yao_cli_parser import build_parser as build_cli_parser +from yao_cli_telemetry import add_telemetry_args, maybe_record_cli_event ROOT = Path(__file__).resolve().parent.parent @@ -691,6 +692,8 @@ def command_adoption_drift(args: argparse.Namespace) -> int: cmd.extend(["--activation-type", args.activation_type]) cmd.extend(["--outcome", args.outcome]) cmd.extend(["--failure-type", args.failure_type]) + cmd.extend(["--source", args.source]) + cmd.extend(["--command", args.telemetry_command]) if args.timestamp: cmd.extend(["--timestamp", args.timestamp]) if args.skill_name: @@ -1182,13 +1185,20 @@ def command_check_update(args: argparse.Namespace) -> int: def build_parser() -> argparse.ArgumentParser: - return build_cli_parser({name: value for name, value in globals().items() if name.startswith("command_")}) + parser = build_cli_parser({name: value for name, value in globals().items() if name.startswith("command_")}) + add_telemetry_args(parser) + return parser def main() -> None: parser = build_parser() args = parser.parse_args() - raise SystemExit(args.func(args)) + returncode = 2 + try: + returncode = args.func(args) + finally: + maybe_record_cli_event(ROOT, args, returncode) + raise SystemExit(returncode) if __name__ == "__main__": diff --git a/scripts/yao_cli_parser.py b/scripts/yao_cli_parser.py index c4fe44b..caa5274 100644 --- a/scripts/yao_cli_parser.py +++ b/scripts/yao_cli_parser.py @@ -277,6 +277,8 @@ def build_parser(command_handlers: dict[str, Callable[[argparse.Namespace], int] ], default="none", ) + adoption_drift_cmd.add_argument("--source", choices=["external", "manual", "unknown", "yao_cli"], default="manual") + adoption_drift_cmd.add_argument("--command", dest="telemetry_command", default="unknown") adoption_drift_cmd.add_argument("--timestamp") adoption_drift_cmd.add_argument("--skill-name") adoption_drift_cmd.add_argument("--version") diff --git a/scripts/yao_cli_telemetry.py b/scripts/yao_cli_telemetry.py new file mode 100644 index 0000000..4bb54b1 --- /dev/null +++ b/scripts/yao_cli_telemetry.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +"""Metadata-only CLI telemetry helpers for yao.py.""" + +import argparse +import os +import sys +from pathlib import Path +from typing import Any + +from render_adoption_drift_report import append_event, normalize_event, skill_defaults, utc_now + + +SCRIPT_INTERFACE = "internal-module" +SCRIPT_INTERFACE_REASON = "Imported by yao.py to record opt-in metadata-only CLI run telemetry." + +ENABLE_ENV = "YAO_CLI_TELEMETRY" +EVENTS_ENV = "YAO_CLI_TELEMETRY_EVENTS" +TRUTHY = {"1", "true", "yes", "on"} +FALSY = {"0", "false", "no", "off"} + + +def add_telemetry_args(parser: argparse.ArgumentParser) -> None: + parser.add_argument( + "--record-cli-telemetry", + action="store_true", + help="Record metadata-only yao.py command telemetry to reports/telemetry_events.jsonl.", + ) + parser.add_argument( + "--no-cli-telemetry", + action="store_true", + help="Disable yao.py command telemetry even when YAO_CLI_TELEMETRY is enabled.", + ) + parser.add_argument( + "--telemetry-events-jsonl", + help="Override the local metadata-only telemetry JSONL path.", + ) + + +def telemetry_enabled(args: argparse.Namespace, environ: dict[str, str] | None = None) -> bool: + environ = environ or os.environ + if getattr(args, "no_cli_telemetry", False): + return False + if getattr(args, "record_cli_telemetry", False): + return True + raw = environ.get(ENABLE_ENV, "").strip().lower() + if raw in TRUTHY: + return True + if raw in FALSY: + return False + return False + + +def telemetry_path(root: Path, args: argparse.Namespace, environ: dict[str, str] | None = None) -> Path: + environ = environ or os.environ + configured = getattr(args, "telemetry_events_jsonl", None) or environ.get(EVENTS_ENV) + if configured: + return Path(configured).expanduser().resolve() + return root / "reports" / "telemetry_events.jsonl" + + +def normalize_command_name(value: Any) -> str: + raw = str(value or "unknown") + lowered = raw.strip().lower() + safe = "".join(char for char in lowered if char.isalnum() or char in {"-", "_"}) + return safe[:64] or "unknown" + + +def cli_event(root: Path, args: argparse.Namespace, returncode: int) -> dict[str, str]: + defaults = skill_defaults(root) + ok = returncode == 0 + return { + "event": "script_run", + "skill": defaults["skill"], + "version": defaults["version"], + "activation_type": "manual", + "outcome": "accepted" if ok else "failed", + "failure_type": "none" if ok else "script_error", + "timestamp": utc_now(), + "source": "yao_cli", + "command": normalize_command_name(getattr(args, "command", "unknown")), + } + + +def maybe_record_cli_event(root: Path, args: argparse.Namespace, returncode: int) -> None: + if not telemetry_enabled(args): + return + path = telemetry_path(root, args) + event, failures = normalize_event(cli_event(root, args, returncode), skill_defaults(root), "yao-cli") + if failures or event is None: + sys.stderr.write(f"Telemetry skipped: {'; '.join(failures)}\n") + return + try: + append_event(path, event) + except OSError as exc: + sys.stderr.write(f"Telemetry skipped: {exc}\n") diff --git a/skill-ir/examples/yao-meta-skill.json b/skill-ir/examples/yao-meta-skill.json index 4615898..7594699 100644 --- a/skill-ir/examples/yao-meta-skill.json +++ b/skill-ir/examples/yao-meta-skill.json @@ -176,7 +176,8 @@ "scripts/verify_package.py", "scripts/yao.py", "scripts/yao_cli_config.py", - "scripts/yao_cli_parser.py" + "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py" ], "assets": [ "templates/basic_skill.md.j2", diff --git a/skill_atlas/catalog.json b/skill_atlas/catalog.json index 4b3b556..8b95c36 100644 --- a/skill_atlas/catalog.json +++ b/skill_atlas/catalog.json @@ -89,6 +89,7 @@ "scripts/yao.py", "scripts/yao_cli_config.py", "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py", "references/artifact-design-doctrine.md", "references/authoring-discipline.md", "references/distribution-registry-method.md", diff --git a/tests/verify_adoption_drift.py b/tests/verify_adoption_drift.py index fb2e739..03bdf2f 100644 --- a/tests/verify_adoption_drift.py +++ b/tests/verify_adoption_drift.py @@ -103,6 +103,10 @@ def main() -> None: "failed", "--failure-type", "script_error", + "--source", + "yao_cli", + "--command", + "validate", "--timestamp", "2026-06-13T10:04:00Z", ) @@ -115,11 +119,14 @@ def main() -> None: assert summary["bad_output_count"] == 1, summary assert summary["script_error_count"] == 1, summary assert summary["risk_band"] == "high", summary + assert summary["source_types"]["yao_cli"] == 1, summary + assert summary["command_counts"]["validate"] == 1, summary assert final["payload"]["privacy_contract"]["raw_content_allowed"] is False, final assert final["payload"]["next_iteration_candidates"], final markdown = (skill_dir / "reports" / "adoption_drift_report.md").read_text(encoding="utf-8") assert "metadata-only telemetry" in markdown, markdown assert "Raw user prompts" in markdown, markdown + assert "source=`yao_cli` command=`validate`" in markdown, markdown unsafe_events = TMP / "unsafe_events.jsonl" unsafe_events.write_text( diff --git a/tests/verify_trust_check.py b/tests/verify_trust_check.py index 0348df2..ae96867 100644 --- a/tests/verify_trust_check.py +++ b/tests/verify_trust_check.py @@ -97,6 +97,7 @@ def main() -> None: "scripts/skill_report_model.py", "scripts/yao_cli_config.py", "scripts/yao_cli_parser.py", + "scripts/yao_cli_telemetry.py", ]: assert script_map[internal_module]["interface"] == "internal-module", script_map[internal_module] assert script_map[internal_module]["interface_declared"], script_map[internal_module] @@ -110,6 +111,7 @@ def main() -> None: assert "skill_report_model.py" not in warning_text, payload["warnings"] assert "yao_cli_config.py" not in warning_text, payload["warnings"] assert "yao_cli_parser.py" not in warning_text, payload["warnings"] + assert "yao_cli_telemetry.py" not in warning_text, payload["warnings"] assert "render_context_reports.py" not in warning_text, payload["warnings"] assert "render_social_preview.py" not in warning_text, payload["warnings"] assert "Network-capable scripts require bounded host policy" not in warning_text, payload["warnings"] diff --git a/tests/verify_yao_cli.py b/tests/verify_yao_cli.py index 1b84046..fe1eace 100644 --- a/tests/verify_yao_cli.py +++ b/tests/verify_yao_cli.py @@ -16,13 +16,35 @@ import yao_cli_parser # noqa: E402 def run(*args: str, input_text: str | None = None) -> dict: + env = dict(os.environ) + env["YAO_CLI_TELEMETRY"] = "0" + env.pop("YAO_CLI_TELEMETRY_EVENTS", None) proc = subprocess.run( [sys.executable, str(CLI), *args], cwd=ROOT, capture_output=True, text=True, input=input_text, - env=os.environ, + env=env, + ) + payload = json.loads(proc.stdout) + return { + "ok": proc.returncode == 0, + "returncode": proc.returncode, + "payload": payload, + "stderr": proc.stderr, + } + + +def run_with_env(extra_env: dict[str, str], *args: str) -> dict: + env = dict(os.environ) + env.update(extra_env) + proc = subprocess.run( + [sys.executable, str(CLI), *args], + cwd=ROOT, + capture_output=True, + text=True, + env=env, ) payload = json.loads(proc.stdout) return { @@ -51,6 +73,7 @@ def main() -> None: parser_help = yao_cli_module.build_parser().format_help() assert "quickstart" in parser_help, parser_help assert "review-studio" in parser_help, parser_help + assert "--record-cli-telemetry" in parser_help, parser_help init_result = run("init", "cli-demo-skill", "--description", "CLI demo skill.", "--output-dir", str(tmp_root)) assert init_result["ok"], init_result @@ -96,6 +119,33 @@ def main() -> None: assert init_skill_ir["name"] == "cli-demo-skill", init_skill_ir assert init_skill_ir["trigger_samples"] >= 1, init_skill_ir + telemetry_log = tmp_root / "cli-telemetry-events.jsonl" + telemetry_env = { + "YAO_CLI_TELEMETRY": "1", + "YAO_CLI_TELEMETRY_EVENTS": str(telemetry_log), + } + telemetry_ok = run_with_env(telemetry_env, "validate", str(created)) + assert telemetry_ok["ok"], telemetry_ok + telemetry_fail = run_with_env( + telemetry_env, + "output-exec", + "--runner-command", + json.dumps([sys.executable, str(ROOT / "scripts" / "local_output_eval_runner.py")]), + "--provider-runner", + "openai", + ) + assert not telemetry_fail["ok"], telemetry_fail + telemetry_events = [json.loads(line) for line in telemetry_log.read_text(encoding="utf-8").splitlines()] + assert len(telemetry_events) == 2, telemetry_events + assert telemetry_events[0]["event"] == "script_run", telemetry_events + assert telemetry_events[0]["source"] == "yao_cli", telemetry_events + assert telemetry_events[0]["command"] == "validate", telemetry_events + assert telemetry_events[0]["outcome"] == "accepted", telemetry_events + assert telemetry_events[0]["failure_type"] == "none", telemetry_events + assert telemetry_events[1]["command"] == "output-exec", telemetry_events + assert telemetry_events[1]["outcome"] == "failed", telemetry_events + assert telemetry_events[1]["failure_type"] == "script_error", telemetry_events + quickstart_result = run( "quickstart", "--output-dir",
路径Path作用Role类型Type
SKILL.mdSkill 入口文件Skill entrypoint文件file
README.md人类可读使用说明Human-readable usage guide文件file
agents/interface.yaml跨平台接口元数据Neutral interface metadata文件file
manifest.json生命周期与打包元数据Lifecycle and portability metadata文件file
references扩展指导与复用资料Extended guidance and reusable notes目录folder
scripts确定性脚本或本地工具Deterministic helpers or local tooling目录folder
evals触发与质量检查Trigger and quality checks目录folder
reports生成的证据与总结报告Generated evidence and overview artifacts目录folder