Telemetry And Drift Method

Telemetry turns real use into the next iteration queue. It must stay local-first and metadata-only by default.

When To Use

Use the telemetry drift loop when a skill is production, library, governed, team-distributed, or repeatedly invoked by more than one workflow.

Do not collect raw prompts, model outputs, transcripts, notes, messages, or private files. If a reviewer needs examples, store anonymized fixtures separately and cite them as eval evidence, not telemetry.

Event Contract

The local event stream is reports/telemetry_events.jsonl. It is intentionally narrow:

{
  "event": "skill_activation",
  "skill": "example-skill",
  "version": "2.0.0",
  "source": "yao_cli",
  "command": "quickstart",
  "activation_type": "implicit",
  "outcome": "accepted",
  "failure_type": "none",
  "timestamp": "2026-06-13T10:00:00Z"
}

Allowed events: skill_activation, skill_output, script_run, review_event.

Allowed sources: manual, yao_cli, external, unknown.

Allowed outcomes: accepted, edited, rejected, missed, failed, reviewed, unknown.

Allowed failure types: wrong_trigger, under_trigger, bad_output, missing_resource, script_error, review_overdue, none.

source and command are metadata fields. They may identify that yao.py ran quickstart, validate, output-exec, or another subcommand, but they must not include arguments, prompt text, file content, model output, transcripts, or reviewer notes.

CLI Capture

scripts/yao.py can record metadata-only script_run events automatically. It is opt-in to keep release evidence reproducible and avoid surprising local writes:

YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate .

Optional destination override:

YAO_CLI_TELEMETRY=1 \
YAO_CLI_TELEMETRY_EVENTS=/tmp/yao-telemetry.jsonl \
python3 scripts/yao.py output-exec

Equivalent global flags are available before the subcommand:

python3 scripts/yao.py --record-cli-telemetry validate .
python3 scripts/yao.py --no-cli-telemetry validate .

Successful CLI runs record event=script_run, source=yao_cli, outcome=accepted, and failure_type=none. Failed CLI runs record outcome=failed and failure_type=script_error. The command name is normalized to the subcommand only; command arguments are never recorded.

Privacy Rule

The raw JSONL event log is local evidence and should not be distributed in skill packages. The distributable artifact is the aggregate report:

reports/adoption_drift_report.json
reports/adoption_drift_report.md

Package builders should exclude reports/telemetry_events.jsonl. The root repository also ignores this raw event stream so local usage evidence does not become ordinary source history by accident.

Release Interpretation

no-data: acceptable for a first scaffold, but a warning for governed release review.
low: events exist and no drift failure signal is present.
medium: at least one missed trigger, wrong trigger, bad output, script error, or overdue review signal exists.
high: several drift signals are present; convert them into eval cases or governance actions before calling the skill release-ready.

Iteration Loop

Capture metadata-only events locally, either manually with adoption-drift --record-event or automatically with opt-in yao.py CLI capture.
Render reports/adoption_drift_report.md.
Convert missed triggers into trigger eval cases.
Convert bad outputs into Output Eval assertions and failure taxonomy entries.
Convert script errors into non-interactive smoke tests.
Feed review-overdue signals back into Skill Atlas and owner review.
Let Skill Atlas read only reports/adoption_drift_report.json and publish portfolio-level skill_atlas/drift_signals.json.

Review Studio Role

Review Studio should show the aggregate telemetry gate as an operating loop, not as raw logs. A blocker means the telemetry contract was violated. A warning means the evidence is absent or the drift signal needs a follow-up case.

Skill Atlas Role

Skill Atlas uses aggregate adoption drift reports to rank portfolio work. It should surface no-data warnings for actionable production/library/governed skills, and drift warnings for missed triggers, wrong triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not inspect raw JSONL telemetry or use non-actionable example/fixture signals as release blockers.

4.4 KiB Raw Blame History