yao-meta-skill/references/telemetry-drift-method.md

# Telemetry And Drift Method

Telemetry turns real use into the next iteration queue. It must stay local-first and metadata-only by default.

## When To Use

Use the telemetry drift loop when a skill is production, library, governed, team-distributed, or repeatedly invoked by more than one workflow.

Do not collect raw prompts, model outputs, transcripts, notes, messages, or private files. If a reviewer needs examples, store anonymized fixtures separately and cite them as eval evidence, not telemetry.

## Event Contract

The local event stream is `reports/telemetry_events.jsonl`. It is intentionally narrow:

```json
{
  "event": "skill_activation",
  "skill": "example-skill",
  "version": "2.0.0",
  "source": "yao_cli",
  "command": "quickstart",
  "activation_type": "implicit",
  "outcome": "accepted",
  "failure_type": "none",
  "timestamp": "2026-06-13T10:00:00Z"
}
```

Allowed events: `skill_activation`, `skill_output`, `script_run`, `review_event`.

Allowed sources: `manual`, `yao_cli`, `external`, `unknown`.

Allowed outcomes: `accepted`, `edited`, `rejected`, `missed`, `failed`, `reviewed`, `unknown`.

Allowed failure types: `wrong_trigger`, `under_trigger`, `bad_output`, `missing_resource`, `script_error`, `review_overdue`, `none`.

`source` and `command` are metadata fields. They may identify that `yao.py` ran `quickstart`, `validate`, `output-exec`, or another subcommand, but they must not include arguments, prompt text, file content, model output, transcripts, or reviewer notes.

## CLI Capture

`scripts/yao.py` can record metadata-only `script_run` events automatically. It is opt-in to keep release evidence reproducible and avoid surprising local writes:

```bash
YAO_CLI_TELEMETRY=1 python3 scripts/yao.py validate .
```

Optional destination override:

```bash
YAO_CLI_TELEMETRY=1 \
YAO_CLI_TELEMETRY_EVENTS=/tmp/yao-telemetry.jsonl \
python3 scripts/yao.py output-exec
```

Equivalent global flags are available before the subcommand:

```bash
python3 scripts/yao.py --record-cli-telemetry validate .
python3 scripts/yao.py --no-cli-telemetry validate .
```

Successful CLI runs record `event=script_run`, `source=yao_cli`, `outcome=accepted`, and `failure_type=none`. Failed CLI runs record `outcome=failed` and `failure_type=script_error`. The command name is normalized to the subcommand only; command arguments are never recorded.

## External Client Emit

External clients, browser extensions, editor adapters, or wrapper scripts can emit one sanitized event at a time into a local spool before importing it into the aggregate drift report:

```bash
python3 scripts/yao.py telemetry-emit . \
  --event skill_activation \
  --activation-type explicit \
  --outcome accepted \
  --command browser-extension
```

By default this writes to `.yao/telemetry_spool/external_events.jsonl`. Use `--output-jsonl` when a client needs a different local handoff path:

```bash
python3 scripts/yao.py telemetry-emit . \
  --output-jsonl /tmp/external-client-events.jsonl \
  --event skill_output \
  --activation-type manual \
  --outcome edited \
  --command browser-plugin
```

Use `--dry-run` to validate a proposed event without writing to the spool. The emitter uses the same metadata-only contract as import: no prompt, input, output, transcript, message, note, raw text, arguments, or unknown fields are accepted.

After a client finishes a batch, import the spool:

```bash
python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
```

## External Client Import

External clients, browser extensions, editor adapters, or wrapper scripts may hand off already-sanitized JSONL through `telemetry-import`:

```bash
python3 scripts/yao.py telemetry-import . \
  --input-jsonl /tmp/external-client-events.jsonl \
  --command browser-extension
```

The importer defaults missing `source` to `external` and missing `command` to `external-client`. It validates the entire JSONL file before writing anything. If any line includes a raw content field, unsupported source, unsupported outcome, unsupported failure type, unknown field, malformed JSON, or an unsafe command name, the whole import is rejected and the existing local event stream is left untouched.

Use `--dry-run` to validate an external batch without writing `reports/telemetry_events.jsonl` or refreshing aggregate reports:

```bash
python3 scripts/yao.py telemetry-import . --input-jsonl /tmp/external-client-events.jsonl --dry-run
```

## Privacy Rule

The raw JSONL event log is local evidence and should not be distributed in skill packages. The distributable artifact is the aggregate report:

- `reports/adoption_drift_report.json`
- `reports/adoption_drift_report.md`

Package builders should exclude `reports/telemetry_events.jsonl`. The root repository also ignores this raw event stream so local usage evidence does not become ordinary source history by accident.

## Release Interpretation

- `no-data`: acceptable for a first scaffold, but a warning for governed release review.
- `low`: events exist and no drift failure signal is present.
- `medium`: at least one missed trigger, wrong trigger, bad output, script error, or overdue review signal exists.
- `high`: several drift signals are present; convert them into eval cases or governance actions before calling the skill release-ready.

## Iteration Loop

1. Capture metadata-only events locally, either manually with `adoption-drift --record-event`, automatically with opt-in `yao.py` CLI capture, through `telemetry-emit` client hooks, or through validated external JSONL import.
2. Render `reports/adoption_drift_report.md`.
3. Convert missed triggers into trigger eval cases.
4. Convert bad outputs into Output Eval assertions and failure taxonomy entries.
5. Convert script errors into non-interactive smoke tests.
6. Feed review-overdue signals back into Skill Atlas and owner review.
7. Let Skill Atlas read only `reports/adoption_drift_report.json` and publish portfolio-level `skill_atlas/drift_signals.json`.

## Review Studio Role

Review Studio should show the aggregate telemetry gate as an operating loop, not as raw logs. A blocker means the telemetry contract was violated. A warning means the evidence is absent or the drift signal needs a follow-up case.

## Skill Atlas Role

Skill Atlas uses aggregate adoption drift reports to rank portfolio work. It should surface no-data warnings for actionable production/library/governed skills, and drift warnings for missed triggers, wrong triggers, bad outputs, missing resources, script errors, and review-overdue counts. It must not inspect raw JSONL telemetry or use non-actionable example/fixture signals as release blockers.