Evidence Operations

# World-Class Operator Runbook

A single operating page for collecting the remaining human and external evidence. It coordinates action, but does not accept evidence or change the ledger.

Pending 4
Awaiting 4
Ready 0
Source 12/19
Queue 2/2
Steps 6/6
Gate 4/5
Blocked 7
Invalid 0

## Fast Path

1. Run the real external or human work for one evidence item.
2. Generate and fill the matching submission draft.
3. Validate intake and inspect the submission review queue.
4. Refresh the ledger and run the claim guard before making any completion claim.

## Coordination Plan

These steps tell the assistant what to run and where the user must supply real provider, reviewer, client, or telemetry input. Every row is operational guidance only and keeps counts_as_completion false.

Step | Evidence | Owner | User | User action | Assistant action | Command | Pass condition
--- | --- | --- | --- | --- | --- | --- | ---

`prepare-evidence-session`

`all`

assistant + user

`true`

Confirm provider access, reviewer availability, target client path, and telemetry client path before collection starts.

Run preflight and prepare submission drafts without accepting them as evidence.

`python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts`

Preflight lists the same pending evidence keys and no credential values are printed.

`collect-provider-holdout`

`provider-holdout`

assistant + operator with provider credentials

`true`

Provide the selected provider API key through an environment variable and confirm the provider, model, endpoint, and API format to use.

Run provider-backed output execution, verify aggregate timing and token metadata, then prepare the evidence packet.

`python3 scripts/yao.py output-exec . --provider-runner <openai|deepseek> --provider-model <model> --timeout-seconds 60`

reports/output_execution_runs.json has model_executed_count > 0 and token_observed_count > 0.

`collect-human-adjudication`

`human-adjudication`

human reviewer + assistant

`true`

Open the blind review kit, choose winners for all pairs, add reviewer metadata and reasons, and keep the answer key hidden until decisions are saved.

Generate the review kit, import decisions, validate integrity, and prepare the human evidence packet.

`python3 scripts/yao.py output-review-kit . && python3 scripts/yao.py output-review .`

reports/output_review_adjudication.json has pending_count == 0 and ready_for_human_evidence == true.

`collect-native-permission-enforcement`

`native-permission-enforcement`

target client or installer integrator + assistant

`true`

Select a real target client or external installer guard that can enforce declared capabilities instead of metadata-only fallback.

Run runtime permission probes, package verification, install simulation, and prepare the native enforcement evidence packet.

`python3 scripts/yao.py runtime-permissions . --package-dir dist`

reports/runtime_permission_probes.json has native_enforcement_count > 0 and failure_count == 0.

`collect-native-client-telemetry`

`native-client-telemetry`

real client integrator + assistant

`true`

Install the native host manifest in a real Browser, Chrome, IDE, or provider client and trigger a metadata-only event.

Generate native host assets, import the external event JSONL, refresh adoption drift, and prepare the telemetry evidence packet.

`python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl --source external`

reports/adoption_drift_report.json has source_types.external > 0 and adoption_sample_count > 0.

`review-and-release-gate`

`all`

assistant + ledger reviewer

`true`

Approve only validated evidence packets and confirm the release wording after the claim guard passes.

Run intake, submission review, ledger, claim guard, benchmark, evidence consistency, Review Studio, and CI before final publish.

`python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions && python3 scripts/yao.py world-class-claim-guard . && make ci-test`

Ledger ready_to_claim_world_class, benchmark public_claim_ready, claim guard violation_count == 0, Review Studio has no blockers, and CI passes.

## Phase Queue

- Unblock access `unblock-access: 4 / 4` Assign a real reviewer identity before claiming human adjudication.
- Collect source `collect-source: 7 / 7` Set reviewer_attestation only after choices are completed before opening the answer key.

## Evidence Items

external · awaiting-submission

### Provider Holdout

No evidence packet has been submitted for review.

Owner
  operator with provider credentials
Ledger
  `pending`
Blocked
  `0`
Queue
  `1`
Submission
  `evidence/world_class/submissions/provider-holdout.json`

#### Phase Queue

- Unblock access `unblock-access: 1 / 1` Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.

#### Source Runbook

- Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
- For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60
- For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

#### Commands

- prepare submission `python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions`
- validate intake `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
- review queue `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
- refresh ledger `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
- guard claim `python3 scripts/yao.py world-class-claim-guard .`

#### Must Collect

- provider-backed model run
- observed timing
- observed token metadata

#### Success Checks

- reports/output_execution_runs.json summary.model_executed_count > 0
- reports/output_execution_runs.json summary.timing_observed_count > 0
- reports/output_execution_runs.json summary.token_observed_count > 0
- reports/skill_os2_audit.json item provider-holdout status becomes pass

#### Privacy

- Do not commit provider credentials or environment dumps.
- The output execution report records output hashes and aggregate run metadata, not raw provider prompts.

#### Next Source Actions

- No blocked source checks.

#### Source Evidence Snapshot

- Provider model run `model_executed_count: 10 / >0` Run provider-backed output-exec with real credentials.
- Timing observed `timing_observed_count: 10 / >0` Provider execution should record timing metadata.
- Token usage observed `token_observed_count: 10 / >0` Provider execution should return non-estimated token usage.

human · awaiting-submission

### Human Adjudication

No evidence packet has been submitted for review.

Owner
  human reviewer
Ledger
  `pending`
Blocked
  `5`
Queue
  `2`
Submission
  `evidence/world_class/submissions/human-adjudication.json`

#### Phase Queue

- Unblock access `unblock-access: 1 / 1` Assign a real reviewer identity before claiming human adjudication.
- Collect source `collect-source: 5 / 5` Set reviewer_attestation only after choices are completed before opening the answer key.

#### Source Runbook

- python3 scripts/yao.py output-review-kit --write-template
- Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
- python3 scripts/adjudicate_output_review.py --write-template
- Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
- python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication
- python3 scripts/yao.py output-review
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

#### Commands

- prepare submission `python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions`
- validate intake `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
- review queue `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
- refresh ledger `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
- guard claim `python3 scripts/yao.py world-class-claim-guard .`

#### Must Collect

- real reviewer identity
- blind A/B decisions
- answer key unopened until decisions exist

#### Success Checks

- reports/output_review_adjudication.json summary.pending_count == 0
- reports/output_review_adjudication.json summary.judgment_count == summary.pair_count
- reports/output_review_adjudication.json summary.invalid_decision_count == 0
- reports/output_review_adjudication.json summary.reviewer_metadata_present is true
- reports/output_review_adjudication.json summary.blind_review_attested is true
- reports/output_review_adjudication.json review_integrity.blind_pack_sha256 exists and matches reports/output_review_decisions.json
- reports/output_review_adjudication.json pairs and reviewer_checklist store prompt_sha256, not raw prompt text
- reports/output_review_adjudication.json summary.ready_for_human_evidence is true
- reports/skill_os2_audit.json item human-adjudication status becomes pass

#### Privacy

- Reviewer decisions should not include raw user data or private customer detail.
- Reviewer reasons must be rubric-based and must not include raw user data or private customer detail.
- The decision importer rejects raw prompt, output, transcript, message, and answer-key fields.
- The adjudication evidence stores prompt_sha256 instead of raw prompt text.
- The decision and adjudication artifacts preserve blind_pack_sha256 so reviewers can audit exactly which pack was judged.
- Keep the answer key separate until after decisions are recorded.

#### Next Source Actions

- Record a reviewer choice and reason for every pair.
- Every pair needs one valid human judgment.
- Record reviewer and reviewed_at before adjudication can count.
- Set reviewer_attestation only after choices are completed before opening the answer key.
- Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.

#### Source Evidence Snapshot

- Review pairs exist `pair_count: 5 / >0` Generate the blind A/B review pack.
- No pending decisions `pending_count: 5 / ==0` Record a reviewer choice and reason for every pair.
- Judgments complete `judgment_count: 0 / ==pair_count` Every pair needs one valid human judgment.
- No invalid decisions `invalid_decision_count: 0 / ==0` Fix malformed winner/confidence entries.
- Reviewer metadata `reviewer_metadata_present: False / true` Record reviewer and reviewed_at before adjudication can count.
- Reason required `reason_required: True / true` Keep reason mandatory for every imported or direct reviewer decision.
- Blind review attested `blind_review_attested: False / true` Set reviewer_attestation only after choices are completed before opening the answer key.
- Raw content attested `raw_content_excluded_attested: True / true` Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content.
- Raw content blocked `raw_content_allowed: False / false` Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages.
- Human evidence ready `ready_for_human_evidence: False / true` Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.

external · awaiting-submission

### Native Permission Enforcement

No evidence packet has been submitted for review.

Owner
  target client or installer integrator
Ledger
  `pending`
Blocked
  `1`
Queue
  `2`
Submission
  `evidence/world_class/submissions/native-permission-enforcement.json`

#### Phase Queue

- Unblock access `unblock-access: 1 / 1` Attach a real target-client or external installer runtime guard; metadata fallback is not enough.
- Collect source `collect-source: 1 / 1` Collect real target-client or external runtime guard proof.

#### Source Runbook

- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
- Update the generated target adapter only when the guard is actually enforced by that target.
- python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
- python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
- python3 scripts/yao.py runtime-permissions . --package-dir dist
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

#### Commands

- prepare submission `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions`
- validate intake `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
- review queue `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
- refresh ledger `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
- guard claim `python3 scripts/yao.py world-class-claim-guard .`

#### Must Collect

- real target client or external installer runtime guard
- native enforcement flag or externally accepted guard proof
- residual risk retained for fallback targets

#### Success Checks

- reports/runtime_permission_probes.json summary.native_enforcement_count > 0
- reports/runtime_permission_probes.json summary.failure_count == 0
- reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
- reports/skill_os2_audit.json item native-permission-enforcement status becomes pass

#### Privacy

- Do not mark native_enforcement true for metadata-only fallbacks.
- Keep residual risks visible for targets that still rely on operator enforcement.

#### Next Source Actions

- Collect real target-client or external runtime guard proof.

#### Source Evidence Snapshot

- Native enforcement `native_enforcement_count: 0 / >0` Collect real target-client or external runtime guard proof.
- Probe failures `failure_count: 0 / ==0` Runtime permission probes must stay clean.
- Installer support `installer_enforcement_ready: True / true` Installer enforcement is supporting evidence, not native proof.

external · awaiting-submission

### Native Client Telemetry

No evidence packet has been submitted for review.

Owner
  Browser/Chrome/IDE/provider client integrator
Ledger
  `pending`
Blocked
  `1`
Queue
  `2`
Submission
  `evidence/world_class/submissions/native-client-telemetry.json`

#### Phase Queue

- Unblock access `unblock-access: 1 / 1` Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.
- Collect source `collect-source: 1 / 1` Import at least one metadata-only event from a real client.

#### Source Runbook

- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
- Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
- python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
- python3 scripts/yao.py skill-atlas --workspace-root .
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

#### Commands

- prepare submission `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions`
- validate intake `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
- review queue `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
- refresh ledger `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
- guard claim `python3 scripts/yao.py world-class-claim-guard .`

#### Must Collect

- real external client source
- metadata-only event
- local-first import path

#### Success Checks

- reports/adoption_drift_report.json summary.source_types.external > 0
- reports/adoption_drift_report.json summary.adoption_sample_count > 0
- reports/skill_os2_audit.json item native-client-telemetry status becomes pass

#### Privacy

- Telemetry must remain metadata-only and local-first.
- Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.

#### Next Source Actions

- Import at least one metadata-only event from a real client.

#### Source Evidence Snapshot

- External events `external_source_events: 0 / >0` Import at least one metadata-only event from a real client.
- Adoption sample `adoption_sample_count: 1 / >0` Telemetry must include adoption outcome evidence.
- Raw content blocked `raw_content_allowed: False / false` Telemetry must stay metadata-only.

## Release Gate

Decision blocked-until-evidence-accepted
Ready false
Blocked 4/5
Counts false

Run make ci-test in a clean worktree and verify GitHub Actions before converting the PR out of Draft.

Check | Current | Expected | Status | Artifact
--- | --- | --- | --- | ---
World-class ledger ready | `evidence-pending` | `ready_to_claim_world_class == true` | `blocked` | `reports/world_class_evidence_ledger.json`
Claim guard clean | `violations 0; ledger ready False` | `violation_count == 0 and ledger_ready_to_claim_world_class == true` | `blocked` | `reports/world_class_claim_guard.json`
Benchmark public claim ready | `public_claim_ready False` | `public_claim_ready == true` | `blocked` | `reports/benchmark_reproducibility.json`
Review Studio clean | `blockers 0; warnings 3` | `blocker_count == 0 and warning_count == 0` | `blocked` | `reports/review-studio.json`
Evidence consistency clean | `consistent` | `decision == consistent and fail_count == 0` | `pass` | `reports/evidence_consistency.json`

## Boundary

- Planned work, draft packets, metadata fallback, pending human decisions, and local command runners do not count as completion.
- Valid intake means ready for submission review; ledger review still requires passing source evidence.
- The world-class ledger and claim guard remain the source of truth.