World-Class Operator Runbook
A single operating page for collecting the remaining human and external evidence. It coordinates action, but does not accept evidence or change the ledger.
Fast Path
- Run the real external or human work for one evidence item.
- Generate and fill the matching submission draft.
- Validate intake and inspect the submission review queue.
- Refresh the ledger and run the claim guard before making any completion claim.
Evidence Items
Provider Holdout
No evidence packet has been submitted for review.
- Owner
- operator with provider credentials
- Ledger
pending- Submission
evidence/world_class/submissions/provider-holdout.json
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
Must Collect
- provider-backed model run
- observed timing
- observed token metadata
Success Checks
- reports/output_execution_runs.json summary.model_executed_count > 0
- reports/output_execution_runs.json summary.timing_observed_count > 0
- reports/output_execution_runs.json summary.token_observed_count > 0
- reports/skill_os2_audit.json item provider-holdout status becomes pass
Privacy
- Do not commit provider credentials or environment dumps.
- The output execution report records output hashes and aggregate run metadata, not raw provider prompts.
Human Adjudication
No evidence packet has been submitted for review.
- Owner
- human reviewer
- Ledger
pending- Submission
evidence/world_class/submissions/human-adjudication.json
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
Must Collect
- real reviewer identity
- blind A/B decisions
- answer key unopened until decisions exist
Success Checks
- reports/output_review_adjudication.json summary.pending_count == 0
- reports/output_review_adjudication.json summary.judgment_count == summary.pair_count
- reports/output_review_adjudication.json summary.invalid_decision_count == 0
- reports/skill_os2_audit.json item human-adjudication status becomes pass
Privacy
- Reviewer decisions should not include raw user data or private customer detail.
- Keep the answer key separate until after decisions are recorded.
Native Permission Enforcement
No evidence packet has been submitted for review.
- Owner
- target client or installer integrator
- Ledger
pending- Submission
evidence/world_class/submissions/native-permission-enforcement.json
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
Must Collect
- real target client or external installer runtime guard
- native enforcement flag or externally accepted guard proof
- residual risk retained for fallback targets
Success Checks
- reports/runtime_permission_probes.json summary.native_enforcement_count > 0
- reports/runtime_permission_probes.json summary.failure_count == 0
- reports/runtime_permission_probes.json summary.installer_enforcement_pass_count records local installer enforcement but does not replace native evidence
- reports/skill_os2_audit.json item native-permission-enforcement status becomes pass
Privacy
- Do not mark native_enforcement true for metadata-only fallbacks.
- Keep residual risks visible for targets that still rely on operator enforcement.
Native Client Telemetry
No evidence packet has been submitted for review.
- Owner
- Browser/Chrome/IDE/provider client integrator
- Ledger
pending- Submission
evidence/world_class/submissions/native-client-telemetry.json
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
Must Collect
- real external client source
- metadata-only event
- local-first import path
Success Checks
- reports/adoption_drift_report.json summary.source_types.external > 0
- reports/adoption_drift_report.json summary.adoption_sample_count > 0
- reports/skill_os2_audit.json item native-client-telemetry status becomes pass
Privacy
- Telemetry must remain metadata-only and local-first.
- Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
Boundary
- Planned work, draft packets, metadata fallback, pending human decisions, and local command runners do not count as completion.
- Valid intake means ready for submission review; ledger review still requires passing source evidence.
- The world-class ledger and claim guard remain the source of truth.