World-Class Evidence Preflight
This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.
Submission Kit
Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied.
- submissions directory:
evidence/world_class/submissions - drafts count as evidence:
false - preflight accepts evidence:
false
Evidence Queue
Provider Holdout
Prechecks
file
Output eval cases
- Current
present- Status
- pass
- Action
- Keep output holdout cases available before provider execution.
file
Provider runner
- Current
present- Status
- pass
- Action
- Use the provider runner instead of the local command runner for model-backed evidence.
env
Provider credential
- Current
not-set- Status
- missing
- Action
- Set OPENAI_API_KEY in the operator shell; never commit or print the value.
env
Provider model
- Current
not-set- Status
- optional
- Action
- Optionally set YAO_OUTPUT_EVAL_MODEL; the runbook defaults to gpt-4.1-mini.
Source Checks
model_executed_count
Provider model run
- Current
0- Expected
>0- Status
- blocked
- Action
- Run provider-backed output-exec with real credentials.
timing_observed_count
Timing observed
- Current
10- Expected
>0- Status
- pass
- Action
- Provider execution should record timing metadata.
token_observed_count
Token usage observed
- Current
0- Expected
>0- Status
- blocked
- Action
- Provider execution should return non-estimated token usage.
Runbook
- YAO_OUTPUT_EVAL_MODEL=gpt-4.1-mini OPENAI_API_KEY=<redacted> python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Human Adjudication
Prechecks
file
Blind review kit
- Current
present- Status
- pass
- Action
- Open the blind review kit and record real reviewer choices.
file
Decision template
- Current
present- Status
- pass
- Action
- Fill winner_variant values with real A/B decisions.
human
Human reviewer
- Current
external-human-action- Status
- human-required
- Action
- Assign a real reviewer identity before claiming human adjudication.
Source Checks
pair_count
Review pairs exist
- Current
5- Expected
>0- Status
- pass
- Action
- Generate the blind A/B review pack.
pending_count
No pending decisions
- Current
5- Expected
==0- Status
- blocked
- Action
- Record a reviewer choice for every pair.
judgment_count
Judgments complete
- Current
0- Expected
==pair_count- Status
- blocked
- Action
- Every pair needs one valid human judgment.
invalid_decision_count
No invalid decisions
- Current
0- Expected
==0- Status
- pass
- Action
- Fix malformed winner/confidence entries.
Runbook
- python3 scripts/yao.py output-review-kit --write-template
- Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
- python3 scripts/adjudicate_output_review.py --write-template
- Edit reports/output_review_decisions.json with winner_variant values and reviewer metadata.
- python3 scripts/yao.py output-review
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Native Permission Enforcement
Prechecks
file
Permission policy
- Current
present- Status
- pass
- Action
- Keep approved high-permission capabilities explicit.
file
Runtime probes
- Current
present- Status
- pass
- Action
- Refresh runtime permission probes after packaging changes.
external
Native guard
- Current
external-integration-required- Status
- external-required
- Action
- Attach a real target-client or external installer runtime guard; metadata fallback is not enough.
Source Checks
native_enforcement_count
Native enforcement
- Current
0- Expected
>0- Status
- blocked
- Action
- Collect real target-client or external runtime guard proof.
failure_count
Probe failures
- Current
0- Expected
==0- Status
- pass
- Action
- Runtime permission probes must stay clean.
installer_enforcement_ready
Installer support
- Current
True- Expected
true- Status
- pass
- Action
- Installer enforcement is supporting evidence, not native proof.
Runbook
- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
- Update the generated target adapter only when the guard is actually enforced by that target.
- python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
- python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
- python3 scripts/yao.py runtime-permissions . --package-dir dist
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Native Client Telemetry
Prechecks
file
Native telemetry host
- Current
present- Status
- pass
- Action
- Use the native host to receive metadata-only client events.
file
Hook recipes
- Current
present- Status
- pass
- Action
- Refresh telemetry hook recipes before external client installation.
external
External client
- Current
external-integration-required- Status
- external-required
- Action
- Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.
Source Checks
external_source_events
External events
- Current
0- Expected
>0- Status
- blocked
- Action
- Import at least one metadata-only event from a real client.
adoption_sample_count
Adoption sample
- Current
0- Expected
>0- Status
- blocked
- Action
- Telemetry must include adoption outcome evidence.
raw_content_allowed
Raw content blocked
- Current
False- Expected
false- Status
- pass
- Action
- Telemetry must stay metadata-only.
Runbook
- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
- Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
- python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
- python3 scripts/yao.py skill-atlas --workspace-root .
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Safety Boundary
- Environment variables are displayed only as set or not-set; secret values are never printed.
- Human-required and external-required states are operator work, not accepted evidence.
- The world-class ledger remains the only source of truth for ready_to_claim_world_class.