Evidence Collection

World-Class Evidence Preflight

This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.

Decisioncollection-preflight-blocked

Pending4

Ready1

Blocked3

Source6/13

Submission Kit

Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.

submissions directory: evidence/world_class/submissions
drafts count as evidence: false
artifact prefill counts as evidence: false
preflight accepts evidence: false

Submission refs

submission-ref

7/7 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

31/31 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

submission-ref rows are the paths expected in artifact_refs; supporting-evidence rows stay available for audit context.

Evidence Queue

external · blocked

Provider Holdout

Evidence: provider-holdout
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/provider-holdout.json

Next Action

Set OPENAI_API_KEY in the operator shell; never commit or print the value.

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

1/1 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Output eval cases

Current: present
Status: pass
Action: Keep output holdout cases available before provider execution.

file Provider runner

Current: present
Status: pass
Action: Use the provider runner instead of the local command runner for model-backed evidence.

env Provider credential

Current: not-set
Status: missing
Action: Set OPENAI_API_KEY in the operator shell; never commit or print the value.

env Provider model

Current: not-set
Status: optional
Action: Optionally set YAO_OUTPUT_EVAL_MODEL; the runbook defaults to gpt-4.1-mini.

Source Checks

model_executed_count Provider model run

Current: 0
Expected: >0
Status: blocked
Action: Run provider-backed output-exec with real credentials.

timing_observed_count Timing observed

Current: 10
Expected: >0
Status: pass
Action: Provider execution should record timing metadata.

token_observed_count Token usage observed

Current: 0
Expected: >0
Status: blocked
Action: Provider execution should return non-estimated token usage.

Runbook

Set OPENAI_API_KEY in the operator shell before running provider evidence; never commit or print the value.
export YAO_OUTPUT_EVAL_MODEL=${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini}
python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

human · ready-for-human-review

Human Adjudication

Evidence: human-adjudication
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/human-adjudication.json

Next Action

Assign a real reviewer identity before claiming human adjudication.

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

8/8 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Blind review kit

Current: present
Status: pass
Action: Open the blind review kit and record real reviewer choices.

file Decision template

Current: present
Status: pass
Action: Import real A/B decisions with `python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --run-adjudication`.

file Decision importer

Current: present
Status: pass
Action: Use the importer to reject raw content fields and normalize reviewer decisions before adjudication.

human Human reviewer

Current: external-human-action
Status: human-required
Action: Assign a real reviewer identity before claiming human adjudication.

Source Checks

pair_count Review pairs exist

Current: 5
Expected: >0
Status: pass
Action: Generate the blind A/B review pack.

pending_count No pending decisions

Current: 5
Expected: ==0
Status: blocked
Action: Record a reviewer choice for every pair.

judgment_count Judgments complete

Current: 0
Expected: ==pair_count
Status: blocked
Action: Every pair needs one valid human judgment.

invalid_decision_count No invalid decisions

Current: 0
Expected: ==0
Status: pass
Action: Fix malformed winner/confidence entries.

Runbook

python3 scripts/yao.py output-review-kit --write-template
Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
python3 scripts/adjudicate_output_review.py --write-template
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with case_id, winner_variant, confidence, and reason only.
python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --run-adjudication
python3 scripts/yao.py output-review
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

Native Permission Enforcement

Evidence: native-permission-enforcement
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-permission-enforcement.json

Next Action

Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

11/11 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Permission policy

Current: present
Status: pass
Action: Keep approved high-permission capabilities explicit.

file Runtime probes

Current: present
Status: pass
Action: Refresh runtime permission probes after packaging changes.

external Native guard

Current: external-integration-required
Status: external-required
Action: Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

Source Checks

native_enforcement_count Native enforcement

Current: 0
Expected: >0
Status: blocked
Action: Collect real target-client or external runtime guard proof.

failure_count Probe failures

Current: 0
Expected: ==0
Status: pass
Action: Runtime permission probes must stay clean.

installer_enforcement_ready Installer support

Current: True
Expected: true
Status: pass
Action: Installer enforcement is supporting evidence, not native proof.

Runbook

Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
Update the generated target adapter only when the guard is actually enforced by that target.
python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
python3 scripts/yao.py runtime-permissions . --package-dir dist
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

Native Client Telemetry

Evidence: native-client-telemetry
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-client-telemetry.json

Next Action

Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Native telemetry host

Current: present
Status: pass
Action: Use the native host to receive metadata-only client events.

file Hook recipes

Current: present
Status: pass
Action: Refresh telemetry hook recipes before external client installation.

external External client

Current: external-integration-required
Status: external-required
Action: Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

Source Checks

external_source_events External events

Current: 0
Expected: >0
Status: blocked
Action: Import at least one metadata-only event from a real client.

adoption_sample_count Adoption sample

Current: 0
Expected: >0
Status: blocked
Action: Telemetry must include adoption outcome evidence.

raw_content_allowed Raw content blocked

Current: False
Expected: false
Status: pass
Action: Telemetry must stay metadata-only.

Runbook

python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
python3 scripts/yao.py skill-atlas --workspace-root .
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

Safety Boundary

Environment variables are displayed only as set or not-set; secret values are never printed.
Human-required and external-required states are operator work, not accepted evidence.
The world-class ledger remains the only source of truth for ready_to_claim_world_class.