Evidence Collection

World-Class Evidence Preflight

This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.

Decisioncollection-preflight-blocked

Pending4

Ready1

Blocked3

Source12/19

Repairs11/11

Phases2/2

Submission Kit

Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.

submissions directory: evidence/world_class/submissions
drafts count as evidence: false
artifact prefill counts as evidence: false
preflight accepts evidence: false

Submission refs

submission-ref

7/7 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

31/31 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

submission-ref rows are the paths expected in artifact_refs; supporting-evidence rows stay available for audit context.

Phase Queue

Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later source collection. Phase queue rows remain procedural and never count as accepted evidence.

#20 · blocked

Unblock access

Phase: unblock-access
Rows: 4 / 4 blocked
Owners: Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator
Evidence: human-adjudication, native-client-telemetry, native-permission-enforcement, provider-holdout
Next: Assign a real reviewer identity before claiming human adjudication.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Counts: does not count as completion

#40 · blocked

Collect source

Phase: collect-source
Rows: 7 / 7 blocked
Owners: Browser/Chrome/IDE/provider client integrator, human reviewer, target client or installer integrator
Evidence: human-adjudication, native-client-telemetry, native-permission-enforcement
Next: Set reviewer_attestation only after choices are completed before opening the answer key.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Counts: does not count as completion

Evidence Queue

external · blocked

Provider Holdout

Evidence: provider-holdout
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/provider-holdout.json

Next Action

Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

1/1 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Output eval cases

Current: present
Status: pass
Action: Keep output holdout cases available before provider execution.

file Provider runner

Current: present
Status: pass
Action: Use the provider runner instead of the local command runner for model-backed evidence.

env_any Provider credential

Current: not-set
Status: missing
Action: Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.

env Provider model

Current: not-set
Status: optional
Action: Optionally set YAO_OUTPUT_EVAL_MODEL, or pass --provider-model for the selected provider.

Source Checks

model_executed_count Provider model run

Current: 10
Expected: >0
Status: pass
Action: Run provider-backed output-exec with real credentials.

timing_observed_count Timing observed

Current: 10
Expected: >0
Status: pass
Action: Provider execution should record timing metadata.

token_observed_count Token usage observed

Current: 10
Expected: >0
Status: pass
Action: Provider execution should return non-estimated token usage.

Repair Rows

precheck provider-api-key

Priority: 20
Phase: unblock-access
Owner: operator with provider credentials
Status: blocked
Reason: Required env_any precheck is missing.
Action: Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

Runbook

Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60
For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

human · ready-for-human-review

Human Adjudication

Evidence: human-adjudication
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/human-adjudication.json

Next Action

Assign a real reviewer identity before claiming human adjudication.

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

8/8 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Blind review kit

Current: present
Status: pass
Action: Open the blind review kit and record real reviewer choices with required rationale.

file Decision template

Current: present
Status: pass
Action: Import real A/B decisions with reviewer, reviewed_at, winner_variant, confidence, reason, and truthful blind-review attestation via `python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication`.

file Decision importer

Current: present
Status: pass
Action: Use the importer to reject raw content fields and normalize reviewer decisions before adjudication.

human Human reviewer

Current: external-human-action
Status: human-required
Action: Assign a real reviewer identity before claiming human adjudication.

Source Checks

pair_count Review pairs exist

Current: 5
Expected: >0
Status: pass
Action: Generate the blind A/B review pack.

pending_count No pending decisions

Current: 5
Expected: ==0
Status: blocked
Action: Record a reviewer choice and reason for every pair.

judgment_count Judgments complete

Current: 0
Expected: ==pair_count
Status: blocked
Action: Every pair needs one valid human judgment.

invalid_decision_count No invalid decisions

Current: 0
Expected: ==0
Status: pass
Action: Fix malformed winner/confidence entries.

reviewer_metadata_present Reviewer metadata

Current: False
Expected: true
Status: blocked
Action: Record reviewer and reviewed_at before adjudication can count.

reason_required Reason required

Current: True
Expected: true
Status: pass
Action: Keep reason mandatory for every imported or direct reviewer decision.

blind_review_attested Blind review attested

Current: False
Expected: true
Status: blocked
Action: Set reviewer_attestation only after choices are completed before opening the answer key.

raw_content_excluded_attested Raw content attested

Current: True
Expected: true
Status: pass
Action: Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content.

raw_content_allowed Raw content blocked

Current: False
Expected: false
Status: pass
Action: Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages.

ready_for_human_evidence Human evidence ready

Current: False
Expected: true
Status: blocked
Action: Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.

Repair Rows

precheck human-reviewer

Priority: 20
Phase: unblock-access
Owner: human reviewer
Status: blocked
Reason: Required human precheck is human-required.
Action: Assign a real reviewer identity before claiming human adjudication.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check blind_review_attested

Priority: 40
Phase: collect-source
Owner: human reviewer
Status: blocked
Reason: Current value False does not satisfy 'true'.
Action: Set reviewer_attestation only after choices are completed before opening the answer key.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check judgment_count

Priority: 40
Phase: collect-source
Owner: human reviewer
Status: blocked
Reason: Current value 0 does not satisfy '==pair_count'.
Action: Every pair needs one valid human judgment.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check pending_count

Priority: 40
Phase: collect-source
Owner: human reviewer
Status: blocked
Reason: Current value 5 does not satisfy '==0'.
Action: Record a reviewer choice and reason for every pair.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check ready_for_human_evidence

Priority: 40
Phase: collect-source
Owner: human reviewer
Status: blocked
Reason: Current value False does not satisfy 'true'.
Action: Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check reviewer_metadata_present

Priority: 40
Phase: collect-source
Owner: human reviewer
Status: blocked
Reason: Current value False does not satisfy 'true'.
Action: Record reviewer and reviewed_at before adjudication can count.
Verify: python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

Runbook

python3 scripts/yao.py output-review-kit --write-template
Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
python3 scripts/adjudicate_output_review.py --write-template
Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication
python3 scripts/yao.py output-review
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

Native Permission Enforcement

Evidence: native-permission-enforcement
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-permission-enforcement.json

Next Action

Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

11/11 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Permission policy

Current: present
Status: pass
Action: Keep approved high-permission capabilities explicit.

file Runtime probes

Current: present
Status: pass
Action: Refresh runtime permission probes after packaging changes.

external Native guard

Current: external-integration-required
Status: external-required
Action: Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

Source Checks

native_enforcement_count Native enforcement

Current: 0
Expected: >0
Status: blocked
Action: Collect real target-client or external runtime guard proof.

failure_count Probe failures

Current: 0
Expected: ==0
Status: pass
Action: Runtime permission probes must stay clean.

installer_enforcement_ready Installer support

Current: True
Expected: true
Status: pass
Action: Installer enforcement is supporting evidence, not native proof.

Repair Rows

precheck native-guard

Priority: 20
Phase: unblock-access
Owner: target client or installer integrator
Status: blocked
Reason: Required external precheck is external-required.
Action: Attach a real target-client or external installer runtime guard; metadata fallback is not enough.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check native_enforcement_count

Priority: 40
Phase: collect-source
Owner: target client or installer integrator
Status: blocked
Reason: Current value 0 does not satisfy '>0'.
Action: Collect real target-client or external runtime guard proof.
Verify: python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

Runbook

Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
Update the generated target adapter only when the guard is actually enforced by that target.
python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
python3 scripts/yao.py runtime-permissions . --package-dir dist
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

Native Client Telemetry

Evidence: native-client-telemetry
Ledger: pending
Intake: awaiting-submission
Review: awaiting-submission
Draft: evidence/world_class/submissions/native-client-telemetry.json

Next Action

Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions

python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts

Artifact Roles

Submission refs

submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: true

Supporting evidence

supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: false

Prechecks

file Native telemetry host

Current: present
Status: pass
Action: Use the native host to receive metadata-only client events.

file Hook recipes

Current: present
Status: pass
Action: Refresh telemetry hook recipes before external client installation.

external External client

Current: external-integration-required
Status: external-required
Action: Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

Source Checks

external_source_events External events

Current: 0
Expected: >0
Status: blocked
Action: Import at least one metadata-only event from a real client.

adoption_sample_count Adoption sample

Current: 1
Expected: >0
Status: pass
Action: Telemetry must include adoption outcome evidence.

raw_content_allowed Raw content blocked

Current: False
Expected: false
Status: pass
Action: Telemetry must stay metadata-only.

Repair Rows

precheck external-client

Priority: 20
Phase: unblock-access
Owner: Browser/Chrome/IDE/provider client integrator
Status: blocked
Reason: Required external precheck is external-required.
Action: Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.
Verify: python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

source-check external_source_events

Priority: 40
Phase: collect-source
Owner: Browser/Chrome/IDE/provider client integrator
Status: blocked
Reason: Current value 0 does not satisfy '>0'.
Action: Import at least one metadata-only event from a real client.
Verify: python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions
Evidence: does not count as completion

Runbook

python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
python3 scripts/yao.py skill-atlas --workspace-root .
python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

Safety Boundary

Environment variables are displayed only as set or not-set; secret values are never printed.
Human-required and external-required states are operator work, not accepted evidence.
The world-class ledger remains the only source of truth for ready_to_claim_world_class.