Evidence Collection

# World-Class Evidence Preflight

This operator view shows which external and human evidence is still blocked, which commands prepare editable submission drafts, and why preflight never counts as accepted evidence.

Decision collection-preflight-blocked
Pending 4
Ready 1
Blocked 3
Source 12/19
Repairs 11/11
Phases 2/2

## Submission Kit

Generate drafts only after real provider, human-review, native-permission, or native-client work exists. Drafts remain non-evidence until valid aggregate artifact refs and SHA-256 digests are supplied. Artifact prefill is convenience data only.

- submissions directory: `evidence/world_class/submissions`
- drafts count as evidence: `false`
- artifact prefill counts as evidence: `false`
- preflight accepts evidence: `false`

Submission refs

### submission-ref

7/7 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: `true`
Supporting evidence

### supporting-evidence

31/31 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: `false`

`submission-ref` rows are the paths expected in `artifact_refs`; `supporting-evidence` rows stay available for audit context.

## Commands

- prepare submission `python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions`
- prepare prefilled submission `python3 scripts/yao.py world-class-submission-kit . --output-dir evidence/world_class/submissions --prefill-artifacts`
- validate intake `python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions`
- submission review `python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions`
- refresh ledger `python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions`
- guard claim `python3 scripts/yao.py world-class-claim-guard .`

## Phase Queue

Repair rows are grouped into execution phases so operators can clear the first blocker before moving to later source collection. Phase queue rows remain procedural and never count as accepted evidence.

#20 · blocked

### Unblock access

Phase
  `unblock-access`
Rows
  4 / 4 blocked
Owners
  Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator
Evidence
  human-adjudication, native-client-telemetry, native-permission-enforcement, provider-holdout
Next
  Assign a real reviewer identity before claiming human adjudication.
Verify
  `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Counts
  does not count as completion

#40 · blocked

### Collect source

Phase
  `collect-source`
Rows
  7 / 7 blocked
Owners
  Browser/Chrome/IDE/provider client integrator, human reviewer, target client or installer integrator
Evidence
  human-adjudication, native-client-telemetry, native-permission-enforcement
Next
  Set reviewer_attestation only after choices are completed before opening the answer key.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Counts
  does not count as completion

## Evidence Queue

external · blocked

### Provider Holdout

Evidence
  `provider-holdout`
Ledger
  pending
Intake
  awaiting-submission
Review
  awaiting-submission
Draft
  `evidence/world_class/submissions/provider-holdout.json`

#### Next Action

Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.

`python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions` `python3 scripts/yao.py world-class-submission-kit . --evidence-key provider-holdout --output-dir evidence/world_class/submissions --prefill-artifacts`

#### Artifact Roles

Submission refs

### submission-ref

1/1 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: `true`
Supporting evidence

### supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: `false`

#### Prechecks

file Output eval cases

Current
  `present`
Status
  pass
Action
  Keep output holdout cases available before provider execution.

file Provider runner

Current
  `present`
Status
  pass
Action
  Use the provider runner instead of the local command runner for model-backed evidence.

env_any Provider credential

Current
  `not-set`
Status
  missing
Action
  Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.

env Provider model

Current
  `not-set`
Status
  optional
Action
  Optionally set YAO_OUTPUT_EVAL_MODEL, or pass --provider-model for the selected provider.

#### Source Checks

model_executed_count Provider model run

Current
  `10`
Expected
  `>0`
Status
  pass
Action
  Run provider-backed output-exec with real credentials.

timing_observed_count Timing observed

Current
  `10`
Expected
  `>0`
Status
  pass
Action
  Provider execution should record timing metadata.

token_observed_count Token usage observed

Current
  `10`
Expected
  `>0`
Status
  pass
Action
  Provider execution should return non-estimated token usage.

#### Repair Rows

precheck provider-api-key

Priority
  20
Phase
  unblock-access
Owner
  operator with provider credentials
Status
  blocked
Reason
  Required env_any precheck is missing.
Action
  Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
Verify
  `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

#### Runbook

- Set one provider API key in the operator shell, such as OPENAI_API_KEY or DEEPSEEK_API_KEY; never commit or print the value.
- For OpenAI Responses: python3 scripts/yao.py output-exec --provider-runner openai --provider-model ${YAO_OUTPUT_EVAL_MODEL:-gpt-4.1-mini} --timeout-seconds 60
- For DeepSeek Chat Completions: python3 scripts/yao.py output-exec --provider-runner deepseek --provider-model deepseek-v4-flash --provider-api-format chat-completions --provider-thinking disabled --api-key-env DEEPSEEK_API_KEY --timeout-seconds 120
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/provider-holdout.intake.json to evidence/world_class/submissions/provider-holdout.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

human · ready-for-human-review

### Human Adjudication

Evidence
  `human-adjudication`
Ledger
  pending
Intake
  awaiting-submission
Review
  awaiting-submission
Draft
  `evidence/world_class/submissions/human-adjudication.json`

#### Next Action

Assign a real reviewer identity before claiming human adjudication.

`python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions` `python3 scripts/yao.py world-class-submission-kit . --evidence-key human-adjudication --output-dir evidence/world_class/submissions --prefill-artifacts`

#### Artifact Roles

Submission refs

### submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: `true`
Supporting evidence

### supporting-evidence

8/8 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: `false`

#### Prechecks

file Blind review kit

Current
  `present`
Status
  pass
Action
  Open the blind review kit and record real reviewer choices with required rationale.

file Decision template

Current
  `present`
Status
  pass
Action
  Import real A/B decisions with reviewer, reviewed_at, winner_variant, confidence, reason, and truthful blind-review attestation via `python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication`.

file Decision importer

Current
  `present`
Status
  pass
Action
  Use the importer to reject raw content fields and normalize reviewer decisions before adjudication.

human Human reviewer

Current
  `external-human-action`
Status
  human-required
Action
  Assign a real reviewer identity before claiming human adjudication.

#### Source Checks

pair_count Review pairs exist

Current
  `5`
Expected
  `>0`
Status
  pass
Action
  Generate the blind A/B review pack.

pending_count No pending decisions

Current
  `5`
Expected
  `==0`
Status
  blocked
Action
  Record a reviewer choice and reason for every pair.

judgment_count Judgments complete

Current
  `0`
Expected
  `==pair_count`
Status
  blocked
Action
  Every pair needs one valid human judgment.

invalid_decision_count No invalid decisions

Current
  `0`
Expected
  `==0`
Status
  pass
Action
  Fix malformed winner/confidence entries.

reviewer_metadata_present Reviewer metadata

Current
  `False`
Expected
  `true`
Status
  blocked
Action
  Record reviewer and reviewed_at before adjudication can count.

reason_required Reason required

Current
  `True`
Expected
  `true`
Status
  pass
Action
  Keep reason mandatory for every imported or direct reviewer decision.

blind_review_attested Blind review attested

Current
  `False`
Expected
  `true`
Status
  blocked
Action
  Set reviewer_attestation only after choices are completed before opening the answer key.

raw_content_excluded_attested Raw content attested

Current
  `True`
Expected
  `true`
Status
  pass
Action
  Attest that reviewer decisions exclude raw prompts, outputs, transcripts, messages, and private user content.

raw_content_allowed Raw content blocked

Current
  `False`
Expected
  `false`
Status
  pass
Action
  Adjudication evidence should store prompt hashes and reviewer metadata, not raw prompts, outputs, transcripts, or messages.

ready_for_human_evidence Human evidence ready

Current
  `False`
Expected
  `true`
Status
  blocked
Action
  Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.

#### Repair Rows

precheck human-reviewer

Priority
  20
Phase
  unblock-access
Owner
  human reviewer
Status
  blocked
Reason
  Required human precheck is human-required.
Action
  Assign a real reviewer identity before claiming human adjudication.
Verify
  `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check blind_review_attested

Priority
  40
Phase
  collect-source
Owner
  human reviewer
Status
  blocked
Reason
  Current value False does not satisfy 'true'.
Action
  Set reviewer_attestation only after choices are completed before opening the answer key.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check judgment_count

Priority
  40
Phase
  collect-source
Owner
  human reviewer
Status
  blocked
Reason
  Current value 0 does not satisfy '==pair_count'.
Action
  Every pair needs one valid human judgment.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check pending_count

Priority
  40
Phase
  collect-source
Owner
  human reviewer
Status
  blocked
Reason
  Current value 5 does not satisfy '==0'.
Action
  Record a reviewer choice and reason for every pair.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check ready_for_human_evidence

Priority
  40
Phase
  collect-source
Owner
  human reviewer
Status
  blocked
Reason
  Current value False does not satisfy 'true'.
Action
  Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check reviewer_metadata_present

Priority
  40
Phase
  collect-source
Owner
  human reviewer
Status
  blocked
Reason
  Current value False does not satisfy 'true'.
Action
  Record reviewer and reviewed_at before adjudication can count.
Verify
  `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

#### Runbook

- python3 scripts/yao.py output-review-kit --write-template
- Open reports/output_review_kit.md and choose A or B for each pair without opening the answer key.
- python3 scripts/adjudicate_output_review.py --write-template
- Record reviewer choices in a separate JSON, JSONL, or CSV decision source with reviewer, reviewed_at, case_id, winner_variant, confidence, required reason, and truthful reviewer_attestation only.
- python3 scripts/yao.py output-review-import --input <reviewer-decisions.json> --blind-review-attested --run-adjudication
- python3 scripts/yao.py output-review
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/human-adjudication.intake.json to evidence/world_class/submissions/human-adjudication.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

### Native Permission Enforcement

Evidence
  `native-permission-enforcement`
Ledger
  pending
Intake
  awaiting-submission
Review
  awaiting-submission
Draft
  `evidence/world_class/submissions/native-permission-enforcement.json`

#### Next Action

Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

`python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions` `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions --prefill-artifacts`

#### Artifact Roles

Submission refs

### submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: `true`
Supporting evidence

### supporting-evidence

11/11 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: `false`

#### Prechecks

file Permission policy

Current
  `present`
Status
  pass
Action
  Keep approved high-permission capabilities explicit.

file Runtime probes

Current
  `present`
Status
  pass
Action
  Refresh runtime permission probes after packaging changes.

external Native guard

Current
  `external-integration-required`
Status
  external-required
Action
  Attach a real target-client or external installer runtime guard; metadata fallback is not enough.

#### Source Checks

native_enforcement_count Native enforcement

Current
  `0`
Expected
  `>0`
Status
  blocked
Action
  Collect real target-client or external runtime guard proof.

failure_count Probe failures

Current
  `0`
Expected
  `==0`
Status
  pass
Action
  Runtime permission probes must stay clean.

installer_enforcement_ready Installer support

Current
  `True`
Expected
  `true`
Status
  pass
Action
  Installer enforcement is supporting evidence, not native proof.

#### Repair Rows

precheck native-guard

Priority
  20
Phase
  unblock-access
Owner
  target client or installer integrator
Status
  blocked
Reason
  Required external precheck is external-required.
Action
  Attach a real target-client or external installer runtime guard; metadata fallback is not enough.
Verify
  `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check native_enforcement_count

Priority
  40
Phase
  collect-source
Owner
  target client or installer integrator
Status
  blocked
Reason
  Current value 0 does not satisfy '>0'.
Action
  Collect real target-client or external runtime guard proof.
Verify
  `python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

#### Runbook

- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
- Update the generated target adapter only when the guard is actually enforced by that target.
- python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
- python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
- python3 scripts/yao.py runtime-permissions . --package-dir dist
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

external · blocked

### Native Client Telemetry

Evidence
  `native-client-telemetry`
Ledger
  pending
Intake
  awaiting-submission
Review
  awaiting-submission
Draft
  `evidence/world_class/submissions/native-client-telemetry.json`

#### Next Action

Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

`python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions` `python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions --prefill-artifacts`

#### Artifact Roles

Submission refs

### submission-ref

2/2 ready

Rows marked submission-ref are the aggregate paths expected in artifact_refs.

copy to artifact_refs: `true`
Supporting evidence

### supporting-evidence

6/6 ready

Supporting-evidence rows help reviewers audit the packet but do not all need to be copied into artifact_refs.

copy to artifact_refs: `false`

#### Prechecks

file Native telemetry host

Current
  `present`
Status
  pass
Action
  Use the native host to receive metadata-only client events.

file Hook recipes

Current
  `present`
Status
  pass
Action
  Refresh telemetry hook recipes before external client installation.

external External client

Current
  `external-integration-required`
Status
  external-required
Action
  Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.

#### Source Checks

external_source_events External events

Current
  `0`
Expected
  `>0`
Status
  blocked
Action
  Import at least one metadata-only event from a real client.

adoption_sample_count Adoption sample

Current
  `1`
Expected
  `>0`
Status
  pass
Action
  Telemetry must include adoption outcome evidence.

raw_content_allowed Raw content blocked

Current
  `False`
Expected
  `false`
Status
  pass
Action
  Telemetry must stay metadata-only.

#### Repair Rows

precheck external-client

Priority
  20
Phase
  unblock-access
Owner
  Browser/Chrome/IDE/provider client integrator
Status
  blocked
Reason
  Required external precheck is external-required.
Action
  Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.
Verify
  `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

source-check external_source_events

Priority
  40
Phase
  collect-source
Owner
  Browser/Chrome/IDE/provider client integrator
Status
  blocked
Reason
  Current value 0 does not satisfy '>0'.
Action
  Import at least one metadata-only event from a real client.
Verify
  `python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions`
Evidence
  does not count as completion

#### Runbook

- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
- Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
- python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
- python3 scripts/yao.py skill-atlas --workspace-root .
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions

## Safety Boundary

- Environment variables are displayed only as set or not-set; secret values are never printed.
- Human-required and external-required states are operator work, not accepted evidence.
- The world-class ledger remains the only source of truth for ready_to_claim_world_class.