@@ -103,8 +108,10 @@
Ownertarget client or installer integrator
Ledgerpending
Blocked1
+
Queue2
Submissionevidence/world_class/submissions/native-permission-enforcement.json
+
Phase Queue
- Unblock access
unblock-access: 1 / 1Attach a real target-client or external installer runtime guard; metadata fallback is not enough. - Collect source
collect-source: 1 / 1Collect real target-client or external runtime guard proof.
Source Runbook
- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
- Update the generated target adapter only when the guard is actually enforced by that target.
- python3 scripts/yao.py package . --platform openai --platform claude --platform generic --platform vscode --output-dir dist --zip
- python3 scripts/yao.py install-simulate . --package-dir dist --install-root dist/install-simulation
- python3 scripts/yao.py runtime-permissions . --package-dir dist
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-permission-enforcement.intake.json to evidence/world_class/submissions/native-permission-enforcement.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-permission-enforcement --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
@@ -120,9 +127,11 @@
- Owner
- Browser/Chrome/IDE/provider client integrator
- Ledger
pending- Blocked
2- Blocked
1- Queue
2- Submission
evidence/world_class/submissions/native-client-telemetry.json
+
Phase Queue
- Unblock access
unblock-access: 1 / 1Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. - Collect source
collect-source: 1 / 1Import at least one metadata-only event from a real client.
Source Runbook
- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://<extension-id>/
- Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.
- python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl
- python3 scripts/yao.py skill-atlas --workspace-root .
- python3 scripts/yao.py skill-os2-audit . --generated-at <YYYY-MM-DD>
- Copy evidence/world_class/templates/native-client-telemetry.intake.json to evidence/world_class/submissions/native-client-telemetry.json and fill only real evidence fields.
- python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions
Commands
- prepare submission
python3 scripts/yao.py world-class-submission-kit . --evidence-key native-client-telemetry --output-dir evidence/world_class/submissions - validate intake
python3 scripts/yao.py world-class-intake . --submissions-dir evidence/world_class/submissions - review queue
python3 scripts/yao.py world-class-submission-review . --submissions-dir evidence/world_class/submissions - refresh ledger
python3 scripts/yao.py world-class-ledger . --submissions-dir evidence/world_class/submissions - guard claim
python3 scripts/yao.py world-class-claim-guard .
@@ -130,8 +139,8 @@
Success Checks
- reports/adoption_drift_report.json summary.source_types.external > 0
- reports/adoption_drift_report.json summary.adoption_sample_count > 0
- reports/skill_os2_audit.json item native-client-telemetry status becomes pass
Privacy
- Telemetry must remain metadata-only and local-first.
- Do not package reports/telemetry_events.jsonl or any raw prompt, output, transcript, note, or message field.
-
Next Source Actions
- Import at least one metadata-only event from a real client.
- Telemetry must include adoption outcome evidence.
-
Source Evidence Snapshot
- External events
external_source_events: 0 / >0Import at least one metadata-only event from a real client. - Adoption sample
adoption_sample_count: 0 / >0Telemetry must include adoption outcome evidence. - Raw content blocked
raw_content_allowed: False / falseTelemetry must stay metadata-only.
+
Next Source Actions
- Import at least one metadata-only event from a real client.
+
Source Evidence Snapshot
- External events
external_source_events: 0 / >0Import at least one metadata-only event from a real client. - Adoption sample
adoption_sample_count: 1 / >0Telemetry must include adoption outcome evidence. - Raw content blocked
raw_content_allowed: False / falseTelemetry must stay metadata-only.
Boundary
- Planned work, draft packets, metadata fallback, pending human decisions, and local command runners do not count as completion.
- Valid intake means ready for submission review; ledger review still requires passing source evidence.
- The world-class ledger and claim guard remain the source of truth.
diff --git a/reports/world_class_operator_runbook.json b/reports/world_class_operator_runbook.json
index 0c82d92b..c29707f6 100644
--- a/reports/world_class_operator_runbook.json
+++ b/reports/world_class_operator_runbook.json
@@ -12,8 +12,17 @@
"valid_packet_source_incomplete_count": 0,
"invalid_submission_count": 0,
"source_check_count": 19,
- "source_pass_count": 9,
- "source_blocked_count": 10,
+ "source_pass_count": 10,
+ "source_blocked_count": 9,
+ "repair_checklist_count": 13,
+ "repair_blocked_count": 13,
+ "phase_queue_count": 2,
+ "phase_queue_blocked_count": 2,
+ "phase_queue_row_count": 13,
+ "phase_queue_next_phase": "unblock-access",
+ "phase_queue_next_action_id": "human-adjudication-precheck-human-reviewer",
+ "phase_queue_next_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "phase_queue_counts_as_completion": false,
"ready_to_claim_world_class": false,
"runbook_counts_as_completion": false,
"decision": "collect-evidence"
@@ -121,6 +130,144 @@
"Run provider-backed output-exec with real credentials.",
"Provider execution should return non-estimated token usage."
],
+ "repair_checklist": [
+ {
+ "action_id": "provider-holdout-precheck-openai-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "openai-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env precheck is missing.",
+ "next_action": "Set OPENAI_API_KEY in the operator shell; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-model_executed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "model_executed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Run provider-backed output-exec with real credentials.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-token_observed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "token_observed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Provider execution should return non-estimated token usage.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "repair_blocked_count": 3,
+ "repair_counts_as_completion": false,
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-precheck-openai-api-key",
+ "next_action": "Set OPENAI_API_KEY in the operator shell; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "provider-holdout-precheck-openai-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "openai-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env precheck is missing.",
+ "next_action": "Set OPENAI_API_KEY in the operator shell; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 2,
+ "row_count": 2,
+ "owners": [
+ "operator with provider credentials"
+ ],
+ "evidence_keys": [
+ "provider-holdout"
+ ],
+ "next_action_id": "provider-holdout-source-check-model_executed_count",
+ "next_action": "Run provider-backed output-exec with real credentials.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "provider-holdout-source-check-model_executed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "model_executed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Run provider-backed output-exec with real credentials.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-token_observed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "token_observed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Provider execution should return non-estimated token usage.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
+ "phase_queue_blocked_count": 2,
+ "phase_queue_counts_as_completion": false,
"submission_state": {
"status": "missing",
"path": "evidence/world_class/submissions/provider-holdout.json",
@@ -335,6 +482,234 @@
"Set reviewer_attestation only after choices are completed before opening the answer key.",
"Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints."
],
+ "repair_checklist": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "repair_blocked_count": 6,
+ "repair_counts_as_completion": false,
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 5,
+ "row_count": 5,
+ "owners": [
+ "human reviewer"
+ ],
+ "evidence_keys": [
+ "human-adjudication"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
+ "phase_queue_blocked_count": 2,
+ "phase_queue_counts_as_completion": false,
"submission_state": {
"status": "missing",
"path": "evidence/world_class/submissions/human-adjudication.json",
@@ -459,6 +834,114 @@
"next_source_actions": [
"Collect real target-client or external runtime guard proof."
],
+ "repair_checklist": [
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "repair_blocked_count": 2,
+ "repair_counts_as_completion": false,
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-precheck-native-guard",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "native-permission-enforcement"
+ ],
+ "next_action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
+ "phase_queue_blocked_count": 2,
+ "phase_queue_counts_as_completion": false,
"submission_state": {
"status": "missing",
"path": "evidence/world_class/submissions/native-permission-enforcement.json",
@@ -487,7 +970,7 @@
"review_state": "awaiting-submission",
"source_accepted": false,
"objective": "Import production metadata-only events from a real external client into the local drift loop.",
- "current": "external source events 0; adoption samples 0",
+ "current": "external source events 0; adoption samples 1",
"execution_runbook": [
"python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension://
/",
"Install the generated native messaging manifest for the real client and send at least one accepted skill_activation or skill_output event.",
@@ -535,7 +1018,7 @@
],
"observed_state": {
"external_source_events": 0,
- "adoption_sample_count": 0,
+ "adoption_sample_count": 1,
"raw_content_allowed": false,
"risk_band": "low",
"accepted": false
@@ -556,8 +1039,8 @@
"label": "Adoption sample",
"field": "adoption_sample_count",
"expected": ">0",
- "actual": 0,
- "status": "blocked",
+ "actual": 1,
+ "status": "pass",
"source_accepted": false,
"next_action": "Telemetry must include adoption outcome evidence."
},
@@ -572,11 +1055,118 @@
"next_action": "Telemetry must stay metadata-only."
}
],
- "blocked_source_check_count": 2,
+ "blocked_source_check_count": 1,
"next_source_actions": [
- "Import at least one metadata-only event from a real client.",
- "Telemetry must include adoption outcome evidence."
+ "Import at least one metadata-only event from a real client."
],
+ "repair_checklist": [
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "repair_blocked_count": 2,
+ "repair_counts_as_completion": false,
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-precheck-external-client",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 1,
+ "row_count": 1,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator"
+ ],
+ "evidence_keys": [
+ "native-client-telemetry"
+ ],
+ "next_action_id": "native-client-telemetry-source-check-external_source_events",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
+ "phase_queue_blocked_count": 2,
+ "phase_queue_counts_as_completion": false,
"submission_state": {
"status": "missing",
"path": "evidence/world_class/submissions/native-client-telemetry.json",
@@ -596,9 +1186,456 @@
}
}
],
+ "repair_checklist": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-precheck-openai-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "openai-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env precheck is missing.",
+ "next_action": "Set OPENAI_API_KEY in the operator shell; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-model_executed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "model_executed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Run provider-backed output-exec with real credentials.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-token_observed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "token_observed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Provider execution should return non-estimated token usage.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ],
+ "phase_queue": [
+ {
+ "phase": "unblock-access",
+ "label": "Unblock access",
+ "priority": 20,
+ "status": "blocked",
+ "blocked_count": 4,
+ "row_count": 4,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-precheck-human-reviewer",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-precheck-human-reviewer",
+ "evidence_key": "human-adjudication",
+ "repair_type": "precheck",
+ "target": "human-reviewer",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Required human precheck is human-required.",
+ "next_action": "Assign a real reviewer identity before claiming human adjudication.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-precheck-external-client",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "precheck",
+ "target": "external-client",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-precheck-native-guard",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "precheck",
+ "target": "native-guard",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Required external precheck is external-required.",
+ "next_action": "Attach a real target-client or external installer runtime guard; metadata fallback is not enough.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-precheck-openai-api-key",
+ "evidence_key": "provider-holdout",
+ "repair_type": "precheck",
+ "target": "openai-api-key",
+ "phase": "unblock-access",
+ "priority": 20,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Required env precheck is missing.",
+ "next_action": "Set OPENAI_API_KEY in the operator shell; never commit or print the value.",
+ "verification_command": "python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "prechecks",
+ "counts_as_completion": false
+ }
+ ]
+ },
+ {
+ "phase": "collect-source",
+ "label": "Collect source",
+ "priority": 40,
+ "status": "blocked",
+ "blocked_count": 9,
+ "row_count": 9,
+ "owners": [
+ "Browser/Chrome/IDE/provider client integrator",
+ "human reviewer",
+ "operator with provider credentials",
+ "target client or installer integrator"
+ ],
+ "evidence_keys": [
+ "human-adjudication",
+ "native-client-telemetry",
+ "native-permission-enforcement",
+ "provider-holdout"
+ ],
+ "next_action_id": "human-adjudication-source-check-blind_review_attested",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "counts_as_completion": false,
+ "rows": [
+ {
+ "action_id": "human-adjudication-source-check-blind_review_attested",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "blind_review_attested",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Set reviewer_attestation only after choices are completed before opening the answer key.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-judgment_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "judgment_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '==pair_count'.",
+ "next_action": "Every pair needs one valid human judgment.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-pending_count",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "pending_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value 5 does not satisfy '==0'.",
+ "next_action": "Record a reviewer choice and reason for every pair.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-ready_for_human_evidence",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "ready_for_human_evidence",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Complete all reviewer decisions with metadata and rationale, plus blind-review attestation and integrity fingerprints.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "human-adjudication-source-check-reviewer_metadata_present",
+ "evidence_key": "human-adjudication",
+ "repair_type": "source-check",
+ "target": "reviewer_metadata_present",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "human reviewer",
+ "status": "blocked",
+ "blocking_reason": "Current value False does not satisfy 'true'.",
+ "next_action": "Record reviewer and reviewed_at before adjudication can count.",
+ "verification_command": "python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-client-telemetry-source-check-external_source_events",
+ "evidence_key": "native-client-telemetry",
+ "repair_type": "source-check",
+ "target": "external_source_events",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "Browser/Chrome/IDE/provider client integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Import at least one metadata-only event from a real client.",
+ "verification_command": "python3 scripts/yao.py telemetry-import . --input-jsonl .yao/telemetry_spool/external_events.jsonl && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "native-permission-enforcement-source-check-native_enforcement_count",
+ "evidence_key": "native-permission-enforcement",
+ "repair_type": "source-check",
+ "target": "native_enforcement_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "target client or installer integrator",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Collect real target-client or external runtime guard proof.",
+ "verification_command": "python3 scripts/yao.py runtime-permissions . --package-dir dist && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-model_executed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "model_executed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Run provider-backed output-exec with real credentials.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ },
+ {
+ "action_id": "provider-holdout-source-check-token_observed_count",
+ "evidence_key": "provider-holdout",
+ "repair_type": "source-check",
+ "target": "token_observed_count",
+ "phase": "collect-source",
+ "priority": 40,
+ "owner": "operator with provider credentials",
+ "status": "blocked",
+ "blocking_reason": "Current value 0 does not satisfy '>0'.",
+ "next_action": "Provider execution should return non-estimated token usage.",
+ "verification_command": "python3 scripts/yao.py output-exec --provider-runner openai --timeout-seconds 60 && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions",
+ "source": "source_checklist",
+ "counts_as_completion": false
+ }
+ ]
+ }
+ ],
"source_reports": {
"ledger": "reports/world_class_evidence_ledger.json",
"intake": "reports/world_class_evidence_intake.json",
+ "preflight": "reports/world_class_evidence_preflight.json",
"submission_review": "reports/world_class_submission_review.json",
"claim_guard": "reports/world_class_claim_guard.json"
},
diff --git a/reports/world_class_operator_runbook.md b/reports/world_class_operator_runbook.md
index 25c0dde6..09376291 100644
--- a/reports/world_class_operator_runbook.md
+++ b/reports/world_class_operator_runbook.md
@@ -11,6 +11,9 @@ Generated at: `2026-06-17`
- pending: `4`
- awaiting submission: `4`
- ready for ledger review: `0`
+- phase queue: `2` blocked / `2` phases
+- phase queue rows: `13`
+- phase queue counts as completion: `false`
This runbook coordinates evidence collection only. It does not accept submissions or make world-class completion true.
@@ -21,6 +24,13 @@ This runbook coordinates evidence collection only. It does not accept submission
3. Replace template-only fields with aggregate evidence and provenance.
4. Validate intake, review the queue, refresh the ledger, then run the claim guard.
+## Phase Queue
+
+| Phase | Status | Rows | Blocked | Owners | Next action | Verify |
+| --- | --- | ---: | ---: | --- | --- | --- |
+| `unblock-access` | `blocked` | `4` | `4` | Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator | Assign a real reviewer identity before claiming human adjudication. | `python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` |
+| `collect-source` | `blocked` | `9` | `9` | Browser/Chrome/IDE/provider client integrator, human reviewer, operator with provider credentials, target client or installer integrator | Set reviewer_attestation only after choices are completed before opening the answer key. | `python3 scripts/yao.py output-review && python3 scripts/yao.py world-class-preflight . --submissions-dir evidence/world_class/submissions` |
+
## Evidence Items
| Evidence | Ledger | Intake | Review | Blocked checks | Next source action | Owner |
@@ -28,16 +38,25 @@ This runbook coordinates evidence collection only. It does not accept submission
| `provider-holdout` | `pending` | `awaiting-submission` | `awaiting-submission` | `2` | Run provider-backed output-exec with real credentials. | operator with provider credentials |
| `human-adjudication` | `pending` | `awaiting-submission` | `awaiting-submission` | `5` | Record a reviewer choice and reason for every pair. | human reviewer |
| `native-permission-enforcement` | `pending` | `awaiting-submission` | `awaiting-submission` | `1` | Collect real target-client or external runtime guard proof. | target client or installer integrator |
-| `native-client-telemetry` | `pending` | `awaiting-submission` | `awaiting-submission` | `2` | Import at least one metadata-only event from a real client. | Browser/Chrome/IDE/provider client integrator |
+| `native-client-telemetry` | `pending` | `awaiting-submission` | `awaiting-submission` | `1` | Import at least one metadata-only event from a real client. | Browser/Chrome/IDE/provider client integrator |
## Provider Holdout
- objective: Collect at least one provider-backed output-eval holdout run with model, timing, and token metadata.
- blocking reason: No evidence packet has been submitted for review.
- blocked source checks: `2`
+- repair rows: `3` blocked
+- phase queue: `2` blocked phases
- submission: `evidence/world_class/submissions/provider-holdout.json`
- template: `evidence/world_class/templates/provider-holdout.intake.json`
+### Phase Queue
+
+| Phase | Status | Rows | Blocked | Next action |
+| --- | --- | ---: | ---: | --- |
+| `unblock-access` | `blocked` | `1` | `1` | Set OPENAI_API_KEY in the operator shell; never commit or print the value. |
+| `collect-source` | `blocked` | `2` | `2` | Run provider-backed output-exec with real credentials. |
+
### Source Runbook
- Set OPENAI_API_KEY in the operator shell before running provider evidence; never commit or print the value.
@@ -101,9 +120,18 @@ This runbook coordinates evidence collection only. It does not accept submission
- objective: Record real blind A/B reviewer decisions before claiming human output review completion.
- blocking reason: No evidence packet has been submitted for review.
- blocked source checks: `5`
+- repair rows: `6` blocked
+- phase queue: `2` blocked phases
- submission: `evidence/world_class/submissions/human-adjudication.json`
- template: `evidence/world_class/templates/human-adjudication.intake.json`
+### Phase Queue
+
+| Phase | Status | Rows | Blocked | Next action |
+| --- | --- | ---: | ---: | --- |
+| `unblock-access` | `blocked` | `1` | `1` | Assign a real reviewer identity before claiming human adjudication. |
+| `collect-source` | `blocked` | `5` | `5` | Set reviewer_attestation only after choices are completed before opening the answer key. |
+
### Source Runbook
- python3 scripts/yao.py output-review-kit --write-template
@@ -192,9 +220,18 @@ This runbook coordinates evidence collection only. It does not accept submission
- objective: Prove at least one real target client or external installer runtime guard enforces approved high-permission capabilities.
- blocking reason: No evidence packet has been submitted for review.
- blocked source checks: `1`
+- repair rows: `2` blocked
+- phase queue: `2` blocked phases
- submission: `evidence/world_class/submissions/native-permission-enforcement.json`
- template: `evidence/world_class/templates/native-permission-enforcement.intake.json`
+### Phase Queue
+
+| Phase | Status | Rows | Blocked | Next action |
+| --- | --- | ---: | ---: | --- |
+| `unblock-access` | `blocked` | `1` | `1` | Attach a real target-client or external installer runtime guard; metadata fallback is not enough. |
+| `collect-source` | `blocked` | `1` | `1` | Collect real target-client or external runtime guard proof. |
+
### Source Runbook
- Implement or connect a real target client or external installer runtime guard that blocks undeclared network, file_write, or subprocess capabilities.
@@ -261,10 +298,19 @@ This runbook coordinates evidence collection only. It does not accept submission
- objective: Import production metadata-only events from a real external client into the local drift loop.
- blocking reason: No evidence packet has been submitted for review.
-- blocked source checks: `2`
+- blocked source checks: `1`
+- repair rows: `2` blocked
+- phase queue: `2` blocked phases
- submission: `evidence/world_class/submissions/native-client-telemetry.json`
- template: `evidence/world_class/templates/native-client-telemetry.intake.json`
+### Phase Queue
+
+| Phase | Status | Rows | Blocked | Next action |
+| --- | --- | ---: | ---: | --- |
+| `unblock-access` | `blocked` | `1` | `1` | Install a real Browser, Chrome, IDE, or provider client that emits metadata-only events. |
+| `collect-source` | `blocked` | `1` | `1` | Import at least one metadata-only event from a real client. |
+
### Source Runbook
- python3 scripts/telemetry_native_host.py . --write-launcher /tmp/yao-telemetry-host.sh --write-manifest /tmp/yao-telemetry-host.json --allowed-origin chrome-extension:///
@@ -314,14 +360,13 @@ This runbook coordinates evidence collection only. It does not accept submission
### Next Source Actions
- Import at least one metadata-only event from a real client.
-- Telemetry must include adoption outcome evidence.
### Source Evidence Snapshot
| Check | Current | Expected | Status | Next action |
| --- | --- | --- | --- | --- |
| External events | `0` | `>0` | `blocked` | Import at least one metadata-only event from a real client. |
-| Adoption sample | `0` | `>0` | `blocked` | Telemetry must include adoption outcome evidence. |
+| Adoption sample | `1` | `>0` | `pass` | Telemetry must include adoption outcome evidence. |
| Raw content blocked | `False` | `false` | `pass` | Telemetry must stay metadata-only. |
## Boundary
diff --git a/reports/world_class_submission_review.json b/reports/world_class_submission_review.json
index 4c9a96e6..3c09e53d 100644
--- a/reports/world_class_submission_review.json
+++ b/reports/world_class_submission_review.json
@@ -14,8 +14,8 @@
"unmatched_submission_count": 0,
"invalid_submission_count": 0,
"source_check_count": 19,
- "source_pass_count": 9,
- "source_blocked_count": 10,
+ "source_pass_count": 10,
+ "source_blocked_count": 9,
"review_counts_submission_as_completion": false,
"summary": {
"review_item_count": 4,
@@ -27,8 +27,8 @@
"unmatched_submission_count": 0,
"invalid_submission_count": 0,
"source_check_count": 19,
- "source_pass_count": 9,
- "source_blocked_count": 10,
+ "source_pass_count": 10,
+ "source_blocked_count": 9,
"ready_to_claim_world_class": false,
"review_counts_submission_as_completion": false,
"decision": "awaiting-submissions"
@@ -378,7 +378,7 @@
"intake_errors": [],
"observed_state": {
"external_source_events": 0,
- "adoption_sample_count": 0,
+ "adoption_sample_count": 1,
"raw_content_allowed": false,
"risk_band": "low",
"accepted": false
@@ -399,8 +399,8 @@
"label": "Adoption sample",
"field": "adoption_sample_count",
"expected": ">0",
- "actual": 0,
- "status": "blocked",
+ "actual": 1,
+ "status": "pass",
"source_accepted": false,
"next_action": "Telemetry must include adoption outcome evidence."
},
@@ -416,8 +416,8 @@
}
],
"source_check_count": 3,
- "source_pass_count": 1,
- "source_blocked_count": 2,
+ "source_pass_count": 2,
+ "source_blocked_count": 1,
"success_checks": [
"reports/adoption_drift_report.json summary.source_types.external > 0",
"reports/adoption_drift_report.json summary.adoption_sample_count > 0",
diff --git a/reports/world_class_submission_review.md b/reports/world_class_submission_review.md
index a6b7a5d5..2dda5c01 100644
--- a/reports/world_class_submission_review.md
+++ b/reports/world_class_submission_review.md
@@ -153,7 +153,7 @@ This report is a read-only reviewer queue. It does not accept evidence or make w
#### Source Checks
- External events: 0 / >0 => blocked
-- Adoption sample: 0 / >0 => blocked
+- Adoption sample: 1 / >0 => pass
- Raw content blocked: False / false => pass
#### Completion Assertions
diff --git a/scripts/evidence_consistency_phase_queue.py b/scripts/evidence_consistency_phase_queue.py
index 1e957c7f..007cc454 100644
--- a/scripts/evidence_consistency_phase_queue.py
+++ b/scripts/evidence_consistency_phase_queue.py
@@ -74,9 +74,27 @@ def keyed_phase_queue_signatures_from_items(items: dict[str, dict[str, Any]], qu
return signatures
+def any_phase_queue_counts_as_completion(
+ *queues: list[dict[str, Any]],
+ item_maps: dict[str, dict[str, Any]] | None = None,
+) -> bool:
+ item_maps = item_maps or {}
+ for queue in queues:
+ for item in queue:
+ if isinstance(item, dict) and item.get("counts_as_completion") is True:
+ return True
+ for item in item_maps.values():
+ queue = item.get("phase_queue", []) if isinstance(item.get("phase_queue", []), list) else []
+ for row in queue:
+ if isinstance(row, dict) and row.get("counts_as_completion") is True:
+ return True
+ return False
+
+
def build_phase_queue_consistency_check(
*,
world_class_preflight: dict[str, Any],
+ world_class_operator_runbook: dict[str, Any],
review_studio: dict[str, Any],
report_paths: dict[str, str],
) -> dict[str, Any]:
@@ -88,6 +106,7 @@ def build_phase_queue_consistency_check(
expected_queue = build_phase_queue(repair_rows)
expected_summary = summarize_phase_queue(expected_queue)
preflight_items = keyed_preflight_items(world_class_preflight)
+ operator_runbook_items = keyed_preflight_items(world_class_operator_runbook)
review_steps = world_class_review_action_steps(review_studio)
expected = {
"summary": summary_signature(expected_summary),
@@ -106,14 +125,35 @@ def build_phase_queue_consistency_check(
if isinstance(world_class_preflight.get("phase_queue", []), list)
else []
),
+ "operator_runbook_summary": summary_signature(
+ world_class_operator_runbook.get("summary", {})
+ if isinstance(world_class_operator_runbook.get("summary", {}), dict)
+ else {}
+ ),
+ "operator_runbook_top_level_phase_queue": phase_queue_signature(
+ world_class_operator_runbook.get("phase_queue", [])
+ if isinstance(world_class_operator_runbook.get("phase_queue", []), list)
+ else []
+ ),
"item_phase_queues": keyed_phase_queue_signatures_from_items(preflight_items, "phase_queue"),
+ "operator_runbook_phase_queues": keyed_phase_queue_signatures_from_items(
+ operator_runbook_items,
+ "phase_queue",
+ ),
"review_studio_phase_queues": keyed_phase_queue_signatures_from_items(review_steps, "phase_queue"),
- "phase_queue_counts_as_completion": any(
- item.get("counts_as_completion") is True
- for item in world_class_preflight.get("phase_queue", [])
- if isinstance(item, dict)
+ "phase_queue_counts_as_completion": any_phase_queue_counts_as_completion(
+ world_class_preflight.get("phase_queue", [])
+ if isinstance(world_class_preflight.get("phase_queue", []), list)
+ else [],
+ world_class_operator_runbook.get("phase_queue", [])
+ if isinstance(world_class_operator_runbook.get("phase_queue", []), list)
+ else [],
+ item_maps={**preflight_items, **operator_runbook_items, **review_steps},
),
}
+ expected["operator_runbook_summary"] = expected["summary"]
+ expected["operator_runbook_top_level_phase_queue"] = expected["top_level_phase_queue"]
+ expected["operator_runbook_phase_queues"] = expected["item_phase_queues"]
expected["review_studio_phase_queues"] = expected["item_phase_queues"]
return {
"key": "world-class-phase-queue-consistency",
@@ -121,9 +161,13 @@ def build_phase_queue_consistency_check(
"status": "pass" if expected == actual else "fail",
"expected": expected,
"actual": actual,
- "paths": [report_paths["world_class_preflight"], report_paths["review_studio"]],
+ "paths": [
+ report_paths["world_class_preflight"],
+ report_paths["world_class_operator_runbook"],
+ report_paths["review_studio"],
+ ],
"detail": (
- "Phase queues must be derived from repair rows in preflight and mirrored into Review Studio "
- "without counting queue guidance as completion evidence."
+ "Phase queues must be derived from repair rows in preflight and mirrored into the operator runbook "
+ "and Review Studio without counting queue guidance as completion evidence."
),
}
diff --git a/scripts/render_evidence_consistency.py b/scripts/render_evidence_consistency.py
index 1b09c201..358c7f93 100644
--- a/scripts/render_evidence_consistency.py
+++ b/scripts/render_evidence_consistency.py
@@ -404,6 +404,7 @@ def build_report(skill_dir: Path, generated_at: str) -> dict[str, Any]:
checks.append(
build_phase_queue_consistency_check(
world_class_preflight=world_class_preflight,
+ world_class_operator_runbook=world_class_operator_runbook,
review_studio=review_studio,
report_paths=REQUIRED_REPORTS,
)
diff --git a/scripts/render_world_class_operator_runbook.py b/scripts/render_world_class_operator_runbook.py
index 162b23f6..1fba377c 100644
--- a/scripts/render_world_class_operator_runbook.py
+++ b/scripts/render_world_class_operator_runbook.py
@@ -8,6 +8,7 @@ from typing import Any
from html_rendering import html_text
from render_world_class_evidence_intake import build_intake
from render_world_class_evidence_ledger import build_ledger
+from render_world_class_preflight import build_preflight
from render_world_class_submission_review import build_submission_review
@@ -38,6 +39,7 @@ def build_runbook_item(
entry: dict[str, Any],
checklist: dict[str, Any],
review_item: dict[str, Any],
+ preflight_item: dict[str, Any],
) -> dict[str, Any]:
commands = checklist.get("commands", {}) if isinstance(checklist.get("commands", {}), dict) else {}
must_collect = checklist.get("must_collect", {}) if isinstance(checklist.get("must_collect", {}), dict) else {}
@@ -82,6 +84,24 @@ def build_runbook_item(
"source_checklist": source_checklist,
"blocked_source_check_count": len(blocked_source_checks),
"next_source_actions": next_source_actions,
+ "repair_checklist": preflight_item.get("repair_checklist", [])
+ if isinstance(preflight_item.get("repair_checklist", []), list)
+ else [],
+ "repair_blocked_count": sum(
+ 1
+ for row in preflight_item.get("repair_checklist", [])
+ if isinstance(row, dict) and row.get("status") != "ready"
+ ),
+ "repair_counts_as_completion": False,
+ "phase_queue": preflight_item.get("phase_queue", [])
+ if isinstance(preflight_item.get("phase_queue", []), list)
+ else [],
+ "phase_queue_blocked_count": sum(
+ 1
+ for row in preflight_item.get("phase_queue", [])
+ if isinstance(row, dict) and row.get("status") != "ready"
+ ),
+ "phase_queue_counts_as_completion": False,
"submission_state": entry.get("submission_state", {}),
"anti_overclaim": entry.get("anti_overclaim", {}),
}
@@ -92,14 +112,22 @@ def build_operator_runbook(skill_dir: Path, generated_at: str, submissions_dir:
ledger = build_ledger(skill_dir, generated_at, submissions_dir=submissions_dir)
intake = build_intake(skill_dir, generated_at, submissions_dir=submissions_dir)
review = build_submission_review(skill_dir, generated_at, submissions_dir=submissions_dir)
+ preflight = build_preflight(skill_dir, generated_at, submissions_dir=submissions_dir)
checklist_by_key = by_key(intake.get("operator_checklist", []), "evidence_key")
review_by_key = by_key(review.get("items", []), "evidence_key")
+ preflight_by_key = by_key(preflight.get("items", []), "evidence_key")
items = [
- build_runbook_item(entry, checklist_by_key.get(str(entry.get("key", "")), {}), review_by_key.get(str(entry.get("key", "")), {}))
+ build_runbook_item(
+ entry,
+ checklist_by_key.get(str(entry.get("key", "")), {}),
+ review_by_key.get(str(entry.get("key", "")), {}),
+ preflight_by_key.get(str(entry.get("key", "")), {}),
+ )
for entry in ledger.get("entries", [])
]
summary = ledger.get("summary", {})
review_summary = review.get("summary", {})
+ preflight_summary = preflight.get("summary", {}) if isinstance(preflight.get("summary", {}), dict) else {}
return {
"schema_version": "1.0",
"ok": True,
@@ -116,6 +144,15 @@ def build_operator_runbook(skill_dir: Path, generated_at: str, submissions_dir:
"source_check_count": review_summary.get("source_check_count", 0),
"source_pass_count": review_summary.get("source_pass_count", 0),
"source_blocked_count": review_summary.get("source_blocked_count", 0),
+ "repair_checklist_count": preflight_summary.get("repair_checklist_count", 0),
+ "repair_blocked_count": preflight_summary.get("repair_blocked_count", 0),
+ "phase_queue_count": preflight_summary.get("phase_queue_count", 0),
+ "phase_queue_blocked_count": preflight_summary.get("phase_queue_blocked_count", 0),
+ "phase_queue_row_count": preflight_summary.get("phase_queue_row_count", 0),
+ "phase_queue_next_phase": preflight_summary.get("phase_queue_next_phase", ""),
+ "phase_queue_next_action_id": preflight_summary.get("phase_queue_next_action_id", ""),
+ "phase_queue_next_command": preflight_summary.get("phase_queue_next_command", ""),
+ "phase_queue_counts_as_completion": False,
"ready_to_claim_world_class": summary.get("ready_to_claim_world_class") is True,
"runbook_counts_as_completion": False,
"decision": "ready-for-completion-audit" if summary.get("ready_to_claim_world_class") is True else "collect-evidence",
@@ -125,9 +162,14 @@ def build_operator_runbook(skill_dir: Path, generated_at: str, submissions_dir:
"runbook_counts_submission_as_completion": False,
},
"items": items,
+ "repair_checklist": preflight.get("repair_checklist", [])
+ if isinstance(preflight.get("repair_checklist", []), list)
+ else [],
+ "phase_queue": preflight.get("phase_queue", []) if isinstance(preflight.get("phase_queue", []), list) else [],
"source_reports": {
"ledger": "reports/world_class_evidence_ledger.json",
"intake": "reports/world_class_evidence_intake.json",
+ "preflight": "reports/world_class_evidence_preflight.json",
"submission_review": "reports/world_class_submission_review.json",
"claim_guard": "reports/world_class_claim_guard.json",
},
@@ -161,6 +203,9 @@ def render_markdown(report: dict[str, Any]) -> str:
f"- pending: `{summary['pending_count']}`",
f"- awaiting submission: `{summary['awaiting_submission_count']}`",
f"- ready for ledger review: `{summary['ready_for_ledger_review_count']}`",
+ f"- phase queue: `{summary['phase_queue_blocked_count']}` blocked / `{summary['phase_queue_count']}` phases",
+ f"- phase queue rows: `{summary['phase_queue_row_count']}`",
+ f"- phase queue counts as completion: `{str(summary['phase_queue_counts_as_completion']).lower()}`",
"",
"This runbook coordinates evidence collection only. It does not accept submissions or make world-class completion true.",
"",
@@ -171,11 +216,30 @@ def render_markdown(report: dict[str, Any]) -> str:
"3. Replace template-only fields with aggregate evidence and provenance.",
"4. Validate intake, review the queue, refresh the ledger, then run the claim guard.",
"",
- "## Evidence Items",
- "",
- "| Evidence | Ledger | Intake | Review | Blocked checks | Next source action | Owner |",
- "| --- | --- | --- | --- | ---: | --- | --- |",
]
+ lines.extend(
+ [
+ "## Phase Queue",
+ "",
+ "| Phase | Status | Rows | Blocked | Owners | Next action | Verify |",
+ "| --- | --- | ---: | ---: | --- | --- | --- |",
+ ]
+ )
+ for row in report.get("phase_queue", []):
+ owners = ", ".join(str(owner) for owner in row.get("owners", []))
+ lines.append(
+ f"| `{row.get('phase', '')}` | `{row.get('status', '')}` | `{row.get('row_count', 0)}` | "
+ f"`{row.get('blocked_count', 0)}` | {owners} | {row.get('next_action', '')} | `{row.get('verification_command', '')}` |"
+ )
+ lines.append("")
+ lines.extend(
+ [
+ "## Evidence Items",
+ "",
+ "| Evidence | Ledger | Intake | Review | Blocked checks | Next source action | Owner |",
+ "| --- | --- | --- | --- | ---: | --- | --- |",
+ ]
+ )
for item in report["items"]:
next_action = item.get("next_source_actions", ["none"])[0] if item.get("next_source_actions") else "none"
lines.append(
@@ -191,8 +255,28 @@ def render_markdown(report: dict[str, Any]) -> str:
f"- objective: {item['objective']}",
f"- blocking reason: {item['blocking_reason']}",
f"- blocked source checks: `{item.get('blocked_source_check_count', 0)}`",
+ f"- repair rows: `{item.get('repair_blocked_count', 0)}` blocked",
+ f"- phase queue: `{item.get('phase_queue_blocked_count', 0)}` blocked phases",
f"- submission: `{item['submission_path'] or 'missing'}`",
f"- template: `{item['template_path'] or 'missing'}`",
+ "",
+ "### Phase Queue",
+ "",
+ "| Phase | Status | Rows | Blocked | Next action |",
+ "| --- | --- | ---: | ---: | --- |",
+ ]
+ )
+ item_phase_queue = item.get("phase_queue", [])
+ if item_phase_queue:
+ for row in item_phase_queue:
+ lines.append(
+ f"| `{row.get('phase', '')}` | `{row.get('status', '')}` | `{row.get('row_count', 0)}` | "
+ f"`{row.get('blocked_count', 0)}` | {row.get('next_action', '')} |"
+ )
+ else:
+ lines.append("| No phase queue listed. | `n/a` | `0` | `0` | n/a |")
+ lines.extend(
+ [
"",
"### Source Runbook",
"",
@@ -267,6 +351,21 @@ def html_source_checks(rows: list[dict[str, Any]]) -> str:
return ""
+def html_phase_queue(rows: list[dict[str, Any]]) -> str:
+ if not rows:
+ return "No phase queue listed.
"
+ items = []
+ for row in rows:
+ items.append(
+ ""
+ f"{html_text(row.get('label', row.get('phase', '')))}"
+ f"{html_text(row.get('phase', ''))}: {html_text(row.get('blocked_count', 0))} / {html_text(row.get('row_count', 0))}"
+ f"{html_text(row.get('next_action', ''))}"
+ ""
+ )
+ return ""
+
+
def render_html_item(item: dict[str, Any]) -> str:
commands = "".join(
f"{html_text(label.replace('_', ' '))}{html_text(command)}"
@@ -281,8 +380,10 @@ def render_html_item(item: dict[str, Any]) -> str:
Owner{html_text(item['owner'])}
Ledger{html_text(item['ledger_status'])}
Blocked{html_text(item.get('blocked_source_check_count', 0))}
+ Queue{html_text(item.get('phase_queue_blocked_count', 0))}
Submission{html_text(item['submission_path'])}
+ Phase Queue
{html_phase_queue(item.get('phase_queue', []))}
Source Runbook
{html_list(item.get('execution_runbook', []), 'No source runbook listed.')}
@@ -303,6 +404,7 @@ def render_html(report: dict[str, Any]) -> str:
("Awaiting", summary["awaiting_submission_count"]),
("Ready", summary["ready_for_ledger_review_count"]),
("Source", f"{summary.get('source_pass_count', 0)}/{summary.get('source_check_count', 0)}"),
+ ("Queue", f"{summary.get('phase_queue_blocked_count', 0)}/{summary.get('phase_queue_count', 0)}"),
("Blocked", summary.get("source_blocked_count", 0)),
("Invalid", summary["invalid_submission_count"]),
]
@@ -370,6 +472,7 @@ def render_html(report: dict[str, Any]) -> str:
{stat_html}
Fast Path
- Run the real external or human work for one evidence item.
- Generate and fill the matching submission draft.
- Validate intake and inspect the submission review queue.
- Refresh the ledger and run the claim guard before making any completion claim.
+
Phase Queue
{html_phase_queue(report.get('phase_queue', []))}
Evidence Items
{item_html}
Boundary
- Planned work, draft packets, metadata fallback, pending human decisions, and local command runners do not count as completion.
- Valid intake means ready for submission review; ledger review still requires passing source evidence.
- The world-class ledger and claim guard remain the source of truth.
diff --git a/tests/verify_evidence_consistency.py b/tests/verify_evidence_consistency.py
index 4a14d54d..4b06826f 100644
--- a/tests/verify_evidence_consistency.py
+++ b/tests/verify_evidence_consistency.py
@@ -187,7 +187,11 @@ def main() -> None:
]
phase_queue_actual = checks["world-class-phase-queue-consistency"]["actual"]
assert phase_queue_actual["summary"]["phase_queue_count"] == 2, phase_queue_actual
- assert phase_queue_actual["summary"]["phase_queue_row_count"] == 13, phase_queue_actual
+ assert phase_queue_actual["summary"]["phase_queue_row_count"] >= 13, phase_queue_actual
+ assert phase_queue_actual["summary"] == phase_queue_actual["operator_runbook_summary"], phase_queue_actual
+ assert phase_queue_actual["top_level_phase_queue"] == phase_queue_actual[
+ "operator_runbook_top_level_phase_queue"
+ ], phase_queue_actual
assert phase_queue_actual["phase_queue_counts_as_completion"] is False, phase_queue_actual
assert set(phase_queue_actual["item_phase_queues"]) == {
"provider-holdout",
@@ -195,6 +199,9 @@ def main() -> None:
"native-permission-enforcement",
"native-client-telemetry",
}, phase_queue_actual
+ assert phase_queue_actual["operator_runbook_phase_queues"] == phase_queue_actual["item_phase_queues"], (
+ phase_queue_actual
+ )
assert phase_queue_actual["review_studio_phase_queues"] == phase_queue_actual["item_phase_queues"], (
phase_queue_actual
)
@@ -361,6 +368,15 @@ def main() -> None:
phase_queue_payload["phase_queue"][0]["counts_as_completion"] = True
phase_queue_payload["items"][0]["phase_queue"][0]["row_count"] = 999
phase_queue_path.write_text(json.dumps(phase_queue_payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ operator_phase_queue_path = phase_queue_drift_root / "reports" / "world_class_operator_runbook.json"
+ operator_phase_queue_payload = json.loads(operator_phase_queue_path.read_text(encoding="utf-8"))
+ operator_phase_queue_payload["summary"]["phase_queue_count"] = 99
+ operator_phase_queue_payload["phase_queue"][0]["row_count"] = 999
+ operator_phase_queue_payload["items"][0]["phase_queue"][0]["row_count"] = 999
+ operator_phase_queue_path.write_text(
+ json.dumps(operator_phase_queue_payload, ensure_ascii=False, indent=2) + "\n",
+ encoding="utf-8",
+ )
phase_queue_drift_proc = run(
[
sys.executable,
diff --git a/tests/verify_world_class_operator_runbook.py b/tests/verify_world_class_operator_runbook.py
index 91bd16cd..132425fb 100644
--- a/tests/verify_world_class_operator_runbook.py
+++ b/tests/verify_world_class_operator_runbook.py
@@ -111,8 +111,19 @@ def main() -> None:
assert summary["source_check_count"] >= 13, summary
assert summary["source_pass_count"] + summary["source_blocked_count"] == summary["source_check_count"], summary
assert summary["source_blocked_count"] >= 6, summary
+ assert summary["repair_checklist_count"] >= summary["source_blocked_count"], summary
+ assert summary["phase_queue_count"] == 2, summary
+ assert summary["phase_queue_blocked_count"] == 2, summary
+ assert summary["phase_queue_row_count"] == summary["repair_checklist_count"], summary
+ assert summary["phase_queue_next_phase"] == "unblock-access", summary
+ assert summary["phase_queue_counts_as_completion"] is False, summary
assert summary["ready_to_claim_world_class"] is False, summary
assert summary["runbook_counts_as_completion"] is False, summary
+ assert payload["repair_checklist"], payload
+ assert len(payload["phase_queue"]) == summary["phase_queue_count"], payload["phase_queue"]
+ assert sum(item["row_count"] for item in payload["phase_queue"]) == summary["phase_queue_row_count"], payload[
+ "phase_queue"
+ ]
items = {item["evidence_key"]: item for item in payload["items"]}
assert set(items) == {
"provider-holdout",
@@ -124,6 +135,11 @@ def main() -> None:
assert provider["review_state"] == "awaiting-submission", provider
assert provider["source_accepted"] is False, provider
assert provider["blocked_source_check_count"] == 2, provider
+ assert provider["repair_blocked_count"] == 3, provider
+ assert provider["repair_counts_as_completion"] is False, provider
+ assert provider["phase_queue_blocked_count"] == 2, provider
+ assert provider["phase_queue_counts_as_completion"] is False, provider
+ assert [item["phase"] for item in provider["phase_queue"]] == ["unblock-access", "collect-source"], provider
assert any("output-exec --provider-runner openai" in step for step in provider["execution_runbook"]), provider
assert not any("
" in step or "OPENAI_API_KEY=" in step for step in provider["execution_runbook"]), provider
assert "Run provider-backed output-exec with real credentials." in provider["next_source_actions"], provider
@@ -146,10 +162,14 @@ def main() -> None:
markdown = output_md.read_text(encoding="utf-8")
assert "World-Class Operator Runbook" in markdown, markdown
assert "runbook counts as completion: `false`" in markdown, markdown
+ assert "phase queue counts as completion: `false`" in markdown, markdown
+ assert "## Phase Queue" in markdown, markdown
+ assert "| `unblock-access` | `blocked` |" in markdown, markdown
assert "Valid intake means ready for submission review; ledger review still requires passing source evidence." in markdown, markdown
assert "| Evidence | Ledger | Intake | Review | Blocked checks | Next source action | Owner |" in markdown, markdown
assert "| `provider-holdout` | `pending` | `awaiting-submission` | `awaiting-submission` | `2` | Run provider-backed output-exec with real credentials." in markdown, markdown
assert "Source Runbook" in markdown, markdown
+ assert "### Phase Queue" in markdown, markdown
assert "output-exec --provider-runner openai" in markdown, markdown
assert "" not in markdown, markdown
assert "OPENAI_API_KEY=" not in markdown, markdown
@@ -164,8 +184,11 @@ def main() -> None:
assert "position:sticky" in html, html
assert "Ready0" in html, html
assert "Invalid0" in html, html
+ assert f"Queue{summary['phase_queue_blocked_count']}/{summary['phase_queue_count']}" in html, html
assert f"Blocked{summary['source_blocked_count']}" in html, html
assert "Blocked2" in html, html
+ assert "Queue2" in html, html
+ assert "Phase Queue" in html, html
assert "Next Source Actions" in html, html
assert "Source Runbook" in html, html
assert "output-exec --provider-runner openai" in html, html
@@ -208,12 +231,16 @@ def main() -> None:
assert submitted_summary["ready_for_ledger_review_count"] == 0, submitted_summary
assert submitted_summary["source_pass_count"] + submitted_summary["source_blocked_count"] == submitted_summary["source_check_count"], submitted_summary
assert submitted_summary["source_blocked_count"] >= 6, submitted_summary
+ assert submitted_summary["phase_queue_count"] == 2, submitted_summary
+ assert submitted_summary["phase_queue_blocked_count"] == 2, submitted_summary
+ assert submitted_summary["phase_queue_counts_as_completion"] is False, submitted_summary
assert submitted_summary["ready_to_claim_world_class"] is False, submitted_summary
submitted_provider = {item["evidence_key"]: item for item in submitted["items"]}["provider-holdout"]
assert submitted_provider["intake_readiness"] == "fix-submission", submitted_provider
assert submitted_provider["review_state"] == "fix-submission", submitted_provider
assert submitted_provider["source_accepted"] is False, submitted_provider
assert submitted_provider["blocked_source_check_count"] == 2, submitted_provider
+ assert submitted_provider["phase_queue_blocked_count"] == 2, submitted_provider
assert submitted_provider["next_source_actions"] == provider["next_source_actions"], submitted_provider
assert "tests/tmp_world_class_operator_runbook/valid_submissions" in submitted_provider["commands"]["validate_intake"], submitted_provider
assert "tests/tmp_world_class_operator_runbook/valid_submissions" in submitted_provider["commands"]["review_queue"], submitted_provider