Blind A/B Human Review

# Reviewer cockpit for output quality decisions

Compare visible Variant A and Variant B outputs, fill the decision file, then run adjudication. The answer key is intentionally hidden from this page.

Cases 5
Ready 0
Pending 5
Invalid 0

## Review Flow

1. Open reports/output_blind_review_pack.md or this kit and compare Variant A vs Variant B for each case.
2. Record choices in reports/output_review_decisions.json without opening the answer key.
3. Use winner_variant A or B, confidence from 0 to 1, and a short reason for every case.
4. Run python3 scripts/yao.py output-review after choices are recorded.
5. Refresh python3 scripts/yao.py review-studio . before asking for release approval.

## Privacy

- The answer key is intentionally withheld from this kit.
- Do not inspect hidden labels or expected winners before decisions are recorded.
- Do not paste private user data into decision reasons.
- Pending decisions must stay pending instead of being counted as human agreement.

Case 01

### skill-package-contract

Awaiting

Turn this repeated workflow into a reusable team skill package.

#### Rubric

- has-entrypoint

  Output names the routeable Skill entrypoint.

  1.0
- has-interface

  Output includes neutral interface metadata.

  1.0
- has-report

  Output points reviewer to generated overview report.

  1.0
- has-resource-boundary

  Output preserves entrypoint/resource separation.

  1.0

#### Variant A

    Create a routeable SKILL.md, agents/interface.yaml, reports/skill-overview.html, and a focused eval plan. Keep the root entrypoint lean, move durable guidance into references/, scripts into scripts/, and evidence into reports/.

#### Variant B

    I can write a prompt for that workflow and include a short checklist.

Case 02

### output-eval-expectation

Awaiting

Upgrade this production skill so we know whether the generated output is better than baseline.

#### Rubric

- has-baseline-comparison

  Output explicitly compares with-skill and baseline outputs.

  1.0
- has-assertion-grading

  Output includes objective assertion grading.

  1.0
- has-scorecard

  Output produces a scorecard report path.

  1.0
- has-failure-taxonomy

  Output records failure taxonomy.

  1.0

#### Variant A

    Add Output Eval Lab cases with baseline_output and with_skill_output, run assertion grading, report with-skill vs baseline pass-rate delta, and record failure taxonomy plus next fixes in reports/output_quality_scorecard.md.

#### Variant B

    Add more examples and run the trigger eval again.

Case 03

### ir-before-packaging

Awaiting

Package this skill for OpenAI, Claude, Agent Skills, and generic targets.

#### Rubric

- has-ir-first

  Output requires Skill IR before packaging.

  1.0
- has-core-ir-fields

  Output lists core IR contract fields.

  1.0
- has-targets

  Output names requested runtime targets.

  1.0
- has-semantic-preservation

  Output says targets must preserve the capability contract.

  1.0

#### Variant A

    Export folders for each platform and adjust files as needed.

#### Variant B

    Export Skill IR first with job_to_be_done, trigger_surface, workflow, resources, eval_plan, risk, and governance. Then compile or package targets from the IR so OpenAI, Claude, Agent Skills, and generic packages preserve the same capability contract.

Case 04

### near-neighbor-boundary

Awaiting

I only need a one-off summary of these notes, not a reusable process.

#### Rubric

- declines-skill

  Output refuses unnecessary skill creation for one-off work.

  1.0
- names-near-neighbor

  Output labels the request as near-neighbor instead of owned work.

  1.0
- requires-reuse-signal

  Output asks for repeat-use evidence before packaging.

  1.0

#### Variant A

    Create a SKILL.md and a reusable workflow anyway so future notes can use it.

#### Variant B

    Do not create a skill for this one-off request. Treat it as a near-neighbor: answer the summary directly unless the user confirms repeated use, shared ownership, or a reusable output contract.

Case 05

### file-backed-governed-package

Awaiting

Turn the attached release brief source into a governed skill package.

#### Rubric

- uses-file-backed-evidence

  Output names file-backed source evidence.

  1.0
- has-governance

  Output preserves governed ownership metadata.

  1.0
- has-output-contract

  Output preserves output and rollback boundaries.

  1.0
- has-trust-and-scorecard

  Output requires trust and output scorecard artifacts.

  1.0
- does-not-invent-evidence

  Output forbids invented launch evidence.

  1.0

#### Variant A

    Draft a release announcement with the changelog, support notes, and owner name.

#### Variant B

    Use the file-backed fixture as source evidence, then create a governed skill package with SKILL.md, agents/interface.yaml, owner, review cadence, input_files, output contract, rollback boundary, trust report, and reports/output_quality_scorecard.md. Mark missing launch metrics as missing evidence instead of inventing them.

## Decision Template

Use this shape in reports/output_review_decisions.json; leave a case blank when the reviewer is not ready.

    {
      "schema_version": "1.0",
      "reviewer": "",
      "reviewed_at": "",
      "decisions": [
        {
          "case_id": "skill-package-contract",
          "winner_variant": "",
          "confidence": null,
          "reason": ""
        },
        {
          "case_id": "output-eval-expectation",
          "winner_variant": "",
          "confidence": null,
          "reason": ""
        },
        {
          "case_id": "ir-before-packaging",
          "winner_variant": "",
          "confidence": null,
          "reason": ""
        },
        {
          "case_id": "near-neighbor-boundary",
          "winner_variant": "",
          "confidence": null,
          "reason": ""
        },
        {
          "case_id": "file-backed-governed-package",
          "winner_variant": "",
          "confidence": null,
          "reason": ""
        }
      ]
    }