trycua--cua/libs/cua-driver/docs/e2e-ci-reporting.md

Rust E2E CI Reporting

Rust owns behavioral case declarations, observations, result classification, and report validation. OS runners build the environment, execute Rust targets, and upload the validated artifacts.

Canonical Invocation

The contributor-facing invocation runs the complete matrix on every OS and takes no suite selector:

Windows: scripts/ci/windows/run-rust-e2e.ps1 -RequireGui
Linux:   scripts/ci/linux/run-rust-e2e.sh
macOS:   scripts/ci/macos/run-rust-e2e.sh

Workflows set a private lane value to fan that matrix into shared, native, and capture jobs so one failure does not hide another lane. These are execution partitions, not public suite choices or separate behavioral sources of truth.

Execution Order

Each runner follows the same sequence:

1. Resolve one immutable source SHA and check out that SHA in every lane.
2. Build the Rust driver and required repo-local fixtures.
3. Run e2e_environment_preflight_test once; it verifies git rev-parse HEAD against the workflow's resolved SHA.
4. Abort before behavioral cells if the desktop, permissions, fixture, capture, or video lifecycle is unavailable.
5. Run the selected Rust integration targets.
6. Validate declarations, results, and evidence with cua-e2e-report.
7. Upload the report, logs, and recordings.

The preflight prevents a single TCC or desktop-session problem from appearing as the same failure on every behavioral cell.

Artifact Files

Each OS lane writes:

artifacts/cua-driver/<platform>/
|-- environment.jsonl
|-- cases.jsonl
|-- results.jsonl
|-- summary.md
|-- environment-preflight.log
|-- <rust-target>.log
`-- recordings/<cell-label>-pid<process>-<sequence>/
    |-- recording.mp4
    |-- trajectory.json
    |-- session.json
    |-- cursor.jsonl
    `-- turn-*/
        |-- action.json
        |-- evidence.json
        |-- before_state.json
        |-- before.png
        |-- after_state.json
        |-- after.png
        |-- app_state.json
        |-- screenshot.png
        `-- click.png (click-family turns)

cases.jsonl is the executed catalog. results.jsonl contains one result for every declared cell. In canonical mode, the reporter rejects duplicates, missing results, undeclared results, contradictory statuses, missing required videos, and missing or invalid expected turn evidence. An evidence.json classification normally makes capture failure auditable without making it pass. The one narrow exception is a successful Windows bring_to_front restoration: the before image may be target_minimized, and the after image may be capture_failed, only when the action reports success, after-state accessibility data was captured, and the required trajectory and finalized MP4 remain present. Any other missing image still fails closed.

The testkit prepares each cell's artifact directory before fixture setup but does not start recording.mp4 yet. On Windows GitHub-hosted runners it obtains the test process's inherited top-level console with GetConsoleWindow, verifies the HostedComputeAgent/runner title and console class, then minimizes it with ShowWindow(SW_MINIMIZE). Fixture readiness and the required foreground or sentinel-occluded posture are established after that cleanup. The cell then calls start_behavior_recording, re-establishes the sentinel posture after the capture backend attaches, and clears setup-only focus events. It dispatches the action only after this observation boundary, then keeps recording through oracle collection.

trajectory.json records the behavioral video phase as pending, started, or finalized, its start/baseline/finalization timestamps, and the hosted-runner console cleanup status. Strict reporting requires finalized. A setup or posture failure before the boundary remains in the test result and logs, and the testkit writes recording-error.txt; it does not create a misleading behavioral clip.

Environment Record

The preflight emits exactly one record:

{
  "schema": "cua-e2e-environment/v2",
  "platform": "macos",
  "display_server": "quartz",
  "source_sha": "0123456789abcdef0123456789abcdef01234567",
  "status": "ready",
  "duration_ms": 912,
  "message": ""
}

An error record produces an environment section in summary.md and aborts the lane before case declarations are executed. The same source SHA appears in the successful typed summary.

Case And Result Contract

A case declaration uses cua-e2e-case/v2. A result flattens the same contract fields into cua-e2e-result/v2 and adds the observation:

{
  "schema": "cua-e2e-result/v2",
  "cell_id": "windows-tauri-left-click-ax-background",
  "platform": "windows",
  "display_server": "win32",
  "harness": "tauri",
  "toolkit": "platform-webview",
  "action": "left_click",
  "targeting": "ax",
  "delivery": "background",
  "scope": "window",
  "driver_route": "uia_invoke",
  "expected_behavior": { "kind": "deliver" },
  "oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],
  "test_status": "pass",
  "observed_behavior": "delivered",
  "passed_oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],
  "duration_ms": 1482,
  "message": "",
  "evidence": {
    "video": "recordings/windows-tauri-left-click-ax-background-pid123-001/recording.mp4",
    "trajectory": "recordings/windows-tauri-left-click-ax-background-pid123-001/trajectory.json"
  }
}

targeting describes how the action identifies its target: ax, px, page, or not_applicable. It is separate from screenshot/tree capture.

Classification Rules

test_status answers whether the cell met its declared contract. observed_behavior records what the driver did.

Expected	Observed	Required oracles	Status
Deliver	Delivered	Passed	Pass
Deliver	Refused	Any	Fail
Deliver	No effect or error	Any	Fail
Refuse	Allowed structured refusal	Passed	Pass
Refuse	Different refusal code	Any	Fail
Refuse	Delivered	Any	Fail pending contract review

The structured refusal enum currently recognizes background_unavailable, background_occluded, and background_uipi_blocked. Each refusal case declares the exact subset allowed by its controlled setup. String-prefix or message-substring matching is not accepted.

A refusal contract also requires focus, z-order, and leaked-input observations. An honest refusal does not satisfy a case whose contract requires delivery.

Summary Ownership

cua-e2e-report is the only behavioral Markdown renderer. Shell and PowerShell runners must not parse test ... ok output or create host=cargo and host=lane rows. Cargo logs remain available for unit-test annotations and lane diagnostics, outside the behavioral population.

The summary reports delivered passes, refused passes, failures, and skips separately. Shared and native targets emit the same typed records; a compile or runner failure that prevents declaration remains visible as a failed job and cannot be mistaken for a green behavioral row.

Evidence Links

Every canonical behavioral cell records a validated MP4 and trajectory. The recording directory also contains its session metadata and turn-level action and screenshot files; Cargo target logs remain lane-level diagnostics. GitHub cannot deep-link to a file inside a multi-cell artifact archive, so each summary row displays the exact MP4 path as a link to the owning lane archive while the trajectory rollup provides bulk-download links and video counts.