trycua--cua/libs/cua-driver/docs/test-harnesses-guide.md

CUA Driver Test Harnesses Guide

This contributor guide explains where the CUA Driver tests live, how to run them, and where to add a new case. For current coverage and platform gaps, use test-matrix.md and action-support.md.

The Short Version

There are two main test layers:

1. Unit and protocol tests exercise Rust code and the public MCP/CLI contract without launching a real application.
2. Harness E2E tests launch a small application built from this repository, drive it through the Rust driver, and verify an external application or desktop state.

The Rust tests are the source of truth. Python tests, old shell runners, and historical recording scripts are not part of the canonical E2E path.

The canonical E2E command on every OS runs the complete matrix and takes no suite selector:

Linux:  scripts/ci/linux/run-rust-e2e.sh
Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui
macOS:  scripts/ci/macos/run-rust-e2e.sh

The OS workflow may fan the complete matrix out into independent jobs for reporting and failure isolation. That is an execution detail; contributors should think of it as one canonical suite.

Repository Map

cua/
|-- libs/cua-driver/
|   |-- rust/
|   |   |-- crates/
|   |   |   |-- cua-driver/          Rust driver and integration tests
|   |   |   |-- cua-driver-core/     Shared driver logic and unit tests
|   |   |   |-- cua-driver-testkit/  Shared Rust E2E helpers and evidence capture
|   |   |   |-- platform-linux/      Linux backend
|   |   |   |-- platform-macos/      macOS backend
|   |   |   |-- platform-windows/    Windows backend
|   |   |   `-- cursor-overlay/      Cursor evidence helper
|   |   |-- test-apps/               Ignored staged harness binaries
|   |   `-- Cargo.toml               Rust workspace
|   |-- tests/
|   |   |-- fixtures/
|   |   |   |-- shared/web/           Shared web page and external markers
|   |   |   |-- apps/                 Repo-local fixture sources
|   |   |   `-- build/                macOS/Linux/Windows fixture builders
|   |   `-- runners/                  Auxiliary VM and sandbox entrypoints
|   `-- docs/                         Test matrix, reporting, and contributor docs
`-- scripts/ci/
    |-- linux/run-rust-e2e.sh         Linux canonical runner
    |-- windows/run-rust-e2e.ps1      Windows canonical runner
    `-- macos/run-rust-e2e.sh         macOS canonical runner

The important separation is:

Layer	Owns	Does not own
Rust integration test	Scenarios, driver calls, assertions, action metadata	OS setup and fixture compilation
cua-driver-testkit	Session helpers, fixture launching, screenshots, recordings, trajectories	The scenario list
Fixture app	Visible controls and externally observable state markers	Driver correctness assertions
OS runner	Build environment, user session, test selection, artifact collection	Test behavior definitions

There is deliberately no second Python E2E implementation that the Rust suite has to mirror.

How One E2E Test Works

Every canonical E2E cell follows this shape:

OS runner
  -> builds the Rust driver and repo-local fixture
  -> starts or connects to a real desktop session
  -> Rust testkit starts one harness application
  -> Rust test discovers the target window
  -> get_window_state provides accessibility tree and screenshot
  -> action addresses a target by AX element index or PX coordinates
  -> delivery is foreground or background where the action supports it
  -> fixture state, focus, pixels, or protocol response is checked
  -> testkit writes video, screenshots, trajectory, logs, and result data

A tool returning ok is not enough to pass an E2E cell. The fixture must show that the action happened, or the test must verify a documented structured refusal and the absence of focus or input side effects.

The Test Layers

Unit and Protocol Tests

These run without a repo-local GUI application and normally run without --ignored:

Location or prefix	What it proves
rust/crates/*/src/**	Core driver, platform-independent logic, schemas, and helpers
protocol_*_test.rs	MCP handshake, tool calls, sessions, media, and errors
schema_*_test.rs	Shared schema and backend consistency
transport_config_persistence_test.rs	CLI/MCP configuration persistence
protocol_element_token_test.rs	Element-token protocol behavior

These tests should be fast, deterministic, and safe to run on ordinary CI workers. They do not prove that a real click, key, scroll, or background input reached an application.

Harness E2E Tests

These are Rust integration tests under:

libs/cua-driver/rust/crates/cua-driver/tests/

Most are marked #[ignore] because they require a desktop, built fixtures, and platform permissions. They are selected by the OS runner rather than the ordinary unit command.

The canonical E2E suite has two behavior owners:

Owner	Purpose
Shared app	Same web behavior tested through Electron and Tauri
Native harness	Toolkit-specific controls and native window behavior

WebView, CDP, and page-tool integration stays inside the shared or native owner that exercises it. It is not a third public test family or command.

Delivery is not a test family. It is a dimension on each action row: an action is tested in foreground and background modes whenever the driver and OS support both. Capture and desktop scope are separate environment checks, while focus preservation is a cross-cutting oracle that can be attached to any action row. Focus, z-order, cursor, and desktop-state checks are cross-cutting invariants, not a separate family. These names describe responsibilities, not separate sources of truth or commands to run instead of the selector-free canonical invocation.

What The Complete Run Includes

Windows

Runner: scripts/ci/windows/run-rust-e2e.ps1

Runner area	Rust test	Real harness or app
Shared app matrix	cross_platform_behavior_test.rs	Electron and Tauri
Native controls	harness_wpf_test.rs	Repo-local WPF app
Native controls	harness_winui3_test.rs	Repo-local WinUI3 app
Web integration	harness_web_test.rs	WebView2 and Electron
Capture contract	capture_contract_test.rs	WPF plus driver tree/image output
Launch contract	launch_windows_test.rs	Repo-local Electron launch and focus behavior
Agent cursor	agent_cursor_windows_test.rs	Source-built cursor overlay and pixel evidence
Desktop scope	desktop_scope_windows_test.rs	Windowless desktop input and scope rejection

Cross-cutting instrumentation used by these rows includes the testkit DesktopObserver, capture validation, cursor evidence, and desktop-scope checks. These invariants are attached to their owning action rows rather than run as a separate test family.

Windows is currently the broadest native matrix. It covers UIA controls, web integration, background focus checks, and Windows-specific input routes. The desktop observer is attached to shared and native action rows wherever background delivery is tested.

macOS

Runner: scripts/ci/macos/run-rust-e2e.sh

Runner area	Rust test	Real harness or app
Shared app matrix	cross_platform_behavior_test.rs	Electron and Tauri
Native web matrix	cross_platform_behavior_test.rs	Repo-local WKWebView host
Native controls	harness_appkit_test.rs	Repo-local AppKit app
Native controls	harness_swiftui_test.rs	Repo-local SwiftUI app
Installed app launch	installed_app_launch_macos_test.rs	Calculator and TextEdit
Installed app AX delivery	installed_app_textedit_macos_test.rs	TextEdit
Capture contract	capture_contract_test.rs	Installed driver and macOS capture APIs
Desktop scope	desktop_scope_macos_test.rs	macOS window and desktop scope

The WKWebView host runs the same typed shared-web catalog as Electron and Tauri. Calculator and TextEdit add typed supporting rows for built-in app launch focus and native Cocoa background value delivery. They run in the canonical logged-in macOS lane, but they do not replace repo-local fixtures.

Linux

Runner: scripts/ci/linux/run-rust-e2e.sh

Runner area	Rust test	Real harness or app
Shared app matrix	cross_platform_behavior_test.rs	Electron and Tauri
Native controls	harness_gtk3_test.rs	Repo-local GTK3 app
Capture contract	capture_contract_test.rs	Linux capture backend
Desktop scope	desktop_scope_linux_test.rs	X11/Wayland desktop scope

Linux has separate X11 and Wayland concerns. Nix supplies the reproducible build and desktop environment, but the E2E test still needs an actual X11 or Wayland session. Linux does not need GIF output; MP4, screenshots, accessibility trees, trajectories, and logs are the useful evidence.

Wayland results are compositor-specific. The hosted lane uses Sway to prove wlroots protocols. GNOME requires the optional WinRects Shell helper for authoritative frame and buffer geometry, observation, capture, and verified target activation. A portal/libei grant persists until the user revokes it, so subsequent driver processes do not reopen the consent dialog. KDE requires a future target-addressable KWin adapter; portal availability by itself is not evidence that input can be sent safely to a named window.

AX, PX, and Delivery

See action-support.md for the current Windows, macOS, and Linux delivery, refusal, and unproven-action ledger.

These terms describe different dimensions:

Term	Meaning
AX	Address a target through its accessibility/UI automation element
PX	Address a target by screen coordinates or pointer geometry
Foreground	The target may be brought to the foreground for delivery
Background	The target should receive the action without being raised or stealing focus
Window scope	Capture or action is limited to one target window
Desktop scope	Capture or action covers the full desktop

The shared and native action matrices should test left click, right click, double click, typing, keys, hotkeys, scroll, child windows, and drag across AX/PX and foreground/background combinations where the driver supports them. Unsupported background routes require an explicit refusal contract with an allowed structured code and desktop-side-effect oracles. A refusal fails a cell that requires delivery. There should not be a separate "delivery" family whose only purpose is to repeat those same actions in the background.

Native harness rows use the same typed case/result contract as the shared matrix. Current native set_value rows declare background delivery because their contract includes no-focus and no-raise observations; actions without a delivery concept use not_applicable explicitly.

Cross-Cutting Invariants

The desktop observer is cross-cutting test instrumentation. It answers the same question for any action, harness, or catalog area:

Did the driver perform or reject the operation without disturbing the user's foreground application or desktop?

cua-driver-testkit::DesktopObserver owns the shared interface. Native Windows, macOS, and Linux backends snapshot foreground-window, target z-order, cursor, and leaked-input state before and after an action. Background rows opt into the observer directly; there is no special guard suite.

Invariant or scenario	What it checks
Background click/type/key	The target action does not move focus away from the user's foreground window
Minimized app launch	launch_app(start_minimized=true) does not raise the new app
Background hotkey	A keyboard chord does not steal focus
Child-window click	A target-created window does not unexpectedly become foreground
Background screenshot	Reading the target does not change focus or z-order
Agent cursor visibility	The cursor appears in the captured pixels when enabled and moved

A focus assertion can prove "no focus steal" while failing to prove that a click changed the target application state. An action row must therefore check both the target's external state and, when background delivery is under test, the cross-cutting desktop observer.

These tests require a real interactive Windows user desktop. They reject Session 0, locked desktops, and disconnected RDP sessions. Without CUA_REQUIRE_GUI=1, an unusable desktop can self-skip for local development; the canonical Windows runner enables the hard-failure behavior.

Evidence

Canonical GUI runs are expected to produce evidence per test cell:

artifacts/cua-driver/<os>/
|-- recordings/<cell-label>-pid<pid>-<sequence>/recording.mp4
|-- recordings/<cell-label>-pid<pid>-<sequence>/trajectory.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before_state.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/before.png
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after_state.json
|-- recordings/<cell-label>-pid<pid>-<sequence>/turn-*/after.png
|-- cases.jsonl
|-- environment.jsonl
|-- results.jsonl
|-- summary.md
`-- <rust-target>.log

The GitHub Actions summary contains one row per meaningful behavioral cell, including its OS, harness, action, AX/PX targeting, delivery mode, driver route, expected and observed behavior, oracles, and one evidence link. The link uses the exact video path as its label and opens the owning lane archive. Unit tests need normal test output and logs; they do not need desktop video.

What Is Implemented Today

• Rust owns the canonical scenario definitions and external-state assertions.
• Electron and Tauri use the same shared web fixture across supported OSs.
• Native Windows, macOS, and Linux harnesses are repo-local applications built from source.
• The three OS runners use a selector-free command for the complete matrix.
• Shared and native harness owners emit the same typed v2 result records.
• Canonical GUI rows collect a trajectory and MP4, validated before reporting.
• Canonical GUI rows require parseable pre/post state and non-empty pre/post target-window images for each targeted turn. The one narrow exception is a successful Windows bring_to_front restore whose minimized target has no pre-action image or whose host capture remains unavailable afterward: it must retain the successful action response, captured post-action accessibility state, an explicit capture classification, trajectory, and MP4. Any other missing expected evidence fails the report.
• Per-cell video starts after fixture readiness and foreground/background posture. A 300 ms baseline precedes dispatch, and capture continues through external oracle collection. trajectory.json must finish with behavior_video.status = "finalized".
• Windows hosted runs use GetConsoleWindow to select the inherited HostedComputeAgent/runner console, verify its identity, and minimize it through ShowWindow(SW_MINIMIZE) before fixture or sentinel posture is established. The sentinel remains a separate test fixture and is reasserted after console cleanup.
• Strict lane preflights fail on missing fixtures, desktop access, permissions, accessibility, capture, or recording support instead of silently skipping.
• GitHub summaries link every evidence-bearing row to its lane archive and display the exact recording path. The trajectory path remains in the typed evidence and archive.
• Unit/protocol tests remain separate from interactive E2E tests.

What Still Needs Implementation

The remaining work is platform coverage and validation, not another test hierarchy:

1. Broaden native action rows. The shared web matrix covers every declared AX/PX and foreground/background cell. AppKit, SwiftUI, WPF, WinUI3, WebView2, and non-GTK3 Linux toolkits still have unproven native combinations listed in action-support.md.
2. Preserve exact-source validation. Every accepted platform run must record one immutable source SHA and retain the typed evidence contract.
3. Close representative-desktop gaps. Hosted Sway passes the complete Electron, Tauri, GTK3, capture, and desktop-scope catalogs. A real GNOME 46 session passes GTK3, capture, and desktop scope, but still needs the shared renderer catalog and portal-video parity. Plasma 6 still needs a verified KWin activation adapter and its first accepted behavioral lane. Issue #1922 tracks the grouped backend work.
4. Add representative toolkit surfaces. GTK4, Qt5/Qt6, VTE, VCL, and GL canvases remain optional real-app gaps; shared Electron/Tauri coverage does not substitute for those native stacks.
5. Flake cleanup. Replace remaining fixed native waits with external-state polling and add fixture reset tokens before reusing a harness process.

File Convergence Plan

The goal is not to put every assertion into one enormous test file. The goal is to give each behavior one clear owner and make cross-cutting evidence reusable.

Target Ownership

rust/crates/cua-driver-testkit/src/
`-- observer.rs                     Cross-OS desktop-side-effect interface

rust/crates/cua-driver/tests/
|-- cross_platform_behavior_test.rs Shared Electron/Tauri action matrix
|-- harness_wpf_test.rs             Windows WPF action rows
|-- harness_winui3_test.rs          Windows WinUI3 action rows
|-- harness_web_test.rs             WebView2/Electron page and CDP rows
|-- harness_appkit_test.rs          macOS AppKit action rows
|-- harness_swiftui_test.rs         macOS SwiftUI action rows
|-- harness_gtk3_test.rs            Linux GTK3 action rows
|-- capture_contract_test.rs        Tree and screenshot read contract
|-- desktop_scope_<os>_test.rs      Window/desktop scope invariants
`-- protocol_*_test.rs              Protocol and schema tests

The desktop observer is a helper, not a test family. An action row invokes it when the row is testing background delivery. The row then records both outcomes:

1. Did the target application state change, or did the driver return the documented structured refusal?
2. Did focus, z-order, cursor, and desktop state remain within the contract?

Focus, z-order, cursor, and input-leak assertions belong to the typed action rows that exercise them; launch, capture, cursor, and desktop-scope contracts retain their narrowly owned scenarios. The canonical runner is the only user-facing command; lane selectors are internal diagnostics.

Contributor Workflow

When adding a new scenario:

1. Add or update the repo-local fixture and its external state marker.
2. Add the Rust scenario under rust/crates/cua-driver/tests/.
3. Declare AX/PX addressing, foreground/background delivery, scope, and oracle.
4. Add the scenario to docs/test-matrix.md and this guide when it changes the cross-OS structure.
5. Update only the OS runner selection when the test is platform-specific.
6. Run the smallest Rust test locally, then run the OS command before calling the matrix complete.

The goal is one understandable Rust E2E model across platforms, with platform-specific harnesses where the OS genuinely differs.