trycua--cua/samples/driver/multi-cursor/README.md

Multi-cursor background computer-use demo — "National Records System"

A fleet of deliberately legacy-looking government records terminals (navy banner, UNCLASSIFIED // FOR OFFICIAL USE ONLY strip, function-key bar, green-screen records grid, status line) — the kind of internal agency app that, in the age of AI, has no automation integration. cua-driver automates them anyway.

One human action in the foreground "master" terminal is replayed onto four background terminals at the same time, each driven by its own cua-driver session = its own uniquely-coloured agent cursor — with no window ever raised and the user's mouse never moved.

It also proves cua-driver works with or without an accessibility tree: the five windows span five UI frameworks, and cua-driver's default dispatch auto-selects UIA-Invoke where an a11y tree exists and falls back to pixel/pointer-injection where it doesn't.

Layout (each window = ½ work-width × ½ work-height)

The four corners tile the taskbar-safe work area into quadrants; the master is centered, overlapping all four:

 ┌────────────────────────┬────────────────────────┐
 │ Win32 GDI (NO a11y)     │ WinForms (.NET)         │
 │            crimson ●    │            amber ●      │
 │           ┌────────────────────────┐             │
 │           │ MASTER — Win32 controls │  ← you      │
 ├───────────│ (foreground, overlaps)  │─────────────┤
 │ WPF (XAML)│                         │ Electron    │
 │           └────────────────────────┘ mint_lime ●  │
 │            aqua ●       │            (Chromium)    │
 └────────────────────────┴────────────────────────┘

Click SUBMIT (or type a subject name then submit) in the center master: four coloured cursors glide onto the four corner terminals and commit the same record there — concurrently, in the background. Watch each corner's green-screen records grid grow and its RECORDS: counter tick up, without any corner ever coming to the front.

Frameworks (and what they exercise)

Window	Framework	Accessibility	cua-driver path
TL	Win32 + GDI (custom-drawn)	none	pixel hit-test → PostMessage / pointer injection
TR	.NET WinForms	MSAA/UIA	UIA Invoke
BL	.NET WPF	UIA (XAML)	UIA Invoke (no foreground steal via WS_EX_NOACTIVATE)
BR	Electron	UIA (Chromium)	UIA Invoke
Center	Win32 standard controls	MSAA	(foreground; the human drives it)

Build

# from this directory
cargo build                                   # legacy-app + orchestrator (Rust)
dotnet build dotnet/winforms/winforms.csproj  # WinForms
dotnet build dotnet/wpf/wpf.csproj            # WPF
npm install --prefix electron                 # Electron (downloads electron once)

Also build the driver once (repo root workspace):

cargo build -p cua-driver --manifest-path ..\..\libs\cua-driver\rust\Cargo.toml

Run

.\target\debug\orchestrator.exe            # human-driven: click/type in the center
.\target\debug\orchestrator.exe --auto     # self-playing: drives a TYPE+CLICK every few seconds

The orchestrator starts the cua-driver daemon, launches + positions all five windows, and fans every center action out to the four corners over four concurrent cua-driver call sessions (crimson / amber / aqua / mint_lime → four cursor colours). Close the center window (or kill the orchestrator) to tear everything down — a Windows Job Object kills the whole tree, so nothing is orphaned.

Env overrides

CUA_DRIVER_EXE, LEGACY_APP_EXE, WINFORMS_EXE, WPF_EXE, ELECTRON_DIR.

How the coloured cursors work

cua-driver assigns each session a cursor colour by name (palette-name sessions like crimson pick that colour directly). Passing "session":"<color>" on each click/type_text call routes it to that session's overlay cursor, which glides to the target. Four sessions → four cursors animating at once. See docs/windows-background-input-re-plan.md for the no-z-raise mechanism.