chore: import upstream snapshot with attribution

plugins/plugin-computeruse/docs/ANDROID_CONSTRAINTS.md

@@ -0,0 +1,215 @@
+# Android computer-use — constraints, capabilities, and validation checklist
+
+## What works in a consumer APK
+
+| Capability | Permission | Notes |
+|---|---|---|
+| AccessibilityService — view tree + gesture dispatch | `BIND_ACCESSIBILITY_SERVICE` (user must enable in Settings) | Survives Advanced Protection Mode in most OEM ROMs |
+| MediaProjection — screen frame capture | `FOREGROUND_SERVICE_MEDIA_PROJECTION` + user consent dialog | 1 Hz default; foreground service required |
+| UsageStatsManager — app history + foreground app | `PACKAGE_USAGE_STATS` (user must enable in Settings > Usage Access, no runtime prompt) | 24-hour window; 5-min scan for foreground app |
+| Camera2 — JPEG/RGBA frames | `CAMERA` (runtime permission) | Service-friendly; no Activity or SurfaceView needed |
+| onTrimMemory → MemoryArbiter pressure | None (ComponentCallbacks2) | Fires at TRIM_MEMORY_RUNNING_LOW and TRIM_MEMORY_RUNNING_CRITICAL |
+
+## Pixel/Google Android assistant entry points
+
+For normal Google Android and Play distribution, Eliza uses Google App
+Actions and Android shortcuts only:
+
+- `packages/app-core/platforms/android/app/src/main/res/xml/shortcuts.xml`
+  declares `actions.intent.OPEN_APP_FEATURE`, `CREATE_MESSAGE`,
+  and `GET_THING`.
+- `OPEN_APP_FEATURE` is the static feature surface for chat/ask, voice,
+  LifeOps daily brief, LifeOps task creation, and LifeOps tasks.
+- `CREATE_MESSAGE` and `GET_THING` route free-form ask/chat text to the
+  app's chat deep link.
+- Static shortcuts use source-tagged deep links and are bound to
+  `OPEN_APP_FEATURE` inline inventory.
+- Each App Actions capability keeps a fallback fulfillment without a
+  required parameter. The build validator rejects regressions because
+  Assistant may invoke vague feature/search/message requests.
+Unsupported BIIs such as `actions.intent.CREATE_THING` are intentionally
+not declared; LifeOps task creation is a feature-open flow and any
+mutation must still go through runtime confirmation and the ScheduledTask
+path.
+
+Mapping by flow:
+
+| Flow | Supported App Actions / shortcut entry |
+|---|---|
+| Ask/chat | `CREATE_MESSAGE`, `GET_THING`, `eliza_app_action_chat` |
+| Voice chat | `OPEN_APP_FEATURE` inline inventory, `eliza_app_action_voice` |
+| LifeOps daily brief | `OPEN_APP_FEATURE` inline inventory, `eliza_app_action_daily_brief` |
+| Create LifeOps task | `OPEN_APP_FEATURE` inline inventory, `eliza_app_action_new_task` |
+| View LifeOps tasks | `OPEN_APP_FEATURE` inline inventory, `eliza_app_action_tasks` |
+
+There is no Play-compatible default-assistant handoff for the Pixel build.
+Normal Google Android entry goes through App Actions/static shortcuts/deep
+links. AOSP-only `ROLE_ASSISTANT` and `ACTION_ASSIST` behavior stays out of
+the Play build.
+
+The Play-compatible `android-cloud` build must not request or expose
+default-assistant/system-only powers. Keep `ACTION_ASSIST`,
+`VOICE_COMMAND`, `ROLE_ASSISTANT`, `BIND_VOICE_INTERACTION`,
+usage-stats appop permissions, SMS/call default-role components, boot
+receivers, battery-optimization exemption, MediaProjection foreground
+services, and special-use foreground services out of that build.
+AOSP/default-assistant behavior belongs only to `android-system` or
+sideload-only validation builds.
+
+Current Android docs describe App Actions as `shortcuts.xml`
+capabilities registered on the launcher activity; Gemini/Assistant
+interoperability for general apps is through those App Actions and
+shortcuts. The navigation-app Gemini/Assistant intent formats are
+navigation-specific and are not a general personal-assistant integration
+surface for this app.
+
+## What requires a system-app build (AOSP flavor)
+
+| Capability | Mechanism | Permission |
+|---|---|---|
+| High-fidelity screen capture | `SurfaceControl.captureDisplay()` | `READ_FRAME_BUFFER` (`signature|privileged`) |
+| High-fidelity input injection | `InputManager.injectInputEvent()` | `INJECT_EVENTS` (`signature|privileged`) |
+| Full process enumeration | `IActivityManager.getRunningAppProcesses()` via AIDL | `REAL_GET_TASKS` (`signature`) |
+
+See `AOSP_SYSTEM_APP.md` for the privileged build path.
+
+## Advanced Protection Mode caveat
+
+When Advanced Protection Mode (APM) is active on Pixel 9+ and some OEM variants:
+
+- AccessibilityService registered with `featureAccessibility` (not `featureGeneric`) survives.
+  `ElizaAccessibilityService` is registered correctly.
+- Third-party AccessibilityServices with broad event masks may be killed on APM devices
+  even when re-enabled in Settings. If the service is repeatedly stopped, check `adb logcat`
+  for `AccessibilityManagerService` or `android.safetycenter` entries.
+
+## lmkd survival strategy
+
+The Linux low-memory killer daemon (lmkd) uses oom_score_adj to prioritize kills.
+Two mitigations are active:
+
+1. `ScreenCaptureService` is a foreground service — lmkd ranks foreground services
+   below cached apps; they survive until memory is critically exhausted.
+2. `onTrimMemory` → `capacitorPressureSource.dispatch()` — WS1 MemoryArbiter receives
+   the pressure signal and proactively unloads lower-priority model handles
+   (transcribe, vision-describe) before the OOM killer fires.
+
+## Manual on-device validation checklist
+
+Run this against a physical Android device (API 24+ for gesture dispatch; API 29+ for
+`FOREGROUND_SERVICE_MEDIA_PROJECTION`). Cuttlefish x86_64 emulator is acceptable for
+smoke-testing, but the x86_64 JNI patch must be present (see WS4 llama-cpp-capacitor patch).
+
+### 1. Permissions setup
+
+- [ ] Install the APK and open the app.
+- [ ] Grant `CAMERA` runtime permission when prompted.
+- [ ] Navigate to Settings > Accessibility > Eliza > enable the service.
+  Verify `ElizaAccessibilityService.instance` is non-null via:
+  `adb shell dumpsys accessibility | grep -i eliza`
+- [ ] Navigate to Settings > Digital Wellbeing (or Settings > Security > Usage Access)
+  > Eliza > enable Usage Access.
+
+### 2. AccessibilityService — view tree
+
+```
+adb shell am start com.example.app   # open any app
+curl -X POST http://localhost:1337/api/computer-use/getAccessibilityTree
+```
+Expected: JSON array with `[{id, role, label, bbox, actions}]` entries.
+Verify `role` values are Android class names (e.g. `android.widget.Button`).
+
+### 3. Gesture dispatch
+
+```
+curl -X POST http://localhost:1337/api/computer-use/dispatchGesture \
+  -d '{"type":"tap","x":540,"y":960}'
+```
+Expected: `{"ok":true}` and the tap is visible on screen.
+
+```
+curl -X POST http://localhost:1337/api/computer-use/dispatchGesture \
+  -d '{"type":"swipe","x":540,"y":1600,"x2":540,"y2":400,"durationMs":400}'
+```
+Expected: `{"ok":true}` and the list scrolls up.
+
+### 4. Global actions
+
+```
+curl -X POST http://localhost:1337/api/computer-use/performGlobalAction -d '{"action":"home"}'
+curl -X POST http://localhost:1337/api/computer-use/performGlobalAction -d '{"action":"recents"}'
+curl -X POST http://localhost:1337/api/computer-use/performGlobalAction -d '{"action":"back"}'
+curl -X POST http://localhost:1337/api/computer-use/performGlobalAction -d '{"action":"notifications"}'
+```
+Expected: each action is visually confirmed on device.
+
+### 5. MediaProjection — screen capture
+
+```
+curl -X POST http://localhost:1337/api/computer-use/startMediaProjection -d '{"fps":1}'
+```
+Expected: system consent dialog appears. Accept it.
+
+```
+curl -X GET http://localhost:1337/api/computer-use/captureFrame
+```
+Expected: `{ok:true, data:{jpegBase64:"...", width:..., height:..., timestampMs:...}}`.
+Verify `jpegBase64` decodes to a valid JPEG of the current screen.
+
+```
+curl -X POST http://localhost:1337/api/computer-use/stopMediaProjection
+```
+Expected: `{ok:true, data:{stopped:true}}`.
+
+### 6. UsageStats — app enumeration
+
+```
+curl -X GET http://localhost:1337/api/computer-use/enumerateApps
+```
+Expected: JSON array of `{packageName, label, lastUsedMs, totalForegroundMs, isForeground}`.
+Verify `isForeground:true` for the frontmost app.
+If you receive `{ok:false, code:"permission_denied"}`, confirm Usage Access is enabled.
+
+### 7. Camera capture
+
+```
+curl -X POST http://localhost:1337/api/computer-use/startCamera -d '{"fps":1}'
+```
+Expected: `{ok:true, data:{cameras:"[...]"}}` with at least one camera entry.
+
+```
+curl -X GET http://localhost:1337/api/computer-use/captureFrameCamera
+```
+Expected: `{ok:true, data:{jpegBase64:"..."}}`.
+
+```
+curl -X POST http://localhost:1337/api/computer-use/stopCamera
+```
+
+### 8. Memory pressure dispatch
+
+Open a memory-intensive app or use `adb shell am send-trim-memory $(pidof ai.eliza.eliza) 80`
+to simulate TRIM_MEMORY_RUNNING_CRITICAL.
+
+Expected: the JS console (or logcat for bridge events) shows:
+`[capacitorPressureSource] dispatching pressure: critical`
+followed by MemoryArbiter eviction log entries.
+
+Verify via `GET /api/training/auto/config` that arbiter pressure state transitions to `critical`.
+
+### 9. App Actions / static shortcuts
+
+Use a Play/Assistant-capable Pixel or Google Android device signed into
+the same account used by the App Actions test tool.
+
+- [ ] Confirm the launcher activity registers `@xml/shortcuts`:
+  `aapt dump xmltree app-release.aab base/manifest/AndroidManifest.xml`.
+- [ ] Confirm `shortcuts.xml` contains `OPEN_APP_FEATURE`,
+  `CREATE_MESSAGE`, and `GET_THING`, with no unsupported
+  `actions.intent.CREATE_THING`.
+- [ ] Confirm generated shortcuts use the app package and URL scheme for
+  the current brand; no `ai.elizaos.app`, `app.eliza`, or stale `eliza://`
+  value should remain after rewriting.
+- [ ] Trigger chat/ask, voice, new task, daily brief, and tasks from the Assistant
+  preview. Expected: Eliza opens via a source-tagged deep link and any
+  LifeOps mutation goes through the app/runtime `ScheduledTask` path.

plugins/plugin-computeruse/docs/AOSP_SYSTEM_APP.md

@@ -0,0 +1,112 @@
+# AOSP system-app deployment (computer-use on the Eliza AOSP fork)
+
+> Status: **design + contract**. The desktop CUA surface (capture, input, scene,
+> Brain→Cascade→dispatch) is production; the AOSP path is partially built —
+> `createAospInputActor()` returns `null` in consumer builds today
+> (`src/actor/aosp-input-actor.ts`), and the privileged bridge below is the
+> remaining work. This doc is the contract that wiring must satisfy. See
+> [`ANDROID_CONSTRAINTS.md`](./ANDROID_CONSTRAINTS.md) for the honest scope on
+> stock (non-system) Android, and `docs/android-aosp-validation.json` for the
+> release evidence manifest this path is gated on.
+
+## Why a system app
+
+On stock Android an app cannot freely read the framebuffer of *other* apps or
+inject input into them — those are `signature|privileged` permissions. The Eliza
+AOSP fork ships the agent as a **platform-signed system app** (or a privileged
+companion service), which unlocks the two capabilities a computer-use agent
+needs:
+
+| Capability | Stock Android | AOSP system app |
+|---|---|---|
+| Full-screen capture | `MediaProjection` (per-session user consent, own-app-biased) | `SurfaceControl.captureDisplay` / `READ_FRAME_BUFFER` (no prompt, any display) |
+| Global input injection | none (only own UI / a11y gestures) | `InputManager.injectInputEvent` with `INJECT_EVENTS` |
+| Element grounding | `AccessibilityService` tree (opt-in) | `AccessibilityService` tree + OCR, always available |
+
+## Manifest / signing
+
+```xml
+<!-- Platform-signed; placed on the system image (priv-app). -->
+<uses-permission android:name="android.permission.READ_FRAME_BUFFER" />
+<uses-permission android:name="android.permission.INJECT_EVENTS" />
+<uses-permission android:name="android.permission.ACCESS_SURFACE_FLINGER" />
+<application android:sharedUserId="android.uid.system" ... />
+```
+
+- Sign with the platform key; install under `/system/priv-app` (or
+  `/product/priv-app`) with a matching `privapp-permissions` allowlist entry for
+  the two signature permissions.
+- Non-system builds **must** degrade to the `MediaProjection` + `AccessibilityService`
+  path (see ANDROID_CONSTRAINTS.md). The plugin feature-detects at runtime and
+  never assumes the privileged path.
+
+## Capture — `ScreenState` source
+
+The AOSP capture feeds the same per-display frame contract the desktop scene
+uses (`DisplayCapture { display, frame: PNG }`, see `src/platform/capture.ts`):
+
+- **Privileged**: `SurfaceControl.captureDisplay(DisplayCaptureArgs)` →
+  `HardwareBuffer` → PNG. No user prompt; supports secondary/virtual displays.
+- **Fallback**: `MediaProjection` + `ImageReader` (one-time user consent), wired
+  in `src/mobile/android-scene.ts` / `mobile-screen-capture.ts`.
+
+Both paths populate the Android scene (`android-scene.ts`) with display bounds so
+multi-display coordinate translation (`src/platform/coords.ts`) works unchanged.
+
+## Input — `AospPrivilegedInputBridge`
+
+`src/actor/aosp-input-actor.ts` is the seam. `createAospInputActor()` returns
+`null` today; the privileged bridge must implement the `ComputerInterface` verbs
+via `InputManager.injectInputEvent`:
+
+| Verb | Implementation |
+|---|---|
+| `click` / `double_click` / `long_press` | `MotionEvent` ACTION_DOWN→UP (→DOWN→UP) at (x,y), `INJECT_INPUT_EVENT_MODE_ASYNC` |
+| `drag` | ACTION_DOWN → interpolated ACTION_MOVE waypoints → ACTION_UP (mirror the desktop manual-drag in `nut-driver.ts`) |
+| `scroll` | per-notch ACTION_SCROLL or MOVE deltas (mirror the desktop per-notch split) |
+| `type` | `KeyEvent` stream or `InputConnection` commit |
+| `key` / `key_combo` | `KeyEvent` with meta-state |
+
+Coordinates are **logical display pixels** (same contract the empirical Windows
+probe established for nutjs); the bridge applies the display's density only if
+`injectInputEvent` is found to operate in physical pixels on the target ROM —
+verify empirically before adding any scale, exactly as documented for the
+desktop driver.
+
+## Grounding — Brain-less, Actor-only (cheap by construction)
+
+There is **no mobile Brain** (no full-frame VLM reasoning loop on device). AOSP
+uses the **Actor-only** path: `OcrCoordinateGroundingActor` resolves a target by
+matching `Scene.ocr` (OCR boxes) and `Scene.ax` (`AccessibilityNodeInfo`
+clickables) — exactly the structured readout `GET_SCREEN` produces (#9105 M2).
+This means AOSP gets computer-use **without a remote VLM**:
+
+1. `android-scene.ts` builds `Scene` from MediaProjection/SurfaceControl capture
+   + the `AccessibilityNodeInfo` tree + OCR.
+2. OCR provider: register **PaddleOCR PP-OCRv5 mobile (Paddle-Lite)** as the
+   AOSP `CoordOcrProvider` via `registerCoordOcrProvider` (the same seam
+   plugin-vision uses on desktop, #9105 M1) — on-device, ARM-optimized, zero LLM
+   tokens.
+3. `Cascade` grounds via `resolveReference` (OCR/AX text-match) with the
+   per-Scene grounding cache (#9105 M5); `dispatch` routes the concrete coords to
+   the `AospPrivilegedInputBridge`.
+
+## Remaining work (tracked in #9105 M6)
+
+- [ ] Implement `AospPrivilegedInputBridge` and return it from
+      `createAospInputActor()` when the privileged permissions are held.
+- [ ] `SurfaceControl.captureDisplay` capture path + `MediaProjection` fallback
+      feeding `android-scene.ts`.
+- [ ] Register Paddle-Lite PP-OCRv5 as the AOSP `CoordOcrProvider`.
+- [ ] Empirically verify `injectInputEvent` coordinate space (logical vs
+      physical) on the target ROM before any density scaling.
+- [ ] Complete `docs/android-aosp-validation.json` evidence (capture + a click
+      landed via injection + an OCR-grounded tap) and gate the release on it.
+
+## Security
+
+All destructive actions still pass through `ComputerUseApprovalManager`
+(`smart_approve` by default). A platform-signed agent with `INJECT_EVENTS` is a
+high-privilege component — keep approval gating on, scope the priv-app allowlist
+to exactly the two signature permissions, and never expose the injection bridge
+over an unauthenticated IPC surface.

plugins/plugin-computeruse/docs/IOS_CONSTRAINTS.md

@@ -0,0 +1,233 @@
+# iOS Computer-Use — Honest Scope
+
+Apple does not let third-party apps drive other apps. There is no equivalent
+to Android's `MediaProjection` for full-screen capture without a user prompt,
+no `ScreenCaptureKit` (that is macOS only), and no system-wide accessibility
+API that a sandboxed iOS app can use to inject events into other apps.
+
+This document spells out what *is* possible on iOS, exactly how each surface
+behaves, and the manual validation checklist that has to be run on a real
+device before any of this ships.
+
+## What works
+
+### 1. ReplayKit foreground capture (own app)
+
+`RPScreenRecorder.shared().startCapture(handler:completionHandler:)` returns
+`CMSampleBuffer`s of *the host app's* window. Useful for "show me what's on
+the Eliza screen right now."
+
+```swift
+RPScreenRecorder.shared().startCapture(handler: { sampleBuffer, type, error in
+    // Receive frames for own-app capture only.
+}, completionHandler: { error in
+    // Setup result; ReplayKit shows a system permission prompt on first use.
+})
+```
+
+Constraints:
+
+- Frame delivery rate is the display refresh rate; the bridge throttles to a
+  caller-supplied `frameRate` (default 1Hz).
+- Hard cap of 30 seconds per session. Past that, the work has to move to a
+  `BGProcessingTask`.
+
+### 2. ReplayKit broadcast extension (system-wide capture)
+
+A separate target with a ~50MB memory ceiling. Streams frames over an App
+Group shared container into the main app. The user must start the broadcast
+themselves via the system share-sheet picker — apps cannot programmatically
+launch the extension.
+
+```swift
+override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer,
+                                  with sampleBufferType: RPSampleBufferType) {
+    // Compress + dump into the App Group container.
+}
+```
+
+**iOS 26 / 26.1 beta regression**: extensions are killed within ~3 seconds
+even when memory headroom is fine. We surface this as
+`extension_died` from `broadcastExtensionHandshake` so callers can fall back
+to foreground capture. Track Apple's feedback status before shipping this
+target on iOS 26.
+
+### 3. Apple Vision OCR
+
+`VNRecognizeTextRequest` is on-device, free, supports ~30 languages, and
+typically runs sub-300ms on modern devices.
+
+```swift
+let request = VNRecognizeTextRequest { request, error in
+    let observations = request.results as? [VNRecognizedTextObservation]
+    // Top candidate per observation; each has bounding box, text, confidence.
+}
+request.recognitionLevel = .accurate
+request.recognitionLanguages = ["en-US"]
+try VNImageRequestHandler(cgImage: cg).perform([request])
+```
+
+This is the OCR provider the WS6 scene-builder will pick up on iOS. The
+provider interface is in
+`eliza/plugins/plugin-computeruse/src/mobile/ocr-provider.ts`.
+
+### 4. App Intents
+
+The only sanctioned way to drive other apps. Each target app must expose
+intents (Shortcuts-style). We support invocation via x-callback URL schemes
+for the system apps in the static registry
+(`ios-app-intent-registry.ts`):
+
+- Mail — `mailto:` with `subject` / `body` / `cc` / `bcc`
+- Messages — `sms:` with `body`
+- Maps — `http://maps.apple.com/?daddr=...&dirflg=...`
+- Safari — open URL
+
+For richer intents (Notes append, Reminders add, Music play with a query)
+the user has to donate the action via Shortcuts; we can then invoke via
+`AppIntent` on iOS 16+. The bridge's `appIntentList` returns the runtime
+list of donated intents this app sees.
+
+### 5. UIAccessibility (own-app reading only)
+
+`accessibilitySnapshot` walks the key window's view hierarchy and returns
+`accessibilityLabel` / `accessibilityValue` / role. iOS gives us no way to
+read another app's UIAccessibility tree.
+
+### 6. Apple Foundation Models (iOS 26+)
+
+Apple ships an on-device LLM under the `FoundationModels` framework when
+Apple Intelligence is enabled. We expose this as an *opportunistic*
+fast-path. If unavailable, the existing llama-cpp-capacitor local Eliza-1
+vision path stays as the default.
+
+Entitlement and Info.plist updates are listed below.
+
+## What does NOT work
+
+Stock iOS does not allow any of the following from a third-party app, and
+we do not pretend otherwise:
+
+- **Driving other apps' UI**. No cross-app input synthesis, no
+  `MediaProjection` / `ScreenCaptureKit` equivalent, no system-wide
+  accessibility event injection.
+- **Listing other running apps' processes**. The kernel hides this from
+  third-party apps; there is no `ps`-equivalent.
+- **Persistent background inference past ~30s**, except via
+  `BGProcessingTask` (opportunistic, OS-scheduled, not guaranteed).
+
+If a feature spec calls for any of the above, escalate it as not feasible
+on iOS rather than inventing a workaround that will get the app rejected
+or silently broken by Apple in the next OS update.
+
+## Entitlements + Info.plist
+
+Add the following to `apps/app/ios/App/App/App.entitlements`:
+
+```xml
+<key>com.apple.developer.kernel.increased-memory-limit</key>
+<true/>
+<key>com.apple.developer.kernel.extended-virtual-addressing</key>
+<true/>
+```
+
+Add the following to `apps/app/ios/App/App/Info.plist`:
+
+```xml
+<key>NSScreenCaptureDescription</key>
+<string>Captures the Eliza app window when you ask it to see the screen.</string>
+<key>NSAppleEventsUsageDescription</key>
+<string>Sends Shortcuts to other apps when you authorize an action.</string>
+```
+
+The screen capture string is shown on ReplayKit's first-launch system
+prompt. The Apple Events string is shown only if the host (this app) ever
+needs to drive AppleScript via the Catalyst surface; iOS apps that only
+use `UIApplication.shared.open(url:)` to invoke x-callback intents do not
+need it, but it costs nothing to include for forward compatibility with
+Mac Catalyst builds.
+
+## Validation checklist
+
+The iOS bridge contract is covered by TypeScript tests, but physical iOS
+device behavior is not proven by this repository alone. The required evidence
+manifest is `docs/ios-device-validation.json`; keep it in
+`requires_device_evidence` until a real device run records device/build
+metadata, artifacts, and per-method results. Release gates that require
+physical proof should run:
+
+```bash
+bun run --cwd plugins/plugin-computeruse validate:ios-device-evidence -- --require-complete
+```
+
+Without `--require-complete`, the same command validates that the manifest
+still tracks every required method and evidence field. Before shipping any
+iOS bridge release, complete the manifest for:
+
+### Simulator vs device
+
+| Surface                          | Simulator | Real device |
+| -------------------------------- | --------- | ----------- |
+| ReplayKit foreground capture     | Partial   | Required    |
+| Broadcast extension              | No        | Required    |
+| Vision OCR                       | Yes       | Required    |
+| App Intents (x-callback)         | Partial   | Required    |
+| Accessibility snapshot           | Yes       | Required    |
+| Foundation Models                | No        | Required    |
+| Memory pressure probe            | Limited   | Required    |
+
+### Per-method checklist
+
+For each method below, run on at least one A14-or-later iPhone running the
+target iOS (currently iOS 26.1, with iOS 17.6 as the floor):
+
+1. **`probe()`** — assert `data.platform === "ios"`, `osVersion` matches the
+   device, all six capability bits are present.
+2. **`replayKitForegroundStart`** — assert the system permission prompt is
+   shown on first use; on accept, `replayKitForegroundDrain` returns frames
+   with non-empty `jpegBase64` and the device's screen resolution.
+3. **`broadcastExtensionHandshake`** — bundle the extension target via Xcode,
+   tap the share-sheet broadcast picker, and assert `broadcastActive`
+   transitions to `true`. On iOS 26 betas, verify `regression.observed`
+   eventually flips to `true` after the extension dies.
+4. **`visionOcr`** — render a known PNG with the string `"WS9 OCR SMOKE"`
+   into the app, run OCR with `recognitionLevel: "accurate"`, and assert
+   `fullText` contains the source string (case-insensitive).
+5. **`appIntentList`** — verify the returned list is empty for a fresh
+   install and grows after the user donates intents via Shortcuts.
+6. **`appIntentInvoke`** with `com.apple.mobilesafari.open-url` — assert
+   Safari opens to the provided URL. With `com.apple.MobileSMS.send-message`
+   — assert Messages opens with the recipient and body pre-filled.
+7. **`accessibilitySnapshot`** — assert the returned tree's top-level node
+   has `role !== "labeled"` and `children.length > 0` on a populated screen.
+8. **`foundationModelGenerate`** — on a device with Apple Intelligence
+   enabled, assert a short prompt returns non-empty text. With AI disabled,
+   assert the call resolves with `foundation_model_unavailable`.
+9. **`memoryPressureProbe`** — invoke `UIApplication.shared.performMemoryWarning()`
+   (debug-only) and assert the next probe call returns
+   `severity >= 0.7` with `lastWarningAt` set.
+
+## Hand-off to other workstreams
+
+- **WS1 (memory-pressure arbiter)** consumes `MemoryPressureSample` via the
+  shared `IPressureSignal` contract in `ios-bridge.ts`. The bridge is the
+  producer; the arbiter is the consumer. WS1 will subscribe to push events
+  via the bridge once that channel lands; for now it polls `memoryPressureProbe`.
+- **WS6 (scene-builder OCR)** picks up the iOS Vision provider through
+  `selectOcrProvider()` in `ocr-provider.ts`. Register the provider at app
+  boot when running on iOS:
+  ```ts
+  registerOcrProvider(
+    createIosVisionOcrProvider(() => Capacitor.Plugins.ComputerUse),
+  );
+  ```
+
+## Apple documentation references
+
+- ReplayKit — https://developer.apple.com/documentation/replaykit
+- Vision Text Recognition — https://developer.apple.com/documentation/vision/recognizing_text_in_images
+- App Intents — https://developer.apple.com/documentation/appintents
+- Foundation Models — https://developer.apple.com/documentation/foundationmodels (iOS 26)
+- App Groups — https://developer.apple.com/documentation/xcode/configuring-app-groups
+- `os_proc_available_memory()` — https://developer.apple.com/documentation/foundation/process_info/3743117-os_proc_available_memory
+- Background Tasks — https://developer.apple.com/documentation/backgroundtasks

plugins/plugin-computeruse/docs/MOBILE_ASSISTANT_ROUTING.md

@@ -0,0 +1,174 @@
+# Siri, App Actions, and LifeOps Routing
+
+Assistant entry surfaces are launch and capture surfaces. They are not a
+second task engine.
+
+Siri/App Shortcuts, Android App Actions, Android assistant-role intents, and
+desktop shortcut surfaces should route the user's utterance or tap into the
+Eliza app/runtime. If the user asks for a reminder, check-in, follow-up,
+watcher, recap, or approval, the runtime must create or update a LifeOps
+`ScheduledTask`. Native mobile code may deliver notifications for tasks that
+already exist, but it must not create native-only reminder state that bypasses
+LifeOps.
+
+## Intended Flow
+
+1. The OS entry surface opens Eliza with a source-tagged deep link or assistant
+   intent payload.
+2. The app/runtime normalizes the payload into the same planner path used by
+   chat and device-bus requests.
+3. LifeOps creates or updates a `ScheduledTask` through the app/runtime.
+4. Native notification APIs only deliver output for that task or wake the
+   runtime so the scheduled-task runner can decide what fires.
+
+## Platform Contracts
+
+| Platform | Entry surface | Required routing contract |
+|---|---|---|
+| Mac | Shortcuts/deep link/menu-bar voice entry | Opens the app/runtime; LifeOps mutations are persisted as `ScheduledTask` records. |
+| iOS | Siri/App Shortcuts/App Intents | Uses App Intents or app deep links to hand off to the app/runtime. No cross-app UI driving, and no native-only reminder store. |
+| Android consumer | Static shortcuts + App Actions | `shortcuts.xml` and assistant intents open the app custom scheme; LifeOps scheduling happens after runtime routing. |
+| Android AOSP | `ROLE_ASSISTANT` + privileged system app | Assistant role may wake Eliza more directly; AOSP-only accessibility/notification-listener services are available, but scheduled behavior still goes through the same `ScheduledTask` runner. |
+
+Android consumer App Actions coverage:
+
+| Flow | BII/static shortcut |
+|---|---|
+| Ask/chat | `actions.intent.CREATE_MESSAGE`, `actions.intent.GET_THING`, `eliza_app_action_chat` |
+| Voice chat | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_voice` |
+| LifeOps daily brief | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_daily_brief` |
+| LifeOps task creation | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_new_task` |
+| LifeOps task list | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_tasks` |
+
+Each Android App Actions capability keeps a fallback fulfillment without a
+required parameter. Feature-specific requests route through
+`elizaos://feature/open?...`; vague requests fall back to the chat route so
+the runtime can disambiguate instead of executing native-only behavior.
+
+The Play/Pixel build does not request `ROLE_ASSISTANT`,
+`ACTION_ASSIST`, `VOICE_COMMAND`, or `BIND_VOICE_INTERACTION`; those are
+reserved for AOSP/system validation builds. It also avoids unsupported BIIs
+such as `actions.intent.CREATE_THING`; task creation is modeled as opening
+the LifeOps task feature and letting the app/runtime confirm any mutation.
+
+## External Assistant Landscape
+
+Checked 2026-05-14 against primary vendor docs:
+
+- OpenAI/ChatGPT on Apple platforms is exposed through Apple's ChatGPT
+  extension inside Apple Intelligence and Siri settings, not as a third-party
+  Siri API pattern for other apps to copy. See
+  <https://help.openai.com/en/articles/10263570-apple-intelligence-siri-faq>
+  and
+  <https://help.openai.com/en/articles/10269382-setting-up-chatgpt-with-apple-intelligence>.
+- Apple's supported third-party integration path is App Intents/App Shortcuts.
+  App Intents expose app capabilities to Shortcuts, Spotlight, Siri, widgets,
+  controls, and Apple Intelligence experiences. See
+  <https://developer.apple.com/documentation/appintents/app-intents>,
+  <https://developer.apple.com/shortcuts/>, and
+  <https://developer.apple.com/documentation/appintents/app-intent-domains>.
+- Claude uses the same Apple-native pattern: an "Ask Claude" App Intent
+  available from Siri, Spotlight, the Share menu, and Shortcuts. See
+  <https://support.anthropic.com/en/articles/10263469-using-claude-app-intents-shortcuts-and-widgets-on-ios>.
+- Perplexity exposes iOS voice assistant entry through Shortcuts, Lock Screen
+  controls, and the Action Button, including "Ask Perplexity" and "Start voice
+  conversation" shortcut actions. See
+  <https://www.perplexity.ai/help-center/en/articles/11132456-how-to-use-the-perplexity-voice-assistant-for-ios>.
+- Grok/X official docs currently document Grok availability on iOS, Android,
+  and web, including voice and text interactions, but do not document a public
+  Siri/App Intent shortcut contract comparable to Claude's. See
+  <https://help.x.com/en/using-x/about-grok>.
+- Google Assistant/App Actions are driven by capabilities in `shortcuts.xml`.
+  Assistant matches a built-in intent declared there and launches the app to
+  the requested screen. See
+  <https://developer.android.com/develop/devices/assistant/action-schema>.
+
+## Validation Checklist
+
+### Mac
+
+- [ ] Trigger the Mac shortcut/voice entry surface with a plain chat request
+  and confirm it opens the app/runtime instead of executing native-only logic.
+- [ ] Ask for a one-off reminder and verify a LifeOps `ScheduledTask` record is
+  created with `kind: "reminder"`.
+- [ ] Ask for a recurring check-in or follow-up and verify cadence lives in the
+  `ScheduledTask.trigger`/relationship edge, not in a platform notification.
+
+### iOS
+
+- [ ] Verify Siri/App Shortcuts open Eliza or invoke the App Intent handoff.
+- [ ] Verify `appIntentList` only reports local app/donated intents and does
+  not claim cross-app enumeration.
+- [ ] Ask Siri for a LifeOps reminder/check-in/follow-up and verify the app
+  creates a `ScheduledTask` through the runtime.
+- [ ] Confirm native `UNUserNotificationCenter` entries, if any, reference an
+  existing task or deep link back into the app.
+
+### Android Consumer
+
+- [ ] Verify `AndroidManifest.xml` registers `@xml/shortcuts` on `MainActivity`.
+- [ ] Verify the `eliza_app_action_chat` static shortcut opens the app chat
+  route.
+- [ ] Trigger an App Action and confirm the app/runtime receives the request.
+- [ ] Ask for a reminder/check-in/follow-up and verify LifeOps stores a
+  `ScheduledTask`; no Java/Kotlin-only reminder table or notification-only
+  schedule is created.
+
+### Android AOSP Assistant Role
+
+- [ ] On the system image, confirm `ROLE_ASSISTANT` resolves to Eliza.
+- [ ] Trigger `android.intent.action.ASSIST` and `VOICE_COMMAND`; both should
+  reach Eliza's app/runtime.
+- [ ] Confirm `ElizaAccessibilityService` and
+  `ElizaNotificationListenerService` are present only on the AOSP/system APK,
+  not the Play/Pixel cloud APK.
+- [ ] Confirm privileged capture/input capabilities do not change LifeOps
+  persistence: reminders/check-ins/follow-ups still use `ScheduledTask`.
+
+## Static Coverage
+
+- `ios-bridge.test.ts` checks that the TypeScript iOS AppIntent registry stays
+  aligned with the native x-callback switch.
+- `android-bridge.test.ts` checks that Android assistant/App Actions source
+  files route into app deep links rather than native notification creation.
+
+## Current Status and Remaining Work
+
+Implemented static/build coverage:
+
+- iOS App Shortcuts/App Intents are present in the native iOS target and hand
+  off source-tagged deep links to the app runtime.
+- Android App Actions metadata is present in `shortcuts.xml`, is registered on
+  `MainActivity`, and is rewritten by `run-mobile-build.mjs` to the configured
+  package and URL scheme.
+- Android assistant-role launch uses the same template deep-link scheme as the
+  other Android bridge activities, so white-label builds are rewritten
+  consistently.
+- Android AOSP system builds declare `ElizaAccessibilityService` and
+  `ElizaNotificationListenerService`; `android-cloud` strips those services,
+  their Java sources, and the accessibility-service XML resource.
+- Assistant launch payloads with captured text are consumed by the chat surface
+  and sent through the normal chat/planner path with source metadata. LifeOps
+  task/reminder deep links with text route to chat first so `ScheduledTask`
+  creation remains owned by the LifeOps runtime.
+
+Live-device validation is tracked through evidence manifests and platform
+release gates:
+
+- macOS shortcut/voice entry has installer/verifier scripts, but the final Siri
+  phrase creation and spoken-phrase result are still per-device manual checks.
+- iOS Siri/App Shortcuts must be installed on a physical device and recorded
+  in `docs/ios-device-validation.json` alongside the spoken phrases in
+  `ElizaAppShortcuts.swift`.
+- Android consumer App Actions require a Play/Assistant-capable device or test
+  environment to confirm Assistant fulfillment, not just static XML validity.
+- Android AOSP assistant-role behavior still needs a system-image validation
+  pass.
+
+Known remaining product test:
+
+- Static tests prove source routing and build-time wiring, but they do not prove
+  the planner produced the intended `ScheduledTask`. The remaining product test
+  is live end-to-end validation: speak or invoke a LifeOps reminder/check-in,
+  confirm the assistant launch payload sends through chat/planner, and verify a
+  LifeOps `ScheduledTask` record exists.

plugins/plugin-computeruse/docs/MULTI_MONITOR.md

@@ -0,0 +1,123 @@
+# Multi-monitor capture & coordinate contract (WS5)
+
+This plugin treats every physical display as an independent scene. There is
+NO virtual-desktop coordinate space exposed to the agent or the model.
+
+## Display enumeration
+
+`platform/displays.ts` returns the live attached set:
+
+```ts
+{
+  id: number,                          // OS-stable handle or 0-based index
+  bounds: [x, y, w, h],                // OS-global pixel space
+  scaleFactor: number,                 // 1 on Linux/Win, >1 on retina
+  primary: boolean,
+  name: string                         // e.g. "eDP-1", "DISPLAY1"
+}
+```
+
+Per-OS source:
+
+| OS              | Source                                                                  |
+| --------------- | ----------------------------------------------------------------------- |
+| Linux X11       | `xrandr --listmonitors`                                                 |
+| Linux Wayland   | `hyprctl monitors -j` / `swaymsg -t get_outputs` (else falls back to X) |
+| macOS           | `system_profiler SPDisplaysDataType -json`, then JXA `CGMainDisplayID`  |
+| Windows         | PowerShell `[Screen]::AllScreens`                                       |
+
+Native sidecars (Swift ScreenCaptureKit, DXGI/WGC, Rust libdisplay) are a
+follow-up — the interface is shaped to absorb them without changing callers.
+
+## Per-display capture
+
+`platform/capture.ts` is the canonical capture entry point:
+
+```ts
+captureDisplay(id):       Promise<{ display, frame: PNG-Buffer }>
+captureAllDisplays():     Promise<DisplayCapture[]>
+captureDisplayRegion():   Promise<DisplayCapture>     // local-to-display region
+```
+
+`frame` is at backing-store resolution. On a 2× retina display reporting a
+2560×1440 logical bounds with `scaleFactor: 2`, the PNG is 5120×2880.
+
+The legacy single-display `captureScreenshot()` from `screenshot.ts` is still
+exported for back-compat, but new code should prefer the per-display path.
+
+## Coordinate contract
+
+Every coordinate-bearing action accepts:
+
+```ts
+{
+  displayId: number,           // which display the coords belong to
+  coordinate: [x, y],          // LOCAL to that display
+  coordSource?: "logical"|"backing"  // default "logical"
+}
+```
+
+`platform/coords.ts::localToGlobal` translates to OS-global before the input
+driver fires. The model never sees OS-global coords.
+
+### Per-OS translation
+
+- **Linux/X11**: global = display.x + local.x. Pixels-to-pixels, no DPI.
+- **Windows (PerMonitorV2)**: same. Process MUST be PerMonitorV2 DPI-aware
+  (manifest entry — Electrobun handles this for the desktop app).
+- **macOS**: Quartz event coords are in *points* (logical pixels), not the
+  backing-store resolution of the capture. If the model used the raw retina
+  capture to choose coords, it must declare `coordSource: "backing"` so the
+  translator divides by `scaleFactor` before adding the display origin.
+
+### Why local-first
+
+1. Local coords match the model's actual visual context — it analyzed one
+   capture, not a stitched virtual desktop.
+2. Virtual-desktop coords are a perpetual bug source: negative origins
+   when secondary is left-of-primary, mixed DPI tiles, X11 vs CG origin
+   conventions.
+3. `displayId` is opaque to the model — it just echoes whatever the
+   `displays[]` provider gave it.
+
+### Backwards compatibility
+
+If `displayId` is omitted, the service:
+
+1. On single-display hosts: silently defaults to primary (debug-log only).
+2. On multi-display hosts: warns once per process, then defaults to
+   primary. Plan: this fallback will be removed once all in-tree callers
+   are migrated.
+
+## Ultrawide
+
+> 21:9 displays are NOT sliced. Each `displayId` is one scene. The
+aspect-aware patcher in WS6/WS7 sends the whole frame at the model's
+`max_pixels` budget; M-RoPE preserves aspect inside the model.
+
+## Provider surface
+
+`computerState` provider includes `data.displays: DisplayDescriptor[]` and
+in-text `computer_use.displays`. The planner reads this to pick a target
+display before issuing any coordinate-bearing COMPUTER_USE action.
+
+## What's manually validated
+
+This Linux test host has a single display. Automated tests cover:
+
+- `parseXrandrMonitors()` — string → DisplayInfo[] with golden fixtures.
+- `parseHyprlandMonitors()`, `parseSwayOutputs()` — JSON → DisplayInfo[].
+- `parseSystemProfilerDisplays()` — macOS JSON → DisplayInfo[].
+- `parseWindowsScreens()` — PowerShell JSON → DisplayInfo[].
+- `localToGlobal` / `globalToLocal` round-trip.
+
+Real multi-monitor capture & input injection on each OS still needs a
+manual rig:
+
+| What                                  | OS      | Manual check                                         |
+| ------------------------------------- | ------- | ---------------------------------------------------- |
+| `screencapture -D 2`                  | macOS   | dual display rig, capture each independently         |
+| Retina backing-store scale            | macOS   | scaleFactor=2, capture is 2× logical bounds          |
+| WGC / DXGI vs PerMonitorV2 DPI        | Windows | per-monitor DPI mix, click lands on correct pixel    |
+| Wayland portal capture                | Linux   | GNOME 45+ / KDE 6 — `platform/wayland-portal.ts` captures through xdg-desktop-portal; per-display crop still depends on compositor support |
+| Hyprland / Sway parser                | Linux   | run `hyprctl monitors -j` against live compositor    |

plugins/plugin-computeruse/docs/TEST_LANES_COMPUTERUSE_VISION.md

@@ -0,0 +1,148 @@
+# Test lanes — computeruse × vision (per-OS)
+
+How the `@elizaos/plugin-computeruse` + `@elizaos/plugin-vision` test surface is
+organized across the unit lane and the real-driver lane, what runs where, the
+per-OS host requirements, and the **Windows non-interactive-session gotcha** that
+makes input-effect tests fail outside a logged-in desktop. Companion to the
+CUA × Vision EPIC (#9105) and the trycua/cua parity tracker (#9170).
+
+## Lanes at a glance
+
+| Lane | Selects | Runs the real OS? | Command |
+|------|---------|-------------------|---------|
+| **Unit / component** | every `*.test.ts` **except** `*.real`/`*.live`/`*.e2e` | No (mocked) — but platform-gated `*.test.ts` self-skip on the wrong OS | `bun run --cwd plugins/plugin-computeruse test` · `bun run --cwd plugins/plugin-vision test` |
+| **Real-driver / live** | `*.real.test.ts` / `*.live.test.ts` (repo-wide) | **Yes** — nutjs / PowerShell / WinRT / Apple Vision / xdotool against the host | shared config `packages/test/vitest/real.config.ts` (post-merge lane via `packages/scripts/run-all-tests.mjs`) |
+| **Standalone probe** | a hand-written `.mts` importing `src/platform/*.js` | **Yes** — fastest Windows real smoke | `bun plugins/plugin-computeruse/<probe>.mts` |
+
+### Unit lane
+
+Each plugin's `vitest.config.ts` **excludes** `**/*.real.test.{ts,tsx}`,
+`**/*.live.test.{ts,tsx}`, and `**/*.e2e.*`. So `bun run --cwd plugins/plugin-computeruse test`
+never touches the real OS.
+
+Platform-specific behavior that *can* be exercised without a real desktop still
+lives in the unit lane as a regular `*.test.ts` gated with
+`it.skipIf(platform() !== "win32")(...)`. These **run on Windows in the normal
+lane** and self-skip elsewhere. Examples:
+
+- `plugin-vision/src/ocr-service-windows.test.ts` — renders a PNG and OCRs it
+  through the real **`Windows.Media.Ocr`** WinRT engine (works headless).
+- `plugin-computeruse/src/__tests__/cua-parity-surface.test.ts` — Windows
+  clipboard-write command regression (`Set-Clipboard -Value
+  [Console]::In.ReadToEnd()`, not `$input | Set-Clipboard`).
+
+### Real-driver / live lane
+
+`*.real.test.ts` files are **only** picked up by the shared real config
+`packages/test/vitest/real.config.ts` (include globs `**/*.real.test.ts`,
+`**/*.live.test.ts`). They drive the real input/capture/OCR stack. A
+`fail-on-silent-skip` setup (`packages/test/vitest/fail-on-silent-skip.setup.ts`)
+**fails any test that silently skips**, so a `.real` test must either run or be
+explicitly excluded — it cannot quietly no-op.
+
+Run a single real test against the host (the lane's include globs are repo-wide,
+so pass a file path to scope it):
+
+```bash
+bunx vitest run plugins/plugin-computeruse/src/__tests__/cua-parity-input.real.test.ts \
+  --config packages/test/vitest/real.config.ts
+```
+
+`ELIZA_CI_REAL=1` additionally drops credential/upstream-gated reals
+(e.g. `computeruse.real.test.ts`, whose headless-browser path needs a display).
+
+### Standalone probe (proven Windows smoke)
+
+For a fast Windows real-driver check, a standalone `.mts` that imports the
+platform modules directly and is run with `bun` beats the vitest real lane
+(no config graph, instant feedback). Pattern:
+
+```ts
+// parity-probe.mts — bun plugins/plugin-computeruse/parity-probe.mts
+import { legacyGetCursorPosition } from "./src/platform/desktop.js";
+import { readClipboard, writeClipboard } from "./src/platform/clipboard.js";
+console.log(legacyGetCursorPosition());          // live WinForms read
+await writeClipboard("hi"); console.log(await readClipboard());
+```
+
+Keep probes out of commits (delete after use) — they are smoke checks, not the
+suite.
+
+## Input driver selection
+
+`ELIZA_COMPUTERUSE_DRIVER` selects the input backend:
+
+- `nutjs` (default) — `@nut-tree-fork/nut-js` native bindings.
+- `legacy` — per-OS shell tools: PowerShell (Windows), `cliclick` (macOS),
+  `xdotool` (Linux).
+
+nutjs auto-falls-back to `legacy` when the native module fails to load. Note the
+read path is asymmetric on Windows: `driverGetCursorPosition` always uses the
+WinForms OS query on Windows because nutjs `mouse.getPosition()` returns a stale
+constant there (see #9165).
+
+## Per-OS host requirements
+
+| OS | Capture | Input | OCR | Notes |
+|----|---------|-------|-----|-------|
+| **Windows** | PowerShell / nutjs | nutjs (SendInput) or legacy PowerShell | **`Windows.Media.Ocr`** (WinRT via `powershell` 5.1, **not** pwsh 7) — 0 tokens | **Input needs an interactive desktop** (see gotcha). Read/clipboard/OCR work headless. |
+| **macOS** | `screencapture -D` (retina 2× backing store) | nutjs or `cliclick` | Apple Vision | Needs Accessibility + Screen Recording grants; headful. |
+| **Linux** | X11 `import`/`scrot` (Xvfb headful); Wayland `xdg-desktop-portal` screenshot sidecar | nutjs or `xdotool` on X11 | docTR / PaddleOCR | Wayland capture uses `python3` + `gdbus`; Wayland still lacks AT-SPI in many compositors, so grounding can be OCR-only. Clipboard needs `wl-clipboard`/`xclip`. |
+| **AOSP** | MediaProjection | privileged input bridge | Paddle-Lite | Emulator + system-app path. See `AOSP_SYSTEM_APP.md`. |
+
+## ⚠️ Windows non-interactive-session gotcha (verified 2026-06-23)
+
+In a **Session-0 / RDP-disconnected / service** Windows session, programmatic
+cursor **movement and button/key presses are silent no-ops**, while reads,
+clipboard, and OCR work. Verified directly on the Windows backend:
+
+| Operation | Non-interactive session |
+|-----------|-------------------------|
+| `legacyGetCursorPosition()` (WinForms read) | ✅ live, accurate |
+| nutjs `mouse.setPosition` / `mouse.move` | ❌ no-op (cursor unchanged) |
+| PowerShell `SetCursorPos` | ❌ no-op |
+| nutjs `pressButton`/`releaseButton`, `pressKey`/`releaseKey` (SendInput) | ❌ no-op (no foreground desktop to receive input) |
+| clipboard read/write round-trip | ✅ works headless |
+| `Windows.Media.Ocr` (rendered PNG → text + word boxes) | ✅ works headless |
+
+**Consequence for the real lane:** any `*.real.test.ts` that asserts an *input
+effect* — e.g. `cua-parity-input.real.test.ts`'s `move → get_cursor_position`
+round-trip — will **fail** (not skip) on a non-interactive Windows runner. Run
+against this backend it produces:
+
+```
+× get_cursor_position reflects driverMouseMove
+  AssertionError: expected 388 to be less than or equal to 2   # cursor never moved
+✓ clipboard write/read round-trips
+```
+
+This is an **environment** failure, not a code defect. The Windows real-driver
+CI lane therefore MUST run on an **interactive, logged-in desktop** (autologon +
+unlocked session; the agent on a real desktop), not as a service / in Session 0.
+Read-only reals (cursor read, clipboard, OCR) are safe headless and can run on
+any Windows runner.
+
+## What is validated on Windows today
+
+Confirmed working on the Windows backend (2026-06-23):
+
+- **Cursor read** — `driverGetCursorPosition` returns the live position (WinForms).
+- **Clipboard** — read/write round-trips a newline/unicode/quote payload
+  (the `Set-Clipboard -Value [Console]::In.ReadToEnd()` fix; #9165).
+- **OCR** — native `Windows.Media.Ocr` extracts rendered text with per-word boxes
+  (`ocr-service-windows.test.ts`, 5/5; #9121 M4a, wired to `COMPUTER_USE ocr` /
+  `detect_elements` via #9173 M7).
+
+Input-effect verification (move/click/drag/press fidelity, DPI, multi-monitor)
+requires an interactive Windows desktop and is gated to that lane.
+
+## Release evidence
+
+Per-OS evidence manifests live next to this doc
+(`windows-desktop-validation.json`, `macos-desktop-validation.json`,
+`linux-desktop-validation.json`, `android-*-validation.json`,
+`ios-device-validation.json`) and are checked by
+`bun run --cwd plugins/plugin-computeruse validate:<os>-desktop-evidence`.
+Keep incomplete live-device checks under `requires_device_evidence`; only use
+`validate:platform-evidence -- --require-complete` for release gates that truly
+have artifacts for every required platform check.

plugins/plugin-computeruse/docs/TRYCUA_PARITY_AUDIT.md

@@ -0,0 +1,176 @@
+# Computer-Use × Vision — trycua/cua parity audit (current state)
+
+Authoritative, **current-state** review of where elizaOS `plugin-computeruse` +
+`plugin-vision` stand against [trycua/cua](https://github.com/trycua/cua) and the
+six other computer-use / OCR references, refreshed **after M3 / M3.5 / M4a / M5 /
+M6 / M7 / M8** landed on `develop` (the older `reports/trycua-*.txt` snapshot
+predates M7+M8 and is stale).
+
+- Tracking issues: **#9105** (EPIC — integration) and **#9170** (parity + tests
+  on Windows/Linux/macOS/AOSP).
+- Test-lane reference: [`TEST_LANES_COMPUTERUSE_VISION.md`](./TEST_LANES_COMPUTERUSE_VISION.md).
+- Raw research snapshots: `reports/cua-external-summary.txt`,
+  `reports/trycua-matrix.txt`, `reports/trycua-gaps.txt`.
+
+---
+
+## 1. External-repo review (what to adopt, what is N/A)
+
+| Repo | What it is | OCR | Local vision model | Continuous/low-token screen | Net takeaway for us |
+|------|-----------|-----|--------------------|------------------------------|---------------------|
+| **trycua/cua** | ~50-verb computer-server (WS/HTTP/MCP) over 6 VM/sandbox providers + agent SDK (loop registry, predict_step/predict_click, Set-of-Marks/OmniParser grounding, callback middleware) | model-side (hosted) | no | discrete capture→act loop | **The parity target.** Verb gaps are small/additive; the real work is the architecture: loop registry (M10), SoM overlay (M9), provider/RPC matrix (M13), eval harness (M14). |
+| **coasty-ai/open-computer-use** (Apache-2.0) | OSS+SaaS CUA; web + Electron + MCP | classical (pytesseract, Linux VM only) | no (hosted "Coasty CUA") | discrete loop, JPEG q70, HD price penalty | **Adopt:** cross-platform input-correctness (DPI scale, off-primary cursor) — *already taken in M3.5*; predict/ground separation — *M5*; stateful session to cut tokens; avoid runtime-compiled inline C# in the PS fallback (AMSI fingerprint). |
+| **injaneity/pi-computer-use** (MIT) | macOS-only; AX-tree grounding, **no vision/OCR** | none (AX tree + CDP) | none | pull-based AX snapshots | **Adopt:** accessibility-tree grounding tier to skip OCR for most clicks — *partially in scene a11y-provider*; stale-ref reacquisition; batched multi-action → one post-state; GET_SCREEN envelope with explicit scale/coordinateSpace/stateId. |
+| **domdomegg/computer-use-mcp** (MIT) | thin local MCP mirror of Anthropic's computer-use tool | none (model does it) | none | pull-based, downscaled | **Adopt:** cursor-crosshair drawn into the frame; hard-cap to 1568px/~1.15MP + re-encode before any cloud model; validated bidirectional image↔logical scale; `xdotool --clearmodifiers`. |
+| **vercel-labs/ai-sdk-computer-use** (no license — reference only) | Next.js demo, Claude drives a Vercel Sandbox | none | none (cloud Claude) | noVNC live for human; pull screenshots for model | **Adopt (patterns, not code):** `prunedMessages()` token-frugal history (redact prior frames, keep latest) — *aligns with M3 dHash + M11*; pull-based, skip re-screenshot after blind type/key; typed image tool-results; Anthropic `cacheControl` on the stable preamble; provider-defined-tool path when the model is Anthropic. |
+| **bracesproul/gen-ui-computer-use** (MIT) | LangGraph Generative-UI showcase over OpenAI CUA + Scrapybara | none | none (OpenAI CUA) | hosted VNC iframe + per-action audit shots | **Adopt:** before/after action hooks streaming typed UI cards; clean live-view vs audit-trail split; content-addressed screenshot store (we already have the better sha256 media-store) + collapsed-by-default frames; live pause/approve/abort controls. |
+| **baidu/Unlimited-OCR** (MIT) | one-shot long-horizon **document** OCR (DeepSeek-OCR + DeepSeek-V2 MoE, GGUF-able) | VLM end-to-end | yes (3B/0.5B-active) | n/a (batch doc parser) | **Adopt later:** a candidate **local** OCR/VLM for `plugin-vision` once exposed as GGUF — fits the "local, token-frugal" goal. **Not** a screen/CUA tool; document-parsing only. |
+
+**Cross-cutting design conclusions already encoded in our roadmap:** OCR + element
+detection belong in `plugin-vision` and feed `plugin-computeruse` through the
+`CoordOcrProvider` seam (M1); screen understanding must be **local + token-frugal**
+(GGUF only, no ONNX) with dHash-gated re-description (M3); a `GET_SCREEN` action
+returns a typed image + structured elements (M2).
+
+---
+
+## 2. Verb / capability parity matrix (vs trycua/cua, on `develop` today)
+
+Legend: ✅ have · 🟡 partial · ❌ missing · ⛔ N/A (deliberately not chased)
+
+### Pointer / keyboard / screen (host drivers)
+| trycua capability | elizaOS status | Where |
+|---|---|---|
+| screenshot (per-display, native res, quality/downscale) | ✅ | `platform/capture.ts`, `screenshot-quality.ts` |
+| click / right_click / double_click | ✅ | `COMPUTER_USE` |
+| **middle_click** | ✅ (M8) | `nut-driver.ts` `nutMiddleClick` |
+| click_with_modifiers | ✅ | |
+| mouse_move | ✅ | |
+| **mouse_down / mouse_up** (button-param press-hold) | ✅ (M8) | `driver.ts` `driverMouseDown/Up(x,y,button)` |
+| **key_down / key_up** (press-hold; bare modifiers) | ✅ (M8 + #9189) | `resolveKeyCode` now maps `MODIFIER_KEYS` |
+| type / key / key_combo | ✅ | |
+| scroll (per-notch + raw) | ✅ (M3.5 per-notch pacing = parity-plus) | |
+| drag (start→end) | ✅ | |
+| **drag (multi-point polyline path)** | ✅ (M8) | `driver.ts` `driverDragPath` → `nutDragPath`/`densifyDragPath` |
+| get_cursor_position | ✅ (M3.5) | win32 uses `System.Windows.Forms.Cursor` (nutjs getPosition is stale) |
+| get_screen_size / to_screen(shot)_coordinates | ✅ | multi-display + DPI transforms |
+| get_environment / desktop info | 🟡 | `get_environment` yes; `get_desktop_environment` (for driver selection) ❌ minor |
+
+### Vision / grounding
+| trycua capability | elizaOS status | Where |
+|---|---|---|
+| **ocr** (full-screen, coord-bearing) | ✅ (M7) | `COMPUTER_USE ocr` → `getCoordOcrProvider()` (plugin-vision OCR) |
+| **detect_elements** | ✅ (M7) | routes through scene element registry + CoordOcr |
+| **GET_SCREEN action** (typed image + elements, token-frugal) | ✅ (M2) | `plugin-vision/src/get-screen.ts`, `get-screen-elements.ts` |
+| low-token continuous description | ✅ (M3) | dHash-gated `DirtyTileDescriber` + frame-dHash skip + token counters |
+| predict / ground split (cheap ground vs full predict) | 🟡 (M5) | `OcrCoordinateGroundingActor` + per-Scene grounding cache; no model-string registry yet |
+| accessibility-tree grounding tier | ✅ | scene `a11y-provider` (Win UIA / macOS AX / Linux AT-SPI) |
+| OmniParser/icon detection (GGUF, no ONNX) | 🟡 | `yolo-detector.ts` (YOLOv8n COCO GGUF via `native/yolo.cpp`) |
+| **Set-of-Marks (numbered overlay) grounding** | ✅ (M9) | `plugin-vision/src/som.ts` + `set-of-marks-provider.ts`; registered via `registerSetOfMarksProvider` seam in `mobile/ocr-provider.ts` |
+
+### Windows / files / process / shell
+| trycua capability | elizaOS status | Where |
+|---|---|---|
+| window list/focus/switch/arrange/move/min/max/restore/close | ✅ | `WINDOW` action |
+| get_current_window_id / get_application_windows(app) | ❌ (M12) | `scene.focused_window` is adjacent but no discrete getter verb |
+| get_window_size / get_window_position / set_window_size | ❌ (M12) | `WINDOW_MOVE` sets position only |
+| open(target) / launch(app,args)→pid | ❌ (M12) | |
+| filesystem verbs (exists/list/read_text/write_text/read_bytes/write_bytes/delete/create_dir/get_file_size) | 🟡 (M12) | internal `file-ops.ts` + docker-backend wire ops; **not** exposed as `COMPUTER_USE` verbs |
+| run_command (CommandResult) | 🟡 (M12) | `terminal.ts` one-shot + docker-backend; not a host `COMPUTER_USE` verb by design |
+| PTY / interactive terminal | ❌ low-pri | one-shot only |
+| clipboard read/write | ✅ (M3.5) | `CLIPBOARD` action |
+
+### Architecture / platform
+| trycua capability | elizaOS status | Notes |
+|---|---|---|
+| agent-loop registry (model-string → loop) | 🟡 (M10) | hardcoded Brain→Cascade→Actor→dispatch |
+| callback middleware (budget/image-retention/trajectory/operator-normalize) | 🟡 (M11) | approval-manager + token counters + log-line trajectory events |
+| VM/sandbox provider matrix | 🟡 (M13) | Docker backend only; WSB + QEMU are the opt-in targets worth adding |
+| daemon/RPC seam (`{command,params}→{success,result}`) | ❌ (M13) | in-process service; routes only for approvals |
+| MCP server seam | ❌ | one optional MCP seam is the relevant subset of cua's 2 servers / 35 tools |
+| eval harness (ScreenSpot / OSWorld / GET-commands matrix) | 🟡 (M14) | `osworld/` adapter + `benchmarks/eliza-1/vision-cua-e2e`; not yet per-OS scenario-wired |
+| AOSP touch + hardware keys + multitouch | 🟡 | Capacitor bridge tap/swipe + globalAction(back/home/recents); `multitouch_gesture` ❌ |
+
+### Explicitly N/A (do not chase)
+`set_wallpaper` (niche) · `playwright_exec` / `browser_execute` (**unconditionally
+disabled**, GHSA-rcvr-766c-4phv) · cloud managed sandbox · the full 2-MCP-server /
+35-driver-tool surface · PII anonymization · AOSP gRPC `:8554` backend (CI-only).
+
+---
+
+## 3. Test-coverage matrix (every-verb, by lane and platform)
+
+Lanes: **unit** = default Vitest (runs on all OSes), **real** = `*.real.test.ts`
+gated `platform()==="win32"` and excluded from the default run (Windows only),
+**e2e**, **scenario** (scenario-runner / issue-evidence), **probe** = standalone
+`bun` driver check.
+
+| Capability group | unit (all OS) | Windows real/probe | Linux | macOS | AOSP |
+|---|---|---|---|---|---|
+| pointer click family + modifiers | `cua-parity-surface`, `computer-interface`, `dispatch`, `use-computer-action` | `cua-parity-input.real`, `computeruse.real`, `service.real`, probes | ⚠️ no real lane | ⚠️ no real lane | n/a |
+| M8 middle/down-up + key down-up + drag-path | `cua-parity-surface`, `nut-driver-input` (incl. modifier guard) | `cua-parity-input.real` + this-session probe (**verified**) | ⚠️ no real lane | ⚠️ no real lane | n/a |
+| ocr / detect_elements (M7) | `ocr-adapter-coord-seam`, `ocr-with-coords`, `computeruse-ocr-bridge` | `service.real`, OCR bridge | unit only | `ocr-service-apple-vision` (unit) | n/a |
+| GET_SCREEN / scene / dHash (M2/M3) | `get-screen`, `get-screen-elements`, `dirty-tile-describer`, `screen-state`, `dhash`, `scene-*` | scene multimon/probe | unit only | unit only | `android-scene` |
+| windows / clipboard / cursor | `platform-capabilities`, `windows-powershell-safety` (static) | `windows-list.real`, `clipboard` (win32), cursor probe | unit only | unit only | n/a |
+| files / terminal | `security-file-target` | `file-ops.real`, `terminal.real` | unit only | unit only | n/a |
+| OCR engines | `ocr-service`, `ocr-service-windows`, `yolo-detector` | `ocr-service-windows` (win32) | `ocr-service` (docTR) | `ocr-service-apple-vision` | n/a |
+| autonomous loop | `cascade`, `brain`, `dispatch`, `computer-use-agent` | `computer-use-agent.real`, `runtime.live.e2e` | unit only | unit only | `mobile-cascade`, `aosp-input-actor` |
+| benchmarks | `osworld-action-converter` | `benchmark/osworld-*.real` | — | — | — |
+| approvals / progress | `computer-use-approval-relay` | `routes-e2e` | scenario `8912-computeruse-progress-approvals-*` | — | — |
+| mobile/AOSP | — | — | — | — | `android-bridge`, `android-trajectory`, `ios-*`, device-evidence manifests |
+
+### The single biggest "tests on all platforms" hole
+The real-driver lane is **`win32`-only** (`const RUN = platform()==="win32"`). The
+nutjs + legacy `xdotool`/`cliclick` code paths exist for Linux/macOS but **no
+equivalent real-driver lane executes them** — and because the gate skips silently,
+a Linux/macOS regression would not turn a lane red. This is exactly how the M8
+`key_down("shift")` bug (fixed in #9189) reached `develop`: M8's real test was
+authored on a host without a Windows machine, so the gated case never ran.
+
+**Recommended (M14):** a Linux Xvfb headful lane and a macOS headful lane that run
+the parity verbs against the real driver, plus a machine-checkable matrix derived
+from the action enum so a newly-added trycua verb that lacks a per-OS test fails CI.
+
+---
+
+## 4. Per-platform verification status
+
+| OS | Driver | Status |
+|----|--------|--------|
+| **Windows 11** | nutjs (default) + legacy PowerShell | **Primary, verified.** All M8 verbs + key_down modifier fix dispatched against the live desktop this cycle (cursor read back via WinForms). OCR via `Windows.Media.Ocr` (M4a). |
+| **Linux** | nutjs + legacy `xdotool` (X11; AT-SPI/Wayland fallback) | Code paths present + unit-tested; **no real-driver lane runs** → needs Xvfb headful CI. Clipboard needs `wl-clipboard`/`xclip` (asserts clear error when absent). |
+| **macOS** | nutjs + legacy `cliclick`; retina backing-store 2× transforms | Code paths present + unit-tested (incl. Apple Vision OCR); **no real-driver lane runs** → needs headful macOS CI; assert retina coordinate transforms. |
+| **AOSP / Android** | Capacitor bridge + `MobileComputerInterface` | tap/swipe + globalAction(back/home/recents) implemented + unit-tested; device-evidence manifests exist; `multitouch_gesture` missing; system-app path is CI-only. |
+
+---
+
+## 5. Remaining roadmap (prioritized)
+
+1. **M14 cross-platform real lanes** *(highest leverage for the stated goal)* —
+   The enum-derived parity coverage guard (`cua-parity-coverage.test.ts`) is
+   **landed** and runs on all platforms in the default lane, and the real lane is
+   no longer win32-hardcoded. The remaining piece is the **Linux Xvfb + macOS
+   headful CI lanes** that actually invoke the real config (requires a
+   `.github/workflows/**` change / `workflow` token scope). Until those exist,
+   real-driver *actuation* is verified on Windows only.
+2. **M12 additive verbs** — window getters (`get_current_window_id`,
+   `get_application_windows`, `get/set_window_size`, `get_window_position`),
+   `open`/`launch`, and the filesystem/`run_command` verbs (wrap the existing
+   internal `file-ops.ts`/`terminal.ts` as gated `COMPUTER_USE` verbs).
+3. **M10 agent-loop registry** + predict_step/predict_click split (extend the Actor
+   registry; keep approval gating).
+4. **M11 callback middleware** — budget cap, image-retention (only-N-recent, aligns
+   with M3), operator-normalizer, trajectory.
+5. **M13 provider matrix + RPC seam** — Windows Sandbox (WSB) + QEMU behind a
+   remote-guest `{command,params}→{success,result}` RPC over the route/WS seam.
+6. **AOSP `multitouch_gesture`** (MT-Protocol-B) + an optional MCP server seam.
+
+**Done since this audit was first written:** M9 Set-of-Marks (numbered-overlay
+grounding) landed — `plugin-vision/src/som.ts` + `set-of-marks-provider.ts`,
+registered via the `registerSetOfMarksProvider` seam; 23/23 unit tests green on
+Windows.
+
+> **Process note for contributors:** the `*.real.test.ts` lanes only execute on a
+> Windows host. Before trusting any newly-landed computeruse milestone, run its
+> verbs against the real driver on Windows (or the future Linux/macOS lanes) — a
+> green default-lane run does **not** mean the platform paths work. See §3.

plugins/plugin-computeruse/docs/android-aosp-validation.json

@@ -0,0 +1,107 @@
+{
+  "schemaVersion": 1,
+  "platform": "android-aosp",
+  "status": "requires_device_evidence",
+  "target": {
+    "minimumApi": "system image capable of ROLE_ASSISTANT and privileged permission grants",
+    "imageType": "AOSP/system validation image",
+    "requiredBuildFlavor": "android-system or sideload-only system APK",
+    "privilegedPermissions": [
+      "READ_FRAME_BUFFER",
+      "INJECT_EVENTS",
+      "REAL_GET_TASKS"
+    ]
+  },
+  "evidence": {
+    "imageName": null,
+    "androidVersion": null,
+    "apiLevel": null,
+    "buildId": null,
+    "validatedAt": null,
+    "validator": null,
+    "artifacts": []
+  },
+  "checks": [
+    {
+      "id": "assistantRole",
+      "method": "ROLE_ASSISTANT resolution",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "Eliza is installed as or granted the assistant role",
+        "role manager resolves Eliza for assistant requests",
+        "consumer/Play build remains free of ROLE_ASSISTANT requirements"
+      ]
+    },
+    {
+      "id": "assistVoiceCommand",
+      "method": "ACTION_ASSIST/VOICE_COMMAND",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "android.intent.action.ASSIST reaches Eliza runtime",
+        "android.intent.action.VOICE_COMMAND reaches Eliza runtime",
+        "payload source metadata is preserved"
+      ]
+    },
+    {
+      "id": "privilegedCapture",
+      "method": "SurfaceControl.captureDisplay",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "READ_FRAME_BUFFER is granted only to the system build",
+        "capture returns a non-empty frame",
+        "frame visually matches the current display"
+      ]
+    },
+    {
+      "id": "privilegedInput",
+      "method": "InputManager.injectInputEvent",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "INJECT_EVENTS is granted only to the system build",
+        "tap injection returns success",
+        "swipe injection returns success",
+        "unsupported text/key actions are rejected with explicit errors"
+      ]
+    },
+    {
+      "id": "processEnumeration",
+      "method": "IActivityManager.getRunningAppProcesses",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "REAL_GET_TASKS is granted only to the system build",
+        "running processes are enumerated",
+        "foreground process is identifiable"
+      ]
+    },
+    {
+      "id": "serviceFlavorSeparation",
+      "method": "build flavor inspection",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "ElizaAccessibilityService is present only where expected",
+        "ElizaNotificationListenerService is present only where expected",
+        "system-only Java/Kotlin sources are excluded from android-cloud"
+      ]
+    },
+    {
+      "id": "consumerBuildStripping",
+      "method": "consumer artifact inspection",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "android-cloud strips privileged services",
+        "android-cloud strips accessibility-service XML resource",
+        "android-cloud has no privileged permission declarations"
+      ]
+    },
+    {
+      "id": "lifeOpsPersistence",
+      "method": "assistant-to-runtime handoff",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "AOSP assistant requests enter the app/runtime planner",
+        "reminders/check-ins/follow-ups persist as ScheduledTask records",
+        "privileged capture/input does not create native-only task state"
+      ]
+    }
+  ]
+}

plugins/plugin-computeruse/docs/android-device-validation.json

@@ -0,0 +1,126 @@
+{
+  "schemaVersion": 1,
+  "platform": "android-consumer",
+  "status": "requires_device_evidence",
+  "target": {
+    "minimumApi": "24 for Accessibility gesture dispatch; 29 for MediaProjection foreground service",
+    "minimumAndroid": "Android 10 for full capture validation; Android 7+ for basic bridge validation",
+    "requiredBuildFlavor": "android-cloud or Play-compatible consumer APK",
+    "distribution": "Google Android / Pixel / Play-compatible build; no privileged system permissions"
+  },
+  "evidence": {
+    "deviceModel": null,
+    "androidVersion": null,
+    "apiLevel": null,
+    "buildId": null,
+    "validatedAt": null,
+    "validator": null,
+    "artifacts": []
+  },
+  "checks": [
+    {
+      "id": "permissionsSetup",
+      "method": "manual-device-setup",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "CAMERA runtime permission is granted",
+        "Eliza AccessibilityService is enabled and visible in dumpsys accessibility",
+        "Usage Access is enabled for Eliza",
+        "consumer build does not request privileged assistant/system permissions"
+      ]
+    },
+    {
+      "id": "accessibilityTree",
+      "method": "getAccessibilityTree",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "foreground app has a populated accessibility tree",
+        "nodes include id, role, label, bbox, and actions",
+        "role values are Android class names"
+      ]
+    },
+    {
+      "id": "gestureDispatch",
+      "method": "dispatchGesture",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "tap returns ok:true and is visibly applied",
+        "swipe returns ok:true and scrolls the target content",
+        "coordinates are screen-pixel coordinates"
+      ]
+    },
+    {
+      "id": "globalActions",
+      "method": "performGlobalAction",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "back action is visually confirmed",
+        "home action is visually confirmed",
+        "recents action is visually confirmed",
+        "notifications action is visually confirmed"
+      ]
+    },
+    {
+      "id": "mediaProjectionCapture",
+      "method": "startMediaProjection/captureFrame/stopMediaProjection",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "system consent dialog appears and is accepted",
+        "captureFrame returns ok:true with non-empty jpegBase64",
+        "decoded JPEG matches the current screen",
+        "stopMediaProjection returns stopped:true"
+      ]
+    },
+    {
+      "id": "usageStatsEnumeration",
+      "method": "enumerateApps",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "Usage Access grant is honored",
+        "enumerateApps returns packageName, label, lastUsedMs, totalForegroundMs, and isForeground",
+        "frontmost app is marked isForeground:true"
+      ]
+    },
+    {
+      "id": "cameraCapture",
+      "method": "startCamera/captureFrameCamera/stopCamera",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "startCamera returns at least one camera",
+        "captureFrameCamera returns non-empty jpegBase64",
+        "stopCamera succeeds"
+      ]
+    },
+    {
+      "id": "memoryPressureDispatch",
+      "method": "onTrimMemory/dispatchMemoryPressure",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "TRIM_MEMORY_RUNNING_LOW or CRITICAL is triggered",
+        "bridge dispatches a pressure sample",
+        "MemoryArbiter pressure state transitions to low or critical"
+      ]
+    },
+    {
+      "id": "appActionsShortcuts",
+      "method": "Assistant/App Actions/static shortcuts",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "launcher activity registers @xml/shortcuts",
+        "OPEN_APP_FEATURE, CREATE_MESSAGE, and GET_THING are present",
+        "Assistant preview triggers chat, voice, daily brief, new task, and tasks routes",
+        "deep links are source-tagged and brand-rewritten"
+      ]
+    },
+    {
+      "id": "lifeOpsScheduledTaskHandoff",
+      "method": "assistant-to-runtime handoff",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "reminder/check-in/follow-up request enters the app/runtime planner",
+        "LifeOps creates or updates a ScheduledTask record",
+        "no native-only reminder table or notification-only schedule is created"
+      ]
+    }
+  ]
+}

plugins/plugin-computeruse/docs/ios-device-validation.json

@@ -0,0 +1,113 @@
+{
+  "schemaVersion": 1,
+  "platform": "ios",
+  "status": "requires_device_evidence",
+  "target": {
+    "minimumIos": "17.6",
+    "primaryIos": "26.1",
+    "minimumDeviceClass": "A14-or-later iPhone"
+  },
+  "evidence": {
+    "deviceModel": null,
+    "iosVersion": null,
+    "buildId": null,
+    "validatedAt": null,
+    "validator": null,
+    "artifacts": []
+  },
+  "checks": [
+    {
+      "id": "probe",
+      "method": "probe",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "platform is ios",
+        "osVersion matches the device",
+        "all six capability bits are present"
+      ]
+    },
+    {
+      "id": "replayKitForegroundStart",
+      "method": "replayKitForegroundStart",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "first-use permission prompt is shown",
+        "accepted capture returns non-empty jpegBase64 frames",
+        "frame resolution matches the device screen"
+      ]
+    },
+    {
+      "id": "broadcastExtensionHandshake",
+      "method": "broadcastExtensionHandshake",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "extension target is bundled",
+        "share-sheet broadcast picker starts a session",
+        "broadcastActive transitions to true",
+        "iOS 26 beta extension-death regression status is recorded when applicable"
+      ]
+    },
+    {
+      "id": "visionOcr",
+      "method": "visionOcr",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "known PNG with WS9 OCR SMOKE is rendered",
+        "accurate OCR returns fullText containing the source string"
+      ]
+    },
+    {
+      "id": "appIntentList",
+      "method": "appIntentList",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "fresh install returns an empty donated-intent list",
+        "list grows after donation through Shortcuts"
+      ]
+    },
+    {
+      "id": "appIntentInvokeSafari",
+      "method": "appIntentInvoke",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "com.apple.mobilesafari.open-url opens Safari to the provided URL"
+      ]
+    },
+    {
+      "id": "appIntentInvokeMessages",
+      "method": "appIntentInvoke",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "com.apple.MobileSMS.send-message opens Messages with recipient and body pre-filled"
+      ]
+    },
+    {
+      "id": "accessibilitySnapshot",
+      "method": "accessibilitySnapshot",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "top-level node role is not labeled",
+        "top-level node has children on a populated screen"
+      ]
+    },
+    {
+      "id": "foundationModelGenerate",
+      "method": "foundationModelGenerate",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "Apple Intelligence enabled returns non-empty generated text",
+        "Apple Intelligence disabled returns foundation_model_unavailable"
+      ]
+    },
+    {
+      "id": "memoryPressureProbe",
+      "method": "memoryPressureProbe",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "debug memory warning is invoked",
+        "next probe returns severity >= 0.7",
+        "lastWarningAt is set"
+      ]
+    }
+  ]
+}

plugins/plugin-computeruse/docs/linux-desktop-validation.json

@@ -0,0 +1,114 @@
+{
+  "schemaVersion": 1,
+  "platform": "linux-desktop",
+  "status": "requires_device_evidence",
+  "target": {
+    "minimumDistribution": "Ubuntu 22.04 or equivalent modern desktop Linux",
+    "displayServer": "X11 required for xdotool/wmctrl path; Wayland portal capture remains a separate manual check",
+    "driver": "nutjs preferred; legacy xdotool/scrot/wmctrl fallback allowed"
+  },
+  "evidence": {
+    "machineId": null,
+    "distribution": null,
+    "kernelVersion": null,
+    "displayServer": null,
+    "buildId": null,
+    "validatedAt": null,
+    "validator": null,
+    "artifacts": []
+  },
+  "checks": [
+    {
+      "id": "capabilityProbe",
+      "method": "detectPlatformCapabilities/getCapabilities",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "platform is linux",
+        "screenshot, computerUse, windowList, browser, terminal, fileSystem, and clipboard capabilities are reported",
+        "missing optional tools produce explicit unavailable/tool messages"
+      ]
+    },
+    {
+      "id": "dependencyProbe",
+      "method": "ensure-platform-deps / commandExists",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "xdotool or nutjs driver availability is recorded",
+        "screenshot backend availability is recorded",
+        "wmctrl or xdotool window-list backend availability is recorded",
+        "wl-clipboard, xclip, or xsel clipboard backend availability is recorded"
+      ]
+    },
+    {
+      "id": "screenshotCapture",
+      "method": "captureDisplay/capturePrimaryDisplay",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "primary display capture returns non-empty PNG bytes",
+        "captured dimensions match display metadata",
+        "screenshot artifact is attached"
+      ]
+    },
+    {
+      "id": "mouseKeyboardInput",
+      "method": "ComputerUseService desktop actions",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "mouse_move succeeds",
+        "click or double_click succeeds on a harmless target",
+        "type and key actions succeed in a controlled text field",
+        "post-action screenshot is captured when configured"
+      ]
+    },
+    {
+      "id": "windowListFocus",
+      "method": "listWindows/focusWindow",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "listWindows returns visible windows with app/process metadata",
+        "focusWindow focuses a safe target window",
+        "window operation failures include dependency guidance"
+      ]
+    },
+    {
+      "id": "browserAutomation",
+      "method": "browser open/navigate/get/screenshot/close",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "browser target opens or attaches to a supported Chromium browser",
+        "navigate/get/screenshot succeed",
+        "browser cleanup closes the test tab"
+      ]
+    },
+    {
+      "id": "clipboardRoundTrip",
+      "method": "readClipboard/writeClipboard",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "configured Linux clipboard backend writes a test token",
+        "configured Linux clipboard backend reads the same test token",
+        "missing clipboard backend returns ClipboardUnavailableError"
+      ]
+    },
+    {
+      "id": "terminalSafety",
+      "method": "terminal execution safety",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "allowed harmless shell command succeeds",
+        "dangerous or blocked command is rejected",
+        "timeout behavior is recorded"
+      ]
+    },
+    {
+      "id": "approvalMode",
+      "method": "ComputerUseApprovalManager",
+      "status": "requires_device_evidence",
+      "requiredEvidence": [
+        "smart_approve auto-approves read-only actions",
+        "destructive actions require approval",
+        "full_control/approve_all/off modes behave as configured"
+      ]
+    }
+  ]
+}

plugins/plugin-computeruse/docs/macos-desktop-validation.json

@@ -0,0 +1,123 @@
+{
+  "schemaVersion": 1,
+  "platform": "macos-desktop",
+  "status": "passed",
+  "target": {
+    "minimumMacos": "macOS 13 or newer",
+    "requiredPermissions": ["Screen Recording", "Accessibility"],
+    "driver": "nutjs preferred; legacy cliclick/AppleScript fallback allowed"
+  },
+  "evidence": {
+    "machineModel": "Mac16,5",
+    "macosVersion": "26.2",
+    "buildId": "25C56",
+    "validatedAt": "2026-06-26T22:47:17.507Z",
+    "validator": "bun scripts/capture-macos-desktop-evidence.mjs (08255e4731)",
+    "artifacts": [
+      ".github/issue-evidence/9581-macos-desktop-cua/screenshot-primary.png",
+      ".github/issue-evidence/9581-macos-desktop-cua/browser-evidence.png",
+      ".github/issue-evidence/9581-macos-desktop-cua/approval-full-control.txt",
+      ".github/issue-evidence/9581-macos-desktop-cua/macos-desktop-validation.json",
+      ".github/issue-evidence/9581-macos-desktop-cua/report.json",
+      ".github/issue-evidence/9581-macos-desktop-cua/README.md",
+      ".github/issue-evidence/9581-macos-desktop-cua/manifest.json"
+    ]
+  },
+  "checks": [
+    {
+      "id": "capabilityProbe",
+      "method": "detectPlatformCapabilities/getCapabilities",
+      "status": "passed",
+      "requiredEvidence": [
+        "platform is darwin",
+        "reported capabilities: screenshot=available:screencapture (built-in); computerUse=available:cliclick; windowList=available:AppleScript System Events; browser=available:puppeteer-core (Chromium detected); terminal=available:/bin/zsh; fileSystem=available:node:fs; clipboard=available:pbpaste / pbcopy (built-in)",
+        "service capabilities match darwin probe for screenshot=true, windowList=true"
+      ]
+    },
+    {
+      "id": "screenRecordingPermission",
+      "method": "screencapture permission gate",
+      "status": "passed",
+      "requiredEvidence": [
+        "granted Screen Recording permission allowed screenshot capture",
+        "permission-denied screenshot errors are classified by the service when TCC blocks capture",
+        "captured nonblank artifact .github/issue-evidence/9581-macos-desktop-cua/screenshot-primary.png"
+      ]
+    },
+    {
+      "id": "screenshotCapture",
+      "method": "captureDisplay/capturePrimaryDisplay",
+      "status": "passed",
+      "requiredEvidence": [
+        "primary display capture returned 2502814 PNG bytes",
+        "captured dimensions 3456x2234 match display 0",
+        "screenshot artifact .github/issue-evidence/9581-macos-desktop-cua/screenshot-primary.png"
+      ]
+    },
+    {
+      "id": "accessibilityPermission",
+      "method": "Accessibility/TCC gate",
+      "status": "passed",
+      "requiredEvidence": [
+        "granted Accessibility permission allowed TextEdit focus, window bounds, click, key_combo, and type operations",
+        "missing Accessibility permission is classified by desktop service errors when TCC blocks input or System Events access"
+      ]
+    },
+    {
+      "id": "mouseKeyboardInput",
+      "method": "ComputerUseService desktop actions",
+      "status": "passed",
+      "requiredEvidence": [
+        "mouse_move succeeded at 1348,581 on display 0",
+        "click succeeded on a controlled TextEdit document",
+        "key_combo cmd+a succeeded in the controlled text field",
+        "type wrote and verified marker macos-cua-1782514041177",
+        "post-action screenshots were requested by service configuration"
+      ]
+    },
+    {
+      "id": "windowListFocus",
+      "method": "listWindows/focusWindow",
+      "status": "passed",
+      "requiredEvidence": [
+        "listWindows returned 15 visible window(s)",
+        "focusWindow/switchWindow succeeded for Simulator:iPhone 16 Pro",
+        "window operation errors retain permission guidance through service failure results"
+      ]
+    },
+    {
+      "id": "browserAutomation",
+      "method": "browser open/navigate/get/screenshot/close",
+      "status": "passed",
+      "requiredEvidence": [
+        "browser target opened data:text/html,<html><head><title>macOS CUA Evidence</title></head><body><main><h1>macOS CUA Evidence</h1><button id='go'>Ready</button><input id='field' value=''></main></body></html>",
+        "browser_get_dom returned the local evidence page",
+        "browser_get_clickables returned 2 element(s)",
+        "browser screenshot artifact .github/issue-evidence/9581-macos-desktop-cua/browser-evidence.png (1280x900)",
+        "browser cleanup closed the test browser"
+      ]
+    },
+    {
+      "id": "clipboardRoundTrip",
+      "method": "readClipboard/writeClipboard",
+      "status": "passed",
+      "requiredEvidence": [
+        "pbcopy wrote test marker macos-clipboard-1782514118884",
+        "pbpaste read the same test marker",
+        "large payload cap and command failures remain covered by unit tests"
+      ]
+    },
+    {
+      "id": "approvalMode",
+      "method": "ComputerUseApprovalManager",
+      "status": "passed",
+      "requiredEvidence": [
+        "smart_approve auto-approves read-only actions",
+        "smart_approve queues destructive file_write until approval",
+        "full_control auto-approves destructive actions",
+        "approve_all does not auto-approve read-only actions",
+        "off mode denies actions"
+      ]
+    }
+  ]
+}

plugins/plugin-computeruse/docs/windows-desktop-validation.json

@@ -0,0 +1,127 @@
+{
+  "schemaVersion": 1,
+  "platform": "windows-desktop",
+  "status": "passed",
+  "target": {
+    "minimumWindows": "Windows 10 or newer",
+    "driver": "nutjs preferred; legacy PowerShell/user32 fallback allowed",
+    "shell": "PowerShell -NoProfile"
+  },
+  "evidence": {
+    "machineModel": "QEMU Standard PC (Q35 + ICH9, 2009)",
+    "windowsVersion": "Microsoft Windows 11 Pro 10.0.26200",
+    "buildId": "26200",
+    "validatedAt": "2026-06-29T16:56:32.389Z",
+    "validator": "bun scripts/capture-windows-desktop-evidence.mjs (03e0b1fc94)",
+    "artifacts": [
+      ".github/issue-evidence/9581-windows-cua/screenshot-primary.png",
+      ".github/issue-evidence/9581-windows-cua/browser-evidence.png",
+      ".github/issue-evidence/9581-windows-cua/approval-full-control.txt",
+      ".github/issue-evidence/9581-windows-cua/cua-windows-desktop-capture.jpg",
+      ".github/issue-evidence/9581-windows-cua/windows-desktop-validation.json",
+      ".github/issue-evidence/9581-windows-cua/report.json",
+      ".github/issue-evidence/9581-windows-cua/CAPTURE_README.md",
+      ".github/issue-evidence/9581-windows-cua/manifest.json"
+    ]
+  },
+  "checks": [
+    {
+      "id": "capabilityProbe",
+      "method": "detectPlatformCapabilities/getCapabilities",
+      "status": "passed",
+      "requiredEvidence": [
+        "platform is win32",
+        "reported capabilities: screenshot=available:PowerShell System.Drawing; computerUse=available:PowerShell user32.dll; windowList=available:PowerShell Get-Process; browser=available:puppeteer-core (Chromium detected); terminal=available:powershell.exe; fileSystem=available:node:fs; clipboard=available:PowerShell Get-Clipboard",
+        "PowerShell-backed tools identified (screenshot=PowerShell System.Drawing, windowList=PowerShell Get-Process, clipboard=PowerShell Get-Clipboard)"
+      ]
+    },
+    {
+      "id": "screenshotCapture",
+      "method": "captureDisplay/capturePrimaryDisplay",
+      "status": "passed",
+      "requiredEvidence": [
+        "primary display capture returned 845567 PNG bytes",
+        "captured dimensions 1728x1052 match display 0",
+        "screenshot artifact .github/issue-evidence/9581-windows-cua/screenshot-primary.png"
+      ]
+    },
+    {
+      "id": "mouseKeyboardInput",
+      "method": "ComputerUseService desktop actions",
+      "status": "passed",
+      "requiredEvidence": [
+        "launched controlled Notepad (pid 14796); resolved window [46656] \"*RESEARCH-LIST.txt - Notepad\"",
+        "mouse_move succeeded at 758,648 on display 0",
+        "click succeeded inside the controlled Notepad text area",
+        "key_combo ctrl+a succeeded in the controlled text field",
+        "type wrote and verified marker eliza-win-cua-1782752194983 (read back via select-all/copy)",
+        "post-action screenshots were requested by service configuration"
+      ]
+    },
+    {
+      "id": "windowListFocus",
+      "method": "listWindows/focusWindow",
+      "status": "passed",
+      "requiredEvidence": [
+        "listWindows returned 3 visible window(s) with id + title metadata (finding #2 resolved — not 0)",
+        "3 window(s) had a usable id + title; focusWindow/switchWindow succeeded for [52064] \"Claude\"",
+        "window operation failures surface actionable diagnostics through the service result"
+      ]
+    },
+    {
+      "id": "browserAutomation",
+      "method": "browser open/navigate/get/screenshot/close",
+      "status": "passed",
+      "requiredEvidence": [
+        "browser target opened the evidence data URL",
+        "browser_get_dom returned the local evidence page",
+        "browser_get_clickables returned 2 element(s)",
+        "browser screenshot artifact .github/issue-evidence/9581-windows-cua/browser-evidence.png (1280x900)",
+        "browser cleanup closed the test browser"
+      ]
+    },
+    {
+      "id": "clipboardRoundTrip",
+      "method": "readClipboard/writeClipboard",
+      "status": "passed",
+      "requiredEvidence": [
+        "Set-Clipboard wrote test marker windows-clipboard-1782752210096",
+        "Get-Clipboard -Raw read the same test marker (trailing newline normalized)",
+        "large payload cap and command failures remain covered by unit tests"
+      ]
+    },
+    {
+      "id": "terminalSafety",
+      "method": "PowerShell terminal execution safety",
+      "status": "passed",
+      "requiredEvidence": [
+        "allowed harmless PowerShell command 'echo eliza-term-1782752210157' succeeded and echoed the token",
+        "dangerous 'Remove-Item -Recurse 'C:\\'' was rejected: Command blocked: PowerShell recursive deletion of drive root.",
+        "a 1s timeout killed a 5s Start-Sleep in 1045ms (timeout enforced)"
+      ]
+    },
+    {
+      "id": "approvalMode",
+      "method": "ComputerUseApprovalManager",
+      "status": "passed",
+      "requiredEvidence": [
+        "smart_approve auto-approves read-only actions",
+        "smart_approve queues destructive file_write until approval",
+        "full_control auto-approves destructive actions",
+        "approve_all does not auto-approve read-only actions",
+        "off mode denies actions"
+      ]
+    },
+    {
+      "id": "windowsHardeningRegression",
+      "method": "Windows hardening evidence replay",
+      "status": "passed",
+      "requiredEvidence": [
+        "primary-screen-size PowerShell command runs Add-Type System.Windows.Forms before using [System.Windows.Forms.Screen]",
+        "degenerate 0x0 capture region rejected with: Invalid capture region: width and height must be positive (got 0x0).",
+        "UNC network path blocked and a safe local temp path allowed by validateFilePath",
+        "linked real-driver screenshot evidence: .github/issue-evidence/9581-windows-cua/cua-windows-desktop-capture.jpg, .github/issue-evidence/9581-windows-cua/screenshot-primary.png"
+      ]
+    }
+  ]
+}