# @elizaos/plugin-computeruse Desktop automation for elizaOS agents — screenshots, mouse/keyboard control, browser CDP automation, window management, and a multi-display scene model. Ported from [`coasty-ai/open-computer-use`](https://github.com/coasty-ai/open-computer-use) (Apache 2.0). ## Purpose / role Adds real desktop control to an Eliza agent: taking screenshots, clicking/typing/scrolling, managing windows, automating web browsers via CDP (puppeteer-core), and building a structured `Scene` from displays, accessibility tree, OCR, and process list so the agent can reason about what is visible. Opt-in: the plugin auto-enables only when `COMPUTER_USE_ENABLED=1` is set (see `autoEnable` in `src/index.ts`). Requires a headful display session on macOS/Linux; headless browser mode is supported. File operations belong to the FILE action; shell/terminal access belongs to the SHELL action — this plugin does not expose them. ## Plugin surface ### Actions | Name | File | What it does | |------|------|--------------| | `COMPUTER_USE` | `src/actions/use-computer.ts` | Umbrella desktop action: `screenshot`, `click`, `click_with_modifiers`, `double_click`, `right_click`, `mouse_move`, `middle_click`, `mouse_down`, `mouse_up`, `type`, `key`, `key_combo`, `key_down`, `key_up`, `scroll`, `drag` (single segment or multi-point `path`), `get_cursor_position`, `detect_elements`, `ocr`, `open` (file/URL/folder via the OS default handler), `launch` (start an app → returns pid). `mouse_down/up` + `key_down/up` are real press-and-hold primitives (nutjs `pressButton`/`pressKey` without release — back hold-drags, marquee, held modifiers); they require the nutjs driver. Requires `OWNER` role. Subactions are promoted to virtual top-level actions (`COMPUTER_USE_CLICK`, etc.) via `promoteSubactionsToActions`. | | `WINDOW` | `src/actions/window.ts` | Window management: `list`, `focus`, `switch`, `arrange`, `move`, `minimize`, `maximize`, `restore`, `close`, `get_current_window_id`, `get_application_windows`, `set_bounds` (position + size). Also promoted to `WINDOW_FOCUS`, etc. | | `CLIPBOARD` | `src/actions/clipboard.ts` | Host clipboard read/write (`read`, `write`) — trycua/cua parity. Promoted to `CLIPBOARD_READ` / `CLIPBOARD_WRITE`. macOS pbcopy/pbpaste, Linux wl-clipboard/xclip, Windows PowerShell `Get-Clipboard` / `Set-Clipboard`. | | `COMPUTER_USE_AGENT` | `src/actions/use-computer-agent.ts` | High-level "give me a goal, click my way there" loop (WS7). Selects an agent loop by model string (#9170 M10, default `local-grounder` = Brain → Cascade) and runs predictStep → dispatch up to `maxSteps`, through a callback-middleware pipeline (#9170 M11: operator-normalizer + trajectory by default; `maxDurationMs` budget cap + `imageRetentionLast` window opt-in). Emits trajectory events as structured log lines and on `report.trajectory`. | ### Providers | Name | File | What it injects | |------|------|----------------| | `computerState` | `src/providers/computer-state.ts` | Platform info, screen dimensions, available capabilities, recent actions, approval queue. Gate: `browser/files/terminal/automation/admin` contexts. | | `scene` | `src/providers/scene.ts` | Live desktop scene (displays, focused window, apps, OCR boxes, AX nodes, VLM annotations) via `SceneBuilder`. Refreshed once per turn; serialized as token-efficient JSON fence. Gate: `browser/automation/admin` contexts. | ### Services | Name | `serviceType` | File | What it does | |------|--------------|------|--------------| | `ComputerUseService` | `"computeruse"` | `src/services/computer-use-service.ts` | Central service: input dispatch, screenshot capture, browser CDP session, window ops, approval-manager wiring, `SceneBuilder` lifecycle. | | `VisionContextProvider` | `"vision-context"` | `src/services/vision-context-provider.ts` | Surfaces a `VisionContext` snapshot (open apps, focused window, recent actions, current task goal) for downstream consumers (e.g. plugin-vision). | ### Routes All paths are under `/api/computer-use/` and implemented in `src/routes/computer-use-compat-routes.ts`: | Method | Path | Description | |--------|------|-------------| | GET | `/api/computer-use/approvals` | List pending approval requests | | GET | `/api/computer-use/approvals/stream` | SSE stream of approval events (public) | | POST | `/api/computer-use/approval-mode` | Change the active `COMPUTER_USE_APPROVAL_MODE` | | POST | `/api/computer-use/approvals/:id` | Approve or deny a pending action | ## Layout ``` src/ index.ts Plugin entry — assembles and exports computerUsePlugin types.ts All shared types (ApprovalMode, ComputerUseConfig, DesktopActionParams, …) approval-manager.ts ComputerUseApprovalManager — queues pending actions, applies approval mode register-routes.ts Route registration helper actions/ use-computer.ts COMPUTER_USE parent action + subaction table use-computer-agent.ts COMPUTER_USE_AGENT (WS7 autonomous loop) window.ts WINDOW parent action window-handlers.ts Per-verb handlers called by window.ts clipboard.ts clipboardAction (CLIPBOARD parent action) — registered in index.ts; promoted to CLIPBOARD_READ / CLIPBOARD_WRITE helpers.ts resolveActionParams, buildScreenshotAttachment, … actor/ WS7 autonomous desktop loop brain.ts Brain — sends display PNGs to IMAGE_DESCRIPTION model, produces BrainOutput cascade.ts Cascade (ScreenSeekeR) — Brain → Actor → ProposedAction actor.ts Actor interface + OsAtlasProActor + OcrCoordinateGroundingActor dispatch.ts dispatch() — executes a ProposedAction via ComputerInterface computer-interface.ts ComputerInterface abstraction + makeComputerInterface() aosp-input-actor.ts AOSP-specific actor types.ts BrainOutput, ProposedAction, … index.ts Public re-exports platform/ browser.ts Puppeteer-core CDP browser automation capture.ts captureDisplay / captureAllDisplays displays.ts listDisplays / getPrimaryDisplay driver.ts driverClick / driverType / … (nutjs or legacy shell) nut-driver.ts @nut-tree-fork/nut-js implementation windows-list.ts listWindows / focusWindow / arrangeWindows / getActiveWindow / resizeWindow / … launch.ts openTarget (open file/URL/folder) / launchApp (spawn app → pid) clipboard.ts OS clipboard read/write a11y.ts Accessibility tree query coords.ts localToGlobal coordinate translation wayland-portal.ts xdg-desktop-portal screenshot sidecar for Wayland capabilities.ts detectPlatformCapabilities desktop.ts High-level desktop helpers helpers.ts Shared platform utilities permissions.ts classifyPermissionDeniedError process-list.ts listProcesses / parsePsOutput screenshot-quality.ts Quality / compression settings for screenshots terminal.ts Terminal session management (internal; not exposed as action) file-ops.ts File primitives (internal; not exposed as action) screenshot.ts Low-level screencapture wrappers security.ts Path + command security checks providers/ computer-state.ts computerStateProvider scene.ts sceneProvider routes/ computer-use-routes.ts handleComputerUseRoutes (full route table) computer-use-compat-routes.ts computerUseRouteHandler() (compat wrapper used by plugin entry) sandbox-routes.ts handleSandboxRoute scene/ scene-builder.ts SceneBuilder — composes displays + a11y + OCR into Scene scene-types.ts Scene, SceneApp, SceneAppWindow, SceneAxNode, … a11y-provider.ts DarwinAccessibilityProvider / LinuxAccessibilityProvider / … apps.ts enumerateApps / joinAppsAndWindows dhash.ts Perceptual hash / dirty-block diffing for change detection screen-state.ts ScreenState + ScreenStateStore — one shared capture/turn (dHash/blockGrid + change events) ocr-adapter.ts OcrProvider / CoordOcrProvider adapter seam serialize.ts serializeSceneForPrompt — token-efficient JSON fence services/ computer-use-service.ts ComputerUseService (serviceType = "computeruse") vision-context-provider.ts VisionContextProvider (serviceType = "vision-context") desktop-control.ts Low-level desktop control primitives + DesktopControl* types index.ts Barrel re-exports for services/ mobile/ ocr-provider.ts OcrProvider / CoordOcrProvider interfaces (plugin-vision contributes impls) ios-bridge.ts iOS computer-use bridge ios-computer-interface.ts iOS-specific ComputerInterface implementation ios-app-intent-registry.ts Registry of iOS app intents for automation android-bridge.ts AOSP input bridge android-scene.ts Android scene capture and representation android-trajectory.ts Android action trajectory recording mobile-computer-interface.ts MobileComputerInterface mobile-screen-capture.ts Screen capture abstraction for mobile targets index.ts Mobile public surface osworld/ adapter.ts OSWorld benchmark adapter action-converter.ts OSWorld action → ComputerInterface translation types.ts OSWorld-specific type definitions parity/ trycua/cua parity tooling (#9170 M14) parity-matrix.ts machine-checkable capability matrix + validateParityMatrix (guards drift vs the live action surface) screenspot.ts ScreenSpot grounding harness (point-in-bbox scorer + score fold) index.ts Public re-exports sandbox/ sandbox-driver.ts Sandbox driver (backend-agnostic) docker-backend.ts Docker backend remote-guest.ts Generic {command,params}→{success,result} RPC seam + RemoteGuestBackend base (#9170 M13) wsb-backend.ts Windows Sandbox provider (RemoteGuestBackend) qemu-backend.ts QEMU provider (RemoteGuestBackend) surface-types.ts Shared surface type definitions types.ts Sandbox-specific types index.ts Public re-exports security/ browser-script-policy.ts GHSA-rcvr-766c-4phv — browser_execute disabled by default ``` ## Commands Scripts are defined in `package.json`; run them from the repo root with `bun run --cwd`: ```bash bun run --cwd plugins/plugin-computeruse build # build package artifacts bun run --cwd plugins/plugin-computeruse typecheck # TypeScript typecheck bun run --cwd plugins/plugin-computeruse lint # mutating Biome check bun run --cwd plugins/plugin-computeruse lint:check # read-only Biome check bun run --cwd plugins/plugin-computeruse format # write formatting bun run --cwd plugins/plugin-computeruse format:check # read-only formatting check bun run --cwd plugins/plugin-computeruse test # run package tests bun run --cwd plugins/plugin-computeruse capture:macos-desktop-evidence # macos-desktop-evidence evidence capture bun run --cwd plugins/plugin-computeruse capture:windows-desktop-evidence # windows-desktop-evidence evidence capture bun run --cwd plugins/plugin-computeruse postinstall # postinstall setup bun run --cwd plugins/plugin-computeruse record:windows-cua-input # bun scripts/record-windows-cua-input.mjs bun run --cwd plugins/plugin-computeruse validate:android-aosp-evidence # android-aosp-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:android-device-evidence # android-device-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:ios-device-evidence # ios-device-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:linux-desktop-evidence # linux-desktop-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:macos-desktop-evidence # macos-desktop-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:platform-evidence # platform-evidence evidence validation bun run --cwd plugins/plugin-computeruse validate:windows-desktop-evidence # windows-desktop-evidence evidence validation ``` ## Config / env vars All read via `runtime.getSetting()` / `process.env`. Core vars are declared in `package.json#agentConfig.pluginParameters`; sandbox vars are read directly. | Env var | Type | Default | Required | Description | |---------|------|---------|----------|-------------| | `COMPUTER_USE_ENABLED` | boolean | `false` | No | Master toggle; also controls `autoEnable` | | `COMPUTER_USE_SCREENSHOT_AFTER_ACTION` | boolean | `true` | No | Auto-capture screenshot after each desktop action | | `COMPUTER_USE_ACTION_TIMEOUT_MS` | number | `10000` | No | Per-action timeout in ms | | `COMPUTER_USE_APPROVAL_MODE` | enum | `"smart_approve"` | No | `full_control` / `smart_approve` / `approve_all` / `off` | | `COMPUTER_USE_BROWSER_HEADLESS` | boolean | `false` | No | Headless browser (useful in CI) | | `ELIZA_COMPUTERUSE_DRIVER` | enum | `"nutjs"` | No | Input driver: `nutjs` (@nut-tree-fork/nut-js) or `legacy` (cliclick/xdotool/PowerShell) | | `COMPUTER_USE_MODE` | enum | `"yolo"` | No | Runtime mode: `yolo` (direct desktop) or `sandbox` (Docker-isolated). Alias: `COMPUTERUSE_MODE` | | `COMPUTER_USE_SANDBOX_BACKEND` | enum | — | No | Sandbox backend when `COMPUTER_USE_MODE=sandbox`: `"docker"`, `"wsb"` (Windows Sandbox), or `"qemu"`. `docker`/`qemu` need `COMPUTER_USE_SANDBOX_IMAGE`; `wsb` is imageless. Alias: `COMPUTERUSE_SANDBOX_BACKEND` | | `COMPUTER_USE_SANDBOX_RPC_URL` | string | — | No | VM-provider (wsb/qemu) in-guest computer-server RPC URL (#9170 M13). Default `http://127.0.0.1:/cua`. Alias: `COMPUTERUSE_SANDBOX_RPC_URL` | | `COMPUTER_USE_SANDBOX_RPC_PORT` | number | `8000` | No | Host-forwarded guest RPC port for wsb/qemu. Alias: `COMPUTERUSE_SANDBOX_RPC_PORT` | | `COMPUTER_USE_SANDBOX_IMAGE` | string | — | No | Docker image to use for sandbox mode. Alias: `COMPUTERUSE_SANDBOX_IMAGE` | Power-user escape hatches read directly via `process.env` (not declared as plugin parameters): | Env var | Type | Default | Description | |---------|------|---------|-------------| | `COMPUTERUSE_PS_HOST` | `0` to disable | enabled | Windows-only warm PowerShell host (`ps-host.ts`) that amortizes the cold-spawn AV tax. Set `0` to force every spawn through one-shot `powershell.exe`. | | `ELIZA_COMPUTERUSE_PS_TIMEOUT_MS` | number | unset | Windows-only **floor** for every PowerShell/WinRT spawn budget (capture, clipboard, ps-host startup, window enumeration/ops in `windows-list.ts`). Applied via `psSpawnTimeoutMs` — only ever RAISES a call site's default, never lowers it. Raise it on Defender-heavy hosts where cold `powershell.exe` spawns exceed the default budgets and false-fail with `ETIMEDOUT` (#9581). Mirrors `ELIZA_WAYLAND_PORTAL_TIMEOUT_MS`. | `BROWSER_EXECUTE_DISABLED` is declared in `package.json#agentConfig.pluginParameters` but is **inert**: `browser_execute` is unconditionally disabled in `src/security/browser-script-policy.ts` (`isBrowserExecuteAllowed()` always returns `false`, GHSA-rcvr-766c-4phv). No setting re-enables it. ## How to extend ### Add a new desktop action verb to COMPUTER_USE 1. Add the verb string to the `action` enum in `src/types.ts` (`DesktopActionType`). 2. Add a handler branch in the `ComputerUseService` dispatch switch in `src/services/computer-use-service.ts`. 3. Add the low-level platform function in the appropriate `src/platform/*.ts` file. 4. Update the `parameters[].schema.enum` in `src/actions/use-computer.ts` so the planner sees it. 5. `promoteSubactionsToActions` will auto-promote `COMPUTER_USE_` — no extra registration needed. ### Add a new window operation Follow the same pattern in `src/actions/window.ts` / `src/actions/window-handlers.ts` / `src/platform/windows-list.ts`. ### Add a new provider 1. Create `src/providers/.ts` exporting a `Provider` object. 2. Import and add it to the `providers` array in `src/index.ts`. ### Add a route Add a `Route` object to `computerUseRoutes` in `src/index.ts`, implement the handler in `src/routes/`. ### Register an OCR provider (from another plugin) Call the module-level `registerCoordOcrProvider(provider)` exported from `src/mobile/ocr-provider.ts` (not a method on `ComputerUseService`). plugin-vision does this at boot to contribute its hierarchical OCR adapter. ### Register a Set-of-Marks provider (from another plugin) Call `registerSetOfMarksProvider(provider)` (same module, `src/mobile/ocr-provider.ts`). When registered, `detect_elements` prefers it: it returns a deduplicated, **1-indexed** set of numbered marks (GGUF YOLO icons + OCR text fused, icon-over-text suppression + NMS) plus an optional numbered-overlay PNG under `data.setOfMarks.overlay`. Each mark carries a `center` click target the VLM's chosen number resolves to. plugin-vision contributes the implementation (`som.ts` + `set-of-marks-provider.ts`); with no provider registered, `detect_elements` falls back to OCR-only text elements. ### Register an agent loop (model-string → loop) `COMPUTER_USE_AGENT` selects its loop from a model string via the registry in `src/actor/agent-loop.ts` (#9170 M10) — trycua/cua parity. Every loop implements the same `predictStep` (observe + plan the next action) / `predictClick` (ground a target to a coordinate) seam. The built-in `local-grounder` wraps the existing Brain → Cascade (ScreenSeekeR) and exposes the M5 grounding cache through `predictClick`; it is the match-anything fallback. Register a provider-specific loop (Anthropic / OpenAI `computer-use-preview` / a remote grounder) with `registerAgentLoop({ name, matches: matchesModelFamily("anthropic"), create, priority })`; the runner reads the active model string from the `COMPUTER_USE_AGENT_LOOP` setting/env (default `local-grounder`). ## Conventions / gotchas - **Approval flow**: every destructive action passes through `ComputerUseApprovalManager`. The default mode is `smart_approve` — only read-only `SAFE_COMMANDS` auto-approve, and destructive verbs (terminal execute, file write/delete) require explicit human approval. Switch to `full_control` (auto-approve everything), `approve_all`, or `off` (deny all) via env or the `/api/computer-use/approval-mode` route. - **`browser_execute` is always disabled** (GHSA-rcvr-766c-4phv) — `isBrowserExecuteAllowed()` returns `false` unconditionally; no setting re-enables it. Use `dom`, `clickables`, `click`, `type`, `navigate`, `screenshot` browser subactions instead. - **Coordinate system**: each display has its own local coordinate space. `src/platform/coords.ts` translates local→global when needed. Always pass `displayId` when targeting a specific monitor. - **nutjs native bindings**: `@nut-tree-fork/nut-js` requires native compilation. If the build fails, set `ELIZA_COMPUTERUSE_DRIVER=legacy` to fall back to shell tools. - **Scene is per-turn**: `sceneProvider` calls `SceneBuilder.onAgentTurn()` once per turn. Code that needs the scene outside a provider turn should call `ComputerUseService.getCurrentScene()` or `refreshScene("active")`. - **WS7 trajectory events**: `COMPUTER_USE_AGENT` emits `logger.info` lines with `evt: "computeruse.agent.step"`. These are picked up by plugin-trajectory-logger via log capture — no direct dependency. - **Platform evidence**: `docs/ios-device-validation.json`, `docs/android-device-validation.json`, `docs/android-aosp-validation.json`, `docs/macos-desktop-validation.json`, `docs/linux-desktop-validation.json`, and `docs/windows-desktop-validation.json` are the release evidence manifests. Keep incomplete live-device checks in `requires_device_evidence`; use `validate:platform-evidence -- --require-complete` only for release gates that truly have artifacts for every required platform check. - **Mobile surface**: `src/mobile/` is real but constrained. Read `docs/IOS_CONSTRAINTS.md` and `docs/ANDROID_CONSTRAINTS.md` before touching mobile code. - **OSWorld benchmark**: `src/osworld/` adapts the plugin to the OSWorld desktop benchmark format. Not part of normal agent runtime. - **Parity matrix (#9170 M14)**: `src/parity/parity-matrix.ts` is the machine-checkable trycua/cua capability map. `validateParityMatrix(actionNames)` is asserted in the test suite — adding a verb to the matrix without registering it (or renaming a promoted action) fails CI, so the map can't silently drift. `src/parity/screenspot.ts` scores click-grounding (point-in-bbox) for any grounder. - **Further reading**: `docs/MULTI_MONITOR.md`, `docs/SCENE_BUILDER.md`, `docs/MOBILE_ASSISTANT_ROUTING.md`, `docs/AOSP_SYSTEM_APP.md`, `docs/TEST_LANES_COMPUTERUSE_VISION.md` (unit vs real-driver lanes, per-OS reqs, and the Windows non-interactive-session input gotcha). ## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests > The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it. > Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it > works **without reading the code**, from the artifacts you attach. This applies to **every** > feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof. - **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then **open the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence (`packages/scenario-runner/bin/eliza-scenarios run --report `). - **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that drive the *real* path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test **does not count**. If the real model/device/chain/connector/account is hard to reach, **make it reachable — that is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change. - **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of the entire feature or view, start to finish (`bun run test:e2e:record`). - **Manually review every artifact the change touches** — never just the green check: client logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out, before/after full-page screenshots, **and the domain artifacts listed below for this package.** - **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every blocker by the **hard path**: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted. Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `
`); attach each evidence type **or** explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed behavior, **re-capture** evidence; stale proof is worse than none. **Capture & manually review for this package — CLI / tooling:** - The real command/flow invocation transcript (args in, stdout/stderr, exit code) and the artifacts it generated (files, scaffolds, manifests, screenshots/recordings). - Failure paths: bad args, missing deps, partial state, permission/network errors. - A recording/log of the actual run end to end — not a unit test of one helper. - Any model interaction captured as a live trajectory and reviewed.