1jehuang--jcode/docs/proposals/computer-use-maximal-control.md

Roadmap: maximal macOS control for the computer tool

Goal: give the agent as much reliable control over macOS as the platform allows, including background control that does not disturb what the user is looking at.

This builds on the v1 computer tool (PR #345): screenshot, coordinate mouse/keyboard, scroll, AX-tree read, cursor, permission check.

Everything below is implementable in Rust with crates that are already in the lockfile or available on crates.io (accessibility-sys, screencapturekit, objc2-app-kit, core-graphics). No Swift/ObjC build step.

---

The one hard constraint

macOS has one HID cursor and one keyboard-focus shared by the whole login session. Synthetic coordinate input (CGEvent) is therefore always visible: it moves the real cursor and types into the focused app.

Background / not-in-view control must avoid CGEvent and instead go through:

1. Accessibility (AX) actions - act on a specific element by reference.
2. Apple Events / scripting - drive scriptable apps with no UI.
3. Per-window capture - "see" a window without raising it.

True simultaneous "you work + I work independently" needs a separate display or login session (see Tier 4).

---

Tier 0 - done (v1, PR #345)

• screenshot (main display, point/pixel scale aware)
• move / click / double_click / right_click / drag / scroll
• type / key (chords)
• ui (AX tree read via osascript)
• cursor, check_permissions

Tier 1 - AX semantic actions ← highest leverage for background control

Read + act on elements by reference, no cursor movement, target app need not be frontmost. Uses accessibility-sys (AXUIElementPerformAction, AXUIElementSetAttributeValue, AXUIElementCopyElementAtPosition, AXUIElementCopyAttributeValue).

• find_element { role?, title?, value?, pid?, app? } -> stable element handles
• element_at { x, y } -> element under a point (AXUIElementCopyElementAtPosition)
• press { element } -> AXPress (click a button in a background window)
• set_value { element, value } -> type into a field without focus
• get_value { element }
• perform_action { element, ax_action } -> any advertised AX action
• select_menu { app, path: ["File","Export…"] } -> drive the menu bar of any app

Handle format: pid + AX path (index chain) or a session-scoped element id cache, so the model can act structurally instead of by pixels.

Why it matters: this is the actual "click things you're not looking at" capability.

Tier 2 - app / window / system management

Mostly objc2-app-kit (NSWorkspace, NSRunningApplication) + AX window attributes + CoreGraphics window list.

• list_apps / activate_app { app } / hide_app / quit_app
• list_windows { pid? } (CGWindowList) with ids, titles, bounds, on/off-screen
• focus_window / move_window / resize_window / minimize_window / close_window (AX window actions - can target background windows)
• window_screenshot { window_id } -> capture a specific window even if occluded (CGWindowListCreateImage now, ScreenCaptureKit later)
• spaces awareness (which Space an app is on; activating may switch Spaces - visible)

Tier 3 - clipboard, input fidelity, observation

• get_clipboard / set_clipboard { text } (NSPasteboard via objc2-app-kit)
• key_down / key_up (hold modifiers, game-style input)
• type_into { element, text } (AX set value + confirm) for reliability over blind typing
• wait_for { element|condition, timeout } using AXObserver* notifications (e.g. wait for a sheet to appear) instead of sleep-and-poll
• paste_type { text } - set clipboard + Cmd-V for fast/large text entry

Tier 4 - true background / parallel operation (advanced)

These give genuinely off-screen, non-interfering control. Higher setup cost.

• Apple Events scripting bridge: run_applescript { script } / run_jxa. Fully headless for scriptable apps (Mail, Notes, Safari, Finder, Music, System Settings panes, Terminal, many pro apps). No cursor, no focus. Per-app Automation permission (prompts on first use).
• Virtual / headless display: route the agent's cursor+windows to a second (virtual) display the user isn't looking at. Options: a virtual display driver (e.g. BetterDisplay/CGVirtualDisplay private API) or a real unused monitor. Lets the agent move windows there and use coordinate input without touching the user's screen.
• Separate login / Screen Sharing session: a second macOS session has its own cursor and focus; the agent drives that one. Strongest isolation, most setup.
• Shortcuts integration: invoke the user's Shortcuts automations (shortcuts run …) as high-level, sanctioned actions.

Tier 5 - sensors / extras (optional, opt-in)

• ocr { region|window } via Vision framework (read text in images / non-AX apps).
• screen_record { seconds } short clips via ScreenCaptureKit.
• Audio in/out control, notifications, do_not_disturb toggling via scripting.
• Camera/mic are separate TCC permissions; keep strictly opt-in.

---

Permissions (TCC) - the gatekeeping reality

Permission	Unlocks	Auto-grantable?
Accessibility	CGEvent input, all AX read/act, window control	No - user toggles once (we can prompt + deep-link)
Screen Recording	screenshots, window/ocr capture	Request API exists (CGRequestScreenCaptureAccess)
Automation (Apple Events)	scripting each app	Prompts per target app on first send
Input Monitoring	reading global input stream (only if we add capture)	Request API exists

Plan: a request_permissions action that calls AXIsProcessTrustedWithOptions(prompt=true) (adds jcode to the list + shows the dialog) and deep-links to the exact System Settings pane, then polls AXIsProcessTrusted(). One prompt + one toggle; never zero-touch for Accessibility (Apple's anti-malware boundary).

Important: the permission attaches to the host binary/terminal running jcode. For a stable experience we likely want a signed jcode.app with a fixed bundle id so the grant persists across updates (otherwise each new binary path re-prompts).

Safety model (high blast radius)

• Gated like bash: refuses early if required permission missing.
• dry_run on mutating actions: resolve + report target without acting.
• Prefer AX semantic actions over blind coordinate clicks (auditable, robust).
• Screenshot/element echo on destructive coordinate clicks.
• No global input capture unless explicitly enabled (keeps us out of Input Monitoring by default).
• Per-action audit log; optional allowlist/denylist of target apps.

Suggested build order

1. Tier 1 (AX actions) - biggest capability jump, enables background control.
2. Tier 2 window mgmt + per-window screenshot - "see and act on hidden windows".
3. Tier 3 clipboard + AXObserver waits - reliability.
4. run_applescript/JXA bridge (Tier 4) - headless scripting for many apps.
5. Virtual-display / second-session (Tier 4) - true parallel, non-interfering.
6. Signed jcode.app bundle for durable permissions.
7. Vision OCR (Tier 5) as needed.

Crates

• accessibility-sys 0.2 (AX read/act/observe) - on crates.io
• screencapturekit 7 (modern capture) - on crates.io; core-graphics window list as fallback
• objc2-app-kit / objc2-foundation 0.3 - already in lockfile (NSWorkspace, NSPasteboard)
• core-graphics 0.23 - already a direct dep (CGEvent, CGWindowList, CGDisplay)

---

Tool interface design (decided)

Single tool, progressive disclosure

One computer tool, action-dispatched (like browser). To keep always-on context flat regardless of how many tiers exist, the schema uses progressive disclosure:

• Always-on core (~370 tokens, measured with tiktoken cl100k_base): screenshot, ui, ocr, click, type, key, press, set_value, run_applescript, setup, discover.
• discover { category } returns full specs for advanced actions on demand (mouse|keyboard|ax|windows|apps|clipboard|scripting|displays|system|all), ~130 tokens per category, paid only when used.
• Shared handle types (element, window_id, region) defined once and reused, so params do not multiply with actions.

Measured always-on cost:

Design	Actions visible	Always-on tokens
Current v1 tool	12	~720
Flat, all tiers (~46 actions)	46	~1,020
Progressive core	11	~370

Background control is a property of the mechanism, not the tier: CGEvent = visible; AX actions (press/set_value/select_menu) + Apple Events = background.

setup / check_permissions action

A first-class setup action that:

1. Reports status of every requirement: Accessibility (AXIsProcessTrusted), Screen Recording (CGPreflightScreenCaptureAccess), Automation (per-app, via first Apple Event), plus install/bundle health.
2. Requests what it can programmatically:
   • AXIsProcessTrustedWithOptions(prompt=true) — shows the Accessibility dialog and pre-adds jcode to the list (toggled off).
   • CGRequestScreenCaptureAccess() — prompts for Screen Recording.
   • First Apple Event to a target app — triggers its Automation prompt.
3. Deep-links to the exact System Settings pane for anything still missing:
   • x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility
   • …?Privacy_ScreenCapture
   • …?Privacy_Automation
4. Polls AXIsProcessTrusted() until granted, then reports "ready".

Hard limit: the Accessibility toggle itself cannot be flipped by any API (Apple anti-malware boundary). tccutil can only reset, not grant. So the best achievable UX is "one or two prompts + one toggle," never zero-touch.

Durable permissions: signed app bundle

TCC permissions attach to the running binary's identity. A bare dev/cli binary changes path/signature across updates, so macOS re-prompts every time. To make a grant stick:

• Ship/install jcode as a signed .app bundle with a stable bundle id (e.g. com.jcode.app) and a Designated Requirement, so the Accessibility / Screen Recording grant persists across updates.
• setup should detect "running from an unstable/unsigned path" and offer to install the proper bundle, so the user grants once.

Build order (updated)

1. Progressive-disclosure refactor of the v1 tool (core + discover).
2. setup action (check + request + deep-link + poll).
3. Tier 1 AX actions (background control).
4. Tier 2 window/app management + per-window screenshot.
5. Tier 3 clipboard + AXObserver waits.
6. run_applescript/JXA bridge (Tier 4 headless scripting).
7. Signed app bundle for durable permissions.
8. Tier 5 OCR (Vision). (Camera/audio intentionally excluded.)
9. Virtual-display / second-session for true parallel work (advanced).