1jehuang--jcode/docs/proposals/computer-use-tool.md

# Proposal: native `computer` tool for macOS computer use

## Summary

Add a single native tool, **`computer`**, that lets the agent observe and control
the macOS GUI — screenshots, the accessibility (AX) tree, mouse/keyboard input,
window/app management, and clipboard — through one `action`-dispatched interface.

This mirrors the existing **`browser`** tool (`crates/jcode-app-core/src/tool/browser.rs`):
one registered tool, an `action: String` that selects a sub-operation, with optional
typed params. It gives jcode a closed control loop (*see screen → decide → act*)
without depending on a browser or external automation tooling.

## Motivation

- The agent can already drive a browser; it cannot drive native macOS apps, system
  UI, or anything outside the browser sandbox.
- "Computer use" agents need exactly three primitives: **read the screen**, **read
  UI structure**, and **synthesize input**. macOS exposes all three through the
  Accessibility / Quartz Event Services / ScreenCaptureKit stack.
- A single, well-scoped tool keeps the tool surface small and the permission story
  in one place.

## Architecture

```
crates/jcode-macos-control/        (new) cfg(target_os = "macos") platform crate
  └─ AX (accessibility-sys), CGEvent (core-graphics),
     CoreFoundation (core-foundation), screenshots (ScreenCaptureKit / CGDisplay),
     app/window control (objc2 + objc2-app-kit), clipboard (objc2 NSPasteboard)

crates/jcode-app-core/src/tool/computer.rs   (new) ComputerTool
  └─ thin dispatch layer: parse input -> call jcode-macos-control -> ToolOutput
  └─ registered in crates/jcode-app-core/src/tool/mod.rs base_tools()
```

- All native APIs are reached through existing Rust bindings (`objc2`,
  `accessibility-sys`, `core-graphics`, `core-foundation`) — **no Swift/ObjC build
  step**.
- On non-macOS targets the tool still registers but every action returns a clean
  `unsupported on this platform` error, so the tool list stays stable across OSes.
- `screenshot` returns its image via `ToolOutput::with_image` (base64), matching how
  `browser` returns screenshots today.

## Permissions (the important part)

macOS splits this across **four** TCC permissions. Programmatic *request* support
differs per permission:

| Permission | Used for | Programmatic request |
|---|---|---|
| **Accessibility** | drive other apps' UI, inject `CGEvent` input | ⚠️ prompt + deep-link only; user must toggle |
| **Screen Recording** | screenshots / `get_ui_tree` of some apps | ✅ `CGRequestScreenCaptureAccess()` |
| **Input Monitoring** | reading the global input stream | ✅ `IOHIDRequestAccess(...)` |
| **Automation** (Apple Events) | scripting cooperating apps | ✅ prompts on first send, per target app |

**Accessibility is the one that cannot be auto-granted** (Apple's anti-malware
boundary), and it is required for input injection. Best achievable flow, exposed via
the `request_permissions` action:

1. `AXIsProcessTrustedWithOptions([kAXTrustedCheckOptionPrompt: true])` — shows the
   system dialog *and auto-adds jcode to the Accessibility list* (toggled off).
2. Deep-link to the exact pane:
   `open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"`.
3. Poll `AXIsProcessTrusted()` until granted, then report ready.

So the experience becomes **one prompt + one toggle**, not "go hunt in Settings" —
but never zero-touch for Accessibility.

`check_permissions` reports the current state of all four so the agent can tell the
user precisely what is missing before attempting control.

## Actions

`action` (required) selects the operation. Params below are optional and validated
per action.

**Permissions**
- `check_permissions` → status of accessibility / screen-recording / input-monitoring
- `request_permissions` → prompt + deep-link flow above

**Observe**
- `screenshot` — `{ display?, window_id?, region? }` → image
- `get_ui_tree` — `{ pid? | frontmost, max_depth? }` → serialized AX tree (role, title, value, position, size, actions)
- `find_element` — `{ role?, title?, value?, pid? }` → matching elements + their identifiers
- `element_at` — `{ x, y }` → element under the point

**Mouse**
- `move` — `{ x, y }`
- `click` / `double_click` / `right_click` — `{ x?, y? }` (current position if omitted)
- `drag` — `{ from: [x,y], to: [x,y] }`
- `scroll` — `{ x?, y?, dx, dy }`

**Keyboard**
- `type` — `{ text }`
- `key` — `{ keys: "cmd+shift+4" }` (chord)
- `key_down` / `key_up` — `{ key }`

**Semantic AX (preferred over raw input when available)**
- `press` — `{ element }` (AXPress)
- `set_value` — `{ element, value }`
- `get_value` — `{ element }`
- `perform_action` — `{ element, ax_action }`
- `select_menu` — `{ app, path: ["File", "Export…"] }`

**Window / app**
- `list_apps`, `activate_app` `{ app }`
- `list_windows` `{ pid? }`, `focus_window` `{ window }`
- `move_window` `{ window, x, y }`, `resize_window` `{ window, w, h }`
- `minimize_window` / `close_window` `{ window }`

**Clipboard**
- `get_clipboard`, `set_clipboard` `{ text }`

> Element identifiers: `find_element` / `get_ui_tree` return stable-enough handles
> (e.g. `pid` + AX path or a session-scoped element id) that semantic actions accept,
> so the agent can act structurally instead of by pixel coordinates when possible.

## Safety

Computer use is high-blast-radius, so:

- **Permission-gated** like other powerful tools; refuses early with a clear message
  if Accessibility/Screen Recording is missing.
- **`dry_run` param** on mutating actions — resolves and reports the target without
  acting.
- **Screenshot-assisted confirmation** for destructive coordinate clicks (return the
  region/element being targeted).
- **No global input *capture*** in v1 (we synthesize input but do not log the user's
  keystrokes), keeping us out of Input Monitoring unless a future feature needs it.
- Prefer **semantic AX actions** over blind coordinate input wherever the element is
  resolvable — more robust and more auditable.

## Implementation plan

1. `jcode-macos-control` crate: permissions, screenshot, AX read, AX action,
   CGEvent input, window/app control, clipboard. Unit-test the pure parts
   (input parsing, chord parsing, tree serialization).
2. `ComputerTool` in `tool/computer.rs`: input struct + `action` dispatch +
   schema + description; register `"computer"` in `tool/mod.rs` `base_tools()`.
3. Default-off / gated rollout + docs in `docs/`.
4. Follow-up: Windows/Linux backends behind the same tool surface.

## Open questions

- Element handle format — `pid`+AX-path vs an opaque session-scoped id cache?
- Should `request_permissions` block-and-poll, or return immediately with status and
  let the agent re-check?
- Default enablement: opt-in flag vs always-registered-but-gated?