chore: import upstream snapshot with attribution
FreeBSD Smoke / FreeBSD Smoke (x86_64) (push) Has been cancelled
CI / Quality Guardrails (push) Has been cancelled
CI / Build & Test (macos-latest) (push) Has been cancelled
CI / Build & Test (ubuntu-latest) (push) Has been cancelled
CI / Build & Test (windows-latest) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / PowerShell Syntax (push) Has been cancelled
CI / Windows Cross-Target Check (Linux) (push) Has been cancelled
FreeBSD Smoke / FreeBSD Smoke (x86_64) (push) Has been cancelled
CI / Quality Guardrails (push) Has been cancelled
CI / Build & Test (macos-latest) (push) Has been cancelled
CI / Build & Test (ubuntu-latest) (push) Has been cancelled
CI / Build & Test (windows-latest) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / PowerShell Syntax (push) Has been cancelled
CI / Windows Cross-Target Check (Linux) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,174 @@
|
||||
# Cross-Device Worktree and Build Sync (Proposed)
|
||||
|
||||
Status: Proposed (design only, not implemented)
|
||||
|
||||
## Problem
|
||||
|
||||
A user develops jcode (and other repos) on multiple machines, e.g. a MacBook
|
||||
(aarch64-darwin) and a Linux laptop. On a single machine, `selfdev build` +
|
||||
reload means every local jcode instance runs the new version, and multiple
|
||||
agents can share one worktree because the server mediates edits and tracks
|
||||
conflicts (`FileTouchService`, `file_activity.rs`).
|
||||
|
||||
Split across machines, this breaks:
|
||||
|
||||
- A `selfdev build` on the Linux laptop does not update the MacBook's jcode,
|
||||
and vice versa.
|
||||
- There is no shared worktree, so changes made on one machine are invisible
|
||||
to agents/sessions on the other until manually pushed/pulled.
|
||||
|
||||
Goal: make multiple machines behave like one logical worktree + one logical
|
||||
build channel, the same way multiple agents already share one worktree on a
|
||||
single machine.
|
||||
|
||||
## Existing building blocks (verified in code)
|
||||
|
||||
| Building block | Where | Why it matters |
|
||||
| --- | --- | --- |
|
||||
| Arch-independent version identity | `jcode-build-support/src/source_state.rs` (`SourceState::version_label`, `fingerprint`) | Fingerprint hashes full commit hash + status + `diff --binary HEAD` + untracked contents. Identical source trees on two machines produce the same label, even though binaries differ per-arch. |
|
||||
| Auto-reload on newer binary | `server/util.rs::server_has_newer_binary`, `reload_exec_target` | Mtime-based channel scan. A peer-triggered local build that publishes to `builds/current` triggers the existing reload flow with zero changes. |
|
||||
| Pull → build → install → exec | `session_rebuild.rs` | Already implements the receiving side's pipeline shape. |
|
||||
| Network door, same protocol | `jcode-base/src/gateway.rs` (WS + plain HTTP on :7643) | Remote clients speak the identical newline-JSON protocol as Unix-socket clients. Plain HTTP handler (`/pair`, `/health`) is a natural place for `/peer/*` endpoints. |
|
||||
| NAT-friendly device event bus | `server/jade_relay.rs` | Device IDs, heartbeats, long-polled command events. Works when machines cannot reach each other directly. |
|
||||
| Server-side tool execution | server architecture | Tools (bash, edit) run in the server process; a remote client attaching to another machine's server gets the full multi-agent-one-worktree behavior, including conflict warnings. |
|
||||
| Remote build precedent | `scripts/remote_build.sh` | rsync + ssh + sync-back pattern. |
|
||||
|
||||
### Known gap: repo identity across machines
|
||||
|
||||
`repo_scope_key` / `worktree_scope_key` hash the *local canonical path* of the
|
||||
git common dir / worktree. These will never match across machines
|
||||
(`/Users/jeremy/...` vs `/home/jeremy/...`). Cross-device features must key
|
||||
repos by something portable: normalized origin URL, or an explicit repo name
|
||||
in config (`[sync] repo_id = "jcode"`), falling back to origin URL hash.
|
||||
|
||||
## Design tensions
|
||||
|
||||
1. **Binaries cannot be shared** across darwin-aarch64 and linux-x86_64.
|
||||
"Same version everywhere" must mean *same source state, built per machine*,
|
||||
with `version_label` as the cross-machine equality check.
|
||||
2. **Not all writes are server-mediated.** Tool edits flow through the server,
|
||||
but `bash`, editors, and `cargo` mutate the worktree invisibly. Cross-device
|
||||
sync therefore needs a byte-level capture mechanism (git plumbing snapshot
|
||||
and/or fs watcher), not just tool-event forwarding.
|
||||
3. **Laptops go offline.** Pure live-sync (mutagen/syncthing style) has poor
|
||||
conflict semantics. Git-based convergence (per-device sync refs, merges)
|
||||
handles offline divergence honestly.
|
||||
|
||||
## Mechanism: shipping a worktree state without touching HEAD
|
||||
|
||||
To capture a possibly-dirty worktree atomically without moving the user's
|
||||
HEAD or index:
|
||||
|
||||
```
|
||||
GIT_INDEX_FILE=$tmp git add -A # tracked + untracked into temp index
|
||||
tree=$(GIT_INDEX_FILE=$tmp git write-tree)
|
||||
commit=$(git commit-tree $tree -p HEAD -m "jcode sync: <device> <fingerprint>")
|
||||
git push origin $commit:refs/jcode/sync/<device>
|
||||
```
|
||||
|
||||
- Atomic, content-addressed, includes untracked files, excludes gitignored.
|
||||
- Receiver fetches the ref, applies it (checkout into worktree or materialize
|
||||
diff), then verifies `current_source_state().fingerprint` matches the
|
||||
beacon's fingerprint, guaranteeing byte-exact reproduction.
|
||||
- The sender's worktree/index/HEAD never move.
|
||||
|
||||
## Phased plan
|
||||
|
||||
### Phase B - selfdev build parity (do first; kills the stated pain)
|
||||
|
||||
After a successful `selfdev build` + publish on machine X:
|
||||
|
||||
1. Compute `SourceState` (already done by the build pipeline).
|
||||
2. Snapshot the worktree to `refs/jcode/sync/<device>` (mechanism above) and
|
||||
push to the shared git remote. Clean trees can skip the snapshot and use
|
||||
the existing commit.
|
||||
3. Announce a **version beacon** `{repo_id, version_label, fingerprint,
|
||||
full_hash, sync_ref, device, timestamp}`:
|
||||
- Fast path: HTTP POST to peer gateways (`/peer/version-beacon`) over
|
||||
Tailscale.
|
||||
- Fallback: jade relay device event (works through NAT).
|
||||
- Slow path: peer polls sync refs on the git remote.
|
||||
|
||||
Machine Y's server runs a small peer-sync task (same shape as
|
||||
`jade_relay::spawn_if_configured`):
|
||||
|
||||
1. Receives beacon; ignores if `version_label` matches what it already runs
|
||||
or recently applied (**echo suppression**, prevents rebuild ping-pong).
|
||||
2. Policy gate, default conservative:
|
||||
- Worktree clean AND local HEAD is an ancestor of the beacon commit
|
||||
→ auto-apply: fetch, advance, build via the selfdev build queue
|
||||
(native arch), publish. The existing `server_has_newer_binary()` poll
|
||||
then auto-reloads.
|
||||
- Otherwise → do not clobber. Surface in TUI/status:
|
||||
`peer build available: <label> from <device> (blocked: local changes)`
|
||||
with a one-keystroke accept.
|
||||
3. After publish, Y's `version_label` equals X's. Cross-device parity is
|
||||
verifiable by comparing labels (e.g. in `selfdev status` and the beacon
|
||||
acks).
|
||||
|
||||
Config sketch:
|
||||
|
||||
```toml
|
||||
[sync]
|
||||
enabled = true
|
||||
repo_id = "jcode" # portable repo identity
|
||||
peers = ["macbook.tail-net.ts.net:7643"]
|
||||
auto_apply = "clean-ff-only" # off | clean-ff-only | always-notify
|
||||
```
|
||||
|
||||
### Phase A - hub attach (one authoritative worktree when both online)
|
||||
|
||||
`jcode attach <host>`: TUI connects to the peer machine's server through the
|
||||
existing gateway WS. Because tools execute server-side, the attached client
|
||||
participates fully in that machine's worktree, conflict tracking included.
|
||||
|
||||
Work items:
|
||||
|
||||
- Client transport: WS stream in place of Unix socket (bridge already exists
|
||||
server-side; needs a client-side counterpart).
|
||||
- Pairing/auth UX for a trusted personal device (DeviceRegistry exists).
|
||||
- Audit client-side local-disk reads that assume the session's filesystem,
|
||||
e.g. `jcode-tui/src/tui/ui_file_diff.rs:270` (`std::fs::read_to_string` of
|
||||
the diffed file). These need server RPCs (a `read_file` control request) or
|
||||
graceful degradation.
|
||||
|
||||
### Phase C - true worktree federation (long-term)
|
||||
|
||||
Generalize Phase B's snapshot + Phase A's peering into continuous two-way
|
||||
sync:
|
||||
|
||||
- **Data plane:** throttled auto-snapshots to per-device sync refs, triggered
|
||||
by (a) server-mediated tool edits, (b) an fs watcher for bash/editor/cargo
|
||||
writes, (c) timers. Peers fetch refs directly (ssh/Tailscale) or via the
|
||||
shared remote.
|
||||
- **Convergence:** if local HEAD/state is an ancestor → fast-forward apply.
|
||||
If diverged → keep both snapshots, mark the repo "split", and let the
|
||||
harness spawn an agent to perform the merge (the cross-device analog of
|
||||
the server managing same-machine conflicts).
|
||||
- **Coordination plane:** federate `FileTouchService` events over the peer
|
||||
link so agents on both machines see "another agent edited lines 40-60"
|
||||
warnings across devices. Optionally add advisory write leases for hot
|
||||
files.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Syncthing/mutagen for the worktree:** simple, but byte-level conflicts
|
||||
(`.sync-conflict` files), no atomicity across multi-file edits, and target
|
||||
dirs / build artifacts need careful exclusion. Git-plumbing snapshots give
|
||||
atomic, content-addressed, mergeable states using semantics git users
|
||||
already understand.
|
||||
- **Always build on one machine + copy binaries:** broken by arch mismatch;
|
||||
cross-compiling darwin from linux (and vice versa) is not worth the
|
||||
toolchain cost given both machines have working local toolchains.
|
||||
- **NFS/SSHFS shared worktree:** punishes offline use and IDE/file-watcher
|
||||
performance; a non-starter for laptops.
|
||||
|
||||
## Suggested order of implementation
|
||||
|
||||
1. Phase B beacon + receiver with `auto_apply = "clean-ff-only"`, git-remote
|
||||
polling only (no new ports), `selfdev status` showing peer parity.
|
||||
2. Add gateway `/peer/version-beacon` fast path + TUI notification for the
|
||||
blocked case.
|
||||
3. Phase A `jcode attach` (WS client transport + pairing + file-read RPC).
|
||||
4. Phase C federation, reusing the beacon snapshot machinery and the peer
|
||||
link from A.
|
||||
@@ -0,0 +1,229 @@
|
||||
# Roadmap: maximal macOS control for the `computer` tool
|
||||
|
||||
Goal: give the agent as much reliable control over macOS as the platform allows,
|
||||
including **background control that does not disturb what the user is looking at**.
|
||||
|
||||
This builds on the v1 `computer` tool (PR #345): screenshot, coordinate
|
||||
mouse/keyboard, scroll, AX-tree read, cursor, permission check.
|
||||
|
||||
Everything below is implementable in Rust with crates that are already in the
|
||||
lockfile or available on crates.io (`accessibility-sys`, `screencapturekit`,
|
||||
`objc2-app-kit`, `core-graphics`). No Swift/ObjC build step.
|
||||
|
||||
---
|
||||
|
||||
## The one hard constraint
|
||||
|
||||
macOS has **one HID cursor and one keyboard-focus** shared by the whole login
|
||||
session. Synthetic *coordinate* input (CGEvent) is therefore always visible: it
|
||||
moves the real cursor and types into the focused app.
|
||||
|
||||
**Background / not-in-view control must avoid CGEvent** and instead go through:
|
||||
|
||||
1. **Accessibility (AX) actions** - act on a specific element by reference.
|
||||
2. **Apple Events / scripting** - drive scriptable apps with no UI.
|
||||
3. **Per-window capture** - "see" a window without raising it.
|
||||
|
||||
True simultaneous "you work + I work independently" needs a **separate display
|
||||
or login session** (see Tier 4).
|
||||
|
||||
---
|
||||
|
||||
## Tier 0 - done (v1, PR #345)
|
||||
|
||||
- `screenshot` (main display, point/pixel scale aware)
|
||||
- `move` / `click` / `double_click` / `right_click` / `drag` / `scroll`
|
||||
- `type` / `key` (chords)
|
||||
- `ui` (AX tree read via osascript)
|
||||
- `cursor`, `check_permissions`
|
||||
|
||||
## Tier 1 - AX semantic actions ← highest leverage for background control
|
||||
|
||||
Read + act on elements by reference, no cursor movement, target app need not be
|
||||
frontmost. Uses `accessibility-sys` (`AXUIElementPerformAction`,
|
||||
`AXUIElementSetAttributeValue`, `AXUIElementCopyElementAtPosition`,
|
||||
`AXUIElementCopyAttributeValue`).
|
||||
|
||||
- `find_element { role?, title?, value?, pid?, app? }` -> stable element handles
|
||||
- `element_at { x, y }` -> element under a point (AXUIElementCopyElementAtPosition)
|
||||
- `press { element }` -> `AXPress` (click a button in a background window)
|
||||
- `set_value { element, value }` -> type into a field without focus
|
||||
- `get_value { element }`
|
||||
- `perform_action { element, ax_action }` -> any advertised AX action
|
||||
- `select_menu { app, path: ["File","Export…"] }` -> drive the menu bar of any app
|
||||
|
||||
Handle format: `pid` + AX path (index chain) or a session-scoped element id cache,
|
||||
so the model can act structurally instead of by pixels.
|
||||
|
||||
Why it matters: this is the actual "click things you're not looking at" capability.
|
||||
|
||||
## Tier 2 - app / window / system management
|
||||
|
||||
Mostly `objc2-app-kit` (`NSWorkspace`, `NSRunningApplication`) + AX window
|
||||
attributes + CoreGraphics window list.
|
||||
|
||||
- `list_apps` / `activate_app { app }` / `hide_app` / `quit_app`
|
||||
- `list_windows { pid? }` (CGWindowList) with ids, titles, bounds, on/off-screen
|
||||
- `focus_window` / `move_window` / `resize_window` / `minimize_window` / `close_window`
|
||||
(AX window actions - can target background windows)
|
||||
- `window_screenshot { window_id }` -> capture a specific window even if occluded
|
||||
(`CGWindowListCreateImage` now, ScreenCaptureKit later)
|
||||
- `spaces` awareness (which Space an app is on; activating may switch Spaces - visible)
|
||||
|
||||
## Tier 3 - clipboard, input fidelity, observation
|
||||
|
||||
- `get_clipboard` / `set_clipboard { text }` (`NSPasteboard` via objc2-app-kit)
|
||||
- `key_down` / `key_up` (hold modifiers, game-style input)
|
||||
- `type_into { element, text }` (AX set value + confirm) for reliability over blind typing
|
||||
- `wait_for { element|condition, timeout }` using `AXObserver*` notifications
|
||||
(e.g. wait for a sheet to appear) instead of sleep-and-poll
|
||||
- `paste_type { text }` - set clipboard + Cmd-V for fast/large text entry
|
||||
|
||||
## Tier 4 - true background / parallel operation (advanced)
|
||||
|
||||
These give genuinely off-screen, non-interfering control. Higher setup cost.
|
||||
|
||||
- **Apple Events scripting bridge**: `run_applescript { script }` / `run_jxa`.
|
||||
Fully headless for scriptable apps (Mail, Notes, Safari, Finder, Music, System
|
||||
Settings panes, Terminal, many pro apps). No cursor, no focus. Per-app
|
||||
Automation permission (prompts on first use).
|
||||
- **Virtual / headless display**: route the agent's cursor+windows to a second
|
||||
(virtual) display the user isn't looking at. Options: a virtual display driver
|
||||
(e.g. BetterDisplay/`CGVirtualDisplay` private API) or a real unused monitor.
|
||||
Lets the agent move windows there and use coordinate input without touching the
|
||||
user's screen.
|
||||
- **Separate login / Screen Sharing session**: a second macOS session has its own
|
||||
cursor and focus; the agent drives that one. Strongest isolation, most setup.
|
||||
- **Shortcuts integration**: invoke the user's `Shortcuts` automations
|
||||
(`shortcuts run …`) as high-level, sanctioned actions.
|
||||
|
||||
## Tier 5 - sensors / extras (optional, opt-in)
|
||||
|
||||
- `ocr { region|window }` via Vision framework (read text in images / non-AX apps).
|
||||
- `screen_record { seconds }` short clips via ScreenCaptureKit.
|
||||
- Audio in/out control, notifications, `do_not_disturb` toggling via scripting.
|
||||
- Camera/mic are separate TCC permissions; keep strictly opt-in.
|
||||
|
||||
---
|
||||
|
||||
## Permissions (TCC) - the gatekeeping reality
|
||||
|
||||
| Permission | Unlocks | Auto-grantable? |
|
||||
|---|---|---|
|
||||
| **Accessibility** | CGEvent input, all AX read/act, window control | No - user toggles once (we can prompt + deep-link) |
|
||||
| **Screen Recording** | screenshots, window/ocr capture | Request API exists (`CGRequestScreenCaptureAccess`) |
|
||||
| **Automation (Apple Events)** | scripting each app | Prompts per target app on first send |
|
||||
| **Input Monitoring** | reading global input stream (only if we add capture) | Request API exists |
|
||||
|
||||
Plan: a `request_permissions` action that calls
|
||||
`AXIsProcessTrustedWithOptions(prompt=true)` (adds jcode to the list + shows the
|
||||
dialog) and deep-links to the exact System Settings pane, then polls
|
||||
`AXIsProcessTrusted()`. One prompt + one toggle; never zero-touch for Accessibility
|
||||
(Apple's anti-malware boundary).
|
||||
|
||||
Important: the permission attaches to the **host binary/terminal** running jcode.
|
||||
For a stable experience we likely want a signed jcode.app with a fixed bundle id so
|
||||
the grant persists across updates (otherwise each new binary path re-prompts).
|
||||
|
||||
## Safety model (high blast radius)
|
||||
|
||||
- Gated like `bash`: refuses early if required permission missing.
|
||||
- `dry_run` on mutating actions: resolve + report target without acting.
|
||||
- Prefer AX semantic actions over blind coordinate clicks (auditable, robust).
|
||||
- Screenshot/element echo on destructive coordinate clicks.
|
||||
- No global input *capture* unless explicitly enabled (keeps us out of Input
|
||||
Monitoring by default).
|
||||
- Per-action audit log; optional allowlist/denylist of target apps.
|
||||
|
||||
## Suggested build order
|
||||
|
||||
1. **Tier 1 (AX actions)** - biggest capability jump, enables background control.
|
||||
2. **Tier 2 window mgmt + per-window screenshot** - "see and act on hidden windows".
|
||||
3. **Tier 3 clipboard + AXObserver waits** - reliability.
|
||||
4. **`run_applescript`/JXA bridge (Tier 4)** - headless scripting for many apps.
|
||||
5. **Virtual-display / second-session (Tier 4)** - true parallel, non-interfering.
|
||||
6. Signed jcode.app bundle for durable permissions.
|
||||
7. Vision OCR (Tier 5) as needed.
|
||||
|
||||
## Crates
|
||||
|
||||
- `accessibility-sys` 0.2 (AX read/act/observe) - on crates.io
|
||||
- `screencapturekit` 7 (modern capture) - on crates.io; `core-graphics` window list as fallback
|
||||
- `objc2-app-kit` / `objc2-foundation` 0.3 - already in lockfile (NSWorkspace, NSPasteboard)
|
||||
- `core-graphics` 0.23 - already a direct dep (CGEvent, CGWindowList, CGDisplay)
|
||||
|
||||
---
|
||||
|
||||
## Tool interface design (decided)
|
||||
|
||||
### Single tool, progressive disclosure
|
||||
|
||||
One `computer` tool, `action`-dispatched (like `browser`). To keep always-on
|
||||
context flat regardless of how many tiers exist, the schema uses **progressive
|
||||
disclosure**:
|
||||
|
||||
- **Always-on core (~370 tokens, measured with tiktoken cl100k_base):**
|
||||
`screenshot, ui, ocr, click, type, key, press, set_value, run_applescript,
|
||||
setup, discover`.
|
||||
- **`discover { category }`** returns full specs for advanced actions on demand
|
||||
(`mouse|keyboard|ax|windows|apps|clipboard|scripting|displays|system|all`),
|
||||
~130 tokens per category, paid only when used.
|
||||
- **Shared handle types** (`element`, `window_id`, `region`) defined once and
|
||||
reused, so params do not multiply with actions.
|
||||
|
||||
Measured always-on cost:
|
||||
|
||||
| Design | Actions visible | Always-on tokens |
|
||||
|---|---|---|
|
||||
| Current v1 tool | 12 | ~720 |
|
||||
| Flat, all tiers (~46 actions) | 46 | ~1,020 |
|
||||
| **Progressive core** | 11 | **~370** |
|
||||
|
||||
Background control is a property of the *mechanism*, not the tier: CGEvent =
|
||||
visible; **AX actions (press/set_value/select_menu) + Apple Events = background**.
|
||||
|
||||
### `setup` / `check_permissions` action
|
||||
|
||||
A first-class `setup` action that:
|
||||
|
||||
1. **Reports** status of every requirement: Accessibility (`AXIsProcessTrusted`),
|
||||
Screen Recording (`CGPreflightScreenCaptureAccess`), Automation (per-app, via
|
||||
first Apple Event), plus install/bundle health.
|
||||
2. **Requests** what it can programmatically:
|
||||
- `AXIsProcessTrustedWithOptions(prompt=true)` — shows the Accessibility dialog
|
||||
and pre-adds jcode to the list (toggled off).
|
||||
- `CGRequestScreenCaptureAccess()` — prompts for Screen Recording.
|
||||
- First Apple Event to a target app — triggers its Automation prompt.
|
||||
3. **Deep-links** to the exact System Settings pane for anything still missing:
|
||||
- `x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility`
|
||||
- `…?Privacy_ScreenCapture`
|
||||
- `…?Privacy_Automation`
|
||||
4. **Polls** `AXIsProcessTrusted()` until granted, then reports "ready".
|
||||
|
||||
**Hard limit:** the Accessibility *toggle itself cannot be flipped by any API*
|
||||
(Apple anti-malware boundary). `tccutil` can only reset, not grant. So the best
|
||||
achievable UX is **"one or two prompts + one toggle,"** never zero-touch.
|
||||
|
||||
### Durable permissions: signed app bundle
|
||||
|
||||
TCC permissions attach to the **running binary's identity**. A bare dev/cli binary
|
||||
changes path/signature across updates, so macOS re-prompts every time. To make a
|
||||
grant stick:
|
||||
|
||||
- Ship/install jcode as a **signed `.app` bundle with a stable bundle id**
|
||||
(e.g. `com.jcode.app`) and a Designated Requirement, so the Accessibility /
|
||||
Screen Recording grant persists across updates.
|
||||
- `setup` should detect "running from an unstable/unsigned path" and offer to
|
||||
install the proper bundle, so the user grants **once**.
|
||||
|
||||
### Build order (updated)
|
||||
|
||||
1. Progressive-disclosure refactor of the v1 tool (core + `discover`).
|
||||
2. `setup` action (check + request + deep-link + poll).
|
||||
3. Tier 1 AX actions (background control).
|
||||
4. Tier 2 window/app management + per-window screenshot.
|
||||
5. Tier 3 clipboard + AXObserver waits.
|
||||
6. `run_applescript`/JXA bridge (Tier 4 headless scripting).
|
||||
7. Signed app bundle for durable permissions.
|
||||
8. Tier 5 OCR (Vision). (Camera/audio intentionally excluded.)
|
||||
9. Virtual-display / second-session for true parallel work (advanced).
|
||||
@@ -0,0 +1,149 @@
|
||||
# Proposal: native `computer` tool for macOS computer use
|
||||
|
||||
## Summary
|
||||
|
||||
Add a single native tool, **`computer`**, that lets the agent observe and control
|
||||
the macOS GUI — screenshots, the accessibility (AX) tree, mouse/keyboard input,
|
||||
window/app management, and clipboard — through one `action`-dispatched interface.
|
||||
|
||||
This mirrors the existing **`browser`** tool (`crates/jcode-app-core/src/tool/browser.rs`):
|
||||
one registered tool, an `action: String` that selects a sub-operation, with optional
|
||||
typed params. It gives jcode a closed control loop (*see screen → decide → act*)
|
||||
without depending on a browser or external automation tooling.
|
||||
|
||||
## Motivation
|
||||
|
||||
- The agent can already drive a browser; it cannot drive native macOS apps, system
|
||||
UI, or anything outside the browser sandbox.
|
||||
- "Computer use" agents need exactly three primitives: **read the screen**, **read
|
||||
UI structure**, and **synthesize input**. macOS exposes all three through the
|
||||
Accessibility / Quartz Event Services / ScreenCaptureKit stack.
|
||||
- A single, well-scoped tool keeps the tool surface small and the permission story
|
||||
in one place.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
crates/jcode-macos-control/ (new) cfg(target_os = "macos") platform crate
|
||||
└─ AX (accessibility-sys), CGEvent (core-graphics),
|
||||
CoreFoundation (core-foundation), screenshots (ScreenCaptureKit / CGDisplay),
|
||||
app/window control (objc2 + objc2-app-kit), clipboard (objc2 NSPasteboard)
|
||||
|
||||
crates/jcode-app-core/src/tool/computer.rs (new) ComputerTool
|
||||
└─ thin dispatch layer: parse input -> call jcode-macos-control -> ToolOutput
|
||||
└─ registered in crates/jcode-app-core/src/tool/mod.rs base_tools()
|
||||
```
|
||||
|
||||
- All native APIs are reached through existing Rust bindings (`objc2`,
|
||||
`accessibility-sys`, `core-graphics`, `core-foundation`) — **no Swift/ObjC build
|
||||
step**.
|
||||
- On non-macOS targets the tool still registers but every action returns a clean
|
||||
`unsupported on this platform` error, so the tool list stays stable across OSes.
|
||||
- `screenshot` returns its image via `ToolOutput::with_image` (base64), matching how
|
||||
`browser` returns screenshots today.
|
||||
|
||||
## Permissions (the important part)
|
||||
|
||||
macOS splits this across **four** TCC permissions. Programmatic *request* support
|
||||
differs per permission:
|
||||
|
||||
| Permission | Used for | Programmatic request |
|
||||
|---|---|---|
|
||||
| **Accessibility** | drive other apps' UI, inject `CGEvent` input | ⚠️ prompt + deep-link only; user must toggle |
|
||||
| **Screen Recording** | screenshots / `get_ui_tree` of some apps | ✅ `CGRequestScreenCaptureAccess()` |
|
||||
| **Input Monitoring** | reading the global input stream | ✅ `IOHIDRequestAccess(...)` |
|
||||
| **Automation** (Apple Events) | scripting cooperating apps | ✅ prompts on first send, per target app |
|
||||
|
||||
**Accessibility is the one that cannot be auto-granted** (Apple's anti-malware
|
||||
boundary), and it is required for input injection. Best achievable flow, exposed via
|
||||
the `request_permissions` action:
|
||||
|
||||
1. `AXIsProcessTrustedWithOptions([kAXTrustedCheckOptionPrompt: true])` — shows the
|
||||
system dialog *and auto-adds jcode to the Accessibility list* (toggled off).
|
||||
2. Deep-link to the exact pane:
|
||||
`open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"`.
|
||||
3. Poll `AXIsProcessTrusted()` until granted, then report ready.
|
||||
|
||||
So the experience becomes **one prompt + one toggle**, not "go hunt in Settings" —
|
||||
but never zero-touch for Accessibility.
|
||||
|
||||
`check_permissions` reports the current state of all four so the agent can tell the
|
||||
user precisely what is missing before attempting control.
|
||||
|
||||
## Actions
|
||||
|
||||
`action` (required) selects the operation. Params below are optional and validated
|
||||
per action.
|
||||
|
||||
**Permissions**
|
||||
- `check_permissions` → status of accessibility / screen-recording / input-monitoring
|
||||
- `request_permissions` → prompt + deep-link flow above
|
||||
|
||||
**Observe**
|
||||
- `screenshot` — `{ display?, window_id?, region? }` → image
|
||||
- `get_ui_tree` — `{ pid? | frontmost, max_depth? }` → serialized AX tree (role, title, value, position, size, actions)
|
||||
- `find_element` — `{ role?, title?, value?, pid? }` → matching elements + their identifiers
|
||||
- `element_at` — `{ x, y }` → element under the point
|
||||
|
||||
**Mouse**
|
||||
- `move` — `{ x, y }`
|
||||
- `click` / `double_click` / `right_click` — `{ x?, y? }` (current position if omitted)
|
||||
- `drag` — `{ from: [x,y], to: [x,y] }`
|
||||
- `scroll` — `{ x?, y?, dx, dy }`
|
||||
|
||||
**Keyboard**
|
||||
- `type` — `{ text }`
|
||||
- `key` — `{ keys: "cmd+shift+4" }` (chord)
|
||||
- `key_down` / `key_up` — `{ key }`
|
||||
|
||||
**Semantic AX (preferred over raw input when available)**
|
||||
- `press` — `{ element }` (AXPress)
|
||||
- `set_value` — `{ element, value }`
|
||||
- `get_value` — `{ element }`
|
||||
- `perform_action` — `{ element, ax_action }`
|
||||
- `select_menu` — `{ app, path: ["File", "Export…"] }`
|
||||
|
||||
**Window / app**
|
||||
- `list_apps`, `activate_app` `{ app }`
|
||||
- `list_windows` `{ pid? }`, `focus_window` `{ window }`
|
||||
- `move_window` `{ window, x, y }`, `resize_window` `{ window, w, h }`
|
||||
- `minimize_window` / `close_window` `{ window }`
|
||||
|
||||
**Clipboard**
|
||||
- `get_clipboard`, `set_clipboard` `{ text }`
|
||||
|
||||
> Element identifiers: `find_element` / `get_ui_tree` return stable-enough handles
|
||||
> (e.g. `pid` + AX path or a session-scoped element id) that semantic actions accept,
|
||||
> so the agent can act structurally instead of by pixel coordinates when possible.
|
||||
|
||||
## Safety
|
||||
|
||||
Computer use is high-blast-radius, so:
|
||||
|
||||
- **Permission-gated** like other powerful tools; refuses early with a clear message
|
||||
if Accessibility/Screen Recording is missing.
|
||||
- **`dry_run` param** on mutating actions — resolves and reports the target without
|
||||
acting.
|
||||
- **Screenshot-assisted confirmation** for destructive coordinate clicks (return the
|
||||
region/element being targeted).
|
||||
- **No global input *capture*** in v1 (we synthesize input but do not log the user's
|
||||
keystrokes), keeping us out of Input Monitoring unless a future feature needs it.
|
||||
- Prefer **semantic AX actions** over blind coordinate input wherever the element is
|
||||
resolvable — more robust and more auditable.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
1. `jcode-macos-control` crate: permissions, screenshot, AX read, AX action,
|
||||
CGEvent input, window/app control, clipboard. Unit-test the pure parts
|
||||
(input parsing, chord parsing, tree serialization).
|
||||
2. `ComputerTool` in `tool/computer.rs`: input struct + `action` dispatch +
|
||||
schema + description; register `"computer"` in `tool/mod.rs` `base_tools()`.
|
||||
3. Default-off / gated rollout + docs in `docs/`.
|
||||
4. Follow-up: Windows/Linux backends behind the same tool surface.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Element handle format — `pid`+AX-path vs an opaque session-scoped id cache?
|
||||
- Should `request_permissions` block-and-poll, or return immediately with status and
|
||||
let the agent re-check?
|
||||
- Default enablement: opt-in flag vs always-registered-but-gated?
|
||||
Reference in New Issue
Block a user