elizaos--eliza/plugins/plugin-computeruse/docs/IOS_CONSTRAINTS.md

iOS Computer-Use — Honest Scope

Apple does not let third-party apps drive other apps. There is no equivalent to Android's MediaProjection for full-screen capture without a user prompt, no ScreenCaptureKit (that is macOS only), and no system-wide accessibility API that a sandboxed iOS app can use to inject events into other apps.

This document spells out what is possible on iOS, exactly how each surface behaves, and the manual validation checklist that has to be run on a real device before any of this ships.

What works

1. ReplayKit foreground capture (own app)

RPScreenRecorder.shared().startCapture(handler:completionHandler:) returns CMSampleBuffers of the host app's window. Useful for "show me what's on the Eliza screen right now."

RPScreenRecorder.shared().startCapture(handler: { sampleBuffer, type, error in
    // Receive frames for own-app capture only.
}, completionHandler: { error in
    // Setup result; ReplayKit shows a system permission prompt on first use.
})

Constraints:

• Frame delivery rate is the display refresh rate; the bridge throttles to a caller-supplied frameRate (default 1Hz).
• Hard cap of 30 seconds per session. Past that, the work has to move to a BGProcessingTask.

2. ReplayKit broadcast extension (system-wide capture)

A separate target with a ~50MB memory ceiling. Streams frames over an App Group shared container into the main app. The user must start the broadcast themselves via the system share-sheet picker — apps cannot programmatically launch the extension.

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer,
                                  with sampleBufferType: RPSampleBufferType) {
    // Compress + dump into the App Group container.
}

iOS 26 / 26.1 beta regression: extensions are killed within ~3 seconds even when memory headroom is fine. We surface this as extension_died from broadcastExtensionHandshake so callers can fall back to foreground capture. Track Apple's feedback status before shipping this target on iOS 26.

3. Apple Vision OCR

VNRecognizeTextRequest is on-device, free, supports ~30 languages, and typically runs sub-300ms on modern devices.

let request = VNRecognizeTextRequest { request, error in
    let observations = request.results as? [VNRecognizedTextObservation]
    // Top candidate per observation; each has bounding box, text, confidence.
}
request.recognitionLevel = .accurate
request.recognitionLanguages = ["en-US"]
try VNImageRequestHandler(cgImage: cg).perform([request])

This is the OCR provider the WS6 scene-builder will pick up on iOS. The provider interface is in eliza/plugins/plugin-computeruse/src/mobile/ocr-provider.ts.

4. App Intents

The only sanctioned way to drive other apps. Each target app must expose intents (Shortcuts-style). We support invocation via x-callback URL schemes for the system apps in the static registry (ios-app-intent-registry.ts):

• Mail — mailto: with subject / body / cc / bcc
• Messages — sms: with body
• Maps — http://maps.apple.com/?daddr=...&dirflg=...
• Safari — open URL

For richer intents (Notes append, Reminders add, Music play with a query) the user has to donate the action via Shortcuts; we can then invoke via AppIntent on iOS 16+. The bridge's appIntentList returns the runtime list of donated intents this app sees.

5. UIAccessibility (own-app reading only)

accessibilitySnapshot walks the key window's view hierarchy and returns accessibilityLabel / accessibilityValue / role. iOS gives us no way to read another app's UIAccessibility tree.

6. Apple Foundation Models (iOS 26+)

Apple ships an on-device LLM under the FoundationModels framework when Apple Intelligence is enabled. We expose this as an opportunistic fast-path. If unavailable, the existing llama-cpp-capacitor local Eliza-1 vision path stays as the default.

Entitlement and Info.plist updates are listed below.

What does NOT work

Stock iOS does not allow any of the following from a third-party app, and we do not pretend otherwise:

• Driving other apps' UI. No cross-app input synthesis, no MediaProjection / ScreenCaptureKit equivalent, no system-wide accessibility event injection.
• Listing other running apps' processes. The kernel hides this from third-party apps; there is no ps-equivalent.
• Persistent background inference past ~30s, except via BGProcessingTask (opportunistic, OS-scheduled, not guaranteed).

If a feature spec calls for any of the above, escalate it as not feasible on iOS rather than inventing a workaround that will get the app rejected or silently broken by Apple in the next OS update.

Entitlements + Info.plist

Add the following to apps/app/ios/App/App/App.entitlements:

<key>com.apple.developer.kernel.increased-memory-limit</key>
<true/>
<key>com.apple.developer.kernel.extended-virtual-addressing</key>
<true/>

Add the following to apps/app/ios/App/App/Info.plist:

<key>NSScreenCaptureDescription</key>
<string>Captures the Eliza app window when you ask it to see the screen.</string>
<key>NSAppleEventsUsageDescription</key>
<string>Sends Shortcuts to other apps when you authorize an action.</string>

The screen capture string is shown on ReplayKit's first-launch system prompt. The Apple Events string is shown only if the host (this app) ever needs to drive AppleScript via the Catalyst surface; iOS apps that only use UIApplication.shared.open(url:) to invoke x-callback intents do not need it, but it costs nothing to include for forward compatibility with Mac Catalyst builds.

Validation checklist

The iOS bridge contract is covered by TypeScript tests, but physical iOS device behavior is not proven by this repository alone. The required evidence manifest is docs/ios-device-validation.json; keep it in requires_device_evidence until a real device run records device/build metadata, artifacts, and per-method results. Release gates that require physical proof should run:

bun run --cwd plugins/plugin-computeruse validate:ios-device-evidence -- --require-complete

Without --require-complete, the same command validates that the manifest still tracks every required method and evidence field. Before shipping any iOS bridge release, complete the manifest for:

Simulator vs device

Surface	Simulator	Real device
ReplayKit foreground capture	Partial	Required
Broadcast extension	No	Required
Vision OCR	Yes	Required
App Intents (x-callback)	Partial	Required
Accessibility snapshot	Yes	Required
Foundation Models	No	Required
Memory pressure probe	Limited	Required

Per-method checklist

For each method below, run on at least one A14-or-later iPhone running the target iOS (currently iOS 26.1, with iOS 17.6 as the floor):

1. probe() — assert data.platform === "ios", osVersion matches the device, all six capability bits are present.
2. replayKitForegroundStart — assert the system permission prompt is shown on first use; on accept, replayKitForegroundDrain returns frames with non-empty jpegBase64 and the device's screen resolution.
3. broadcastExtensionHandshake — bundle the extension target via Xcode, tap the share-sheet broadcast picker, and assert broadcastActive transitions to true. On iOS 26 betas, verify regression.observed eventually flips to true after the extension dies.
4. visionOcr — render a known PNG with the string "WS9 OCR SMOKE" into the app, run OCR with recognitionLevel: "accurate", and assert fullText contains the source string (case-insensitive).
5. appIntentList — verify the returned list is empty for a fresh install and grows after the user donates intents via Shortcuts.
6. appIntentInvoke with com.apple.mobilesafari.open-url — assert Safari opens to the provided URL. With com.apple.MobileSMS.send-message — assert Messages opens with the recipient and body pre-filled.
7. accessibilitySnapshot — assert the returned tree's top-level node has role !== "labeled" and children.length > 0 on a populated screen.
8. foundationModelGenerate — on a device with Apple Intelligence enabled, assert a short prompt returns non-empty text. With AI disabled, assert the call resolves with foundation_model_unavailable.
9. memoryPressureProbe — invoke UIApplication.shared.performMemoryWarning() (debug-only) and assert the next probe call returns severity >= 0.7 with lastWarningAt set.

Hand-off to other workstreams

• WS1 (memory-pressure arbiter) consumes MemoryPressureSample via the shared IPressureSignal contract in ios-bridge.ts. The bridge is the producer; the arbiter is the consumer. WS1 will subscribe to push events via the bridge once that channel lands; for now it polls memoryPressureProbe.
• WS6 (scene-builder OCR) picks up the iOS Vision provider through selectOcrProvider() in ocr-provider.ts. Register the provider at app boot when running on iOS:

  registerOcrProvider(
    createIosVisionOcrProvider(() => Capacitor.Plugins.ComputerUse),
  );
  

Apple documentation references

• ReplayKit — https://developer.apple.com/documentation/replaykit
• Vision Text Recognition — https://developer.apple.com/documentation/vision/recognizing_text_in_images
• App Intents — https://developer.apple.com/documentation/appintents
• Foundation Models — https://developer.apple.com/documentation/foundationmodels (iOS 26)
• App Groups — https://developer.apple.com/documentation/xcode/configuring-app-groups
• os_proc_available_memory() — https://developer.apple.com/documentation/foundation/process_info/3743117-os_proc_available_memory
• Background Tasks — https://developer.apple.com/documentation/backgroundtasks