elizaos--eliza/plugins/plugin-computeruse/docs/AOSP_SYSTEM_APP.md

AOSP system-app deployment (computer-use on the Eliza AOSP fork)

Status: design + contract. The desktop CUA surface (capture, input, scene, Brain→Cascade→dispatch) is production; the AOSP path is partially built — createAospInputActor() returns null in consumer builds today (src/actor/aosp-input-actor.ts), and the privileged bridge below is the remaining work. This doc is the contract that wiring must satisfy. See ANDROID_CONSTRAINTS.md for the honest scope on stock (non-system) Android, and docs/android-aosp-validation.json for the release evidence manifest this path is gated on.

Why a system app

On stock Android an app cannot freely read the framebuffer of other apps or inject input into them — those are signature|privileged permissions. The Eliza AOSP fork ships the agent as a platform-signed system app (or a privileged companion service), which unlocks the two capabilities a computer-use agent needs:

Capability	Stock Android	AOSP system app
Full-screen capture	MediaProjection (per-session user consent, own-app-biased)	SurfaceControl.captureDisplay / READ_FRAME_BUFFER (no prompt, any display)
Global input injection	none (only own UI / a11y gestures)	InputManager.injectInputEvent with INJECT_EVENTS
Element grounding	AccessibilityService tree (opt-in)	AccessibilityService tree + OCR, always available

Manifest / signing

<!-- Platform-signed; placed on the system image (priv-app). -->
<uses-permission android:name="android.permission.READ_FRAME_BUFFER" />
<uses-permission android:name="android.permission.INJECT_EVENTS" />
<uses-permission android:name="android.permission.ACCESS_SURFACE_FLINGER" />
<application android:sharedUserId="android.uid.system" ... />

• Sign with the platform key; install under /system/priv-app (or /product/priv-app) with a matching privapp-permissions allowlist entry for the two signature permissions.
• Non-system builds must degrade to the MediaProjection + AccessibilityService path (see ANDROID_CONSTRAINTS.md). The plugin feature-detects at runtime and never assumes the privileged path.

Capture — ScreenState source

The AOSP capture feeds the same per-display frame contract the desktop scene uses (DisplayCapture { display, frame: PNG }, see src/platform/capture.ts):

• Privileged: SurfaceControl.captureDisplay(DisplayCaptureArgs) → HardwareBuffer → PNG. No user prompt; supports secondary/virtual displays.
• Fallback: MediaProjection + ImageReader (one-time user consent), wired in src/mobile/android-scene.ts / mobile-screen-capture.ts.

Both paths populate the Android scene (android-scene.ts) with display bounds so multi-display coordinate translation (src/platform/coords.ts) works unchanged.

Input — AospPrivilegedInputBridge

src/actor/aosp-input-actor.ts is the seam. createAospInputActor() returns null today; the privileged bridge must implement the ComputerInterface verbs via InputManager.injectInputEvent:

Verb	Implementation
click / double_click / long_press	MotionEvent ACTION_DOWN→UP (→DOWN→UP) at (x,y), INJECT_INPUT_EVENT_MODE_ASYNC
drag	ACTION_DOWN → interpolated ACTION_MOVE waypoints → ACTION_UP (mirror the desktop manual-drag in nut-driver.ts)
scroll	per-notch ACTION_SCROLL or MOVE deltas (mirror the desktop per-notch split)
type	KeyEvent stream or InputConnection commit
key / key_combo	KeyEvent with meta-state

Coordinates are logical display pixels (same contract the empirical Windows probe established for nutjs); the bridge applies the display's density only if injectInputEvent is found to operate in physical pixels on the target ROM — verify empirically before adding any scale, exactly as documented for the desktop driver.

Grounding — Brain-less, Actor-only (cheap by construction)

There is no mobile Brain (no full-frame VLM reasoning loop on device). AOSP uses the Actor-only path: OcrCoordinateGroundingActor resolves a target by matching Scene.ocr (OCR boxes) and Scene.ax (AccessibilityNodeInfo clickables) — exactly the structured readout GET_SCREEN produces (#9105 M2). This means AOSP gets computer-use without a remote VLM:

1. android-scene.ts builds Scene from MediaProjection/SurfaceControl capture
   • the AccessibilityNodeInfo tree + OCR.
2. OCR provider: register PaddleOCR PP-OCRv5 mobile (Paddle-Lite) as the AOSP CoordOcrProvider via registerCoordOcrProvider (the same seam plugin-vision uses on desktop, #9105 M1) — on-device, ARM-optimized, zero LLM tokens.
3. Cascade grounds via resolveReference (OCR/AX text-match) with the per-Scene grounding cache (#9105 M5); dispatch routes the concrete coords to the AospPrivilegedInputBridge.

Remaining work (tracked in #9105 M6)

• Implement AospPrivilegedInputBridge and return it from createAospInputActor() when the privileged permissions are held.
• SurfaceControl.captureDisplay capture path + MediaProjection fallback feeding android-scene.ts.
• Register Paddle-Lite PP-OCRv5 as the AOSP CoordOcrProvider.
• Empirically verify injectInputEvent coordinate space (logical vs physical) on the target ROM before any density scaling.
• Complete docs/android-aosp-validation.json evidence (capture + a click landed via injection + an OCR-grounded tap) and gate the release on it.

Security

All destructive actions still pass through ComputerUseApprovalManager (smart_approve by default). A platform-signed agent with INJECT_EVENTS is a high-privilege component — keep approval gating on, scope the priv-app allowlist to exactly the two signature permissions, and never expose the injection bridge over an unauthenticated IPC surface.