elizaos--eliza/plugins/plugin-computeruse/docs/MOBILE_ASSISTANT_ROUTING.md

# Siri, App Actions, and LifeOps Routing

Assistant entry surfaces are launch and capture surfaces. They are not a
second task engine.

Siri/App Shortcuts, Android App Actions, Android assistant-role intents, and
desktop shortcut surfaces should route the user's utterance or tap into the
Eliza app/runtime. If the user asks for a reminder, check-in, follow-up,
watcher, recap, or approval, the runtime must create or update a LifeOps
`ScheduledTask`. Native mobile code may deliver notifications for tasks that
already exist, but it must not create native-only reminder state that bypasses
LifeOps.

## Intended Flow

1. The OS entry surface opens Eliza with a source-tagged deep link or assistant
   intent payload.
2. The app/runtime normalizes the payload into the same planner path used by
   chat and device-bus requests.
3. LifeOps creates or updates a `ScheduledTask` through the app/runtime.
4. Native notification APIs only deliver output for that task or wake the
   runtime so the scheduled-task runner can decide what fires.

## Platform Contracts

| Platform | Entry surface | Required routing contract |
|---|---|---|
| Mac | Shortcuts/deep link/menu-bar voice entry | Opens the app/runtime; LifeOps mutations are persisted as `ScheduledTask` records. |
| iOS | Siri/App Shortcuts/App Intents | Uses App Intents or app deep links to hand off to the app/runtime. No cross-app UI driving, and no native-only reminder store. |
| Android consumer | Static shortcuts + App Actions | `shortcuts.xml` and assistant intents open the app custom scheme; LifeOps scheduling happens after runtime routing. |
| Android AOSP | `ROLE_ASSISTANT` + privileged system app | Assistant role may wake Eliza more directly; AOSP-only accessibility/notification-listener services are available, but scheduled behavior still goes through the same `ScheduledTask` runner. |

Android consumer App Actions coverage:

| Flow | BII/static shortcut |
|---|---|
| Ask/chat | `actions.intent.CREATE_MESSAGE`, `actions.intent.GET_THING`, `eliza_app_action_chat` |
| Voice chat | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_voice` |
| LifeOps daily brief | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_daily_brief` |
| LifeOps task creation | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_new_task` |
| LifeOps task list | `actions.intent.OPEN_APP_FEATURE` inline inventory, `eliza_app_action_tasks` |

Each Android App Actions capability keeps a fallback fulfillment without a
required parameter. Feature-specific requests route through
`elizaos://feature/open?...`; vague requests fall back to the chat route so
the runtime can disambiguate instead of executing native-only behavior.

The Play/Pixel build does not request `ROLE_ASSISTANT`,
`ACTION_ASSIST`, `VOICE_COMMAND`, or `BIND_VOICE_INTERACTION`; those are
reserved for AOSP/system validation builds. It also avoids unsupported BIIs
such as `actions.intent.CREATE_THING`; task creation is modeled as opening
the LifeOps task feature and letting the app/runtime confirm any mutation.

## External Assistant Landscape

Checked 2026-05-14 against primary vendor docs:

- OpenAI/ChatGPT on Apple platforms is exposed through Apple's ChatGPT
  extension inside Apple Intelligence and Siri settings, not as a third-party
  Siri API pattern for other apps to copy. See
  <https://help.openai.com/en/articles/10263570-apple-intelligence-siri-faq>
  and
  <https://help.openai.com/en/articles/10269382-setting-up-chatgpt-with-apple-intelligence>.
- Apple's supported third-party integration path is App Intents/App Shortcuts.
  App Intents expose app capabilities to Shortcuts, Spotlight, Siri, widgets,
  controls, and Apple Intelligence experiences. See
  <https://developer.apple.com/documentation/appintents/app-intents>,
  <https://developer.apple.com/shortcuts/>, and
  <https://developer.apple.com/documentation/appintents/app-intent-domains>.
- Claude uses the same Apple-native pattern: an "Ask Claude" App Intent
  available from Siri, Spotlight, the Share menu, and Shortcuts. See
  <https://support.anthropic.com/en/articles/10263469-using-claude-app-intents-shortcuts-and-widgets-on-ios>.
- Perplexity exposes iOS voice assistant entry through Shortcuts, Lock Screen
  controls, and the Action Button, including "Ask Perplexity" and "Start voice
  conversation" shortcut actions. See
  <https://www.perplexity.ai/help-center/en/articles/11132456-how-to-use-the-perplexity-voice-assistant-for-ios>.
- Grok/X official docs currently document Grok availability on iOS, Android,
  and web, including voice and text interactions, but do not document a public
  Siri/App Intent shortcut contract comparable to Claude's. See
  <https://help.x.com/en/using-x/about-grok>.
- Google Assistant/App Actions are driven by capabilities in `shortcuts.xml`.
  Assistant matches a built-in intent declared there and launches the app to
  the requested screen. See
  <https://developer.android.com/develop/devices/assistant/action-schema>.

## Validation Checklist

### Mac

- [ ] Trigger the Mac shortcut/voice entry surface with a plain chat request
  and confirm it opens the app/runtime instead of executing native-only logic.
- [ ] Ask for a one-off reminder and verify a LifeOps `ScheduledTask` record is
  created with `kind: "reminder"`.
- [ ] Ask for a recurring check-in or follow-up and verify cadence lives in the
  `ScheduledTask.trigger`/relationship edge, not in a platform notification.

### iOS

- [ ] Verify Siri/App Shortcuts open Eliza or invoke the App Intent handoff.
- [ ] Verify `appIntentList` only reports local app/donated intents and does
  not claim cross-app enumeration.
- [ ] Ask Siri for a LifeOps reminder/check-in/follow-up and verify the app
  creates a `ScheduledTask` through the runtime.
- [ ] Confirm native `UNUserNotificationCenter` entries, if any, reference an
  existing task or deep link back into the app.

### Android Consumer

- [ ] Verify `AndroidManifest.xml` registers `@xml/shortcuts` on `MainActivity`.
- [ ] Verify the `eliza_app_action_chat` static shortcut opens the app chat
  route.
- [ ] Trigger an App Action and confirm the app/runtime receives the request.
- [ ] Ask for a reminder/check-in/follow-up and verify LifeOps stores a
  `ScheduledTask`; no Java/Kotlin-only reminder table or notification-only
  schedule is created.

### Android AOSP Assistant Role

- [ ] On the system image, confirm `ROLE_ASSISTANT` resolves to Eliza.
- [ ] Trigger `android.intent.action.ASSIST` and `VOICE_COMMAND`; both should
  reach Eliza's app/runtime.
- [ ] Confirm `ElizaAccessibilityService` and
  `ElizaNotificationListenerService` are present only on the AOSP/system APK,
  not the Play/Pixel cloud APK.
- [ ] Confirm privileged capture/input capabilities do not change LifeOps
  persistence: reminders/check-ins/follow-ups still use `ScheduledTask`.

## Static Coverage

- `ios-bridge.test.ts` checks that the TypeScript iOS AppIntent registry stays
  aligned with the native x-callback switch.
- `android-bridge.test.ts` checks that Android assistant/App Actions source
  files route into app deep links rather than native notification creation.

## Current Status and Remaining Work

Implemented static/build coverage:

- iOS App Shortcuts/App Intents are present in the native iOS target and hand
  off source-tagged deep links to the app runtime.
- Android App Actions metadata is present in `shortcuts.xml`, is registered on
  `MainActivity`, and is rewritten by `run-mobile-build.mjs` to the configured
  package and URL scheme.
- Android assistant-role launch uses the same template deep-link scheme as the
  other Android bridge activities, so white-label builds are rewritten
  consistently.
- Android AOSP system builds declare `ElizaAccessibilityService` and
  `ElizaNotificationListenerService`; `android-cloud` strips those services,
  their Java sources, and the accessibility-service XML resource.
- Assistant launch payloads with captured text are consumed by the chat surface
  and sent through the normal chat/planner path with source metadata. LifeOps
  task/reminder deep links with text route to chat first so `ScheduledTask`
  creation remains owned by the LifeOps runtime.

Live-device validation is tracked through evidence manifests and platform
release gates:

- macOS shortcut/voice entry has installer/verifier scripts, but the final Siri
  phrase creation and spoken-phrase result are still per-device manual checks.
- iOS Siri/App Shortcuts must be installed on a physical device and recorded
  in `docs/ios-device-validation.json` alongside the spoken phrases in
  `ElizaAppShortcuts.swift`.
- Android consumer App Actions require a Play/Assistant-capable device or test
  environment to confirm Assistant fulfillment, not just static XML validity.
- Android AOSP assistant-role behavior still needs a system-image validation
  pass.

Known remaining product test:

- Static tests prove source routing and build-time wiring, but they do not prove
  the planner produced the intended `ScheduledTask`. The remaining product test
  is live end-to-end validation: speak or invoke a LifeOps reminder/check-in,
  confirm the assistant launch payload sends through chat/planner, and verify a
  LifeOps `ScheduledTask` record exists.