elizaos--eliza/plugins/plugin-remote-desktop/AGENTS.md

@elizaos/plugin-remote-desktop

Owner-only remote desktop session control for Eliza agents.

Purpose / role

Lets the owner connect to the agent's host machine from another device (typically a phone) over Tailscale VNC, Tailscale SSH, or an ngrok TCP tunnel, gated by an explicit confirmation step and (in cloud mode) a 6-digit pairing code. The plugin is opt-in — add it to the agent's plugin list.

This plugin was extracted from @elizaos/plugin-personal-assistant as part of the LifeOps decomposition. It owns the full remote-desktop implementation: the REMOTE_DESKTOP action, the backend-detection engine, and the in-process RemoteSessionService control plane. PA re-exports remoteDesktopAction + detectRemoteDesktopBackend via a thin shim for back-compat and loads this plugin at init (ensureLifeOpsRemoteDesktopPluginRegistered). Exactly one plugin registers REMOTE_DESKTOP — this one.

Plugin surface

Action

• REMOTE_DESKTOP (src/actions/remote-desktop.ts) — umbrella action with op-based dispatch (start / status / end / list / revoke). Role gate: OWNER. Contexts: browser, automation, settings, admin, terminal. Sets suppressPostActionContinuation: true so the planner does not chain a follow-up turn after a session is opened. start is gated by requireConfirmation (LLM-supplied confirmed:true is never authoritative — the first turn always asks the owner).

No providers. No services. No schema. Session state is owned by the in-process RemoteSessionService (src/remote/); it is in-memory plus a JSON file under resolveStateDir()/lifeops/remote-sessions.json.

Layout

src/
  index.ts                       Plugin export; re-exports action + engine + service + types
  plugin.ts                      Plugin object (actions: [remoteDesktopAction])
  types.ts                       Canonical types (RemoteDesktopSession, RemoteSession, ...)
  actions/
    remote-desktop.ts            REMOTE_DESKTOP umbrella action (real handler; resolveActionArgs dispatch)
  lifeops/
    remote-desktop.ts            Backend detection (Tailscale/ngrok/VNC) + in-process session store
  remote/
    remote-session-service.ts    Control-plane service (lifecycle, pairing-code gate, data-plane handoff)
    pairing-code.ts              6-digit rolling one-time pairing codes

Commands

bun run --cwd plugins/plugin-remote-desktop build        # bun build → dist/ (ESM) + tsc --emitDeclarationOnly
bun run --cwd plugins/plugin-remote-desktop dev          # hot-rebuild via build.ts
bun run --cwd plugins/plugin-remote-desktop test         # vitest run
bun run --cwd plugins/plugin-remote-desktop typecheck    # tsgo --noEmit
bun run --cwd plugins/plugin-remote-desktop check        # typecheck + test
bun run --cwd plugins/plugin-remote-desktop clean        # rm -rf dist .turbo

Config / env vars

Variable	Where used	Required
ELIZA_REMOTE_LOCAL_MODE	RemoteSessionService.startSession — 1 skips pairing-code requirement	No
ELIZA_STATE_DIR	resolveStateDir() — overrides the default base directory for lifeops/remote-sessions.json	No
ELIZA_TEST_REMOTE_DESKTOP_BACKEND	Force mock mode for tests	No

All variables are read by this plugin's engine (src/lifeops/remote-desktop.ts) and control-plane service (src/remote/remote-session-service.ts).

Migration mapping (LifeOps decomposition — complete)

The remote-desktop domain moved here from @elizaos/plugin-personal-assistant. PA now keeps only a thin re-export shim at src/actions/remote-desktop.ts.

Location (this plugin)	Former source in @elizaos/plugin-personal-assistant
src/actions/remote-desktop.ts	src/actions/remote-desktop.ts
src/lifeops/remote-desktop.ts	src/lifeops/remote-desktop.ts
src/remote/remote-session-service.ts	src/remote/remote-session-service.ts
src/remote/pairing-code.ts	src/remote/pairing-code.ts
src/types.ts	inline in the engine + service (now canonical here)

How to extend

Add a new subaction to REMOTE_DESKTOP:

1. Add the name to RemoteDesktopSubaction in src/types.ts.
2. Add an entry to the SUBACTIONS map in src/actions/remote-desktop.ts, write a handle<Op> function, add a case to the dispatch switch.
3. Extend the parameters array on remoteDesktopAction if the op needs new params.

Add a new backend:

1. Add the backend tag to RemoteDesktopBackend in src/types.ts.
2. Add a probe<Backend> and start<Backend>Session helper in src/lifeops/remote-desktop.ts.
3. Extend detectRemoteDesktopBackend and backendAvailable to cover the new backend.

Conventions / gotchas

• OWNER role gate. REMOTE_DESKTOP will not fire for non-owner entities. Check the runtime's role system if the action is unexpectedly unavailable.
• confirmed: true is mandatory for start. RemoteSessionService.startSession throws RemoteSessionError("NOT_CONFIRMED") otherwise. The action uses requireConfirmation from @elizaos/core to surface a confirmation prompt to the owner.
• Pairing codes are one-time. Each issuePairingCode() call rotates the code; consume() is single-use.
• suppressPostActionContinuation: true — opening a remote session is consumed out-of-band (a VNC viewer / SSH client), so the planner should not chain another turn.
• start with no data plane returns DATA_PLANE_NOT_CONFIGURED. The control plane authorizes the session and persists it, but ingressUrl is null until Tailscale (T9b) or the Eliza Cloud tunnel is wired — that explicit absence is surfaced, not papered over.
• No business computation in this plugin's surface. Session state and ingress URL come from RemoteSessionService; the action just shapes the ActionResult for the agent.
• No console.* in server code. Use @elizaos/core's logger; prefix with [remote-desktop] and attach context objects on errors.
• See root AGENTS.md for repo-wide architecture commandments, logger conventions, ESM rules, and naming.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — CLI / tooling:

• The real command/flow invocation transcript (args in, stdout/stderr, exit code) and the artifacts it generated (files, scaffolds, manifests, screenshots/recordings).
• Failure paths: bad args, missing deps, partial state, permission/network errors.
• A recording/log of the actual run end to end — not a unit test of one helper.
• Any model interaction captured as a live trajectory and reviewed.