elizaos--eliza/plugins/plugin-screenshare/AGENTS.md

@elizaos/plugin-screenshare

Adds authenticated desktop screen-streaming and remote mouse/keyboard control to an Eliza agent.

Purpose / role

This plugin exposes the local machine's desktop as a streamable, interactable session accessible from any authenticated viewer. It is loaded as an elizaOS app plugin (kind: app, launchType: connect) and registered through gatePluginSessionForHostedApp, which gates the plugin surface behind a valid app session. No agent actions or providers are registered — the plugin surface is entirely HTTP routes plus one adaptive UI view.

Plugin surface

Views (registered in plugin.views)

The package ships one dashboard GUI declaration from a single authored source.

id	label	modalities	componentExport	path	bundlePath
screenshare	Screen Share	gui	ScreenshareView	/screenshare	dist/views/bundle.js

ScreenshareView (components/ScreenshareView.tsx) is the GUI data wrapper. It owns the live operator data (capability fetch/poll, host start/stop/rotate, copy/open-viewer, remote connect, refresh) and renders the presentational ScreenshareSpatialView (components/ScreenshareSpatialView.tsx) inside a SpatialSurface. Views are bundled via Vite into dist/views/bundle.js (entry ui/screenshare-view-bundle.ts, which re-exports ScreenshareView plus the interact capability handler).

HTTP routes (handled via handleAppRoutes)

All routes are under /api/apps/screenshare/:

Method	Path	Auth	Description
GET	/api/apps/screenshare/viewer	none	Serves the self-contained viewer HTML page
GET	/api/apps/screenshare/capabilities	none	Reports platform + desktop control capabilities
GET	/api/apps/screenshare/windows	none	Lists open desktop windows via plugin-computeruse
GET	/api/apps/screenshare/sessions	none	Lists all tracked screenshare sessions (public fields only)
POST	/api/apps/screenshare/session	rate-limited (10/min per IP)	Creates a new screenshare session; returns { session, token, viewerUrl }
GET	/api/apps/screenshare/session/:id	session token	Returns public session state
GET	/api/apps/screenshare/session/:id/frame	session token	Captures and streams a PNG screenshot
POST	/api/apps/screenshare/session/:id/input	session token	Sends mouse/keyboard input to the desktop
POST	/api/apps/screenshare/session/:id/stop	session token	Stops the session

Token is accepted via: ?token= query param, X-Screenshare-Token header, or Authorization: Bearer <token> header. Token is also accepted in the JSON request body for POST endpoints.

App lifecycle hooks (exported from routes.ts)

• prepareLaunch — creates/retrieves the local session, builds the viewer URL
• resolveLaunchSession — returns initial AppSessionState
• refreshRunSession — returns current AppSessionState or null if session is gone/stopped
• stopRun — stops the session by ID

Registered interact capabilities (via interact export in ui/screenshare-interact.ts)

• terminal-screenshare-state
• terminal-screenshare-start
• terminal-screenshare-session
• terminal-screenshare-stop
• terminal-screenshare-input
• terminal-screenshare-viewer-url

Layout

src/
  index.ts                     Plugin definition; wraps raw plugin with gatePluginSessionForHostedApp
  routes.ts                    All HTTP route handlers + app lifecycle hooks (prepareLaunch, stopRun, etc.)
  session-store.ts             In-process session store (globalThis-keyed); session CRUD + capability detection
  components/
    ScreenshareView.tsx              GUI data wrapper (owns fetch/poll + host/remote actions); renders ScreenshareSpatialView in a SpatialSurface
    ScreenshareView.test.tsx         Unit tests for ScreenshareView (the GUI surface)
    ScreenshareSpatialView.tsx       The one presentational spatial component for the GUI route
    ScreenshareSpatialView.test.tsx  Unit tests for ScreenshareSpatialView
  ui/
    screenshare-view-bundle.ts                 View bundle entry: re-exports ScreenshareView + interact
    screenshare-helpers.ts                     Shared data contracts + fetch helpers (used by ScreenshareView and interact)
    screenshare-interact.ts                    interact() capability handler (kept out of the component file)
    screenshare-interact.test.ts               Tests for interact() (all capabilities + dispatch types + guards)
    screenshare-capabilities.contract.test.ts  Contract test: real detectDesktopControlCapabilities() shape vs the helper types

Commands

bun run --cwd plugins/plugin-screenshare build         # tsup + vite views + tsc types
bun run --cwd plugins/plugin-screenshare build:js      # tsup (runtime bundle only)
bun run --cwd plugins/plugin-screenshare build:views   # Vite (UI bundle for views)
bun run --cwd plugins/plugin-screenshare build:types   # tsc declarations
bun run --cwd plugins/plugin-screenshare test          # vitest run
bun run --cwd plugins/plugin-screenshare clean         # rm -rf dist

Config / env vars

This plugin reads no env vars directly. Desktop control capability detection is delegated entirely to @elizaos/plugin-computeruse (detectDesktopControlCapabilities, getDesktopPlatformName). Runtime capability checks cover:

• headfulGui — whether a display server is available
• screenshot — whether screenshot capture is available
• computerUse — whether mouse/keyboard control is available

The GET /capabilities route reports all three capability flags along with which tool backs each one. If any are unavailable, prepareLaunch emits warning-level AppLaunchDiagnostic entries so the elizaOS app system can surface them.

How to extend

Add a new route: Add a branch in handleAppRoutes in src/routes.ts. Follow the pattern of reading and validating query/body params, calling session-store helpers or @elizaos/plugin-computeruse primitives, and replying with ctx.json / ctx.error.

Add a new input type: Add a branch in executeInput in src/routes.ts. Dispatch to the appropriate @elizaos/plugin-computeruse primitive (e.g. performDesktopScroll, performDesktopKeypress).

Add a new view capability: Add a branch in the interact function in src/ui/screenshare-interact.ts.

Add UI controls: Edit the one presentational ScreenshareSpatialView (src/components/ScreenshareSpatialView.tsx) and wire any new control through ScreenshareView's onAction handler (src/components/ScreenshareView.tsx).

Conventions / gotchas

• No actions or providers. All intelligence surface is in HTTP routes + UI views. The elizaOS agent does not get any new actions from this plugin.
• Session store is in-process, not persisted. session-store.ts stores sessions in globalThis under Symbol.for("elizaos.app-screenshare.session-store"). Sessions are lost on process restart.
• One active local session at a time. createScreenshareSession stops the previous local session before creating a new one. Use getOrCreateLocalScreenshareSession to reuse an existing active session instead.
• Token is secret. ScreenshareSession.token (a 24-byte base64url random) is never included in ScreensharePublicSession. Callers must separately track the token returned at session creation.
• Rate limiting is in-process. sessionCreateRateLimitExceeded limits POST /session to 10 requests per IP per minute using an in-process map. It does not survive process restart and does not apply to other routes.
• Keypress character allowlist. Keys sent via type: "keypress" must match /^[A-Za-z0-9+_.,: -]+$/ and be under 128 characters. Special names like Enter, Escape, Tab, Up, Down, Left, Right, Backspace are mapped in the viewer JS.
• Desktop primitives come from @elizaos/plugin-computeruse. Do not reach into OS APIs directly. All screenshot and input calls go through that package.
• Views build separately from the runtime. build:js (tsup) and build:views (Vite) are independent steps. Both must run for a complete build.
• Viewer HTML is inline. renderViewerHtml() in routes.ts returns a self-contained HTML string with embedded CSS and JS — no external assets needed. It polls for frames at 500 ms intervals.
• One GUI view. ScreenshareView is the single authored shipped view. ScreenshareSpatialView is the presentational source; do not reintroduce a parallel operator-surface duplicate.
• interact is a separate module. The view capability handler lives in ui/screenshare-interact.ts, not in a component file, so component files export only React components (keeping them Fast-Refresh-compatible).
• See root AGENTS.md for architecture rules (logger-only, ESM, naming, layer boundaries) that apply repo-wide.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — native / on-device bridge:

• The capability run on a real device or simulator — not desktop Chromium against a mocked bridge (see #9967/#9580): device logs + the captured output (photo, OCR text, detection boxes, transcript, sensor reading).
• Parity vs the reference implementation where one exists (e.g. the Python/Ultralytics reference), with the numeric tolerances actually met.
• Permission-denied, no-hardware, and background/foreground lifecycle paths.
• A short recording of the on-device run; confirm the build under test is yours (versionName / a known on-screen change), not a stale install.