elizaos--eliza/plugins/plugin-streaming/AGENTS.md

@elizaos/plugin-streaming

RTMP live streaming for Eliza agents: Twitch, YouTube Live, X (Twitter), pump.fun, custom ingest URLs, and named extra ingests.

Purpose / role

Adds RTMP streaming control to an Eliza agent. The agent can start, stop, and check status of live streams via the STREAM action. The plugin manages the FFmpeg pipeline, screen/audio capture, overlay layout persistence, and TTS-to-stream audio bridging. It is opt-in: auto-enable.ts activates it only when at least one streaming destination is configured under the streaming key in agent config (or via env vars).

Plugin surface

Actions

• STREAM — start/stop/status for twitch, youtube, x, pumpfun; dispatches to POST /api/stream/live, POST /api/stream/offline, GET /api/stream/status; roleGate: ADMIN; similes: START_STREAM, STOP_STREAM, GET_STREAM_STATUS, GO_LIVE, GO_OFFLINE, STREAM_STATUS, IS_LIVE.

Providers

• streamStatus — per-turn provider; fetches GET /api/stream/status for all four platforms and emits JSON context; contexts: ["media", "automation"].

Services / singletons (not registered as elizaOS Services)

• streamManager (StreamManager) — FFmpeg lifecycle; exported singleton from public API; video input modes: pipe, avfoundation, screen, x11grab, file, testsrc; audio sources: silent, system, microphone, tts, file path; volume/mute; auto-restart with exponential backoff.
• ttsStreamBridge (TtsStreamBridge) — internal singleton (not re-exported from package index); generates TTS (ElevenLabs, OpenAI, Edge TTS, local inference) and feeds PCM s16le 24 kHz mono into FFmpeg pipe:3; attach/detach lifecycle follows streamManager.start()/stop().

Route handler (not registered in Plugin.routes; consumed externally)

• handleStreamRoute — handles /api/stream/* and /api/streaming/* endpoints: frame ingest, MJPEG preview, stream start/stop, status, volume, mute/unmute, destination management, settings persistence, source switching.
• handleTtsRoutes — handles /api/tts/config, /api/tts/elevenlabs, /api/tts/local-inference.

Layout

plugins/plugin-streaming/
  package.json            npm metadata, agentConfig env var declarations
  auto-enable.ts          shouldEnable() — activates when a dest is configured
  src/
    index.ts              Plugin export: streamingPlugin (actions, providers, config)
                          Platform preset configs (TWITCH_CFG, YOUTUBE_CFG, X_CFG, PUMPFUN_CFG)
                          Destination factory exports: createTwitchDestination,
                            createYoutubeDestination, createXStreamDestination,
                            createPumpfunDestination, createCustomRtmpDestination,
                            createNamedRtmpDestination
    core.ts               Shared types (StreamingDestination, StreamingPluginConfig,
                            OverlayLayoutData, StreamingBackend)
                          buildPresetLayout() — overlay widget layout factory
                          createStreamingDestination() — direct RTMP destination
                          createCloudRelayDestination() — cloud-relay destination
                          createStreamingPlugin() — per-platform plugin+factory bundle
                          buildStreamOpAction() — builds the STREAM action
                          streamStatusProvider — the streamStatus provider
                          resolveStreamingBackend() — direct vs. cloud selection logic
    api/
      stream-routes.ts    handleStreamRoute(); detectCaptureMode(); ensureXvfb()
                          MJPEG frame store; pipeline start/stop coordination
      stream-persistence.ts  readStreamSettings, writeStreamSettings, readOverlayLayout,
                              writeOverlayLayout, seedOverlayDefaults, validateStreamSettings,
                              getHeadlessCaptureConfig, parseDestinationQuery, safeDestId
      stream-route-state.ts  StreamRouteState interface (destinations map, streamManager ref, etc.)
      streaming-types.ts     Re-exports StreamingDestination, OverlayLayoutData,
                              OverlayWidgetInstance from core.ts (single nominal type)
      streaming-text.ts      mergeStreamingText(), resolveStreamingUpdate() — token-stream merge
      tts-routes.ts          handleTtsRoutes(); local inference + ElevenLabs + config route
    services/
      stream-manager.ts   StreamManager class; streamManager singleton; StreamConfig, AudioSource types
      tts-stream-bridge.ts  TtsStreamBridge class; ttsStreamBridge singleton; resolveTtsConfig();
                             getTtsProviderStatus(); TtsConfig, ResolvedTtsConfig, ITtsStreamBridge

Commands

bun run --cwd plugins/plugin-streaming build        # tsup + declaration emit
bun run --cwd plugins/plugin-streaming dev          # tsup --watch
bun run --cwd plugins/plugin-streaming test         # vitest run
bun run --cwd plugins/plugin-streaming typecheck    # tsgo --noEmit
bun run --cwd plugins/plugin-streaming lint         # biome check --write --unsafe src
bun run --cwd plugins/plugin-streaming lint:check   # biome check src
bun run --cwd plugins/plugin-streaming format       # biome format --write .
bun run --cwd plugins/plugin-streaming clean        # rm dist .turbo tsconfig artifacts

Config / env vars

All are optional. The plugin stays inactive if no streaming destination is configured.

Env var	Purpose
TWITCH_STREAM_KEY	Twitch RTMP stream key
YOUTUBE_STREAM_KEY	YouTube Live stream key
YOUTUBE_RTMP_URL	Override YouTube RTMP ingest URL
X_STREAM_KEY	X (Twitter) stream key
X_RTMP_URL	X RTMP ingest URL (required with X_STREAM_KEY)
PUMPFUN_STREAM_KEY	pump.fun stream key
PUMPFUN_RTMP_URL	pump.fun RTMP ingest URL
CUSTOM_RTMP_URL	Custom RTMP ingest URL
CUSTOM_RTMP_KEY	Custom RTMP stream key
<PLATFORM>_STREAMING_BACKEND	direct | cloud | auto (default auto); e.g. TWITCH_STREAMING_BACKEND
ELIZAOS_CLOUD_API_KEY	Eliza Cloud API key — required for cloud relay mode
ELIZAOS_CLOUD_BASE_URL	Override Eliza Cloud base URL (default https://www.elizacloud.ai/api/v1)
ELIZAOS_CLOUD_ENABLED	Enable cloud connection check for auto backend selection
ELIZAOS_CLOUD_USE_TTS	Set to true to force-enable Eliza Cloud TTS in the stream bridge; set to false to force-disable
ELIZA_CLOUD_TTS_DISABLED	Set to true to prevent the TTS bridge from using Eliza Cloud TTS
STREAM_MODE	Override capture mode: pipe, x11grab, avfoundation, screen, file
STREAM_AUDIO_SOURCE	Audio source: silent, system, microphone, tts, or file path
STREAM_AUDIO_DEVICE	Platform-specific audio device identifier
STREAM_VOLUME	Volume 0–100 (default 80)
STREAM_DISPLAY	X11 display for x11grab mode (default :99)
DISPLAY	X11 display identifier; auto-detected for x11grab capture mode and managed by ensureXvfb()
STREAM_VIDEO_DEVICE	avfoundation video device index (default 3)
STREAM_CAPTURE_URL	URL for headless browser capture (file/x11grab fallback)
STREAM_THEME	Overlay theme (fallback if not in stream-settings.json)
STREAM_AVATAR_INDEX	Overlay avatar index (fallback if not in stream-settings.json)
ELEVENLABS_API_KEY	ElevenLabs API key (TTS audio for streams)
OPENAI_API_KEY	OpenAI API key (TTS audio for streams)
SERVER_PORT / PORT	HTTP server port used to build internal API URLs (default 2138)
ELIZA_DATA_DIR	Data root; stream settings + overlay layouts land in <dir>/stream/

Overlay layout and visual settings are persisted in <ELIZA_DATA_DIR>/stream/ (default ./data/stream/). Per-destination layout files are named overlay-layout-<destId>.json.

How to extend

Add a new action:

1. Write the action object (implementing Action from @elizaos/core) in src/core.ts or a new file under src/api/.
2. Add it to streamingPlugin.actions array in src/index.ts.

Add a new provider:

1. Write the provider object (implementing Provider) in src/core.ts or a new file.
2. Add it to streamingPlugin.providers array in src/index.ts.

Add a new streaming platform preset:

1. Define a StreamingPluginConfig constant in src/index.ts (see TWITCH_CFG pattern).
2. Call createStreamingPlugin(cfg) to get { plugin, createDestination }.
3. Export a create<Platform>Destination factory function.
4. Optionally add the platform to STREAMING_PLATFORMS in src/core.ts so STREAM action and streamStatus provider cover it.

Add a new stream route: Add a conditional branch in handleStreamRoute() in src/api/stream-routes.ts. The function returns true when it handles the request, false to fall through.

Conventions / gotchas

• FFmpeg is a hard runtime dependency. streamManager.start() runs ffmpeg -version at startup and throws with an install hint if not found. There is no bundled binary.
• Cloud relay vs. direct: resolveStreamingBackend() auto-selects cloud relay when Eliza Cloud is connected AND no local stream key is set. Force a mode with <PLATFORM>_STREAMING_BACKEND=direct|cloud.
• TTS audio uses pipe:3. TtsStreamBridge attaches to FFmpeg's 4th stdio fd. The PCM format is fixed: s16le, 24 kHz, mono. Audio is decoded via a second FFmpeg subprocess spawned per speak call.
• handleStreamRoute and handleTtsRoutes are not in Plugin.routes. They are imperative route-handler functions intended to be wired into an HTTP server by the consuming runtime. The plugin object registers only actions and providers.
• Overlay layouts are seeded from destination.defaultOverlayLayout on first stream start per destination, then persisted as JSON files. Subsequent starts read from the file.
• streaming-text.ts (mergeStreamingText, resolveStreamingUpdate) is a standalone utility for de-duplicating overlapping token-stream chunks; it has no dependency on streaming state.
• auto-enable.ts is a separate entry point referenced by the elizaOS plugin auto-enable system. It must stay import-free from the full plugin runtime (no @elizaos/core service imports).
• node-edge-tts is an optional peer dependency for Edge TTS; the bridge catches the import error and surfaces a clear message if it is not installed.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — voice / audio:

• Captured audio of the real round-trip (STT in, TTS out) plus the transcript, with a narrated walkthrough of what is happening.
• Latency, barge-in/interruption, and wake-word behavior measured on real audio — across platforms, not Linux-x64-synthetic only (see #9958).
• The model trajectory for any LLM turn inside the loop.
• Failure paths: no mic, silence, noise, overlapping speech, network drop mid-stream.