# @elizaos/plugin-streaming RTMP live streaming for Eliza agents: Twitch, YouTube Live, X (Twitter), pump.fun, custom ingest URLs, and named extra ingests. ## Purpose / role Adds RTMP streaming control to an Eliza agent. The agent can start, stop, and check status of live streams via the `STREAM` action. The plugin manages the FFmpeg pipeline, screen/audio capture, overlay layout persistence, and TTS-to-stream audio bridging. It is opt-in: `auto-enable.ts` activates it only when at least one streaming destination is configured under the `streaming` key in agent config (or via env vars). ## Plugin surface **Actions** - `STREAM` — start/stop/status for twitch, youtube, x, pumpfun; dispatches to `POST /api/stream/live`, `POST /api/stream/offline`, `GET /api/stream/status`; `roleGate: ADMIN`; similes: `START_STREAM`, `STOP_STREAM`, `GET_STREAM_STATUS`, `GO_LIVE`, `GO_OFFLINE`, `STREAM_STATUS`, `IS_LIVE`. **Providers** - `streamStatus` — per-turn provider; fetches `GET /api/stream/status` for all four platforms and emits JSON context; `contexts: ["media", "automation"]`. **Services / singletons** (not registered as elizaOS Services) - `streamManager` (`StreamManager`) — FFmpeg lifecycle; exported singleton from public API; video input modes: `pipe`, `avfoundation`, `screen`, `x11grab`, `file`, `testsrc`; audio sources: `silent`, `system`, `microphone`, `tts`, file path; volume/mute; auto-restart with exponential backoff. - `ttsStreamBridge` (`TtsStreamBridge`) — internal singleton (not re-exported from package index); generates TTS (ElevenLabs, OpenAI, Edge TTS, local inference) and feeds PCM s16le 24 kHz mono into FFmpeg pipe:3; attach/detach lifecycle follows `streamManager.start()/stop()`. **Route handler** (not registered in `Plugin.routes`; consumed externally) - `handleStreamRoute` — handles `/api/stream/*` and `/api/streaming/*` endpoints: frame ingest, MJPEG preview, stream start/stop, status, volume, mute/unmute, destination management, settings persistence, source switching. - `handleTtsRoutes` — handles `/api/tts/config`, `/api/tts/elevenlabs`, `/api/tts/local-inference`. ## Layout ``` plugins/plugin-streaming/ package.json npm metadata, agentConfig env var declarations auto-enable.ts shouldEnable() — activates when a dest is configured src/ index.ts Plugin export: streamingPlugin (actions, providers, config) Platform preset configs (TWITCH_CFG, YOUTUBE_CFG, X_CFG, PUMPFUN_CFG) Destination factory exports: createTwitchDestination, createYoutubeDestination, createXStreamDestination, createPumpfunDestination, createCustomRtmpDestination, createNamedRtmpDestination core.ts Shared types (StreamingDestination, StreamingPluginConfig, OverlayLayoutData, StreamingBackend) buildPresetLayout() — overlay widget layout factory createStreamingDestination() — direct RTMP destination createCloudRelayDestination() — cloud-relay destination createStreamingPlugin() — per-platform plugin+factory bundle buildStreamOpAction() — builds the STREAM action streamStatusProvider — the streamStatus provider resolveStreamingBackend() — direct vs. cloud selection logic api/ stream-routes.ts handleStreamRoute(); detectCaptureMode(); ensureXvfb() MJPEG frame store; pipeline start/stop coordination stream-persistence.ts readStreamSettings, writeStreamSettings, readOverlayLayout, writeOverlayLayout, seedOverlayDefaults, validateStreamSettings, getHeadlessCaptureConfig, parseDestinationQuery, safeDestId stream-route-state.ts StreamRouteState interface (destinations map, streamManager ref, etc.) streaming-types.ts Re-exports StreamingDestination, OverlayLayoutData, OverlayWidgetInstance from core.ts (single nominal type) streaming-text.ts mergeStreamingText(), resolveStreamingUpdate() — token-stream merge tts-routes.ts handleTtsRoutes(); local inference + ElevenLabs + config route services/ stream-manager.ts StreamManager class; streamManager singleton; StreamConfig, AudioSource types tts-stream-bridge.ts TtsStreamBridge class; ttsStreamBridge singleton; resolveTtsConfig(); getTtsProviderStatus(); TtsConfig, ResolvedTtsConfig, ITtsStreamBridge ``` ## Commands ```bash bun run --cwd plugins/plugin-streaming build # tsup + declaration emit bun run --cwd plugins/plugin-streaming dev # tsup --watch bun run --cwd plugins/plugin-streaming test # vitest run bun run --cwd plugins/plugin-streaming typecheck # tsgo --noEmit bun run --cwd plugins/plugin-streaming lint # biome check --write --unsafe src bun run --cwd plugins/plugin-streaming lint:check # biome check src bun run --cwd plugins/plugin-streaming format # biome format --write . bun run --cwd plugins/plugin-streaming clean # rm dist .turbo tsconfig artifacts ``` ## Config / env vars All are optional. The plugin stays inactive if no streaming destination is configured. | Env var | Purpose | |---|---| | `TWITCH_STREAM_KEY` | Twitch RTMP stream key | | `YOUTUBE_STREAM_KEY` | YouTube Live stream key | | `YOUTUBE_RTMP_URL` | Override YouTube RTMP ingest URL | | `X_STREAM_KEY` | X (Twitter) stream key | | `X_RTMP_URL` | X RTMP ingest URL (required with X_STREAM_KEY) | | `PUMPFUN_STREAM_KEY` | pump.fun stream key | | `PUMPFUN_RTMP_URL` | pump.fun RTMP ingest URL | | `CUSTOM_RTMP_URL` | Custom RTMP ingest URL | | `CUSTOM_RTMP_KEY` | Custom RTMP stream key | | `_STREAMING_BACKEND` | `direct` \| `cloud` \| `auto` (default `auto`); e.g. `TWITCH_STREAMING_BACKEND` | | `ELIZAOS_CLOUD_API_KEY` | Eliza Cloud API key — required for cloud relay mode | | `ELIZAOS_CLOUD_BASE_URL` | Override Eliza Cloud base URL (default `https://www.elizacloud.ai/api/v1`) | | `ELIZAOS_CLOUD_ENABLED` | Enable cloud connection check for auto backend selection | | `ELIZAOS_CLOUD_USE_TTS` | Set to `true` to force-enable Eliza Cloud TTS in the stream bridge; set to `false` to force-disable | | `ELIZA_CLOUD_TTS_DISABLED` | Set to `true` to prevent the TTS bridge from using Eliza Cloud TTS | | `STREAM_MODE` | Override capture mode: `pipe`, `x11grab`, `avfoundation`, `screen`, `file` | | `STREAM_AUDIO_SOURCE` | Audio source: `silent`, `system`, `microphone`, `tts`, or file path | | `STREAM_AUDIO_DEVICE` | Platform-specific audio device identifier | | `STREAM_VOLUME` | Volume 0–100 (default `80`) | | `STREAM_DISPLAY` | X11 display for x11grab mode (default `:99`) | | `DISPLAY` | X11 display identifier; auto-detected for x11grab capture mode and managed by `ensureXvfb()` | | `STREAM_VIDEO_DEVICE` | avfoundation video device index (default `3`) | | `STREAM_CAPTURE_URL` | URL for headless browser capture (file/x11grab fallback) | | `STREAM_THEME` | Overlay theme (fallback if not in stream-settings.json) | | `STREAM_AVATAR_INDEX` | Overlay avatar index (fallback if not in stream-settings.json) | | `ELEVENLABS_API_KEY` | ElevenLabs API key (TTS audio for streams) | | `OPENAI_API_KEY` | OpenAI API key (TTS audio for streams) | | `SERVER_PORT` / `PORT` | HTTP server port used to build internal API URLs (default `2138`) | | `ELIZA_DATA_DIR` | Data root; stream settings + overlay layouts land in `/stream/` | Overlay layout and visual settings are persisted in `/stream/` (default `./data/stream/`). Per-destination layout files are named `overlay-layout-.json`. ## How to extend **Add a new action:** 1. Write the action object (implementing `Action` from `@elizaos/core`) in `src/core.ts` or a new file under `src/api/`. 2. Add it to `streamingPlugin.actions` array in `src/index.ts`. **Add a new provider:** 1. Write the provider object (implementing `Provider`) in `src/core.ts` or a new file. 2. Add it to `streamingPlugin.providers` array in `src/index.ts`. **Add a new streaming platform preset:** 1. Define a `StreamingPluginConfig` constant in `src/index.ts` (see `TWITCH_CFG` pattern). 2. Call `createStreamingPlugin(cfg)` to get `{ plugin, createDestination }`. 3. Export a `createDestination` factory function. 4. Optionally add the platform to `STREAMING_PLATFORMS` in `src/core.ts` so `STREAM` action and `streamStatus` provider cover it. **Add a new stream route:** Add a conditional branch in `handleStreamRoute()` in `src/api/stream-routes.ts`. The function returns `true` when it handles the request, `false` to fall through. ## Conventions / gotchas - **FFmpeg is a hard runtime dependency.** `streamManager.start()` runs `ffmpeg -version` at startup and throws with an install hint if not found. There is no bundled binary. - **Cloud relay vs. direct:** `resolveStreamingBackend()` auto-selects cloud relay when Eliza Cloud is connected AND no local stream key is set. Force a mode with `_STREAMING_BACKEND=direct|cloud`. - **TTS audio uses pipe:3.** `TtsStreamBridge` attaches to FFmpeg's 4th stdio fd. The PCM format is fixed: s16le, 24 kHz, mono. Audio is decoded via a second FFmpeg subprocess spawned per speak call. - **`handleStreamRoute` and `handleTtsRoutes` are not in `Plugin.routes`.** They are imperative route-handler functions intended to be wired into an HTTP server by the consuming runtime. The plugin object registers only `actions` and `providers`. - **Overlay layouts** are seeded from `destination.defaultOverlayLayout` on first stream start per destination, then persisted as JSON files. Subsequent starts read from the file. - **`streaming-text.ts`** (`mergeStreamingText`, `resolveStreamingUpdate`) is a standalone utility for de-duplicating overlapping token-stream chunks; it has no dependency on streaming state. - **`auto-enable.ts`** is a separate entry point referenced by the elizaOS plugin auto-enable system. It must stay import-free from the full plugin runtime (no `@elizaos/core` service imports). - **`node-edge-tts`** is an optional peer dependency for Edge TTS; the bridge catches the import error and surfaces a clear message if it is not installed. ## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests > The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it. > Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it > works **without reading the code**, from the artifacts you attach. This applies to **every** > feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof. - **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then **open the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence (`packages/scenario-runner/bin/eliza-scenarios run --report `). - **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that drive the *real* path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test **does not count**. If the real model/device/chain/connector/account is hard to reach, **make it reachable — that is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change. - **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of the entire feature or view, start to finish (`bun run test:e2e:record`). - **Manually review every artifact the change touches** — never just the green check: client logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out, before/after full-page screenshots, **and the domain artifacts listed below for this package.** - **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every blocker by the **hard path**: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted. Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `
`); attach each evidence type **or** explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed behavior, **re-capture** evidence; stale proof is worse than none. **Capture & manually review for this package — voice / audio:** - Captured **audio** of the real round-trip (STT in, TTS out) plus the transcript, with a narrated walkthrough of what is happening. - Latency, barge-in/interruption, and wake-word behavior measured on real audio — across platforms, not Linux-x64-synthetic only (see #9958). - The model trajectory for any LLM turn inside the loop. - Failure paths: no mic, silence, noise, overlapping speech, network drop mid-stream.