elizaos--eliza/plugins/plugin-video/AGENTS.md

@elizaos/plugin-video

Video download, processing, and transcription service for Eliza agents.

Purpose / role

Adds VideoService to an Eliza agent's service registry (ServiceType.VIDEO). The service handles downloading videos from YouTube, Vimeo, and direct MP4 URLs; converting them to audio; extracting transcripts from subtitles or captions; and falling back to an audio transcription service when no subtitles are available. The plugin has no actions, providers, or routes — all capability is exposed through the service interface. It is opt-in; agents must include @elizaos/plugin-video in their plugin list explicitly.

Plugin surface

Kind	Name	Description
Service	VideoService (ServiceType.VIDEO)	Download, transcode, and transcribe video from URLs

No actions, providers, evaluators, routes, or events are registered.

Layout

src/
  index.ts                  Plugin definition; exports default videoPlugin
  services/
    video.ts                VideoService — IVideoService implementation
    binaries.ts             BinaryResolver — yt-dlp download/update lifecycle; ffmpeg path resolution
    binaries.integration.test.ts  Integration tests for BinaryResolver

Key entry: dist/index.js (ESM only).

Commands

Scripts defined in this package's package.json:

bun run --cwd plugins/plugin-video build      # compile via build.ts
bun run --cwd plugins/plugin-video dev        # same as build (watch not set up)
bun run --cwd plugins/plugin-video typecheck  # tsgo --noEmit
bun run --cwd plugins/plugin-video test       # vitest run

Config / env vars

All vars are optional. The plugin falls back to managed-cache download when no overrides are set.

Env var	Effect
ELIZA_YT_DLP_PATH	Use this yt-dlp binary instead of the managed cache or system PATH
ELIZA_YT_DLP_PREFER_PATH	1 to prefer system PATH yt-dlp over the managed cache
ELIZA_DISABLE_YTDLP_AUTOUPDATE	1 to disable automatic yt-dlp self-update on extractor errors
ELIZA_FFMPEG_PATH	Use this ffmpeg binary; overrides system PATH and ffmpeg-static
ELIZA_NODE_BIN / NODE_BINARY	Node executable used when running the bundled ffmpeg installer from non-Node runtimes
ELIZA_BINARIES_DIR	Directory where the managed yt-dlp binary and metadata are cached (default: <stateDir>/binaries)

ELIZA_STATE_DIR / ELIZA_BINARIES_DIR interact: BinaryResolver calls resolveStateDir() from @elizaos/core to find the default binaries directory.

How to extend

Add a new method to VideoService

1. Add the method signature to IVideoService in @elizaos/core (that interface is the public contract).
2. Implement it in src/services/video.ts inside VideoService.
3. Wire any needed yt-dlp flags through this.binaries.runYtDlp(url, flags).

Add a new action that uses VideoService

1. Create src/actions/my-action.ts exporting an Action object.
2. Inside the action handler, call runtime.getService<VideoService>(ServiceType.VIDEO) to get the service.
3. Add the action to the actions array in src/index.ts.

Add a new provider

Same pattern: create src/providers/my-provider.ts, export a Provider, and add it to providers in src/index.ts.

Conventions / gotchas

• Service singleton per runtime. VideoService keeps a serial processingChain promise so concurrent processVideo calls are queued; don't bypass this.
• Cache directory is ./content_cache relative to the process cwd. Files are never auto-deleted; callers are responsible for cleanup if storage is a concern.
• yt-dlp auto-update. On extractor failures matching known error patterns (Sign in to confirm, nsig extraction failed, HTTP Error 403, etc.), BinaryResolver re-downloads yt-dlp from the GitHub releases API and retries. This is throttled to once per hour. Set ELIZA_DISABLE_YTDLP_AUTOUPDATE=1 to disable.
• ffmpeg is required for audio extraction and thumbnail generation. The plugin resolves it from ELIZA_FFMPEG_PATH, then system PATH, then the ffmpeg-static npm package. If the package exists but its binary payload is missing because install scripts were skipped, the resolver runs the package installer once before declaring ffmpeg unavailable.
• Transcription fallback requires ServiceType.TRANSCRIPTION. If no subtitles or captions exist and the video is not categorised as Music, VideoService calls runtime.getService<ITranscriptionService>(ServiceType.TRANSCRIPTION). Load a plugin that registers an ITranscriptionService under ServiceType.TRANSCRIPTION, or this path throws "Transcription service not found".
• Direct MP4 URLs are detected by extension and bypass yt-dlp for metadata; they use a simple HEAD check and fall back to yt-dlp if unreachable.
• BinaryResolver is a singleton (BinaryResolver.instance()). In tests, call BinaryResolver.resetForTests() between cases to avoid state leakage.
• See root AGENTS.md for repo-wide rules (logger-only logging, ESM, architecture commandments, naming).

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — native / on-device bridge:

• The capability run on a real device or simulator — not desktop Chromium against a mocked bridge (see #9967/#9580): device logs + the captured output (photo, OCR text, detection boxes, transcript, sensor reading).
• Parity vs the reference implementation where one exists (e.g. the Python/Ultralytics reference), with the numeric tolerances actually met.
• Permission-denied, no-hardware, and background/foreground lifecycle paths.
• A short recording of the on-device run; confirm the build under test is yours (versionName / a known on-screen change), not a stale install.