elizaos--eliza/plugins/plugin-embeddings/AGENTS.md

@elizaos/plugin-embeddings

Provider-agnostic ("bring your own") TEXT_EMBEDDING provider for elizaOS agents. Points a single set of EMBEDDING_* vars at any OpenAI-compatible /embeddings endpoint, independent of the chat brain.

Purpose / role

Decouples embeddings from text generation. A self-hosted bot whose chat provider serves no good embeddings — Claude (no embeddings API), Cerebras (no embeddings) — can still get high-quality vectors by setting EMBEDDING_BASE_URL / EMBEDDING_API_KEY to a personal OpenAI key, an Eliza Cloud URL, Voyage, or a local TEI / Infinity / vLLM / LM Studio server.

Purely additive. The plugin auto-enables only when EMBEDDING_BASE_URL or EMBEDDING_API_KEY is set (see auto-enable.ts). With neither set it never loads, so existing deployments — which use their chat provider's embedding slot, local inference, or Eliza Cloud — are unaffected.

It registers only the embedding slots — no text/image/audio handlers, no actions, providers, services, or evaluators.

Plugin surface

Model type	Handler	File
ModelType.TEXT_EMBEDDING	handleTextEmbedding	src/models/embedding.ts
ModelType.TEXT_EMBEDDING_BATCH	handleBatchTextEmbedding	src/models/embedding.ts

Both POST { model, input, ...(explicit dimensions ? { dimensions } : {}) } to `${EMBEDDING_BASE_URL}/embeddings` using raw fetch (no @ai-sdk dependency), parse the OpenAI-compatible response, validate the returned width against the configured dimension, and emit a MODEL_USED event.

Registration priority

The plugin registers at priority: 1. The native priority sort for the embedding slot is:

local-inference @ 0  <  plugin-embeddings @ 1  <  Eliza Cloud @ 50

So a bring-your-own endpoint beats a bare local embedder but yields to a paired Eliza Cloud. This is the desired default; override per-slot via the runtime routing preferences when a different precedence is wanted.

Error policy (Commandment 8 / issue #9324)

On any HTTP, config, or response-shape error the handler THROWS — it never returns a zero or fabricated vector, which would silently corrupt the embedding store. The single legitimate synthetic return is the boot dimension-probe: the runtime calls useModel(TEXT_EMBEDDING, null) purely to read .length, so a correctly-sized marker vector ([0.1, 0, 0, …]) is returned for null input only. There is no Cerebras deterministic-fallback branch (dropped from the lifted OpenAI handler) and no default endpoint — a missing EMBEDDING_BASE_URL throws.

Layout

plugins/plugin-embeddings/
  index.ts / index.node.ts / index.browser.ts   Build entrypoints (re-export src/index)
  auto-enable.ts        Manifest entry-point — env-only shouldEnable (no transitive imports)
  build.ts              Bun.build (node + browser + cjs) + tsc declarations
  src/
    index.ts            embeddingsPlugin — models map + init() config validation/logging
    models/
      embedding.ts      handleTextEmbedding + handleBatchTextEmbedding (raw fetch, THROW on error)
      index.ts          Re-exports handlers
    utils/
      config.ts         Provider-neutral getSetting-based getters
      events.ts         emitModelUsageEvent (MODEL_USED)
    types/
      index.ts          EmbeddingResponse, TokenUsage
  __tests__/
    embedding.test.ts   Null-probe width, wire-mocked vector, dimension-mismatch/empty/unsupported throws, batch, VECTOR_DIMS contract
    config.test.ts      Provider-neutral getter resolution + no chat fallback
    auto-enable.test.ts shouldEnable opt-in semantics

Commands

bun run --cwd plugins/plugin-embeddings build        # Bun.build (node + browser + cjs) + tsc d.ts
bun run --cwd plugins/plugin-embeddings dev          # watch build
bun run --cwd plugins/plugin-embeddings test         # vitest unit suite
bun run --cwd plugins/plugin-embeddings typecheck    # tsc --noEmit --noCheck
bun run --cwd plugins/plugin-embeddings lint         # biome check --write --unsafe
bun run --cwd plugins/plugin-embeddings lint:check   # biome check (read-only)
bun run --cwd plugins/plugin-embeddings format       # biome format --write
bun run --cwd plugins/plugin-embeddings clean        # rm -rf dist .turbo …

Config / env vars

All read via getSetting(runtime, key) (runtime/character config first, then process.env), so every value is per-character overridable. There is no fallback to a chat provider's settings (OPENAI_*, ELIZAOS_CLOUD_*, …) — this plugin owns the embedding slot independently.

Var	Required	Default	Purpose
EMBEDDING_BASE_URL	one-of*	—	Base URL of an OpenAI-compatible /embeddings endpoint. No default — unset throws.
EMBEDDING_API_KEY	one-of*	—	Bearer token for the endpoint. Omit for local servers needing no auth.
EMBEDDING_MODEL	no	text-embedding-3-small	Model id sent as the request model field.
EMBEDDING_DIMENSIONS	no	1536	Vector width. When explicitly set, sent as the request dimensions field.
EMBEDDING_BROWSER_URL	no	—	Browser-only server-side proxy URL. In a browser build the Authorization header is sent only when this is set, keeping the key server-side.

* Setting either EMBEDDING_BASE_URL or EMBEDDING_API_KEY is what activates the plugin. For real (non-probe) embedding calls a EMBEDDING_BASE_URL is required or the handler throws.

Supported dimensions

EMBEDDING_DIMENSIONS must be one of the elizaOS VECTOR_DIMS (imported from @elizaos/core):

384, 512, 768, 1024, 1536, 2048, 3072

An unsupported value throws at boot and on every call.

Dimension-switch caveat

The embedding dimension is part of the database vector schema. Changing EMBEDDING_DIMENSIONS (or the model's native width) invalidates existing vectors until the database adapter reclaims old-width embeddings and re-embeds those memories at the active width. SQL adapters do this through clearEmbeddingsOutsideActiveDimension() during runtime boot; custom stores need an equivalent path before searches are trustworthy after a switch.

How to extend

This plugin is intentionally embedding-only. To add another OpenAI-compatible embedding behavior, add a helper in src/models/embedding.ts, re-export from src/models/index.ts, and (if a new slot) wire it into the models map in src/index.ts. Add any new env var to src/utils/config.ts (follow the getSetting pattern) and to agentConfig.pluginParameters in package.json.

Registration / discovery

No central list edit is needed. The plugin lives under the repo plugins/* workspace glob (so bun install symlinks it into node_modules) and declares elizaos.plugin.autoEnableModule in package.json. The agent's plugin-candidate discovery (packages/agent/src/runtime/plugin-resolver.ts → discoverPluginCandidates) walks node_modules and the workspace plugins/ dir, reads each elizaos.plugin manifest, and the auto-enable engine (packages/shared/src/config/plugin-manifest.ts) runs shouldEnable. This is identical to how plugin-lmstudio is wired — neither plugin appears in core-plugins.ts nor as a dep of packages/agent.

Conventions / gotchas

• Raw fetch, no @ai-sdk. Mirrors plugin-openai's transport. The only runtime dependency is @elizaos/core (peer/workspace:*).
• THROW, never fabricate. Any failure throws so the runtime falls through to another provider instead of persisting a corrupt vector.
• Browser key safety. The Authorization header is suppressed in browser builds unless EMBEDDING_BROWSER_URL is set (the proxy injects auth server-side).
• Dual build (node + browser). dist/node/index.node.js and dist/browser/index.browser.js.
• See the repo-root AGENTS.md for logger-only, ESM, naming, and architecture rules.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — model provider:

• A trajectory from a live call to this provider (not the proxy, not a mock): full request, raw response, token usage, finish reason, and streamed chunks.
• Proof of tool/function-calling and structured-output parsing against the real model.
• The error paths exercised: bad key, model-not-found, oversized context, timeout, rate-limit, mid-stream disconnect — plus latency and cost from the real call.
• If no key is available in CI, attach the documented live-run transcript as evidence — never a mocked client passed off as a pass.