evermind-ai--everos/docs/multimodal.md

Multimodal Memory

EverOS turns non-text content — images, PDFs, audio, office documents, HTML, email — into the same structured, searchable memory as plain text. You attach the asset to a message at ingest time; a vision/audio capable LLM parses it into text, and from there it flows through the identical extraction → markdown → index pipeline as any text turn. The result is fully retrievable with the same /search stack.

Table of contents

• How it works
• Prerequisites
  • Install the extra
  • LibreOffice (office documents only)
  • Configure the multimodal LLM
• Supported modalities
• Sending multimodal content
  • Payload: uri vs base64
  • Example: image by URL
  • Example: mixed text + image in one turn
  • Example: inline PDF via base64
  • Example: local file via file://
  • Calling from Python (plain HTTP)
• Configuration reference
• Errors and limits
• Searching multimodal memory

How it works

POST /api/v1/memory/add
  messages[].content = [ ContentItem, ContentItem, ... ]
        │
        │  text items      → used verbatim
        │  non-text items  → multimodal LLM (everalgo-parser)
        ▼
  parsed text merged back into the session buffer (in original order)
        │
        ▼
  boundary detector → extraction LLM → memory cell (MemCell)
        │
        ▼
  markdown (truth)  +  SQLite (state)  +  LanceDB (vector + BM25)
        │
        ▼
  retrievable via /search and /get like any text memory

Each non-text ContentItem is routed through the parser, which calls a separate, vision/audio capable LLM (configured independently from the main extraction [llm], so parsing can target a multimodal endpoint without changing boundary or extraction behaviour). Visual/audio formats (image / pdf / audio / office) always go through that LLM; a few text-bearing formats can be parsed without it (e.g. a plain email with no inline images). The parser returns text; that text takes the place of the asset in the message buffer. Nothing downstream of the parser knows or cares that the content originated as an image or PDF — the raw bytes are not persisted past extraction (the episode and memory cell (MemCell) store only the parsed text).

Prerequisites

Install the extra

Multimodal parsing lives behind an optional dependency group so the base install stays lean:

uv pip install 'everos[multimodal]'    # or: pip install 'everos[multimodal]'

This pulls in everalgo-parser[svg] — the [svg] bundle adds cairosvg so SVG works out of the box.

LibreOffice (office documents only)

Office formats (.doc / .docx / .ppt / .pptx / .xls / .xlsx) are converted to PDF before being fed to the multimodal LLM. The parser shells out to soffice, LibreOffice's headless renderer, so LibreOffice must be present on the server host:

brew install --cask libreoffice          # macOS
sudo apt-get install -y libreoffice       # Debian / Ubuntu

Without LibreOffice, office uploads return 503 (CAPABILITY_UNAVAILABLE) with a clear error message; image / PDF / audio / HTML / email parsing is unaffected.

Configure the multimodal LLM

The parser uses its own LLM section ([multimodal] in everos.toml), independent from [llm]. The model must accept OpenAI image_url parts. Fill in three fields in everos.toml:

[multimodal]
model    = "google/gemini-3-flash-preview"   # must support image_url parts
base_url = "https://openrouter.ai/api/v1"
api_key  = "<your key>"

See Configuration reference for the full field list.

Supported modalities

type	Typical formats	Payload	Notes
text	—	text	Plain text; the string shorthand also maps here
image	PNG / JPG / GIF / WebP / SVG	uri or base64	SVG via the bundled cairosvg
pdf	PDF	uri or base64	—
audio	MP3 / WAV / …	uri or base64	Endpoint must accept audio parts
doc	DOC / DOCX / PPT / PPTX / XLS / XLSX	uri or base64	Requires LibreOffice (converted to PDF first)
html	HTML	uri or base64	To inline HTML as plain text instead, send it as type: "text"
email	EML / MSG	uri or base64	—

A non-text item must carry a fetchable/decodable payload (uri or base64). A non-text item that only carries text returns 415 — the parser has nothing to parse.

Sending multimodal content

Multimodal input is a content array of ContentItem objects on a MessageItem. A bare string content is shorthand for a single text item; switch to the array form when you mix text with non-text assets. Field-level rules are in api.md → ContentItem; the essentials:

Field	Purpose
type	One of the modalities above
text	The literal text — only for type: "text"
uri	http(s):// (fetched server-side) or file:// (read from the server fs)
base64	Inline payload, plain base64 (no data: prefix)
ext	Extension hint ("pdf", "png", …); effectively required for base64
name	Display filename for logs

Carry the payload in exactly one of text / uri / base64.

Payload: uri vs base64

	uri (http(s)://)	base64
Where the bytes live	Fetched transiently at parse time	Held verbatim in the SQLite session buffer until flush
Wire size	URL only	~4/3× the raw size (base64 inflation)
Best for	Large assets, S3/OSS presigned URLs	Small assets, or when no reachable URL exists

Prefer uri for anything large. A multi-MB base64 blob becomes multi-MB of SQLite buffer text for the buffer's lifetime and slows request parsing. The bytes are never persisted past extraction either way — only the parsed text is.

Example: image by URL

TS=$(($(date +%s) * 1000))     # v1 contract: timestamp in ms
curl -X POST http://127.0.0.1:8000/api/v1/memory/add \
  -H 'Content-Type: application/json' \
  -d "{
    \"session_id\": \"mm-001\",
    \"messages\": [
      {
        \"sender_id\": \"alice\",
        \"role\": \"user\",
        \"timestamp\": $TS,
        \"content\": [
          { \"type\": \"image\", \"uri\": \"https://example.com/whiteboard.png\" }
        ]
      }
    ]
  }"

Example: mixed text + image in one turn

{
  "session_id": "mm-001",
  "messages": [
    {
      "sender_id": "alice",
      "role": "user",
      "timestamp": 1748390400000,
      "content": [
        { "type": "text",  "text": "Here's the whiteboard from today's planning session." },
        { "type": "image", "uri": "https://example.com/whiteboard.png", "name": "whiteboard.png" }
      ]
    }
  ]
}

Example: inline PDF via base64

{
  "session_id": "mm-001",
  "messages": [
    {
      "sender_id": "alice",
      "role": "user",
      "timestamp": 1748390400000,
      "content": [
        { "type": "text", "text": "Quarterly report attached." },
        { "type": "pdf",  "base64": "JVBERi0xLjQK...", "ext": "pdf", "name": "q3.pdf" }
      ]
    }
  ]
}

ext is effectively required for base64 payloads — it drives modality dispatch. Without it the server falls back to MIME inference and otherwise 415s.

Example: local file via file://

A file:// URI is read from the server's local filesystem (the path must be reachable by the server process), guardrailed by size and an optional allowlist:

{ "type": "pdf", "uri": "file:///srv/uploads/q3.pdf" }

Guardrails (a violation surfaces as 415):

• the resolved path (symlinks followed) must be an existing regular file;
• size ≤ EVEROS_MULTIMODAL__FILE_URI_MAX_BYTES (default 50 MiB);
• if EVEROS_MULTIMODAL__FILE_URI_ALLOW_DIRS is set, the path must lie within one of the listed roots (unset = any readable file, the local-first default — confine this when exposing the API beyond loopback).

Calling from Python (plain HTTP)

There is no EverOS Python client; call the HTTP API directly with any HTTP library:

import httpx

httpx.post(
    "http://127.0.0.1:8000/api/v1/memory/add",
    json={
        "session_id": "mm-001",
        "messages": [
            {
                "sender_id": "alice",
                "role": "user",
                "timestamp": 1748390400000,
                "content": [
                    {"type": "text", "text": "Here's the whiteboard from today's meeting."},
                    {"type": "image", "uri": "https://example.com/whiteboard.png"},
                ],
            }
        ],
    },
)

Configuration reference

All fields live under [multimodal] in everos.toml. Each can also be overridden via EVEROS_MULTIMODAL__<FIELD> env vars (useful for containers and CI).

Field	Default	Meaning
model	google/gemini-3-flash-preview	Parsing model; must accept image_url parts
base_url	https://openrouter.ai/api/v1	OpenAI-compatible base URL
api_key	— (required)	API key for the endpoint above
max_concurrency	4	Cap on parallel multimodal calls within one extraction
file_uri_max_bytes	52428800 (50 MiB)	Max size of a file:// asset
file_uri_allow_dirs	[] (any)	Allowlisted base dirs for file:// URIs

Errors and limits

Three failure classes behave differently:

Format errors — the uploaded file format is invalid or not recognized. These abort the batch with 415 (UNSUPPORTED_FORMAT):

Condition	HTTP	error.code
Non-text item carries only text (no uri / base64)	415	UNSUPPORTED_FORMAT
Extension / modality the parser has no handler for	415	UNSUPPORTED_FORMAT
base64 without a resolvable ext / MIME to dispatch on	415	UNSUPPORTED_FORMAT
file:// fails a guardrail (missing / non-regular / too large / outside allowlist)	415	UNSUPPORTED_FORMAT

Capability errors — the server is missing a required dependency. These abort the batch with 503 (CAPABILITY_UNAVAILABLE). Unlike transient errors, retrying will not help — admin action is required:

Condition	HTTP	error.code
everos[multimodal] extra not installed	503	CAPABILITY_UNAVAILABLE
Office document but no LibreOffice (soffice) on host	503	CAPABILITY_UNAVAILABLE

Transient LLM errors — the multimodal LLM call failed. These degrade gracefully — the request still returns 200, the affected item is marked parse_status="failed" and contributes no text, and the rest of the batch extracts normally:

Condition	HTTP	Result
Multimodal LLM call fails (timeout / rate-limit / model rejects)	200	That item is skipped; the rest of the batch still extracts

All error responses use the standard error envelope — see api.md → Errors.

Searching multimodal memory

Nothing special is required. Because parsed text is folded into the same episodes and memory cells as text turns, every retrieval method works across multimodal-derived memory unchanged:

curl -X POST http://127.0.0.1:8000/api/v1/memory/search \
  -H 'Content-Type: application/json' \
  -d '{
    "user_id": "alice",
    "query": "whiteboard from the planning session",
    "method": "hybrid"
  }'

keyword, vector, hybrid (default), and agentic all apply — see api.md → SearchMethod.