elizaos--eliza/plugins/plugin-local-inference/native/PRECACHE.md

Eliza-1 precache strategy (text-side KV)

Scope: the local-inference text path (packages/app-core/src/services/ local-inference/). What gets KV-prefilled before the user's tokens land, when, how it's keyed, and how it's invalidated. The drafter co-residency and first-audio filler are cross-referenced (the audio side lives in voice/phrase-cache.ts + the turn controller); only the text-KV pieces are covered here.

The goal is to make "user stops speaking → agent's first token" as close to "decode the user's tokens + sample" as possible — the system prompt, tool/action schema block, and stable provider blocks should already be in the slot's KV by the time STT finishes.

---

What is precached

1. The Stage-1 response-handler stable prefix (renderMessageHandlerStablePrefix)

The bulk of every turn's prompt is the part that does not depend on the user's message: the canonical system prompt, the HANDLE_RESPONSE tool schema block, the available-contexts catalog, and the stable provider blocks (character, capabilities, etc.). @elizaos/core exposes renderMessageHandlerStablePrefix(runtime, roomId) which renders exactly the stable segments of the Stage-1 model input — byte-identical to the messages[0].content (the "system" message) that renderMessageHandlerModelInput would produce for the first turn of a fresh conversation in that room, with the unstable tail (recent dialogue, the current user message) dropped.

This string is what the local engine pre-warms with: a max_tokens: 1, temperature: 0, cache_prompt: true chat completion against the deterministic slot the conversation pins to (MtpLlamaServer. prewarmConversation → localInferenceEngine.prewarmConversation → prewarmResponseHandler in ensure-local-inference-handler.ts). When the real Stage-1 request lands it appends the user turn to the same prefix, so llama-server only forward-passes the new tokens; --slot-prompt-similarity 0.7 covers any minor trailing-instruction drift.

2. The system prefix on slot 0 at model load (conv:__system_prefix__)

At runtime boot, once a local model is resident, prewarmSystemPrefix fires the same stable prefix against the synthetic conversation id __system_prefix__. deriveSlotId("conv:__system_prefix__", parallel) is stable, so this always warms the same slot before any user message — and because llama-server's radix KV is shared across slots, every per-room conversation that opens later inherits the common prefix tokens.

3. Drafter co-residency (cross-ref packages/inference/AGENTS.md §3)

The MTP drafter is mmapped alongside the target by llama-server itself (-md <drafter> --spec-type mtp …) — there is no separate "load drafter" step, so it is "precached" by construction the moment the server starts. Owned by ffi-streaming-backend.ts startup + the MtpDrafterHandle in voice/shared-resources.ts.

4. First-audio filler (cross-ref the voice phrase cache)

Pre-generating a short acknowledgement audio chunk ("one sec", "got it", …) on VAD fire is the audio-side counterpart to the text-KV pre-warm. Owned by voice/phrase-cache.ts (+ cache/voice-preset-default.bin) and the turn controller; not part of this module.

---

When precache fires

Trigger	What's warmed	Call site
Model load / runtime boot	system prefix on conv:__system_prefix__	ensureLocalInferenceHandler → prewarmSystemPrefix (fire-and-forget)
Voice session open / speech-start	Stage-1 stable prefix on conv:<roomId>	the voice turn controller → prewarmResponseHandler(runtime, roomId)
Keep-alive sweep (~4 min, < short TTL)	re-issues the last pre-warm prefix for slots untouched ≥ 80% of the short TTL	MtpLlamaServer keep-alive timer (startKeepAliveTimer)
First real Stage-1 request	(no extra work) appends the user turn to the warm prefix	runV5MessageRuntimeStage1 → useModel(RESPONSE_HANDLER) with conversationId: roomId

prewarmConversation is best-effort by definition: a failure (server not running, render throws, HTTP error) just means the real request cold-prefills — the pre-pre-warm behaviour. It must never paper over a broken cache key with retries; it only saves the prefill cost when the key is sound.

It is a no-op on the in-process node-llama-cpp backend — that backend's session pool already pins by cache key, so there is nothing to pre-warm.

---

How it's keyed

KV reuse cache-key precedence (cache-bridge.ts resolveLocalCacheKey):

1. conv:<conversationId> — strongest. The Stage-1 call site passes conversationId: String(roomId) through buildProviderCachePlan (@elizaos/core/src/runtime/provider-cache-plan.ts → providerOptions.eliza.conversationId), so every turn of a room — and the pre-warm for it — lands on the same KV slot. This is the dominant reuse signal for chat and voice.
2. seg:<hashStablePrefix(promptSegments)> — hash of the longest run of stable: true segments, derived from providerOptions.eliza.promptSegments.
3. pfx:<prefixHash> — providerOptions.eliza.prefixHash (already stable-only via cachePrefixSegments upstream).
4. v5:<prefixHash> — promptCacheKey back-compat fallback.

deriveSlotId(cacheKey, parallel) maps the key into [0, parallel). The conversationRegistry additionally pins each open conversation to the lowest-loaded slot via pickLowestLoadedSlot so two distinct conversations whose keys hash-collide don't thrash each other.

The synthetic conv:__system_prefix__ key is intentionally not a real roomId — it carries no per-room state, only the model-and-config-stable prefix, so a fixed id is correct.

Verified: buildProviderCachePlan does emit conversationId / promptSegments / prefixHash on the voice + response-handler path — runV5MessageRuntimeStage1 (packages/core/src/services/message.ts) passes all three, with conversationId = String(message.roomId). Voice messages travel the VOICE_DM channel with a populated roomId, so the local cache key is conv:<roomId> per conversation with no degradation. No runtime fix was required for key stability — the pre-warm path just has to reuse the same roomId, which prewarmResponseHandler does.

---

How it's invalidated

There is no explicit invalidation step — KV reuse is structural, not event-driven:

• Prompt drift within a turn. The unstable tail (recent dialogue, the current message, timestamps) is not part of the cache key, so it never shifts the slot; llama-server forward-passes whatever doesn't match the cached prefix. --slot-prompt-similarity 0.7 lets a slightly different trailing instruction block still reuse the common prefix.
• System-prompt / tool-schema change. A real change to the character system prompt, the registered actions/contexts, or the messageHandlerTemplate changes the rendered stable prefix → the next pre-warm and the next real request simply forward-pass the new tokens onto whatever the slot held. The optimizer artifacts (OptimizedPromptService) deliberately keep the rendered prompt text stable to avoid churning this prefix.
• Model swap. buildModelHash keys the on-disk slot directory by sha256(target + drafter + cache-types + ctx + parallel)[:16], so switching the active model uses a different directory and the old slot files are orphaned (and aged out by the eviction sweep, below).
• Idle conversation. conversationRegistry.evictIdle() (run on the eviction timer) drops handles untouched past their TTL (default 60 min), persisting each one's KV to <slotDir>/<convId>.long.bin and closing it — so an idle handle is flushed-and-dropped rather than lingering and inflating the parallel high-water mark. A re-open lazy-restores from the .bin.
• On-disk slot/conversation .bin eviction. evictExpired deletes slot/conversation .bin files older than their per-file TTL horizon. The TTL class is encoded in the filename as <base>.<ttl>.bin (slotCacheFileName / parseSlotCacheTtlClass); files written without an encoded class keep the long horizon (the prior global behaviour). Conversation KV persisted on close/checkpoint/shutdown is written .long.bin. The sweep runs every ELIZA_LOCAL_EVICTION_INTERVAL_MS (default 5 min) alongside the evictIdle pass.

---

Tunables

Env var	Default	Effect
ELIZA_LOCAL_EVICTION_INTERVAL_MS	300000	eviction + evictIdle sweep interval (clamped ≥ 60 s)
ELIZA_LOCAL_KEEPALIVE_INTERVAL_MS	240000	keep-alive re-warm sweep interval (clamped ≥ 30 s)
ELIZA_MTP_HOST / port	loopback	the llama-server the pre-warm requests hit

The short / long / extended cache-TTL windows (DEFAULT_CACHE_TTLS — 5 min / 60 min / 24 h) mirror the cloud ephemeral-cache semantics and are what ttlMsForKey resolves a per-file class to. The keep-alive sweep re-warms a slot once it has been untouched for 80% of the short TTL (KEEPALIVE_STALE_FRACTION).