elizaos--eliza/plugins/plugin-local-inference/native/AGENTS.md

AGENTS.md — Eliza-1 inference (kernels + runtime)

This file is the canonical contract for any agent working on the Eliza-1 on-device inference stack. In this repository the native stack lives under plugins/plugin-local-inference/native/; older docs may still call that area packages/inference/. The contract also applies to the local-inference runtime surfaces in packages/ui/src/services/local-inference/, the mtp build hook at packages/app-core/scripts/build-llama-cpp-mtp.mjs, and any mobile/desktop bridges that consume the same artifacts.

The training-side companion is at packages/training/AGENTS.md. Read both before changing anything that crosses the boundary (artifacts, manifest, kernel ABI, GGML pin).

Fork source. The patched llama.cpp ships in-tree as a git submodule at plugins/plugin-local-inference/native/llama.cpp — elizaOS/llama.cpp tracking the v1.2.0-eliza line (resolve the current gitlink via git -C plugins/plugin-local-inference/native/llama.cpp describe --always; the v1.0.0-eliza / 08032d57 pin documented previously is stale — do not copy that pin into new tooling or scripts). git submodule update --init --recursive (which bun install runs) is the canonical checkout. This is the canonical fork: TurboQuant (turbo3/turbo4/turbo3_tcq) + QJL (block_qjl1_256, GGML_OP_ATTN_SCORE_QJL, GGML_OP_FUSED_ATTN_QJL_TBQ) + PolarQuant (block_q4_polar, Q4_POLAR=47) + the eliza Metal/Vulkan/CUDA kernels + MTP spec-decode (--spec-type mtp, the mtp-draft GGUF arch)

• the post-refactor llama-server (server-task.cpp / server-common.cpp with grammar_lazy / json_schema / response_format / prefill_assistant), on upstream b9213. Both build paths consume it: build-llama-cpp-mtp.mjs (desktop/server/Windows/iOS) and aosp/compile-libllama.mjs (Android) default to the submodule checkout. ELIZA_MTP_LLAMA_CPP_REMOTE / _REF (or --cache-dir / --src-dir) still force a standalone clone for fork bisects. (The v1.2.0-eliza line tracks the prior v1.0.0-eliza tree forward, re-tagged on the elizaOS rename chain. A full rebase onto a recent upstream llama.cpp remains a deferred follow-up — not a blocker for structured output (the b9213 base already has grammar_lazy / json_schema / response_format / prefill_assistant); the conflict-prone files are the quant-slot enums in ggml-common.h / ggml.h and the Q1_0 block layout, which upstream redefined incompatibly with the fork's. Before attempting that rebase, write or update a concrete porting plan in the repo docs and verify every referenced path from this directory.)

---

1. What we are building

Eliza-1 is a single product line of on-device fused models. From the user's perspective there is exactly one default option per device tier; they pick a size, they get one bundle, it does text + voice + vision. Underneath it is a manifest-described bundle of GGUF/metallib/SPIR-V files plus kernel capability metadata. There is no "pick the text model" or "pick the TTS" — that is a runtime concern, not a user concern.

Backbones (do not change without explicit human approval):

• Text/vision: Gemma 4 family (E2B/E4B/12B/31B) for the 2B/4B/9B/27B release tiers. Gemma is a dense SWA + shared-KV + PLE + MQA architecture with dual head dims (512 global / 256 swa); the legacy QJL/PolarQuant/TurboQuant KV kernels are head_dim=128 and are NOT required for Gemma (its KV is already minimal) — TurboQuant weight-quant still applies. Every active Eliza-1 tier is vision-capable when its tier-matched vision/mmproj-<tier>.gguf is present and validated. We do not name these as "Gemma" in any user-facing string. Internally, manifests record the upstream lineage and license; the UI shows "Eliza-1 ".

• Voice (TTS): Tier-aware. The active backend per tier is declared in ELIZA_1_VOICE_BACKENDS (packages/shared/src/local-inference/catalog.ts) and is read by the runtime selector at engine arm time. Policy:

  Tier	Backend(s)	Default
  2b / 4b	OmniVoice and Kokoro	OmniVoice
  9b	OmniVoice and Kokoro	OmniVoice
  27b / 27b-256k	OmniVoice only	OmniVoice

  • Kokoro = Kokoro-82M GGUF staged from elizaos/eliza-1 (original hexgrad/Kokoro-82M, Q4_K_M). Fixed voice packs, no per-user cloning. Bundled at tts/kokoro/{kokoro-82m-v1_0-Q4_K_M.gguf,tokenizer.json,voices/<voice>.bin} in each shipping tier. Backend implementation: plugins/plugin-local-inference/src/services/voice/kokoro/.
  • OmniVoice = Qwen3-TTS lineage. Upstream at https://github.com/ServeurpersoCom/omnivoice.cpp, mirrored at https://github.com/elizaOS/omnivoice.cpp. ~200ms TTFB on the fused build. Per-user voice cloning, voice design via attribute keywords, optional omnivoice-singing variant ([singing]/[happy]/[sad]/[whisper]/[angry]/[nervous]/[calm]/ [excited] + non-verbals [laughter]/[sigh]). Bundled at tts/omnivoice-base-<quant>.gguf + tts/omnivoice-tokenizer-<quant>.gguf in tiers that ship it.

  Per Wave-6 user direction (2026-05-10), omnivoice-singing CAN ship as part of default bundles for non-commercial use (CC-compatible terms). Commercial pivot requires re-training on commercially-licensed corpora to clear the CC-BY-NC-SA training-data lineage (GTSinger, RAVDESS, Expresso).

  Canonical voice engine: fused libelizainference. The strategic on-device voice engine is the fused-FFI libelizainference library built directly from the merged llama.cpp fork tree at plugins/plugin-local-inference/native/llama.cpp/tools/omnivoice/. This is what services/voice/ calls; this is what the manifest's voice and asr entries are activated through.

  W3-3 (OmniVoice → llama.cpp literal merge, v1.0.1-eliza, 2026-05-14): the OmniVoice sources, FFI bridge, and streaming optimizations now live INSIDE the fork at tools/omnivoice/. The pre-merge graft path (clone elizaOS/omnivoice.cpp at build time, copy sources into omnivoice/ at fork root, append ELIZA-OMNIVOICE-FUSION-GRAFT-V1 CMake block via packages/app-core/scripts/omnivoice-fuse/ — driven by ELIZA_FUSE_OMNIVOICE=ON) is deprecated and stays for ONE release as a runway. Setting OMNIVOICE_INSIDE_LLAMA_CPP=0 (build-script env) opts back into the legacy graft. The default is the merged path (OMNIVOICE_INSIDE_LLAMA_CPP=1). The build flag is now -DLLAMA_BUILD_OMNIVOICE=ON -DOMNIVOICE_SHARED=ON; ELIZA_FUSE_OMNIVOICE=ON is a back-compat alias that emits a deprecation warning and redirects.

  After the v1.0.2-eliza release, OMNIVOICE_INSIDE_LLAMA_CPP=0 is removed and packages/app-core/scripts/omnivoice-fuse/ is deleted.

  The standalone OmniVoice backend (separate ABI v2 ov_* symbols, separate libomnivoice.{so,dylib,dll}, separate build script native/build-omnivoice.mjs) is legacy; the standalone plugin that once wrapped it has been removed. Do not add a new standalone wrapper; new voice code goes through libelizainference.

  Kokoro and OmniVoice both satisfy the same TtsBackend + StreamingTtsBackend contract — the runtime selector (services/voice/kokoro/runtime-selection.ts) picks one at arm time based on the tier policy, the ELIZA_TTS_BACKEND env override (kokoro|omnivoice|auto), voice-cloning requirements, and measured RTF / TTFB.

• ASR: Local ASR is fused-FFI only and artifact-gated. A default-eligible Gemma 4 release must record real Gemma ASR lineage in the manifest and must not ship Qwen ASR provenance; the runtime and manifest validator reject strict/defaultEligible Qwen ASR stand-ins. Bundled files remain asr/eliza-1-asr.gguf + asr/eliza-1-asr-mmproj.gguf (mmproj sidecar is REQUIRED for the audio-in / text-out path). Activated through libelizainference's eliza_pick_asr_files(). The public Qwen3-ASR artifacts and the qwen3a mtmd backport are compatibility/scaffolding only until verified Gemma ASR artifacts are staged; do not expose them as default/recommended. OmniVoice has no ASR head — do not route ASR through it.

  whisper.cpp is not in the contract — it vendors its own ggml, violating the one-llama.cpp-build / one-GGML-pin contract in §4. All historical transcribeManager.ts / whisper-node references have been removed.

• VAD: Silero VAD (MIT, ~2 MB ONNX). Ships in every voice-enabled bundle. Drives barge-in cancellation; gates ASR to skip silent frames.

• Wake word: openWakeWord (Apache-2.0, ~3 MB). Opt-in, local-mode only. Hidden in cloud mode per three-mode hide-not-disable.

• Embedding: 2b (the entry tier) reuses the active text backbone with --pooling last — no duplicate weights in the mobile/default tier. Larger tiers may ship a dedicated embedding/ artifact (1024-dim Matryoshka, 32k ctx) when the manifest records a real source artifact and evidence. Do not fabricate embedding source repos, and do not silently fall back on larger tiers when the manifest says a dedicated region is required.

• Drafter: MTP ships on every tier, including the 2b entry tier. Speculative decoding is mandatory, not optional (see §3).

Three runtime modes — every code path must work in all three:

Mode	Local models?	Cloud models?	Remote control?
local	yes (default)	optional	exposes itself
cloud	hidden	yes (default)	hidden
remote	via target	no	yes

Settings rules (enforce in UI + API layer, not just docs):

• cloud mode hides every local-model UI surface, every ELIZA_LOCAL_* setting, and the local-inference settings panel entirely. The cloud setting page is the only model-related surface.
• local-only mode (a sub-state of local) hides every cloud setting and every cloud-routed provider. The user must not be able to accidentally route a request to cloud.
• remote mode connects to a local instance only. It must refuse to point at a cloud instance. Changing cloud settings in remote mode mutates the target's cloud settings (i.e. the local agent the remote is controlling), not the remote-control client.

These three modes are not feature flags or A/B variants. They are the top-level shape of the product. New code that adds a model surface MUST state explicitly which modes it lives in, and MUST be removed from modes where it does not belong.

---

2. Single fused bundle per device tier

Eliza-1 ships as one logical bundle per tier. The user sees one download. Internally a bundle is a manifest plus several files, all hosted under the elizalabs HuggingFace org under eliza-1.

Tier matrix (binding)

Tier	Tagline	Text	Voice	Vision	Context	MTP	Quant default
2b	small / low-RAM phones (entry)	2B	OmniVoice + Kokoro	mmproj	128k	yes	TurboQuant Q4 + stock Gemma KV
4b	flagship phones, small desktops	4B	OmniVoice + Kokoro	mmproj	128k	yes	TurboQuant Q4 + stock Gemma KV
9b	desktop / midrange GPU	9B	OmniVoice + Kokoro	mmproj	128k	yes	TurboQuant Q4 + stock Gemma KV
27b	flagship GPU	27B	OmniVoice	mmproj	128k	yes	TurboQuant Q4 + turbo3_tcq
27b-256k	long-context flagship	27B	OmniVoice	mmproj	256k	yes	+ turbo3_tcq (long ctx)

Quant default — Gemma 4 reality (binding, see §1/§3 + catalog.ts). The shipped tiers run Gemma 4 bases. Gemma's KV is already minimal (MQA + windowed-SWA + shared-KV, dual head dims 512 global / 256 swa), so the runtime ships stock KV (f16/q8_0) and the mandatory optimization set is TurboQuant weight-quant (turbo3/turbo4, + turbo3_tcq on the big/long-ctx tiers) + MTP. The legacy head_dim=128 QJL K-cache / PolarQuant (TBQ) V-cache fused-attn kernels are NOT used on Gemma — they were the Qwen3.5/3.6-era KV optimizations. The kernels still ship and pass metal_verify/vulkan_verify, but the decode graph never routes a Gemma KV through them (the dequant-to-F16 hop in src/llama-graph.cpp only fires for QJL1_256/TBQ3_TCQ/Q4_POLAR cache types, which Gemma never allocates). catalog.test.ts ("advertises only safe runtime optimizations for the shipped gemma4 tiers") pins this contract.

Context-length variants (32k / 64k / 128k / 256k) are not separate tiers — they are dimensions inside a tier. A tier's manifest lists which context lengths are available; the runtime picks the largest that fits the device's RAM budget at activation time.

Bundle layout (binding)

All tiers ship in a single HuggingFace mono-repo elizaos/eliza-1, with each tier living under bundles/<tier>/. The manifest is the source of truth; never derive contents from filenames.

elizaos/eliza-1/
  bundles/<tier>/
    eliza-1.manifest.json          # canonical schema, see §6
    text/
      eliza-1-<tier>-<ctx>.gguf    # text (+ inline vision where supported)
    tts/
      # Kokoro fallback shipped on 2b/4b/9b:
      kokoro/kokoro-82m-v1_0-Q4_K_M.gguf
      kokoro/tokenizer.json
      kokoro/voices/<voice>.bin
      # OmniVoice shipped on every active tier. Default install stages a
      # single quant (VOICE_QUANT_BY_TIER); --include-voice-ladder at stage
      # time emits the tier ladder so the downloader can pick the level
      # matching the host's RAM/SoC class at install time. See
      # `voiceQuantLadderForTier()` in
      # packages/shared/src/local-inference/catalog.ts and
      # docs/inference/voice-quant-matrix.md.
      omnivoice-base-<quant>.gguf
      omnivoice-tokenizer-<quant>.gguf
    asr/
      eliza-1-asr.gguf             # eligible local ASR text head
      eliza-1-asr-mmproj.gguf      # local ASR audio projector sidecar
    vad/
      silero-vad-v5.gguf           # native silero-vad-cpp, every tier
    vision/
      mmproj-<tier>.gguf           # 2b/4b/9b/27b/27b-256k
    mtp/
      drafter-<tier>.gguf
      target-meta.json             # acceptance windows, kernel caps
    cache/
      voice-preset-default.bin     # speaker embedding + phrase cache seed
    evals/
      text-eval.json
      voice-rtf.json
      asr-wer.json
      e2e-loop.json
    licenses/
      LICENSE.text
      LICENSE.voice
      LICENSE.asr
      LICENSE.vad
      LICENSE.mtp
      LICENSE.vision
      LICENSE.eliza-1
    checksums/SHA256SUMS
    evidence/release.json
    evidence/platform/<id>.json
    quantization/{turboquant,qjl,polarquant}.json
  README.md                        # mono-repo overview

The runtime default voice quant per tier (the level the runtime selects when no device-class override applies) is the value returned by voiceQuantForTier() in packages/shared/src/local-inference/catalog.ts: Q4_K_M for 2b/4b, Q8_0 for 9b/27b/27b-256k.

The publish ladder per tier (every level that gets staged when --include-voice-ladder is passed) is the value returned by voiceQuantLadderForTier():

• Mobile tiers (2b/4b) publish the narrow OmniVoice ladder Q3_K_M, Q4_K_M, Q5_K_M.
• Larger OmniVoice-shipping tiers (9b/27b/27b-256k) publish Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0.

The downloader picks the level matching the host's memory class at install time (MAX / GOOD / OKAY / POOR per memory-budget.ts). No silent fallback — §3 forbids "try the next smaller one" at runtime; if the resolved level isn't in the bundle, the install fails loudly with an actionable diagnostic.

OmniVoice quant rules (per R6 §5.6 + R8 §2)

• K-quant ladder Q3..Q8 — applies. omnivoice.cpp/tools/quantize.cpp already supports the full Q2_K..Q8_0 set; the curated publish subset is the ladder above.
• PolarQuant on the LM weight bank — applies (OmniVoice's LM head is Qwen3-shaped), but no recipe is wired yet. Landing this requires either grafting Q4_POLAR recognition into OmniVoice's loader via omnivoice-fuse/cmake-graft.mjs, or running PolarQuant before omnivoice.cpp/convert.py. Gated on a measured TTS-MOS comparison.
• V-cache PolarQuant — N/A. OmniVoice has no KV cache between MaskGIT steps; there's no V-cache to compress.
• QJL — conditional, deferred. Only matters for long-form multi-chunk synth where the cumulative cache becomes large. Off the Wave-2 critical path.
• TurboQuant V-cache — same applicability as QJL — N/A absent a KV cache.

A literal single .gguf containing all of text + voice + ASR + vision

• drafter is not the deliverable — that requires either a custom container format or major upstream work in llama.cpp's GGUF graph, and is explicitly out of scope until the bundle ships and is stable. The single user-visible download action IS the deliverable: one click, one progress bar, one bundle on disk.

If that constraint changes (i.e. someone wants a literal one-file artifact later), define an .eliza container format with a manifest + multiple GGUFs concatenated, and update §6 — do not silently change the GGUF schema.

---

3. Mandatory optimizations (never skip, error if missing)

Every Eliza-1 bundle MUST run through every applicable optimization. The runtime MUST refuse to load a bundle that is missing any required artifact for its tier. There is no "fast path that skips X" and no "fallback to unoptimized". A bundle that cannot satisfy the contract must be marked broken in eliza-1.manifest.json and not served from the recommended-models endpoint.

Gemma 4 exception. For Gemma 4 tiers the mandatory set is TurboQuant (weight-quant) + MTP + the Gemma-native SWA/shared-KV/PLE memory settings (swa_full=false, bounded ctx-checkpoints, mmap-on, Per-Layer-Embeddings pinned to CPU on GPU backends). Because Gemma's KV is already minimal (MQA + windowed-SWA + shared-KV), the head_dim=128 QJL/PolarQuant KV kernels are optional on Gemma rather than required — "must run every kernel" below applies to the legacy Qwen-shaped tiers, not Gemma's KV.

Required for ALL tiers

1. TurboQuant on the text model. Q3 for lite, Q3/Q4 for mobile, Q4 for desktop/pro/server. The KV cache MUST use TurboQuant Q3 or Q4 quantization. See vulkan/turbo3.comp, vulkan/turbo4.comp, vulkan/turbo3_tcq.comp, metal/turbo3.metal, metal/turbo4.metal, metal/turbo3_tcq.metal. Verification: verify/metal_verify and verify/vulkan_verify MUST report 8/8 PASS on the target backend for the bundle's dtype before publish.
2. QJL on the K-cache when context > 8k. See vulkan/qjl*.comp and metal/qjl.metal. The reference is packages/native/plugins/qjl-cpu.
3. PolarQuant on the V-cache when context > 8k. See vulkan/polar*.comp and metal/polar.metal. The reference is packages/native/plugins/polarquant-cpu.
4. MTP speculative decoding with the bundle's drafter. Always wired, always running in voice mode. The MTP drafter participates in voice generation — proposed text tokens that survive verification are immediately handed to the TTS pipeline; rejected tokens roll back the TTS chunker (see §4 for the streaming contract).
5. Fused kernels. TurboQuant + QJL + Polar must compile into the same shipped llama.cpp build via the patch hooks in packages/app-core/scripts/build-llama-cpp-mtp.mjs. The runtime MUST log the kernel set on startup; missing kernels = startup error.

Required for desktop/pro/server tiers

6. TCQ trellis-coded quantization for desktop/pro/server and any long-context text variant. turbo3_tcq.comp / turbo3_tcq.metal.
7. CPU-offloaded KV cache for context > 64k where device RAM is insufficient. The runtime MUST implement spill, not just refuse the request.

Failure handling

If a required kernel fails to load, fails verification, or is missing from the build:

• Build time: build-llama-cpp-mtp.mjs MUST exit non-zero, and the published artifact MUST NOT include a "kernels-missing" fallback build. There is no fallback build.
• Runtime: the engine MUST refuse to activate the bundle and surface a structured error to the UI. It MUST NOT silently fall back to unoptimized inference. It MUST NOT log-and-continue.

The Metal and Vulkan kernel patchers run unconditionally for matching build targets. Build outputs can record shipped shader symbols separately from runtime-ready graph dispatch, but only runtime-ready capabilities may satisfy this contract. Treat any builder/runtime that disables a required patch as broken.

---

4. Fused pipeline (mic → speech, end-to-end)

The streaming contract for voice mode. Every Eliza-1 runtime MUST implement this exact graph; integrations that need a subset (e.g. text-only) must reach the same nodes via the same scheduler, not via a parallel codepath.

mic / file → ASR → text tokens
                    ↓
                  scheduler ──→ MTP drafter (proposes N tokens)
                                       ↓
                                  target verifier (text model)
                                       ↓
                              accepted tokens → phrase chunker
                                       ↓                       ↘
                            speaker preset (cached)        rollback queue
                                       ↓                       ↙
                                  OmniVoice TTS  ←── on-reject: cancel chunk
                                       ↓
                                  PCM ring buffer → audio out

Hard requirements

• One process, one llama.cpp build, one GGML pin. Text and voice share the same llama.cpp library. omnivoice.cpp is fused into the same build at the source level (vendored, not a sidecar). If the GGML version pin used by omnivoice.cpp diverges from the text model, the build MUST fail.
• Shared KV cache scheduling, not shared KV memory. Text and voice have their own KV caches (different layer counts, different head configs, different quantizations). What they share is the scheduler, the mmap region for weights, the kernel set, and the memory-budget policy.
• Streaming handoff. When MTP + target produce an accepted text token, the phrase chunker MUST hand the chunk to TTS within the same scheduler tick — no buffering past phrase boundaries. Phrase boundaries are punctuation + a max-N-token cap (configurable per tier).
• Barge-in cancellation. When the mic detects new user speech, the TTS PCM ring buffer MUST drain immediately, the phrase chunker queue MUST flush, and any in-flight TTS forward pass MUST be cancelled at the next kernel boundary.
• Voice cancellation contract (W3-9). One VoiceCancellationToken per voice turn is the canonical handle. It lives in @elizaos/shared/voice/voice-cancellation-token, is owned by the VoiceCancellationCoordinator in plugins/plugin-local-inference/src/services/voice/cancellation-coordinator.ts, and is the sole legitimate way to fan an abort across these four layers:
  1. The runtime's TurnControllerRegistry.abortTurn(roomId, reason) so the planner-loop / action handlers / streaming useModel calls see the abort within one tick.
  2. The LM slot — via the registered slotAbort(slotId, reason) callback (today: HTTP-fetch close on the in-flight stream; on a fork that exposes a slot-cancel REST route, the REST call).
  3. The TTS pipeline — via the registered ttsStop(reason) callback (today: EngineVoiceBridge.triggerBargeIn → audio-sink drain + FFI/HTTP synthesis cancel).
  4. Any fetch / model / FFI consumer of token.signal (a standard AbortSignal). The token is idempotent (first reason wins) and fires every onAbort listener synchronously. Optimistic LM start is gated by OptimisticGenerationPolicy — default true on plugged-in / unknown, false on battery, with explicit user override. Full contract: plugins/plugin-local-inference/docs/voice-cancellation-contract.md.
• Speaker preset caching. The default voice ships as a precomputed speaker embedding in cache/voice-preset-default.bin. Loading a voice MUST NOT re-extract the embedding from raw audio on every startup. A precomputed phrase cache for common assistant utterances ("Sure.", "One moment.", "I can't help with that.") MUST be used as a first-byte-latency win.
• MTP↔TTS coupling. When MTP proposes text tokens that are later rejected by the target, the TTS chunker's rollback queue MUST drop the corresponding (not-yet-spoken) audio chunks. Audio that has already left the ring buffer is gone — design the chunker so this is rare (small chunk = low latency cost on rollback).

What we do NOT do

• We do not run text and voice in two processes communicating over IPC. That regresses memory and adds a 1–10ms scheduling tax per turn.
• We do not run a "TTS-only mode" that skips MTP. MTP is always on. If the user disables speculative decoding for debugging, that is a developer-only flag (ELIZA_MTP_DISABLE=1), it is not a user setting, and it MUST log a loud warning every turn.
• We do not split voice into "fast TTS" and "high-quality TTS" tiers. One voice model per tier, fused, optimized.

---

5. Three modes — code organization

Every entry point that touches a model MUST be classified into one or more of local, cloud, remote. The classification lives in code, not in docs.

• packages/app-core/src/services/local-inference/ is the local and local-only surface. It MUST have a hard import boundary against cloud-only modules.
• Cloud-routing code is in the cloud package and MUST NOT be imported by the local-inference service except through a typed mode-aware router that the runtime mode gates.
• remote mode is implemented as a thin client over the local instance's HTTP API. It does NOT have its own model surfaces — every setting it changes maps to a setting on the target.

Hide-not-disable rule: when a mode hides a setting, the UI must omit the surface entirely, the API must reject mutations to that setting with a 4xx, and the persisted setting must be inert (no background job acts on it). "Hidden" without "inert" is a leak.

---

6. Manifest schema (binding)

eliza-1.manifest.json is the source of truth for every Eliza-1 bundle. The runtime, the recommendation engine, the downloader, the mobile catalogs, and the build script all read this file. Do not let catalogs drift from it — generate them.

{
  "$schema": "https://elizaos.ai/schemas/eliza-1.manifest.v1.json",
  "id": "eliza-1-4b",
  "tier": "4b",
  "version": "1.0.0",
  "publishedAt": "2026-MM-DDTHH:MM:SSZ",
  "lineage": {
    "text": { "base": "gemma-4-E4B", "license": "..." },
    "voice": { "base": "omnivoice-base-Q4_K_M", "license": "..." },
    "drafter": { "base": "mtp-4b-drafter", "license": "..." }
  },
  "files": {
    "text":    [{ "path": "text/eliza-1-4b-128k.gguf", "ctx": 131072, "sha256": "..." }],
    "voice":   [{ "path": "tts/omnivoice-base-Q4_K_M.gguf",   "sha256": "..." }],
    "asr":     [{ "path": "asr/...",                          "sha256": "..." }],
    "vision":  [{ "path": "vision/mmproj-4b.gguf",    "sha256": "..." }],
    "mtp":  [{ "path": "mtp/drafter-4b.gguf",   "sha256": "..." }],
    "cache":   [{ "path": "cache/voice-preset-default.bin",   "sha256": "..." }]
  },
  "kernels": {
    "required": ["turboquant_q4", "qjl", "polarquant", "mtp", "turbo3_tcq"],
    "optional": [],
    "verifiedBackends": {
      "metal":  { "status": "pass", "atCommit": "...", "report": "..." },
      "vulkan": { "status": "pass", "atCommit": "...", "report": "..." },
      "cuda":   { "status": "pass", "atCommit": "...", "report": "..." },
      "cpu":    { "status": "pass", "atCommit": "...", "report": "..." }
    }
  },
  "evals": {
    "textEval":      { "score": 0.0, "passed": true },
    "voiceRtf":      { "rtf": 0.0,   "passed": true },
    "e2eLoopOk":     true,
    "thirtyTurnOk":  true
  },
  "ramBudgetMb": { "min": 7000, "recommended": 9500 },
  "defaultEligible": true
}

Rules:

• Every published bundle MUST have defaultEligible: true only if every required kernel is verified on every supported backend for that tier AND every eval has passed: true. The recommendation engine MUST refuse to surface a bundle with defaultEligible: false as a default.
• HF-search results from outside elizaos/eliza-1 MUST never set defaultEligible: true. They are user-installed customs only.
• The runtime MUST validate the manifest against kernels.required before activating the bundle. A capability mismatch is a hard error.

---

7. HuggingFace publishing & auto-download

Every Eliza-1 release lives at https://huggingface.co/elizaos. The device-side downloader MUST:

1. Read the manifest from the bundle's repo before downloading any weight file. Verify schema version, kernel caps against the device, RAM budget against device hardware. Refuse incompatible bundles with a structured error.
2. Download every file in manifest.files.*. Verify sha256 for each. Resume on partial download.
3. Check the device's available kernels (Metal/Vulkan/CUDA/CPU/MLX/NEON) against manifest.kernels.required. If any required kernel is unavailable on this device, the download MUST be aborted with a structured error before any weight bytes are fetched. There is no "download anyway, hope it works" path.
4. Materialize the bundle to the local cache, run a one-time verify-on-device pass (load → 1-token text generation → 1-phrase voice generation → barge-in cancel test), and only then mark the bundle ready in the local catalog.

Publishing flow (training side, see packages/training/AGENTS.md):

• Training produces text + drafter weights.
• Quantization recipes in packages/training/scripts/quantization/ apply TurboQuant + QJL + Polar.
• A publish script (one of the publish_* scripts in packages/training/scripts/) assembles the bundle, generates the manifest, runs verify/metal_verify + verify/vulkan_verify against the bundle's quantized artifacts, populates kernels.verifiedBackends, runs the eval suite, and pushes to HF.
• The publish script MUST refuse to upload if any required eval fails or any required kernel is unverified.

---

8. Verification gates (what "done" means)

A bundle is shippable when, on each supported backend:

• make -C plugins/plugin-local-inference/native/verify reference-test is clean.
• verify/metal_verify reports 8/8 PASS for turbo3, turbo4, turbo3_tcq, qjl, polar against the bundle's quantized weights (not just synthetic fixtures — fixtures regenerated from the actual shipped weights).
• verify/vulkan_verify reports 8/8 PASS for the same set.
• The CUDA path (where applicable) reproduces the same outputs to the same numerical tolerance.
• A 30-turn end-to-end voice loop runs without crash, without leak, without exceeding manifest.ramBudgetMb.recommended.
• First-token latency, first-audio latency, RTF, ASR WER, peak RSS, thermal/battery (mobile), and MTP acceptance rate are recorded in the manifest's evals block and meet tier-specific gates.

A code change that touches kernels, the build script, the mtp server, or the bundled-models catalog MUST run the relevant subset of these gates locally before merge. CI runs the full set per supported backend nightly.

---

9. Working style

• Scope discipline. The kernels in this directory are a contract. Do not invent new quantization formats, new fusion graphs, or new KV-cache layouts without a written design doc that explains why the existing five (turbo3, turbo4, turbo3_tcq, qjl, polar) are insufficient.
• No defensive code. A missing kernel, a missing manifest field, or a verification failure is a hard error. Do not add fallbacks. Do not log-and-continue. The whole point of the contract is that we ship one optimized path, not three with conditional branches.
• Mirror the references bit-for-bit. Metal/Vulkan kernels MUST produce numerically identical output (within published tolerance) to the C reference in plugins/plugin-local-inference/native/reference/ and to the upstream CUDA implementation in packages/native/plugins/{qjl-cpu,polarquant-cpu} and the elizaOS/llama.cpp fork. New kernels follow the same pattern: ship the C reference and a JSON fixture before shipping the Vulkan/Metal port.
• Hardware verification is non-optional. A "compiles cleanly" badge is not a "passes" badge. The README's verification matrix marks rows as NEEDS HARDWARE until metal_verify / vulkan_verify reports 8/8 on a real device. Do not flip a row to ✓ without that evidence.
• Stay aligned with the training side. Quantization recipes, weight layouts, and bundle structure cross the boundary between training and inference. Read packages/training/AGENTS.md before changing the manifest schema or any quantization op.
• Branding. User-facing strings and logs say Eliza-1. They do not say Qwen, Llama, OmniVoice, MTP, or TurboQuant. Internal logs, stack traces, and developer-mode UI surfaces may reference upstream names — anywhere a user can see, the name is Eliza-1.

---

10. Files to read before making changes

• plugins/plugin-local-inference/README.md — plugin-level runtime and native inference overview.
• packages/ui/src/services/local-inference/README.md — runtime contract for the engine, downloader, recommendation, and routing.
• packages/app-core/scripts/build-llama-cpp-mtp.mjs — the build hook. Every kernel patch lives here. It (and the AOSP cross-compile at packages/app-core/scripts/aosp/compile-libllama.mjs) default to building from the in-repo plugins/plugin-local-inference/native/llama.cpp submodule.
• packages/training/AGENTS.md — the training-side contract, including what the bundle/publish flow expects.
• the repo-root AGENTS.md — repo-wide cleanup mandate and conventions (port handling, scope discipline, elizaOS naming). The non-negotiable architecture rules apply here too: dependencies point inward, no polymorphism for runtime branching in code (kernels are a registry, not an if), no try/catch that swallows.

---

11. ONNX deprecation status (updated Kokoro GGUF, 2026-06-25)

Single on-device runtime: ONE managed library (libelizainference), ONE pipe, no sidecar/subprocess/TCP. No ONNX anywhere in the resolved code path: onnxruntime-node/onnxruntime-web is not a dependency and there are zero live imports in plugin-local-inference/src (enforced by src/services/voice/onnx-import-ban.test.ts). VAD, wake-word, turn-detector (preferred), Kokoro (fused GGUF), OmniVoice, and ASR all resolve through the fork FFI. The three voice classifier heads (Wav2Small emotion, WeSpeaker, pyannote-3) have already dropped ONNX too — but see the emotion-classifier gap called out below: the ONNX runtime was removed before the native GGUF read was wired, so the acoustic emotion read is DEAD at runtime (nothing loads a files.emotion GGUF; the fusion runs text/prosody-only) — a tracked K1 follow-up, not a silent fallback hidden behind a stub.

Single runtime policy: every local-inference model path must flow through ONE managed library (libelizainference) over ONE FFI pipe — no sidecar, subprocess, or TCP server. The elizaOS llama.cpp fork is the primary backend compiled into that library, but the contract is "one managed library, one pipe", not "llama.cpp only". In-process compiled-in backends that link into libelizainference behind the same FFI symbols are compliant — e.g. LiteRT-LM for the Android NPU path, or MLX / CoreML for Apple — because they are the owned backend, not a separate process. Out-of-process OS model services (AICore / Gemini Nano, Apple Foundation Models) remain opportunistic adapters only, never the owned backend, and never satisfy this contract on their own. ONNX (onnxruntime-node / onnxruntime-web) is deprecated and will be removed from the runtime path once all native ports land.

Completed (fork path active — no ONNX in resolved runtime)

Model	Path	Status
OmniVoice TTS	fork FFI libelizainference (tools/omnivoice/)	DONE (W3-3)
Silero VAD	standalone silero-vad-cpp FFI libsilero_vad + vad/silero-vad-v5.gguf	DONE (I1/K7 verified) — vad.ts imports zero onnxruntime-node
hey-eliza wakeword	fork FFI eliza_inference_wakeword_*	DONE (I1/K7 verified) — wake-word.ts imports zero onnxruntime-node
ASR (eligible local ASR)	fork FFI eliza_pick_asr_files()	DONE (runtime) / GATED (Gemma artifacts)
MTP speculative decoding	fork llama-server --spec-type mtp	DONE
Text EOT (Eliza1EotClassifier)	fork node-llama-cpp P(`<	im_end
Fused EOT scorer	fork FFI eliza_inference_eot_* + CompositeEotClassifier	DONE (J1.d) — preferred runtime path; staged GGUF assets remain bundle-only compatibility
Kokoro TTS	fused FFI eliza_inference_kokoro_* + tts/kokoro/kokoro-82m-v1_0-Q4_K_M.gguf	DONE (#9588) — GGUF-only runtime discovery
Wav2Small emotion	ONNX removed; scalar C forward in voice-classifier-cpp/src/voice_emotion.c	ONNX-FREE — but native read NOT wired: acoustic emotion path is DEAD at runtime (K1, see gap below)
WeSpeaker R34-LM	ONNX removed; scalar C forward in voice-classifier-cpp/src/voice_speaker.c	ONNX-FREE (native FFI/GGUF path)
pyannote-3 diarizer	ONNX removed; scalar C forward in voice-classifier-cpp/src/voice_diarizer.c	ONNX-FREE (native FFI/GGUF path)

Compute-gated (native read not yet wired)

Model	Gate	Owner	Est.
Wav2Small emotion (acoustic read)	ONNX gone; the native GGUF forward is NOT bound into libelizainference, and no runtime code loads a files.emotion GGUF, so the acoustic read is dead — the fusion in emotion-attribution.ts runs text/prosody-only. Tracked, not silently swallowed.	K1	bind voice_emotion.c GGUF forward + parity + pipeline promotion
LiveKit EOT (ONNX last-resort)	Historically ONNX was the last-resort in the engine.ts chain (Eliza1Eot → GgmlTD → OnnxTD → Heuristic). With onnxruntime-* no longer a dependency, the OnnxTD leg is unreachable; the chain is Eliza1Eot → GgmlTD → Heuristic.	K7/J1.d	confirm OnnxTD leg is removed from the chain

Rule: onnxruntime-node/onnxruntime-web are no longer dependencies of plugin-local-inference and there must be no live import of either (src/services/voice/onnx-import-ban.test.ts enforces this). Do NOT reintroduce an ONNX runtime import — the compliant path for any remaining head is the native FFI/GGUF forward. For a per-model native promotion, update the manifest runtime to the GGUF runtime, rename the promoted artifact to the canonical manifest path, and run the relevant gate from native/verify/PLATFORM_MATRIX.md.

GAP (emotion acoustic read — flagged 2026-07-03, C15): the ONNX Wav2Small runtime was deleted before the native GGUF read was wired. In production nothing loads a files.emotion GGUF and nothing constructs a VoiceEmotionClassifierOutput, so attributeVoiceEmotion() runs text/prosody-only and the acoustic-fusion branch is dead code. This is flagged as a tracked follow-up (not silently swallowed): wiring the voice_emotion.c GGUF forward through the memory arbiter — plus a runtime activation gate for a shipped-but-unrunnable files.emotion artifact — is the K1 work (out of headless scope: needs the native binding + a parity gate). Until then, no production bundle ships a files.emotion artifact.

HF deprecation runway: ONNX model files remain on HF alongside GGUFs for one release after each native port lands. Do not delete ONNX from HF until the GGUF path has been in production for one release cycle.