elizaos--eliza/plugins/plugin-local-inference/docs/gemma4-cutover-plan.md

Gemma 4 cutover + multi-backend libelizainference — living plan

Status doc for the Gemma 4 cutover campaign. Supersedes the research issue #8794 (Gemma is real + shipped; the open questions there are answered below). Continuously updated as milestones land.

Goal

Cut eliza-1 over from the Qwen3.5/3.6 backbone to Gemma 4 entirely (remove Qwen), and make every platform run the fastest in-process backend behind one FFI pipe (libelizainference):

Backend	Where	§11 status
llama.cpp	everywhere (CPU / CUDA / Vulkan-Mali-Adreno / Metal)	owned pipe (today)
LiteRT-LM (compiled-in)	Android NPU (Tensor / Qualcomm QNN / MediaTek NeuroPilot), opt. desktop/iOS GPU	owned pipe ✅
CoreML / MLX (compiled-in)	Apple (mac first, iOS later)	owned pipe ✅
AICore / Apple Foundation	opportunistic fast-path	external (not owned)

§11 ("single on-device runtime") is reinterpreted from "llama.cpp-only" to "one managed library, one pipe, no sidecar/subprocess/TCP." LiteRT-LM and MLX are embeddable in-process C++ libraries → linked into libelizainference, exposing the same FFI streaming symbols FfiStreamingRunner already drives. AICore is an out-of-process Android system service (Binder IPC) → it stays an opportunistic adapter like apple-foundation.ts, never the owned backend.

Gemma 4 — validated facts (2026)

Shipped 2026-04-02 (MTP drafters 04-16, 12B Unified 06-03).

• Sizes: E2B (~2.3B eff / 5.1B w/ embeddings), E4B (~4.5B / 8B), 12B Unified (11.95B dense, encoder-free, 256K ctx, 262,144 vocab), 31B dense, 26B-A4B (MoE, 4B active). E2B ≈ 1.3 GB Q4, 2–3 GB RAM → mobile tier.
• Architecture: dense; alternating local sliding-window (512/1024) + global full-attention; dual RoPE; Per-Layer Embeddings (PLE); shared KV cache (last N layers reuse earlier KV). No SSM/Gated-DeltaNet. Dual head dims (global_head_dim + SWA head_dim).
• MTP: official separate drafter models (speculative decode ~3×, no quality loss). Our catalog's drafterFile? already supports separate-drafter MTP.
• Modalities: image + audio (USM conformer) + video.
• NPU fit: dense graphs are NPU-delegate-friendly; LiteRT-LM ships pre-converted Gemma 4 .litertlm bundles. Qwen3.5-hybrid is hostile to NPU delegates — the strongest technical reason to cut over.
• Our llama.cpp fork already speaks Gemma 4: LLM_ARCH_GEMMA4 + gemma4.cpp / gemma4a.cpp (audio) / gemma4v.cpp (vision) + converter (text/MoE/vision/ audio). It already implements SWA (LLAMA_SWA_TYPE_STANDARD), shared KV (n_kv_shared_layers → n_layer_kv_from_start reuse cb), PLE (per_layer_tok_embd), and interleaved-SWA KV (llama_kv_cache_iswa).

RAM / performance: llama.cpp is not optimal for Gemma out of the box

Gemma 4 is notorious for RAM blow-up in stock llama.cpp. Root causes + our levers (several are config, not new kernels — "Gemma already does a lot of this"):

1. KV-cache layout (the ~40% fix) — landed upstream ~2026-04-05 (align KV memory layout to Gemma's MHA/grouping). Our merge-base is 2026-05-14, so we likely already have it; the M1 sync to upstream master confirms + grabs the rest.
2. ctx-checkpoints accumulation (#21690) — Gemma-only: server KV checkpoints grow unbounded (64 GB filled in 4–5 × 16K-prompt). Fix: bound ctx-checkpoints (1/0) + -np 1 in our fused defaults. Pure config.
3. Windowed SWA KV (swa_full=false) — llama_kv_cache_iswa already sizes SWA layers to n_swa (512/1024) + n_ubatch when swa_full=false. Most layers are SWA, so this is the dominant KV saving. Default swa_full=false (it's already plumbed through capacitor-llama/types.ts + the iOS shim).
4. PLE memory — per_layer_tok_embd is {n_embd_per_layer·n_layer, n_vocab} (~2.8B params for E2B). With mmap ON the OS pages it from disk (≈ LiteRT's "mmap embeddings, keep on disk until needed"). Never force --no-mmap for Gemma; on GPU backends, pin PLE to CPU/mmap rather than copying it to VRAM. (Watch Vulkan #18317: Vulkan can't run mmap=0.)
5. Prefix-KV reuse caveat (#21468) — Gemma's shared-KV layers log "cache reuse not supported" even with -fa --swa-full. So prefix reuse is partial on shared-KV layers — better than Qwen3.5 (which can't at all) but not free. Our stream_reset_keep work must account for this.

Leverage Google's C++ work, don't reinvent: LiteRT-LM already does PLE-mmap + windowed-SWA + NPU optimally → use it as the in-process backend on capable hardware. For llama.cpp, absorb upstream Gemma memory PRs and set Gemma-aware defaults rather than porting our Qwen kernels blindly.

Where our kernels still add value (M6), re-scoped:

• TurboQuant low-precision weight quant — orthogonal to KV; applies to the dense FFN/attention weights. Keep.
• QJL/PolarQuant KV quant — now applied on top of the windowed SWA KV + the global-attention KV (where 256K context still hurts). Re-parameterize from uniform head_dim=128 to Gemma's dual dims (global ~256 + SWA). Validate it still wins after SWA+shared-KV already shrank the cache.
• Re-run the §8 8/8 kernel verify matrix per buildable backend for the new geometry.

Frozen contracts re-opened (owner-approved 2026-06-22)

1. Training base lock (model_registry.py, training/AGENTS.md) — Qwen → Gemma 4.
2. tokenizerFamily "qwen35"→"gemma4"; vocab 248,320 → 262,144 (memory_calc.py).
3. KV geometry + kernels — head_dim 128 uniform → Gemma dual dims.
4. Same-file MTP NextN head → Gemma separate drafter GGUF.
5. EOT <|im_end|> → <end_of_turn> (3 scorer files).
6. Abliteration Gated-DeltaNet surgery → dense surgery.
7. native/AGENTS.md §11 reinterpreted (single lib + single pipe).

eliza-1 v1 is base-not-fine-tuned (releaseState=base-v1) → v1 cutover is swap base + re-optimize, not a retraining run.

Owner directives (2026-06-22)

• Hard cutover, remove Qwen entirely. Not shipped, no downloaders → no staging.
• Local stack is ONLY eliza-1 (max-optimized Gemma). No generic single-GGUF engine path, no arbitrary-model selection — remove that machinery (closes #8808; M9). Cloud handles anything not local eliza-1.
• HF downloads proxy through Eliza Cloud — the cloud holds the HF token and streams files; no HF keys on local/desktop/mobile, no local token UI (re-scopes #8807; M10).
• HF elizaos/eliza-1 hard cutover to Gemma bundles; purge legacy tiers (0_6b/0_8b/1_7b) (M8).
• Max-optimize per platform: fastest in-process backend behind one FFI (llama.cpp default; LiteRT NPU on Android; CoreML/MLX on Apple); TurboQuant weight-quant + MTP + Gemma-native SWA/shared-KV/PLE + stock KV; QJL/Polar deprioritized (Gemma KV already minimal).
• Related issues: #8808 closed (superseded); #8807 + #8809 kept, addressed in M10 (install cloud-proxy + integrity; memory LRU/dynamic-fit/bench).

Milestones (PR per milestone → develop)

• M1 — llama.cpp upstream sync + PR absorption. Merge ggml-org/master (602 commits ahead of our base; we are 867 ahead with kernels/voice) into the fork; review + absorb all relevant Gemma/MTP/LiteRT/CoreML/AICore/KV/quant PRs (incl. open #21587 Gemma4 BPE SIGSEGV, #24590 Gemma4Assistant memory-fit, #21690 ctx-checkpoints, #21468 cache-reuse). Build + verify Gemma 4 E2B/E4B text+vision+audio+MTP on CPU/CUDA.
• M2 — Code cutover Qwen→Gemma (registry, catalog/types, memory_calc, EOT, abliterate, AGENTS; remove Qwen; tier map E2B/E4B/12B/31B(/26B-A4B); Gemma-aware runtime defaults: swa_full=false, bounded ctx-checkpoints, mmap-on/PLE-on-CPU).
• M3 — Multi-backend FFI seam (backend abstraction + backend-selector). ✅ Done + verified on Linux. src/llm-backend.h (the LlmBackendSession / LlmBackendFactory interfaces + the llm_backend_context_bundle_dir accessor), src/llm-backend-selector.cpp (registry + env/rank selection, inert-by-default), and the non-invasive eliza_inference_llm_stream_* dispatch (one backend branch inserted above each existing llama.cpp/MTP branch). Compiles + links into the default fused libelizainference.so with the FFI pipe intact and the seam inert (no alternate backend registered → byte-for-byte the prior llama.cpp path). Design: docs/multi-backend-ffi-seam.md.
• M4 — LiteRT-LM in-process backend (Android NPU delegate ladder). 🧩 Scaffolded, gated -DELIZA_ENABLE_LITERT (OFF default). src/backends/litert-backend.{h,cpp}: full LlmBackendSession/factory against the researched LiteRT-LM C++ API (Engine/Session, NPU→GPU→CPU ladder, text/*.litertlm probe) behind the gate; a no-SDK stub when OFF. CMake adds the source + SDK include/link knobs (ELIZA_LITERT_SDK_DIR/ELIZA_LITERT_LIBS) when enabled. Every hardware assumption tagged DEVICE-VERIFY; needs a Pixel/NPU device + the LiteRT-LM SDK to build + validate.
• M5 — CoreML/MLX in-process backend (mac first, iOS later). 🧩 Scaffolded, gated -DELIZA_ENABLE_MLX + __APPLE__ (OFF default; FATALs on a non-Apple host). src/backends/mlx-coreml-backend.{h,mm}: MLX-primary (mlx-c decode graph) + CoreML-alternate (stateful MLState KV) sessions behind the gate; a no-SDK stub when OFF. CMake adds the .mm + MLX/CoreML/Metal link knobs when enabled. Tagged DEVICE-VERIFY; needs Apple Silicon + the MLX/CoreML toolchain to build + validate.
• M6 — Kernel re-optimization for Gemma geometry (TurboQuant weight-quant + QJL/Polar KV-quant on windowed/global KV) across CPU/CUDA/Vulkan-Mali/Metal/NPU
  • low-precision + long context; re-verify 8/8.
• M7 — Verification everywhere (web / desktop app / Pixel / bench harnesses + PR_EVIDENCE).

What's verifiable in-session vs needs hardware

• Here (Linux x64 + CUDA): llama.cpp merge + CPU/CUDA builds; Gemma 4 text/vision/audio/MTP gen; code cutover; FFI seam; bench (llama-bench, e2e_loop_bench) on CPU/CUDA; web + desktop-app smoke.
• Needs Mac: Metal kernels, CoreML/MLX backends, iOS.
• Needs Pixel/Android device: Vulkan-Mali kernels, LiteRT NPU (Tensor/QNN), on-device tok/s + RSS. (Prior on-device work used adb/CDP on a Pixel 9a.)

Hardware-gated items are scoped + scaffolded here and marked for device verification; nothing is claimed "verified" without the evidence.

Acceptance criteria

• Fork synced to upstream master; all relevant Gemma/MTP/LiteRT PRs absorbed or rejected-with-reason.
• Gemma 4 runs through libelizainference (text+vision+audio+MTP) on every buildable backend.
• Qwen fully removed from the shipped eliza-1 line.
• Multi-backend selection behind one FFI; LiteRT/MLX/CoreML in-process; AICore/Foundation opportunistic.
• Gemma-aware RAM defaults set; kernels re-optimized + 8/8 verified per buildable backend; low-precision quant validated.
• tok/s + RSS + first-token + MTP-acceptance captured per platform; faster-or-justified vs retired Qwen line.
• Verified on web + desktop app + on-device (as hardware allows; else honestly scoped).
• eliza-1 branding preserved (users never see "Qwen"/"Gemma").