elizaos--eliza/plugins/plugin-local-inference/docs/per-platform-backend-strategy.md

Per-platform on-device inference backend strategy — CoreML / LiteRT-LM / Metal

Status: decision record + current-state evidence (2026-06-24). Answers the recurring question "should iOS/Mac LLM inference move to CoreML or LiteRT-LM, and does llama.cpp give us per-platform backends for max battery/GPU efficiency?"

Short answer: the per-platform backend strategy already exists and is correct. One elizaOS/llama.cpp fork compiles into libelizainference with the right GGML backend per platform. CoreML and LiteRT-LM are not the levers for Apple LLM decode — Metal is, and it is already the shipped, optimized, verified path.

---

1. Per-platform backend table (what is actually built/used)

Platform	GGML backend	Source of truth
macOS (Apple Silicon)	Metal (TurboQuant + eliza kernels embedded)	packages/app-core/scripts/build-llama-cpp-mtp.mjs (-DGGML_METAL=ON, embed metallib)
iOS (arm64)	Metal (static archives)	same build script, ios-arm64-metal / …-metal-fused targets
Android arm64	Vulkan (GPU) + CPU fallback	packages/app-core/scripts/aosp/compile-libllama.mjs
Android x86_64	CPU only	same (Vulkan wired for arm64 only)
Linux / Windows desktop	CUDA (NVIDIA) / CPU	desktop dylib build + cuda_verify

All from one fork → one managed library (libelizainference) → one FFI pipe (native/AGENTS.md §11). This is "per-platform backends for max battery/GPU efficiency." There is no separate per-platform runtime to add.

2. CoreML — wrong tool for LLM decode; correct where it is already used

• LLM autoregressive decode: CoreML/ANE is not viable. CoreML compiles a static graph with fixed tensor shapes; LLM decode grows a KV cache token by token with per-position masks. ANE targets fixed-graph CV/encoder workloads, not streaming text. There is no CoreML LLM-decode backend in llama.cpp, and we have none. The correct Apple LLM backend is Metal/MPS, which handles dynamic KV shapes natively.
• Where CoreML IS used (correctly): fixed-graph models — Kokoro TTS on iOS (kokoro-coreml/*.mlmodelc) and image generation (src/services/imagegen/ CoreML backend). These are encoder/diffusion graphs, the right shape for CoreML.
• Apple Foundation Models (src/backends/apple-foundation.ts): an opportunistic iOS-26 text adapter, never the owned backend (native/AGENTS.md §11). Out-of-process OS model services cannot satisfy the single-runtime contract.

2a. Every voice/vision model is already on its optimal backend — MEASURED, no ANE win to capture

A natural follow-up: the always-on gate models (Silero VAD, openWakeWord) run continuously, so wouldn't the ANE save battery vs the GPU? Measured on this Apple Silicon Mac (ANE present, CoreML native) — the answer is no, on two counts:

1. They don't run on the GPU — both ship on native CPU (pure scalar C, "no ggml link"): silero_vad_runtime.c / wakeword_runtime.c, whose *_active_backend() returns "native-cpu".
2. The ANE is slower for a model this tiny. Ran Silero VAD v5 ONNX (2.3 MB LSTM) 200×32 ms windows, CoreML EP (ANE) vs CPU EP: CPU 0.227 ms/window vs ANE 0.798 ms/window — the ANE is 3.5× slower (identical outputs). ANE dispatch overhead dominates a few-layer LSTM; the ANE wins on large fixed-graph models (Kokoro), not tiny per-frame gates. Evidence: native/verify/evidence/platform/coreml-ane-vs-cpu-voice-gate-2026-06-24.md.

So the architecture already routes every model to its optimal backend by compute profile, and it's verified:

Model	Cadence	Backend	Right call
LLM (Gemma) decode	per-token, dynamic KV	Metal	ANE can't do dynamic decode
Vision mmproj / local ASR	bursty	Metal	GPU-sized bursty work
Kokoro TTS	per-utterance	CoreML (kokoro_5s.mlmodelc)	larger fixed-graph → ANE helps
Silero VAD	always-on	native CPU	tiny LSTM → CPU fastest+lowest-power (measured; ANE 3.5× slower)
openWakeWord	always-on	native CPU	same

Bottom line for "optimize voice for CoreML on iOS": there is no un-captured CoreML/ANE win — Kokoro already uses CoreML where it pays off, and the tiny always-on gates are correctly on the CPU (the ANE would regress them, measured). The system is already optimal. LiteRT-LM (the Android-NPU analogue) would hit the same tiny-model dispatch-overhead wall for the gate models.

3. LiteRT-LM — not used; permitted only as a future in-process Android-NPU backend

• Not present in the resolved code path. No LiteRT/TFLite/MediaPipe LLM runtime is wired.
• The architecture permits LiteRT-LM only as an in-process backend that links into libelizainference behind the same FFI symbols, for the Android NPU path (native/AGENTS.md §11). It is additive to llama.cpp, not a replacement, and is unimplemented.
• It does nothing for Apple. Adopting it would not touch the iOS/Mac story. It is a future Android-NPU optimization, gated on real product demand.

4. The shipped model is Gemma 4 — its optimization set is applied + verified

The current Eliza-1 tiers ship Gemma 4 bases (E2B/E4B/12B/31B → 2b/4b/9b/27b), not the retired Qwen3.5/3.6 line (packages/shared/src/local-inference/catalog.ts). Gemma's KV is already minimal (MQA + windowed-SWA + shared-KV, dual head dims 512/256), so:

• Mandatory set (applied): TurboQuant weight-quant (turbo3/turbo4, +turbo3_tcq on big/long-ctx tiers) + MTP separate-drafter speculative decode
  • Gemma-native memory settings + stock minimal KV (f16/q8_0).
• MTP wiring (applied on the Apple/desktop path): catalog.ts runtimeForTier → active-model.ts (draftMin/draftMax/speculativeSamples) → desktop-fused-ffi-backend-runtime.ts → the FFI's separate-drafter MTP engine (tools/omnivoice/src/eliza-inference-ffi.cpp).
• NOT applicable on Gemma: the legacy head_dim=128 QJL K-cache / PolarQuant(TBQ) V-cache fused-attn kernels. They still ship and pass verification, but the decode graph never routes a Gemma KV through them — the dequant-to-F16 hop in src/llama-graph.cpp:1990 only fires for QJL1_256/TBQ3_TCQ/Q4_POLAR cache types, which Gemma never allocates. catalog.test.ts ("advertises only safe runtime optimizations for the shipped gemma4 tiers") pins this contract.

Implication for issue #8848

#8848's headline item — "fused QJL/TBQ attention is implemented but never routed, leaving a win on the table" — is a legacy-Qwen concern, filed from a Pixel audit of the Qwen-shaped eliza-1-0_8b. It is not on the shipped Gemma critical path: routing it would not speed up the Gemma model (Gemma has no QJL/TBQ cache to route). The live Gemma audit (TurboQuant + MTP + stock KV) is green and tested. The Mali/Vulkan items in #8848 (#4 generic-FA race, #5 prefill matmul) remain real for Android GPU but are independent of the fused-attn routing.

5. Host-side verification evidence (Apple M-series, this machine, 2026-06-24)

Run from plugins/plugin-local-inference/native/verify/ (Metal framework runtime shader compilation — no full Xcode required):

make reference-test        → self-test: turbo3/turbo4/turbo3_tcq/qjl/polar/fused all finite, parity OK
make metal-verify          → turbo3 8/8 · turbo4 8/8 · turbo3_tcq 8/8 · qjl 8/8 · polar PASS (tol 1e-3)
make metal-verify-fused    → fused_attn_qjl_polar 1920/1920 PASS (max_diff 4.8e-7)
                             fused_attn_qjl_polar causal-prefix 1536/1536 PASS (max_diff 9.5e-7)
                             polar_preht use_qjl=0/1 8/8 PASS

Cross-check: verify/PLATFORM_MATRIX.md records the 2026-06-23 Gemma-4 Metal cutover — §8 kernel gate 40/40 PASS incl. fused-attention, and real google/gemma-4-E2B (head_dim 512, MQA, SWA) generation + llama-bench pp512 636 / tg128 23 t/s (FA=1) on the Metal FA path.

Conclusion: the Apple/Metal Gemma path is optimized (TurboQuant + MTP + minimal KV), and every shipped kernel is bit-exact on Apple Silicon. No CoreML/LiteRT pivot is warranted or beneficial for iOS/Mac LLM decode.

6. Genuinely-remaining gated items (not code-fixable from a Mac host)

Item	Issue	Why gated
Gemma MTP drafter GGUF conversion + on-Metal --spec-type draft-mtp gate — DONE 2026-06-24: drafter converted (drafter-2b.gguf), validated on M-series Metal, and the fast-tier draft window fixed (draftMax 4→1 = 1.37–1.66× decode win on the 2B; the prior "regression" was the mistuned window, not the head). A from-scratch H200 head is now only a possible large/slow-tier optimization, not a fast-tier unlock.	#8848 / #9172	n/a — shipped; see docs/gemma4-mtp-drafter-conversion.md (2026-06-24 correction)
Rebuild + republish the prebuilt Android Vulkan fused-lib to eliza-archive — OBSOLETE: builds in CI, no archive. The Android Vulkan fused libelizainference.so is built from source in CI by .github/workflows/build-llama-ffi-android.yml (android-arm64-vulkan-fused via compile-libllama.mjs, zig 0.13 pin from #9598) and consumed directly as a workflow_call artifact — there is no eliza-archive prebuilt dependency and nothing to republish. The Mali mitigation is in source; the Linux/CUDA + Mali-G715 kernel gates are verified on real hardware (#9584/#9715).	#9508	n/a — CI-built
Real-audio GPU CI lane (DER/WER/echo/owner/impostor numbers)	#9454	needs a gpu-cuda-12.6 self-hosted runner + ELEVENLABS_API_KEY
PCM-level AEC3 sample-level echo cancellation (turn-level self-voice gate already shipped)	#9455	net-new DSP feature (adaptive filter + double-talk detect + ERLE corpus)