elizaos--eliza/plugins/plugin-local-inference/docs/multi-backend-ffi-seam.md

Multi-backend libelizainference FFI seam — design doc

Realizes milestone M3 of the Gemma 4 cutover (gemma4-cutover-plan.md) and the M4/M5 backends that land on top of it. The interface this doc describes is already written: src/llm-backend.h. The FFI ABI it plugs into is include/eliza-inference-ffi.h (the streaming-LLM surface, ABI v8+). The governing contract is native/AGENTS.md §11 — "one managed library, one pipe, no sidecar/subprocess/TCP."

1. The problem — one FFI pipe, N in-process runtimes

The streaming-LLM FFI (eliza_inference_llm_stream_open → _prefill → _next* → _close, plus _cancel / _reset / _reset_keep) is one pipe. Every consumer — the Node FFI loader (ffi-bindings.ts), the Capacitor JNI bridge on Android, the Swift bridge on iOS — drives exactly these symbols and knows nothing about which inference runtime answers behind them.

Today that runtime is always the in-tree llama.cpp path: EliLlmStream owns a private llama_context + sampler chain (and an optional eliza_mtp::Engine speculative driver), and the FFI functions call into it directly. That is the right default everywhere — CPU / CUDA / Vulkan-Mali-Adreno / Metal — but it is not the fastest backend on every device:

Backend	Where it wins	§11 status
llama.cpp	everywhere; the reference	owned pipe (today)
LiteRT-LM	Android NPU (Tensor / Qualcomm QNN / MediaTek NeuroPilot); opt. desktop/iOS GPU	owned pipe (M4)
CoreML / MLX	Apple Silicon (mac first, iOS later)	owned pipe (M5)
AICore / Apple Foundation	opportunistic fast-path	external — not owned, not a backend here

Per the §11 reinterpretation, LiteRT-LM and MLX are embeddable in-process C++ libraries — they link into libelizainference and answer the same FFI streaming symbols. They are never a child process or a TCP server. AICore (Android Binder system service) and Apple Foundation Models stay opportunistic adapters on the TS side (backends/apple-foundation.ts and friends); they are out-of-process and are not registered as backends in this seam.

The problem M3 solves: let one FFI pipe be served by more than one in-process runtime, selected per-_open from the platform + bundle + build flags + an env override — without touching the llama.cpp code path, which stays the default and the fallback.

2. Architecture

2.1 The two abstractions

llm-backend.h defines two pure-virtual interfaces. Both mirror the FFI 1:1 so the dispatch is a delegate with zero argument translation.

LlmBackendSession — one per active generation. Its methods are the FFI entry points minus the opaque-handle argument:

int prefill(const int32_t* token_ids, size_t num_tokens, char** out_error);
int next(int32_t* tokens_out, size_t tokens_cap, size_t* num_tokens_out,
         char* text_out, size_t text_cap,
         int32_t* drafter_drafted_out, int32_t* drafter_accepted_out,
         char** out_error);                       // 0=more, 1=final, <0=ELIZA_* err
int cancel();                                     // ELIZA_OK; safe from another thread
int reset();                                      // clear KV + sampler/counters
int reset_keep(int32_t n_keep);                   // prefix-preserving; full-reset fallback => 0
int save_slot(const char*, char**);               // optional; default ELIZA_ERR_INVALID_ARG
int restore_slot(const char*, char**);            // optional; default ELIZA_ERR_INVALID_ARG

Status conventions are identical to the FFI: >= 0 on success, the negative ELIZA_* constants on failure (ELIZA_ERR_CANCELLED on cancel), *out_error heap-allocated for the caller to free. next()'s drafter_*_out carry per-step speculative stats and are 0 when the backend has no drafter. A backend that cannot do prefix reuse MUST make reset_keep() fall back to a full reset and return 0 — never an error.

LlmBackendFactory — one static-lifetime singleton per linked-in runtime:

const char* name() const;                         // "llama.cpp" | "litert-lm" | "mlx-coreml"
bool available() const;                            // compiled in AND deps present on THIS host
bool can_serve(const char* bundle_dir) const;      // backend artifact exists in the bundle
int  preference_rank() const;                      // higher wins; llama.cpp == 0 (implicit fallback)
LlmBackendSession* open(EliInferenceContext*, const eliza_llm_stream_config_t*, char** out_error);

available() and can_serve() are both cheap — no model load. available() returns false when the build gate is OFF or the runtime dependency (NPU delegate, Metal device, linked lib) is absent on this host. can_serve() is a directory probe for the backend-specific artifact under the bundle's text/ dir.

open() receives the EliInferenceContext* but treats it as opaque — the struct is only forward-declared in llm-backend.h. A backend reads the bundle root through the one seam accessor llm_backend_context_bundle_dir(ctx) (declared in llm-backend.h, defined in the FFI translation unit where the struct is complete) and re-resolves its artifact from there. There is no bundle-dir caching between can_serve() and open(): both probe from the same root, so the selection path carries no mutable cross-call state.

2.2 Bundle artifact discovery (can_serve)

The text model lives under <bundle_dir>/text/ (§2 bundle layout). Each backend recognizes its own artifact there:

Backend	can_serve(bundle_dir) true when
llama.cpp	(implicit) text/*.gguf present — it is the fallback, never selected by rank
LiteRT-LM	text/*.litertlm present (Google ships pre-converted Gemma 4 .litertlm bundles)
MLX/CoreML	text/*.mlpackage or text/*.mlmodelc or an MLX weights dir present

The manifest is the source of truth for bundle contents; can_serve is a fast on-disk existence check that lets the selector skip a backend whose artifact was not staged for this tier. A bundle that ships only text/*.gguf is served by llama.cpp on every platform with no behavior change.

2.3 The selector

llm_backend_select(bundle_dir, cfg, out_error) runs at _open time. Resolution order:

1. ELIZA_LLM_BACKEND env — a HARD select (case-insensitive name() match).
   • "llama.cpp" / "llamacpp" → returns nullptr, *out_error == nullptr: forces the in-tree path.
   • Any other name that is not registered+available(), or that cannot can_serve(bundle_dir), is a hard error: returns nullptr and sets *out_error, so the FFI aborts rather than silently falling back to llama.cpp. An explicit backend request that can't be honored must fail loudly (§9 no-defensive-fallback).
2. No env override — auto-select. Among registered backends that are available() and can_serve(bundle_dir), pick the highest preference_rank(). If none qualifies, return nullptr (*out_error == nullptr) → use the in-tree llama.cpp path.

The return contract is the load-bearing detail:

Return	*out_error	Meaning
non-null factory	—	open a session on this backend
nullptr	nullptr	use the in-tree llama.cpp path (not an error)
nullptr	non-null	hard failure — propagate, abort the _open

llm_backend_register_builtins() registers every compiled-in factory once (idempotent, gated by the -DELIZA_ENABLE_* options); the FFI translation unit calls it at first _open.

2.4 Non-invasive FFI dispatch

The llama.cpp path is untouched and stays the default. Each eliza_inference_llm_stream_* function gets one if (stream-has-backend) branch inserted above the existing branches — never replacing them.

EliLlmStream today holds { lctx, sampler, mtp, ... } and the streaming functions already branch if (stream->mtp) { ... } else { plain llama.cpp }. M3 adds one field and one branch above that:

struct EliLlmStream {
    LlmBackendSession* backend = nullptr;  // non-null => alternate runtime owns this session
    llama_context*     lctx    = nullptr;  // in-tree llama.cpp path (unchanged)
    /* sampler, eliza_mtp::Engine* mtp, mtp_first_token, mtp_step_buf, ... unchanged */
};

_open:

char* sel_err = nullptr;
LlmBackendFactory* f = llm_backend_select(ctx->bundle_dir, cfg, &sel_err);
if (sel_err) { /* hard error: free stream, set *out_error, return NULL */ }
if (f) {
    stream->backend = f->open(ctx, cfg, out_error);
    if (!stream->backend) { /* free stream, return NULL */ }
    return stream;                         // alternate backend session — done
}
/* f == nullptr && no error: fall through to the EXISTING llama.cpp/MTP open path */

Every other streaming function gets the same top branch and otherwise runs unchanged:

int eliza_inference_llm_stream_next(EliLlmStream* s, /* ... */) {
    if (s->backend) return s->backend->next(/* args forwarded verbatim */);
    if (s->mtp)     { /* existing MTP path — unchanged */ }
    /* existing plain llama.cpp path — unchanged */
}

_close deletes stream->backend (the FFI owns the session per the interface contract) and then tears down lctx/sampler/mtp as today. _cancel / _reset / _reset_keep / _save_slot / _restore_slot follow the identical "backend-branch-on-top" shape.

Consequence: a build with no alternate backend compiled in has stream->backend == nullptr on every path, llm_backend_select() returns nullptr/no-error, and the library behaves exactly as before — byte-for-byte the current llama.cpp + MTP code path. The seam is inert until a backend gate is turned on AND the bundle ships that backend's artifact AND it's selected.

3. Build flags — each backend is a compiled-out stub when its gate is off

Two CMake options, both default OFF:

Flag	Backend	Links	Registers
-DELIZA_ENABLE_LITERT	LiteRT-LM	LiteRT-LM in-process C++ lib + NPU delegates	litert-lm factory
-DELIZA_ENABLE_MLX	MLX/CoreML	MLX / CoreML in-process libs (Apple only)	mlx-coreml factory

When a gate is off:

• The backend's factory translation unit is either not compiled, or compiled as a stub whose available() returns false and whose open() returns nullptr + an error. llm_backend_register_builtins() only registers the factories whose gate is on.
• No alternate-runtime headers/libs enter the link line. The default libelizainference (no gates) links only the in-tree llama.cpp tree — unchanged from today.
• llm_backend_select() finds no available() candidate → returns nullptr → llama.cpp path. A stale ELIZA_LLM_BACKEND=litert-lm on a build without the gate is a clean hard error (env names an unavailable backend), not a silent llama.cpp fallback.

This keeps M3 (the seam + selector + stubs) landable and verifiable on Linux x64 before the device-only backend bodies (M4/M5) exist. The stubs compile out cleanly; the seam compiles in.

ELIZA_ENABLE_LITERT / ELIZA_ENABLE_MLX join the existing per-feature gates (ELIZA_ENABLE_KOKORO, ELIZA_ENABLE_VISION) and follow the same "absent symbol / *_supported() == 0" degradation convention the ABI already uses.

4. Verifiability matrix

Honest scoping per the cutover plan's "verifiable in-session vs needs hardware" split. Nothing is claimed verified without the evidence.

Verifiable here (Linux x64 + CUDA)

Item	How
The seam compiles	Build libelizainference with the M3 patch; the if (stream->backend) branches + EliLlmStream::backend field type-check and link.
llama.cpp still works through the seam	With no gate on, llm_backend_select() returns nullptr/no-error on every _open; run the existing CPU/CUDA text-gen + MTP + llama-bench / e2e_loop_bench and diff against the pre-M3 baseline. Identical output = the seam is inert when it should be.
Selector is unit-testable	Register fake factories (controllable available() / can_serve() / preference_rank()) and assert the resolution table: env hard-select (hit / unavailable-error / llama.cpp→nullptr), rank ordering, and the nullptr/*out_error tri-state. No model load, no device.
Scaffolds compile-out cleanly	Build with -DELIZA_ENABLE_LITERT=ON / -DELIZA_ENABLE_MLX=ON on Linux x64 where the runtime deps are absent: the factories compile, register_builtins() registers them, available() returns false (no NPU / no Metal), and selection still falls through to llama.cpp. Confirms the gate wiring without the device.

Device-gated (scoped + scaffolded, NOT claimed here)

Item	Needs	Owner milestone
Gemma 4 → .litertlm conversion + bundle staging	LiteRT-LM toolchain; Pixel	M4
LiteRT NPU delegate ladder (Tensor → QNN → NeuroPilot → GPU/CPU)	Pixel / QNN / NeuroPilot device	M4
LiteRT on-device tok/s, first-token, peak RSS	Pixel	M4 / M7
Gemma 4 → .mlpackage / .mlmodelc / MLX-weights conversion	Mac; CoreML/MLX toolchain	M5
MLX/CoreML on-device tok/s, RSS, first-token	Mac (mac first, iOS later)	M5 / M7
next() drafter stats parity vs llama.cpp MTP	per-backend device run	M6 / M7

The M3 PR ships: the interface (already written), the selector + its unit tests, the non-invasive FFI dispatch branches, the two build gates, and compile-out stubs for both backends. The backend bodies (delegate ladders, weight loaders, real prefill/next) are M4/M5 and land behind their device gates.

5. Mapping to cutover-plan acceptance criteria

From gemma4-cutover-plan.md "Acceptance criteria":

Criterion	How this seam satisfies it
"Multi-backend selection behind one FFI; LiteRT/MLX/CoreML in-process; AICore/Foundation opportunistic."	LlmBackendFactory + llm_backend_select() are the multi-backend selection, behind the single eliza_inference_llm_stream_* pipe. LiteRT/MLX register as in-process factories; AICore/Foundation are explicitly not registered here (TS adapters).
"Gemma 4 runs through libelizainference (text+vision+audio+MTP) on every buildable backend."	The seam is the text-path mechanism by which a non-llama.cpp backend serves Gemma; llama.cpp remains the buildable-everywhere path (rank 0 fallback). Vision/audio/MTP stay on their existing fused surfaces.
"tok/s + RSS + first-token + MTP-acceptance captured per platform; faster-or-justified vs the retired Qwen line."	next()'s drafter_drafted_out / drafter_accepted_out give per-step MTP-acceptance for any backend that drafts; per-platform tok/s/RSS/first-token are the M4/M5/M7 device measurements gated above.
"Verified on web + desktop app + on-device (as hardware allows; else honestly scoped)."	§4: the seam + selector + compile-out are verified on Linux x64+CUDA in-session; the LiteRT/MLX bodies are scoped to M4/M5/M7 device runs and not claimed before evidence.
"eliza-1 branding preserved (users never see Qwen/Gemma)."	Unchanged — the backend name() values (llama.cpp, litert-lm, mlx-coreml) and ELIZA_LLM_BACKEND are developer-facing only; no user-visible string changes.

M3 milestone exit

M3 (this seam) is done when, on Linux x64 + CUDA:

• libelizainference builds with the dispatch branches; default (no-gate) build is byte-for-byte the pre-M3 llama.cpp path on e2e_loop_bench.
• The selector unit tests pass the full resolution table (env hard-select, rank ordering, the nullptr/*out_error tri-state).
• -DELIZA_ENABLE_LITERT=ON and -DELIZA_ENABLE_MLX=ON builds compile, register, report available() == false on this host, and fall through to llama.cpp.
• No consumer (ffi-bindings.ts, JNI, Swift) changes — the FFI ABI is unchanged (the seam is entirely below the ABI surface).

M4 (LiteRT body) and M5 (MLX/CoreML body) build on this seam behind their device gates and are out of M3's scope.