elizaos--eliza/plugins/plugin-native-talkmode/android/LIVE_PIPELINE.md

# Android live voice pipeline: `audioFrame` → speaker-attributed turns

This document maps the **end-to-end on-device** path that turns the Android
`audioFrame` PCM stream (see [`AUDIO_FRAMES.md`](./AUDIO_FRAMES.md)) into live,
VAD-segmented, speaker-attributed voice turns.

There are two transports, and they target different builds. For the **normal
Android APK** (`ai.elizaos.app`) the canonical path is the **in-process
JNI/bionic host** — the four fused voice classifiers run inside the Capacitor
app process via `libelizainference.so`, with no separate agent process and no
HTTP hop. The legacy **musl bun-agent transport** remains only for the
privileged AOSP build (where the embedded bun agent runs platform-signed); it is
not the path the normal APK uses.

## The canonical pipeline (normal APK: in-process JNI/bionic host)

```
 Android native AudioRecord (plugin-native-talkmode, Kotlin)
   │  emits `audioFrame` Capacitor event: base64 LE-s16 16 kHz mono PCM,
   │  20 ms/frame, { sampleRate, channels, samples, rms, timestamp, frameIndex }
   ▼
 Capacitor WebView (JS renderer)
   │  TalkMode.addListener("audioFrame", …)  →  JniVoicePipeline
   │    (packages/ui/src/voice/jni-voice-pipeline.ts)
   │  batches ~1 s of frames → ElizaVoice.pipelineProcess({ handle, pcm16 })
   ▼
 ElizaVoice JNI host  (ai.elizaos.app process, BIONIC — same process, NO agent)
   │  packages/app-core/platforms/android/app/src/main/elizavoice-jni/
   │  libelizavoicejni.so → libelizainference.so (fused, ABI v7, all four runtimes)
   │  1. native VAD hot-loop + turn segmentation (ported VadDetector state machine):
   │       streams the PCM through eliza_inference_vad_process, applies the
   │       onset/offset/pause/end-hangover thresholds, buffers the turn PCM
   │       (+ pre-roll) between speech-start and speech-end — ZERO per-512-window
   │       JS↔native bridge calls.
   │  2. on speech-end: eliza_inference_speaker_embed (256-d WeSpeaker embedding)
   │       + eliza_inference_diariz_segment (293 pyannote frame labels), natively.
   │  3. returns a turn-level result (base64 embedding + int8 labels) to JS.
   ▼
 JS (JniVoicePipeline)
   │  decodes the embedding + labels, runs the injected speaker resolver
   │  (embedding → enrolled entity) and buildVoiceTurnSignal (the ambient gate),
   │  and surfaces a JniAttributedTurn.
   ▼
 voiceTurnSignal  → the `core.voice_turn_signal` server gate decides
                     whether the agent speaks (owner / bystander / wake word).
```

The native ops the JNI host wraps (all `eliza_inference_*`, fused into the one
`libelizainference.so`):

| Stage | JNI surface | native runtime |
|---|---|---|
| Silero VAD (turn segmentation) | `vad open/processBatch/reset/close` + the streaming `pipeline*` | `eliza_inference_vad_*` |
| openWakeWord ("hey eliza") | `wakeword open/scoreBatch/reset/close` | `eliza_inference_wakeword_*` |
| WeSpeaker encoder (speaker embedding) | `speaker open/embed/close` | `eliza_inference_speaker_*` |
| pyannote diarizer (segment by speaker) | `diariz open/segment/close` | `eliza_inference_diariz_*` |

Each resolves its GGUF from the on-device bundle
(`<files>/eliza-1/bundle/{vad,wake,speaker,diariz}/…`). The split is: the VAD
hot-loop + turn segmentation + speaker/diariz forward passes run natively; the
speaker-match-against-enrolled-profiles + the ambient gate stay in JS (per-turn,
infrequent).

## What is verified on-device (Pixel 9a, `ai.elizaos.app`)

The whole pipeline runs in the bionic app process — proven via CDP + logcat
(the in-process verification channel; every line is emitted by the
`ai.elizaos.app` pid, never the agent):

- **ABI + capability**: `eliza_inference_abi_version() = 7`, and all four
  classifiers report supported in-process: `vad=1 wakeword=1 speaker=1 diariz=1`.
- **Full pipeline on real speech** (`freeman.wav` + 2 s trailing silence, fed in
  1 s batches via `pipelineSelfTest`):
  ```
  pipelineSelfTest: feeding 308224 samples (19.26s), chunk=16000
  TURN jni_0: samples=285184 (17.82s) | speaker: embDim=256 norm=1.0000 |
              diariz: frames=293 distinctClasses=1
  ```
  The 17.82 s turn (< 19.26 s fed) is a **real VAD speech-end** firing at the
  silence boundary — turn segmentation, the 256-d L2-normalized speaker
  embedding, and the 293-frame diarizer all ran in-process.
- **Wake-word** (`wakewordSelfTest`, "hey eliza" clip vs silence):
  `posMax=1.0000 negMax=0.0000`.

The JS-side consumer (`JniVoicePipeline`) has a host unit test
(`packages/ui/src/voice/jni-voice-pipeline.test.ts`, 5 cases): lifecycle,
runtime-unavailable refusal, frame batching (one bridge call per ~1 s), turn
embedding/label decode, and the confident-bystander suppression gate.

The platform-agnostic agent-side consumer
(`plugins/plugin-local-inference/src/services/voice/audio-frame-consumer.ts`)
also has its host unit test + the real-model smoke
(`packages/app-core/scripts/voice-attribution-smoke.ts`), shared by both
transports.

## On-device verification surface

`window.__jniVoice` (installed on Android by `main.tsx` →
`installJniVoiceHarness`) drives the in-process pipeline from CDP:

```
window.__jniVoice.start()   → open native pipeline + start mic + pump
window.__jniVoice.status()  → { running, framesSent, turnsObserved, abi, recentTurns }
window.__jniVoice.stop()    → stop capture, flush the open turn, free handles
```

## Legacy: musl bun-agent transport (AOSP build only)

Before the JNI host, the four classifiers ran in the **embedded bun agent**
(a separate musl process), reached from the WebView by POSTing batched frames to
`POST /api/voice/audio-frames` (`AudioFramePump` →
`LiveDiarizationSession` → `AudioFrameConsumer`). That path dlopened standalone
musl libs (`libsilero_vad.so`, `libvoice_classifier.so`, cross-compiled with
`zig cc --target=aarch64-linux-musl`) via bun:ffi, pointed at by the
`ELIZA_SILERO_VAD_LIB` / `ELIZA_VOICE_CLASSIFIER_LIB` env vars
`ElizaAgentService` exports when those `.so` are present in `nativeLibraryDir`.

This transport is **superseded for the normal APK** by the JNI/bionic host
above. The standalone musl voice `.so` and their zig cross-build script
(`packages/native/scripts/build-voice-libs-android-musl.mjs`) have been removed.
The musl `libllama.so` (the text agent — a separate concern) is untouched, and
the bun-agent voice path remains available on the privileged AOSP build (out of
scope here — to be unified with the JNI host later).