elizaos--eliza/plugins/plugin-native-talkmode/android/AUDIO_FRAMES.md

Android raw-PCM frame capture (audioFrame)

Adds an opt-in native AudioRecord path to TalkModePlugin.kt that streams 16 kHz mono 16-bit PCM frames to JS as a new audioFrame Capacitor event, so a JS/bun consumer can run diarization / frame-VAD / wake-word on Android. The default SpeechRecognizer (SODA) STT path is untouched and remains the default.

Why a new path was needed

Android STT (android.speech.SpeechRecognizer) yields transcript strings only — onBufferReceived is a documented no-op on SODA, so there is no raw PCM, and diarization/VAD/wake-word can't run. The new path captures PCM directly with AudioRecord.

The mic-contention constraint and the chosen design

Android lets only one capture client own a given audio input at a time. A parallel AudioRecord while SpeechRecognizer holds the mic does not get the mic — AudioRecord either fails to reach RECORDSTATE_RECORDING or reads silence. The three options from the brief:

• (a) AudioRecord-only mode replacing SpeechRecognizer (PCM -> local-inference ASR).
• (b) Coexistence, if the device permits it.
• (c) A distinct "diarization mode" that suspends SpeechRecognizer while AudioRecord runs.

Chosen: (c). startAudioFrames detects an active SpeechRecognizer session and suspends it (recognizer.cancel() + destroy(), cancel the restart/silence jobs) before opening AudioRecord; stopAudioFrames releases AudioRecord and resumes SpeechRecognizer if a session is still active (enabled && !stopRequested). This is the cleanest viable path that does NOT regress the working SODA STT default: it is a separate opt-in method; start() is unchanged and frame capture is off unless explicitly requested.

Coexistence (b) was measured infeasible on the Pixel 9a (see verification): AudioRecord reaches RECORDING and frames flow only AFTER STT is suspended; the design proactively suspends rather than relying on concurrent capture.

Option (a) — AudioRecord-only + local transcription — is a strict superset of this PCM capture (the PCM produced here is exactly what a local-ASR consumer would feed to local ASR). It is left to the JS consumer: this layer's job is to deliver verified PCM frames; transcription backend choice lives above it.

API (added)

• startAudioFrames({ sampleRate = 16000, frameMs = 20 }) → { started, sampleRate, frameSamples, suspendedStt, error? }
• stopAudioFrames() → void
• isCapturingAudioFrames() → { capturing }
• Event audioFrame: { pcm16 (base64 LE s16 mono), sampleRate, channels: 1, samples, rms (0..1), timestamp (elapsedRealtime ms), frameIndex }

The AudioRecord is opened on VOICE_RECOGNITION (preprocessing-light, what diarization wants), falling back to MIC. The read loop runs on Dispatchers.IO; each frame is little-endian-packed, RMS-measured, base64-encoded, and emitted via notifyListeners. Lifecycle is released in stop() and handleOnDestroy().

On-device verification (Pixel 9a, 53081JEBF11586)

Driven via CDP against the WebView Capacitor context (TalkMode plugin), with mic permission granted:

1. PCM frames reach JS (frame-only): startAudioFrames({16000, 20ms}) → started:true, sampleRate:16000, frameSamples:320. 196 frames in 4 s (~49 fps = correct for 20 ms frames), samples:320, channels:1, sampleRate:16000, base64 length 856 (= 640 PCM bytes), RMS varied per frame (min 0.00007, max 0.00679, avg 0.00465 → live audio, not a silence stub), frameIndex strictly monotonic, capturing true→false across start/stop.
2. Mic contention / suspend-resume: start() (STT, state:listening) → startAudioFrames() returned suspendedStt:true and captured 149 frames (avg RMS 0.00485) while STT was suspended → stopAudioFrames() resumed STT (state:listening, enabled:true). Proves AudioRecord only gets the mic once SpeechRecognizer is suspended, and STT cleanly resumes.
3. No STT regression: default start() → started:true, state:listening, enabled:true; isCapturingAudioFrames stays false (frame capture is off unless explicitly requested).

Build: :elizaos-capacitor-talkmode:compileDebugKotlin + :app:assembleDebug (BUILD SUCCESSFUL); APK contains startAudioFrames/audioFrame in the dex; installed + verified on-device.