wehub-resource-sync
338 lines
17 KiB
Markdown
338 lines
17 KiB
Markdown
# FunASR on llama.cpp / GGUF — Design Document
|
||
|
||
This document describes the design of the `runtime/llama.cpp` directory: a C++ /
|
||
ggml runtime that runs FunASR models (Fun-ASR-Nano, SenseVoiceSmall, Paraformer)
|
||
without PyTorch, on CPU and edge devices, with quantized GGUF weights. It is the
|
||
counterpart of [whisper.cpp](https://github.com/ggml-org/whisper.cpp) for FunASR.
|
||
|
||
It is written to be read without the source: it explains *why* the runtime exists,
|
||
*how* each model maps onto ggml, the GGUF weight format, the numerical-fidelity and
|
||
validation methodology, the non-obvious gotchas discovered during the port, and the
|
||
roadmap.
|
||
|
||
> This is the **shared design document** for the FunASR-on-llama.cpp effort and is
|
||
> kept identical across the FunASR family repos (modelscope/FunASR, Fun-ASR,
|
||
> SenseVoice). The three models share one ggml SAN-M encoder / FSMN / fbank
|
||
> foundation, so the design is documented once here in full; a single-model repo
|
||
> ships only the relevant model directory (§2) but the shared design still applies.
|
||
|
||
---
|
||
|
||
## 1. Motivation
|
||
|
||
FunASR's reference inference runs on PyTorch (and vLLM for the LLM-based models) on
|
||
GPU. That is the right tool for a server that batches many requests and wants to
|
||
saturate a GPU. It is the wrong tool when there is **no GPU and no Python**: a
|
||
laptop, a phone, a Raspberry Pi, an embedded C/C++ application, an offline desktop
|
||
app. There, you want a single self-contained binary, a few hundred MB of quantized
|
||
weights, and CPU SIMD.
|
||
|
||
[llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml is the de-facto runtime
|
||
for that world (Ollama, LM Studio, whisper.cpp all build on it). Porting FunASR to
|
||
ggml + GGUF makes FunASR run anywhere llama.cpp runs, dramatically widening the
|
||
deployment surface.
|
||
|
||
| | PyTorch / vLLM (existing) | this runtime (llama.cpp) |
|
||
|---|---|---|
|
||
| target | GPU server, high QPS | CPU / edge / embedded |
|
||
| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |
|
||
| weights | HF fp16/bf16 safetensors | GGUF, 2–8 bit quantization |
|
||
| key tech | PagedAttention, continuous batching | quantization, mmap, CPU SIMD |
|
||
| best for | online service, batch eval | offline, on-device, embedded |
|
||
|
||
These are complementary, not competing: cloud serving stays on vLLM; this runtime
|
||
covers the on-device / offline case.
|
||
|
||
---
|
||
|
||
## 2. System overview
|
||
|
||
Three models are supported. They share more than they differ — all three use the
|
||
same **SAN-M encoder**, the same **FSMN memory block**, the same **kaldi-compatible
|
||
fbank front end**, and the same ggml building blocks.
|
||
|
||
```
|
||
┌─────────────────────── shared C++ / ggml ───────────────────────┐
|
||
audio.wav (16k mono) ──► kaldi 80-mel fbank + LFR(7/6) ──► SAN-M encoder (50 layers, ggml)
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
│ encoder_out [T, 512]
|
||
┌───────────────────────────────────┼───────────────────────────────────┐
|
||
Fun-ASR-Nano SenseVoiceSmall Paraformer
|
||
adaptor → audio embeds + 4 query tokens CIF predictor (host)
|
||
→ inject into Qwen3-0.6B CTC head → greedy CTC → SAN-M decoder (cross-attn)
|
||
(llama_decode embd path) → SentencePiece → argmax → tokens.json
|
||
→ text → text → text
|
||
```
|
||
|
||
| model | head / decoder | autoregressive? | output units |
|
||
|---|---|---|---|
|
||
| Fun-ASR-Nano | adaptor + Qwen3-0.6B LLM | yes (LLM) | Qwen3 BPE |
|
||
| SenseVoiceSmall | CTC | no | spectok BPE (25055) |
|
||
| Paraformer | CIF + SAN-M decoder | no (parallel) | char/BPE (8404) |
|
||
|
||
Directory layout:
|
||
```
|
||
runtime/llama.cpp/
|
||
README.md overview
|
||
DESIGN.md this document
|
||
fun-asr-nano/ funasr-cli, funasr-encoder, funasr-embd, export_encoder_gguf.py
|
||
sensevoice/ funasr-sensevoice, export_sensevoice_gguf.py, detok.py
|
||
paraformer/ funasr-paraformer, export_paraformer_gguf.py, detok_paraformer.py
|
||
```
|
||
Each model dir holds the llama.cpp example sources (drop-in under `examples/`), a
|
||
GGUF export script, and a model-specific README.
|
||
|
||
---
|
||
|
||
## 3. Audio front end (kaldi fbank in C++)
|
||
|
||
All models use FunASR's `WavFrontend`: kaldi-compatible 80-bin log-mel fbank with a
|
||
hamming window (25 ms / 10 ms), pre-emphasis 0.97, DC removal, 512-pt FFT, then
|
||
**Low-Frame-Rate (LFR)** stacking of 7 frames with stride 6 → a 560-dim feature
|
||
per output frame.
|
||
|
||
The C++ implementation (`compute_fbank`) reproduces this exactly:
|
||
1. upscale the waveform by 32768 (FunASR feeds int16-range samples to kaldi),
|
||
2. per frame: remove DC offset, pre-emphasis, hamming window, zero-pad to 512,
|
||
3. radix-2 FFT, power spectrum, 80 triangular mel filters (kaldi mel: `1127·ln(1+f/700)`,
|
||
low 20 Hz, high 8000 Hz), log floor `FLT_EPSILON`,
|
||
4. LFR: left-pad 3 copies of frame 0, stack 7 frames stride 6 → 560-dim.
|
||
|
||
**Validation:** vs torchaudio kaldi.fbank (dither=0), cosine **1.000000**,
|
||
max_abs_diff 1.75e-3.
|
||
|
||
**Gotcha — dither.** FunASR's frontend uses `dither=1.0` by default, which adds
|
||
random noise per sample, so the fbank (and everything downstream) is *non-deterministic*
|
||
in the reference. The C++ front end uses dither=0 (deterministic). The model is
|
||
robust to this; it accounts for the small (<1%) cosine gap seen when comparing
|
||
against a dithered reference.
|
||
|
||
---
|
||
|
||
## 4. The SAN-M encoder in ggml
|
||
|
||
The SenseVoice/Paraformer encoder is a 50-layer (Paraformer) or 50+20-layer
|
||
(SenseVoice, with extra `tp_encoders`) **SAN-M** stack. Each layer is pre-norm:
|
||
|
||
```
|
||
x → LN → SAN-M self-attention → +residual → LN → FFN(relu) → +residual
|
||
```
|
||
|
||
SAN-M self-attention = standard multi-head attention **plus** an FSMN memory branch
|
||
that runs in parallel on the value projection and is added to the attention output:
|
||
|
||
```
|
||
q,k,v = split(linear_q_k_v(x)) # one fused projection
|
||
fsmn = FSMN(v) # depthwise conv over time + residual
|
||
attn = softmax(qkᵀ/√d)·v → linear_out
|
||
out = attn + fsmn
|
||
```
|
||
|
||
### 4.1 FSMN as an exact f32 shift-accumulate (design decision)
|
||
|
||
FSMN is a per-channel (depthwise) 1-D convolution over time with a symmetric
|
||
kernel (size 11). ggml has `ggml_conv_1d_dw`, but it (a) requires the kernel in
|
||
F16 and (b) is flagged as "very likely wrong for some cases" upstream. Both are
|
||
unacceptable for a faithful port.
|
||
|
||
Instead FSMN is implemented as an **exact f32 shift-accumulate**: the kernel is
|
||
exported as `[K, D]`, the value tensor is zero-padded by `(K-1)/2` on each side
|
||
along time, and the output is `Σ_j kernel[:,j] ⊙ pad(v)[:, t+j]` plus the residual.
|
||
This is 11 element-wise multiply-adds — exact in f32, no F16 rounding, no dependence
|
||
on the questionable conv kernel. It dropped the full-encoder max_abs_diff vs PyTorch
|
||
from 2.93 to **0.0052**.
|
||
|
||
### 4.2 Position encoding & input scaling
|
||
|
||
Input is pre-scaled by `√(d_model)=√512` then a sinusoidal position encoding is
|
||
added, with **depth = the input feature dim (560)** and **positions starting at 1**
|
||
(not 0) — both quirks of the FunASR encoder that must be matched exactly.
|
||
|
||
### 4.3 LayerNorm
|
||
|
||
eps = 1e-5 everywhere.
|
||
|
||
**Validation:** first layer cosine 1.0 (max_abs_diff 1.8e-4); full encoder cosine
|
||
**1.000000**, max_abs_diff 5.2e-3 (f32).
|
||
|
||
---
|
||
|
||
## 5. Per-model design
|
||
|
||
### 5.1 Fun-ASR-Nano (encoder + adaptor + LLM)
|
||
|
||
Pipeline: `fbank → encoder → adaptor → audio embeds [T', 1024] → inject into
|
||
Qwen3-0.6B → text`.
|
||
|
||
- **LLM half is native.** Qwen3 is supported by llama.cpp, so the extracted
|
||
Qwen3-0.6B converts to GGUF with the stock `convert_hf_to_gguf.py` and runs
|
||
unchanged.
|
||
- **Embedding injection.** The audio embeddings are fed into the LLM through
|
||
`llama_decode`'s embedding-input path — exactly how llava/mtmd inject vision
|
||
embeddings. The integrated CLI builds the prompt as a *mixed* sequence:
|
||
`[prefix tokens | audio embeds | suffix tokens]`, where prefix/suffix are fed as
|
||
token ids (llama.cpp embeds them internally; `llama_tokenize(parse_special=true)`
|
||
reproduces the exact 18-token prefix) and the audio slot is fed as embeddings.
|
||
- **Low-frame-rate truncation (critical).** The adaptor emits `T'` frames, but the
|
||
model only uses the first `fake_token_len` of them as audio tokens, where
|
||
`fake_token_len` derives from the fbank length by a 3-stage `÷2` formula
|
||
(≈ T'/8). Feeding all `T'` frames is out-of-distribution and makes the LLM loop.
|
||
- **Chunking.** Decoding a long (e.g. 60 s) clip as one segment is OOD and triggers
|
||
greedy repetition; the CLI's `--chunk 15` splits into windows with a fresh KV per
|
||
window, dropping micro-CER from ~29% to ~9.5%.
|
||
- **Numerics.** The adaptor output has large magnitude (std ≈ 28, |max| ≈ 1187), so
|
||
fp16 can overflow; the runtime uses f32/f16 weights with f32 activations.
|
||
|
||
### 5.2 SenseVoiceSmall (encoder + CTC)
|
||
|
||
Pipeline: `fbank → prepend 4 query tokens → encoder → CTC head → greedy CTC →
|
||
SentencePiece`.
|
||
|
||
- **Query tokens.** Four learned embeddings are prepended: `[language(auto), event,
|
||
emotion, textnorm]` (indices `[0,1,2,15]` for auto/woitn). They are 560-dim and
|
||
prepended *before* the encoder's `√512` scaling and position encoding.
|
||
- **CTC decode.** `argmax → collapse consecutive → drop blank(0)` → ids → SentencePiece.
|
||
- **Gotcha — no CMVN at inference.** SenseVoice's `inference()` feeds the **raw**
|
||
log-mel fbank to the encoder; it does **not** apply `am.mvn` CMVN (that code path
|
||
is unused at inference). Applying CMVN makes the model predict `<|nospeech|>`.
|
||
|
||
**Validation:** CTC token ids **identical** to PyTorch (108/108 on a clip); text
|
||
matches `AutoModel` exactly.
|
||
|
||
### 5.3 Paraformer (encoder + CIF + decoder, non-autoregressive)
|
||
|
||
Pipeline: `fbank → CMVN → encoder → CIF predictor → acoustic embeds [N, 512] →
|
||
SAN-M decoder (cross-attn to encoder) → argmax → tokens.json`.
|
||
|
||
- **CMVN IS applied** here (unlike SenseVoice): `(fbank + shift)·scale`, per-dim
|
||
(560), from `am.mvn`.
|
||
- **CIF predictor (runs on host).** Continuous Integrate-and-Fire: a 1-D conv
|
||
(k=3) + residual + relu + linear → sigmoid → per-frame weight α; then a sequential
|
||
integrate-and-fire loop emits one acoustic embedding each time the running α-sum
|
||
crosses 1.0. This both decides the token count and produces the decoder input. It
|
||
is inherently sequential, so it runs in plain C++ (cheap: ~0.5 G MACs); the
|
||
encoder and decoder run in ggml.
|
||
- **SAN-M decoder (ggml).** 16 layers, each: `FFN → FSMN self-attention →
|
||
cross-attention to the encoder output`. The self-attention is **FSMN-only** (no
|
||
QK attention); cross-attention has q from the decoder slots and k,v from a fused
|
||
`linear_k_v` of the encoder output. A 17th `decoders3` layer is FFN-only. The
|
||
decoder FFN has an internal LayerNorm and the second linear has no bias. The
|
||
layer ordering (FFN *before* the attention inside the residual) is unusual and is
|
||
matched exactly.
|
||
|
||
**Validation:** decoded text **identical** to `AutoModel`; CIF token count exact
|
||
(105/105). Encoder cosine 0.997 (residual is the reference's random dither).
|
||
|
||
**Gotcha — `am.mvn` has three bracketed blocks.** `[Splice idx]`, `[AddShift=shift]`,
|
||
`[Rescale=scale]`. The shift/scale are the two 560-length vectors; naively taking
|
||
the first two blocks grabs `[0]` as the shift and mis-scales everything, which makes
|
||
CIF emit ~4× too few tokens. Parse by length.
|
||
|
||
---
|
||
|
||
## 6. GGUF conversion & weight layout
|
||
|
||
Each model has an `export_*_gguf.py` that packs weights + architecture metadata into
|
||
a single GGUF.
|
||
|
||
- Tensor names are kept verbatim from the checkpoint (e.g.
|
||
`encoder.encoders.3.norm1.weight`); the C++ looks them up by name.
|
||
- **FSMN kernels** are transposed from `(D,1,K)` to `[K,D]` at export so the C++
|
||
shift-accumulate can take a contiguous per-tap `[D]` vector.
|
||
- **CMVN** (`am.mvn`) is parsed to `cmvn.shift` / `cmvn.scale` tensors (Paraformer
|
||
uses them; SenseVoice ships them but the runtime ignores them).
|
||
- **Quantization.** `--wtype f16` stores the 2-D matmul weights as F16 (norms,
|
||
biases, FSMN kernels stay f32), halving the encoder GGUF (e.g. 935 → 469 MB) with
|
||
cosine 0.999999. The Qwen3 LLM uses the standard llama.cpp quantizer (Q8_0 / Q4_K_M).
|
||
|
||
| file | model | dtype | size |
|
||
|---|---|---|---|
|
||
| funasr-encoder.gguf | Nano | f32 / f16 | 935 / 469 MB |
|
||
| qwen3-0.6b-q8_0.gguf | Nano LLM | Q8_0 | 805 MB |
|
||
| sensevoice-small.gguf | SenseVoice | f32 | 936 MB |
|
||
| paraformer.gguf | Paraformer | f32 | 863 MB |
|
||
|
||
---
|
||
|
||
## 7. Numerical fidelity & validation methodology
|
||
|
||
The port is validated **stage by stage** against the PyTorch reference, using golden
|
||
dumps (fbank, encoder output, adaptor/CIF output, logits/ids) compared by cosine
|
||
similarity and max-abs-diff, then end-to-end by transcription text / CER.
|
||
|
||
Summary of results (benchmark clip / set):
|
||
|
||
| stage | metric |
|
||
|---|---|
|
||
| kaldi fbank vs torchaudio | cosine 1.000000 |
|
||
| SAN-M encoder (full) vs PyTorch | cosine 1.000000, max_abs_diff 5e-3 (f32) |
|
||
| SenseVoice CTC ids | identical (108/108) |
|
||
| Paraformer text / token count | identical / 105 = 105 |
|
||
| Fun-ASR-Nano end-to-end CER (same conditions) | C++ 11.68% vs PyTorch 11.70% (Δ0.02%) |
|
||
|
||
**Why not bit-exact tokens everywhere?** Greedy decoding is chaotic: a ~5e-3
|
||
difference (from ggml-CPU vs torch-GPU matmul summation order) can flip a token on
|
||
a borderline frame, and over a long sequence the paths diverge — *this also happens
|
||
between PyTorch's own GPU and CPU*. What is faithful and what we verify is (a) the
|
||
per-tensor numerics (cosine 1.0) and (b) the **aggregate CER**, which matches the
|
||
reference under identical conditions.
|
||
|
||
**fp16 caution.** The Fun-ASR-Nano adaptor output magnitude (std ≈ 28) can overflow
|
||
fp16; the audio path is kept in f32 (weights may be f16, activations f32).
|
||
|
||
---
|
||
|
||
## 8. Performance
|
||
|
||
CPU, 8 threads, a 44 s clip:
|
||
- Encoder (50 layers): ~1.2 s. Paraformer decoder: ~0.5 s.
|
||
- Fun-ASR-Nano end-to-end (with LLM): ~7 s.
|
||
- Fully-quantized footprint (f16 encoder + Q8 LLM) ≈ 1.3 GB.
|
||
|
||
These are first-correctness numbers; quantizing the encoder and threading/batching
|
||
the front end are open optimizations.
|
||
|
||
---
|
||
|
||
## 9. Design decisions & trade-offs
|
||
|
||
- **ggml for encoder/decoder, host C++ for CIF.** The neural matmul-heavy parts run
|
||
in ggml (SIMD, future GPU backends); CIF is a sequential scalar loop with data-
|
||
dependent control flow, so it is clearer and not slower in plain C++.
|
||
- **Exact f32 FSMN instead of `ggml_conv_1d_dw`.** Correctness and f32 precision
|
||
over reusing a flagged, F16-only op (§4.1).
|
||
- **Prompt as tokens, not a Python embedding table.** The integrated CLI tokenizes
|
||
the prompt with `llama_tokenize` and lets llama.cpp embed it, so no embedding
|
||
matrix needs to be shipped or matched (Fun-ASR-Nano).
|
||
- **f32 by default, f16/Q8 opt-in.** f32 is the faithful default; quantization is a
|
||
size/latency lever the user opts into. (Interestingly, Q8 on the LLM slightly
|
||
*helps* greedy stability by regularizing away from repetition loops.)
|
||
- **Per-model self-contained example dirs.** Mirrors llama.cpp's `examples/` layout
|
||
so each builds as a drop-in target; the shared code is duplicated rather than
|
||
factored to keep each example independently buildable.
|
||
|
||
---
|
||
|
||
## 10. Limitations & roadmap
|
||
|
||
- **WAV input** assumes 16 kHz mono PCM16; arbitrary formats / resampling are TODO.
|
||
- **VAD.** Long audio needs segmentation; today Fun-ASR-Nano uses fixed `--chunk`
|
||
windows. A real FSMN-VAD front end would close the last ~1.3% CER gap to the
|
||
production VAD-segmented number and is the highest-value next step.
|
||
- **Single packaged GGUF** (encoder + adaptor + LLM in one file) and a one-command
|
||
converter.
|
||
- **Encoder/decoder quantization** (Q8 via gguf-py quants), streaming, timestamps
|
||
(Paraformer CIF peaks give alignment; SenseVoice/Nano via CTC).
|
||
- **Upstream.** The example sources are drop-in for llama.cpp; upstreaming the
|
||
runtime to ggml-org/llama.cpp (as whisper.cpp-style tools) is a separate track.
|
||
|
||
---
|
||
|
||
## 11. Reproducing the validation
|
||
|
||
Each model dir's README has the build + convert + run quickstart. The export
|
||
scripts read a standard FunASR checkpoint (`model.pt` + `config.yaml` + `am.mvn` /
|
||
tokenizer). To reproduce a stage comparison, dump the corresponding PyTorch tensor
|
||
(`model.encode`, `model.calc_predictor`, …) and compare with cosine / max-abs-diff;
|
||
the numbers in §7 should reproduce within dither noise.
|