wehub-resource-sync
102 lines
3.9 KiB
Markdown
102 lines
3.9 KiB
Markdown
# SenseVoiceSmall on llama.cpp / GGUF
|
|
|
|
Run **SenseVoiceSmall** on the [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
|
/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. Like
|
|
[whisper.cpp](https://github.com/ggml-org/whisper.cpp), but for SenseVoice.
|
|
|
|
## Why this exists
|
|
|
|
SenseVoiceSmall normally runs on PyTorch / ONNX / libtorch. This runtime ports it
|
|
to **ggml + GGUF** so it can run CPU-only, offline, embedded in a C/C++ app, with
|
|
quantized weights. Use it on laptops / phones / edge boxes where there is no GPU
|
|
and no Python. (For high-QPS GPU serving, the PyTorch/vLLM path is still the way.)
|
|
|
|
## Architecture
|
|
|
|
SenseVoiceSmall = **SAN-M encoder (70 layers) + CTC head** — no LLM, no autoregression.
|
|
The whole pipeline runs in C++:
|
|
|
|
```
|
|
audio.wav (16k mono)
|
|
│ kaldi 80-mel fbank + LFR (C++)
|
|
▼
|
|
features [T, 560]
|
|
│ prepend 4 query tokens [lang, event, emotion, itn]
|
|
▼
|
|
[4 + T, 560]
|
|
│ SAN-M encoder (ggml) ── sensevoice-small.gguf
|
|
▼
|
|
encoder out [4+T, 512]
|
|
│ CTC head (Linear 512→25055) → greedy CTC (argmax, dedup, drop blank)
|
|
▼
|
|
token ids
|
|
│ SentencePiece detok (detok.py)
|
|
▼
|
|
<|zh|><|NEUTRAL|><|Speech|><|woitn|> transcription...
|
|
```
|
|
|
|
The SAN-M encoder is the same architecture as Fun-ASR-Nano's, so the ggml forward
|
|
is shared between the two runtimes.
|
|
|
|
## Quickstart
|
|
|
|
**1. Build:**
|
|
```bash
|
|
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
|
cp -r /path/to/runtime/llama.cpp/funasr-sensevoice examples/
|
|
echo 'add_subdirectory(funasr-sensevoice)' >> examples/CMakeLists.txt
|
|
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
|
|
cmake --build build -j --target llama-funasr-sensevoice
|
|
```
|
|
|
|
**2. Convert weights** (needs the checkpoint, e.g. `FunAudioLLM/SenseVoiceSmall`):
|
|
```bash
|
|
python runtime/llama.cpp/export_sensevoice_gguf.py \
|
|
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
|
--out sensevoice-small.gguf # f32, ~936 MB
|
|
python runtime/llama.cpp/export_sensevoice_gguf.py --wtype f16 \
|
|
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
|
--out sensevoice-small-f16.gguf # half size
|
|
```
|
|
|
|
**3. Transcribe:**
|
|
```bash
|
|
build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a audio.wav # prints transcription text
|
|
# --keep-tags keeps the <|lang|>/<|emotion|>/<|event|> tags; --ids prints raw CTC ids
|
|
```
|
|
Expected output:
|
|
```
|
|
我想问我在滨海新区有房我一直没有照顾孩子...你觉得这是正常的想法吗
|
|
```
|
|
The leading `<|...|>` tags are the predicted language / emotion / event / ITN.
|
|
|
|
## Accuracy & validation
|
|
|
|
- **CTC token ids (C++) vs PyTorch:** **identical** (108/108 on a benchmark clip).
|
|
- **Detokenized text:** matches the FunASR `AutoModel` output **exactly**.
|
|
- Encoder validated against PyTorch (shared with Fun-ASR-Nano runtime): cosine 1.0.
|
|
- Encode time ≈ **1.3 s** on CPU for a 44 s clip.
|
|
|
|
## Tips & gotchas
|
|
|
|
- **No CMVN at inference.** SenseVoice `inference()` feeds the **raw** log-mel fbank
|
|
to the encoder; it does **not** apply `am.mvn`. Applying CMVN makes the model
|
|
predict `<|nospeech|>`. (The export script reads `am.mvn` for completeness but the
|
|
runtime does not use it.)
|
|
- **Query tokens (4)** are prepended from `embed.weight`, default indices
|
|
`[language=auto(0), event=1, emotion=2, textnorm=woitn(15)]`. Change them for a
|
|
fixed language or to enable ITN (`withitn=14`).
|
|
- **WAV input** assumes 16 kHz mono PCM16.
|
|
- LayerNorm eps = 1e-5; FSMN = exact f32 shift-accumulate; fbank matches torchaudio.
|
|
|
|
## Files
|
|
```
|
|
funasr-sensevoice/ ggml runtime: WAV → CTC token ids
|
|
export_sensevoice_gguf.py export encoder + CTC head + query embeddings to GGUF
|
|
detok.py SentencePiece id → text (bpe model ships with the checkpoint)
|
|
```
|
|
|
|
## Roadmap
|
|
- Built-in SentencePiece detok (drop the Python step); arbitrary WAV formats;
|
|
encoder Q8 quantization; timestamps.
|