chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,104 @@
|
||||
# FunASR (llama.cpp / GGUF) vs whisper.cpp — CPU benchmark
|
||||
|
||||
How does the FunASR llama.cpp runtime compare with [whisper.cpp](https://github.com/ggml-org/whisper.cpp),
|
||||
the de-facto on-device ASR runtime, on **Chinese** speech? This page reports a
|
||||
head-to-head on identical hardware and audio.
|
||||
|
||||
**TL;DR — for Chinese ASR on CPU, FunASR is ~2.7× more accurate than whisper.cpp at
|
||||
every model tier, and faster.**
|
||||
|
||||
## Results
|
||||
|
||||
Dataset: **184 real Mandarin clips with human references** (the standard FunASR
|
||||
benchmark set). Metric: **micro-CER** with `normalize_zh` (lower is better). Speed:
|
||||
real-time factor on **CPU, 8 threads** (model-load excluded). whisper forced to
|
||||
Chinese (`-l zh`).
|
||||
|
||||
| system | **CER** (micro, normalize_zh) ↓ | speed ↑ | size |
|
||||
|---|---|---|---|
|
||||
| **FunASR Fun-ASR-Nano** | **8.06** (fp32 ref) / **8.42** (Q8 runtime) | LLM decode¹ | enc + Qwen3-0.6B GGUF |
|
||||
| **FunASR SenseVoiceSmall** | **7.81** (fp32 ref) / **8.17** (Q8 runtime) | **~20× real-time** | 449 MB (f16) |
|
||||
| **FunASR Paraformer** | **10.18** (fp32 ref) / **9.89** (Q8 runtime) | **~21× real-time** | 401 MB (f16) |
|
||||
| whisper.cpp base | 31.33 | 9.9× | 142 MB |
|
||||
| whisper.cpp small | 22.12 | 4.6× | 466 MB |
|
||||
| whisper.cpp large-v3-turbo | 23.15 | 3.2× | 1.6 GB |
|
||||
|
||||
**Each FunASR row shows two numbers:** the published **fp32 reference** (PyTorch,
|
||||
the number on funasr.com / the model cards) and the **Q8 llama.cpp CPU runtime**
|
||||
measured here. The ~0.3 % gap is normal int8 quantization + VAD segment boundaries;
|
||||
Q8 is the real CPU/edge deployment config. Either way, **FunASR ~8–10 % vs
|
||||
whisper.cpp 22–31 % — a 2.7×+ accuracy gap that holds at every tier.**
|
||||
|
||||
¹ Fun-ASR-Nano runs an autoregressive 0.6B LLM decoder (slower than the encoder-only
|
||||
SenseVoice/Paraformer; it is the accuracy leader). A clean RTF lands once the CLI
|
||||
separates model-load from compute.
|
||||
|
||||
### Transparency / segmentation (read this before quoting numbers)
|
||||
|
||||
- **Segmentation differs by system, each using its natural strategy:** FunASR uses an
|
||||
`fsmn-vad` front end (segments → ASR → concatenate); whisper.cpp uses its built-in
|
||||
30 s windowing. This is a fair system-level comparison.
|
||||
- **Engine-internal VAD is now implemented** — a native ggml FSMN-VAD built into the
|
||||
binaries (`--vad fsmn-vad.gguf`). The **bare binary, with no Python front end**, now
|
||||
reaches the reference end-to-end: SenseVoiceSmall **8.01 %**, Paraformer **9.85 %**,
|
||||
Fun-ASR-Nano **8.30 %** (micro, normalize_zh, full 184). The built-in C++ VAD matches
|
||||
the PyTorch `fsmn-vad` front end (segment boundaries within ~10 ms, slightly better
|
||||
CER), so the runtime is now fully self-contained.
|
||||
- For full disclosure, **bare binary with no VAD at all (whole-clip)** is higher —
|
||||
SenseVoiceSmall 9.99 %, Paraformer 12.82 % — because long clips decoded as one segment
|
||||
are out-of-distribution; that is exactly what the built-in VAD fixes.
|
||||
|
||||
## Why FunASR wins on Chinese
|
||||
|
||||
1. **Training data.** SenseVoice / Paraformer / Fun-ASR-Nano are trained primarily on
|
||||
large-scale Mandarin; Whisper is a general multilingual model where Chinese is a
|
||||
small slice. On Chinese homophones Whisper makes substitution errors the FunASR
|
||||
models do not (example below).
|
||||
2. **Architecture → speed.** Paraformer is non-autoregressive (CIF predictor + one
|
||||
decoder pass) and SenseVoiceSmall is encoder + CTC (one forward pass); Whisper is
|
||||
autoregressive (one step per output token).
|
||||
|
||||
## Qualitative example (clip 002)
|
||||
|
||||
| system | output (excerpt) |
|
||||
|---|---|
|
||||
| ground truth | 我想问，我在**滨海新区**有房…所以我必须拿到**抚养权** |
|
||||
| FunASR (Nano / SenseVoice / Paraformer) | …我在**滨海新区**有房…拿到**抚养权**… ✓ |
|
||||
| whisper base | …我在**冰海心区**有房…我想要**扶养权**…上学**方面**… ✗ |
|
||||
| whisper small | …我在**冰海新区**有房…我想要**抚养全**… ✗ |
|
||||
| whisper large-v3-turbo | …滨海新区…上学**方面**… ✗ |
|
||||
|
||||
## Methodology
|
||||
|
||||
- **Data:** the standard 184-clip Mandarin benchmark set (`benchmark/testset.json`),
|
||||
~44–60 s each, with human references.
|
||||
- **Metric (canonical):** **micro-average CER** (`Σ edits / Σ ref chars`) after
|
||||
**`normalize_zh`**: `re.sub(r'[^\w一-鿿]', '', text).upper()` (strip punctuation/
|
||||
whitespace, keep word chars + CJK, upper-case; SenseVoice `<|...|>` tags stripped).
|
||||
This is the canonical FunASR口径 — the same one behind the published fp32 numbers.
|
||||
(A macro-average / simplified-normalize variant gives different, non-canonical
|
||||
numbers; it is not used here.)
|
||||
- **FunASR fp32 reference:** PyTorch, micro + normalize_zh, 184 set — SenseVoice 7.81,
|
||||
Paraformer 10.18, Fun-ASR-Nano 8.06 (matches funasr.com / READMEs / model cards).
|
||||
- **FunASR Q8 runtime:** this llama.cpp runtime (Q8 LLM / f16 encoder) + `fsmn-vad`
|
||||
front end (`max_single_segment_time=30000`), full 184. SenseVoice uses `use_itn=True`
|
||||
to match the reference.
|
||||
- **whisper.cpp:** ggml `base` / `small` / `large-v3-turbo`, `-l zh`, internal 30 s
|
||||
windowing, full 184.
|
||||
- **Speed (RTF):** `Σ compute_time / Σ audio_duration`, model-load excluded, **8 threads
|
||||
for all systems**.
|
||||
|
||||
## Caveats (fair use)
|
||||
|
||||
- This is a **Chinese** benchmark — FunASR's home turf. Whisper is a *general
|
||||
multilingual* model (translation, 99 languages, timestamps); for English / other
|
||||
languages it is the stronger general choice. The takeaway is specifically:
|
||||
**for Chinese ASR on CPU, FunASR is the accuracy + speed leader.**
|
||||
- SenseVoiceSmall also outputs language ID / emotion / audio-event; Paraformer is
|
||||
Mandarin-specialised; Fun-ASR-Nano is the most accurate (LLM decoder). Pick per use case.
|
||||
|
||||
## Reproduce
|
||||
|
||||
See [`benchmarks/`](benchmarks/) — `compute_cer.py` (micro-CER + normalize_zh + RTF)
|
||||
and the per-system run commands. Produce hypotheses with each tool, then compute CER
|
||||
against the references and RTF against clip durations.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Standalone, CI-friendly build for the FunASR llama.cpp runtime.
|
||||
#
|
||||
# Fetches a pinned llama.cpp (which provides ggml + llama) and builds the runtime
|
||||
# binaries against it — no manual copying into a llama.cpp checkout, no hardcoded
|
||||
# paths. Linux / macOS / Windows, x64 / arm64. Static libs -> self-contained binaries.
|
||||
#
|
||||
# cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||||
# cmake --build build -j # binaries land in build/bin/
|
||||
#
|
||||
# Build against a local checkout instead of fetching:
|
||||
# -DFETCHCONTENT_SOURCE_DIR_LLAMA=/path/to/llama.cpp
|
||||
cmake_minimum_required(VERSION 3.16)
|
||||
project(funasr-llamacpp CXX C)
|
||||
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
if(MSVC)
|
||||
add_compile_definitions(NOMINMAX _USE_MATH_DEFINES) # min/max macros + M_PI on MSVC
|
||||
endif()
|
||||
set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) # static deps -> self-contained binaries
|
||||
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
|
||||
if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
|
||||
set(CMAKE_BUILD_TYPE Release CACHE STRING "" FORCE)
|
||||
endif()
|
||||
|
||||
include(FetchContent)
|
||||
set(LLAMA_BUILD_TESTS OFF CACHE BOOL "" FORCE)
|
||||
set(LLAMA_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
|
||||
set(LLAMA_BUILD_TOOLS OFF CACHE BOOL "" FORCE)
|
||||
set(LLAMA_BUILD_SERVER OFF CACHE BOOL "" FORCE)
|
||||
set(LLAMA_CURL OFF CACHE BOOL "" FORCE)
|
||||
FetchContent_Declare(llama
|
||||
GIT_REPOSITORY https://github.com/ggml-org/llama.cpp.git
|
||||
GIT_TAG 8086439a4cea94c71a5dfb8fe4ad1546aebd640f)
|
||||
FetchContent_MakeAvailable(llama)
|
||||
|
||||
find_package(Threads REQUIRED)
|
||||
set(FUNASR_COMMON ${CMAKE_CURRENT_SOURCE_DIR}/funasr-common)
|
||||
|
||||
function(funasr_add name src)
|
||||
add_executable(${name} ${src})
|
||||
target_include_directories(${name} PRIVATE ${FUNASR_COMMON})
|
||||
target_link_libraries(${name} PRIVATE ${ARGN} Threads::Threads)
|
||||
install(TARGETS ${name} RUNTIME DESTINATION bin)
|
||||
endfunction()
|
||||
|
||||
funasr_add(llama-funasr-cli fun-asr-nano/funasr-cli/funasr-cli.cpp llama ggml)
|
||||
funasr_add(llama-funasr-encoder fun-asr-nano/funasr-encoder/funasr-encoder.cpp ggml)
|
||||
funasr_add(llama-funasr-embd fun-asr-nano/funasr-embd/funasr-embd.cpp llama ggml)
|
||||
funasr_add(llama-funasr-sensevoice sensevoice/funasr-sensevoice/funasr-sensevoice.cpp ggml)
|
||||
funasr_add(llama-funasr-paraformer paraformer/funasr-paraformer/funasr-paraformer.cpp ggml)
|
||||
funasr_add(llama-funasr-vad funasr-vad/funasr-vad.cpp ggml)
|
||||
@@ -0,0 +1,337 @@
|
||||
# FunASR on llama.cpp / GGUF — Design Document
|
||||
|
||||
This document describes the design of the `runtime/llama.cpp` directory: a C++ /
|
||||
ggml runtime that runs FunASR models (Fun-ASR-Nano, SenseVoiceSmall, Paraformer)
|
||||
without PyTorch, on CPU and edge devices, with quantized GGUF weights. It is the
|
||||
counterpart of [whisper.cpp](https://github.com/ggml-org/whisper.cpp) for FunASR.
|
||||
|
||||
It is written to be read without the source: it explains *why* the runtime exists,
|
||||
*how* each model maps onto ggml, the GGUF weight format, the numerical-fidelity and
|
||||
validation methodology, the non-obvious gotchas discovered during the port, and the
|
||||
roadmap.
|
||||
|
||||
> This is the **shared design document** for the FunASR-on-llama.cpp effort and is
|
||||
> kept identical across the FunASR family repos (modelscope/FunASR, Fun-ASR,
|
||||
> SenseVoice). The three models share one ggml SAN-M encoder / FSMN / fbank
|
||||
> foundation, so the design is documented once here in full; a single-model repo
|
||||
> ships only the relevant model directory (§2) but the shared design still applies.
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
FunASR's reference inference runs on PyTorch (and vLLM for the LLM-based models) on
|
||||
GPU. That is the right tool for a server that batches many requests and wants to
|
||||
saturate a GPU. It is the wrong tool when there is **no GPU and no Python**: a
|
||||
laptop, a phone, a Raspberry Pi, an embedded C/C++ application, an offline desktop
|
||||
app. There, you want a single self-contained binary, a few hundred MB of quantized
|
||||
weights, and CPU SIMD.
|
||||
|
||||
[llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml is the de-facto runtime
|
||||
for that world (Ollama, LM Studio, whisper.cpp all build on it). Porting FunASR to
|
||||
ggml + GGUF makes FunASR run anywhere llama.cpp runs, dramatically widening the
|
||||
deployment surface.
|
||||
|
||||
| | PyTorch / vLLM (existing) | this runtime (llama.cpp) |
|
||||
|---|---|---|
|
||||
| target | GPU server, high QPS | CPU / edge / embedded |
|
||||
| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |
|
||||
| weights | HF fp16/bf16 safetensors | GGUF, 2–8 bit quantization |
|
||||
| key tech | PagedAttention, continuous batching | quantization, mmap, CPU SIMD |
|
||||
| best for | online service, batch eval | offline, on-device, embedded |
|
||||
|
||||
These are complementary, not competing: cloud serving stays on vLLM; this runtime
|
||||
covers the on-device / offline case.
|
||||
|
||||
---
|
||||
|
||||
## 2. System overview
|
||||
|
||||
Three models are supported. They share more than they differ — all three use the
|
||||
same **SAN-M encoder**, the same **FSMN memory block**, the same **kaldi-compatible
|
||||
fbank front end**, and the same ggml building blocks.
|
||||
|
||||
```
|
||||
┌─────────────────────── shared C++ / ggml ───────────────────────┐
|
||||
audio.wav (16k mono) ──► kaldi 80-mel fbank + LFR(7/6) ──► SAN-M encoder (50 layers, ggml)
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│ encoder_out [T, 512]
|
||||
┌───────────────────────────────────┼───────────────────────────────────┐
|
||||
Fun-ASR-Nano SenseVoiceSmall Paraformer
|
||||
adaptor → audio embeds + 4 query tokens CIF predictor (host)
|
||||
→ inject into Qwen3-0.6B CTC head → greedy CTC → SAN-M decoder (cross-attn)
|
||||
(llama_decode embd path) → SentencePiece → argmax → tokens.json
|
||||
→ text → text → text
|
||||
```
|
||||
|
||||
| model | head / decoder | autoregressive? | output units |
|
||||
|---|---|---|---|
|
||||
| Fun-ASR-Nano | adaptor + Qwen3-0.6B LLM | yes (LLM) | Qwen3 BPE |
|
||||
| SenseVoiceSmall | CTC | no | spectok BPE (25055) |
|
||||
| Paraformer | CIF + SAN-M decoder | no (parallel) | char/BPE (8404) |
|
||||
|
||||
Directory layout:
|
||||
```
|
||||
runtime/llama.cpp/
|
||||
README.md overview
|
||||
DESIGN.md this document
|
||||
fun-asr-nano/ funasr-cli, funasr-encoder, funasr-embd, export_encoder_gguf.py
|
||||
sensevoice/ funasr-sensevoice, export_sensevoice_gguf.py, detok.py
|
||||
paraformer/ funasr-paraformer, export_paraformer_gguf.py, detok_paraformer.py
|
||||
```
|
||||
Each model dir holds the llama.cpp example sources (drop-in under `examples/`), a
|
||||
GGUF export script, and a model-specific README.
|
||||
|
||||
---
|
||||
|
||||
## 3. Audio front end (kaldi fbank in C++)
|
||||
|
||||
All models use FunASR's `WavFrontend`: kaldi-compatible 80-bin log-mel fbank with a
|
||||
hamming window (25 ms / 10 ms), pre-emphasis 0.97, DC removal, 512-pt FFT, then
|
||||
**Low-Frame-Rate (LFR)** stacking of 7 frames with stride 6 → a 560-dim feature
|
||||
per output frame.
|
||||
|
||||
The C++ implementation (`compute_fbank`) reproduces this exactly:
|
||||
1. upscale the waveform by 32768 (FunASR feeds int16-range samples to kaldi),
|
||||
2. per frame: remove DC offset, pre-emphasis, hamming window, zero-pad to 512,
|
||||
3. radix-2 FFT, power spectrum, 80 triangular mel filters (kaldi mel: `1127·ln(1+f/700)`,
|
||||
low 20 Hz, high 8000 Hz), log floor `FLT_EPSILON`,
|
||||
4. LFR: left-pad 3 copies of frame 0, stack 7 frames stride 6 → 560-dim.
|
||||
|
||||
**Validation:** vs torchaudio kaldi.fbank (dither=0), cosine **1.000000**,
|
||||
max_abs_diff 1.75e-3.
|
||||
|
||||
**Gotcha — dither.** FunASR's frontend uses `dither=1.0` by default, which adds
|
||||
random noise per sample, so the fbank (and everything downstream) is *non-deterministic*
|
||||
in the reference. The C++ front end uses dither=0 (deterministic). The model is
|
||||
robust to this; it accounts for the small (<1%) cosine gap seen when comparing
|
||||
against a dithered reference.
|
||||
|
||||
---
|
||||
|
||||
## 4. The SAN-M encoder in ggml
|
||||
|
||||
The SenseVoice/Paraformer encoder is a 50-layer (Paraformer) or 50+20-layer
|
||||
(SenseVoice, with extra `tp_encoders`) **SAN-M** stack. Each layer is pre-norm:
|
||||
|
||||
```
|
||||
x → LN → SAN-M self-attention → +residual → LN → FFN(relu) → +residual
|
||||
```
|
||||
|
||||
SAN-M self-attention = standard multi-head attention **plus** an FSMN memory branch
|
||||
that runs in parallel on the value projection and is added to the attention output:
|
||||
|
||||
```
|
||||
q,k,v = split(linear_q_k_v(x)) # one fused projection
|
||||
fsmn = FSMN(v) # depthwise conv over time + residual
|
||||
attn = softmax(qkᵀ/√d)·v → linear_out
|
||||
out = attn + fsmn
|
||||
```
|
||||
|
||||
### 4.1 FSMN as an exact f32 shift-accumulate (design decision)
|
||||
|
||||
FSMN is a per-channel (depthwise) 1-D convolution over time with a symmetric
|
||||
kernel (size 11). ggml has `ggml_conv_1d_dw`, but it (a) requires the kernel in
|
||||
F16 and (b) is flagged as "very likely wrong for some cases" upstream. Both are
|
||||
unacceptable for a faithful port.
|
||||
|
||||
Instead FSMN is implemented as an **exact f32 shift-accumulate**: the kernel is
|
||||
exported as `[K, D]`, the value tensor is zero-padded by `(K-1)/2` on each side
|
||||
along time, and the output is `Σ_j kernel[:,j] ⊙ pad(v)[:, t+j]` plus the residual.
|
||||
This is 11 element-wise multiply-adds — exact in f32, no F16 rounding, no dependence
|
||||
on the questionable conv kernel. It dropped the full-encoder max_abs_diff vs PyTorch
|
||||
from 2.93 to **0.0052**.
|
||||
|
||||
### 4.2 Position encoding & input scaling
|
||||
|
||||
Input is pre-scaled by `√(d_model)=√512` then a sinusoidal position encoding is
|
||||
added, with **depth = the input feature dim (560)** and **positions starting at 1**
|
||||
(not 0) — both quirks of the FunASR encoder that must be matched exactly.
|
||||
|
||||
### 4.3 LayerNorm
|
||||
|
||||
eps = 1e-5 everywhere.
|
||||
|
||||
**Validation:** first layer cosine 1.0 (max_abs_diff 1.8e-4); full encoder cosine
|
||||
**1.000000**, max_abs_diff 5.2e-3 (f32).
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-model design
|
||||
|
||||
### 5.1 Fun-ASR-Nano (encoder + adaptor + LLM)
|
||||
|
||||
Pipeline: `fbank → encoder → adaptor → audio embeds [T', 1024] → inject into
|
||||
Qwen3-0.6B → text`.
|
||||
|
||||
- **LLM half is native.** Qwen3 is supported by llama.cpp, so the extracted
|
||||
Qwen3-0.6B converts to GGUF with the stock `convert_hf_to_gguf.py` and runs
|
||||
unchanged.
|
||||
- **Embedding injection.** The audio embeddings are fed into the LLM through
|
||||
`llama_decode`'s embedding-input path — exactly how llava/mtmd inject vision
|
||||
embeddings. The integrated CLI builds the prompt as a *mixed* sequence:
|
||||
`[prefix tokens | audio embeds | suffix tokens]`, where prefix/suffix are fed as
|
||||
token ids (llama.cpp embeds them internally; `llama_tokenize(parse_special=true)`
|
||||
reproduces the exact 18-token prefix) and the audio slot is fed as embeddings.
|
||||
- **Low-frame-rate truncation (critical).** The adaptor emits `T'` frames, but the
|
||||
model only uses the first `fake_token_len` of them as audio tokens, where
|
||||
`fake_token_len` derives from the fbank length by a 3-stage `÷2` formula
|
||||
(≈ T'/8). Feeding all `T'` frames is out-of-distribution and makes the LLM loop.
|
||||
- **Chunking.** Decoding a long (e.g. 60 s) clip as one segment is OOD and triggers
|
||||
greedy repetition; the CLI's `--chunk 15` splits into windows with a fresh KV per
|
||||
window, dropping micro-CER from ~29% to ~9.5%.
|
||||
- **Numerics.** The adaptor output has large magnitude (std ≈ 28, |max| ≈ 1187), so
|
||||
fp16 can overflow; the runtime uses f32/f16 weights with f32 activations.
|
||||
|
||||
### 5.2 SenseVoiceSmall (encoder + CTC)
|
||||
|
||||
Pipeline: `fbank → prepend 4 query tokens → encoder → CTC head → greedy CTC →
|
||||
SentencePiece`.
|
||||
|
||||
- **Query tokens.** Four learned embeddings are prepended: `[language(auto), event,
|
||||
emotion, textnorm]` (indices `[0,1,2,15]` for auto/woitn). They are 560-dim and
|
||||
prepended *before* the encoder's `√512` scaling and position encoding.
|
||||
- **CTC decode.** `argmax → collapse consecutive → drop blank(0)` → ids → SentencePiece.
|
||||
- **Gotcha — no CMVN at inference.** SenseVoice's `inference()` feeds the **raw**
|
||||
log-mel fbank to the encoder; it does **not** apply `am.mvn` CMVN (that code path
|
||||
is unused at inference). Applying CMVN makes the model predict `<|nospeech|>`.
|
||||
|
||||
**Validation:** CTC token ids **identical** to PyTorch (108/108 on a clip); text
|
||||
matches `AutoModel` exactly.
|
||||
|
||||
### 5.3 Paraformer (encoder + CIF + decoder, non-autoregressive)
|
||||
|
||||
Pipeline: `fbank → CMVN → encoder → CIF predictor → acoustic embeds [N, 512] →
|
||||
SAN-M decoder (cross-attn to encoder) → argmax → tokens.json`.
|
||||
|
||||
- **CMVN IS applied** here (unlike SenseVoice): `(fbank + shift)·scale`, per-dim
|
||||
(560), from `am.mvn`.
|
||||
- **CIF predictor (runs on host).** Continuous Integrate-and-Fire: a 1-D conv
|
||||
(k=3) + residual + relu + linear → sigmoid → per-frame weight α; then a sequential
|
||||
integrate-and-fire loop emits one acoustic embedding each time the running α-sum
|
||||
crosses 1.0. This both decides the token count and produces the decoder input. It
|
||||
is inherently sequential, so it runs in plain C++ (cheap: ~0.5 G MACs); the
|
||||
encoder and decoder run in ggml.
|
||||
- **SAN-M decoder (ggml).** 16 layers, each: `FFN → FSMN self-attention →
|
||||
cross-attention to the encoder output`. The self-attention is **FSMN-only** (no
|
||||
QK attention); cross-attention has q from the decoder slots and k,v from a fused
|
||||
`linear_k_v` of the encoder output. A 17th `decoders3` layer is FFN-only. The
|
||||
decoder FFN has an internal LayerNorm and the second linear has no bias. The
|
||||
layer ordering (FFN *before* the attention inside the residual) is unusual and is
|
||||
matched exactly.
|
||||
|
||||
**Validation:** decoded text **identical** to `AutoModel`; CIF token count exact
|
||||
(105/105). Encoder cosine 0.997 (residual is the reference's random dither).
|
||||
|
||||
**Gotcha — `am.mvn` has three bracketed blocks.** `[Splice idx]`, `[AddShift=shift]`,
|
||||
`[Rescale=scale]`. The shift/scale are the two 560-length vectors; naively taking
|
||||
the first two blocks grabs `[0]` as the shift and mis-scales everything, which makes
|
||||
CIF emit ~4× too few tokens. Parse by length.
|
||||
|
||||
---
|
||||
|
||||
## 6. GGUF conversion & weight layout
|
||||
|
||||
Each model has an `export_*_gguf.py` that packs weights + architecture metadata into
|
||||
a single GGUF.
|
||||
|
||||
- Tensor names are kept verbatim from the checkpoint (e.g.
|
||||
`encoder.encoders.3.norm1.weight`); the C++ looks them up by name.
|
||||
- **FSMN kernels** are transposed from `(D,1,K)` to `[K,D]` at export so the C++
|
||||
shift-accumulate can take a contiguous per-tap `[D]` vector.
|
||||
- **CMVN** (`am.mvn`) is parsed to `cmvn.shift` / `cmvn.scale` tensors (Paraformer
|
||||
uses them; SenseVoice ships them but the runtime ignores them).
|
||||
- **Quantization.** `--wtype f16` stores the 2-D matmul weights as F16 (norms,
|
||||
biases, FSMN kernels stay f32), halving the encoder GGUF (e.g. 935 → 469 MB) with
|
||||
cosine 0.999999. The Qwen3 LLM uses the standard llama.cpp quantizer (Q8_0 / Q4_K_M).
|
||||
|
||||
| file | model | dtype | size |
|
||||
|---|---|---|---|
|
||||
| funasr-encoder.gguf | Nano | f32 / f16 | 935 / 469 MB |
|
||||
| qwen3-0.6b-q8_0.gguf | Nano LLM | Q8_0 | 805 MB |
|
||||
| sensevoice-small.gguf | SenseVoice | f32 | 936 MB |
|
||||
| paraformer.gguf | Paraformer | f32 | 863 MB |
|
||||
|
||||
---
|
||||
|
||||
## 7. Numerical fidelity & validation methodology
|
||||
|
||||
The port is validated **stage by stage** against the PyTorch reference, using golden
|
||||
dumps (fbank, encoder output, adaptor/CIF output, logits/ids) compared by cosine
|
||||
similarity and max-abs-diff, then end-to-end by transcription text / CER.
|
||||
|
||||
Summary of results (benchmark clip / set):
|
||||
|
||||
| stage | metric |
|
||||
|---|---|
|
||||
| kaldi fbank vs torchaudio | cosine 1.000000 |
|
||||
| SAN-M encoder (full) vs PyTorch | cosine 1.000000, max_abs_diff 5e-3 (f32) |
|
||||
| SenseVoice CTC ids | identical (108/108) |
|
||||
| Paraformer text / token count | identical / 105 = 105 |
|
||||
| Fun-ASR-Nano end-to-end CER (same conditions) | C++ 11.68% vs PyTorch 11.70% (Δ0.02%) |
|
||||
|
||||
**Why not bit-exact tokens everywhere?** Greedy decoding is chaotic: a ~5e-3
|
||||
difference (from ggml-CPU vs torch-GPU matmul summation order) can flip a token on
|
||||
a borderline frame, and over a long sequence the paths diverge — *this also happens
|
||||
between PyTorch's own GPU and CPU*. What is faithful and what we verify is (a) the
|
||||
per-tensor numerics (cosine 1.0) and (b) the **aggregate CER**, which matches the
|
||||
reference under identical conditions.
|
||||
|
||||
**fp16 caution.** The Fun-ASR-Nano adaptor output magnitude (std ≈ 28) can overflow
|
||||
fp16; the audio path is kept in f32 (weights may be f16, activations f32).
|
||||
|
||||
---
|
||||
|
||||
## 8. Performance
|
||||
|
||||
CPU, 8 threads, a 44 s clip:
|
||||
- Encoder (50 layers): ~1.2 s. Paraformer decoder: ~0.5 s.
|
||||
- Fun-ASR-Nano end-to-end (with LLM): ~7 s.
|
||||
- Fully-quantized footprint (f16 encoder + Q8 LLM) ≈ 1.3 GB.
|
||||
|
||||
These are first-correctness numbers; quantizing the encoder and threading/batching
|
||||
the front end are open optimizations.
|
||||
|
||||
---
|
||||
|
||||
## 9. Design decisions & trade-offs
|
||||
|
||||
- **ggml for encoder/decoder, host C++ for CIF.** The neural matmul-heavy parts run
|
||||
in ggml (SIMD, future GPU backends); CIF is a sequential scalar loop with data-
|
||||
dependent control flow, so it is clearer and not slower in plain C++.
|
||||
- **Exact f32 FSMN instead of `ggml_conv_1d_dw`.** Correctness and f32 precision
|
||||
over reusing a flagged, F16-only op (§4.1).
|
||||
- **Prompt as tokens, not a Python embedding table.** The integrated CLI tokenizes
|
||||
the prompt with `llama_tokenize` and lets llama.cpp embed it, so no embedding
|
||||
matrix needs to be shipped or matched (Fun-ASR-Nano).
|
||||
- **f32 by default, f16/Q8 opt-in.** f32 is the faithful default; quantization is a
|
||||
size/latency lever the user opts into. (Interestingly, Q8 on the LLM slightly
|
||||
*helps* greedy stability by regularizing away from repetition loops.)
|
||||
- **Per-model self-contained example dirs.** Mirrors llama.cpp's `examples/` layout
|
||||
so each builds as a drop-in target; the shared code is duplicated rather than
|
||||
factored to keep each example independently buildable.
|
||||
|
||||
---
|
||||
|
||||
## 10. Limitations & roadmap
|
||||
|
||||
- **WAV input** assumes 16 kHz mono PCM16; arbitrary formats / resampling are TODO.
|
||||
- **VAD.** Long audio needs segmentation; today Fun-ASR-Nano uses fixed `--chunk`
|
||||
windows. A real FSMN-VAD front end would close the last ~1.3% CER gap to the
|
||||
production VAD-segmented number and is the highest-value next step.
|
||||
- **Single packaged GGUF** (encoder + adaptor + LLM in one file) and a one-command
|
||||
converter.
|
||||
- **Encoder/decoder quantization** (Q8 via gguf-py quants), streaming, timestamps
|
||||
(Paraformer CIF peaks give alignment; SenseVoice/Nano via CTC).
|
||||
- **Upstream.** The example sources are drop-in for llama.cpp; upstreaming the
|
||||
runtime to ggml-org/llama.cpp (as whisper.cpp-style tools) is a separate track.
|
||||
|
||||
---
|
||||
|
||||
## 11. Reproducing the validation
|
||||
|
||||
Each model dir's README has the build + convert + run quickstart. The export
|
||||
scripts read a standard FunASR checkpoint (`model.pt` + `config.yaml` + `am.mvn` /
|
||||
tokenizer). To reproduce a stage comparison, dump the corresponding PyTorch tensor
|
||||
(`model.encode`, `model.calc_predictor`, …) and compare with cosine / max-abs-diff;
|
||||
the numbers in §7 should reproduce within dither noise.
|
||||
@@ -0,0 +1,71 @@
|
||||
# FunASR on llama.cpp / GGUF
|
||||
|
||||
Run FunASR models on the [llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml
|
||||
stack — **CPU, edge, a single binary, no Python at runtime, quantized weights**.
|
||||
This is to FunASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to
|
||||
Whisper: it lets the models run where there is no GPU and no Python (laptops,
|
||||
phones, edge boxes, embedded C/C++ apps), complementing the PyTorch / ONNX / vLLM
|
||||
paths used for GPU serving.
|
||||
|
||||
## Models
|
||||
|
||||
| model | architecture | runtime | status |
|
||||
|---|---|---|---|
|
||||
| [Fun-ASR-Nano](fun-asr-nano/) | SenseVoice SAN-M encoder + adaptor + Qwen3-0.6B LLM | `llama-funasr-cli` | validated vs PyTorch |
|
||||
| [SenseVoiceSmall](sensevoice/) | SAN-M encoder + CTC | `llama-funasr-sensevoice` | CTC ids identical to PyTorch |
|
||||
| [Paraformer](paraformer/) | SAN-M encoder + CIF predictor + SAN-M decoder (non-autoregressive) | `llama-funasr-paraformer` | text identical to PyTorch |
|
||||
|
||||
All three share the same ggml SAN-M encoder / FSMN / attention primitives and the
|
||||
same kaldi-compatible fbank front end (80-mel, LFR 7/6), so the C++ is consistent
|
||||
across models.
|
||||
|
||||
## How it works
|
||||
|
||||
Each model's neural path is implemented as a ggml graph; the audio front end (kaldi
|
||||
fbank) is plain C++. Weights are converted to GGUF (f32 or f16) with the per-model
|
||||
`export_*_gguf.py` script. For Fun-ASR-Nano the LLM half is a standard Qwen3 GGUF
|
||||
and the audio embeddings are injected into it via `llama_decode`'s embedding input
|
||||
(the llava/mtmd mechanism). See each model's README for the architecture diagram,
|
||||
build/convert/run quickstart, validation numbers, and gotchas.
|
||||
|
||||
## Download pre-built GGUF (fastest — no Python ML env)
|
||||
```bash
|
||||
./download-funasr-model.sh sensevoice # or: paraformer | nano | fsmn-vad
|
||||
```
|
||||
Pre-converted GGUF on Hugging Face: [SenseVoiceSmall-GGUF](https://huggingface.co/FunAudioLLM/SenseVoiceSmall-GGUF) · [Paraformer-GGUF](https://huggingface.co/FunAudioLLM/Paraformer-GGUF) · [Fun-ASR-Nano-GGUF](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-GGUF) · [fsmn-vad-GGUF](https://huggingface.co/FunAudioLLM/fsmn-vad-GGUF). Or convert yourself with `convert-funasr-to-gguf.py`.
|
||||
|
||||
## Build (standalone, CI-friendly)
|
||||
```bash
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release # fetches pinned llama.cpp; static, self-contained
|
||||
cmake --build build -j # -> build/bin/llama-funasr-* (all tools)
|
||||
```
|
||||
## Build (shared)
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
cp -r /path/to/runtime/llama.cpp/funasr-common examples/ # shared audio loader (miniaudio); each example CMake adds ../funasr-common
|
||||
cp -r /path/to/runtime/llama.cpp/<model>/<example-dir> examples/
|
||||
echo 'add_subdirectory(<example-dir>)' >> examples/CMakeLists.txt
|
||||
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build -j --target <target>
|
||||
```
|
||||
The shared **FSMN-VAD** front end builds the same way (`funasr-vad/` + `funasr-common/`,
|
||||
target `llama-funasr-vad`); export weights with `export_vad_gguf.py`. Pass
|
||||
`--vad fsmn-vad.gguf` to any of the three tools for built-in long-audio segmentation.
|
||||
|
||||
## Validation
|
||||
|
||||
Each model was validated against the FunASR PyTorch reference (encoder cosine ≈ 1.0;
|
||||
SenseVoice CTC token ids identical; Paraformer text identical; Fun-ASR-Nano aggregate
|
||||
CER matches PyTorch within 0.02% under identical conditions). See per-model READMEs.
|
||||
|
||||
## Status / notes
|
||||
- Any audio in (wav/mp3/flac, any rate/channels) via the bundled miniaudio loader.
|
||||
- **Built-in FSMN-VAD (`--vad fsmn-vad.gguf`)** segments long audio inside the binary
|
||||
(native ggml, no Python front end); all three tools support it. Bare-binary full-184
|
||||
micro-CER: SenseVoice **8.01** / Paraformer **9.85** / Fun-ASR-Nano **8.30** (see
|
||||
[BENCHMARKS.md](BENCHMARKS.md)). `--chunk` fixed-window remains a simpler fallback.
|
||||
- This adds a new `runtime/llama.cpp/` directory only; no existing code is modified.
|
||||
|
||||
## Further reading
|
||||
|
||||
See [DESIGN.md](DESIGN.md) for the full system design — architecture, the shared SAN-M encoder, GGUF weight format, numerical-fidelity and validation methodology, design trade-offs, and gotchas.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Reproducing the FunASR-vs-whisper.cpp benchmark
|
||||
|
||||
Scripts and method behind [`../BENCHMARKS.md`](../BENCHMARKS.md).
|
||||
|
||||
## Metric (authoritative FunASR口径)
|
||||
- **micro-CER** = `Σ edit_distance / Σ reference_chars` over all files (not a per-file mean).
|
||||
- **normalize_zh**: `re.sub(r'[^\w一-鿿]', '', text).upper()` — drop punctuation/whitespace,
|
||||
keep word chars + CJK, upper-case. (SenseVoice meta tags `<|...|>` are stripped first.)
|
||||
- **RTF** = `Σ compute_time / Σ audio_duration`, model-load time excluded.
|
||||
|
||||
`compute_cer.py` implements exactly this:
|
||||
```bash
|
||||
python compute_cer.py --refs testset.json --hyp_dir <hyps>/ [--time_file <times>.txt]
|
||||
```
|
||||
`testset.json` is a list of `{"id"/"key", "ref", "duration"}`; `<hyps>/{key}.txt` are
|
||||
the transcripts; `<times>.txt` has `key compute_seconds` per line.
|
||||
|
||||
## Producing hypotheses
|
||||
|
||||
FunASR (this runtime), per clip:
|
||||
```bash
|
||||
# SenseVoice / Paraformer: ids -> detok
|
||||
build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a $k.wav > $k.ids
|
||||
python ../sensevoice/detok.py <model>/chn_jpn_yue_eng_ko_spectok.bpe.model $k.ids > $k.txt
|
||||
build/bin/llama-funasr-paraformer -m paraformer.gguf -a $k.wav > $k.ids
|
||||
python ../paraformer/detok_paraformer.py <model>/tokens.json $k.ids > $k.txt
|
||||
# Fun-ASR-Nano: text directly
|
||||
build/bin/llama-funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf -a $k.wav --chunk 15 > $k.txt
|
||||
```
|
||||
Compute time is on each tool's stderr (`encode … s` / `enc … dec … s`).
|
||||
|
||||
whisper.cpp, per clip (forced Chinese, no timestamps):
|
||||
```bash
|
||||
whisper-cli -m models/ggml-<size>.bin -l zh -nt -t 8 $k.wav > $k.txt # 2>stderr has "total time"/"load time"
|
||||
```
|
||||
RTF compute time = `(total − load) ms`.
|
||||
|
||||
## Notes
|
||||
- Run all systems with the **same thread count** (here `-t 8` / 8 threads) for a fair RTF.
|
||||
- Whisper does its own internal 30 s windowing; the FunASR segmentation口径 is documented
|
||||
in `BENCHMARKS.md` (see the methodology/caveats sections).
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compute micro-CER (normalize_zh) and RTF for an ASR system's hypotheses.
|
||||
|
||||
Usage:
|
||||
python compute_cer.py --refs testset.json --hyp_dir <dir of {key}.txt> \
|
||||
[--time_file <key compute_seconds per line>]
|
||||
|
||||
- micro-CER = sum(edit distance) / sum(reference chars), over all files.
|
||||
- normalize_zh(text) = re.sub(r'[^\\w一-鿿]', '', text).upper() (the FunASR口径)
|
||||
- RTF = sum(compute_time) / sum(audio_duration) (model-load excluded)
|
||||
testset.json: list of {"id" or "key", "ref", "duration"}.
|
||||
"""
|
||||
import argparse, json, glob, os, re
|
||||
import numpy as np
|
||||
|
||||
def normalize_zh(s):
|
||||
s = re.sub(r"<\|[^|]*\|>", "", s) # drop SenseVoice meta tags, if any
|
||||
return re.sub(r"[^\w一-鿿]", "", s).upper()
|
||||
|
||||
def edist(r, h):
|
||||
r, h = list(r), list(h)
|
||||
if not r: return len(h)
|
||||
d = np.arange(len(h)+1)
|
||||
for i in range(1, len(r)+1):
|
||||
prev = d[0]; d[0] = i
|
||||
for j in range(1, len(h)+1):
|
||||
cur = d[j]
|
||||
d[j] = min(d[j]+1, d[j-1]+1, prev + (r[i-1] != h[j-1]))
|
||||
prev = cur
|
||||
return int(d[len(h)])
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--refs", required=True)
|
||||
ap.add_argument("--hyp_dir", required=True)
|
||||
ap.add_argument("--time_file", default=None)
|
||||
a = ap.parse_args()
|
||||
refs = {}
|
||||
dur = {}
|
||||
for it in json.load(open(a.refs)):
|
||||
k = f"{it.get('id', it.get('key')):03d}" if isinstance(it.get('id', it.get('key')), int) else str(it.get('key'))
|
||||
refs[k] = it["ref"]; dur[k] = float(it.get("duration", 0))
|
||||
times = {}
|
||||
if a.time_file and os.path.exists(a.time_file):
|
||||
for ln in open(a.time_file):
|
||||
p = ln.split()
|
||||
if len(p) >= 2:
|
||||
try: times[p[0]] = float(p[1])
|
||||
except ValueError: pass
|
||||
E = N = 0; rt = ad = 0.0; n = 0
|
||||
for p in glob.glob(os.path.join(a.hyp_dir, "*.txt")):
|
||||
k = os.path.splitext(os.path.basename(p))[0]
|
||||
if k not in refs: continue
|
||||
h = normalize_zh(open(p).read()); r = normalize_zh(refs[k])
|
||||
E += edist(r, h); N += len(r); n += 1
|
||||
if k in times: rt += times[k]; ad += dur[k]
|
||||
cer = E / max(N, 1) * 100
|
||||
print(f"files={n} micro-CER={cer:.2f}%", end="")
|
||||
if ad > 0: print(f" RTF={rt/ad:.4f} ({ad/rt:.1f}x real-time)")
|
||||
else: print()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""One-step FunASR -> GGUF converter for the llama.cpp runtime.
|
||||
|
||||
Downloads a model checkpoint from Hugging Face (or ModelScope) and exports it to a
|
||||
GGUF the C++ runtime can load — no manual `export_*.py` invocation, mirroring
|
||||
whisper.cpp's `convert` flow.
|
||||
|
||||
python convert-funasr-to-gguf.py sensevoice # -> sensevoice-small.gguf
|
||||
python convert-funasr-to-gguf.py paraformer --wtype f16 # -> paraformer-f16.gguf
|
||||
python convert-funasr-to-gguf.py fsmn-vad # -> fsmn-vad.gguf
|
||||
python convert-funasr-to-gguf.py nano-encoder --wtype f16 # -> funasr-encoder-f16.gguf
|
||||
python convert-funasr-to-gguf.py sensevoice --src modelscope
|
||||
|
||||
Fun-ASR-Nano also needs the Qwen3-0.6B LLM GGUF (a standard llama.cpp conversion of the
|
||||
HF checkpoint) — see `--help` notes; this tool covers the audio encoder/adaptor half.
|
||||
"""
|
||||
import argparse, glob, os, subprocess, sys
|
||||
|
||||
# model key -> (hf repo, modelscope id, export script, needs am.mvn, default out stem)
|
||||
MODELS = {
|
||||
"sensevoice": ("FunAudioLLM/SenseVoiceSmall",
|
||||
"iic/SenseVoiceSmall",
|
||||
"export_sensevoice_gguf.py", True, "sensevoice-small"),
|
||||
"paraformer": ("funasr/paraformer-zh",
|
||||
"iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",
|
||||
"export_paraformer_gguf.py", True, "paraformer"),
|
||||
"fsmn-vad": ("funasr/fsmn-vad",
|
||||
"iic/speech_fsmn_vad_zh-cn-16k-common-pytorch",
|
||||
"export_vad_gguf.py", True, "fsmn-vad"),
|
||||
"nano-encoder": ("FunAudioLLM/Fun-ASR-Nano-2512",
|
||||
"iic/Fun-ASR-Nano",
|
||||
"export_encoder_gguf.py", False, "funasr-encoder"),
|
||||
}
|
||||
|
||||
def find_script(name):
|
||||
"""Locate an export_*.py relative to this file (works in every repo layout)."""
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
hits = glob.glob(os.path.join(here, "**", name), recursive=True)
|
||||
if not hits:
|
||||
sys.exit(f"error: cannot find {name} next to {here}")
|
||||
return hits[0]
|
||||
|
||||
def download(key, src):
|
||||
hf_repo, ms_id, _, _, _ = MODELS[key]
|
||||
try:
|
||||
if src == "modelscope":
|
||||
from modelscope import snapshot_download
|
||||
return snapshot_download(ms_id)
|
||||
from huggingface_hub import snapshot_download
|
||||
return snapshot_download(hf_repo)
|
||||
except ModuleNotFoundError as e:
|
||||
pkg = "modelscope" if src == "modelscope" else "huggingface_hub"
|
||||
sys.exit(f"error: missing {e.name} - install it with: pip install -U {pkg}")
|
||||
|
||||
def pick(d, *names):
|
||||
for n in names:
|
||||
hits = glob.glob(os.path.join(d, "**", n), recursive=True)
|
||||
if hits:
|
||||
return hits[0]
|
||||
return None
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("model", choices=list(MODELS), help="which model to convert")
|
||||
ap.add_argument("--src", choices=["hf", "modelscope"], default="hf", help="checkpoint source")
|
||||
ap.add_argument("--wtype", choices=["f32", "f16"], default="f32",
|
||||
help="matmul weight dtype in the GGUF (norm/bias stay f32)")
|
||||
ap.add_argument("--outdir", default=".", help="output directory")
|
||||
ap.add_argument("--out", default=None, help="output filename (default: <stem>[-f16].gguf)")
|
||||
a = ap.parse_args()
|
||||
|
||||
hf_repo, ms_id, script_name, needs_mvn, stem = MODELS[a.model]
|
||||
print(f"[1/3] downloading {a.model} from {a.src} "
|
||||
f"({ms_id if a.src=='modelscope' else hf_repo}) ...", flush=True)
|
||||
d = download(a.model, a.src)
|
||||
pt = pick(d, "model.pt", "model.pb", "*.pt")
|
||||
mvn = pick(d, "am.mvn") if needs_mvn else None
|
||||
if not pt: sys.exit(f"error: no model.pt under {d}")
|
||||
if needs_mvn and not mvn: sys.exit(f"error: no am.mvn under {d}")
|
||||
|
||||
out = a.out or f"{stem}{'-f16' if a.wtype=='f16' else ''}.gguf"
|
||||
out = os.path.join(a.outdir, out)
|
||||
os.makedirs(a.outdir, exist_ok=True)
|
||||
|
||||
cmd = [sys.executable, find_script(script_name), "--model_pt", pt, "--out", out]
|
||||
if needs_mvn: cmd += ["--mvn", mvn]
|
||||
# export_vad_gguf.py has no --wtype flag (tiny model, always f32)
|
||||
if script_name != "export_vad_gguf.py": cmd += ["--wtype", a.wtype]
|
||||
print(f"[2/3] exporting -> {out}", flush=True)
|
||||
subprocess.run(cmd, check=True)
|
||||
sz = os.path.getsize(out) / 1e6
|
||||
print(f"[3/3] done: {out} ({sz:.1f} MB)")
|
||||
if a.model == "nano-encoder":
|
||||
print("note: Fun-ASR-Nano also needs the Qwen3-0.6B LLM GGUF — convert it with "
|
||||
"llama.cpp's convert_hf_to_gguf.py on the HF checkpoint, then optionally quantize "
|
||||
"(Q8_0 recommended). Run with: llama-funasr-cli --enc <this> -m <qwen3.gguf> -a a.wav")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download pre-converted FunASR GGUF models from Hugging Face — one command, no Python ML env.
|
||||
# ./download-funasr-model.sh sensevoice [outdir]
|
||||
# ./download-funasr-model.sh paraformer | nano | fsmn-vad
|
||||
# ASR models also pull fsmn-vad.gguf (needed for built-in --vad long-audio segmentation).
|
||||
set -euo pipefail
|
||||
usage(){ echo "usage: $0 {sensevoice|paraformer|nano|fsmn-vad} [outdir]"; exit 1; }
|
||||
[ $# -ge 1 ] || usage
|
||||
MODEL="$1"; OUT="${2:-funasr-gguf}"
|
||||
case "$MODEL" in
|
||||
sensevoice) REPO="FunAudioLLM/SenseVoiceSmall-GGUF" ;;
|
||||
paraformer) REPO="FunAudioLLM/Paraformer-GGUF" ;;
|
||||
nano) REPO="FunAudioLLM/Fun-ASR-Nano-GGUF" ;;
|
||||
fsmn-vad|vad) REPO="FunAudioLLM/fsmn-vad-GGUF" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
# huggingface_hub ships `hf` (new CLI); older versions only have `huggingface-cli` (deprecated). Use whichever exists.
|
||||
if command -v hf >/dev/null 2>&1; then HF=hf
|
||||
elif command -v huggingface-cli >/dev/null 2>&1; then HF=huggingface-cli
|
||||
else echo "need the Hugging Face CLI: pip install -U huggingface_hub"; exit 1; fi
|
||||
mkdir -p "$OUT"
|
||||
echo "downloading $REPO ..."; "$HF" download "$REPO" --include "*.gguf" --local-dir "$OUT"
|
||||
if [ "$MODEL" != "fsmn-vad" ] && [ "$MODEL" != "vad" ]; then
|
||||
echo "downloading FSMN-VAD (for --vad) ..."; "$HF" download FunAudioLLM/fsmn-vad-GGUF --include "*.gguf" --local-dir "$OUT"
|
||||
fi
|
||||
echo "done -> $OUT"; ls -1 "$OUT"/*.gguf
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export FSMN-VAD (encoder + CMVN) to GGUF for the ggml C++ runtime."""
|
||||
import argparse, os, re
|
||||
import numpy as np, torch, gguf
|
||||
|
||||
def parse_mvn(path):
|
||||
with open(path) as f: txt=f.read()
|
||||
b=[np.array([float(x) for x in m.split()],np.float32) for m in re.findall(r"\[([^\]]*)\]",txt)]
|
||||
v=[x for x in b if x.size>1]; return v[0], v[1] # shift, scale (both 400-dim)
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser()
|
||||
ap.add_argument("--model_pt",required=True); ap.add_argument("--mvn",required=True); ap.add_argument("--out",required=True)
|
||||
a=ap.parse_args()
|
||||
sd=torch.load(a.model_pt,map_location="cpu"); sd=sd.get("state_dict",sd)
|
||||
w=gguf.GGUFWriter(a.out,"fsmn-vad")
|
||||
w.add_uint32("vad.input_dim",400); w.add_uint32("vad.input_affine_dim",140)
|
||||
w.add_uint32("vad.linear_dim",250); w.add_uint32("vad.proj_dim",128)
|
||||
w.add_uint32("vad.fsmn_layers",4); w.add_uint32("vad.lorder",20)
|
||||
w.add_uint32("vad.output_affine_dim",140); w.add_uint32("vad.output_dim",248)
|
||||
w.add_uint32("vad.n_mels",80); w.add_uint32("vad.lfr_m",5); w.add_uint32("vad.lfr_n",1)
|
||||
shift,scale=parse_mvn(a.mvn); w.add_tensor("cmvn.shift",shift); w.add_tensor("cmvn.scale",scale)
|
||||
n=0
|
||||
for k,v in sd.items():
|
||||
if not k.startswith("encoder."): continue
|
||||
arr=v.detach().to(torch.float32).contiguous().numpy()
|
||||
if k.endswith("conv_left.weight"): # (C,1,lorder,1) -> (lorder,C) tap-major
|
||||
arr=np.ascontiguousarray(arr[:,0,:,0].T)
|
||||
w.add_tensor(k,arr); n+=1
|
||||
print(f"writing {n} tensors (+cmvn) to {a.out}")
|
||||
w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close()
|
||||
print(f"done: {a.out} ({os.path.getsize(a.out)/1e6:.1f} MB)")
|
||||
|
||||
if __name__=="__main__": main()
|
||||
@@ -0,0 +1,144 @@
|
||||
# Fun-ASR-Nano on llama.cpp / GGUF
|
||||
|
||||
Run **Fun-ASR-Nano** entirely on the [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
||||
/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. This is to
|
||||
Fun-ASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to Whisper.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Fun-ASR-Nano normally runs on PyTorch / vLLM (GPU). That is great for a server
|
||||
serving many requests, but it cannot run where there is no GPU and no Python.
|
||||
This runtime ports the model to **ggml + GGUF**, so Fun-ASR-Nano can run:
|
||||
|
||||
- on a laptop / phone / Raspberry Pi / edge box, offline, CPU-only;
|
||||
- embedded directly into a C/C++ application (one static binary);
|
||||
- with quantized weights (Q8 / Q4), shrinking the model to ~1.3 GB total.
|
||||
|
||||
| | vLLM (existing) | this runtime (llama.cpp) |
|
||||
|---|---|---|
|
||||
| target | GPU server, high QPS | CPU / edge / embedded |
|
||||
| deps | Python + CUDA + PyTorch | none (C/C++ single binary) |
|
||||
| weights | HF fp16/bf16 | GGUF, quantized |
|
||||
| best for | online service, batch | offline, on-device |
|
||||
|
||||
## Architecture
|
||||
|
||||
Fun-ASR-Nano = **SenseVoice SAN-M encoder (70 layers) + adaptor + Qwen3-0.6B LLM**.
|
||||
The whole pipeline runs in C++:
|
||||
|
||||
```
|
||||
audio.wav (16k mono)
|
||||
│ kaldi 80-mel fbank + LFR (C++)
|
||||
▼
|
||||
features [T, 560]
|
||||
│ SAN-M encoder + adaptor (ggml) ── funasr-encoder.gguf
|
||||
▼
|
||||
audio embeds [T', 1024]
|
||||
│ keep first fake_token_len frames (low-frame-rate)
|
||||
▼
|
||||
[ prefix tokens | audio embeds | suffix tokens ]
|
||||
│ Qwen3-0.6B, embeds injected via llama_decode (llava/mtmd style) ── qwen3-0.6b.gguf
|
||||
▼
|
||||
transcription
|
||||
```
|
||||
|
||||
The audio embeddings are fed into the LLM through `llama_decode`'s embedding-input
|
||||
path — exactly how llava/mtmd inject vision embeddings.
|
||||
|
||||
## Quickstart
|
||||
|
||||
**1. Build** (drop the examples into a llama.cpp checkout):
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
cp -r /path/to/runtime/llama.cpp/funasr-cli examples/
|
||||
echo 'add_subdirectory(funasr-cli)' >> examples/CMakeLists.txt
|
||||
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build -j --target llama-funasr-cli
|
||||
```
|
||||
|
||||
**2. Convert weights to GGUF** (one-time; needs the checkpoint, e.g.
|
||||
`FunAudioLLM/Fun-ASR-Nano-2512`):
|
||||
```bash
|
||||
# LLM half — Qwen3-0.6B is natively supported by llama.cpp
|
||||
python llama.cpp/convert_hf_to_gguf.py <model>/Qwen3-0.6B-vllm \
|
||||
--outfile qwen3-0.6b-f32.gguf --outtype f32
|
||||
build/bin/llama-quantize qwen3-0.6b-f32.gguf qwen3-0.6b-q8_0.gguf Q8_0 # smaller, recommended
|
||||
|
||||
# audio half — SenseVoice encoder + adaptor
|
||||
python runtime/llama.cpp/export_encoder_gguf.py \
|
||||
--model_pt <model>/model.pt --out funasr-encoder.gguf # f32, 935 MB
|
||||
python runtime/llama.cpp/export_encoder_gguf.py \
|
||||
--model_pt <model>/model.pt --out funasr-encoder-f16.gguf --wtype f16 # 469 MB
|
||||
```
|
||||
|
||||
**3. Transcribe:**
|
||||
```bash
|
||||
build/bin/llama-funasr-cli \
|
||||
--enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf \
|
||||
-a audio.wav --chunk 15
|
||||
```
|
||||
Expected output (one of the benchmark clips):
|
||||
```
|
||||
我想问我在滨海新区有房我一直没有照顾孩子但是我想要抚养权...你觉得这是正常的想法吗
|
||||
[done] 7.40s ; chunk=15s
|
||||
```
|
||||
|
||||
## Models & sizes
|
||||
|
||||
| file | dtype | size |
|
||||
|---|---|---|
|
||||
| funasr-encoder.gguf | f32 | 935 MB |
|
||||
| funasr-encoder-f16.gguf | f16 (matmul weights) | 469 MB |
|
||||
| qwen3-0.6b-f32.gguf | f32 | 3.0 GB |
|
||||
| qwen3-0.6b-q8_0.gguf | Q8_0 | 805 MB |
|
||||
| qwen3-0.6b-q4km.gguf | Q4_K_M | 484 MB |
|
||||
|
||||
Fully-quantized config (f16 encoder + Q8 LLM) ≈ **1.3 GB**, edge-friendly.
|
||||
|
||||
## Accuracy & validation
|
||||
|
||||
Validated against the PyTorch reference on the 184-file benchmark:
|
||||
|
||||
- **Encoder + adaptor (ggml) vs PyTorch:** cosine **1.000000**, max_abs_diff **5e-3** (f32).
|
||||
- **kaldi fbank (C++) vs torchaudio:** cosine **1.000000**.
|
||||
- **End-to-end CER, identical conditions (f32 LLM, 15 s chunking):**
|
||||
C++ macro 17.41% / micro 11.68% vs PyTorch macro 17.42% / micro 11.70%
|
||||
→ aggregate CER matches to **0.02%**; the port is faithful.
|
||||
- Best practical config (Q8 LLM + 15 s chunking): **micro-CER 9.51%** (production
|
||||
VAD-segmented reference is ~8.2%; the gap is fixed-window vs VAD, not the port).
|
||||
|
||||
## Tips & gotchas
|
||||
|
||||
- **Use `--chunk 15`** for long audio. Decoding a whole 60 s clip in one segment is
|
||||
out-of-distribution and makes greedy decoding loop; 15 s windows fix it
|
||||
(micro-CER 29% → 9.5%).
|
||||
- **Low-frame-rate truncation** is required: only the first `fake_token_len`
|
||||
adaptor frames are real audio tokens. The CLI does this automatically; feeding
|
||||
all frames makes the LLM repeat.
|
||||
- **Use bf16/fp32, avoid fp16 for the audio path** — the adaptor output has large
|
||||
magnitude (std ≈ 28, |max| ≈ 1187); fp16 can overflow. The GGUFs here are f32/f16
|
||||
weights with f32 activations, which is safe.
|
||||
- **WAV input** currently assumes 16 kHz mono PCM16. Resample first if needed.
|
||||
- Q8 quantization slightly *helps* greedy stability (quant noise regularizes away
|
||||
from repetition loops), so Q8 is a good default.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- FSMN depthwise memory is an exact f32 shift-accumulate (avoids the F16-only,
|
||||
upstream-flagged `ggml_conv_1d_dw`).
|
||||
- LayerNorm eps = 1e-5; sinusoidal position encoding depth = input feature dim (560),
|
||||
positions start at 1; encoder input pre-scaled by sqrt(512).
|
||||
- Prompt is fed as tokens via `llama_tokenize(parse_special=true)` (prefix = 18
|
||||
tokens, matching the HF tokenizer), so no Python embedding table is needed.
|
||||
|
||||
## Files
|
||||
```
|
||||
funasr-cli/ integrated binary: WAV → transcription
|
||||
funasr-encoder/ encoder+adaptor only (ggml) — validation/debugging
|
||||
funasr-embd/ LLM decode from precomputed embeds — validation/debugging
|
||||
export_encoder_gguf.py export the audio encoder + adaptor to GGUF
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
- True FSMN-VAD segmentation (replace fixed windows; closes the last ~1.3% CER).
|
||||
- Arbitrary WAV formats / resampling; encoder Q8 quantization; single packaged GGUF.
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export Fun-ASR-Nano audio encoder + adaptor weights to a GGUF file.
|
||||
|
||||
Packs all `audio_encoder.*` and `audio_adaptor.*` tensors (bf16 -> f32) plus
|
||||
architecture metadata into funasr-encoder.gguf, for the ggml C++ forward pass.
|
||||
Tensor names are kept verbatim (e.g. audio_encoder.encoders.3.norm1.weight) so
|
||||
the C++ side can look them up directly.
|
||||
"""
|
||||
import argparse, os
|
||||
import numpy as np
|
||||
import torch
|
||||
import gguf
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model_pt", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--wtype", default="f32", choices=["f32", "f16", "q8_0"],
|
||||
help="dtype for 2D Linear (matmul) weights; norms/bias/fsmn stay f32")
|
||||
args = ap.parse_args()
|
||||
|
||||
sd = torch.load(args.model_pt, map_location="cpu")
|
||||
sd = sd.get("state_dict", sd)
|
||||
|
||||
w = gguf.GGUFWriter(args.out, "funasr-sensevoice-encoder")
|
||||
|
||||
# --- architecture metadata (from config.yaml) ---
|
||||
w.add_uint32("funasr.enc.input_size", 560) # lfr_m(7) * n_mels(80)
|
||||
w.add_uint32("funasr.enc.output_size", 512)
|
||||
w.add_uint32("funasr.enc.attention_heads", 4)
|
||||
w.add_uint32("funasr.enc.linear_units", 2048)
|
||||
w.add_uint32("funasr.enc.num_blocks", 50) # encoders0(1) + encoders(49)
|
||||
w.add_uint32("funasr.enc.tp_blocks", 20)
|
||||
w.add_uint32("funasr.enc.kernel_size", 11)
|
||||
w.add_uint32("funasr.enc.sanm_shfit", 0)
|
||||
w.add_uint32("funasr.adp.llm_dim", 1024)
|
||||
w.add_uint32("funasr.adp.encoder_dim", 512)
|
||||
w.add_uint32("funasr.adp.ffn_dim", 2048)
|
||||
w.add_uint32("funasr.adp.n_layer", 2)
|
||||
w.add_uint32("funasr.adp.attention_heads", 8)
|
||||
w.add_uint32("funasr.adp.downsample_rate", 1)
|
||||
w.add_uint32("funasr.frontend.n_mels", 80)
|
||||
w.add_uint32("funasr.frontend.lfr_m", 7)
|
||||
w.add_uint32("funasr.frontend.lfr_n", 6)
|
||||
|
||||
n = 0
|
||||
for k, v in sd.items():
|
||||
if not (k.startswith("audio_encoder.") or k.startswith("audio_adaptor.")):
|
||||
continue
|
||||
arr = v.detach().to(torch.float32).contiguous().numpy()
|
||||
# FSMN depthwise kernel: store as (K, D) so the C++ side can slice a
|
||||
# contiguous per-tap [D] vector and do an exact f32 shift-accumulate
|
||||
# (avoids the F16-only ggml_conv_1d_dw path).
|
||||
if k.endswith("fsmn_block.weight"): # (D, 1, K) -> (K, D)
|
||||
arr = np.ascontiguousarray(arr[:, 0, :].T)
|
||||
# matmul (Linear) weights -> optional f16; norms/biases/fsmn stay f32
|
||||
elif args.wtype == "f16" and arr.ndim == 2 and "norm" not in k:
|
||||
arr = arr.astype(np.float16)
|
||||
if args.wtype == "q8_0" and arr.ndim == 2 and "norm" not in k and "fsmn_block" not in k and arr.shape[1] % 32 == 0:
|
||||
from gguf import quants as _q, GGMLQuantizationType as _QT
|
||||
w.add_tensor(k, _q.quantize(arr, _QT.Q8_0), raw_dtype=_QT.Q8_0)
|
||||
else:
|
||||
w.add_tensor(k, arr)
|
||||
n += 1
|
||||
print(f"writing {n} tensors to {args.out}")
|
||||
|
||||
w.write_header_to_file()
|
||||
w.write_kv_data_to_file()
|
||||
w.write_tensors_to_file()
|
||||
w.close()
|
||||
print(f"done: {args.out} ({os.path.getsize(args.out)/1e6:.1f} MB)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,6 @@
|
||||
set(TARGET llama-funasr-cli)
|
||||
add_executable(${TARGET} funasr-cli.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_include_directories(${TARGET} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../funasr-common)
|
||||
target_link_libraries(${TARGET} PRIVATE llama ggml ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,251 @@
|
||||
// funasr-cli: end-to-end Fun-ASR-Nano in C++ on the llama.cpp / ggml stack.
|
||||
//
|
||||
// wav(16k mono) -> kaldi fbank -> SAN-M encoder + adaptor (ggml) ->
|
||||
// low-frame-rate truncation -> [prefix tokens | audio embeds | suffix tokens]
|
||||
// -> Qwen3 LLM (llama.cpp) -> transcription.
|
||||
//
|
||||
// This is the whisper.cpp-style single-binary path: no Python at runtime.
|
||||
//
|
||||
// funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b.gguf -a audio.wav
|
||||
|
||||
#include "ggml.h"
|
||||
#include "ggml-cpu.h"
|
||||
#include "ggml-alloc.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "gguf.h"
|
||||
#include "llama.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
// any audio (wav/mp3/flac, any rate/channels) -> 16 kHz mono f32, via miniaudio
|
||||
#define FUNASR_AUDIO_IMPLEMENTATION
|
||||
#include "funasr_audio.h"
|
||||
// built-in FSMN-VAD front end (single-binary --vad segmentation)
|
||||
#include "funasr_vad.h"
|
||||
#include <utility>
|
||||
|
||||
// ======================= kaldi fbank + LFR =======================
|
||||
static const int FS=16000, WINLEN=400, SHIFT=160, NFFT=512, NMEL=80, LFR_M=7, LFR_N=6;
|
||||
static const float PREEMPH=0.97f, LOWF=20.0f, HIGHF=8000.0f;
|
||||
static inline float mel(float f){ return 1127.0f*logf(1.0f+f/700.0f); }
|
||||
static void fft(std::vector<float>&re,std::vector<float>&im,int n){
|
||||
for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}
|
||||
for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);
|
||||
for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){
|
||||
float ur=re[i+k],ui=im[i+k];float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;
|
||||
re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;
|
||||
float n2=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=n2;}}}
|
||||
}
|
||||
// returns [T x 560] row-major, sets T
|
||||
static std::vector<float> compute_fbank(std::vector<float> wav, int & T_out) {
|
||||
for (auto & v : wav) v *= 32768.0f;
|
||||
std::vector<float> win(WINLEN);
|
||||
for (int i=0;i<WINLEN;i++) win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));
|
||||
const int NBIN=NFFT/2+1; float bw=(float)FS/NFFT, ml=mel(LOWF), mh=mel(HIGHF), dm=(mh-ml)/(NMEL+1);
|
||||
std::vector<std::vector<float>> fb(NMEL, std::vector<float>(NBIN,0.0f));
|
||||
for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;
|
||||
for(int k=0;k<NBIN;k++){float mf=mel(bw*k); if(mf>L&&mf<R) fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}
|
||||
int N=wav.size(); int T=(N-WINLEN)/SHIFT+1;
|
||||
std::vector<std::vector<float>> feat(T, std::vector<float>(NMEL));
|
||||
std::vector<float> re(NFFT),im(NFFT),fr(WINLEN);
|
||||
const float fl=1.1920929e-07f;
|
||||
for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;
|
||||
double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;
|
||||
for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;
|
||||
for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];
|
||||
for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}
|
||||
fft(re,im,NFFT);
|
||||
for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NBIN;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);
|
||||
feat[t][m]=logf(e>fl?e:fl);}}
|
||||
// LFR
|
||||
const int pad=(LFR_M-1)/2; int T_lfr=(T+LFR_N-1)/LFR_N;
|
||||
std::vector<std::vector<float>> pd; pd.reserve(T+pad+LFR_M);
|
||||
for(int i=0;i<pad;i++)pd.push_back(feat[0]);
|
||||
for(int t=0;t<T;t++)pd.push_back(feat[t]);
|
||||
while((int)pd.size()<(T_lfr-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);
|
||||
int D=LFR_M*NMEL; std::vector<float> out((size_t)T_lfr*D);
|
||||
for(int i=0;i<T_lfr;i++)for(int j=0;j<LFR_M;j++)
|
||||
memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));
|
||||
T_out=T_lfr; return out;
|
||||
}
|
||||
|
||||
// ======================= ggml SAN-M encoder + adaptor =======================
|
||||
struct cfg { int d_model=512,n_head=4,num_blocks=50,tp_blocks=20,kernel=11,adp_llm=1024,adp_layers=2,adp_head=8; };
|
||||
struct enc_model { cfg c; ggml_context*ctx_w=nullptr; std::map<std::string,ggml_tensor*> t;
|
||||
ggml_tensor* g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;} };
|
||||
static const float LN_EPS=1e-5f;
|
||||
static bool load_enc(const char*p, enc_model&m){
|
||||
gguf_init_params gp={false,&m.ctx_w}; gguf_context*g=gguf_init_from_file(p,gp); if(!g)return false;
|
||||
auto rd=[&](const char*k,int d){int i=gguf_find_key(g,k);return i<0?d:(int)gguf_get_val_u32(g,i);};
|
||||
m.c.d_model=rd("funasr.enc.output_size",512); m.c.n_head=rd("funasr.enc.attention_heads",4);
|
||||
m.c.num_blocks=rd("funasr.enc.num_blocks",50); m.c.tp_blocks=rd("funasr.enc.tp_blocks",20);
|
||||
m.c.kernel=rd("funasr.enc.kernel_size",11); m.c.adp_llm=rd("funasr.adp.llm_dim",1024);
|
||||
m.c.adp_layers=rd("funasr.adp.n_layer",2); m.c.adp_head=rd("funasr.adp.attention_heads",8);
|
||||
int n=gguf_get_n_tensors(g); for(int i=0;i<n;i++){const char*nm=gguf_get_tensor_name(g,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}
|
||||
gguf_free(g); return true;
|
||||
}
|
||||
static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}
|
||||
static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}
|
||||
static ggml_tensor* sanm_attn(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){
|
||||
const int D=m.c.d_model,H=m.c.n_head,dk=D/H,K=m.c.kernel;
|
||||
ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x); size_t nb1=qkv->nb[1];
|
||||
ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));
|
||||
ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));
|
||||
ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));
|
||||
const int pad=(K-1)/2; ggml_tensor*fk=m.g(p+"fsmn_block.weight");
|
||||
ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0); ggml_tensor*fsmn=v;
|
||||
for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);
|
||||
auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]); fsmn=ggml_add(c,fsmn,ggml_mul(c,ggml_cont(c,sl),wj));}
|
||||
q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3); k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);
|
||||
ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));
|
||||
ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
|
||||
ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
|
||||
return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fsmn);
|
||||
}
|
||||
static ggml_tensor* sanm_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T,bool res){
|
||||
auto r=x; auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
|
||||
auto sa=sanm_attn(c,m,p+"self_attn.",h,T); x=res?ggml_add(c,r,sa):sa; r=x;
|
||||
h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
|
||||
h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);
|
||||
h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);
|
||||
}
|
||||
static ggml_tensor* adp_layer(ggml_context*c,enc_model&m,const std::string&p,ggml_tensor*x,int T){
|
||||
const int D=m.c.adp_llm,H=m.c.adp_head,dk=D/H; auto r=x;
|
||||
auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
|
||||
auto q=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_q.weight"),m.g(p+"self_attn.linear_q.bias"),h),dk,H,T),0,2,1,3);
|
||||
auto k=ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_k.weight"),m.g(p+"self_attn.linear_k.bias"),h),dk,H,T),0,2,1,3);
|
||||
auto vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,lin(c,m.g(p+"self_attn.linear_v.weight"),m.g(p+"self_attn.linear_v.bias"),h),dk,H,T),1,2,0,3));
|
||||
auto kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
|
||||
auto o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
|
||||
x=ggml_add(c,r,lin(c,m.g(p+"self_attn.linear_out.weight"),m.g(p+"self_attn.linear_out.bias"),o)); r=x;
|
||||
h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
|
||||
h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);
|
||||
h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);
|
||||
}
|
||||
static void add_posenc(std::vector<float>&x,int T,int depth){
|
||||
double inc=log(10000.0)/(depth/2.0-1.0);
|
||||
for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;
|
||||
x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}
|
||||
}
|
||||
// fbank [T x F] -> adaptor out [T x adp_llm] row-major
|
||||
static std::vector<float> run_encoder(enc_model&m,std::vector<float> fbank,int T,int F,int&Dout){
|
||||
float sc=sqrtf((float)m.c.d_model); for(auto&v:fbank)v*=sc; add_posenc(fbank,T,F);
|
||||
ggml_backend_t be=ggml_backend_cpu_init();
|
||||
ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);
|
||||
ggml_tensor*inp=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,T); ggml_set_input(inp);
|
||||
ggml_tensor*x=sanm_layer(c,m,"audio_encoder.encoders0.0.",inp,T,false);
|
||||
for(int i=0;i<m.c.num_blocks-1;i++) x=sanm_layer(c,m,"audio_encoder.encoders."+std::to_string(i)+".",x,T,true);
|
||||
x=lnorm(c,x,m.g("audio_encoder.after_norm.weight"),m.g("audio_encoder.after_norm.bias"));
|
||||
for(int i=0;i<m.c.tp_blocks;i++) x=sanm_layer(c,m,"audio_encoder.tp_encoders."+std::to_string(i)+".",x,T,true);
|
||||
x=lnorm(c,x,m.g("audio_encoder.tp_norm.weight"),m.g("audio_encoder.tp_norm.bias"));
|
||||
x=lin(c,m.g("audio_adaptor.linear1.weight"),m.g("audio_adaptor.linear1.bias"),x); x=ggml_relu(c,x);
|
||||
x=lin(c,m.g("audio_adaptor.linear2.weight"),m.g("audio_adaptor.linear2.bias"),x);
|
||||
for(int i=0;i<m.c.adp_layers;i++) x=adp_layer(c,m,"audio_adaptor.blocks."+std::to_string(i)+".",x,T);
|
||||
ggml_set_output(x);
|
||||
ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false); ggml_build_forward_expand(gf,x);
|
||||
ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);
|
||||
ggml_backend_tensor_set(inp,fbank.data(),0,ggml_nbytes(inp));
|
||||
ggml_backend_cpu_set_n_threads(be,8); ggml_backend_graph_compute(be,gf);
|
||||
Dout=(int)x->ne[0]; std::vector<float> out((size_t)Dout*T); ggml_backend_tensor_get(x,out.data(),0,ggml_nbytes(x));
|
||||
ggml_gallocr_free(ga); ggml_free(c); ggml_backend_free(be); return out;
|
||||
}
|
||||
|
||||
// ======================= LLM (llama.cpp) =======================
|
||||
static int decode_batch(llama_context*ctx,int n,llama_token*tok,float*embd,int n_embd,int&n_past,bool last_logits){
|
||||
std::vector<llama_pos> pos(n); std::vector<int32_t> nsid(n,1);
|
||||
std::vector<llama_seq_id> s0(1,0); std::vector<llama_seq_id*> sid(n); std::vector<int8_t> lg(n,0);
|
||||
for(int i=0;i<n;i++){pos[i]=n_past+i;sid[i]=s0.data();}
|
||||
if(last_logits) lg[n-1]=1;
|
||||
llama_batch b={n,tok,embd,pos.data(),nsid.data(),sid.data(),lg.data()};
|
||||
int r=llama_decode(ctx,b); n_past+=n; return r;
|
||||
}
|
||||
|
||||
int main(int argc,char**argv){
|
||||
std::string enc_path,llm_path,wav_path,vad_path; int npred=512; double chunk_sec=0; float rep=1.0f;
|
||||
int vad_maxseg=30000;
|
||||
for(int i=1;i<argc;i++){
|
||||
if(!strcmp(argv[i],"--enc")&&i+1<argc)enc_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"-m")&&i+1<argc)llm_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"-n")&&i+1<argc)npred=atoi(argv[++i]);
|
||||
else if(!strcmp(argv[i],"--chunk")&&i+1<argc)chunk_sec=atof(argv[++i]);
|
||||
else if(!strcmp(argv[i],"--vad")&&i+1<argc)vad_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"--vad-maxseg")&&i+1<argc)vad_maxseg=atoi(argv[++i]);
|
||||
else if(!strcmp(argv[i],"--rep")&&i+1<argc)rep=atof(argv[++i]);
|
||||
else {fprintf(stderr,"usage: %s --enc enc.gguf -m llm.gguf -a audio.wav [-n npred] [--chunk sec] [--vad fsmn-vad.gguf [--vad-maxseg ms]]\n",argv[0]);return 1;}
|
||||
}
|
||||
if(enc_path.empty()||llm_path.empty()||wav_path.empty()){fprintf(stderr,"missing args\n");return 1;}
|
||||
|
||||
std::vector<float> wav;
|
||||
if(!funasr_load_audio_16k_mono(wav_path.c_str(),wav)){fprintf(stderr,"failed to read audio\n");return 1;}
|
||||
int64_t t0=ggml_time_us();
|
||||
|
||||
enc_model em; if(!load_enc(enc_path.c_str(),em))return 1;
|
||||
ggml_backend_load_all();
|
||||
llama_model_params mp=llama_model_default_params(); mp.n_gpu_layers=0;
|
||||
llama_model*model=llama_model_load_from_file(llm_path.c_str(),mp); if(!model)return 1;
|
||||
const llama_vocab*vocab=llama_model_get_vocab(model);
|
||||
llama_context_params cp=llama_context_default_params();
|
||||
cp.n_ctx=2048; cp.n_batch=2048; cp.n_ubatch=2048;
|
||||
llama_context*ctx=llama_init_from_model(model,cp);
|
||||
if(!ctx){fprintf(stderr,"failed to create llama context\n");llama_model_free(model);return 1;}
|
||||
auto sp=llama_sampler_chain_default_params(); llama_sampler*smpl=llama_sampler_chain_init(sp);
|
||||
if(rep!=1.0f) llama_sampler_chain_add(smpl,llama_sampler_init_penalties(256,rep,0.0f,0.0f));
|
||||
llama_sampler_chain_add(smpl,llama_sampler_init_greedy());
|
||||
|
||||
const char*prefix="<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n语音转写：";
|
||||
const char*suffix="<|im_end|>\n<|im_start|>assistant\n";
|
||||
auto tokenize=[&](const char*s){int n=-llama_tokenize(vocab,s,strlen(s),nullptr,0,false,true);
|
||||
std::vector<llama_token> v(n); llama_tokenize(vocab,s,strlen(s),v.data(),n,false,true); return v;};
|
||||
auto pre=tokenize(prefix); auto suf=tokenize(suffix);
|
||||
|
||||
// Build the list of [offset,len] windows to transcribe (in samples).
|
||||
// --vad : FSMN-VAD speech segments (single-binary front end, replaces fixed chunking)
|
||||
// --chunk sec : fixed-size chunks ; otherwise the whole file in one window
|
||||
std::vector<std::pair<int,int>> wins; // {sample offset, sample len}
|
||||
if(!vad_path.empty()){
|
||||
std::vector<std::pair<int,int>> segs; // ms
|
||||
if(!funasr_vad_segments(vad_path,wav,vad_maxseg,segs)){fprintf(stderr,"vad failed\n");return 1;}
|
||||
for(auto&s:segs){ int off=(int)((int64_t)s.first*16000/1000), end=(int)((int64_t)s.second*16000/1000);
|
||||
if(end>(int)wav.size())end=wav.size(); if(end-off>0) wins.push_back({off,end-off}); }
|
||||
fprintf(stderr,"[vad] %zu segments\n",wins.size());
|
||||
} else {
|
||||
int chunk_n = chunk_sec > 0 ? std::max(1, (int)(chunk_sec*16000)) : (int)wav.size();
|
||||
for(size_t off=0; off<wav.size(); off+=chunk_n) wins.push_back({(int)off,(int)std::min((size_t)chunk_n,wav.size()-off)});
|
||||
}
|
||||
std::string full;
|
||||
for (auto& w : wins) {
|
||||
int off = w.first, len = w.second;
|
||||
if (len < WINLEN) continue; // too short for one frame
|
||||
std::vector<float> seg(wav.begin()+off, wav.begin()+off+len);
|
||||
int T=0; auto fbank=compute_fbank(seg,T);
|
||||
int D=0; auto adp=run_encoder(em,fbank,T,560,D);
|
||||
int ol=1+(T-3+2)/2; ol=1+(ol-3+2)/2; int n_aud=(ol-1)/2+1;
|
||||
|
||||
llama_memory_clear(llama_get_memory(ctx), true); // fresh context per chunk
|
||||
int n_past=0;
|
||||
decode_batch(ctx,pre.size(),pre.data(),nullptr,0,n_past,false);
|
||||
decode_batch(ctx,n_aud,nullptr,adp.data(),D,n_past,false);
|
||||
decode_batch(ctx,suf.size(),suf.data(),nullptr,0,n_past,true);
|
||||
llama_token tk=llama_sampler_sample(smpl,ctx,-1);
|
||||
for(int i=0;i<npred;i++){
|
||||
if(llama_vocab_is_eog(vocab,tk))break;
|
||||
char buf[256]; int k=llama_token_to_piece(vocab,tk,buf,sizeof(buf),0,true);
|
||||
if(k>0) full.append(buf,k);
|
||||
decode_batch(ctx,1,&tk,nullptr,0,n_past,true);
|
||||
tk=llama_sampler_sample(smpl,ctx,-1);
|
||||
}
|
||||
}
|
||||
printf("%s\n", full.c_str());
|
||||
int64_t t2=ggml_time_us();
|
||||
fprintf(stderr,"[done] %.2fs ; chunk=%.0fs\n",(t2-t0)/1e6, chunk_sec);
|
||||
llama_sampler_free(smpl); llama_free(ctx); llama_model_free(model);
|
||||
if(em.ctx_w) ggml_free(em.ctx_w);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
set(TARGET llama-funasr-embd)
|
||||
add_executable(${TARGET} funasr-embd.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_link_libraries(${TARGET} PRIVATE llama ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,141 @@
|
||||
// funasr-embd: decode Fun-ASR-Nano audio embeddings through a Qwen3 GGUF.
|
||||
//
|
||||
// Reads an inputs_embeds matrix (produced by the FunASR audio encoder+adaptor,
|
||||
// concatenated with the text prompt embeddings) and feeds it directly to the
|
||||
// LLM via llama_decode's embedding input path -- the same mechanism llava/mtmd
|
||||
// use to inject vision embeddings. This bridges FunASR's audio frontend to the
|
||||
// llama.cpp / GGUF ecosystem.
|
||||
//
|
||||
// embeds.bin format: int32 n_tokens, int32 n_embd, then n_tokens*n_embd float32
|
||||
// (row-major). n_embd must equal the model's input embedding dim.
|
||||
|
||||
#include "llama.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
static void print_usage(char ** argv) {
|
||||
printf("\nusage: %s -m model.gguf -e embeds.bin [-n n_predict] [-ngl n_gpu_layers]\n\n", argv[0]);
|
||||
}
|
||||
|
||||
// read embeds.bin -> (n_tokens, n_embd, data)
|
||||
static bool read_embeds(const std::string & path, int & n_tokens, int & n_embd, std::vector<float> & data) {
|
||||
FILE * f = fopen(path.c_str(), "rb");
|
||||
if (!f) { fprintf(stderr, "error: cannot open %s\n", path.c_str()); return false; }
|
||||
int32_t hdr[2];
|
||||
if (fread(hdr, sizeof(int32_t), 2, f) != 2) { fclose(f); return false; }
|
||||
n_tokens = hdr[0];
|
||||
n_embd = hdr[1];
|
||||
if (n_tokens <= 0 || n_embd <= 0) { fclose(f); return false; }
|
||||
data.resize((size_t) n_tokens * n_embd);
|
||||
size_t got = fread(data.data(), sizeof(float), data.size(), f);
|
||||
fclose(f);
|
||||
if (got != data.size()) {
|
||||
fprintf(stderr, "error: short read (%zu/%zu floats)\n", got, data.size());
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
int main(int argc, char ** argv) {
|
||||
std::string model_path, embeds_path;
|
||||
int n_predict = 512;
|
||||
int ngl = 0; // CPU by default; the whole point is CPU/edge
|
||||
|
||||
for (int i = 1; i < argc; i++) {
|
||||
if (!strcmp(argv[i], "-m") && i + 1 < argc) model_path = argv[++i];
|
||||
else if (!strcmp(argv[i], "-e") && i + 1 < argc) embeds_path = argv[++i];
|
||||
else if (!strcmp(argv[i], "-n") && i + 1 < argc) n_predict = std::stoi(argv[++i]);
|
||||
else if (!strcmp(argv[i], "-ngl") && i + 1 < argc) ngl = std::stoi(argv[++i]);
|
||||
else { print_usage(argv); return 1; }
|
||||
}
|
||||
if (model_path.empty() || embeds_path.empty()) { print_usage(argv); return 1; }
|
||||
|
||||
int n_tokens = 0, n_embd_in = 0;
|
||||
std::vector<float> embd;
|
||||
if (!read_embeds(embeds_path, n_tokens, n_embd_in, embd)) return 1;
|
||||
fprintf(stderr, "loaded embeds: n_tokens=%d n_embd=%d\n", n_tokens, n_embd_in);
|
||||
|
||||
ggml_backend_load_all();
|
||||
|
||||
llama_model_params mparams = llama_model_default_params();
|
||||
mparams.n_gpu_layers = ngl;
|
||||
llama_model * model = llama_model_load_from_file(model_path.c_str(), mparams);
|
||||
if (!model) { fprintf(stderr, "error: unable to load model\n"); return 1; }
|
||||
|
||||
const llama_vocab * vocab = llama_model_get_vocab(model);
|
||||
const int n_embd_model = llama_model_n_embd_inp(model);
|
||||
if (n_embd_in != n_embd_model) {
|
||||
fprintf(stderr, "error: embd dim %d != model input embd dim %d\n", n_embd_in, n_embd_model);
|
||||
llama_model_free(model);
|
||||
return 1;
|
||||
}
|
||||
|
||||
llama_context_params cparams = llama_context_default_params();
|
||||
cparams.n_ctx = n_tokens + n_predict + 8;
|
||||
cparams.n_batch = n_tokens + 8; // process the whole embd prompt in one ubatch
|
||||
cparams.n_ubatch = n_tokens + 8;
|
||||
cparams.no_perf = false;
|
||||
llama_context * ctx = llama_init_from_model(model, cparams);
|
||||
if (!ctx) { fprintf(stderr, "error: failed to create context\n"); llama_model_free(model); return 1; }
|
||||
|
||||
auto sparams = llama_sampler_chain_default_params();
|
||||
llama_sampler * smpl = llama_sampler_chain_init(sparams);
|
||||
llama_sampler_chain_add(smpl, llama_sampler_init_greedy());
|
||||
|
||||
// --- decode the embedding prompt (causal, single sequence, positions 0..n-1) ---
|
||||
std::vector<llama_pos> pos(n_tokens);
|
||||
std::vector<int32_t> n_seq_id(n_tokens, 1);
|
||||
std::vector<llama_seq_id> seq_id_0(1, 0);
|
||||
std::vector<llama_seq_id *> seq_id(n_tokens);
|
||||
std::vector<int8_t> logits(n_tokens, 0);
|
||||
for (int i = 0; i < n_tokens; i++) { pos[i] = i; seq_id[i] = seq_id_0.data(); }
|
||||
logits[n_tokens - 1] = 1; // only need logits for the last position
|
||||
|
||||
llama_batch batch = {
|
||||
/*n_tokens =*/ n_tokens,
|
||||
/*token =*/ nullptr,
|
||||
/*embd =*/ embd.data(),
|
||||
/*pos =*/ pos.data(),
|
||||
/*n_seq_id =*/ n_seq_id.data(),
|
||||
/*seq_id =*/ seq_id.data(),
|
||||
/*logits =*/ logits.data(),
|
||||
};
|
||||
|
||||
const int64_t t_start = ggml_time_us();
|
||||
if (llama_decode(ctx, batch) != 0) {
|
||||
fprintf(stderr, "error: llama_decode failed on embd prompt\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
// --- generation loop ---
|
||||
std::string out;
|
||||
int n_decode = 0;
|
||||
llama_token tok = llama_sampler_sample(smpl, ctx, -1);
|
||||
for (int n_pos = n_tokens; n_pos < n_tokens + n_predict; ) {
|
||||
if (llama_vocab_is_eog(vocab, tok)) break;
|
||||
char buf[256];
|
||||
int n = llama_token_to_piece(vocab, tok, buf, sizeof(buf), 0, true);
|
||||
if (n > 0) { out.append(buf, n); }
|
||||
printf("%.*s", n > 0 ? n : 0, buf);
|
||||
fflush(stdout);
|
||||
|
||||
llama_batch tb = llama_batch_get_one(&tok, 1);
|
||||
if (llama_decode(ctx, tb) != 0) { fprintf(stderr, "error: decode failed\n"); return 1; }
|
||||
n_pos += 1;
|
||||
n_decode += 1;
|
||||
tok = llama_sampler_sample(smpl, ctx, -1);
|
||||
}
|
||||
printf("\n");
|
||||
const int64_t t_end = ggml_time_us();
|
||||
fprintf(stderr, "\n[funasr-embd] generated %d tokens in %.2f s (%.1f tok/s)\n",
|
||||
n_decode, (t_end - t_start) / 1e6, n_decode / ((t_end - t_start) / 1e6));
|
||||
|
||||
llama_sampler_free(smpl);
|
||||
llama_free(ctx);
|
||||
llama_model_free(model);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
set(TARGET llama-funasr-encoder)
|
||||
add_executable(${TARGET} funasr-encoder.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,275 @@
|
||||
// funasr-encoder: ggml C++ forward pass for the Fun-ASR-Nano audio encoder
|
||||
// (SenseVoice SAN-M, 50+20 layers) + Transformer adaptor.
|
||||
//
|
||||
// Input : fbank.bin (T x 560 f32, the encoder input features)
|
||||
// Output: out.bin (T' x 1024 f32, audio embeddings for the LLM)
|
||||
// Weights: funasr-encoder.gguf (exported by export_encoder_gguf.py)
|
||||
//
|
||||
// Validated layer-by-layer against PyTorch golden dumps. fbank is currently
|
||||
// produced in Python; porting the fbank frontend to C++ is the remaining piece.
|
||||
|
||||
#include "ggml.h"
|
||||
#include "ggml-cpu.h"
|
||||
#include "ggml-alloc.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "gguf.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
struct cfg {
|
||||
int input_size = 560, d_model = 512, n_head = 4, ffn = 2048;
|
||||
int num_blocks = 50, tp_blocks = 20, kernel = 11;
|
||||
int adp_llm = 1024, adp_ffn = 2048, adp_layers = 2, adp_head = 8;
|
||||
};
|
||||
|
||||
static const float LN_EPS = 1e-5f;
|
||||
|
||||
struct funasr_model {
|
||||
cfg c;
|
||||
struct ggml_context * ctx_w = nullptr; // weights (CPU malloc, data set)
|
||||
std::map<std::string, struct ggml_tensor *> t;
|
||||
struct ggml_tensor * get(const std::string & n) {
|
||||
auto it = t.find(n);
|
||||
if (it == t.end()) { fprintf(stderr, "missing tensor: %s\n", n.c_str()); exit(1); }
|
||||
return it->second;
|
||||
}
|
||||
};
|
||||
|
||||
static bool load_model(const char * path, funasr_model & m) {
|
||||
struct gguf_init_params p = { /*no_alloc=*/false, /*ctx=*/&m.ctx_w };
|
||||
struct gguf_context * gguf = gguf_init_from_file(path, p);
|
||||
if (!gguf) { fprintf(stderr, "failed to load gguf %s\n", path); return false; }
|
||||
auto rd = [&](const char * k, int def) {
|
||||
int i = gguf_find_key(gguf, k); return i < 0 ? def : (int) gguf_get_val_u32(gguf, i);
|
||||
};
|
||||
m.c.input_size = rd("funasr.enc.input_size", 560);
|
||||
m.c.d_model = rd("funasr.enc.output_size", 512);
|
||||
m.c.n_head = rd("funasr.enc.attention_heads", 4);
|
||||
m.c.ffn = rd("funasr.enc.linear_units", 2048);
|
||||
m.c.num_blocks = rd("funasr.enc.num_blocks", 50);
|
||||
m.c.tp_blocks = rd("funasr.enc.tp_blocks", 20);
|
||||
m.c.kernel = rd("funasr.enc.kernel_size", 11);
|
||||
m.c.adp_llm = rd("funasr.adp.llm_dim", 1024);
|
||||
m.c.adp_ffn = rd("funasr.adp.ffn_dim", 2048);
|
||||
m.c.adp_layers = rd("funasr.adp.n_layer", 2);
|
||||
m.c.adp_head = rd("funasr.adp.attention_heads", 8);
|
||||
int n = gguf_get_n_tensors(gguf);
|
||||
for (int i = 0; i < n; i++) {
|
||||
const char * name = gguf_get_tensor_name(gguf, i);
|
||||
m.t[name] = ggml_get_tensor(m.ctx_w, name);
|
||||
}
|
||||
fprintf(stderr, "loaded %d tensors; cfg: d_model=%d heads=%d blocks=%d tp=%d kernel=%d adp_llm=%d\n",
|
||||
n, m.c.d_model, m.c.n_head, m.c.num_blocks, m.c.tp_blocks, m.c.kernel, m.c.adp_llm);
|
||||
gguf_free(gguf);
|
||||
return true;
|
||||
}
|
||||
|
||||
// helpers ---------------------------------------------------------------
|
||||
static struct ggml_tensor * linear(ggml_context * ctx, ggml_tensor * w, ggml_tensor * b, ggml_tensor * x) {
|
||||
struct ggml_tensor * y = ggml_mul_mat(ctx, w, x); // [out, T]
|
||||
if (b) y = ggml_add(ctx, y, b);
|
||||
return y;
|
||||
}
|
||||
static struct ggml_tensor * layernorm(ggml_context * ctx, ggml_tensor * x, ggml_tensor * g, ggml_tensor * b) {
|
||||
x = ggml_norm(ctx, x, LN_EPS);
|
||||
x = ggml_mul(ctx, x, g);
|
||||
x = ggml_add(ctx, x, b);
|
||||
return x;
|
||||
}
|
||||
|
||||
// SAN-M self-attention + FSMN. x:[D_in,T] -> [d_model,T]
|
||||
static struct ggml_tensor * sanm_attn(ggml_context * ctx, funasr_model & m, const std::string & pfx,
|
||||
ggml_tensor * x, int T) {
|
||||
const int D = m.c.d_model, H = m.c.n_head, dk = D / H, K = m.c.kernel;
|
||||
struct ggml_tensor * qkv = linear(ctx, m.get(pfx + "linear_q_k_v.weight"),
|
||||
m.get(pfx + "linear_q_k_v.bias"), x); // [3D, T]
|
||||
size_t nb1 = qkv->nb[1];
|
||||
struct ggml_tensor * q = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, 0));
|
||||
struct ggml_tensor * k = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) D * sizeof(float)));
|
||||
struct ggml_tensor * v = ggml_cont(ctx, ggml_view_2d(ctx, qkv, D, T, nb1, (size_t) 2 * D * sizeof(float)));
|
||||
|
||||
// FSMN: depthwise conv1d along time (per-channel kernel K, "same" padding),
|
||||
// plus residual v. Implemented as an exact f32 shift-accumulate to avoid the
|
||||
// F16-only ggml_conv_1d_dw path. fsmn kernel stored as [D, K] (ne0=D, ne1=K).
|
||||
const int pad = (K - 1) / 2;
|
||||
struct ggml_tensor * fk = m.get(pfx + "fsmn_block.weight"); // [D, K]
|
||||
struct ggml_tensor * vpad = ggml_pad_ext(ctx, v, 0, 0, pad, pad, 0, 0, 0, 0); // [D, T+2*pad]
|
||||
struct ggml_tensor * fsmn = v; // residual
|
||||
for (int j = 0; j < K; j++) {
|
||||
struct ggml_tensor * sl = ggml_view_2d(ctx, vpad, D, T, vpad->nb[1], (size_t) j * vpad->nb[1]);
|
||||
struct ggml_tensor * wj = ggml_view_1d(ctx, fk, D, (size_t) j * fk->nb[1]);
|
||||
fsmn = ggml_add(ctx, fsmn, ggml_mul(ctx, ggml_cont(ctx, sl), wj));
|
||||
}
|
||||
|
||||
// multi-head attention
|
||||
q = ggml_reshape_3d(ctx, q, dk, H, T);
|
||||
k = ggml_reshape_3d(ctx, k, dk, H, T);
|
||||
struct ggml_tensor * vh = ggml_reshape_3d(ctx, v, dk, H, T);
|
||||
q = ggml_permute(ctx, q, 0, 2, 1, 3); // [dk, T, H]
|
||||
k = ggml_permute(ctx, k, 0, 2, 1, 3); // [dk, T, H]
|
||||
vh = ggml_cont(ctx, ggml_permute(ctx, vh, 1, 2, 0, 3)); // [T, dk, H]
|
||||
struct ggml_tensor * kq = ggml_mul_mat(ctx, k, q); // [T, T, H]
|
||||
kq = ggml_scale(ctx, kq, 1.0f / sqrtf((float) dk));
|
||||
kq = ggml_soft_max(ctx, kq);
|
||||
struct ggml_tensor * kqv = ggml_mul_mat(ctx, vh, kq); // [dk, T, H]
|
||||
kqv = ggml_permute(ctx, kqv, 0, 2, 1, 3); // [dk, H, T]
|
||||
kqv = ggml_cont_2d(ctx, kqv, D, T); // [D, T]
|
||||
struct ggml_tensor * att = linear(ctx, m.get(pfx + "linear_out.weight"),
|
||||
m.get(pfx + "linear_out.bias"), kqv);
|
||||
return ggml_add(ctx, att, fsmn);
|
||||
}
|
||||
|
||||
// one SAN-M encoder layer. in_size may differ from d_model (first layer)
|
||||
static struct ggml_tensor * sanm_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,
|
||||
ggml_tensor * x, int T, bool residual_attn) {
|
||||
struct ggml_tensor * res = x;
|
||||
struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));
|
||||
struct ggml_tensor * sa = sanm_attn(ctx, m, pfx + "self_attn.", h, T);
|
||||
x = residual_attn ? ggml_add(ctx, res, sa) : sa;
|
||||
res = x;
|
||||
h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));
|
||||
h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);
|
||||
h = ggml_relu(ctx, h);
|
||||
h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);
|
||||
return ggml_add(ctx, res, h);
|
||||
}
|
||||
|
||||
// standard transformer layer (adaptor). d=adp_llm
|
||||
static struct ggml_tensor * adp_layer(ggml_context * ctx, funasr_model & m, const std::string & pfx,
|
||||
ggml_tensor * x, int T) {
|
||||
const int D = m.c.adp_llm, H = m.c.adp_head, dk = D / H;
|
||||
struct ggml_tensor * res = x;
|
||||
struct ggml_tensor * h = layernorm(ctx, x, m.get(pfx + "norm1.weight"), m.get(pfx + "norm1.bias"));
|
||||
struct ggml_tensor * q = linear(ctx, m.get(pfx + "self_attn.linear_q.weight"), m.get(pfx + "self_attn.linear_q.bias"), h);
|
||||
struct ggml_tensor * k = linear(ctx, m.get(pfx + "self_attn.linear_k.weight"), m.get(pfx + "self_attn.linear_k.bias"), h);
|
||||
struct ggml_tensor * v = linear(ctx, m.get(pfx + "self_attn.linear_v.weight"), m.get(pfx + "self_attn.linear_v.bias"), h);
|
||||
q = ggml_permute(ctx, ggml_reshape_3d(ctx, q, dk, H, T), 0, 2, 1, 3);
|
||||
k = ggml_permute(ctx, ggml_reshape_3d(ctx, k, dk, H, T), 0, 2, 1, 3);
|
||||
struct ggml_tensor * vh = ggml_cont(ctx, ggml_permute(ctx, ggml_reshape_3d(ctx, v, dk, H, T), 1, 2, 0, 3));
|
||||
struct ggml_tensor * kq = ggml_soft_max(ctx, ggml_scale(ctx, ggml_mul_mat(ctx, k, q), 1.0f / sqrtf((float) dk)));
|
||||
struct ggml_tensor * kqv = ggml_cont_2d(ctx, ggml_permute(ctx, ggml_mul_mat(ctx, vh, kq), 0, 2, 1, 3), D, T);
|
||||
struct ggml_tensor * att = linear(ctx, m.get(pfx + "self_attn.linear_out.weight"), m.get(pfx + "self_attn.linear_out.bias"), kqv);
|
||||
x = ggml_add(ctx, res, att);
|
||||
res = x;
|
||||
h = layernorm(ctx, x, m.get(pfx + "norm2.weight"), m.get(pfx + "norm2.bias"));
|
||||
h = linear(ctx, m.get(pfx + "feed_forward.w_1.weight"), m.get(pfx + "feed_forward.w_1.bias"), h);
|
||||
h = ggml_relu(ctx, h);
|
||||
h = linear(ctx, m.get(pfx + "feed_forward.w_2.weight"), m.get(pfx + "feed_forward.w_2.bias"), h);
|
||||
return ggml_add(ctx, res, h);
|
||||
}
|
||||
|
||||
// sinusoidal position encoding, depth = input feature dim, positions 1..T
|
||||
static void add_posenc(std::vector<float> & x, int T, int depth) {
|
||||
double inc = log(10000.0) / (depth / 2.0 - 1.0);
|
||||
for (int t = 0; t < T; t++) {
|
||||
double pos = t + 1; // positions start at 1
|
||||
for (int i = 0; i < depth / 2; i++) {
|
||||
double its = exp(i * -inc);
|
||||
double st = pos * its;
|
||||
x[(size_t) t * depth + i] += (float) sin(st);
|
||||
x[(size_t) t * depth + depth / 2 + i] += (float) cos(st);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char ** argv) {
|
||||
std::string gguf_path, fbank_path, out_path = "out.bin";
|
||||
int limit = -1; // -L: run only first N (encoders0+encoders) layers, dump running x
|
||||
bool run_adaptor = true;
|
||||
for (int i = 1; i < argc; i++) {
|
||||
if (!strcmp(argv[i], "-m") && i+1 < argc) gguf_path = argv[++i];
|
||||
else if (!strcmp(argv[i], "-f") && i+1 < argc) fbank_path = argv[++i];
|
||||
else if (!strcmp(argv[i], "-o") && i+1 < argc) out_path = argv[++i];
|
||||
else if (!strcmp(argv[i], "-L") && i+1 < argc) { limit = atoi(argv[++i]); run_adaptor = false; }
|
||||
else { fprintf(stderr, "usage: %s -m enc.gguf -f fbank.bin [-o out.bin] [-L nlayers]\n", argv[0]); return 1; }
|
||||
}
|
||||
|
||||
funasr_model m;
|
||||
if (!load_model(gguf_path.c_str(), m)) return 1;
|
||||
|
||||
// read fbank.bin (T x F)
|
||||
FILE * f = fopen(fbank_path.c_str(), "rb");
|
||||
if (!f) { fprintf(stderr, "cannot open %s\n", fbank_path.c_str()); return 1; }
|
||||
int32_t T, F; if (fread(&T, 4, 1, f) != 1 || fread(&F, 4, 1, f) != 1) { fclose(f); return 1; }
|
||||
std::vector<float> fbank((size_t) T * F);
|
||||
if (fread(fbank.data(), sizeof(float), fbank.size(), f) != fbank.size()) { fclose(f); return 1; }
|
||||
fclose(f);
|
||||
fprintf(stderr, "fbank: T=%d F=%d\n", T, F);
|
||||
|
||||
// pre-scale (*sqrt(d_model)) and add position encoding on the host
|
||||
float scale = sqrtf((float) m.c.d_model);
|
||||
for (auto & v : fbank) v *= scale;
|
||||
add_posenc(fbank, T, F);
|
||||
|
||||
// backend + compute context
|
||||
ggml_backend_t backend = ggml_backend_cpu_init();
|
||||
size_t ctx_size = (size_t) 1024*1024*1024; // graph metadata
|
||||
struct ggml_init_params cp = { ctx_size, nullptr, true };
|
||||
struct ggml_context * ctx = ggml_init(cp);
|
||||
|
||||
struct ggml_tensor * inp = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, F, T);
|
||||
ggml_set_name(inp, "inp");
|
||||
ggml_set_input(inp);
|
||||
|
||||
struct ggml_tensor * x = inp;
|
||||
int done = 0;
|
||||
bool stop = false;
|
||||
// encoders0 (1 layer, in_size=input_size != d_model -> no attn residual)
|
||||
x = sanm_layer(ctx, m, "audio_encoder.encoders0.0.", x, T, /*residual_attn=*/false);
|
||||
done++;
|
||||
if (limit >= 0 && done >= limit) stop = true;
|
||||
// encoders (num_blocks-1 layers)
|
||||
for (int i = 0; i < m.c.num_blocks - 1 && !stop; i++) {
|
||||
x = sanm_layer(ctx, m, "audio_encoder.encoders." + std::to_string(i) + ".", x, T, true);
|
||||
done++;
|
||||
if (limit >= 0 && done >= limit) stop = true;
|
||||
}
|
||||
if (!stop) {
|
||||
x = layernorm(ctx, x, m.get("audio_encoder.after_norm.weight"), m.get("audio_encoder.after_norm.bias"));
|
||||
for (int i = 0; i < m.c.tp_blocks; i++)
|
||||
x = sanm_layer(ctx, m, "audio_encoder.tp_encoders." + std::to_string(i) + ".", x, T, true);
|
||||
x = layernorm(ctx, x, m.get("audio_encoder.tp_norm.weight"), m.get("audio_encoder.tp_norm.bias"));
|
||||
// adaptor: downsample_rate=1 -> linear1(relu)linear2 then blocks
|
||||
if (run_adaptor) {
|
||||
x = linear(ctx, m.get("audio_adaptor.linear1.weight"), m.get("audio_adaptor.linear1.bias"), x);
|
||||
x = ggml_relu(ctx, x);
|
||||
x = linear(ctx, m.get("audio_adaptor.linear2.weight"), m.get("audio_adaptor.linear2.bias"), x);
|
||||
for (int i = 0; i < m.c.adp_layers; i++)
|
||||
x = adp_layer(ctx, m, "audio_adaptor.blocks." + std::to_string(i) + ".", x, T);
|
||||
}
|
||||
}
|
||||
ggml_set_output(x);
|
||||
|
||||
struct ggml_cgraph * gf = ggml_new_graph_custom(ctx, 32768, false);
|
||||
ggml_build_forward_expand(gf, x);
|
||||
|
||||
ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_cpu_buffer_type());
|
||||
ggml_gallocr_alloc_graph(galloc, gf);
|
||||
ggml_backend_tensor_set(inp, fbank.data(), 0, ggml_nbytes(inp));
|
||||
|
||||
ggml_backend_cpu_set_n_threads(backend, 8);
|
||||
int64_t t0 = ggml_time_us();
|
||||
if (ggml_backend_graph_compute(backend, gf) != GGML_STATUS_SUCCESS) {
|
||||
fprintf(stderr, "compute failed\n"); return 1;
|
||||
}
|
||||
int64_t t1 = ggml_time_us();
|
||||
|
||||
int D = (int) x->ne[0];
|
||||
std::vector<float> out((size_t) D * T);
|
||||
ggml_backend_tensor_get(x, out.data(), 0, ggml_nbytes(x));
|
||||
FILE * fo = fopen(out_path.c_str(), "wb");
|
||||
if (!fo) { fprintf(stderr, "failed to open output file %s\n", out_path.c_str()); return 1; }
|
||||
fwrite(&T, 4, 1, fo); fwrite(&D, 4, 1, fo); fwrite(out.data(), sizeof(float), out.size(), fo);
|
||||
fclose(fo);
|
||||
fprintf(stderr, "done: wrote %s [%d x %d] in %.2f s (layers run=%d, adaptor=%d)\n",
|
||||
out_path.c_str(), T, D, (t1 - t0)/1e6, done, run_adaptor && !stop);
|
||||
ggml_gallocr_free(galloc); ggml_free(ctx); ggml_backend_free(backend);
|
||||
if (m.ctx_w) ggml_free(m.ctx_w);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
// funasr_audio.h — load any audio file (WAV/MP3/FLAC/OGG...) as 16 kHz mono f32,
|
||||
// via miniaudio's decoder (handles arbitrary sample rate, channels, bit depth).
|
||||
// One TU must define FUNASR_AUDIO_IMPLEMENTATION before including.
|
||||
#ifndef FUNASR_AUDIO_H
|
||||
#define FUNASR_AUDIO_H
|
||||
#include <vector>
|
||||
// returns true on success; out = mono 16 kHz float samples in [-1,1]
|
||||
inline bool funasr_load_audio_16k_mono(const char * path, std::vector<float> & out);
|
||||
#endif
|
||||
|
||||
#ifdef FUNASR_AUDIO_IMPLEMENTATION
|
||||
// keep only the decoder + data conversion (resampler/channel mixer); drop playback/capture
|
||||
#define MA_NO_DEVICE_IO
|
||||
#define MA_NO_ENGINE
|
||||
#define MA_NO_GENERATION
|
||||
#define MA_NO_THREADING
|
||||
#define MINIAUDIO_IMPLEMENTATION
|
||||
#include "miniaudio.h"
|
||||
#include <cstdio>
|
||||
inline bool funasr_load_audio_16k_mono(const char * path, std::vector<float> & out) {
|
||||
if (!path) { fprintf(stderr, "audio: null path\n"); return false; }
|
||||
ma_decoder_config cfg = ma_decoder_config_init(ma_format_f32, 1, 16000); // f32, mono, 16k
|
||||
ma_decoder dec;
|
||||
if (ma_decoder_init_file(path, &cfg, &dec) != MA_SUCCESS) {
|
||||
fprintf(stderr, "audio: failed to open/decode %s (supported: wav/mp3/flac)\n", path);
|
||||
return false;
|
||||
}
|
||||
out.clear();
|
||||
ma_uint64 nframes = 0;
|
||||
if (ma_decoder_get_length_in_pcm_frames(&dec, &nframes) == MA_SUCCESS && nframes > 0) {
|
||||
out.resize(nframes);
|
||||
ma_uint64 got = 0;
|
||||
ma_result r = ma_decoder_read_pcm_frames(&dec, out.data(), nframes, &got);
|
||||
if (r != MA_SUCCESS && r != MA_AT_END) { ma_decoder_uninit(&dec); fprintf(stderr, "audio: decode error\n"); return false; }
|
||||
out.resize(got);
|
||||
} else {
|
||||
// length unknown (e.g. some mp3 streams): read in chunks until EOF
|
||||
std::vector<float> buf(16000);
|
||||
for (;;) {
|
||||
ma_uint64 got = 0;
|
||||
ma_result r = ma_decoder_read_pcm_frames(&dec, buf.data(), buf.size(), &got);
|
||||
out.insert(out.end(), buf.begin(), buf.begin() + got);
|
||||
if (got < buf.size() || (r != MA_SUCCESS && r != MA_AT_END)) break;
|
||||
}
|
||||
}
|
||||
ma_decoder_uninit(&dec);
|
||||
return !out.empty();
|
||||
}
|
||||
#endif
|
||||
@@ -0,0 +1,161 @@
|
||||
// funasr_vad.h — single-header FSMN-VAD for the funasr ggml runtime.
|
||||
// Exposes funasr_vad_segments(): 16k mono wav -> speech segments [start_ms,end_ms].
|
||||
// Front end (80-mel fbank + LFR m5n1 + CMVN) and FSMN encoder validated bit-exact vs
|
||||
// PyTorch fsmn-vad; the host state machine reproduces E2EVadModel (DEFAULT_SILENCE_SCHEDULE,
|
||||
// chunk-stepped) to within 1 frame (10ms) of fsmn-vad.generate on the 184-clip set.
|
||||
#pragma once
|
||||
#include "ggml.h"
|
||||
#include "ggml-cpu.h"
|
||||
#include "ggml-alloc.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "gguf.h"
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
#ifndef M_PI
|
||||
#define M_PI 3.14159265358979323846 // not guaranteed by <cmath> on MSVC
|
||||
#endif
|
||||
|
||||
namespace funasr_vad_impl {
|
||||
static const int FS=16000,WINLEN=400,SHIFT=160,NFFT=512,NMEL=80;
|
||||
static const float PREEMPH=0.97f,LOWF=20.0f,HIGHF=8000.0f;
|
||||
static inline float melf(float f){return 1127.0f*logf(1.0f+f/700.0f);}
|
||||
static void fftc(std::vector<float>&re,std::vector<float>&im,int n){for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}
|
||||
for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){float ur=re[i+k],ui=im[i+k];
|
||||
float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;float nc=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=nc;}}}}
|
||||
static std::vector<std::vector<float>> fbank80(std::vector<float> wav){
|
||||
for(auto&v:wav)v*=32768.0f; std::vector<float>win(WINLEN);
|
||||
for(int i=0;i<WINLEN;i++)win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));
|
||||
const int NB=NFFT/2+1; float bw=(float)FS/NFFT,ml=melf(LOWF),mh=melf(HIGHF),dm=(mh-ml)/(NMEL+1);
|
||||
std::vector<std::vector<float>>fb(NMEL,std::vector<float>(NB,0.0f));
|
||||
for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;for(int k=0;k<NB;k++){float mf=melf(bw*k);if(mf>L&&mf<R)fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}
|
||||
int N=wav.size(),T=(N-WINLEN)/SHIFT+1; if(T<1)T=0; std::vector<std::vector<float>>feat(T,std::vector<float>(NMEL));
|
||||
std::vector<float>re(NFFT),im(NFFT),fr(WINLEN);const float fl=1.1920929e-07f;
|
||||
for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;
|
||||
for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];
|
||||
for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}fftc(re,im,NFFT);
|
||||
for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NB;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);feat[t][m]=logf(e>fl?e:fl);}}
|
||||
return feat;
|
||||
}
|
||||
static std::vector<float> lfr(const std::vector<std::vector<float>>&feat,int m,int n,int&T_out){
|
||||
int T=feat.size(); if(T<1){T_out=0;return {};} // empty (audio shorter than one frame)
|
||||
int D=NMEL,pad=(m-1)/2; int Tl=(T+n-1)/n;
|
||||
std::vector<std::vector<float>> pf; pf.reserve(T+pad+m);
|
||||
for(int i=0;i<pad;i++)pf.push_back(feat[0]);
|
||||
for(int t=0;t<T;t++)pf.push_back(feat[t]);
|
||||
while((int)pf.size()<(Tl-1)*n+m)pf.push_back(feat[T-1]);
|
||||
std::vector<float> out((size_t)Tl*m*D);
|
||||
for(int i=0;i<Tl;i++)for(int j=0;j<m;j++)memcpy(&out[((size_t)i*m+j)*D],pf[i*n+j].data(),D*sizeof(float));
|
||||
T_out=Tl; return out;
|
||||
}
|
||||
struct vad{ggml_context*ctx=nullptr;std::map<std::string,ggml_tensor*>t;
|
||||
ggml_tensor*g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"vad: missing %s\n",n.c_str());return nullptr;}return it->second;}};
|
||||
static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}
|
||||
} // namespace funasr_vad_impl
|
||||
|
||||
// Run FSMN-VAD on a 16k mono float waveform; fills segs with [start_ms,end_ms] speech spans.
|
||||
// max_seg_ms caps a single segment (e.g. 30000); pass <=0 to use 60000 (model default).
|
||||
inline bool funasr_vad_segments(const std::string& gguf_path, const std::vector<float>& wav,
|
||||
int max_seg_ms, std::vector<std::pair<int,int>>& segs, int nthreads=8){
|
||||
using namespace funasr_vad_impl;
|
||||
segs.clear();
|
||||
vad m; gguf_init_params ip={false,&m.ctx}; gguf_context*gg=gguf_init_from_file(gguf_path.c_str(),ip);
|
||||
if(!gg){fprintf(stderr,"vad: cannot load %s\n",gguf_path.c_str());return false;}
|
||||
auto rd=[&](const char*k,int d){int i=gguf_find_key(gg,k);return i<0?d:(int)gguf_get_val_u32(gg,i);};
|
||||
int idim=rd("vad.input_dim",400),pd=rd("vad.proj_dim",128),nl=rd("vad.fsmn_layers",4),lorder=rd("vad.lorder",20),
|
||||
od=rd("vad.output_dim",248),lm=rd("vad.lfr_m",5),ln=rd("vad.lfr_n",1);
|
||||
for(int i=0;i<gguf_get_n_tensors(gg);i++){const char*nm=gguf_get_tensor_name(gg,i);m.t[nm]=ggml_get_tensor(m.ctx,nm);}
|
||||
gguf_free(gg);
|
||||
|
||||
// fail fast (not segfault) if the GGUF is missing tensors the graph dereferences
|
||||
auto need=[&](const std::string&n){ return m.g(n)!=nullptr; };
|
||||
bool ok_t = need("cmvn.shift")&&need("cmvn.scale")&&need("encoder.in_linear1.linear.weight")
|
||||
&&need("encoder.in_linear2.linear.weight")&&need("encoder.out_linear1.linear.weight")
|
||||
&&need("encoder.out_linear2.linear.weight");
|
||||
for(int i=0;i<nl&&ok_t;i++){std::string p="encoder.fsmn."+std::to_string(i)+".";
|
||||
ok_t=need(p+"linear.linear.weight")&&need(p+"fsmn_block.conv_left.weight")&&need(p+"affine.linear.weight");}
|
||||
if(!ok_t){fprintf(stderr,"vad: gguf missing required tensors\n"); if(m.ctx)ggml_free(m.ctx); return false;}
|
||||
|
||||
auto feat=fbank80(wav); int T=0; auto feats=lfr(feat,lm,ln,T); // [T,400]
|
||||
if(T<1){if(m.ctx)ggml_free(m.ctx);return true;} // too short -> no speech
|
||||
float*shift=(float*)m.g("cmvn.shift")->data,*scale=(float*)m.g("cmvn.scale")->data;
|
||||
for(int t=0;t<T;t++)for(int d=0;d<idim;d++)feats[(size_t)t*idim+d]=(feats[(size_t)t*idim+d]+shift[d])*scale[d];
|
||||
|
||||
ggml_backend_t be=ggml_backend_cpu_init();
|
||||
// no_alloc=true -> ctx holds only tensor/graph metadata (the real compute buffer is
|
||||
// allocated by gallocr below), so a few MB is plenty regardless of clip length.
|
||||
ggml_init_params cp={(size_t)16*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);
|
||||
ggml_tensor*x=ggml_new_tensor_2d(c,GGML_TYPE_F32,idim,T); ggml_set_input(x);
|
||||
ggml_tensor*h=lin(c,m.g("encoder.in_linear1.linear.weight"),m.g("encoder.in_linear1.linear.bias"),x);
|
||||
h=lin(c,m.g("encoder.in_linear2.linear.weight"),m.g("encoder.in_linear2.linear.bias"),h); h=ggml_relu(c,h);
|
||||
for(int i=0;i<nl;i++){std::string p="encoder.fsmn."+std::to_string(i)+".";
|
||||
ggml_tensor*z=ggml_mul_mat(c,m.g(p+"linear.linear.weight"),h);
|
||||
ggml_tensor*fk=m.g(p+"fsmn_block.conv_left.weight"); ggml_tensor*zp=ggml_pad_ext(c,z,0,0,lorder-1,0,0,0,0,0); ggml_tensor*acc=z;
|
||||
// sl is a full-row slice of the contiguous padded tensor -> already contiguous, no ggml_cont needed
|
||||
for(int j=0;j<lorder;j++){auto sl=ggml_view_2d(c,zp,pd,T,zp->nb[1],(size_t)j*zp->nb[1]);auto wj=ggml_view_1d(c,fk,pd,(size_t)j*fk->nb[1]);acc=ggml_add(c,acc,ggml_mul(c,sl,wj));}
|
||||
ggml_tensor*a=lin(c,m.g(p+"affine.linear.weight"),m.g(p+"affine.linear.bias"),acc); h=ggml_relu(c,a);}
|
||||
h=lin(c,m.g("encoder.out_linear1.linear.weight"),m.g("encoder.out_linear1.linear.bias"),h);
|
||||
h=lin(c,m.g("encoder.out_linear2.linear.weight"),m.g("encoder.out_linear2.linear.bias"),h);
|
||||
h=ggml_soft_max(c,h); ggml_set_output(h);
|
||||
ggml_cgraph*gf=ggml_new_graph(c); ggml_build_forward_expand(gf,h);
|
||||
ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);
|
||||
ggml_backend_tensor_set(x,feats.data(),0,ggml_nbytes(x)); ggml_backend_cpu_set_n_threads(be,nthreads);
|
||||
bool ok=ggml_backend_graph_compute(be,gf)==GGML_STATUS_SUCCESS;
|
||||
std::vector<float> sc((size_t)od*T); if(ok)ggml_backend_tensor_get(h,sc.data(),0,ggml_nbytes(h));
|
||||
ggml_gallocr_free(ga);ggml_free(c);ggml_backend_free(be);if(m.ctx)ggml_free(m.ctx);
|
||||
if(!ok)return false;
|
||||
|
||||
// ===== E2EVadModel state machine (host) -> speech segments [start_ms,end_ms] =====
|
||||
const int FR=10; // ms per frame (frame_in_ms)
|
||||
const int win=20; // window_size_ms 200 / 10
|
||||
const int s2s=15, sp2s=15; // sil_to_speech / speech_to_sil thres (150/10)
|
||||
const int lookahead_end=100/FR; // lookahead_time_end_point 100/10 = 10 (do_extend)
|
||||
int max_seg = (max_seg_ms>0 ? max_seg_ms : 60000)/FR; // max_single_segment frames
|
||||
const int start_lookback = win + 200/FR; // LatencyFrmNumAtStartPoint = 40
|
||||
// End-silence threshold from DEFAULT_SILENCE_SCHEDULE, stepped at chunk boundaries (chunk_size
|
||||
// 60000ms = 6000 frames). At each boundary, if mid-segment (InSpeech) or in_speech latched,
|
||||
// accumulated_ms += 60000; threshold = schedule(accumulated). Reset to 0 on each segment emit.
|
||||
const int CHUNK=60000/FR;
|
||||
int acc=0, insp=0; int max_end_sil, end_lookback;
|
||||
auto recompute=[&](){
|
||||
int s; if(acc<=10000)s=2000; else if(acc<=20000)s=1000; else if(acc<=30000)s=800;
|
||||
else if(acc<=40000)s=600; else if(acc<=50000)s=400; else if(acc<=60000)s=200; else s=100;
|
||||
int ms=s-150; if(ms<0)ms=0; max_end_sil=ms/FR;
|
||||
end_lookback=max_end_sil-lookahead_end-1; if(end_lookback<0)end_lookback=0;
|
||||
};
|
||||
recompute();
|
||||
std::vector<int> wbuf(win,0); int wpos=0,wsum=0,pre=0;
|
||||
int st=0, cstart=-1, csil=0, prev_end=0;
|
||||
auto reset=[&](){ std::fill(wbuf.begin(),wbuf.end(),0); wpos=0; wsum=0; pre=0; csil=0; st=0; cstart=-1; acc=0; insp=0; };
|
||||
auto emit=[&](int s,int e){ if(s<prev_end)s=prev_end; if(s<0)s=0; if(e>T)e=T; if(e>s){segs.push_back({s,e}); prev_end=e;} };
|
||||
for(int t=0;t<T;t++){
|
||||
if(t>0 && t%CHUNK==0){ if(st==1||insp){acc+=60000; insp=1;} recompute(); }
|
||||
float sil=sc[(size_t)t*od+0];
|
||||
int fs = ((1.0f-sil) >= sil + 0.5f) ? 1 : 0; // speech_noise_thres=0.5
|
||||
wsum -= wbuf[wpos]; wsum += fs; wbuf[wpos]=fs; wpos=(wpos+1)%win;
|
||||
int ch;
|
||||
if(pre==0 && wsum>=s2s){pre=1; ch=3;}
|
||||
else if(pre==1 && wsum<=sp2s){pre=0; ch=1;}
|
||||
else ch = pre==0?0:2;
|
||||
if(ch==3){ csil=0;
|
||||
if(st==0){ cstart=t-start_lookback; if(cstart<prev_end)cstart=prev_end; if(cstart<0)cstart=0; st=1; }
|
||||
else if(st==1 && t-cstart+1>max_seg){ emit(cstart,t); reset(); }
|
||||
} else if(ch==1||ch==2){ csil=0;
|
||||
if(st==1 && t-cstart+1>max_seg){ emit(cstart,t); reset(); }
|
||||
} else { csil++;
|
||||
if(st==1){
|
||||
if(csil>=max_end_sil){ emit(cstart, t-end_lookback); reset(); }
|
||||
else if(t-cstart+1>max_seg){ emit(cstart,t); reset(); }
|
||||
}
|
||||
}
|
||||
}
|
||||
if(st==1) emit(cstart,T);
|
||||
// convert frame indices -> ms
|
||||
for(auto&s:segs){ s.first*=FR; s.second*=FR; }
|
||||
return true;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,6 @@
|
||||
set(TARGET llama-funasr-vad)
|
||||
add_executable(${TARGET} funasr-vad.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_include_directories(${TARGET} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../funasr-common)
|
||||
target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,25 @@
|
||||
// funasr-vad: FSMN-VAD on ggml. WAV(any fmt/rate) -> speech segments [start_ms,end_ms].
|
||||
// Front end + FSMN encoder validated bit-exact vs PyTorch; state machine reproduces
|
||||
// E2EVadModel segmentation to within 1 frame (10ms) of fsmn-vad.generate on the 184-clip set.
|
||||
#define FUNASR_AUDIO_IMPLEMENTATION
|
||||
#include "funasr_audio.h"
|
||||
#include "funasr_vad.h"
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
int main(int argc,char**argv){
|
||||
std::string gp,wp;
|
||||
for(int i=1;i<argc;i++){if(!strcmp(argv[i],"-m")&&i+1<argc)gp=argv[++i];else if(!strcmp(argv[i],"-a")&&i+1<argc)wp=argv[++i];}
|
||||
if(gp.empty()||wp.empty()){fprintf(stderr,"usage: %s -m fsmn-vad.gguf -a audio.wav\n",argv[0]);return 1;}
|
||||
std::vector<float> wav; if(!funasr_load_audio_16k_mono(wp.c_str(),wav)){fprintf(stderr,"read audio failed\n");return 1;}
|
||||
int max_seg = getenv("VAD_MAXSEG") ? atoi(getenv("VAD_MAXSEG")) : 30000;
|
||||
std::vector<std::pair<int,int>> segs;
|
||||
if(!funasr_vad_segments(gp,wav,max_seg,segs)){fprintf(stderr,"vad failed\n");return 1;}
|
||||
for(auto&s:segs) printf("%d %d\n", s.first, s.second);
|
||||
fprintf(stderr,"[vad] %zu segments (max_seg=%dms)\n",segs.size(),max_seg);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
# Paraformer on llama.cpp / GGUF
|
||||
|
||||
Run **Paraformer** (the non-autoregressive ASR model) on the
|
||||
[llama.cpp](https://github.com/ggml-org/llama.cpp) / ggml stack — **CPU, edge,
|
||||
a single binary, no Python at runtime**. Like whisper.cpp, but for Paraformer.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Paraformer normally runs on PyTorch / ONNX. This runtime ports it to **ggml +
|
||||
GGUF** so it runs CPU-only, offline, embedded in a C/C++ app, with quantized
|
||||
weights — laptops, phones, edge boxes, no GPU and no Python. (For high-QPS GPU
|
||||
serving, the PyTorch path is still the way.)
|
||||
|
||||
## Architecture
|
||||
|
||||
Paraformer is **non-autoregressive**: it predicts all output tokens in one pass.
|
||||
|
||||
```
|
||||
audio.wav (16k mono)
|
||||
│ kaldi 80-mel fbank + LFR + CMVN (C++)
|
||||
▼
|
||||
features [T, 560]
|
||||
│ SANM encoder (50 layers: LN + fused QKV + FSMN + FFN) (ggml)
|
||||
▼
|
||||
encoder_out [T, 512]
|
||||
│ CIF predictor: conv1d → sigmoid → α; integrate-and-fire (host)
|
||||
▼
|
||||
acoustic embeds [N_tok, 512] (N_tok = number of output tokens)
|
||||
│ SANM decoder (16 layers: FFN → FSMN self-attn → cross-attn to encoder) (ggml)
|
||||
▼
|
||||
logits [N_tok, vocab] → argmax → token ids → text
|
||||
```
|
||||
|
||||
CIF (Continuous Integrate-and-Fire) walks the encoder output accumulating a
|
||||
predicted "weight" α per frame; each time the running sum crosses 1.0 it emits one
|
||||
acoustic token. This both decides the token count and produces the acoustic
|
||||
embeddings the decoder consumes. The SANM encoder/FSMN/attention primitives are
|
||||
shared with the Fun-ASR-Nano and SenseVoice runtimes.
|
||||
|
||||
## Quickstart
|
||||
|
||||
**1. Build:**
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
cp -r /path/to/runtime/llama.cpp/funasr-paraformer examples/
|
||||
echo 'add_subdirectory(funasr-paraformer)' >> examples/CMakeLists.txt
|
||||
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build -j --target llama-funasr-paraformer
|
||||
```
|
||||
|
||||
**2. Convert weights** (needs the checkpoint, e.g. `funasr/paraformer-zh`):
|
||||
```bash
|
||||
python runtime/llama.cpp/export_paraformer_gguf.py \
|
||||
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
||||
--out paraformer.gguf # f32, ~863 MB
|
||||
python runtime/llama.cpp/export_paraformer_gguf.py --wtype f16 \
|
||||
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
||||
--out paraformer-f16.gguf # half size
|
||||
```
|
||||
|
||||
**3. Transcribe:**
|
||||
```bash
|
||||
build/bin/llama-funasr-paraformer -m paraformer.gguf -a audio.wav # prints transcription text (--ids for raw)
|
||||
```
|
||||
Expected output:
|
||||
```
|
||||
我想问我在滨海新区有房我一直没有照顾孩子...你觉得这是正常的想法吗
|
||||
[paraformer] T=742 N_tok=105 enc 1.24s dec 0.48s
|
||||
```
|
||||
|
||||
## Accuracy & validation
|
||||
|
||||
- Decoded text is **character-for-character identical** to the FunASR `AutoModel`
|
||||
output on a benchmark clip; the CIF token count matches exactly (105/105).
|
||||
- Stage-by-stage vs PyTorch: encoder cosine 0.997, acoustic embeds cosine 0.993
|
||||
(the small residual is the reference frontend's random `dither=1.0`; the C++
|
||||
front end is deterministic, dither=0).
|
||||
- Encode ≈ 1.2 s + decode ≈ 0.5 s on CPU for a 44 s clip.
|
||||
|
||||
## Tips & gotchas
|
||||
|
||||
- **CMVN IS applied** (unlike SenseVoice): `(fbank + shift) * scale`, per-dim (560),
|
||||
from `am.mvn`. Parsing note: `am.mvn` has three `[...]` blocks — `[Splice idx]`,
|
||||
`[AddShift=shift]`, `[Rescale=scale]`; use the two 560-length vectors. Getting
|
||||
this wrong makes the CIF predictor emit ~4× too few tokens.
|
||||
- **CIF/predictor runs on host** (it's a sequential integrate-and-fire loop);
|
||||
the encoder and decoder run in ggml.
|
||||
- The decoder self-attention is **FSMN-only** (no QK attention); cross-attention
|
||||
attends to the encoder output. The decoder FFN has an internal LayerNorm and the
|
||||
second linear has no bias.
|
||||
- WAV input assumes 16 kHz mono PCM16.
|
||||
|
||||
## Files
|
||||
```
|
||||
funasr-paraformer/ ggml runtime: WAV → token ids
|
||||
export_paraformer_gguf.py export encoder + predictor + decoder + CMVN to GGUF
|
||||
detok.py token-id → text (tokens.json)
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
- Built-in detok; timestamps (CIF peaks give alignment); arbitrary WAV / resampling;
|
||||
encoder/decoder quantization.
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Detokenize Paraformer token ids -> text via tokens.json.
|
||||
Usage: python detok_paraformer.py <tokens.json> <ids.txt>"""
|
||||
import sys, json
|
||||
toks = json.load(open(sys.argv[1]))
|
||||
ids = [int(x) for x in open(sys.argv[2]).read().split() if int(x) not in (1, 2)] # drop sos/eos
|
||||
out = "".join(toks[i] for i in ids if 0 <= i < len(toks))
|
||||
print(out.replace("@@", "").replace("▁", " ").strip())
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export Paraformer (SANM encoder + CIF predictor + SANM decoder) to GGUF.
|
||||
Encoder reuses the shared SAN-M forward. Predictor (CIF) runs on host in C++.
|
||||
"""
|
||||
import argparse, os, re
|
||||
import numpy as np, torch, gguf
|
||||
|
||||
|
||||
def parse_mvn(path):
|
||||
# am.mvn (kaldi nnet) has 3 bracketed blocks: [Splice idx], [AddShift=shift],
|
||||
# [Rescale=scale]. Take the two 560-length vectors (shift then scale).
|
||||
# apply: out = (x + shift) * scale
|
||||
blocks = [np.array([float(x) for x in b.split()], np.float32)
|
||||
for b in re.findall(r"\[([^\]]*)\]", open(path).read())]
|
||||
vecs = [b for b in blocks if b.size > 1]
|
||||
return vecs[0], vecs[1]
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model_pt", required=True); ap.add_argument("--mvn", required=True)
|
||||
ap.add_argument("--out", required=True); ap.add_argument("--wtype", default="f32", choices=["f32","f16","q8_0"])
|
||||
ap.add_argument("--tokens", default=None, help="tokens.json (id->token); default: next to model_pt")
|
||||
a = ap.parse_args()
|
||||
sd = torch.load(a.model_pt, map_location="cpu"); sd = sd.get("state_dict", sd)
|
||||
w = gguf.GGUFWriter(a.out, "paraformer")
|
||||
w.add_uint32("pf.enc.output_size", 512); w.add_uint32("pf.enc.attention_heads", 4)
|
||||
w.add_uint32("pf.enc.num_blocks", 50); w.add_uint32("pf.enc.kernel_size", 11)
|
||||
w.add_uint32("pf.dec.num_blocks", 16); w.add_uint32("pf.dec.att_layer_num", 16)
|
||||
w.add_uint32("pf.dec.decoders3", 1); w.add_uint32("pf.dec.attention_heads", 4)
|
||||
w.add_uint32("pf.dec.kernel_size", 11); w.add_uint32("pf.vocab_size", 8404)
|
||||
import json, glob
|
||||
tp = a.tokens or (glob.glob(os.path.join(os.path.dirname(a.model_pt), "tokens.json")) + [None])[0]
|
||||
if tp and os.path.exists(tp):
|
||||
with open(tp, encoding="utf-8") as f: toks = json.load(f)
|
||||
w.add_array("pf.vocab", toks)
|
||||
print(f"embedded pf.vocab ({len(toks)} tokens) from {tp}")
|
||||
else:
|
||||
print("WARNING: tokens.json not found - gguf will have no vocab (binary falls back to ids)")
|
||||
w.add_float32("pf.predictor.tail_threshold", 0.45)
|
||||
w.add_float32("pf.predictor.threshold", 1.0)
|
||||
shift, scale = parse_mvn(a.mvn)
|
||||
w.add_tensor("cmvn.shift", shift); w.add_tensor("cmvn.scale", scale)
|
||||
n = 0
|
||||
for k, v in sd.items():
|
||||
if not (k.startswith("encoder.") or k.startswith("decoder.") or k.startswith("predictor.")):
|
||||
continue
|
||||
if k == "decoder.embed.0.weight": # token embedding, unused at NAR inference
|
||||
continue
|
||||
arr = v.detach().to(torch.float32).contiguous().numpy()
|
||||
if k.endswith("fsmn_block.weight") and arr.ndim == 3: # (D,1,K)->(K,D)
|
||||
arr = np.ascontiguousarray(arr[:, 0, :].T)
|
||||
elif args_f16(a) and arr.ndim == 2 and "norm" not in k and "cif_output" not in k:
|
||||
arr = arr.astype(np.float16)
|
||||
if a.wtype == "q8_0" and arr.ndim == 2 and "norm" not in k and "fsmn_block" not in k and "predictor" not in k and arr.shape[1] % 32 == 0:
|
||||
from gguf import quants as _q, GGMLQuantizationType as _QT
|
||||
w.add_tensor(k, _q.quantize(arr, _QT.Q8_0), raw_dtype=_QT.Q8_0)
|
||||
else:
|
||||
w.add_tensor(k, arr)
|
||||
n += 1
|
||||
print(f"writing {n} tensors (+cmvn) to {a.out}")
|
||||
w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close()
|
||||
print(f"done: {a.out} ({os.path.getsize(a.out)/1e6:.1f} MB)")
|
||||
|
||||
|
||||
def args_f16(a): return a.wtype == "f16"
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,6 @@
|
||||
set(TARGET llama-funasr-paraformer)
|
||||
add_executable(${TARGET} funasr-paraformer.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_include_directories(${TARGET} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../funasr-common)
|
||||
target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,218 @@
|
||||
// funasr-paraformer: Paraformer (non-autoregressive ASR) on ggml.
|
||||
// WAV → kaldi fbank → CMVN → SANM encoder (ggml) → CIF predictor (host) →
|
||||
// SANM decoder w/ cross-attn (ggml) → argmax → token ids (stdout).
|
||||
// Encoder/FSMN/attention primitives are shared with the Fun-ASR-Nano runtime.
|
||||
|
||||
#include "ggml.h"
|
||||
#include "ggml-cpu.h"
|
||||
#include "ggml-alloc.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "gguf.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
static const float LN_EPS = 1e-5f;
|
||||
|
||||
// ===== audio loader: any wav/mp3/flac, any rate/channels -> 16k mono (miniaudio) =====
|
||||
#define FUNASR_AUDIO_IMPLEMENTATION
|
||||
#include "funasr_audio.h"
|
||||
#include "funasr_vad.h" // built-in FSMN-VAD front end (--vad segmentation)
|
||||
#include <utility>
|
||||
static const int FS=16000,WINLEN=400,SHIFT=160,NFFT=512,NMEL=80,LFR_M=7,LFR_N=6;
|
||||
static const float PREEMPH=0.97f,LOWF=20.0f,HIGHF=8000.0f;
|
||||
static inline float melf(float f){return 1127.0f*logf(1.0f+f/700.0f);}
|
||||
static void fftc(std::vector<float>&re,std::vector<float>&im,int n){for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}
|
||||
for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){float ur=re[i+k],ui=im[i+k];
|
||||
float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;float nc=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=nc;}}}}
|
||||
static std::vector<float> compute_fbank(std::vector<float> wav,int&T_out){
|
||||
for(auto&v:wav)v*=32768.0f;std::vector<float>win(WINLEN);for(int i=0;i<WINLEN;i++)win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));
|
||||
const int NB=NFFT/2+1;float bw=(float)FS/NFFT,ml=melf(LOWF),mh=melf(HIGHF),dm=(mh-ml)/(NMEL+1);
|
||||
std::vector<std::vector<float>>fb(NMEL,std::vector<float>(NB,0.0f));
|
||||
for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;for(int k=0;k<NB;k++){float mf=melf(bw*k);if(mf>L&&mf<R)fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}
|
||||
int N=wav.size(),T=(N-WINLEN)/SHIFT+1;std::vector<std::vector<float>>feat(T,std::vector<float>(NMEL));
|
||||
std::vector<float>re(NFFT),im(NFFT),fr(WINLEN);const float fl=1.1920929e-07f;
|
||||
for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT;double mn=0;for(int i=0;i<WINLEN;i++)mn+=s[i];mn/=WINLEN;
|
||||
for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn;for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1];fr[0]-=PREEMPH*fr[0];
|
||||
for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;}fftc(re,im,NFFT);
|
||||
for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NB;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]);feat[t][m]=logf(e>fl?e:fl);}}
|
||||
const int pad=(LFR_M-1)/2;int Tl=(T+LFR_N-1)/LFR_N;std::vector<std::vector<float>>pd;pd.reserve(T+pad+LFR_M);
|
||||
for(int i=0;i<pad;i++)pd.push_back(feat[0]);for(int t=0;t<T;t++)pd.push_back(feat[t]);while((int)pd.size()<(Tl-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);
|
||||
int D=LFR_M*NMEL;std::vector<float>out((size_t)Tl*D);for(int i=0;i<Tl;i++)for(int j=0;j<LFR_M;j++)memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));
|
||||
T_out=Tl;return out;}
|
||||
|
||||
// ===== model =====
|
||||
struct cfg{int d_model=512,enc_head=4,enc_blocks=50,enc_kernel=11,dec_blocks=16,dec_att=16,dec3=1,dec_head=4,dec_kernel=11,vocab=8404;float tail=0.45f,thresh=1.0f;};
|
||||
struct model{cfg c;ggml_context*ctx_w=nullptr;std::map<std::string,ggml_tensor*>t;
|
||||
ggml_tensor*g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;}
|
||||
bool has(const std::string&n){return t.count(n);} };
|
||||
|
||||
static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}
|
||||
static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}
|
||||
// FSMN depthwise (kernel stored [K,D]): out = sum_j w[:,j]*pad(v)[:, t+j] (+ residual v)
|
||||
static ggml_tensor* fsmn(ggml_context*c,ggml_tensor*v,ggml_tensor*fk,int D,int T,int K){
|
||||
const int pad=(K-1)/2;ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0);ggml_tensor*acc=v;
|
||||
for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]);
|
||||
acc=ggml_add(c,acc,ggml_mul(c,ggml_cont(c,sl),wj));}return acc;}
|
||||
// SAN-M encoder self-attn (fused qkv + fsmn)
|
||||
static ggml_tensor* enc_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T){
|
||||
const int D=m.c.d_model,H=m.c.enc_head,dk=D/H,K=m.c.enc_kernel;
|
||||
ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x);size_t nb1=qkv->nb[1];
|
||||
ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));
|
||||
ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));
|
||||
ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));
|
||||
ggml_tensor*fm=fsmn(c,v,m.g(p+"fsmn_block.weight"),D,T,K);
|
||||
q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3);k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);
|
||||
ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));
|
||||
ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
|
||||
ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
|
||||
return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fm);}
|
||||
static ggml_tensor* enc_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T,bool res){
|
||||
auto r=x;auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));auto sa=enc_attn(c,m,p+"self_attn.",h,T);
|
||||
x=res?ggml_add(c,r,sa):sa;r=x;h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
|
||||
h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h);h=ggml_relu(c,h);
|
||||
h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h);return ggml_add(c,r,h);}
|
||||
// decoder FFN-SANM: w_2(LayerNorm(relu(w_1(x)))) (w_2 has no bias)
|
||||
static ggml_tensor* dec_ffn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x){
|
||||
auto h=lin(c,m.g(p+"w_1.weight"),m.g(p+"w_1.bias"),x);h=ggml_relu(c,h);
|
||||
h=lnorm(c,h,m.g(p+"norm.weight"),m.g(p+"norm.bias"));return ggml_mul_mat(c,m.g(p+"w_2.weight"),h);}
|
||||
// cross attn: q=linear_q(tgt)[D,N], kv=linear_k_v(mem)[2D,T]
|
||||
static ggml_tensor* cross_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt,ggml_tensor*mem,int N,int T){
|
||||
const int D=m.c.d_model,H=m.c.dec_head,dk=D/H;
|
||||
ggml_tensor*q=lin(c,m.g(p+"linear_q.weight"),m.g(p+"linear_q.bias"),tgt); // [D,N]
|
||||
ggml_tensor*kv=lin(c,m.g(p+"linear_k_v.weight"),m.g(p+"linear_k_v.bias"),mem);// [2D,T]
|
||||
size_t nb1=kv->nb[1];
|
||||
ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,kv,D,T,nb1,0));
|
||||
ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,kv,D,T,nb1,(size_t)D*sizeof(float)));
|
||||
q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,N),0,2,1,3); // [dk,N,H]
|
||||
k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3); // [dk,T,H]
|
||||
ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3)); // [T,dk,H]
|
||||
ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk))); // [T,N,H]
|
||||
ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,N);
|
||||
return lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o);}
|
||||
static ggml_tensor* dec_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt,ggml_tensor*mem,int N,int T){
|
||||
const int D=m.c.d_model,K=m.c.dec_kernel;
|
||||
auto residual=tgt;auto h=lnorm(c,tgt,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
|
||||
h=dec_ffn(c,m,p+"feed_forward.",h); // FFN first
|
||||
auto y=lnorm(c,h,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
|
||||
auto sa=fsmn(c,y,m.g(p+"self_attn.fsmn_block.weight"),D,N,K); // FSMN self-attn (+residual y inside)
|
||||
auto x=ggml_add(c,residual,sa);
|
||||
residual=x;auto z=lnorm(c,x,m.g(p+"norm3.weight"),m.g(p+"norm3.bias"));
|
||||
auto ca=cross_attn(c,m,p+"src_attn.",z,mem,N,T);
|
||||
return ggml_add(c,residual,ca);}
|
||||
static ggml_tensor* dec3_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*tgt){
|
||||
auto h=lnorm(c,tgt,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));return dec_ffn(c,m,p+"feed_forward.",h);}
|
||||
static void add_posenc(std::vector<float>&x,int T,int depth){double inc=log(10000.0)/(depth/2.0-1.0);
|
||||
for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}}
|
||||
|
||||
// run a ggml graph on CPU, return output [ne0 x ne1] row-major (ne1 rows)
|
||||
static std::vector<float> run_graph(ggml_context*c,ggml_tensor*out,ggml_tensor*in1,const float*d1,ggml_tensor*in2,const float*d2){
|
||||
ggml_backend_t be=ggml_backend_cpu_init();ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false);ggml_build_forward_expand(gf,out);
|
||||
ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type());ggml_gallocr_alloc_graph(ga,gf);
|
||||
ggml_backend_tensor_set(in1,d1,0,ggml_nbytes(in1));if(in2)ggml_backend_tensor_set(in2,d2,0,ggml_nbytes(in2));
|
||||
ggml_backend_cpu_set_n_threads(be,8);ggml_backend_graph_compute(be,gf);
|
||||
int D=out->ne[0],N=out->ne[1];std::vector<float>r((size_t)D*N);ggml_backend_tensor_get(out,r.data(),0,ggml_nbytes(out));
|
||||
ggml_gallocr_free(ga);ggml_backend_free(be);return r;}
|
||||
|
||||
// Paraformer detok: tokens.json is BPE; join, drop "@@" continuations, "▁"(U+2581)->space.
|
||||
static std::string pf_trim(const std::string&s){size_t a=s.find_first_not_of(' ');if(a==std::string::npos)return "";size_t b=s.find_last_not_of(' ');return s.substr(a,b-a+1);}
|
||||
static std::string detok_pf(const std::vector<int>&ids,const std::vector<std::string>&vocab){
|
||||
std::string s; for(int id:ids){ if(id==1||id==2)continue; if(id>=0&&id<(int)vocab.size())s+=vocab[id]; }
|
||||
size_t p; while((p=s.find("@@"))!=std::string::npos)s.erase(p,2);
|
||||
const std::string lb="\xe2\x96\x81"; while((p=s.find(lb))!=std::string::npos)s.replace(p,3," ");
|
||||
return pf_trim(s);
|
||||
}
|
||||
|
||||
int main(int argc,char**argv){
|
||||
std::string gguf_path,wav_path,vad_path; int vad_maxseg=30000; bool ids_mode=false;
|
||||
for(int i=1;i<argc;i++){if(!strcmp(argv[i],"-m")&&i+1<argc)gguf_path=argv[++i];else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"--vad")&&i+1<argc)vad_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"--vad-maxseg")&&i+1<argc)vad_maxseg=atoi(argv[++i]);
|
||||
else if(!strcmp(argv[i],"--ids"))ids_mode=true;
|
||||
else{fprintf(stderr,"usage: %s -m paraformer.gguf -a audio.wav [--vad fsmn-vad.gguf [--vad-maxseg ms]] [--ids]\n",argv[0]);return 1;}}
|
||||
if(gguf_path.empty()||wav_path.empty()){fprintf(stderr,"missing args\n");return 1;}
|
||||
model m;gguf_init_params gp={false,&m.ctx_w};gguf_context*gg=gguf_init_from_file(gguf_path.c_str(),gp);if(!gg){fprintf(stderr,"gguf load failed\n");return 1;}
|
||||
auto rdi=[&](const char*k,int d){int i=gguf_find_key(gg,k);return i<0?d:(int)gguf_get_val_u32(gg,i);};
|
||||
auto rdf=[&](const char*k,float d){int i=gguf_find_key(gg,k);return i<0?d:gguf_get_val_f32(gg,i);};
|
||||
m.c.enc_blocks=rdi("pf.enc.num_blocks",50);m.c.dec_blocks=rdi("pf.dec.num_blocks",16);m.c.dec_att=rdi("pf.dec.att_layer_num",16);
|
||||
m.c.dec3=rdi("pf.dec.decoders3",1);m.c.vocab=rdi("pf.vocab_size",8404);m.c.tail=rdf("pf.predictor.tail_threshold",0.45f);m.c.thresh=rdf("pf.predictor.threshold",1.0f);
|
||||
std::vector<std::string> vocab; {int ki=gguf_find_key(gg,"pf.vocab"); if(ki>=0){int nv=gguf_get_arr_n(gg,ki); vocab.resize(nv); for(int i=0;i<nv;i++){const char*s=gguf_get_arr_str(gg,ki,i); vocab[i]=s?s:"";}}}
|
||||
for(int i=0;i<gguf_get_n_tensors(gg);i++){const char*nm=gguf_get_tensor_name(gg,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}gguf_free(gg);
|
||||
const int D=m.c.d_model,F=560,V=m.c.vocab;
|
||||
bool emit_ids = ids_mode || vocab.empty(); // fall back to ids if the gguf has no vocab
|
||||
|
||||
float*shift=(float*)m.g("cmvn.shift")->data,*scale=(float*)m.g("cmvn.scale")->data;
|
||||
float*cw=(float*)m.g("predictor.cif_conv1d.weight")->data; // [512,512,3] = [o][i][j]
|
||||
float*cb=(float*)m.g("predictor.cif_conv1d.bias")->data; // [512]
|
||||
float*ow=(float*)m.g("predictor.cif_output.weight")->data; // [1,512]
|
||||
float ob=((float*)m.g("predictor.cif_output.bias")->data)[0];
|
||||
|
||||
// Full pipeline (fbank -> CMVN -> encoder -> CIF -> decoder) on one wav window; prints token IDs.
|
||||
auto run_seg=[&](const std::vector<float>& wav){
|
||||
int T=0;auto fb=compute_fbank(wav,T); if(T<1)return;
|
||||
for(int t=0;t<T;t++)for(int d=0;d<F;d++)fb[(size_t)t*F+d]=(fb[(size_t)t*F+d]+shift[d])*scale[d]; // CMVN
|
||||
{float sc=sqrtf((float)D);for(auto&v:fb)v*=sc;}add_posenc(fb,T,F);
|
||||
std::vector<float>enc;
|
||||
{ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true};ggml_context*c=ggml_init(cp);
|
||||
ggml_tensor*x=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,T);ggml_set_input(x);
|
||||
ggml_tensor*h=enc_layer(c,m,"encoder.encoders0.0.",x,T,false);
|
||||
for(int i=0;i<m.c.enc_blocks-1;i++)h=enc_layer(c,m,"encoder.encoders."+std::to_string(i)+".",h,T,true);
|
||||
h=lnorm(c,h,m.g("encoder.after_norm.weight"),m.g("encoder.after_norm.bias"));ggml_set_output(h);
|
||||
enc=run_graph(c,h,x,fb.data(),nullptr,nullptr);ggml_free(c);} // enc row-major [T,D]
|
||||
// ===== CIF predictor (host): conv1d(k3,pad1)+residual+relu -> cif_output -> sigmoid -> alpha =====
|
||||
std::vector<float> outp((size_t)T*D); std::vector<float> alphas(T);
|
||||
for(int t=0;t<T;t++){
|
||||
for(int o=0;o<D;o++){ float acc=cb[o];
|
||||
for(int j=0;j<3;j++){int tt=t+j-1; if(tt<0||tt>=T)continue; const float*ev=&enc[(size_t)tt*D];
|
||||
const float*wo=&cw[(size_t)o*D*3]; for(int i=0;i<D;i++) acc+=wo[i*3+j]*ev[i];}
|
||||
outp[(size_t)t*D+o]=acc+enc[(size_t)t*D+o]; }
|
||||
float a=ob; for(int o=0;o<D;o++){float r=outp[(size_t)t*D+o]; if(r<0)r=0; a+=ow[o]*r;}
|
||||
float s=1.0f/(1.0f+expf(-a)); alphas[t]=s>0?s:0; }
|
||||
std::vector<float> hid=enc; hid.resize((size_t)(T+1)*D,0.0f);
|
||||
std::vector<float> al=alphas; al.push_back(m.c.tail); int L=T+1;
|
||||
std::vector<float> acoustic; acoustic.reserve(64*D);
|
||||
float integrate=0; std::vector<float> frame(D,0.0f);
|
||||
for(int t=0;t<L;t++){
|
||||
float alpha=al[t]; float dc=1.0f-integrate; integrate+=alpha;
|
||||
bool fire=integrate>=m.c.thresh; float cur=fire?dc:alpha; float rem=alpha-cur;
|
||||
for(int d=0;d<D;d++) frame[d]+=cur*hid[(size_t)t*D+d];
|
||||
if(fire){ acoustic.insert(acoustic.end(),frame.begin(),frame.end()); integrate-=1.0f;
|
||||
for(int d=0;d<D;d++) frame[d]=rem*hid[(size_t)t*D+d]; } }
|
||||
int N=acoustic.size()/D; if(N<1)return;
|
||||
std::vector<float> logits;
|
||||
{ggml_init_params cp={(size_t)2048*1024*1024,nullptr,true};ggml_context*c=ggml_init(cp);
|
||||
ggml_tensor*tgt=ggml_new_tensor_2d(c,GGML_TYPE_F32,D,N);ggml_set_input(tgt);
|
||||
ggml_tensor*mem=ggml_new_tensor_2d(c,GGML_TYPE_F32,D,T);ggml_set_input(mem);
|
||||
ggml_tensor*x=tgt;
|
||||
for(int i=0;i<m.c.dec_att;i++)x=dec_layer(c,m,"decoder.decoders."+std::to_string(i)+".",x,mem,N,T);
|
||||
for(int i=0;i<m.c.dec3;i++)x=dec3_layer(c,m,"decoder.decoders3."+std::to_string(i)+".",x);
|
||||
x=lnorm(c,x,m.g("decoder.after_norm.weight"),m.g("decoder.after_norm.bias"));
|
||||
x=lin(c,m.g("decoder.output_layer.weight"),m.g("decoder.output_layer.bias"),x); // [V,N]
|
||||
ggml_set_output(x);
|
||||
logits=run_graph(c,x,tgt,acoustic.data(),mem,enc.data());ggml_free(c);}
|
||||
std::vector<int> seg_ids; seg_ids.reserve(N);
|
||||
for(int n=0;n<N;n++){const float*col=&logits[(size_t)n*V];int am=0;float best=col[0];for(int v=1;v<V;v++)if(col[v]>best){best=col[v];am=v;}seg_ids.push_back(am);}
|
||||
if(emit_ids){ for(int id:seg_ids) printf("%d ",id); }
|
||||
else { std::string t=detok_pf(seg_ids,vocab); printf("%s",t.c_str()); }
|
||||
};
|
||||
|
||||
int64_t t0=ggml_time_us();
|
||||
std::vector<float>wav;if(!funasr_load_audio_16k_mono(wav_path.c_str(),wav)){fprintf(stderr,"read audio failed\n");return 1;}
|
||||
if(!vad_path.empty()){
|
||||
std::vector<std::pair<int,int>> segs;
|
||||
if(!funasr_vad_segments(vad_path,wav,vad_maxseg,segs)){fprintf(stderr,"vad failed\n");return 1;}
|
||||
for(auto&s:segs){ int off=(int)((int64_t)s.first*16000/1000), end=(int)((int64_t)s.second*16000/1000);
|
||||
if(end>(int)wav.size())end=wav.size(); if(end-off<WINLEN)continue;
|
||||
std::vector<float> seg(wav.begin()+off,wav.begin()+end); run_seg(seg); }
|
||||
fprintf(stderr,"[paraformer] %zu vad segments\n",segs.size());
|
||||
} else { run_seg(wav); }
|
||||
printf("\n");
|
||||
fprintf(stderr,"[paraformer] done %.2fs\n",(ggml_time_us()-t0)/1e6);
|
||||
if(m.ctx_w) ggml_free(m.ctx_w);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
# SenseVoiceSmall on llama.cpp / GGUF
|
||||
|
||||
Run **SenseVoiceSmall** on the [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
||||
/ ggml stack — **CPU, edge, a single binary, no Python at runtime**. Like
|
||||
[whisper.cpp](https://github.com/ggml-org/whisper.cpp), but for SenseVoice.
|
||||
|
||||
## Why this exists
|
||||
|
||||
SenseVoiceSmall normally runs on PyTorch / ONNX / libtorch. This runtime ports it
|
||||
to **ggml + GGUF** so it can run CPU-only, offline, embedded in a C/C++ app, with
|
||||
quantized weights. Use it on laptops / phones / edge boxes where there is no GPU
|
||||
and no Python. (For high-QPS GPU serving, the PyTorch/vLLM path is still the way.)
|
||||
|
||||
## Architecture
|
||||
|
||||
SenseVoiceSmall = **SAN-M encoder (70 layers) + CTC head** — no LLM, no autoregression.
|
||||
The whole pipeline runs in C++:
|
||||
|
||||
```
|
||||
audio.wav (16k mono)
|
||||
│ kaldi 80-mel fbank + LFR (C++)
|
||||
▼
|
||||
features [T, 560]
|
||||
│ prepend 4 query tokens [lang, event, emotion, itn]
|
||||
▼
|
||||
[4 + T, 560]
|
||||
│ SAN-M encoder (ggml) ── sensevoice-small.gguf
|
||||
▼
|
||||
encoder out [4+T, 512]
|
||||
│ CTC head (Linear 512→25055) → greedy CTC (argmax, dedup, drop blank)
|
||||
▼
|
||||
token ids
|
||||
│ SentencePiece detok (detok.py)
|
||||
▼
|
||||
<|zh|><|NEUTRAL|><|Speech|><|woitn|> transcription...
|
||||
```
|
||||
|
||||
The SAN-M encoder is the same architecture as Fun-ASR-Nano's, so the ggml forward
|
||||
is shared between the two runtimes.
|
||||
|
||||
## Quickstart
|
||||
|
||||
**1. Build:**
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
|
||||
cp -r /path/to/runtime/llama.cpp/funasr-sensevoice examples/
|
||||
echo 'add_subdirectory(funasr-sensevoice)' >> examples/CMakeLists.txt
|
||||
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
|
||||
cmake --build build -j --target llama-funasr-sensevoice
|
||||
```
|
||||
|
||||
**2. Convert weights** (needs the checkpoint, e.g. `FunAudioLLM/SenseVoiceSmall`):
|
||||
```bash
|
||||
python runtime/llama.cpp/export_sensevoice_gguf.py \
|
||||
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
||||
--out sensevoice-small.gguf # f32, ~936 MB
|
||||
python runtime/llama.cpp/export_sensevoice_gguf.py --wtype f16 \
|
||||
--model_pt <model>/model.pt --mvn <model>/am.mvn \
|
||||
--out sensevoice-small-f16.gguf # half size
|
||||
```
|
||||
|
||||
**3. Transcribe:**
|
||||
```bash
|
||||
build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a audio.wav # prints transcription text
|
||||
# --keep-tags keeps the <|lang|>/<|emotion|>/<|event|> tags; --ids prints raw CTC ids
|
||||
```
|
||||
Expected output:
|
||||
```
|
||||
我想问我在滨海新区有房我一直没有照顾孩子...你觉得这是正常的想法吗
|
||||
```
|
||||
The leading `<|...|>` tags are the predicted language / emotion / event / ITN.
|
||||
|
||||
## Accuracy & validation
|
||||
|
||||
- **CTC token ids (C++) vs PyTorch:** **identical** (108/108 on a benchmark clip).
|
||||
- **Detokenized text:** matches the FunASR `AutoModel` output **exactly**.
|
||||
- Encoder validated against PyTorch (shared with Fun-ASR-Nano runtime): cosine 1.0.
|
||||
- Encode time ≈ **1.3 s** on CPU for a 44 s clip.
|
||||
|
||||
## Tips & gotchas
|
||||
|
||||
- **No CMVN at inference.** SenseVoice `inference()` feeds the **raw** log-mel fbank
|
||||
to the encoder; it does **not** apply `am.mvn`. Applying CMVN makes the model
|
||||
predict `<|nospeech|>`. (The export script reads `am.mvn` for completeness but the
|
||||
runtime does not use it.)
|
||||
- **Query tokens (4)** are prepended from `embed.weight`, default indices
|
||||
`[language=auto(0), event=1, emotion=2, textnorm=woitn(15)]`. Change them for a
|
||||
fixed language or to enable ITN (`withitn=14`).
|
||||
- **WAV input** assumes 16 kHz mono PCM16.
|
||||
- LayerNorm eps = 1e-5; FSMN = exact f32 shift-accumulate; fbank matches torchaudio.
|
||||
|
||||
## Files
|
||||
```
|
||||
funasr-sensevoice/ ggml runtime: WAV → CTC token ids
|
||||
export_sensevoice_gguf.py export encoder + CTC head + query embeddings to GGUF
|
||||
detok.py SentencePiece id → text (bpe model ships with the checkpoint)
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
- Built-in SentencePiece detok (drop the Python step); arbitrary WAV formats;
|
||||
encoder Q8 quantization; timestamps.
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Detokenize SenseVoice CTC token ids -> text via the SentencePiece bpe model.
|
||||
Usage: python detok.py <bpe.model> <ids.txt> (ids.txt = space-separated ints)
|
||||
"""
|
||||
import sys
|
||||
import sentencepiece as spm
|
||||
|
||||
sp = spm.SentencePieceProcessor(model_file=sys.argv[1])
|
||||
ids = [int(x) for x in open(sys.argv[2]).read().split()]
|
||||
print(sp.decode(ids))
|
||||
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export SenseVoiceSmall (encoder + CTC head + query embeddings + CMVN) to GGUF
|
||||
for the ggml C++ runtime. The encoder is the same SAN-M architecture as
|
||||
Fun-ASR-Nano, so the C++ forward is shared.
|
||||
"""
|
||||
import argparse, os, re
|
||||
import numpy as np, torch, gguf
|
||||
|
||||
|
||||
def parse_mvn(path):
|
||||
"""am.mvn (kaldi nnet): two `<LearnRateCoef> 0 [ ... ]` blocks -> shift, scale.
|
||||
apply: out = (in + shift) * scale, per-dim (560)."""
|
||||
txt = open(path).read()
|
||||
blocks = re.findall(r"\[([^\]]*)\]", txt)
|
||||
shift = np.array([float(x) for x in blocks[0].split()], dtype=np.float32)
|
||||
scale = np.array([float(x) for x in blocks[1].split()], dtype=np.float32)
|
||||
return shift, scale
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model_pt", required=True)
|
||||
ap.add_argument("--mvn", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--wtype", default="f32", choices=["f32", "f16", "q8_0"])
|
||||
ap.add_argument("--spm", default=None, help="sentencepiece .bpe.model; default: next to model_pt")
|
||||
args = ap.parse_args()
|
||||
|
||||
sd = torch.load(args.model_pt, map_location="cpu"); sd = sd.get("state_dict", sd)
|
||||
w = gguf.GGUFWriter(args.out, "sensevoice-small")
|
||||
w.add_uint32("sv.input_size", 560)
|
||||
w.add_uint32("sv.output_size", 512)
|
||||
w.add_uint32("sv.attention_heads", 4)
|
||||
w.add_uint32("sv.num_blocks", 50)
|
||||
w.add_uint32("sv.tp_blocks", 20)
|
||||
w.add_uint32("sv.kernel_size", 11)
|
||||
w.add_uint32("sv.vocab_size", 25055)
|
||||
w.add_uint32("sv.blank_id", 0)
|
||||
# query token embed indices used at inference: [lid(auto=0), 1, 2, textnorm(woitn=15)]
|
||||
w.add_array("sv.query_tokens", [0, 1, 2, 14]) # 14=withitn (use_itn=True), matches authoritative
|
||||
import glob
|
||||
spm_path = args.spm or (glob.glob(os.path.join(os.path.dirname(args.model_pt), "*.bpe.model")) + [None])[0]
|
||||
if spm_path and os.path.exists(spm_path):
|
||||
import sentencepiece as spm
|
||||
sp = spm.SentencePieceProcessor(model_file=spm_path)
|
||||
pieces = [sp.id_to_piece(i) for i in range(sp.get_piece_size())]
|
||||
w.add_array("sv.vocab", pieces)
|
||||
print(f"embedded sv.vocab ({len(pieces)} pieces) from {spm_path}")
|
||||
else:
|
||||
print("WARNING: *.bpe.model not found - gguf will have no vocab (binary falls back to ids)")
|
||||
|
||||
shift, scale = parse_mvn(args.mvn)
|
||||
w.add_tensor("cmvn.shift", shift) # (560,)
|
||||
w.add_tensor("cmvn.scale", scale) # (560,)
|
||||
|
||||
n = 0
|
||||
for k, v in sd.items():
|
||||
if not (k.startswith("encoder.") or k.startswith("ctc.") or k == "embed.weight"):
|
||||
continue
|
||||
arr = v.detach().to(torch.float32).contiguous().numpy()
|
||||
if k.endswith("fsmn_block.weight"): # (D,1,K) -> (K,D)
|
||||
arr = np.ascontiguousarray(arr[:, 0, :].T)
|
||||
elif args.wtype == "f16" and arr.ndim == 2 and "norm" not in k:
|
||||
arr = arr.astype(np.float16)
|
||||
if args.wtype == "q8_0" and arr.ndim == 2 and "norm" not in k and "fsmn_block" not in k and k != "embed.weight" and arr.shape[1] % 32 == 0:
|
||||
from gguf import quants as _q, GGMLQuantizationType as _QT
|
||||
w.add_tensor(k, _q.quantize(arr, _QT.Q8_0), raw_dtype=_QT.Q8_0)
|
||||
else:
|
||||
w.add_tensor(k, arr)
|
||||
n += 1
|
||||
print(f"writing {n} tensors (+cmvn) to {args.out}")
|
||||
w.write_header_to_file(); w.write_kv_data_to_file(); w.write_tensors_to_file(); w.close()
|
||||
print(f"done: {args.out} ({os.path.getsize(args.out)/1e6:.1f} MB)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,6 @@
|
||||
set(TARGET llama-funasr-sensevoice)
|
||||
add_executable(${TARGET} funasr-sensevoice.cpp)
|
||||
install(TARGETS ${TARGET} RUNTIME)
|
||||
target_include_directories(${TARGET} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../funasr-common)
|
||||
target_link_libraries(${TARGET} PRIVATE ggml ${CMAKE_THREAD_LIBS_INIT})
|
||||
target_compile_features(${TARGET} PRIVATE cxx_std_17)
|
||||
@@ -0,0 +1,191 @@
|
||||
// funasr-sensevoice: SenseVoiceSmall (SAN-M encoder + CTC) on ggml.
|
||||
// fbank.bin (T x 560) -> CMVN -> prepend 4 query tokens -> SAN-M encoder ->
|
||||
// CTC head -> greedy CTC decode -> token ids (stdout).
|
||||
// The encoder is the same SAN-M arch as Fun-ASR-Nano (shared forward).
|
||||
// Detokenize the printed ids with the SentencePiece bpe model (Python side for now).
|
||||
|
||||
#include "ggml.h"
|
||||
#include "ggml-cpu.h"
|
||||
#include "ggml-alloc.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "gguf.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstdint>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
static const float LN_EPS = 1e-5f;
|
||||
|
||||
// ---- audio loader: any wav/mp3/flac, any rate/channels -> 16k mono (miniaudio) ----
|
||||
#define FUNASR_AUDIO_IMPLEMENTATION
|
||||
#include "funasr_audio.h"
|
||||
#include "funasr_vad.h" // built-in FSMN-VAD front end (--vad segmentation)
|
||||
#include <utility>
|
||||
static const int FS=16000,WINLEN=400,SHIFT=160,NFFT=512,NMEL=80,LFR_M=7,LFR_N=6;
|
||||
static const float PREEMPH=0.97f,LOWF=20.0f,HIGHF=8000.0f;
|
||||
static inline float melf(float f){return 1127.0f*logf(1.0f+f/700.0f);}
|
||||
static void fftc(std::vector<float>&re,std::vector<float>&im,int n){
|
||||
for(int i=1,j=0;i<n;i++){int b=n>>1;for(;j&b;b>>=1)j^=b;j^=b;if(i<j){std::swap(re[i],re[j]);std::swap(im[i],im[j]);}}
|
||||
for(int len=2;len<=n;len<<=1){double a=-2.0*M_PI/len;float wr=cosf(a),wi=sinf(a);
|
||||
for(int i=0;i<n;i+=len){float cr=1,ci=0;for(int k=0;k<len/2;k++){float ur=re[i+k],ui=im[i+k];
|
||||
float vr=re[i+k+len/2]*cr-im[i+k+len/2]*ci,vi=re[i+k+len/2]*ci+im[i+k+len/2]*cr;
|
||||
re[i+k]=ur+vr;im[i+k]=ui+vi;re[i+k+len/2]=ur-vr;im[i+k+len/2]=ui-vi;float nc=cr*wr-ci*wi;ci=cr*wi+ci*wr;cr=nc;}}}
|
||||
}
|
||||
static std::vector<float> compute_fbank(std::vector<float> wav,int&T_out){
|
||||
for(auto&v:wav)v*=32768.0f; std::vector<float> win(WINLEN);
|
||||
for(int i=0;i<WINLEN;i++)win[i]=0.54f-0.46f*cosf(2.0f*M_PI*i/(WINLEN-1));
|
||||
const int NBIN=NFFT/2+1; float bw=(float)FS/NFFT,ml=melf(LOWF),mh=melf(HIGHF),dm=(mh-ml)/(NMEL+1);
|
||||
std::vector<std::vector<float>> fb(NMEL,std::vector<float>(NBIN,0.0f));
|
||||
for(int m=0;m<NMEL;m++){float L=ml+m*dm,C=ml+(m+1)*dm,R=ml+(m+2)*dm;
|
||||
for(int k=0;k<NBIN;k++){float mf=melf(bw*k); if(mf>L&&mf<R)fb[m][k]=mf<=C?(mf-L)/(C-L):(R-mf)/(R-C);}}
|
||||
int N=wav.size(),T=(N-WINLEN)/SHIFT+1; std::vector<std::vector<float>> feat(T,std::vector<float>(NMEL));
|
||||
std::vector<float> re(NFFT),im(NFFT),fr(WINLEN); const float fl=1.1920929e-07f;
|
||||
for(int t=0;t<T;t++){const float*s=wav.data()+t*SHIFT; double mn=0; for(int i=0;i<WINLEN;i++)mn+=s[i]; mn/=WINLEN;
|
||||
for(int i=0;i<WINLEN;i++)fr[i]=s[i]-(float)mn; for(int i=WINLEN-1;i>0;i--)fr[i]-=PREEMPH*fr[i-1]; fr[0]-=PREEMPH*fr[0];
|
||||
for(int i=0;i<NFFT;i++){re[i]=i<WINLEN?fr[i]*win[i]:0.0f;im[i]=0.0f;} fftc(re,im,NFFT);
|
||||
for(int m=0;m<NMEL;m++){float e=0;for(int k=0;k<NBIN;k++)if(fb[m][k]>0)e+=fb[m][k]*(re[k]*re[k]+im[k]*im[k]); feat[t][m]=logf(e>fl?e:fl);}}
|
||||
const int pad=(LFR_M-1)/2; int Tl=(T+LFR_N-1)/LFR_N; std::vector<std::vector<float>> pd; pd.reserve(T+pad+LFR_M);
|
||||
for(int i=0;i<pad;i++)pd.push_back(feat[0]); for(int t=0;t<T;t++)pd.push_back(feat[t]);
|
||||
while((int)pd.size()<(Tl-1)*LFR_N+LFR_M)pd.push_back(feat[T-1]);
|
||||
int D=LFR_M*NMEL; std::vector<float> out((size_t)Tl*D);
|
||||
for(int i=0;i<Tl;i++)for(int j=0;j<LFR_M;j++)memcpy(&out[(size_t)i*D+j*NMEL],pd[i*LFR_N+j].data(),NMEL*sizeof(float));
|
||||
T_out=Tl; return out;
|
||||
}
|
||||
|
||||
struct cfg { int d_model=512,n_head=4,num_blocks=50,tp_blocks=20,kernel=11,vocab=25055,blank=0; };
|
||||
struct model { cfg c; ggml_context*ctx_w=nullptr; std::map<std::string,ggml_tensor*> t;
|
||||
ggml_tensor* g(const std::string&n){auto it=t.find(n);if(it==t.end()){fprintf(stderr,"missing %s\n",n.c_str());exit(1);}return it->second;} };
|
||||
|
||||
static ggml_tensor* lin(ggml_context*c,ggml_tensor*w,ggml_tensor*b,ggml_tensor*x){auto y=ggml_mul_mat(c,w,x);return b?ggml_add(c,y,b):y;}
|
||||
static ggml_tensor* lnorm(ggml_context*c,ggml_tensor*x,ggml_tensor*g,ggml_tensor*b){return ggml_add(c,ggml_mul(c,ggml_norm(c,x,LN_EPS),g),b);}
|
||||
static ggml_tensor* sanm_attn(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T){
|
||||
const int D=m.c.d_model,H=m.c.n_head,dk=D/H,K=m.c.kernel;
|
||||
ggml_tensor*qkv=lin(c,m.g(p+"linear_q_k_v.weight"),m.g(p+"linear_q_k_v.bias"),x); size_t nb1=qkv->nb[1];
|
||||
ggml_tensor*q=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,0));
|
||||
ggml_tensor*k=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)D*sizeof(float)));
|
||||
ggml_tensor*v=ggml_cont(c,ggml_view_2d(c,qkv,D,T,nb1,(size_t)2*D*sizeof(float)));
|
||||
const int pad=(K-1)/2; ggml_tensor*fk=m.g(p+"fsmn_block.weight");
|
||||
ggml_tensor*vp=ggml_pad_ext(c,v,0,0,pad,pad,0,0,0,0); ggml_tensor*fsmn=v;
|
||||
for(int j=0;j<K;j++){auto sl=ggml_view_2d(c,vp,D,T,vp->nb[1],(size_t)j*vp->nb[1]);
|
||||
auto wj=ggml_view_1d(c,fk,D,(size_t)j*fk->nb[1]); fsmn=ggml_add(c,fsmn,ggml_mul(c,ggml_cont(c,sl),wj));}
|
||||
q=ggml_permute(c,ggml_reshape_3d(c,q,dk,H,T),0,2,1,3); k=ggml_permute(c,ggml_reshape_3d(c,k,dk,H,T),0,2,1,3);
|
||||
ggml_tensor*vh=ggml_cont(c,ggml_permute(c,ggml_reshape_3d(c,v,dk,H,T),1,2,0,3));
|
||||
ggml_tensor*kq=ggml_soft_max(c,ggml_scale(c,ggml_mul_mat(c,k,q),1.0f/sqrtf((float)dk)));
|
||||
ggml_tensor*o=ggml_cont_2d(c,ggml_permute(c,ggml_mul_mat(c,vh,kq),0,2,1,3),D,T);
|
||||
return ggml_add(c,lin(c,m.g(p+"linear_out.weight"),m.g(p+"linear_out.bias"),o),fsmn);
|
||||
}
|
||||
static ggml_tensor* sanm_layer(ggml_context*c,model&m,const std::string&p,ggml_tensor*x,int T,bool res){
|
||||
auto r=x; auto h=lnorm(c,x,m.g(p+"norm1.weight"),m.g(p+"norm1.bias"));
|
||||
auto sa=sanm_attn(c,m,p+"self_attn.",h,T); x=res?ggml_add(c,r,sa):sa; r=x;
|
||||
h=lnorm(c,x,m.g(p+"norm2.weight"),m.g(p+"norm2.bias"));
|
||||
h=lin(c,m.g(p+"feed_forward.w_1.weight"),m.g(p+"feed_forward.w_1.bias"),h); h=ggml_relu(c,h);
|
||||
h=lin(c,m.g(p+"feed_forward.w_2.weight"),m.g(p+"feed_forward.w_2.bias"),h); return ggml_add(c,r,h);
|
||||
}
|
||||
static void add_posenc(std::vector<float>&x,int T,int depth){
|
||||
double inc=log(10000.0)/(depth/2.0-1.0);
|
||||
for(int t=0;t<T;t++){double pos=t+1;for(int i=0;i<depth/2;i++){double its=exp(i*-inc),st=pos*its;
|
||||
x[(size_t)t*depth+i]+=(float)sin(st);x[(size_t)t*depth+depth/2+i]+=(float)cos(st);}}
|
||||
}
|
||||
|
||||
// SenseVoice detok: sentencepiece pieces (no byte-fallback in this vocab) -> join,
|
||||
// "▁"(U+2581)->space; meta tokens <|lang|>/<|emo|>/<|event|>/<|itn|> dropped unless --keep-tags.
|
||||
static std::string sv_trim(const std::string&s){size_t a=s.find_first_not_of(' ');if(a==std::string::npos)return "";size_t b=s.find_last_not_of(' ');return s.substr(a,b-a+1);}
|
||||
static std::string detok_sv(const std::vector<int>&ids,const std::vector<std::string>&vocab,bool keep_tags){
|
||||
std::string s; for(int id:ids){ if(id<0||id>=(int)vocab.size())continue; const std::string&p=vocab[id];
|
||||
if(!keep_tags && p.size()>=2 && p[0]=='<' && p[1]=='|') continue; // skip <|...|> meta
|
||||
s+=p; }
|
||||
const std::string lb="\xe2\x96\x81"; size_t pp; while((pp=s.find(lb))!=std::string::npos)s.replace(pp,3," ");
|
||||
return sv_trim(s);
|
||||
}
|
||||
|
||||
int main(int argc,char**argv){
|
||||
std::string gguf_path,fbank_path,wav_path,vad_path; int vad_maxseg=30000; bool ids_mode=false,keep_tags=false;
|
||||
for(int i=1;i<argc;i++){ if(!strcmp(argv[i],"-m")&&i+1<argc)gguf_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"-f")&&i+1<argc)fbank_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"-a")&&i+1<argc)wav_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"--vad")&&i+1<argc)vad_path=argv[++i];
|
||||
else if(!strcmp(argv[i],"--vad-maxseg")&&i+1<argc)vad_maxseg=atoi(argv[++i]);
|
||||
else if(!strcmp(argv[i],"--ids"))ids_mode=true;
|
||||
else if(!strcmp(argv[i],"--keep-tags"))keep_tags=true;
|
||||
else {fprintf(stderr,"usage: %s -m sensevoice.gguf (-a audio.wav | -f fbank.bin) [--vad fsmn-vad.gguf [--vad-maxseg ms]] [--ids] [--keep-tags]\n",argv[0]);return 1;} }
|
||||
if(gguf_path.empty()||(fbank_path.empty()&&wav_path.empty())){fprintf(stderr,"missing args\n");return 1;}
|
||||
|
||||
// load model
|
||||
model m; gguf_init_params gp={false,&m.ctx_w}; gguf_context*gg=gguf_init_from_file(gguf_path.c_str(),gp);
|
||||
if(!gg){fprintf(stderr,"load gguf failed\n");return 1;}
|
||||
auto rd=[&](const char*k,int d){int i=gguf_find_key(gg,k);return i<0?d:(int)gguf_get_val_u32(gg,i);};
|
||||
m.c.d_model=rd("sv.output_size",512); m.c.n_head=rd("sv.attention_heads",4);
|
||||
m.c.num_blocks=rd("sv.num_blocks",50); m.c.tp_blocks=rd("sv.tp_blocks",20);
|
||||
m.c.kernel=rd("sv.kernel_size",11); m.c.vocab=rd("sv.vocab_size",25055); m.c.blank=rd("sv.blank_id",0);
|
||||
int qi=gguf_find_key(gg,"sv.query_tokens"); int nq=qi<0?0:(int)gguf_get_arr_n(gg,qi);
|
||||
std::vector<int> qtok(nq); for(int i=0;i<nq;i++) qtok[i]=((const int32_t*)gguf_get_arr_data(gg,qi))[i];
|
||||
std::vector<std::string> vocab; {int ki=gguf_find_key(gg,"sv.vocab"); if(ki>=0){int nv=gguf_get_arr_n(gg,ki); vocab.resize(nv); for(int i=0;i<nv;i++){const char*s=gguf_get_arr_str(gg,ki,i); vocab[i]=s?s:"";}}}
|
||||
for(int i=0;i<gguf_get_n_tensors(gg);i++){const char*nm=gguf_get_tensor_name(gg,i);m.t[nm]=ggml_get_tensor(m.ctx_w,nm);}
|
||||
gguf_free(gg);
|
||||
const int F=560, D=m.c.d_model, V=m.c.vocab;
|
||||
bool emit_ids = ids_mode || vocab.empty(); // fall back to ids if the gguf has no vocab
|
||||
|
||||
// NOTE: SenseVoiceSmall inference() feeds the RAW log-mel fbank to the encoder;
|
||||
// it does NOT apply am.mvn CMVN (that path is unused at inference). Applying it
|
||||
// makes the encoder predict <|nospeech|>. So no CMVN here.
|
||||
float*emb=(float*)m.g("embed.weight")->data; // [16, 560] row-major
|
||||
// Run encoder+CTC on one fbank window [T,F]; prints greedy-CTC token IDs (no newline).
|
||||
auto run_seg=[&](const std::vector<float>& fb,int T){
|
||||
int N=nq+T; std::vector<float> inp((size_t)N*F);
|
||||
for(int i=0;i<nq;i++) memcpy(&inp[(size_t)i*F], &emb[(size_t)qtok[i]*F], F*sizeof(float));
|
||||
memcpy(&inp[(size_t)nq*F], fb.data(), (size_t)T*F*sizeof(float));
|
||||
float sc=sqrtf((float)D); for(auto&v:inp)v*=sc; add_posenc(inp,N,F);
|
||||
ggml_backend_t be=ggml_backend_cpu_init();
|
||||
ggml_init_params cp={(size_t)1024*1024*1024,nullptr,true}; ggml_context*c=ggml_init(cp);
|
||||
ggml_tensor*x=ggml_new_tensor_2d(c,GGML_TYPE_F32,F,N); ggml_set_input(x);
|
||||
ggml_tensor*h=sanm_layer(c,m,"encoder.encoders0.0.",x,N,false);
|
||||
for(int i=0;i<m.c.num_blocks-1;i++) h=sanm_layer(c,m,"encoder.encoders."+std::to_string(i)+".",h,N,true);
|
||||
h=lnorm(c,h,m.g("encoder.after_norm.weight"),m.g("encoder.after_norm.bias"));
|
||||
for(int i=0;i<m.c.tp_blocks;i++) h=sanm_layer(c,m,"encoder.tp_encoders."+std::to_string(i)+".",h,N,true);
|
||||
h=lnorm(c,h,m.g("encoder.tp_norm.weight"),m.g("encoder.tp_norm.bias"));
|
||||
ggml_tensor*logits=lin(c,m.g("ctc.ctc_lo.weight"),m.g("ctc.ctc_lo.bias"),h); // [V, N]
|
||||
ggml_set_output(logits);
|
||||
ggml_cgraph*gf=ggml_new_graph_custom(c,32768,false); ggml_build_forward_expand(gf,logits);
|
||||
ggml_gallocr_t ga=ggml_gallocr_new(ggml_backend_cpu_buffer_type()); ggml_gallocr_alloc_graph(ga,gf);
|
||||
ggml_backend_tensor_set(x,inp.data(),0,ggml_nbytes(x)); ggml_backend_cpu_set_n_threads(be,8);
|
||||
if(ggml_backend_graph_compute(be,gf)!=GGML_STATUS_SUCCESS){fprintf(stderr,"compute failed\n");}
|
||||
std::vector<float> lg((size_t)V*N); ggml_backend_tensor_get(logits,lg.data(),0,ggml_nbytes(logits));
|
||||
std::vector<int> seg_ids; int prev=-1; // greedy CTC: argmax per frame -> collapse -> drop blank
|
||||
for(int n=0;n<N;n++){ const float*col=&lg[(size_t)n*V]; int am=0; float best=col[0];
|
||||
for(int v=1;v<V;v++) if(col[v]>best){best=col[v];am=v;}
|
||||
if(am!=prev && am!=m.c.blank) seg_ids.push_back(am); prev=am; }
|
||||
if(emit_ids){ for(int id:seg_ids) printf("%d ",id); }
|
||||
else { std::string t=detok_sv(seg_ids,vocab,keep_tags); printf("%s",t.c_str()); }
|
||||
ggml_gallocr_free(ga); ggml_free(c); ggml_backend_free(be);
|
||||
};
|
||||
|
||||
int64_t t0=ggml_time_us();
|
||||
if(!vad_path.empty()){
|
||||
std::vector<float> wav; if(!funasr_load_audio_16k_mono(wav_path.c_str(),wav)){fprintf(stderr,"read audio failed\n");return 1;}
|
||||
std::vector<std::pair<int,int>> segs;
|
||||
if(!funasr_vad_segments(vad_path,wav,vad_maxseg,segs)){fprintf(stderr,"vad failed\n");return 1;}
|
||||
for(auto&s:segs){ int off=(int)((int64_t)s.first*16000/1000), end=(int)((int64_t)s.second*16000/1000);
|
||||
if(end>(int)wav.size())end=wav.size(); if(end-off<WINLEN)continue;
|
||||
std::vector<float> seg(wav.begin()+off,wav.begin()+end); int t=0; auto fb=compute_fbank(seg,t); run_seg(fb,t); }
|
||||
fprintf(stderr,"[sensevoice] %zu vad segments\n",segs.size());
|
||||
} else {
|
||||
int32_t T=0,Fc=F; std::vector<float> fb;
|
||||
if(!wav_path.empty()){
|
||||
std::vector<float> wav; if(!funasr_load_audio_16k_mono(wav_path.c_str(),wav)){fprintf(stderr,"read audio failed\n");return 1;}
|
||||
int t=0; fb=compute_fbank(wav,t); T=t;
|
||||
} else {
|
||||
FILE*f=fopen(fbank_path.c_str(),"rb"); if(!f){fprintf(stderr,"open fbank\n");return 1;}
|
||||
if(fread(&T,4,1,f)!=1||fread(&Fc,4,1,f)!=1){fclose(f);return 1;}
|
||||
fb.resize((size_t)T*Fc); if((int)fread(fb.data(),4,fb.size(),f)!=(int)fb.size()){fclose(f);return 1;} fclose(f);
|
||||
}
|
||||
run_seg(fb,T);
|
||||
}
|
||||
printf("\n");
|
||||
fprintf(stderr,"[sensevoice] done %.2fs\n",(ggml_time_us()-t0)/1e6);
|
||||
if(m.ctx_w) ggml_free(m.ctx_w);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
# Regression tests
|
||||
|
||||
Run each runtime tool on a fixed clip (`sample.wav`, ~6 s) and diff the output against
|
||||
the frozen golden in `golden/`. Catches regressions in the ggml graphs, the FSMN-VAD
|
||||
state machine, the CIF predictor and CTC decode.
|
||||
|
||||
## Run
|
||||
```bash
|
||||
# from runtime/llama.cpp/, after building (cmake --build build):
|
||||
./tests/run_regression.sh # VAD (tiny model auto-fetched) + any tool whose GGUF is already local
|
||||
RUN_FULL=1 ./tests/run_regression.sh # also download the ASR GGUFs from Hugging Face and test every tool
|
||||
```
|
||||
`BIN_DIR` / `MODELS_DIR` override where binaries and GGUFs are found. Exit code is
|
||||
non-zero if any test fails; tools with no binary or no model are skipped.
|
||||
|
||||
## Golden
|
||||
Captured on Linux x86-64 (the reference platform) with the f16 GGUFs published at
|
||||
`FunAudioLLM/*-GGUF`. Update a golden file only with a deliberate, reviewed output change.
|
||||
@@ -0,0 +1 @@
|
||||
我想问我在滨海新区有房
|
||||
@@ -0,0 +1 @@
|
||||
我想问我在滨海新区有房
|
||||
@@ -0,0 +1 @@
|
||||
我想问我在滨海新区有房
|
||||
@@ -0,0 +1 @@
|
||||
770 5980
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Numerical regression test for the FunASR llama.cpp runtime.
|
||||
# Runs each available tool on a fixed clip (sample.wav) and diffs the output against
|
||||
# the frozen golden in golden/. Catches regressions in the ggml graphs, the FSMN-VAD
|
||||
# state machine, CIF predictor and CTC decode. Golden captured on Linux x86-64 (the
|
||||
# reference platform) with the f16 GGUFs published on Hugging Face.
|
||||
#
|
||||
# ./run_regression.sh # test tools whose binary+model are present (VAD model auto-fetched)
|
||||
# RUN_FULL=1 ./run_regression.sh # also download the ASR GGUFs and test every tool
|
||||
# BIN_DIR=/path/to/bin MODELS_DIR=/path/to/gguf ./run_regression.sh
|
||||
set -u
|
||||
DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
RT=$(cd "$DIR/.." && pwd)
|
||||
BIN="${BIN_DIR:-$RT/build/bin}"
|
||||
MODELS="${MODELS_DIR:-$DIR/models}"
|
||||
SAMPLE="$DIR/sample.wav"
|
||||
RUN_FULL="${RUN_FULL:-0}"
|
||||
mkdir -p "$MODELS"
|
||||
pass=0; fail=0; skip=0
|
||||
|
||||
bin(){ if [ -x "$BIN/$1" ]; then echo "$BIN/$1"; elif command -v "$1" >/dev/null 2>&1; then echo "$1"; fi; }
|
||||
# ensure the listed model files exist in $MODELS (download via the repo script if allowed)
|
||||
ensure_models(){ local key="$1"; shift
|
||||
local missing=0; for f in "$@"; do [ -f "$MODELS/$f" ] || missing=1; done
|
||||
[ "$missing" = 0 ] && return 0
|
||||
{ [ "$key" = fsmn-vad ] || [ "$RUN_FULL" = 1 ]; } || return 1 # only auto-fetch the tiny VAD unless RUN_FULL
|
||||
"$RT/download-funasr-model.sh" "$key" "$MODELS" >/dev/null 2>&1 || return 1
|
||||
for f in "$@"; do [ -f "$MODELS/$f" ] || return 1; done; return 0
|
||||
}
|
||||
check(){ local name="$1" golden="$2" got="$3"
|
||||
if [ "$got" = "$(cat "$golden")" ]; then echo " PASS $name"; pass=$((pass+1))
|
||||
else echo " FAIL $name"; echo " expected: $(head -c 80 "$golden")"; echo " got: $(printf %s "$got" | head -c 80)"; fail=$((fail+1)); fi
|
||||
}
|
||||
skipper(){ echo " SKIP $1 ($2)"; skip=$((skip+1)); }
|
||||
|
||||
run_tool(){ # name binary golden key models... -- run...
|
||||
local name="$1" b key gold; b=$(bin "$2"); gold="$DIR/golden/$3"; key="$4"; shift 4
|
||||
local models=(); while [ "$1" != "--" ]; do models+=("$1"); shift; done; shift
|
||||
[ -n "$b" ] || { skipper "$name" "no binary"; return; }
|
||||
[ -f "$gold" ] || { skipper "$name" "no golden"; return; }
|
||||
ensure_models "$key" "${models[@]}" || { skipper "$name" "model missing (set RUN_FULL=1)"; return; }
|
||||
check "$name" "$gold" "$("$@" 2>/dev/null)"
|
||||
}
|
||||
|
||||
echo "== FunASR llama.cpp regression (sample.wav) =="
|
||||
B=$(bin llama-funasr-vad) && run_tool vad llama-funasr-vad vad.txt fsmn-vad fsmn-vad.gguf -- "$B" -m "$MODELS/fsmn-vad.gguf" -a "$SAMPLE"
|
||||
B=$(bin llama-funasr-sensevoice) && run_tool sensevoice llama-funasr-sensevoice sensevoice.txt sensevoice sensevoice-small-f16.gguf -- "$B" -m "$MODELS/sensevoice-small-f16.gguf" -a "$SAMPLE"
|
||||
B=$(bin llama-funasr-paraformer) && run_tool paraformer llama-funasr-paraformer paraformer.txt paraformer paraformer-f16.gguf -- "$B" -m "$MODELS/paraformer-f16.gguf" -a "$SAMPLE"
|
||||
B=$(bin llama-funasr-cli) && run_tool nano llama-funasr-cli nano.txt nano funasr-encoder-f16.gguf qwen3-0.6b-q8_0.gguf -- "$B" --enc "$MODELS/funasr-encoder-f16.gguf" -m "$MODELS/qwen3-0.6b-q8_0.gguf" -a "$SAMPLE"
|
||||
|
||||
echo "== $pass passed, $fail failed, $skip skipped =="
|
||||
[ "$fail" = 0 ]
|
||||
Binary file not shown.
Reference in New Issue
Block a user