modelscope--funasr/runtime/llama.cpp/benchmarks

Reproducing the FunASR-vs-whisper.cpp benchmark

Scripts and method behind ../BENCHMARKS.md.

Metric (authoritative FunASR口径)

• micro-CER = Σ edit_distance / Σ reference_chars over all files (not a per-file mean).
• normalize_zh: re.sub(r'[^\w一-鿿]', '', text).upper() — drop punctuation/whitespace, keep word chars + CJK, upper-case. (SenseVoice meta tags <|...|> are stripped first.)
• RTF = Σ compute_time / Σ audio_duration, model-load time excluded.

compute_cer.py implements exactly this:

python compute_cer.py --refs testset.json --hyp_dir <hyps>/ [--time_file <times>.txt]

testset.json is a list of {"id"/"key", "ref", "duration"}; <hyps>/{key}.txt are the transcripts; <times>.txt has key compute_seconds per line.

Producing hypotheses

FunASR (this runtime), per clip:

# SenseVoice / Paraformer: ids -> detok
build/bin/llama-funasr-sensevoice -m sensevoice-small.gguf -a $k.wav > $k.ids
python ../sensevoice/detok.py <model>/chn_jpn_yue_eng_ko_spectok.bpe.model $k.ids > $k.txt
build/bin/llama-funasr-paraformer -m paraformer.gguf -a $k.wav > $k.ids
python ../paraformer/detok_paraformer.py <model>/tokens.json $k.ids > $k.txt
# Fun-ASR-Nano: text directly
build/bin/llama-funasr-cli --enc funasr-encoder.gguf -m qwen3-0.6b-q8_0.gguf -a $k.wav --chunk 15 > $k.txt

Compute time is on each tool's stderr (encode … s / enc … dec … s).

whisper.cpp, per clip (forced Chinese, no timestamps):

whisper-cli -m models/ggml-<size>.bin -l zh -nt -t 8 $k.wav > $k.txt   # 2>stderr has "total time"/"load time"

RTF compute time = (total − load) ms.

Notes

• Run all systems with the same thread count (here -t 8 / 8 threads) for a fair RTF.
• Whisper does its own internal 30 s windowing; the FunASR segmentation口径 is documented in BENCHMARKS.md (see the methodology/caveats sections).