wehub-resource-sync
73 lines
2.9 KiB
Markdown
73 lines
2.9 KiB
Markdown
# FunASR Migration Benchmark Example
|
|
|
|
Use this example when you are comparing FunASR with Whisper, OpenAI audio APIs, or a cloud ASR provider. It runs FunASR over your representative audio set and writes machine-readable results plus a Markdown summary.
|
|
|
|
The script does not claim accuracy by itself. Run your baseline on the same files, then compare transcripts with human review or your normal WER/CER workflow.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
python examples/migration/benchmark_funasr.py \
|
|
--input /path/to/audio_samples \
|
|
--recursive \
|
|
--model iic/SenseVoiceSmall \
|
|
--device cuda \
|
|
--spk-model cam++ \
|
|
--output-dir outputs/funasr_migration_eval \
|
|
--metadata baseline=whisper-large-v3
|
|
```
|
|
|
|
Outputs:
|
|
|
|
- `results.jsonl`: one JSON object per audio file with text, elapsed seconds, audio duration, realtime factor, model, device, and errors.
|
|
- `summary.md`: run configuration, aggregate speed, per-file previews, and next comparison steps.
|
|
|
|
## CPU smoke test
|
|
|
|
For a portable first check, use CPU and a small audio folder:
|
|
|
|
```bash
|
|
python examples/migration/benchmark_funasr.py \
|
|
--input ./samples \
|
|
--model iic/SenseVoiceSmall \
|
|
--device cpu \
|
|
--output-dir outputs/funasr_cpu_smoke
|
|
```
|
|
|
|
|
|
## 中文快速说明
|
|
|
|
这个示例用于从 Whisper、OpenAI 音频 API 或云端 ASR 迁移前的本地评测。请用同一批代表性音频分别跑旧方案和 FunASR，再用人工审阅或 WER/CER 流程比较质量。
|
|
|
|
```bash
|
|
python examples/migration/benchmark_funasr.py \
|
|
--input /path/to/audio_samples \
|
|
--recursive \
|
|
--model iic/SenseVoiceSmall \
|
|
--device cuda \
|
|
--spk-model cam++ \
|
|
--output-dir outputs/funasr_migration_eval \
|
|
--metadata baseline=whisper-large-v3
|
|
```
|
|
|
|
输出文件：
|
|
|
|
- `results.jsonl`：每条音频的文本、耗时、音频时长、实时倍速和错误信息。
|
|
- `summary.md`：运行配置、总体速度、逐文件预览和下一步对比建议。
|
|
|
|
如果结果可以公开，欢迎提交 [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md)，帮助其他用户参考你的硬件、音频领域和质量记录。
|
|
|
|
## What to compare
|
|
|
|
Track the same fields for your old ASR stack and FunASR:
|
|
|
|
| Field | Why it matters |
|
|
|---|---|
|
|
| Audio duration, language, domain, sample rate, speaker count | Keeps the comparison representative. |
|
|
| Model name, version, device, CUDA/PyTorch versions | Makes results reproducible. |
|
|
| Model load time vs inference time | Separates cold start from steady-state throughput. |
|
|
| WER/CER or human review notes | Captures quality beyond speed. |
|
|
| Failed-file rate and error messages | Shows operational risk before rollout. |
|
|
|
|
See the [migration guide](../../docs/migration_from_whisper.md) for the full evaluation and rollout checklist. If you can share results publicly, open a [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) so others can learn from your hardware, audio domain, and quality notes.
|