wehub/modelscope--funasr
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:10 +08:00
commit c397331b1e
3684 changed files with 990993 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Makefile
+21
View File
@@ -0,0 +1,21 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = FunASR
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/README.md
+19
View File
@@ -0,0 +1,19 @@
# FunASR document generation
## Generate HTML
For convenience, we provide users with the ability to generate local HTML manually.
First, you should install the following packages, which is required for building HTML:
```sh
pip3 install -U "funasr[docs]"
```
Then you can generate HTML manually.
```sh
cd docs
make html
```
The generated files are all contained in the "FunASR/docs/_build" directory. You can access the FunASR documentation by simply opening the "html/index.html" file in your browser from this directory.
docs/benchmark/realtime_ws_benchmark.md
+92
View File
@@ -0,0 +1,92 @@
# Realtime WebSocket Benchmark
Use this benchmark when you need to measure the client-observable behavior of
`examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py` under
real streaming traffic. Offline `RTFx` and realtime service latency are
different metrics: this page focuses on first update latency, final latency
after `STOP`, response lag, and multi-client behavior.
The benchmark client accepts only 16 kHz mono PCM16 WAV input. Keeping the input
format strict removes resampling and file decoding from the measurement.
## Start the Service
For long continuous speech or multiple browser clients, start with a bounded
partial window and a moderate partial refresh interval:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
Speaker diarization is disabled by default. Add `--enable-spk` only when the
`spk` field is required, and report that setting with the benchmark result.
## Run a Single Realtime Replay
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 1 \
--output-jsonl realtime_ws_1c.jsonl
```
With pacing enabled, the client sends audio at realtime speed using 100 ms
frames. This is the closest mode to a microphone or browser stream.
## Run Concurrent Replays
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 8 \
--loops 3 \
--chunk-ms 100 \
--language 中文 \
--output-jsonl realtime_ws_8c.jsonl
```
Use a representative audio file. A long, pauseless monologue creates a very
different load shape from turn-taking meetings, because nearly every client is
speaking and triggering partial decodes at the same time.
For an unpaced stress test, add `--no-pace`. Treat that result as a throughput
stress signal, not as user-facing realtime latency.
## Metrics
| Metric | Meaning |
|--------|---------|
| `aggregate_audio_per_wall` | Total input audio seconds across all clients divided by benchmark wall time |
| `first_update_ms_p50/p95` | Time from first audio frame to first result message with `sentences`, `partial`, or `is_final` |
| `final_after_stop_ms_p50/p95` | Time from sending `STOP` to receiving the final result |
| `client_response_lag_ms_p95_max` | Largest per-client p95 of non-final `(client receive time - audio start) - server duration_ms`; useful mainly in paced mode for preview/partial lag |
| `partial_messages` | Count of non-final result messages with a non-empty `partial` |
| `final_messages` | Count of final result messages |
| `errors` | Connection, timeout, protocol, or client-side validation errors |
The script can observe only client-side timing and fields returned by the
server. If you are debugging service internals, collect server logs separately
for queue wait, VAD time, ASR decode time, speaker diarization time, GPU memory,
and GPU utilization.
## Report Template
When publishing a realtime WebSocket benchmark or issue report, include:
| Category | What to record |
|----------|----------------|
| Data | Audio duration, sample rate, language/domain, silence ratio or speaking pattern, and whether the same file was looped |
| Load | `--clients`, `--loops`, `--chunk-ms`, paced or `--no-pace`, and total benchmark wall time |
| Service | `serve_realtime_ws.py` command, `--partial-window-sec`, `--decode-interval`, `--enable-spk`, language, and hotwords |
| Hardware | GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions, CPU model, and available RAM |
| Software | `funasr`, PyTorch, torchaudio, vLLM, Python, OS, and container image if any |
| Output | Summary line, JSONL artifact, server logs, and any failed client IDs |
Do not reuse an offline `RTFx` number as a concurrency claim. For realtime
service sizing, benchmark with the actual traffic shape, especially sentence
length, pause distribution, simultaneous speakers, and whether speaker
diarization is enabled.
docs/benchmark/rtf_reproducibility.md
+107
View File
@@ -0,0 +1,107 @@
# Benchmark RTF and Reproducibility Notes
Use this page when comparing FunASR with Whisper, a cloud ASR provider, a Rust
runtime, or another self-hosted engine. Speed numbers are only useful when the
timing scope, data, model, runtime, and hardware are reported together.
## RTF and RTFx
FunASR benchmark tables usually report throughput as `RTFx`, or "times
realtime":
```text
RTF = processing_time_seconds / input_audio_seconds
RTFx = input_audio_seconds / processing_time_seconds
= 1 / RTF
```
For example, an `RTFx` value of `340` means 340 seconds of input audio are
processed in about 1 second, under that benchmark's data, runtime, batching, and
hardware setup. On the public vLLM table, the 184-file set has 11,541 seconds of
audio, so `340x` corresponds to roughly 34 seconds of measured processing time
for the whole set if the same scope is used:
```text
11541 / 340 = 33.94 seconds
```
Do not compare an offline batch `RTFx` result with streaming first-token latency
or end-to-end product latency. They measure different things.
For realtime WebSocket service sizing, use the
[Realtime WebSocket Benchmark](./realtime_ws_benchmark.md) instead.
## Current Public vLLM Benchmark Scope
The vLLM guide currently reports the following public scope for the Fun-ASR-Nano
and GLM-ASR-Nano table:
| Field | Value |
|-------|-------|
| Audio set | 184 long-form files |
| Total audio duration | 11,541 seconds, about 192 minutes |
| Models | Fun-ASR-Nano and GLM-ASR-Nano |
| Reported metric | CER and `RTFx` throughput |
| Fun-ASR-Nano vLLM batch result | `RTFx 340`, CER `8.20%` |
| Fun-ASR-Nano PyTorch baseline | `RTFx 21`, CER `8.06%` |
| Fun-ASR-Nano offline service without speaker diarization | `RTFx 102`, CER `8.14%` |
| Fun-ASR-Nano offline service with speaker diarization | `RTFx 46`, CER `8.19%` |
The table describes offline throughput on the stated long-form set. It should
not be read as a guarantee for every GPU, batch shape, language mix, streaming
chunk size, or service deployment.
The main website benchmark page is a separate public table for the broader ASR
comparison. It reports 184 long-form Chinese audio files, 11,539 seconds total,
and an NVIDIA H100 80GB HBM3 GPU. Keep the two tables separate when citing
numbers: the website table documents the general ASR benchmark, while the vLLM
guide table documents the Fun-ASR-Nano / GLM-ASR-Nano vLLM throughput rows.
## Required Fields for Reproducible Benchmark Claims
When publishing a FunASR benchmark, include these fields with the number:
| Category | What to record |
|----------|----------------|
| Data | File count, total audio duration, language/domain, sample rate, mono/stereo handling, and whether test files are public |
| Model | Model ID, checkpoint source, model revision or commit, language setting, hotwords, and text normalization |
| Runtime | Python SDK, ONNX, C++, vLLM, llama.cpp/GGUF, API server, or another path |
| Hardware | CPU model and thread count, GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions |
| Software | `funasr`, PyTorch, torchaudio, vLLM, ONNX Runtime, CUDA, Python, and operating system versions |
| Pipeline | VAD, punctuation, speaker diarization, ITN, timestamps, and post-processing on/off |
| Batching | Batch size, `batch_size_s`, concurrent requests, tensor parallel size, chunk size, VAD segment policy |
| Timing scope | Whether timing includes model download, cold start, warmup, file I/O, audio decoding/resampling, VAD, post-processing, and result serialization |
| Quality | CER/WER method, reference normalization, ignored tokens, and failed-file handling |
For official README or website numbers, include the fields above or link to a
report that includes them.
## Suggested Timing Protocol
1. Put all input audio in a manifest or directory and compute total duration
before running ASR.
2. Warm the model once if the published number is intended to represent steady
state throughput. If you include cold start, say so explicitly.
3. Time exactly one scope: model-only, pipeline-only, or end-to-end service.
4. Run the same scope at least three times and report median plus min/max.
5. Keep transcript output, failed-file list, and timing JSON/CSV with the run.
For migration or product evaluation, start from
[`examples/migration/benchmark_funasr.py`](../../examples/migration/benchmark_funasr.py).
It writes per-file timing and a Markdown summary for your own audio set. The
same reporting fields above also apply when you use vLLM, ONNX, C++, GGUF, or a
custom runtime instead of the migration example.
## Comparing with a Rust or Other Custom Runtime
For a fair engine-to-engine comparison:
- use the same audio files and total duration;
- resample and downmix with the same policy;
- keep VAD, punctuation, speaker diarization, and timestamps either all on or
all off;
- compare both speed and quality, because a faster decode path can change CER;
- report `RTFx` and raw processing seconds, not only a relative speedup.
If you can share your result publicly, open a Migration Benchmark Report issue
with the fields above. That makes the comparison useful to other users and gives
maintainers enough context to reproduce or improve the path.
docs/blog_whisper_vs_funasr_zh.md
+110
View File
@@ -0,0 +1,110 @@
# FunASR 实测：比 Whisper 快 30 倍的开源语音识别，还自带说话人分离
> 本文适合正在使用 OpenAI Whisper 做语音转写、但遇到速度慢或中文效果差问题的开发者。
## 背景
OpenAI Whisper 是目前最流行的开源 ASR 模型，但在生产环境中经常遇到几个痛点：
1. **速度慢** — Whisper-large-v3 在 A100 上只有 13x 实时，处理 1 小时音频需要 ~5 分钟
2. **中文方言差** — 对粤语、四川话、上海话等方言识别率很低
3. **缺少说话人分离** — 需要额外集成 pyannote，增加复杂度
4. **没有流式能力** — 无法做实时字幕
这些正是 [FunASR](https://github.com/modelscope/FunASR) 解决的问题。
## 性能对比
测试条件：184 个中文会议录音，总计 192 分钟，NVIDIA A100。
| 模型 | GPU 速度 | CPU 速度 | 说话人 | 流式 |
|------|---------|---------|--------|------|
| **FunASR SenseVoice-Small** | **170x** 实时 | **17x** 实时 | ✅ | ❌ |
| **FunASR Fun-ASR-Nano (vLLM)** | **340x** 实时 | — | ✅ | ✅ |
| Whisper-large-v3-turbo | 46x 实时 | ❌ | ❌ | ❌ |
| Whisper-large-v3 | 13x 实时 | ❌ | ❌ | ❌ |
**关键发现：FunASR 在 CPU 上比 Whisper 在 GPU 上还快。**
## 最大区别：一站式 vs 拼装
用 Whisper 搭建一个完整的转写系统，你需要：
```
Whisper (ASR) + pyannote (说话人) + silero-vad (VAD) +
deepmultilingualpunctuation (标点) + 自己写逻辑拼接
```
用 FunASR，**一次调用搞定**：
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
)
result = model.generate(input="meeting.wav")
# 输出：带说话人、时间戳、标点的结构化结果
```
## OpenAI 兼容 API：零改动迁移
如果你已经在用 OpenAI 的 Whisper API，迁移到 FunASR 不需要改一行客户端代码：
```bash
# 一行启动本地 ASR 服务
pip install torch torchaudio
pip install funasr vllm fastapi uvicorn python-multipart
funasr-server --device cuda
```
```python
# 客户端代码完全不变
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
result = client.audio.transcriptions.create(
model="fun-asr-nano",
file=open("meeting.wav", "rb")
)
```
## 中文方言支持
这是 Whisper 生态完全没有的：
- **7 大方言**：吴语、粤语、闽语、客家话、赣语、湘语、晋语
- **26 种地域口音**：河南、四川、广东、湖北、云南等
- **歌词识别**：音乐背景下的人声转写
## 什么时候选 FunASR？
| 场景 | 推荐 |
|------|------|
| 中文会议转写 | FunASR（方言+说话人） |
| 实时字幕/直播 | FunASR（流式 WebSocket） |
| 批量文件处理 | FunASR（vLLM 340x 加速） |
| 私有化部署 | FunASR（MIT 开源，无费用） |
| 纯英文、少量使用 | Whisper 也可以 |
## 开始使用
```bash
pip install torch torchaudio
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(model="iic/SenseVoiceSmall", device="cuda")
result = model.generate(input="audio.wav")
print(result[0]["text"])
```
更多信息：
- GitHub：https://github.com/modelscope/FunASR（16K+ stars）
- 官网：https://www.funasr.com
- 完整评测：https://modelscope.github.io/FunASR/benchmark.html
- 从 Whisper 迁移指南：https://github.com/modelscope/FunASR/blob/main/docs/migration_from_whisper.md
docs/cli.md
+117
View File
@@ -0,0 +1,117 @@
# Command-Line Interface
FunASR provides an agent-friendly CLI for speech recognition from the terminal. Designed for AI agents (Claude Code, Codex, Cursor), shell scripts, and automation pipelines.
## Installation
```bash
pip install funasr
```
## Basic Usage
```bash
# Transcribe audio (simplest)
funasr audio.wav
# Specify model
funasr audio.wav --model paraformer
# JSON output (structured, parseable)
funasr audio.wav --output-format json
# SRT subtitles
funasr audio.wav --output-format srt --output-dir ./subs
```
## Options
| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--model` | `-m` | sensevoice | Model: sensevoice, paraformer, paraformer-en, fun-asr-nano |
| `--hub` | `-H` | ms | Model hub: ms (ModelScope) or hf (Hugging Face) |
| `--language` | `-l` | auto | Language: zh, en, ja, ko, yue, auto |
| `--device` | | auto | Device: cuda:0, cpu |
| `--output-format` | `-f` | text | Output: text, json, srt, tsv |
| `--output-dir` | `-o` | stdout | Write output files to directory |
| `--timestamps` | | off | Include word-level timestamps |
| `--spk` | | off | Enable speaker diarization |
| `--hotwords` | | none | Comma-separated hotwords |
| `--verbose` | `-v` | off | Show loading/timing info on stderr |
## Output Formats
### text (default)
Plain transcription text, one result per file. Best for piping:
```bash
funasr audio.wav | wc -w
```
### json
Structured output for programmatic use:
```json
{
"text": "欢迎大家来体验达摩院推出的语音识别模型",
"segments": [
{"start": 0, "end": 5540, "text": "欢迎大家来体验达摩院推出的语音识别模型"}
],
"file": "audio.wav",
"model": "sensevoice",
"language": "auto",
"duration_s": 0.29
}
```
### srt
SubRip subtitle format:
```
1
00:00:00,000 --> 00:00:05,540
欢迎大家来体验达摩院推出的语音识别模型
```
### tsv
Tab-separated values (start/end in seconds):
```
start end text
0.000 5.540 欢迎大家来体验达摩院推出的语音识别模型
```
## Advanced Examples
```bash
# Speaker diarization + JSON
funasr meeting.wav --spk --timestamps -f json
# Batch transcribe all WAV files
funasr *.wav --output-format srt --output-dir ./output
# Chinese with hotwords
funasr audio.wav --model paraformer --language zh --hotwords "FunASR,达摩院"
# Pipe to jq for processing
funasr audio.wav -f json | jq '.text'
# Load models from Hugging Face instead of ModelScope
funasr audio.wav --hub hf --model fun-asr-nano
# Use with AI agents
result=$(funasr audio.wav -f json)
echo "$result" | jq -r '.text'
```
## Models
| Model | Languages | Speed | Best for |
|-------|-----------|-------|----------|
| sensevoice | 50+ | ~70ms/10s | General use, multilingual |
| paraformer | zh + mixed | ~60ms/10s | Chinese production (with punctuation) |
| paraformer-en | en | ~60ms/10s | English |
| fun-asr-nano | 31 | varies | Encoder+LLM, complex audio |
## Legacy CLI
The original Hydra-based CLI is available as `funasr-hydra`:
```bash
funasr-hydra ++model=paraformer-zh ++input=audio.wav
```
docs/community_growth_20k.md
+373
View File
@@ -0,0 +1,373 @@
# FunASR Ecosystem Growth Plan: +20,000 GitHub Stars
Baseline on 2026-06-27: the FunASR ecosystem repositories had 31,224 combined GitHub stars. The next target is earning 20,000 additional stars across the ecosystem by 2026-09-30, reaching 51,224+ combined stars through better onboarding, stronger proof, and repeatable launches.
This plan focuses on useful adoption work rather than vanity marketing: if more users can install, evaluate, deploy, and share FunASR successfully, stars follow naturally.
## Repository scope
| Repository | Role in the ecosystem | Growth surface |
|---|---|---|
| `modelscope/FunASR` | Core toolkit, Python package, runtime services, deployment docs | First-run success, production deployment, issue/PR operations |
| `FunAudioLLM/Fun-ASR` | Fun-ASR-Nano model family and LLM-ASR identity | Model cards, Transformers integration, benchmarks, vLLM/GGUF stories |
| `FunAudioLLM/SenseVoice` | CPU-friendly multilingual ASR with emotion and audio events | Quick demos, edge use cases, app integrations |
| `modelscope/FunClip` | Video clipping and transcription workflow | Creator workflows, local GUI/API recipes, showcase stories |
## North-star metrics
- GitHub stars: +20,000 across the four-repository ecosystem by 2026-09-30
- Monthly PyPI downloads: sustained growth after each release
- README quick-start success: first transcription in under 5 minutes
- Deployment success: API server, WebSocket, Docker, and vLLM examples verified on fresh machines
- Community throughput: faster issue triage, more external PRs, more user showcases
## Audience segments
| Audience | What they need | Repository surface that should serve them |
|---|---|---|
| ASR developers | Fast local transcript, model choice, Python API | README quick start, tutorial, model zoo |
| Production teams | Private deployment, streaming, API compatibility, Docker | runtime docs, OpenAI API example, vLLM guide |
| Agent builders | Speech input to Claude/Cursor/LangChain/Dify/AutoGen | MCP server and OpenAI API examples |
| Researchers | Benchmarks, reproducibility, citations, model details | benchmark report, docs, model cards |
| Chinese users | Chinese docs, ModelScope path, deployment in China | README_zh, docs/zh, ModelScope links |
| International users | English docs, Hugging Face path, comparison with Whisper | README, HF models, benchmark story |
## Current campaign snapshot
As of 2026-07-08 07:32 UTC, the ecosystem has 35,109 combined GitHub stars, or 3,885 additional stars since the 31,224 baseline. The remaining gap to the +20,000 target is 16,115 stars by 2026-09-30, which requires roughly 192 stars/day across the remaining 84 days.
| Repository | Stars | Forks | Open issues | Open PRs | Last push |
|---|---:|---:|---:|---:|---|
| `modelscope/FunASR` | 19,041 | 1,912 | 2 | 0 | 2026-07-08 |
| `FunAudioLLM/Fun-ASR` | 1,363 | 134 | 0 | 0 | 2026-07-07 |
| `FunAudioLLM/SenseVoice` | 8,807 | 789 | 0 | 0 | 2026-07-07 |
| `modelscope/FunClip` | 5,898 | 709 | 0 | 0 | 2026-07-07 |
Keep this snapshot fresh during weekly planning. The ecosystem mode also reports the remaining gap, days left to 2026-09-30, and the required daily average:
```bash
python scripts/collect_growth_metrics.py --ecosystem
python scripts/collect_growth_metrics.py --ecosystem --format json
```
## Traffic funnel snapshot
The highest owned conversion surfaces are the repository overviews and Chinese READMEs. Use these surfaces for model-selection clarity, deployment proof, and official ecosystem routing before adding new long-form pages.
14-day GitHub traffic as of 2026-07-07 18:46 UTC:
| Repository | Views | Unique visitors | Clones | Unique cloners | Highest-traffic owned paths |
|---|---:|---:|---:|---:|---|
| `modelscope/FunASR` | 40,588 | 9,650 | 12,635 | 2,144 | overview: 7,243 uniques; `README_zh.md`: 2,521; releases: 469 |
| `FunAudioLLM/Fun-ASR` | 6,173 | 1,926 | 1,023 | 817 | overview: 1,584 uniques; `README_zh.md`: 581; `docs/vllm_guide.md`: 148 |
| `FunAudioLLM/SenseVoice` | 8,917 | 3,477 | 1,028 | 620 | overview: 2,802 uniques; `README_zh.md`: 923; releases: 163 |
| `modelscope/FunClip` | 2,764 | 1,366 | 622 | 455 | overview: 1,187 uniques; `README_zh.md`: 327; UI screenshots: 203 combined |
Top referrer pattern:
- Search and GitHub dominate all four repos, so SEO snippets, GitHub topics, and README first-screen clarity matter more than low-frequency social posts.
- `funasr.com`, `modelscope.github.io`, Hugging Face, ChatGPT, and `ai-bot.cn` are meaningful secondary referrers; keep model cards and homepage links synchronized with README claims.
- FunClip has unusually visible image traffic, so screenshot freshness and UI clarity can influence creator conversion.
## Workstream 1: Convert first-time visitors
- Keep the README first screen focused on speed, supported tasks, one-call usage, and links to docs.
- Keep the benchmark table close to the quick start, with exact benchmark scope and full report link.
- Maintain a short model-selection table: SenseVoice, Paraformer, Fun-ASR-Nano, Qwen3-ASR, GLM-ASR, streaming, punctuation, VAD, speaker, emotion.
- Add visible links to deployment, agent integration, contribution, and discussions.
- Avoid burying install and first inference behind long research history.
## Workstream 2: Make deployment shareable
- Verify `funasr-server --device cuda` on a clean GPU machine and publish the exact command/output.
- Add curl examples for `/v1/audio/transcriptions` and WebSocket streaming.
- Keep Docker image tags explicit and document CPU/GPU differences.
- Add a small deployment decision table: Python API vs OpenAI API vs WebSocket vs Docker vs vLLM vs Triton.
- Turn common deployment failures into FAQ entries.
## Workstream 3: Launch content that earns stars
Use one release note or article per clear story:
- "FunASR is 170x realtime and CPU viable for long audio."
- "Self-hosted OpenAI-compatible speech transcription with one command."
- "Streaming ASR with VAD, punctuation, and speaker diarization."
- "MCP speech input for Claude/Cursor and agent workflows."
- "vLLM acceleration for Fun-ASR-Nano."
- "Chinese dialect, accent, meeting, and industrial ASR examples."
For each launch, prepare:
- GitHub release notes with GIF/screenshot or terminal output
- README update and docs link
- Hugging Face and ModelScope model-card update
- Homepage update on `modelscope.github.io/FunASR`
- Short posts for developer communities
- A pinned GitHub discussion for feedback
## Workstream 4: Community operations
- Use issue templates to collect reproducible environment, audio, and deployment details.
- Triage issues weekly with labels: bug, question, documentation, deployment, enhancement, good first issue.
- Convert repeated support answers into docs within 48 hours.
- Keep 5-10 `good first issue` tasks ready for contributors.
- Thank external contributors in release notes.
Daily maintainer loop:
- Check open issues and PRs in `modelscope/FunASR`, `FunAudioLLM/Fun-ASR`, `FunAudioLLM/SenseVoice`, and `modelscope/FunClip`.
- Prioritize blockers that stop a first transcription, an API server launch, a WebSocket deployment, or a model download.
- Keep external ecosystem PRs unblocked, especially integrations in high-visibility AI, agent, workflow, ASR, and video repositories.
- Turn repeated answers into docs or examples before closing the loop.
- Keep actionable issues labelled as `good first issue`, `help wanted`, or `ready for PR` so contributors can help without guessing.
Use the issue snapshot before each maintainer pass so waiting-on-reporter, waiting-on-contributor, and maintainer-owned items stay separate:
```bash
python scripts/collect_growth_metrics.py --issues
python scripts/collect_growth_metrics.py --issues --format json
```
## Workstream 5: External proof
- Publish reproducible benchmark scripts and raw configuration for the 184-file benchmark.
- Encourage users to share production stories, dashboards, public demos, or benchmark notes through the showcase issue template.
- Add third-party integrations and community projects to README when they are maintained and runnable.
- Keep citations and paper links visible for researchers.
- Track and unblock integrations in Hugging Face Transformers, inference runtimes, agent frameworks, workflow tools, video tools, and speech-recognition libraries.
## External integration tracker
High-visibility external integrations can create more qualified traffic than a one-off announcement because they put FunASR in the workflow users already trust. Track them as an operations queue, not as a comment-ping list.
Refresh this table before weekly planning and after any reviewer or CI change:
```bash
python scripts/collect_growth_metrics.py --integrations
python scripts/collect_growth_metrics.py --integrations --format json
```
Set `GITHUB_TOKEN` when running this from CI or a shared network so GitHub's public API rate limit does not hide the real PR state.
### External adoption issue queue
High-star feature requests and roadmap issues are earlier in the funnel than PRs: they shape whether upstream projects expose a clean local STT contract before any code branch exists. Track them separately from merge-ready PRs and comment only when a maintainer asks for implementation detail, test scope, or provider behavior.
| Adoption issue | Growth reason | Current maintainer action |
|---|---|---|
| `openclaw/openclaw#35835` audio file support in the read tool | Extends a high-visibility agent read path from images to audio, with a natural fallback slot for private OpenAI-compatible STT when the active model cannot consume native audio | Keep native audio attachments separate from transcription fallback: audio-capable models should receive original media, while text-only models can use an explicit `POST /v1/audio/transcriptions` adapter returning transcript plus metadata. FunASR/SenseVoice are useful self-hosted regression backends for that fallback contract. |
| `openclaw/openclaw#40078` silent preflight transcription in mention-gated channels | Turns voice-note history capture into a high-visibility STT workflow where private transcription can enrich context without triggering assistant replies | Track the proposed `preflightTranscribe` gate: server-side STT should run only after channel/attachment checks, write transcript provenance into history, keep response generation mention-gated, and use an OpenAI-compatible transcription backend so hosted and local FunASR/SenseVoice services share the same contract. |
| `openclaw/openclaw#45508` self-hosted STT/TTS provider support in WebChat | Routes OpenClaw WebChat users toward private OpenAI-compatible STT providers instead of browser-only speech APIs | A batch push-to-talk path has been scoped: browser records a blob, the Gateway calls the configured `tools.media.audio` / OpenAI-compatible transcription backend, and streaming can remain a follow-up. Watch for product/API decisions before proposing code. |
| `openclaw/openclaw#46661` custom ASR server configuration | Connects a 382k-star OpenClaw native voice-input request to self-hosted/private ASR, giving FunASR, SenseVoice, and Qwen3-ASR a clean OpenAI-compatible STT provider path | LauraGPT posted the provider-contract note at https://github.com/openclaw/openclaw/issues/46661#issuecomment-4909572545. Track maintainer decisions on platform default behavior, explicit credential isolation for custom endpoints, multipart `{base_url}/audio/transcriptions`, minimum `{ "text": "..." }` responses, optional segment/timestamp metadata, timeout messaging, and fallback tests before proposing code. |
| `openclaw/openclaw#87140` pluggable macOS Push-to-Talk STT backend | Opens a native desktop path for FunASR/SenseVoice as local/private ASR behind OpenClaw's existing PTT UX | Keep Apple Speech as the default and watch for a batch-only provider boundary with explicit capability flags (`batch`, `streaming`, `languages`, `local_only`, `requires_network`). |
| `openclaw/openclaw#98027` editable Control UI dictation | Turns a 382k-star agent UI's ordinary prompt composer into a local/private STT adoption path where FunASR or SenseVoice can sit behind the existing Gateway/media-audio provider boundary | PR is open with `check-docs` failure and missing real microphone -> Gateway -> editable draft proof, so do not call it merge-ready. LauraGPT posted the downstream ASR contract checklist at https://github.com/openclaw/openclaw/pull/98027#issuecomment-4911740942; track whether proof covers local/private transcription provider dispatch, no auto-send, redacted errors, and temp-audio cleanup on success/failure. |
| `openclaw/openclaw#102028` QQBot speech transcription timeout | Hardens a very high-visibility voice-message transcription path so stalled hosted or self-hosted STT providers do not block inbound QQBot attachments indefinitely | PR is open and behind main while checks continue; LauraGPT posted a non-blocking self-hosted STT contract note at https://github.com/openclaw/openclaw/pull/102028#issuecomment-4911671891. Track whether the timeout remains configurable through the public provider/audio model path and whether timeout errors stay distinguishable from auth/model/route errors for slow FunASR/SenseVoice endpoints. |
| `NousResearch/hermes-agent#56989` local STT docs and hardening | Helps self-hosted messaging gateways document fully local voice-note transcription for private deployments | Track whether docs define a backend-neutral `local_command` and local HTTP contract; FunASR/SenseVoice should fit as wrappers or OpenAI-compatible `/v1/audio/transcriptions` services. |
| `NousResearch/hermes-agent#16185` Telegram voice-message STT provider precedence | Turns Telegram voice notes in a high-visibility messaging agent into a direct local/private STT adoption path, where FunASR/SenseVoice can fit through either `local_command` wrappers or OpenAI-compatible transcription servers | Fresh user report says removing the OpenAI STT block made local STT work, suggesting provider precedence/config resolution rather than model failure. LauraGPT posted the backend-neutral provider-boundary checklist at https://github.com/NousResearch/hermes-agent/issues/16185#issuecomment-4911698678; track for a Telegram adapter or gateway fix that logs the selected provider and prevents silent OpenAI fallback. |
| `NousResearch/hermes-agent#60624` Windows ffmpeg discovery for Discord voice | Removes a Windows setup blocker in a high-visibility Discord voice transcription/TTS path before local/private STT providers such as FunASR/SenseVoice can work reliably | PR `NousResearch/hermes-agent#60627` adds a shared ffmpeg resolver for `FFMPEG_PATH`, PATH, and winget installs. LauraGPT linked the PR and validation back to the user report on 2026-07-08 at `https://github.com/NousResearch/hermes-agent/issues/60624#issuecomment-4911355349`; now monitor checks and maintainer feedback. |
| `anomalyco/opencode#33300` desktop voice input | Puts local speech input in a high-visibility coding-agent desktop app where privacy-sensitive users may avoid browser dictation | Watch for an app-side transcription provider abstraction that inserts transcripts into the prompt draft and keeps provider credentials out of the renderer. |
| `ggml-org/llama.cpp#24375` ASR model routing for WebUI mic input | Lets Qwen3-ASR / Fun-ASR-Nano act as the dedicated transcription model for a separate text-only chat model | Track the proposed `/v1/models` input-modality discovery and explicit transcription-model selector; avoid duplicate comments while a contributor is working on the UI behavior. |
| `Tencent/ncnn#6790` Qwen3-ASR ncnn port and multi-platform deployment | Opens a 23k-star mobile/edge inference path for Qwen3-ASR-style local transcription, where successful conversion can feed Fun-ASR-Nano/Qwen3-ASR users looking for lightweight on-device ASR | This is a 2026 Rhino-Bird activity issue, so LauraGPT posted a non-claim technical validation note rather than trying to own the task: https://github.com/Tencent/ncnn/issues/6790#issuecomment-4912104458. Watch for a public `Qwen3-ASR-ncnn` repo or PR, then verify PyTorch parity on fixed preprocessing, normalized CJK text, module-level encoder/projector/decoder drift, and Linux/Windows smoke commands. |
| `Wei-Shaw/sub2api#3754` OpenAI audio endpoint passthrough | Adds a Chinese gateway path for OpenAI-compatible STT/TTS clients and self-hosted FunASR/SenseVoice endpoints | Watch for standard multipart `POST /v1/audio/transcriptions` passthrough, language/model forwarding, and a CJK regression that preserves `language=zh`. |
| `elizaOS/eliza#14807` audio PII redaction pipeline | Makes timed ASR spans part of an agent safety workflow where local CJK/private transcription can be useful | Keep FunASR/SenseVoice positioned as optional verifier backends; avoid coupling the primary redaction path to one ASR engine. |
| `modelcontextprotocol/servers#4299` FunASR speech-to-text MCP server | Keeps FunASR visible in the canonical MCP server discussion for local speech tools used by Claude/Cursor-style clients | The compact upstream server now exists in `examples/mcp_server` with `transcribe_audio`, Dockerfile, `glama.json`, and stdio smoke evidence. LauraGPT posted the upstream-status update at https://github.com/modelcontextprotocol/servers/issues/4299#issuecomment-4901476584; next action is to track whether MCP maintainers prefer an external community-server reference instead of vendoring. |
| `crewAIInc/crewAI#5983` FunASR for voice-enabled agents | Routes a 55k-star multi-agent framework toward a provider-neutral voice command path where FunASR/SenseVoice can be a local OpenAI-compatible transcription backend | LauraGPT revived the stale broad request with a concrete `voice.stt.*` config, multipart `/v1/audio/transcriptions` contract, mock endpoint test shape, and agent-command handoff at https://github.com/crewAIInc/crewAI/issues/5983#issuecomment-4901293351. Watch for maintainer direction before opening a code PR. |
| `Significant-Gravitas/AutoGPT#13347` FunASR as an open-source STT backend | Keeps the 185k-star AutoGPT voice-input discussion anchored on a generic OpenAI-compatible transcription contract rather than a heavyweight FunASR-only dependency | LauraGPT first mapped the existing hard-coded Copilot Whisper route, then opened draft PR `Significant-Gravitas/AutoGPT#13500` and linked it back at https://github.com/Significant-Gravitas/AutoGPT/issues/13347#issuecomment-4908286807. Track whether maintainers prefer env-based `/v1/audio/transcriptions` configurability, a workflow block, or a separate self-hosted transcript pipeline. |
| `chatchat-space/Langchain-Chatchat#5479` FunASR/SenseVoice voice input | Puts FunASR in a 38k-star Chinese RAG/Agent app where local audio upload can feed existing Chat and knowledge-base flows | LauraGPT proposed an OpenAI-compatible ASR endpoint slice, including `asr.base_url`, multipart request shape, minimal `{ "text": "..." }` response, and mocked CI endpoint at https://github.com/chatchat-space/Langchain-Chatchat/issues/5479#issuecomment-4901293565. Monitor stale handling and maintainer appetite for a first audio-upload transcription path. |
| `royshil/obs-localvocal#314` SenseVoice/Paraformer engine option | Opens an OBS live-captioning path where SenseVoice/Paraformer can be evaluated through a cleaner ASR-engine boundary instead of a Whisper-only runtime | Maintainer is interested but capacity-limited until mid/late August. LauraGPT mapped current Whisper coupling points and suggested a facade-only first refactor at https://github.com/royshil/obs-localvocal/issues/314#issuecomment-4909662013; wait for a refactor branch or review request before proposing Sherpa-ONNX/FunASR runtime code. |
| `SevaSk/ecoute#203` FunASR alternative ASR backend | Places FunASR/SenseVoice in a local realtime transcription app where users already compare Whisper-compatible and local ASR backends | LauraGPT recommended a two-step path at https://github.com/SevaSk/ecoute/issues/203#issuecomment-4906862462: first expose an OpenAI-compatible local STT endpoint with `stt_base_url`, optional key, model, and language; later add a native backend only with explicit streaming, language, timestamp, and batch-window capabilities. Monitor for maintainer appetite or an implementation PR before posting again. |
| `521xueweihan/HelloGitHub#3296` FunASR Chinese open-source recommendation | Gives FunASR a high-visibility Chinese developer discovery path in a long-running open-source recommendation channel | LauraGPT posted a 2026-07-08 editor-ready refresh at `https://github.com/521xueweihan/HelloGitHub/issues/3296#issuecomment-4910800381` with current ecosystem star counts, category fit, quick-start shape, OpenAI-compatible endpoint value, and the FunClip related-project angle; wait for editor triage without duplicate bumps. |
| `duixcom/Duix-Avatar#605` FunASR/SenseVoice ASR backend option | Puts FunASR/SenseVoice into a 13k-star digital-human project where low-latency ASR plus emotion/audio-event tags can drive avatar interaction | Maintainers acknowledged the current-contract suggestion and said they noted it for subsequent version iterations. Track for a Duix-side version/update invite, then verify the existing `duix-avatar-asr` `/v1/preprocess_and_tran` contract before pushing any parallel OpenAI-compatible endpoint path. |
| `sgl-project/sglang-omni#924` ASR support roadmap | Coordinates Fun-ASR-Nano serving tasks across cache behavior, batching, evaluation, and benchmark parity in a serving runtime used by ASR/GPU deployment contributors | LauraGPT posted FunASR-side guidance on full-context encoder cache scope, per-sample speech embedding lengths, batching checks, and CJK evaluation coverage; monitor F-PR follow-ups and avoid duplicate roadmap comments. |
| `pollinations/pollinations#5321` next-model voting for STT | Places FunASR/SenseVoice/Fun-ASR-Nano in a high-traffic open inference model request funnel used by app builders who want hosted or OpenAI-compatible transcription models | LauraGPT posted one exact model request on 2026-07-07 under the TTS & STT voting thread; avoid duplicate comments and monitor for maintainer model-selection signals or requests for endpoint/spec details. |
| `cjpais/Handy#1626` Qwen3-ASR Intel Mac AMD Metal fallback | Reduces a visible local-dictation failure mode in a 25k-star desktop app, keeping Qwen3-ASR / SenseVoice users from abandoning the model after a backend-specific crash | Maintainer confirmed `transcribe.cpp` will avoid Metal by default on non-Apple-Silicon Macs in the next release. Watch the release, then verify Intel macOS + AMD Metal + Qwen3-ASR routes to CPU fallback while SenseVoice keeps its working path. |
| `Zackriya-Solutions/meetily#567` Apple Voice Memos drag-to-import | Routes a 20k-star privacy-first meeting assistant toward imported-audio transcription workflows where local STT providers can be evaluated | Treat this as macOS import normalization, not an ASR swap: inspect Voice Memos drag payloads, normalize them to a local `.m4a` path plus title/start-time/duration metadata, reuse the existing import/transcription provider pipeline, and keep FunASR/SenseVoice as later optional providers behind that boundary. GitHub returned `403 Blocked` on the first public comment attempt, so monitor without retrying until there is new maintainer activity or account policy changes. |
| `altic-dev/FluidVoice#547` remote transcription backend | Gives lightweight or Intel Macs a path to consume a LAN/GPU STT server instead of running every model locally | Relevant to FunASR/SenseVoice remote endpoints, but GitHub returned `403 Blocked` on both the first comment attempt and a 2026-07-08 retry after new maintainer/user activity. Maintainers currently framed near-term work as easier local server startup plus future Windows/Linux GPU support, not a remote consumer endpoint; do not retry again unless account policy or repository write policy changes. |
| `freestyle-voice/freestyle#415` custom OpenAI-compatible STT base URL | Routes a voice-dictation app's batch STT provider toward LAN/self-hosted FunASR/SenseVoice endpoints without forcing users through hosted OpenAI URLs | Author accepted LauraGPT's compatibility scope at https://github.com/freestyle-voice/freestyle/issues/415#issuecomment-4911863210: strip trailing `/v1` or slash before appending `/v1`, keep the first pass batch-only on the existing `openai` provider, reuse the current API key/model flow, and test both bare-host and `/v1` base URLs. Wait for the implementation PR before adding more comments. |
| Integration PR | Growth reason | Current maintainer action |
|---|---|---|
| `1c7/chinese-independent-developer#1065` FunClip listing link refresh | Keeps FunClip's existing entry in a 49k-star Chinese independent-developer directory pointed at the current official GitHub repository instead of the old organization homepage | PR updates the accepted FunClip listing's "更多介绍" link from `github.com/alibaba-damo-academy` to `github.com/modelscope/FunClip`; monitor for maintainer acceptance without extra pings. |
| `Kedreamix/Linly-Talker#151` FunASR link refresh | Keeps a 3.3k-star digital-avatar conversational system's ASR docs pointed at the current official FunASR repository, so users comparing Whisper and FunASR land on maintained setup docs | PR updates the root README and `ASR/README.md` references from `github.com/alibaba-damo-academy/FunASR` to `github.com/modelscope/FunASR`; monitor for maintainer acceptance without extra pings. |
| `huggingface/transformers#46180` Fun-ASR-Nano model support | Makes Fun-ASR-Nano usable through the default HF API surface and model docs | Current blocker is an unrelated LightOnOCR shared-cache failure in `tests_processors`; the account cannot rerun Hugging Face Actions jobs, so wait for a maintainer rerun and avoid duplicate comments unless new CI evidence appears. |
| `huggingface/transformers#47111` Qwen3-ASR hotword and language parsing fixes | Keeps the high-visibility Transformers Qwen3-ASR implementation aligned with upstream hotword/context behavior, language hints, and processor training paths used by downstream Fun-ASR-Nano/Qwen3-ASR adopters | The Qwen3-ASR slow job was explicitly rerun and reported no PR-specific failures on head `caf37300fc3d3333c3d7c2162a92a886e424eb1c`; the aggregate CI recap still has unrelated suite failures. Monitor for reviewer requests around hotword serialization, language parsing, and qwen3_asr processor tests before adding any FunASR-side comment. |
| `sgl-project/sglang-omni#898` Fun-ASR serving support | Exposes Fun-ASR-Nano through a high-visibility serving runtime for ASR benchmarks and GPU deployment | The contributor-owned branch is currently dirty; LauraGPT already posted a concrete ASR benchmark rename/docs conflict recipe, and the author replied that conflicts will be resolved during merging. Wait for contributor or maintainer conflict resolution before adding more comments. |
| `vllm-project/vllm#42478` Qwen3-ASR streaming postprocessing | Improves the upstream Qwen3-ASR streaming path used by OpenAI-compatible transcription clients, so downstream apps can consume clean SSE transcript deltas without model-specific `language ...<asr_text>` cleanup | LauraGPT posted downstream validation guidance on 2026-07-08. CI is currently blocked only by vLLM's pre-run gate: the PR needs a maintainer `ready`/`verified` label or an author history of 4+ merged PRs (current log found 2), so wait for maintainer review rather than adding code-change comments; still watch for a no-space CJK regression. |
| `vllm-project/vllm#47729` MOSS-Transcribe-Diarize support | Expands vLLM's OpenAI-compatible `/v1/audio/transcriptions` route for long-form ASR with timestamps and speaker labels, creating another high-visibility comparison point for Qwen3-ASR / Fun-ASR-Nano-style serving behavior | PR is open and `mergeable=true` with `mergeable_state=blocked`; LauraGPT posted a non-blocking downstream ASR contract note at https://github.com/vllm-project/vllm/pull/47729#issuecomment-4912065897. Watch that `response_format=json` keeps the plain `{ "text": "..." }` shape stable, any future `segments` remain additive, CJK long-form audio stays covered, and the new model registration does not alter existing Qwen3-ASR request/response behavior. |
| `tenstorrent/tt-metal#49104` Qwen3-ASR Blackhole/P150 bringup | Opens a hardware-accelerated Qwen3-ASR path with an OpenAI-compatible `/v1/audio/transcriptions` server for Tenstorrent users evaluating speech workloads | PR is open, approved, and process-blocked rather than waiting on FunASR feedback; LauraGPT posted a non-blocking API-contract note at https://github.com/tenstorrent/tt-metal/pull/49104#issuecomment-4911526665. Track merge/release docs so downstream FunASR users can compare TT, CPU, vLLM, and H100 timing scopes correctly. |
| `ray-project/ray#64053` Ray Serve FunASR ASR example | Puts FunASR in production serving docs for teams already using Ray | Monitor review, answer questions quickly, and keep the example command aligned with the current OpenAI-compatible API behavior. |
| `huggingface/optimum-intel#1801` OpenVINO support | Helps CPU and edge users evaluate Fun-ASR on Intel hardware | Current failed jobs are unrelated OpenVINO matrix failures in Pix2Struct, image-text quantization/export, and tiny-random T5; LauraGPT's direct mirror PR `#1856` was closed after maintainers said they will address the cleanup in a preliminary PR, so wait for that maintainer-side update before adding more comments. |
| `huggingface/speech-to-speech#319` SenseVoice STT handler | Adds SenseVoice/FunASR to local open-source voice-agent pipelines where low-latency STT is a core comparison point | Keep lint/import fixes on the PR head, explain optional `speech-to-speech[sensevoice]` install behavior, and answer handler-scope review quickly. |
| `OpenBMB/VoxCPM#349` Windows CUDA installer with SenseVoice fallback | Puts SenseVoice into a 32k-star multilingual TTS app's Windows install and first-run ASR path as the fallback when local Parakeet/CUDA is unavailable | Current head `9f34141cc81e04028c4e62ac652f2a66dd453dfa` is dirty only in `app.py`; LauraGPT posted the conflict recipe and light validation at https://github.com/OpenBMB/VoxCPM/pull/349#issuecomment-4905293176. Wait for author rebase or maintainer action without duplicate comments. |
| `livekit/agents#6176` FunASR/SenseVoice realtime STT plugin | Opens a path into LiveKit's realtime voice-agent ecosystem where local STT is evaluated alongside hosted providers | CI and CLA are green; avoid duplicate pings and monitor for maintainer review on plugin scope, package metadata, or optional dependency expectations. |
| `datajuicer/data-juicer#938` HumanVBench audio/video operators | Places FunASR/SenseVoice-style speech understanding into a 6k-star data processing toolkit used to evaluate human-centric video and multimodal datasets | Current PR is open and mergeable but process-blocked while unit-test jobs are still waiting; LauraGPT posted the FunASR/SenseVoice dependency and validation follow-up at https://github.com/datajuicer/data-juicer/pull/938#issuecomment-4905235851. Wait for CI or maintainer feedback before adding more comments. |
| `mahimairaja/voiceai#16` FunASR and SenseVoice STT resource listing | Places FunASR/SenseVoice in a curated voice-AI resource map for builders comparing STT providers and local speech stacks | Maintainer approved. A pre-existing Stars badge target caused `link-check` to fail; pushed `16c451d` to retarget the badge to the repo homepage, verified `lychee 0.23.0` with the CI args (`480 Total`, `0 Errors`), and posted the green-status note after GitHub reported combined status `success`. Wait for maintainer merge without further pings. |
| `punkpeye/awesome-mcp-servers#7153` FunASR MCP server listing | Exposes FunASR to a high-star MCP discovery list used by agent-tool builders looking for local speech tools | Glama manifest now points at the current `server.json` schema and declares the maintainer for verification; the MCP Dockerfile also carries the official Registry OCI ownership label. Next step is maintainer OAuth/Glama ingestion or a public OCI image plus matching MCP Registry `server.json`, then update the external PR only after a valid score badge or maintainer feedback. |
| `zts212653/clowder-ai#1083` Qwen3-ASR service unification | Puts Qwen3-ASR into a user-facing local STT service slot by making it a `whisper-stt` backend variant instead of a separate, easier-to-misconfigure service | Head `28067864e313cd03dd3d6f4ce0a72ec5b5026b47` reports green CI after fixes for Rosetta hardware detection, Qwen3-ASR install/server dispatch, async backend locking, temp WAV cleanup, and stale setup docs; author status update: https://github.com/zts212653/clowder-ai/pull/1083#issuecomment-4910797452. Wait for maintainer re-review without duplicate comments. |
| `run-llama/llama_index#21958` FunASR endpoint reader | Puts FunASR behind a LlamaIndex reader for OpenAI-compatible transcription endpoints used in RAG and agent pipelines | LauraGPT rechecked head `07a8599deaebe5e7a559d62174e7a872870c2f7e` at https://github.com/run-llama/llama_index/pull/21958#issuecomment-4905345545; author acknowledged the shared-reader pytest caveat at https://github.com/run-llama/llama_index/pull/21958#issuecomment-4905384536. Keep the endpoint contract clear and avoid forcing local `funasr` dependencies into the main package. |
| `run-llama/llama_index#21996` local FunASR reader | Gives LlamaIndex users a local SenseVoice/FunASR reader for private transcription workflows | Keep optional dependencies isolated, verify the reader does not affect default installs, and watch for maintainer guidance on package extras. |
| `mem0ai/mem0#5571` optional FunASR transcription helper | Adds local FunASR transcription to a high-star memory layer used by agent builders | Keep FunASR optional and make examples clear about local model downloads; the current failing Vercel preview is a Mem0 team authorization gate, not a FunASR code failure, so wait for maintainer-side preview access or review. |
| `Significant-Gravitas/AutoGPT#13500` configurable transcription endpoints | Opens a path from AutoGPT's high-visibility agent platform to local FunASR/SenseVoice gateways through the existing OpenAI-compatible transcription route | GitHub Actions are green after the typed `HeadersInit` fix; remaining blockers are the author-controlled CLA, AutoGPT-team Vercel preview authorization, and maintainer review, not FunASR-side code. |
| `harry0703/MoneyPrinterTurbo#1006` shareable video-generation presets | Puts subtitle, language, voice, and provider settings in a 96k-star short-video generator's reusable preset schema, creating the right safety boundary for later local/offline subtitle ASR or FunClip handoff workflows | LauraGPT posted a non-blocking preset-contract note at https://github.com/harry0703/MoneyPrinterTurbo/pull/1006#issuecomment-4912521164: export only stable non-secret choices such as subtitle provider, language, subtitle style, TTS/voice selection, and local/offline capability; keep API keys, bearer tokens, signed URLs, temporary media paths, cached model files, and unsanitized endpoint URLs out of shareable presets. Track review without asking this preset PR to add a FunASR/SenseVoice engine. |
| `harry0703/MoneyPrinterTurbo#911` OpenVINO local subtitle provider | Adds a hardware-accelerated local transcription provider to a 96k-star short-video generator's subtitle pipeline, creating the same `audio -> SRT/timestamped segments` seam that later FunASR/SenseVoice or FunClip workflows can reuse | LauraGPT posted a non-blocking provider-boundary note at https://github.com/harry0703/MoneyPrinterTurbo/pull/911#issuecomment-4912751845. Track whether the PR keeps `subtitle_provider` generic, handles missing OpenVINO packages/model directories before subtitle correction, preserves valid UTF-8 SRT and CJK punctuation, and avoids another settings migration for future `funasr` / `sensevoice` providers. |
| `maximhq/bifrost#5020` diarized OpenAI transcription compatibility | Keeps a high-visibility LLM gateway's OpenAI-compatible `/v1/audio/transcriptions` path safe for diarized STT responses, including downstream clients that compare hosted models with self-hosted FunASR/SenseVoice endpoints | PR is open with green checks and bot review confidence; LauraGPT posted a non-blocking downstream contract note at https://github.com/maximhq/bifrost/pull/5020#issuecomment-4911586742. The fix covers string diarized segment IDs, explicit empty `segments: []`, multipart transcription params, and speaker passthrough; watch merge/release feedback before adding more comments. |
| `getpaseo/paseo#313` FunASR streaming STT provider | Adds a standalone FunASR/SenseVoice server and streaming dictation provider to a 10k-star agent orchestration app | Current head `19d9270f1b55d5a0d59288ff2c4a4e707bd60c9a` is dirty against main; LauraGPT triage at https://github.com/getpaseo/paseo/pull/313#issuecomment-4905606566 documents the conflict set and a `needsFunASR` provider-wiring bug to fix during rebase. Wait for author rebase or maintainer guidance without duplicate pings. |
| `getpaseo/paseo#1634` SenseVoice local STT model support | Adds SenseVoice to a local voice/dictation app with a visible offline STT model catalog and speech CLI path | Current PR is mergeable at head `4cc65ac6fbc9735ac487648bbbd02ee487050f31`; posted focused local validation at https://github.com/getpaseo/paseo/pull/1634#issuecomment-4905550193 covering server deps build, format check, 3 speech Vitest files with 10 tests, and server build. Wait for maintainer review without duplicate pings. |
| `infiniflow/ragflow#16473` FunASR / SenseVoice STT provider | Adds FunASR to a high-star RAG workflow product where local STT can become a visible configuration choice | Track duplicate/conflict cleanup, keep provider naming consistent, and verify that skipped CI is expected rather than a hidden regression. |
| `mudler/LocalAI#10090` FunASR backend | Exposes FunASR through a high-star local AI engine used by self-hosters and edge deployments | Upstream DCO is green; the fork-side `tests` failure is a CodeQL/SARIF upload permission gate on a fork, not a FunASR backend regression. Local CPU smoke validation remains `backend/python/funasr/test.py` with four passing tests, so wait for maintainer-side review or rerun without duplicate pings. |
| `agno-agi/agno#8501` FunASR transcription tool | Places FunASR in a high-star agent platform as a local multilingual transcription tool | Keep issue linkage and formatting gates green, avoid heavy default dependencies, and respond to review scope requests quickly. |
| `GetStream/Vision-Agents#606` FunASR STT plugin | Adds SenseVoice/FunASR to multimodal voice and vision agent examples | Keep package include paths and tests aligned with upstream conventions, then watch review threads for optional dependency or documentation requests. |
| `TEN-framework/ten-framework#2191` FunASR ASR extension | Places SenseVoice/FunASR in a realtime multimodal agent framework with extension discovery | The failing `claude-review` action is a fork-permission gate, not a code failure; wait for maintainer-side review or rerun, and respond only to concrete code/doc feedback. |
| `activepieces/activepieces#13985` FunASR speech recognition piece | Gives no-code workflow users a direct FunASR speech recognition action | Replacement PR for conflicted `#13450` is open and mergeable at head `f9d22ee03c58199c7236e2f1008f083d6f80b4a2`; Greptile marks the additive community piece safe to merge and normal checks are green/skipped. Latest LauraGPT recheck: https://github.com/activepieces/activepieces/pull/13985#issuecomment-4904635178. Remaining blocker is the `license/cla` status, which is author/account-controlled and should not be signed by the operator. |
| `omnigent-ai/omnigent#2093` server-side streaming dictation | Opens a model-agnostic dictation backend in a multi-agent desktop/mobile harness where FunASR/SenseVoice can later fit behind the same local worker or OpenAI-compatible fallback contract | LauraGPT posted the non-blocking FunASR/SenseVoice backend-contract follow-up at https://github.com/omnigent-ai/omnigent/pull/2093#issuecomment-4907140875. Current PR is open and mergeable but blocked by `npm test`, pre-commit, and maintainer approval checks; wait for concrete review or CI logs before more comments. |
| `speaches-ai/speaches#658` FunASR transcription backend | Adds SenseVoice/Paraformer to an OpenAI-compatible local speech API used by self-hosted voice stacks | Current `unstable` state is a fork-workflow approval gate with no check-runs/logs on the head commit; keep the local validation note current and respond only if real CI logs or maintainer review appear. |
| `Uberi/speech_recognition#903` FunASR recognizer | Exposes FunASR through a widely known Python speech-recognition wrapper | Current `unstable` state is a fork-workflow approval gate with no check-runs/logs on the head commit; keep the recognizer optional and lightweight, and be ready with a minimal install/import smoke test if maintainers ask for scope reduction. |
| `fighting41love/funNLP#478` SenseVoice and FunClip speech-section listing | Adds the newer FunASR ecosystem projects to a high-star Chinese NLP resource list with strong domestic discovery value | PR is clean and mergeable; avoid duplicate pings unless maintainers ask for category, wording, or link changes. |
| `josephmisiti/awesome-machine-learning#1339` FunASR Python speech-recognition listing | Places FunASR in a broad high-star machine-learning discovery list where users compare Python ASR toolkits | PR is open but `mergeable_state=blocked`; recheck only for maintainer feedback or a new failed status, and keep the entry description aligned with the core toolkit rather than every model variant. |
| `crownpku/Awesome-Chinese-NLP#32` Speech Recognition and Audio section | Creates a Chinese-NLP discovery path that can route users to FunASR, SenseVoice, Fun-ASR-Nano, and FunClip together | PR is clean and mergeable; monitor without status bumps unless maintainers request section naming or Chinese wording changes. |
| `mahseema/awesome-ai-tools#1403` FunASR tool listing | Complements the FunClip listing in the same AI-tools catalog with the core ASR toolkit | PR is clean and mergeable; keep FunASR and FunClip descriptions distinct so one does not look like a duplicate of the other. |
| `INTERMT/Awesome-PyTorch-Chinese#5` FunASR Chinese PyTorch listing | Adds FunASR to a Chinese PyTorch resource list used by developers looking for local speech models and toolkits | PR is clean and mergeable; monitor for maintainer style feedback and keep the link pointed at the GitHub repo rather than a transient model page. |
| `krzjoa/awesome-python-data-science#99` FunASR computer-audition listing | Adds FunASR to a Python data-science discovery list where speech/audio tooling is compared with broader ML stacks | PR is clean and mergeable; maintain a short toolkit-focused description and avoid duplicate comments unless maintainers ask for category changes. |
| `zzw922cn/awesome-speech-recognition-speech-synthesis-papers#27` FunAudioLLM paper listing | Places Paraformer, FunASR, SenseVoice, and Fun-ASR-Nano papers in a speech-recognition/synthesis paper list used by researchers | PR is clean and mergeable; keep paper links stable and be ready to adjust ordering or paper metadata if requested. |
| `Osmantic/ODS#1639` SenseVoice/FunASR STT backend | Adds a direct local STT backend to an app surface where users can choose open speech engines | PR is mergeable but blocked by review/process state; wait for maintainer feedback and keep any follow-up focused on optional dependency and endpoint behavior. |
| `faroit/awesome-python-scientific-audio#85` FunASR scientific-audio listing | Exposes FunASR to Python users browsing scientific audio and speech-processing toolkits | PR is clean and mergeable; monitor for taxonomy or wording feedback without status bumps. |
| `joewongjc/type4me#207` Qwen3-only local ASR mode | Helps a local dictation app use Qwen3-ASR final transcription without keeping SenseVoice/VAD preview models resident | Keep as draft until a full Xcode/XCTest run or real Qwen3-only ASR smoke test validates the runtime path; do not request review yet. |
| `ga642381/speech-trident#31` SenseVoice model-list entry | Adds SenseVoice to a speech/audio language-model catalog that routes model researchers toward FunASR ecosystem components | PR is clean and mergeable; monitor for citation or category feedback. |
| `vinta/awesome-python#3246` SenseVoice listing | Places SenseVoice in the largest Python discovery list as a practical local speech-recognition option | PR is blocked by maintainer/review gate rather than a known code failure; watch for category or project-quality feedback before changing scope. |
| `RVC-Boss/GPT-SoVITS#2801` Fun-ASR-Nano fallback fix | Improves a high-star voice generation workflow by making Fun-ASR-Nano setup errors fall back cleanly instead of blocking users | PR is clean and mergeable; monitor for maintainer questions and keep the fix narrowly scoped to registration/fallback behavior. |
| `jobbole/awesome-python-cn#141` FunASR Chinese Python listing | Adds FunASR to a long-lived Chinese Python resource list used by domestic developers | PR is clean and mergeable; avoid duplicate comments unless maintainers ask for ordering or Chinese wording changes. |
| `yuekaizhang/Fun-ASR-vllm#21` vLLM prompt embedding dtype fix | Keeps a community Fun-ASR-vLLM GPU inference path stable for users trying Fun-ASR-Nano with vLLM | PR is clean and has a validation note. Monitor for maintainer feedback, keep the change limited to float32 prompt embeddings, and avoid extra status bumps unless runtime questions appear. |
| `openvino-agent/optimum-intel#5` FunASR OpenVINO review cleanup | Preserves an OpenVINO review-structure cleanup branch while the upstream optimum-intel path is maintainer-owned | Keep as a reference branch for review cleanup and do not ping externally unless maintainers ask for the generated-file or modularization details. |
| `EmulationAI/awesome-large-audio-models#19` FunAudioLLM paper listing | Adds Paraformer, FunASR, SenseVoice, and Fun-ASR-Nano papers to a large-audio-models discovery list | README-only PR with validation posted; wait for maintainer review and be ready to adjust paper ordering or citation text. |
| `ddlBoJack/Awesome-Speech-Language-Model#6` Fun-ASR-Nano listing | Routes speech language model readers toward Fun-ASR-Nano as an audio-LLM component | Validation was posted on 2026-07-07; PR is clean with no exposed checks, so monitor without duplicate pings. |
| `LqNoob/Neural-Codec-and-Speech-Language-Models#4` Fun-ASR-Nano and SenseVoice listing | Adds the FunAudioLLM speech understanding models to a neural codec and speech-language-model resource list | Validation was posted on 2026-07-07; wait for maintainer review and keep any follow-up focused on model categorization. |
| `PyTorchKR/oss-landscape#688` FunASR OSS landscape entry | Adds FunASR to a Korean PyTorch and OSS discovery map that can route regional users to the toolkit | PR is README-only, clean, and has no check contexts; avoid low-signal comments unless maintainers ask for metadata or category changes. |
| `metame-ai/awesome-audio-plaza#10` FunASR, SenseVoice, and FunClip ASR listing | Presents the full FunASR ecosystem together in an audio tools plaza used by speech and creator-tool readers | Prior pings are already present; monitor for maintainer feedback and keep future replies limited to link or category changes. |
| `ChristosChristofidis/awesome-deep-learning#317` FunASR framework listing | Adds FunASR to a 28k-star deep-learning discovery list where users compare frameworks and toolkits | Validation was refreshed on 2026-07-07; wait for maintainer review without another status bump. |
| `Hannibal046/Awesome-LLM#623` Fun-ASR-Nano Alibaba-section listing | Routes a large LLM audience to Fun-ASR-Nano as a speech understanding model in the Alibaba ecosystem | Validation was refreshed on 2026-07-07; wait for maintainer review and adjust section placement only if requested. |
| `AiHubCN/Awesome-Chinese-LLM#103` SenseVoice and Fun-ASR-Nano Chinese LLM listing | Adds FunAudioLLM speech models to a high-star Chinese LLM catalog with strong domestic discovery value | Validation was refreshed on 2026-07-07; PR is currently blocked by maintainer review/process state rather than a code failure. |
| `pluja/awesome-privacy#836` FunASR privacy-focused STT listing | Puts fully self-hosted FunASR in front of privacy-conscious users looking for local speech-to-text models | Validation was refreshed on 2026-07-07; wait for maintainer review and keep privacy wording concise. |
| `BradyFU/Awesome-Multimodal-Large-Language-Models#280` Fun-ASR-Nano MLLM listing | Adds Fun-ASR-Nano to a multimodal-LLM discovery list where speech understanding models are compared | Validation was refreshed on 2026-07-07; monitor for maintainer category feedback. |
| `mahmoud/awesome-python-applications#227` FunASR audio application listing | Adds FunASR to a broad Python applications list used by developers looking for usable audio tooling | Validation was refreshed on 2026-07-07; wait for maintainer review without extra pings. |
| `bharathgs/Awesome-pytorch-list#164` FunASR PyTorch speech listing | Adds FunASR to a PyTorch NLP and speech processing list with a broad ML practitioner audience | Validation was refreshed on 2026-07-07; PR is clean and should only need maintainer review. |
| `WangRongsheng/awesome-LLM-resources#162` FunASR and SenseVoice resources | Adds FunASR/SenseVoice to a Chinese LLM resources list where multimodal and speech tool readers compare projects | Validation was refreshed on 2026-07-07; wait for maintainer review and keep future replies focused on link stability. |
| `steven2358/awesome-generative-ai#821` FunASR speech-to-text listing | Routes generative-AI readers looking for STT tools toward the core FunASR repo | PR is clean and mergeable; monitor for section placement feedback. |
| `owainlewis/awesome-artificial-intelligence#243` FunASR audio listing | Adds another broad AI-discovery path for users comparing open-source audio tools | PR is clean and mergeable; keep the description concise and avoid status pings. |
| `ai4s-research/awesome-ai-for-science#69` FunASR science toolkit listing | Creates a discovery path from scientific AI tooling lists to FunASR for transcription and field-recording workflows | Validate the link, keep the description technically accurate, and avoid extra comments unless maintainers ask for category or wording changes. |
| `lukasmasuch/best-of-ml-python#455` FunASR project listing | Adds FunASR to a high-star ranked Python ML discovery list that is refreshed weekly | The PR is mergeable and already has prior pings; keep monitoring without adding more comments unless new maintainer feedback or validation evidence appears. |
| `tensorchord/Awesome-LLMOps#533` FunASR audio foundation model listing | Adds FunASR to an LLMOps discovery list where users look for model-serving and operations-ready audio foundation models | Current PR is README-only, clean, DCO-passing, and has fresh FunASR-side link validation; wait for maintainer review without duplicate pings. |
| `rafska/awesome-local-llm#118` FunASR local CPU ASR listing | Exposes the FunASR llama.cpp/GGUF path to local-LLM users looking for fully local CPU speech recognition | Current PR is README-only, clean, and has fresh link validation; monitor for maintainer feedback without adding another status comment. |
| `krzemienski/awesome-video#102` FunClip AI video tools listing | Adds FunClip to a curated streaming/video tools list where video engineers and creator-tool builders discover AI-assisted clipping and subtitle generation workflows | Sourcery's rerun passes on head `f90be00`; the PR is README-only, mergeable clean, and uses owner/repo-style link text `[modelscope/FunClip]`. LauraGPT posted a concise ready/validation note on 2026-07-08 at `https://github.com/krzemienski/awesome-video/pull/102#issuecomment-4911378040`; now wait for maintainer review without duplicate pings. |
| `CopilotKit/CopilotKit#5863` voice runtime transcription docs | Puts self-hosted OpenAI-compatible ASR behind a 35k-star agent UI voice surface by clarifying that custom `TranscriptionService` implementations can proxy CopilotKit `/transcribe` requests to local or LAN STT endpoints | LauraGPT posted a 2026-07-08 downstream compatibility note at `https://github.com/CopilotKit/CopilotKit/pull/5863#issuecomment-4911282937`, suggesting an optional self-hosted ASR callout for FunASR, SenseVoice, and Qwen3-ASR-style `/v1/audio/transcriptions` deployments. Wait for maintainer/doc-author feedback; do not turn this into a blocking review. |
| `manaflow-ai/cmux#7314` provider-neutral iOS voice backend | Puts a 23k-star AI terminal's iOS Voice Mode and composer dictation behind a recognition-backend seam where local/on-device Apple or Parakeet engines can later coexist with LAN/self-hosted FunASR/SenseVoice/Qwen3-ASR transcription gateways | LauraGPT posted a non-blocking backend-contract note at https://github.com/manaflow-ai/cmux/pull/7314#issuecomment-4912323678: keep engine capability metadata provider-neutral, send exactly one final transcript to `mobile.voice.input`, preserve fail-closed target checks outside the ASR engine, sanitize provider errors, and keep downloaded model state separate from any future remote endpoint settings. Track review without asking this PR to add another engine. |
| `QwenLM/qwen-code#6516` VoiceButton i18n and provider-error boundary | Makes voice dictation states and retry affordances translatable in the same high-visibility Qwen Web Shell surface, which matters for non-English users running local FunASR/SenseVoice STT backends | LauraGPT posted the local/private STT note at https://github.com/QwenLM/qwen-code/pull/6516#issuecomment-4911909587. Latest head `79f3294e651211de4095947eec35e33d94da5c7a` adds locale-cache invalidation after the earlier missed placeholder fix; Ubuntu Node 22 tests and coverage comment passed, while `review-pr` is still in progress. Reviewer `wenshao` downgraded to non-blocking comment at https://github.com/QwenLM/qwen-code/pull/6516#issuecomment-4912381983 and suggested i18n interpolation/timeline tests plus sentinel cleanup. Monitor checks/review; avoid another FunASR comment unless provider-error or voice insertion tests are requested. |
| `agent-of-empires/agent-of-empires#2585` composer action extension point | Adds a provider-agnostic composer action API in an agent UI, creating the right plugin boundary for local FunASR/SenseVoice dictation without hard-coding STT into core | Maintainer agreed with LauraGPT's provider-neutral boundary at https://github.com/agent-of-empires/agent-of-empires/pull/2585#issuecomment-4911979551 and asked the author to rebase. Keep STT-provider code out of this PR; after the extension point is clean, propose or review a plugin that calls an OpenAI-compatible FunASR/SenseVoice `/v1/audio/transcriptions` endpoint. |
| `tmoroney/auto-subs#652` Whisper download failure with SenseVoice fallback | Protects a live subtitle workflow after AutoSubs added SenseVoice: a Chinese-video user is blocked by `large-v3-turbo` whisper.cpp model download failure, not by audio normalization or ASR execution | LauraGPT posted safe triage at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4911843184: avoid unverified ZIP/bypass scripts, keep the Whisper fix in the model-manager/cache path, and use the built-in SenseVoice path from #629 plus the app translation route as the immediate `lang=zh` fallback. A new user asked for 1:1 help, so LauraGPT moved support back into the public issue at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4912472340; after they asked which option is SenseVoice, LauraGPT identified the UI label/model id at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4912653724. Track whether they can see `SenseVoice` / `sense-voice`, whether their build includes the newer engine, and whether Whisper remains isolated to download/cache handling. |
| `OpenWhispr/openwhispr#769` custom endpoint transcription contract | Keeps a privacy-first dictation app's BYOK/custom STT route compatible with OpenAI-style multipart transcription endpoints, the same wire shape used by self-hosted FunASR/SenseVoice gateways | LauraGPT posted the regression-contract checklist at https://github.com/OpenWhispr/openwhispr/issues/769#issuecomment-4909852583 after a user confirmed the OpenRouter fix branch worked. Track for a main/release-side fix that preserves `file`, `model`, optional `language`, optional `response_format=verbose_json`, exact `/audio/transcriptions` base-url joining, safe `audio/webm` handling, and provider-specific JSON/base64 only when required. |
| `mahseema/awesome-ai-tools#1689` FunClip video-tool listing | Adds FunClip to a high-visibility AI tools list where video creators discover clipping and subtitle workflows | Validate the FunClip link, keep the description aligned with local transcription/SRT/AI clipping capabilities, and avoid status pings after evidence is posted. |
Completed external wins:
| Integration | Growth reason | Result |
|---|---|---|
| `xinnan-tech/xiaozhi-esp32-server#3255` configurable FunASR language | Improves a high-star ESP32 voice-agent backend by letting users pin SenseVoice/FunASR language for short utterances | Merged on 2026-06-30 after adding the multi-module management-console database migration requested by maintainers. |
| `tmoroney/auto-subs#629` SenseVoice engine for subtitle workflows | Adds SenseVoice to an on-device subtitle generation app used by video editors and creators | Merged on 2026-06-30 after adding the declarative model manifest, SenseVoice/Canary/Cohere engines, real CTC timestamps, native translation routing, and robustness fixes. |
| `pipecat-ai/pipecat#4844` FunASR local STT service | Puts SenseVoice/FunASR into realtime voice agent pipelines used by builders comparing local STT backends | Merged on 2026-07-02 after adding the local STT service, optional dependency handling, docs, and validation coverage. |
| `liusongxiang/Large-Audio-Models#26` FunAudioLLM audio-language-model listing | Adds FunAudioLLM to a focused audio language model catalog where researchers compare speech and audio LLM projects | Merged on 2026-07-03 with a README-only FunAudioLLM entry. |
| `yuekaizhang/Fun-ASR-vllm#20` deterministic vLLM sampling for ASR | Improves a community Fun-ASR-vLLM inference path by making ASR generation deterministic enough for repeatable evaluation and demos | Merged on 2026-07-07 after switching the vLLM sampling path to deterministic decoding for ASR. Follow-up `#21` continues the prompt-embedding dtype cleanup. |
| `xorbitsai/inference#5140` Fun-ASR-Nano Xinference dependency fix | Unblocks Fun-ASR-Nano deployment through Xinference by moving model specs off an old FunASR git pin that predates `FunASRNano` registration | Merged on 2026-07-07 as `4a625fafa55827ee932f1e2cfcd362b9a7b4aabe`; LauraGPT synced FunASR deployment docs in https://github.com/modelscope/FunASR/pull/3117 and posted the merge follow-up at https://github.com/xorbitsai/inference/pull/5140#issuecomment-4901893926. Track Xinference release notes so FunASR docs can point users at a fixed build. |
| `debpalash/OmniVoice-Studio#1003` OpenAI-compatible ASR backend | Gives an 8k-star local voice/video studio a generic `POST /v1/audio/transcriptions` path that can point at self-hosted FunASR, SenseVoice, or Qwen3-ASR servers before direct Transformers support lands | Merged by maintainer `debpalash` on 2026-07-08 as `5a7d9cc05cda45682b11b5ece35190a5c39bbb6f` after LauraGPT's timeout, fallback-scope, key-clear, and error-sanitization review note. Watch release/user feedback for real FunASR/SenseVoice endpoint smoke results; do not post another follow-up unless those hardening items resurface. |
| `OpenWhispr/openwhispr#1093` transcript-retention fix | Strengthens a privacy-first dictation and meeting transcription app so local/private STT providers do not lose genuine microphone speech when echo evidence is audio-only | Merged on 2026-07-08 as `4d6f2d2e8b4602feb72f1493aff1f8f12de9e3f3` after LauraGPT highlighted the final-text retention contract; watch release feedback and future OpenAI-compatible/local provider work without posting duplicate follow-ups. |
| `elizaOS/eliza#15426` streaming ASR live transcript contract | Lands a high-star agent OS voice UX path where segment preview and final batch transcript semantics can support local/private FunASR or SenseVoice-style ASR providers | Merged on 2026-07-08 as `e8d780cd31d1b5e8a870c20e370da0010ce8eb2d` after LauraGPT called out the final-transcript safety contract at https://github.com/elizaOS/eliza/pull/15426#issuecomment-4911110675. Watch follow-up provider work and keep any FunASR bridge aligned with the final-only fallback contract. |
| `QwenLM/qwen-code#6510` pane-scoped voice dictation restore | Restores voice dictation in a 25k-star Qwen coding-agent Web Shell split-view, creating a visible local/private STT surface when the daemon advertises `voice_transcribe` | Merged on 2026-07-08 as `d8dc8043d6cfd4d7605f83b8a4838fee9dacdffa` after LauraGPT's pane-scoped local/private ASR boundary note at https://github.com/QwenLM/qwen-code/pull/6510#issuecomment-4911877569 and maintainer confirmation at https://github.com/QwenLM/qwen-code/pull/6510#issuecomment-4912022943: capability-gated mic visibility, editable transcript draft without auto-submit, pane-specific routing, and sanitized provider errors for FunASR/SenseVoice/OpenAI-compatible STT backends. |
| `chatmcp/mcpso#1` MCP server directory submission | Routes MCP builders browsing mcp.so toward FunASR's local speech-recognition MCP server | Listed on mcp.so on 2026-07-08 at https://mcp.so/server/mcp-server-funasr/radial-hks after LauraGPT submitted the server in the public directory thread. The MCP README now links the live directory page so Claude/Cursor-style users can discover `transcribe_audio` without reading the growth tracker. |
Operating rules:
- Comment on external PRs only when there is new evidence, a reviewer question, or a concrete unblock; avoid low-signal status bumps.
- Keep each integration's local verification command, CI link, and latest reviewer decision in the weekly notes.
- Promote an integration in FunASR README or release notes only after it is merged or has a maintained, runnable branch.
- When a third-party PR stalls, look for a smaller docs/example PR in that upstream instead of forcing a large runtime integration.
## Tracking cadence
Use the lightweight metrics script before and after major README, homepage, release, or demo updates:
```bash
python scripts/collect_growth_metrics.py
python scripts/collect_growth_metrics.py --format json
```
The script captures GitHub stars, forks, watchers, open issues, open pull requests, latest push time, and the current PyPI version using public APIs. Set `GITHUB_TOKEN` when running it from CI or a shared network to avoid public GitHub API rate limits. Paste the Markdown output into launch notes, weekly community updates, or release retrospectives so the 20k-star effort stays measurable.
## 30-day execution checklist
### Week 1: Repository conversion
- [ ] Verify README quick start on clean CPU and GPU environments.
- [ ] Verify all README links and docs links.
- [ ] Add or refresh CONTRIBUTING, PR template, and issue templates.
- [ ] Add a short FAQ for install/deployment failures.
- [ ] Confirm GitHub repo description and topics mention ASR, speech-recognition, streaming, diarization, OpenAI-compatible API, MCP, and vLLM.
### Week 2: Deployment proof
- [ ] Publish API server curl examples.
- [ ] Publish WebSocket streaming examples.
- [ ] Validate Docker CPU and GPU images.
- [ ] Validate vLLM guide on one GPU server.
- [ ] Record concise terminal outputs for release notes.
### Week 3: Launch package
- [ ] Create a GitHub release focused on one sharp value proposition.
- [ ] Update PyPI release description if needed.
- [ ] Update Hugging Face organization/model cards.
- [ ] Update ModelScope model cards and demos.
- [ ] Update homepage hero and docs entry points.
### Week 4: Distribution and feedback
- [ ] Share the release in relevant developer communities.
- [ ] Pin a GitHub discussion for the release.
- [ ] Triage all new issues within 48 hours.
- [ ] Convert top 3 support questions into docs.
- [ ] Track stars, PyPI downloads, issue volume, and docs traffic.
## Release-note template
````markdown
## Why this release matters
FunASR now lets you ...
## Try it in 60 seconds
```bash
pip install -U funasr
funasr-server --device cuda
```
## Highlights
- ...
- ...
- ...
## Verified environments
- GPU:
- CPU:
- Docker:
## Links
- Quick start:
- Deployment guide:
- Benchmark:
- Discussion:
````
## 中文摘要
目标不是单纯“求 Star”，而是让更多用户能更快完成安装、转写、部署和分享。当前增长目标按 FunASR / Fun-ASR / SenseVoice / FunClip 四仓生态统一跟踪：以 2026-06-27 的 31,224 combined stars 为基线，到 2026-09-30 累计新增 20,000 stars。最有效的抓手是：首屏转化、可复现 benchmark、OpenAI 兼容 API/流式/vLLM/Docker 部署闭环、HF/ModelScope/官网同步更新、高星生态集成，以及高效的 issue/PR 社区运营。
docs/community_projects.md
+40
View File
@@ -0,0 +1,40 @@
# FunASR Community Projects
This page collects maintained community integrations and experiments around FunASR models. These projects are useful for users exploring deployment paths beyond the official Python runtime, but they are not official FunASR implementations unless explicitly noted.
## VAD runtimes
| Project | What it provides | Notes |
|---|---|---|
| [vad-burn](https://github.com/di-osc/vad-burn) | FSMN VAD inference in pure Rust with Python bindings. It supports offline VAD, streaming VAD, CPU-only inference, and automatic download of the default `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` model from ModelScope. | Community project from [#3106](https://github.com/modelscope/FunASR/issues/3106). Useful for Rust services, CLI tools, desktop apps, or Python pipelines that need FSMN VAD without depending on the original Python runtime. |
`vad-burn` reports the following benchmark on `assets/vad_example.wav` (16 kHz mono PCM, 70.47 seconds) on a MacBook Pro M1 with the Burn Flex CPU backend:
| Mode | Avg time | RTF | Speedup |
|---|---:|---:|---:|
| FSMN VAD offline | 73.631 ms | 0.001045 | 957.08x |
| FSMN VAD streaming, 600 ms chunks | 198.425 ms | 0.002816 | 355.15x |
Minimal Python usage from the project:
```python
from vad_burn import FsmnVadModel, VadOptions
vad = FsmnVadModel.from_modelscope()
segments = vad.detect(samples, 16000, VadOptions())
stream = vad.new_stream(VadOptions())
for chunk in chunks:
segments = stream.push(chunk, 16000)
final_segments = stream.finish()
```
## Add your project
If you maintain a FunASR integration, open a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md) with:
- repository link and maintenance status
- supported FunASR model or runtime path
- install and minimal usage instructions
- benchmark or validation details when available
- a note about whether the project is official, community-maintained, or experimental
docs/conf.py
+67
View File
@@ -0,0 +1,67 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = "FunASR"
copyright = "2022, Speech Lab, Alibaba Group"
author = "Speech Lab, Alibaba Group"
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"nbsphinx",
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.viewcode",
"sphinx.ext.mathjax",
"sphinx.ext.todo",
# "sphinxarg.ext",
"sphinx_markdown_tables",
"recommonmark",
"sphinx_rtd_theme",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
source_suffix = [".rst", ".md"]
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "sphinx_rtd_theme"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
docs/deployment_matrix.md
+111
View File
@@ -0,0 +1,111 @@
# FunASR Deployment Matrix
Use this page to choose the shortest deployment path for a product, demo, benchmark, or internal workflow. Start with the smallest surface that satisfies the job, then move to heavier runtimes only when throughput, latency, or integration requirements demand it.
## Quick decision table
| Path | Best for | Start here | Operational notes |
|---|---|---|---|
| Colab notebook | Browser smoke tests, first evaluation, shareable demos | [Colab quickstart](../examples/colab/) | No local setup; first run downloads model files, GPU runtime is faster. |
| Python API | Notebooks, offline jobs, first model evaluation | [README quick start](../README.md#quick-start) | Lowest ceremony; caller owns batching, retries, and files. |
| OpenAI-compatible API | Private speech API, agents, Dify/LangChain/AutoGen-style clients | [OpenAI API example](../examples/openai_api/) | Easiest integration for apps that already support OpenAI audio APIs. |
| Xinference | Teams that already operate Xinference model serving | [Xinference repository](https://github.com/xorbitsai/inference) | Use a Xinference version containing [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140) so Fun-ASR-Nano uses packaged `funasr~=1.3.0` instead of the old pinned git commit. |
| Docker Compose API | Reproducible local smoke test or small internal service | [OpenAI API Docker docs](../examples/openai_api/#docker-deployment) | CPU by default; adapt the image before using CUDA in containers. |
| Kubernetes API | Internal speech API for cluster services | [Kubernetes template](../examples/openai_api/kubernetes/) | Starts as private `ClusterIP`; add auth, TLS, network policy, and GPU scheduling before broader exposure. |
| Runtime WebSocket service | Live captions, meetings, call-center streams | [Runtime service docs](../runtime/readme.md) | Use when partial results, endpointing, or long-lived audio streams matter. |
| ONNX/C++ runtime | High-concurrency CPU services or embedded realtime ASR | [ONNX runtime docs](../runtime/onnxruntime/readme.md) | Keep this path when latency/concurrency is already proven; add text post-processing for fixed business terms before moving to GPU LLMs. |
| vLLM acceleration | Higher-throughput LLM-based ASR with Fun-ASR-Nano | [vLLM guide](./vllm_guide.md) | Use for LLM decoder throughput; does not apply to non-autoregressive Paraformer. |
| MCP server | Claude/Cursor/desktop agent speech tools | [MCP example](../examples/mcp_server/) | Good when the ASR result should be exposed as a local tool. |
| Subtitle generator | SRT/VTT from long audio or video | [Subtitle example](../examples/subtitle/) | Use verbose segments and speaker labels when readability matters. |
| Batch ASR script | Archives, meetings, datasets, repeated offline runs | [Batch example](../examples/batch_asr_improved.py) | Add queueing, manifests, and retry logs for production use. |
| Triton runtime | Specialized high-performance serving | [Triton runtime docs](../runtime/triton_gpu/README.md) | Heavier setup; choose when your team already operates Triton/GPU serving. |
## Common choices
### I want to try FunASR in five minutes
Use the [Colab quickstart](../examples/colab/) when you want a browser-only smoke test, or use the Python API from the README for local work. It is the shortest route for validating installation, model download, device selection, and basic output shape. If you are unsure which model to start with, use the [model selection guide](./model_selection.md).
### I want a local replacement for cloud transcription
Use the OpenAI-compatible API. It exposes `/v1/audio/transcriptions`, `/v1/models`, `/health`, and Swagger docs. Start with `sensevoice`, run `examples/openai_api/smoke_test.sh` or `examples/openai_api/smoke_test.py`, then connect existing SDK or HTTP clients using [client recipes](../examples/openai_api/CLIENTS.md) and [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md). For browser upload or microphone demos, use the [Gradio browser demo](../examples/openai_api/GRADIO.md). For Dify, n8n, HTTP nodes, or webhook workers, follow the [workflow recipes](../examples/openai_api/WORKFLOWS.md). For API gateways, developer portals, and schema-driven imports, use the [OpenAPI spec](../examples/openai_api/OPENAPI.md). Before sharing the service, review the [security and gateway guide](../examples/openai_api/SECURITY.md).
### I already run Xinference
Use Xinference when your stack already standardizes on its model registry,
virtualenv isolation, and serving lifecycle. Make sure your Xinference build
includes [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140);
that update moved the Fun-ASR-Nano model specs from an old FunASR git SHA to the
packaged `funasr~=1.3.0` dependency line. For first-time FunASR evaluation or
agent-oriented OpenAI-compatible transcription, start with the native FunASR
OpenAI API example above instead.
### I want a repeatable container demo
Use `examples/openai_api/docker-compose.yml` for a CPU-mode smoke test:
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
Keep CPU mode until you have a CUDA-capable PyTorch/FunASR image. After that, set `FUNASR_DEVICE=cuda` and verify with the same smoke test. Use `python examples/openai_api/smoke_test.py --base-url http://localhost:8000` on systems without bash/curl.
### I want an internal Kubernetes service
Use the [Kubernetes template](../examples/openai_api/kubernetes/) for a private `ClusterIP` OpenAI-compatible API with persistent model cache, `/health` probes, and a port-forward smoke-test path. Keep the CPU default until you have a CUDA-capable image and cluster GPU scheduling in place.
### I need streaming or live captioning
Use the runtime WebSocket service. Validate chunk size, VAD, endpointing, punctuation, speaker diarization, reconnect behavior, and client backpressure with real audio before production rollout.
### I already have high-concurrency ONNX and need hotwords
Do not move an entire working CPU ONNX/C++ realtime stack to a GPU LLM only
because the current path lacks hotwords. First decide what "hotword" means in
your product:
- For fixed company names, product names, or common misrecognitions, keep the
ONNX path and add deterministic text post-processing on final results. This
preserves proven CPU concurrency and is easier to audit.
- For true decoder-time biasing, pronunciation ambiguity, or multilingual
accuracy gaps, test a GPU path such as Fun-ASR-Nano or Qwen3-ASR, but size it
against your target first-token latency, final latency, audio chunk length,
VAD policy, and simultaneous speaking sessions.
- If both are needed, use the GPU model only where it wins on quality or
language coverage, and keep ONNX for the high-volume traffic that already
meets latency and accuracy requirements.
When opening a deployment issue, include your current ONNX concurrency, CPU
cores, target terms, acceptable end-to-end latency, GPU model, model command,
and whether hotwords are post-recognition corrections or decoder-time biasing.
### I need more LLM-based ASR throughput
Use the vLLM path for Fun-ASR-Nano. Benchmark with your own audio distribution and watch GPU memory, tensor parallel size, first-token latency, and warmup time.
### I want to run Fun-ASR-Nano on Ascend NPU
Fun-ASR-Nano's LLM-based path is documented and validated for CUDA/vLLM, standard PyTorch CPU/GPU runs, and CPU/edge GGUF runtimes. Ascend NPU (`torch_npu`) is still not an official production runtime for this model. Do not assume that a SenseVoice or Paraformer NPU deployment means Fun-ASR-Nano will also work, because Nano also exercises the Qwen decoder, `inputs_embeds`, and backend-specific autocast/operator paths.
A community smoke test in [#3034](https://github.com/modelscope/FunASR/issues/3034) confirmed that the PyTorch `AutoModel(..., device="npu:*")` path can now enter the NPU backend after the autocast device fix, and produced the expected transcript on a 310P3 with `torch_npu 2.5.1` / CANN 8.5.1. That run was much slower than CPU (`rtf_avg` about 124 on NPU vs about 1.9 on CPU), so treat it as compatibility evidence, not a performance recommendation. The same report showed `AutoModelVLLM` on `vllm_ascend 0.9.2rc1` failing inside Qwen3 rotary embedding / `TransData` operator support; debug that path as a vLLM-Ascend runtime/operator issue and capture logs with `ASCEND_LAUNCH_BLOCKING=1`.
If you are adapting this backend, keep the first PR narrow: start with `torch.bfloat16`, capture `torch` / `torch_npu` / CANN / driver / NPU model, separate PyTorch `AutoModel` from `AutoModelVLLM`, and attach the minimal command plus full stack trace.
## Readiness checklist
- Pick a model alias and pin it in deployment notes.
- Record FunASR version, model version, device, CUDA/PyTorch version, Docker image tag, and command line.
- Run a short public smoke sample and at least one realistic private sample.
- Log audio duration, model, device, latency, response format, and error type for every request.
- Add upload-size limits, authentication, TLS, and rate limits before exposing an API outside a trusted network; use the [security and gateway guide](../examples/openai_api/SECURITY.md) to plan the boundary.
- For hotword or correction requirements, record whether the change is
deterministic post-processing or decoder-time biasing, then benchmark quality
and latency before replacing an existing runtime.
- For streaming, test silence, noise, overlapping speakers, long sessions, reconnects, and slow clients.
- For benchmark claims, include input duration, hardware, batch size, model, runtime path, and whether model download/warmup time is excluded.
## When to open an issue
Use [Deployment Help](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) for runtime, Docker, vLLM, Triton, Android, browser, or agent integration problems. Include your deployment path, exact command/config, logs, model, device, and audio characteristics.
docs/deployment_matrix_ja.md
+53
View File
@@ -0,0 +1,53 @@
# FunASR デプロイ選択マトリクス
プロダクト、デモ、ベンチマーク、社内ワークフローに合わせて最短のデプロイ経路を選ぶためのガイドです。まずは要件を満たす最小構成から始め、throughput、latency、integration 要件が明確になったら重い runtime に移行してください。
## クイック判断表
| Path | 向いている用途 | 最初に読むもの | 運用メモ |
|---|---|---|---|
| Colab notebook | ブラウザ smoke test、初回評価、共有 demo | [Colab クイックスタート](../examples/colab/README_ja.md) | ローカル環境不要。初回はモデルをダウンロードし、GPU runtime の方が高速です。 |
| Python API | Notebook、offline job、最初の model evaluation | [README quick start](../README_ja.md#クイックスタート) | 最小構成。batching、retry、file 管理は呼び出し側で扱います。 |
| OpenAI 互換 API | Private speech API、Agent、Dify/LangChain/AutoGen style clients | [OpenAI API example](../examples/openai_api/README_ja.md) | OpenAI audio API に対応した既存 app に最も接続しやすい経路です。 |
| Docker Compose API | 再現可能な local smoke test、小さな internal service | [OpenAI API Docker docs](../examples/openai_api/README_ja.md#docker-デプロイ) | デフォルトは CPU。CUDA を使う前に CUDA-capable image へ調整してください。 |
| Kubernetes API | Cluster service 向け internal speech API | [Kubernetes template](../examples/openai_api/kubernetes/) | private `ClusterIP` から開始。公開範囲を広げる前に auth、TLS、network policy、GPU scheduling を追加します。 |
| Runtime WebSocket service | Live captions、meeting、call-center stream | [Runtime service docs](../runtime/readme.md) | partial result、endpointing、long-lived audio stream が重要な場合に使います。 |
| vLLM acceleration | Fun-ASR-Nano の LLM-based ASR throughput 向上 | [vLLM guide](./vllm_guide.md) | LLM decoder throughput 向け。non-autoregressive Paraformer には適用しません。 |
| MCP server | Claude/Cursor/desktop agent の speech tool | [MCP example](../examples/mcp_server/) | ASR 結果を local tool として Agent に渡したい場合に便利です。 |
| Subtitle generator | 長時間 audio/video から SRT/VTT 作成 | [Subtitle example](../examples/subtitle/) | readability が重要な場合は verbose segment と speaker label を使います。 |
| Batch ASR script | Archive、meeting、dataset、繰り返し offline run | [Batch example](../examples/batch_asr_improved.py) | production では queue、manifest、retry log を追加してください。 |
## よくある選択
### 5分で FunASR を試したい
ブラウザだけで試すなら [Colab クイックスタート](../examples/colab/README_ja.md) を使います。ローカルで作業する場合は README の Python API から始めます。どのモデルを使うか迷う場合は [モデル選択ガイド](./model_selection_ja.md) を参照してください。
### Cloud transcription の local replacement が欲しい
OpenAI 互換 API を使います。`/v1/audio/transcriptions`、`/v1/models`、`/health`、Swagger docs を提供します。まず `sensevoice` で smoke test を実行し、既存 SDK や HTTP client を [OpenAI API example](../examples/openai_api/README_ja.md) に合わせて接続してください。
### 再現可能な container demo が欲しい
`examples/openai_api/docker-compose.yml` を CPU mode の smoke test として使います。
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
CUDA を使う場合は CUDA-capable PyTorch/FunASR image を作成してから `FUNASR_DEVICE=cuda` に変更し、同じ smoke test で確認します。
### Streaming または live captioning が必要
Runtime WebSocket service を使います。本番投入前に chunk size、VAD、endpointing、punctuation、speaker diarization、reconnect、client backpressure を実音声で検証してください。
## Readiness checklist
- model alias を決め、deployment note に固定します。
- FunASR version、model version、device、CUDA/PyTorch version、Docker image tag、command line を記録します。
- public smoke sample と realistic private sample を少なくとも 1 つずつ実行します。
- request ごとに audio duration、model、device、latency、response format、error type をログ化します。
- trusted network の外へ API を出す前に upload-size limit、authentication、TLS、rate limit を入れます。[Security guide](../examples/openai_api/SECURITY.md) も確認してください。
- 詰まったら deployment path、command/config、logs、model、device、audio characteristics を添えて [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) を開いてください。
docs/deployment_matrix_ko.md
+53
View File
@@ -0,0 +1,53 @@
# FunASR 배포 선택 매트릭스
제품, 데모, 벤치마크, 내부 워크플로에 맞는 가장 짧은 배포 경로를 고르기 위한 가이드입니다. 먼저 요구를 만족하는 최소 구성에서 시작하고, throughput, latency, integration 요구가 명확해질 때 더 무거운 runtime으로 이동하세요.
## 빠른 결정 표
| Path | 적합한 용도 | 시작 문서 | 운영 메모 |
|---|---|---|---|
| Colab notebook | 브라우저 smoke test, 첫 평가, 공유 가능한 demo | [Colab 빠른 시작](../examples/colab/README_ko.md) | 로컬 환경이 필요 없습니다. 첫 실행은 모델을 다운로드하며 GPU runtime이 더 빠릅니다. |
| Python API | Notebook, offline job, 첫 model evaluation | [README quick start](../README_ko.md#빠른-시작) | 가장 단순한 경로입니다. batching, retry, file 관리는 호출 측에서 담당합니다. |
| OpenAI 호환 API | Private speech API, Agent, Dify/LangChain/AutoGen style clients | [OpenAI API example](../examples/openai_api/README_ko.md) | OpenAI audio API를 이미 지원하는 앱에 가장 쉽게 연결됩니다. |
| Docker Compose API | 재현 가능한 local smoke test, 작은 internal service | [OpenAI API Docker docs](../examples/openai_api/README_ko.md#docker-배포) | 기본은 CPU입니다. CUDA를 쓰기 전에 CUDA-capable image로 조정하세요. |
| Kubernetes API | Cluster service용 internal speech API | [Kubernetes template](../examples/openai_api/kubernetes/) | private `ClusterIP`부터 시작합니다. 범위를 넓히기 전에 auth, TLS, network policy, GPU scheduling을 추가하세요. |
| Runtime WebSocket service | Live captions, meeting, call-center stream | [Runtime service docs](../runtime/readme.md) | partial result, endpointing, long-lived audio stream이 중요할 때 사용합니다. |
| vLLM acceleration | Fun-ASR-Nano의 LLM-based ASR throughput 향상 | [vLLM guide](./vllm_guide.md) | LLM decoder throughput용입니다. non-autoregressive Paraformer에는 적용되지 않습니다. |
| MCP server | Claude/Cursor/desktop agent speech tool | [MCP example](../examples/mcp_server/) | ASR 결과를 local tool로 Agent에 전달할 때 유용합니다. |
| Subtitle generator | 긴 audio/video에서 SRT/VTT 생성 | [Subtitle example](../examples/subtitle/) | readability가 중요하면 verbose segment와 speaker label을 사용합니다. |
| Batch ASR script | Archive, meeting, dataset, 반복 offline run | [Batch example](../examples/batch_asr_improved.py) | production에서는 queue, manifest, retry log를 추가하세요. |
## 자주 쓰는 선택
### 5분 안에 FunASR을 시험하고 싶다
브라우저만으로 확인하려면 [Colab 빠른 시작](../examples/colab/README_ko.md)을 사용하세요. 로컬에서 작업하려면 README의 Python API부터 시작합니다. 어떤 모델을 고를지 고민된다면 [모델 선택 가이드](./model_selection_ko.md)를 참고하세요.
### Cloud transcription의 local replacement가 필요하다
OpenAI 호환 API를 사용하세요. `/v1/audio/transcriptions`, `/v1/models`, `/health`, Swagger docs를 제공합니다. 먼저 `sensevoice`로 smoke test를 실행하고 기존 SDK나 HTTP client를 [OpenAI API example](../examples/openai_api/README_ko.md)에 맞춰 연결하세요.
### 재현 가능한 container demo가 필요하다
`examples/openai_api/docker-compose.yml`을 CPU mode smoke test로 사용합니다.
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
CUDA를 사용하려면 CUDA-capable PyTorch/FunASR image를 만든 뒤 `FUNASR_DEVICE=cuda`로 바꾸고 같은 smoke test로 확인하세요.
### Streaming 또는 live captioning이 필요하다
Runtime WebSocket service를 사용하세요. production 전에 chunk size, VAD, endpointing, punctuation, speaker diarization, reconnect, client backpressure를 실제 오디오로 검증하세요.
## Readiness checklist
- model alias를 정하고 deployment note에 고정합니다.
- FunASR version, model version, device, CUDA/PyTorch version, Docker image tag, command line을 기록합니다.
- public smoke sample과 realistic private sample을 최소 1개씩 실행합니다.
- request마다 audio duration, model, device, latency, response format, error type을 로깅합니다.
- trusted network 밖으로 API를 노출하기 전에 upload-size limit, authentication, TLS, rate limit을 넣습니다. [Security guide](../examples/openai_api/SECURITY.md)도 확인하세요.
- 막히면 deployment path, command/config, logs, model, device, audio characteristics를 포함해 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)를 열어 주세요.
docs/deployment_matrix_zh.md
+92
View File
@@ -0,0 +1,92 @@
# FunASR 部署选型表
这个页面帮助你为产品、demo、benchmark 或内部工作流选择最短部署路径。先选择能满足目标的最小方案，只有在吞吐、延迟或集成方式有明确要求时，再切换到更重的运行时。
## 快速决策表
| 路径 | 适合场景 | 从这里开始 | 运维提示 |
|---|---|---|---|
| Colab Notebook | 浏览器 smoke test、首次评估、可分享 demo | [Colab 快速体验](../examples/colab/README_zh.md) | 不需要本地环境；首次运行会下载模型，GPU runtime 更快。 |
| Python API | Notebook、离线任务、首次模型评测 | [README 快速开始](../README_zh.md#快速开始) | 最简单；调用方自己负责批处理、重试和文件管理。 |
| OpenAI 兼容 API | 私有语音 API、Agent、Dify/LangChain/AutoGen 风格客户端 | [OpenAI API 示例](../examples/openai_api/README_zh.md) | 已支持 OpenAI audio API 的应用最容易接入。 |
| Xinference | 已经使用 Xinference 统一管理模型服务的团队 | [Xinference 仓库](https://github.com/xorbitsai/inference) | 使用包含 [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140) 的版本或 commit，确保 Fun-ASR-Nano 使用打包发布的 `funasr~=1.3.0`，而不是旧的 git commit pin。 |
| Docker Compose API | 可复现本地 smoke test 或小型内部服务 | [OpenAI API Docker 文档](../examples/openai_api/README_zh.md) | 默认 CPU；容器里使用 CUDA 前需要先适配 CUDA-capable 镜像。 |
| Kubernetes API | 集群内私有语音 API | [Kubernetes 模板](../examples/openai_api/kubernetes/README_zh.md) | 默认私有 `ClusterIP`；对外开放前补齐鉴权、TLS、网络策略和 GPU 调度。 |
| Runtime WebSocket 服务 | 实时字幕、会议、客服流式音频 | [Runtime 服务文档](../runtime/readme_cn.md) | 需要中间结果、断句或长连接音频流时选择。 |
| ONNX/C++ Runtime | 高并发 CPU 服务或嵌入式实时 ASR | [ONNX Runtime 文档](../runtime/onnxruntime/readme.md) | 如果延迟和并发已经验证，不要轻易替换；固定业务词优先做文本后处理。 |
| vLLM 加速 | Fun-ASR-Nano 等 LLM-based ASR 高吞吐 | [vLLM 指南](./vllm_guide.md) | 适合 LLM 解码吞吐；不适用于非自回归 Paraformer。 |
| MCP 服务 | Claude/Cursor/桌面 Agent 语音工具 | [MCP 示例](../examples/mcp_server/) | 适合把 ASR 结果暴露成一个本地工具。 |
| 字幕生成 | 从长音频或视频生成 SRT/VTT | [字幕示例](../examples/subtitle/) | 需要可读性时使用 verbose segments 和说话人标签。 |
| 批处理脚本 | 录音归档、会议纪要、数据集处理 | [批处理示例](../examples/batch_asr_improved.py) | 生产使用时建议增加队列、manifest 和重试日志。 |
| Triton Runtime | 专门的高性能推理服务 | [Triton 文档](../runtime/triton_gpu/README.md) | 配置更重；适合已经在运维 Triton/GPU serving 的团队。 |
## 常见选择
### 我想五分钟内试跑 FunASR
如果只想在浏览器里 smoke test，可以先用 [Colab 快速体验](../examples/colab/README_zh.md)；本地工作再使用 README 里的 Python API。它是验证安装、模型下载、设备选择和基础输出格式的最短路径。如果还不确定先用哪个模型，请看 [模型选择指南](./model_selection_zh.md)。
### 我想替代云端转写服务
使用 OpenAI 兼容 API。它提供 `/v1/audio/transcriptions`、`/v1/models`、`/health` 和 Swagger docs。先用 `sensevoice` 跑通 `examples/openai_api/smoke_test.sh` 或 `examples/openai_api/smoke_test.py`，再根据 [客户端配方](../examples/openai_api/CLIENTS.md) 和 [JavaScript/TypeScript 配方](../examples/openai_api/JAVASCRIPT_zh.md) 接入 SDK 或 HTTP 客户端。浏览器上传或麦克风 demo 可使用 [Gradio 浏览器 Demo](../examples/openai_api/GRADIO_zh.md)。Dify、n8n、HTTP 节点或 webhook worker 可参考 [工作流配方](../examples/openai_api/WORKFLOWS_zh.md)。API 网关、开发者门户或按 schema 导入时可使用 [OpenAPI 规范](../examples/openai_api/OPENAPI_zh.md)。跨团队共享服务前，请先阅读 [安全与网关指南](../examples/openai_api/SECURITY_zh.md)。
### 我已经在使用 Xinference
如果你的系统已经用 Xinference 管理模型注册、virtualenv 隔离和服务生命周期，可以选择 Xinference 路径。请确认使用的 Xinference 版本或 commit 包含 [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140)；该更新把 Fun-ASR-Nano model spec 从旧的 FunASR git SHA 改为打包发布的 `funasr~=1.3.0` 依赖。首次评估 FunASR，或需要面向 Agent 的 OpenAI 兼容转写接口时，仍建议先从上面的 FunASR 原生 OpenAI API 示例开始。
### 我想要可复现的容器 demo
使用 `examples/openai_api/docker-compose.yml` 跑 CPU smoke test：
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
在没有 CUDA-capable PyTorch/FunASR 镜像前保持 CPU 模式。准备好 CUDA 镜像后，再设置 `FUNASR_DEVICE=cuda` 并用同一个 smoke test 验证。没有 bash/curl 时可运行 `python examples/openai_api/smoke_test.py --base-url http://localhost:8000`。
### 我想部署集群内服务
使用 [Kubernetes 模板](../examples/openai_api/kubernetes/README_zh.md) 部署私有 `ClusterIP` OpenAI 兼容 API，包含持久化模型缓存、`/health` probes 和 port-forward smoke test 路径。在没有 CUDA-capable 镜像和集群 GPU 调度前，请保持默认 CPU 模式。
### 我需要流式识别或实时字幕
使用 Runtime WebSocket 服务。上线前请用真实音频验证 chunk size、VAD、断句、标点、说话人分离、重连行为和客户端背压。
### 我已有高并发 ONNX，但需要热词
不要只因为当前 CPU ONNX/C++ 实时链路缺少热词，就把整条已经跑稳的链路迁到 GPU 大模型。先确认产品里的“热词”到底是哪一种：
- 如果是公司名、产品名、专有名词或固定误识别纠错，优先保留 ONNX 链路，在最终文本上做确定性后处理。这样能保留已经验证过的 CPU 并发能力，也更容易审计。
- 如果必须在解码阶段影响识别，或者确实存在多语言/语义理解上的质量缺口，再测试 Fun-ASR-Nano、Qwen3-ASR 等 GPU 路径；但要按首包延迟、尾包延迟、音频切片长度、VAD 策略和同时说话路数重新压测。
- 如果两类需求都有，只在 GPU 模型质量或语言覆盖明显更好的流量上使用 GPU，其他高吞吐稳定流量继续走 ONNX。
开部署 issue 时，请附上当前 ONNX 并发、CPU 核数、目标热词、可接受端到端延迟、GPU 型号、模型启动命令，以及热词是“识别后纠错”还是“解码阶段偏置”。
### 我需要更高的 LLM-based ASR 吞吐
Fun-ASR-Nano 走 vLLM 路径。请用自己的音频分布做 benchmark，并关注 GPU 显存、tensor parallel size、首 token 延迟和 warmup 时间。
### 我想在昇腾 NPU 上跑 Fun-ASR-Nano
Fun-ASR-Nano 的 LLM-based 路径目前主要按 CUDA/vLLM、标准 PyTorch CPU/GPU，以及 CPU/边缘 GGUF runtime 记录和验证；Ascend NPU（`torch_npu`）仍不是这个模型的官方生产运行时。不要因为 SenseVoice 或 Paraformer 能在 NPU 上跑，就默认 Fun-ASR-Nano 也能直接跑通，因为 Nano 还会经过 Qwen 解码器、`inputs_embeds`，以及后端相关的 autocast / 算子路径。
[#3034](https://github.com/modelscope/FunASR/issues/3034) 里的社区复测表明：修复 autocast device 之后，PyTorch `AutoModel(..., device="npu:*")` 路径已经能进入 NPU 后端，并在 310P3、`torch_npu 2.5.1`、CANN 8.5.1 环境里产出正确文本。但该 smoke run 明显慢于 CPU（NPU `rtf_avg` 约 124，CPU 约 1.9），所以这只能作为兼容性证据，不是性能推荐。同一报告里，`AutoModelVLLM` + `vllm_ascend 0.9.2rc1` 仍在 Qwen3 rotary embedding / `TransData` 算子支持处失败；这条路径应按 vLLM-Ascend runtime / 算子兼容问题继续排查，并建议带 `ASCEND_LAUNCH_BLOCKING=1` 复现日志。
如果你正在适配这个后端，请保持第一版范围很小：先从 `torch.bfloat16` 开始，记录 `torch` / `torch_npu` / CANN / 驱动 / NPU 型号，把 PyTorch `AutoModel` 和 `AutoModelVLLM` 分开验证，并在 PR 或 deployment issue 里附上最小命令和完整 stack trace。
## 上线检查清单
- 选择模型 alias，并写入部署说明。
- 记录 FunASR 版本、模型版本、设备、CUDA/PyTorch 版本、Docker 镜像 tag 和启动命令。
- 跑一个公开短音频 smoke sample，再跑至少一个真实私有样本。
- 每次请求记录音频时长、模型、设备、延迟、响应格式和错误类型。
- API 暴露到可信网络外之前，增加上传大小限制、鉴权、TLS 和限流；可用 [安全与网关指南](../examples/openai_api/SECURITY_zh.md) 规划边界。
- 热词或纠错需求要先记录它是确定性后处理，还是解码阶段偏置；替换已有 runtime 前同时 benchmark 质量和延迟。
- 流式场景需要测试静音、噪声、多人重叠、长连接、重连和慢客户端。
- 发布 benchmark 结论时，说明输入时长、硬件、batch size、模型、运行路径，以及是否排除模型下载和 warmup 时间。
## 什么时候开 issue
Runtime、Docker、vLLM、Triton、Android、浏览器或 Agent 集成问题，请使用 [Deployment Help](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)。请附上部署路径、完整命令/config、日志、模型、设备和音频特征。
docs/images/China_Telecom.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

docs/images/DeepScience.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.9 KiB

docs/images/RapidAI.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

docs/images/XVERSE.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

docs/images/aihealthx.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/images/alibaba.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

docs/images/damo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

docs/images/dingding.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

docs/images/funasr_logo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 624 KiB

docs/images/funasr_overview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

docs/images/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 653 KiB

docs/images/nwpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

docs/images/wechat.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 187 KiB

docs/index.rst
+133
View File
@@ -0,0 +1,133 @@
.. Funasr documentation master file, created by
sphinx-quickstart on Tues Dec 6 19:05:00 2022.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
FunASR: A Fundamental End-to-End Speech Recognition Toolkit
============================================================
.. image:: ./images/funasr_logo.jpg
FunASR hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on `ModelScope <https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition>`_, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！
Overview
============================================================
.. image:: ./images/funasr_overview.png
.. toctree::
:maxdepth: 1
:caption: Installation
./installation/installation.md
./installation/docker.md
.. toctree::
:maxdepth: 5
:caption: Quick Start
./funasr/quick_start.md
.. toctree::
:maxdepth: 1
:caption: Academic Egs
./academic_recipe/asr_recipe.md
./academic_recipe/punc_recipe.md
./academic_recipe/vad_recipe.md
./academic_recipe/sv_recipe.md
./academic_recipe/sd_recipe.md
.. toctree::
:maxdepth: 1
:caption: ModelScope Egs
./modelscope_pipeline/quick_start.md
./egs_modelscope/asr/TEMPLATE/README.md
./egs_modelscope/vad/TEMPLATE/README.md
./egs_modelscope/punctuation/TEMPLATE/README.md
./egs_modelscope/tp/TEMPLATE/README.md
./modelscope_pipeline/sv_pipeline.md
./modelscope_pipeline/sd_pipeline.md
./modelscope_pipeline/itn_pipeline.md
.. toctree::
:maxdepth: 1
:caption: Huggingface Egs
Undo
.. toctree::
:maxdepth: 1
:caption: Model Zoo
./model_zoo/modelscope_models.md
./model_zoo/huggingface_models.md
.. toctree::
:maxdepth: 1
:caption: Runtime and Service
./deployment_matrix.md
./deployment_matrix_zh.md
./runtime/readme.md
./runtime/docs/SDK_tutorial_online.md
./runtime/docs/SDK_tutorial.md
./runtime/html5/readme.md
.. toctree::
:maxdepth: 1
:caption: Benchmark and Leaderboard
./benchmark/rtf_reproducibility.md
./benchmark/realtime_ws_benchmark.md
./benchmark/benchmark_onnx.md
./benchmark/benchmark_onnx_cpp.md
./benchmark/benchmark_libtorch.md
./benchmark/benchmark_pipeline_cer.md
.. toctree::
:maxdepth: 1
:caption: Funasr Library
./reference/build_task.md
.. toctree::
:maxdepth: 1
:caption: Papers
./reference/papers.md
.. toctree::
:maxdepth: 1
:caption: Application
./model_selection.md
./model_selection_zh.md
./migration_from_whisper.md
./migration_from_whisper_zh.md
./cli.md
./use_case_showcase.md
./use_case_showcase_zh.md
./community_projects.md
./reference/application.md
.. toctree::
:maxdepth: 1
:caption: FAQ and Troubleshooting
./reference/FQA.md
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
docs/installation/docker.md
+72
View File
@@ -0,0 +1,72 @@
([简体中文](./docker_zh.md)|English)
# Docker
## Install Docker
### Ubuntu
```shell
curl -fsSL https://test.docker.com -o test-docker.sh
sudo sh test-docker.sh
```
### Debian
```shell
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
### CentOS
```shell
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
```
### MacOS
```shell
brew install --cask --appdir=/Applications docker
```
### Windows
Ref to [docs](https://docs.docker.com/desktop/install/windows-install/)
## Start Docker
```shell
sudo systemctl start docker
```
## Download image
### Image Hub
#### CPU
`registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-cpu-0.4.1`
#### GPU
`registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch1.11.0-tf1.15.5-1.8.1`
### Pull Image
```shell
sudo docker pull <image-name>:<tag>
```
### Check Image
```shell
sudo docker images
```
## Run Docker
```shell
# cpu
sudo docker run -itd --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
# gpu
sudo docker run -itd --gpus all --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
sudo docker exec -it funasr /bin/bash
```
## Stop Docker
```shell
exit
sudo docker ps
sudo docker stop funasr
```
docs/installation/docker_zh.md
+72
View File
@@ -0,0 +1,72 @@
(简体中文|[English](./docker.md))
# Docker
## 安装Docker
### Ubuntu
```shell
curl -fsSL https://test.docker.com -o test-docker.sh
sudo sh test-docker.sh
```
### Debian
```shell
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
### CentOS
```shell
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
```
### MacOS
```shell
brew install --cask --appdir=/Applications docker
```
### Windows
请参考[文档](https://docs.docker.com/desktop/install/windows-install/)
## 启动Docker
```shell
sudo systemctl start docker
```
## 下载Docker镜像
### 镜像仓库
#### CPU
`registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-cpu-0.4.1`
#### GPU
`registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch1.11.0-tf1.15.5-1.8.1`
### 拉取镜像
```shell
sudo docker pull <image-name>:<tag>
```
### 查看镜像
```shell
sudo docker images
```
## 运行Docker
```shell
# cpu
sudo docker run -itd --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
# gpu
sudo docker run -itd --gpus all --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
sudo docker exec -it funasr /bin/bash
```
## 停止Docker
```shell
exit
sudo docker ps
sudo docker stop funasr
```
docs/installation/installation.md
Executable
+74
View File
@@ -0,0 +1,74 @@
([简体中文](./installation_zh.md)|English)
<p align="left">
<a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Python->=3.8,<=3.13-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>
</p>
## Installation
### Install Conda (Optional):
#### Linux
```sh
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Mac
```sh
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
# For M1 chip
# wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX*
source ~/.zashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Windows
Ref to [docs](https://docs.conda.io/en/latest/miniconda.html#windows-installers)
### Install Pytorch (version >= 1.11.0):
```sh
pip3 install torch torchaudio
```
If there exists CUDAs in your environments, you should install the pytorch with the version matching the CUDA. The matching list could be found in [docs](https://pytorch.org/get-started/previous-versions/).
### Install funasr
#### Install from pip
```shell
pip3 install -U funasr
# For the users in China, you could install with the command:
# pip3 install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
#### Or install from source code
``` sh
git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip3 install -e ./
# For the users in China, you could install with the command:
# pip3 install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### Install modelscope (Optional)
If you want to use the pretrained models in ModelScope, you should install the modelscope:
```shell
pip3 install -U modelscope
# For the users in China, you could install with the command:
# pip3 install -U modelscope -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### FQA
- For installation on MAC M1 chip, the following error may happen:
- - _cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
```shell
pip uninstall cffi pycparser
ARCHFLAGS="-arch arm64" pip install cffi pycparser --compile --no-cache-dir
```
docs/installation/installation_zh.md
Executable
+75
View File
@@ -0,0 +1,75 @@
(简体中文|[English](./installation.md))
<p align="left">
<a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Python->=3.8,<=3.13-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>
</p>
## 安装
### 安装Conda（可选）：
#### Linux
```sh
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Mac
```sh
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
# For M1 chip
# wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX*
source ~/.zashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Windows
Ref to [docs](https://docs.conda.io/en/latest/miniconda.html#windows-installers)
### 安装Pytorch（版本 >= 1.11.0）：
```sh
pip3 install torch torchaudio
```
如果您的环境中存在CUDAs，则应安装与CUDA匹配版本的pytorch，匹配列表可在文档中找到（[文档](https://pytorch.org/get-started/previous-versions/)）。
### 安装funasr
#### 从pip安装
```shell
pip3 install -U funasr
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
#### 或者从源代码安装
``` sh
git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip3 install -e ./
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### 安装modelscope（可选）
如果您想要使用ModelScope中的预训练模型，则应安装modelscope:
```shell
pip3 install -U modelscope
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -U modelscope -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### 常见问题解答
- 在MAC M1芯片上安装时，可能会出现以下错误：
- - _cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
```shell
pip uninstall cffi pycparser
ARCHFLAGS="-arch arm64" pip install cffi pycparser --compile --no-cache-dir
```
docs/m2met2/Baseline.md
+38
View File
@@ -0,0 +1,38 @@
# Baseline
## Overview
We will release an E2E SA-ASR baseline conducted on [FunASR](https://github.com/modelscope/FunASR) at the time according to the timeline. The model architecture is shown in Figure 3. The SpeakerEncoder is initialized with a pre-trained speaker verification model from ModelScope. This speaker verification model is also be used to extract the speaker embedding in the speaker profile.
![model archietecture](images/sa_asr_arch.png)
## Quick start
To run the baseline, first you need to install FunASR and ModelScope. ([installation](https://github.com/modelscope/FunASR#installation))
There are two startup scripts, `run.sh` for training and evaluating on the old eval and test sets, and `run_m2met_2023_infer.sh` for inference on the new test set of the Multi-Channel Multi-Party Meeting Transcription 2.0 ([M2MeT2.0](https://alibaba-damo-academy.github.io/FunASR/m2met2/index.html)) Challenge.
Before running `run.sh`, you must manually download and unpack the [AliMeeting](http://www.openslr.org/119/) corpus and place it in the `./dataset` directory:
```shell
dataset
|—— Eval_Ali_far
|—— Eval_Ali_near
|—— Test_Ali_far
|—— Test_Ali_near
|—— Train_Ali_far
|—— Train_Ali_near
```
Before running `run_m2met_2023_infer.sh`, you need to place the new test set `Test_2023_Ali_far` (to be released after the challenge starts) in the `./dataset` directory, which contains only raw audios. Then put the given `wav.scp`, `wav_raw.scp`, `segments`, `utt2spk` and `spk2utt` in the `./data/Test_2023_Ali_far` directory.
```shell
data/Test_2023_Ali_far
|—— wav.scp
|—— wav_raw.scp
|—— segments
|—— utt2spk
|—— spk2utt
```
For more details you can see [here](https://github.com/modelscope/FunASR/blob/main/egs/alimeeting/sa-asr/README.md)
## Baseline results
The results of the baseline system are shown in Table 3. The speaker profile adopts the oracle speaker embedding during training. However, due to the lack of oracle speaker label during evaluation, the speaker profile provided by an additional spectral clustering is used. Meanwhile, the results of using the oracle speaker profile on Eval and Test Set are also provided to show the impact of speaker profile accuracy.
| |SI-CER(%) |cpCER(%) |
|:---------------|:------------:|----------:|
|oracle profile |32.72 |42.92 |
|cluster profile|32.73 |49.37 |
docs/m2met2/Challenge_result.md
+14
View File
@@ -0,0 +1,14 @@
# Challenge Result
The following table shows the final results of the competition, where Sub-track1 represents the sub-track under fixed training condition and Sub-track 2 represents the sub-track under the open training condition. All result in this table is cp-CER (%). The rankings in the table are the combined rankings of the two sub-tracks as all teams' submissions met the requirements of the sub-track under fixed training condition.
| Rank &nbsp; &nbsp; | Team Name &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | Sub-track1 &nbsp; &nbsp; | Sub-track2 &nbsp; &nbsp; | paper |
|------|----------------------|------------|------------|------------------------|
| 1 | Ximalaya Speech Team | 11.27 | 11.27 | |
| 2 | 小马达 | 18.64 | 18.64 | |
| 3 | AIzyzx | 22.83 | 22.83 | |
| 4 | AsrSpeeder | / | 23.51 | |
| 5 | zyxlhz | 24.82 | 24.82 | |
| 6 | CMCAI | 26.11 | / | |
| 7 | Volcspeech | 34.21 | 34.21 | |
| 8 | 鉴往知来 | 40.14 | 40.14 | |
| 9 | baseline | 41.55 | 41.55 | |
| 10 | DAICT | 41.64 | | |
docs/m2met2/Contact.md
+4
View File
@@ -0,0 +1,4 @@
# Contact
If you have any questions about M2MeT2.0 challenge, please contact us by
- email: [m2met.alimeeting@gmail.com](mailto:m2met.alimeeting@gmail.com)
docs/m2met2/Dataset.md
+26
View File
@@ -0,0 +1,26 @@
# Datasets
## Overview of training data
In the fixed training condition, the training dataset is restricted to three publicly available corpora, namely, AliMeeting, AISHELL-4, and CN-Celeb. To evaluate the performance of the models trained on these datasets, we will release a new Test set called Test-2023 for scoring and ranking. We will describe the AliMeeting dataset and the Test-2023 set in detail.
## Detail of AliMeeting corpus
AliMeeting contains 118.75 hours of speech data in total. The dataset is divided into 104.75 hours for training (Train), 4 hours for evaluation (Eval) and 10 hours as test set (Test) for scoring and ranking. Specifically, the Train, Eval and Test sets contain 212, 8 and 20 sessions, respectively. Each session consists of a 15 to 30-minute discussion by a group of participants. The total number of participants in Train, Eval and Test sets is 456, 25 and 60, respectively, with balanced gender coverage.
The dataset is collected in 13 meeting venues, which are categorized into three types: small, medium, and large rooms with sizes ranging from 8 m$^{2}$ to 55 m$^{2}$. Different rooms give us a variety of acoustic properties and layouts. The detailed parameters of each meeting venue will be released together with the Train data. The type of wall material of the meeting venues covers cement, glass, etc. Other furnishings in meeting venues include sofa, TV, blackboard, fan, air conditioner, plants, etc. During recording, the participants of the meeting sit around the microphone array which is placed on the table and conduct a natural conversation. The microphone-speaker distance ranges from 0.3 m to 5.0 m. All participants are native Chinese speakers speaking Mandarin without strong accents. During the meeting, various kinds of indoor noise including but not limited to clicking, keyboard, door opening/closing, fan, bubble noise, etc., are made naturally. For both Train and Eval sets, the participants are required to remain in the same position during recording. There is no speaker overlap between the Train and Eval set. An example of the recording venue from the Train set is shown in Fig 1.
![meeting room](images/meeting_room.png)
The number of participants within one meeting session ranges from 2 to 4. To ensure the coverage of different overlap ratios, we select various meeting topics during recording, including medical treatment, education, business, organization management, industrial production and other daily routine meetings. The average speech overlap ratio of Train, Eval and Test sets are 42.27\%, 34.76\% and 42.8\%, respectively. More details of AliMeeting are shown in Table 1. A detailed overlap ratio distribution of meeting sessions with different numbers of speakers in the Train, Eval and Test set is shown in Table 2.
![dataset detail](images/dataset_details.png)
The Test-2023 set consists of 20 sessions that were recorded in an identical acoustic setting to that of the AliMeeting corpus. Each meeting session in the Test-2023 dataset comprises between 2 and 4 participants, thereby sharing a similar configuration with the AliMeeting test set.
We also record the near-field signal of each participant using a headset microphone and ensure that only the participant's own speech is recorded and transcribed. It is worth noting that the far-field audio recorded by the microphone array and the near-field audio recorded by the headset microphone will be synchronized to a common timeline range.
All transcriptions of the speech data are prepared in TextGrid format for each session, which contains the information of the session duration, speaker information (number of speaker, speaker-id, gender, etc.), the total number of segments of each speaker, the timestamp and transcription of each segment, etc.
## Get the data
The three dataset for training mentioned above can be downloaded at [OpenSLR](https://openslr.org/resources.php). The participants can download via the following links. Particularly, in the baseline we provide convenient data preparation scripts for AliMeeting corpus.
- [AliMeeting](https://openslr.org/119/)
- [AISHELL-4](https://openslr.org/111/)
- [CN-Celeb](https://openslr.org/82/)
Now, the new test set is available [here](https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/AliMeeting/openlr/Test_2023_Ali.tar.gz)
docs/m2met2/Introduction.md
+28
View File
@@ -0,0 +1,28 @@
# Introduction
## Call for participation
Automatic speech recognition (ASR) and speaker diarization have made significant strides in recent years, resulting in a surge of speech technology applications across various domains. However, meetings present unique challenges to speech technologies due to their complex acoustic conditions and diverse speaking styles, including overlapping speech, variable numbers of speakers, far-field signals in large conference rooms, and environmental noise and reverberation.
Over the years, several challenges have been organized to advance the development of meeting transcription, including the Rich Transcription evaluation and Computational Hearing in Multisource Environments (CHIME) challenges. The latest iteration of the CHIME challenge has a particular focus on distant automatic speech recognition and developing systems that can generalize across various array topologies and application scenarios. However, while progress has been made in English meeting transcription, language differences remain a significant barrier to achieving comparable results in non-English languages, such as Mandarin. The Multimodal Information Based Speech Processing (MISP) and Multi-Channel Multi-Party Meeting Transcription (M2MeT) challenges have been instrumental in advancing Mandarin meeting transcription. The MISP challenge seeks to address the problem of audio-visual distant multi-microphone signal processing in everyday home environments, while the M2MeT challenge focuses on tackling the speech overlap issue in offline meeting rooms.
The ICASSP2022 M2MeT challenge focuses on meeting scenarios, and it comprises two main tasks: speaker diarization and multi-speaker automatic speech recognition. The former involves identifying who spoke when in the meeting, while the latter aims to transcribe speech from multiple speakers simultaneously, which poses significant technical difficulties due to overlapping speech and acoustic interferences.
Building on the success of the previous M2MeT challenge, we are excited to propose the M2MeT2.0 challenge as an ASRU 2023 challenge special session. In the original M2MeT challenge, the evaluation metric was speaker-independent, which meant that the transcription could be determined, but not the corresponding speaker. To address this limitation and further advance the current multi-talker ASR system towards practicality, the M2MeT2.0 challenge proposes the speaker-attributed ASR task with two sub-tracks: fixed and open training conditions. The speaker-attribute automatic speech recognition (ASR) task aims to tackle the practical and challenging problem of identifying "who spoke what at when". To facilitate reproducible research in this field, we offer a comprehensive overview of the dataset, rules, evaluation metrics, and baseline systems. Furthermore, we will release a carefully curated test set, comprising approximately 10 hours of audio, according to the timeline. The new test set is designed to enable researchers to validate and compare their models' performance and advance the state of the art in this area.
## Timeline(AOE Time)
- $ April~29, 2023: $ Challenge and registration open.
- $ May~11, 2023: $ Baseline release.
- $ May~22, 2023: $ Registration deadline, the due date for participants to join the Challenge.
- $ June~16, 2023: $ Test data release and leaderboard open.
- $ June~20, 2023: $ Final submission deadline and leaderboar close.
- $ June~26, 2023: $ Evaluation result and ranking release.
- $ July~3, 2023: $ Deadline for paper submission.
- $ July~10, 2023: $ Deadline for final paper submission.
- $ December~12\ to\ 16, 2023: $ ASRU Workshop and Challenge Session.
## Guidelines
Interested participants, whether from academia or industry, must register for the challenge by completing the Google form below. The deadline for registration is May 22, 2023.
[M2MeT2.0 Registration](https://docs.google.com/forms/d/e/1FAIpQLSf77T9vAl7Ym-u5g8gXu18SBofoWRaFShBo26Ym0-HDxHW9PQ/viewform?usp=sf_link)
Within three working days, the challenge organizer will send email invitations to eligible teams to participate in the challenge. All qualified teams are required to adhere to the challenge rules, which will be published on the challenge page. Prior to the ranking release time, each participant must submit a system description document detailing their approach and methods. The organizer will select the top ranking submissions to be included in the ASRU2023 Proceedings.
docs/m2met2/Makefile
+20
View File
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/m2met2/Organizers.md
+48
View File
@@ -0,0 +1,48 @@
# Organizers
***Lei Xie, Professor, AISHELL foundation, China***
Email: [lxie@nwpu.edu.cn](mailto:lxie@nwpu.edu.cn)
<img src="images/lxie.jpeg" alt="lxie" width="20%">
***Kong Aik Lee, Senior Scientist at Institute for Infocomm Research, A\*Star, Singapore***
Email: [kongaik.lee@ieee.org](mailto:kongaik.lee@ieee.org)
<img src="images/kong.png" alt="kong" width="20%">
***Zhijie Yan, Principal Engineer at Alibaba, China***
Email: [zhijie.yzj@alibaba-inc.com](mailto:zhijie.yzj@alibaba-inc.com)
<img src="images/zhijie.jpg" alt="zhijie" width="20%">
***Shiliang Zhang, Senior Engineer at Alibaba, China***
Email: [sly.zsl@alibaba-inc.com](mailto:sly.zsl@alibaba-inc.com)
<img src="images/zsl.JPG" alt="zsl" width="20%">
***Yanmin Qian, Professor, Shanghai Jiao Tong University, China***
Email: [yanminqian@sjtu.edu.cn](mailto:yanminqian@sjtu.edu.cn)
<img src="images/qian.jpeg" alt="qian" width="20%">
***Zhuo Chen, Applied Scientist in Microsoft, USA***
Email: [zhuc@microsoft.com](mailto:zhuc@microsoft.com)
<img src="images/chenzhuo.jpg" alt="chenzhuo" width="20%">
***Jian Wu, Applied Scientist in Microsoft, USA***
Email: [wujian@microsoft.com](mailto:wujian@microsoft.com)
<img src="images/wujian.jpg" alt="wujian" width="20%">
***Hui Bu, CEO, AISHELL foundation, China***
Email: [buhui@aishelldata.com](mailto:buhui@aishelldata.com)
<img src="images/buhui.jpeg" alt="buhui" width="20%">
docs/m2met2/Rules.md
+14
View File
@@ -0,0 +1,14 @@
# Rules
All participants should adhere to the following rules to be eligible for the challenge.
- Data augmentation is allowed on the original training dataset, including, but not limited to, adding noise or reverberation, speed perturbation and tone change.
- Participants are permitted to use the Eval set for model training, but it is not allowed to use the Test set for this purpose. Instead, the Test set should only be utilized for parameter tuning and model selection. Any use of the Test-2023 dataset that violates these rules is strictly prohibited, including but not limited to the use of the Test set for fine-tuning or training the model.
- If the cpCER of the two systems on the Test dataset are the same, the system with lower computation complexity will be judged as the superior one.
- If the forced alignment is used to obtain the frame-level classification label, the forced alignment model must be trained on the basis of the data allowed by the corresponding sub-track.
- Shallow fusion is allowed to the end-to-end approaches, e.g., LAS, RNNT and Transformer, but the training data of the shallow fusion language model can only come from the transcripts of the allowed training dataset.
- The right of final interpretation belongs to the organizer. In case of special circumstances, the organizer will coordinate the interpretation.
docs/m2met2/Track_setting_and_evaluation.md
+17
View File
@@ -0,0 +1,17 @@
# Track & Evaluation
## Speaker-Attributed ASR
The speaker-attributed ASR task poses a unique challenge of transcribing speech from multiple speakers and assigning a speaker label to the transcription. Figure 2 illustrates the difference between the speaker-attributed ASR task and the multi-speaker ASR task. This track allows for the use of the AliMeeting, Aishell4, and Cn-Celeb datasets as constrained data sources during both training and evaluation. The AliMeeting dataset, which was used in the M2MeT challenge, includes Train, Eval, and Test sets. Additionally, a new Test-2023 set, consisting of approximately 10 hours of meeting data recorded in an identical acoustic setting as the AliMeeting corpus, will be released soon for challenge scoring and ranking. It's worth noting that the organizers will not provide the near-field audio, transcriptions, or oracle timestamps of the Test-2023 set. Instead, segments containing multiple speakers will be provided, which can be obtained using a simple voice activity detection (VAD) model.
![task difference](images/task_diff.png)
## Evaluation metric
The accuracy of a speaker-attributed ASR system is evaluated using the concatenated minimum permutation character error rate (cpCER) metric. The calculation of cpCER involves three steps. Firstly, the reference and hypothesis transcriptions from each speaker in a session are concatenated in chronological order. Secondly, the character error rate (CER) is calculated between the concatenated reference and hypothesis transcriptions, and this process is repeated for all possible speaker permutations. Finally, the permutation with the lowest CER is selected as the cpCER for that session. TThe CER is obtained by dividing the total number of insertions (Ins), substitutions (Sub), and deletions(Del) of characters required to transform the ASR output into the reference transcript by the total number of characters in the reference transcript. Specifically, CER is calculated by:
$$\text{CER} = \frac {\mathcal N_{\text{Ins}} + \mathcal N_{\text{Sub}} + \mathcal N_{\text{Del}} }{\mathcal N_{\text{Total}}} \times 100\%,$$
where $\mathcal N_{\text{Ins}}$, $\mathcal N_{\text{Sub}}$, $\mathcal N_{\text{Del}}$ are the character number of the three errors, and $\mathcal N_{\text{Total}}$ is the total number of characters.
## Sub-track arrangement
### Sub-track I (Fixed Training Condition):
Participants are required to use only the fixed-constrained data (i.e., AliMeeting, Aishell-4, and CN-Celeb) for system development. The usage of any additional data is strictly prohibited. However, participants can use open-source pre-trained models from third-party sources, such as [Hugging Face](https://huggingface.co/models) and [ModelScope](https://www.modelscope.cn/models), provided that the list of utilized models is clearly stated in the final system description paper.
### Sub-track II (Open Training Condition):
Participants are allowed to use any publicly available data set, privately recorded data, and manual simulation to build their systems in addition to the fixed-constrained data. They can also utilize any open-source pre-trained models, but it is mandatory to provide a clear list of the data and models used in the final system description paper. If manually simulated data is used, a detailed description of the data simulation scheme must be provided.
docs/m2met2/conf.py
+44
View File
@@ -0,0 +1,44 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = "MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0"
copyright = "2023, Speech Lab, Alibaba Group; ASLP Group, Northwestern Polytechnical University"
author = "Speech Lab, Alibaba Group; Audio, Speech and Language Processing Group, Northwestern Polytechnical University"
extensions = [
"nbsphinx",
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.viewcode",
"sphinx.ext.mathjax",
"sphinx.ext.todo",
# "sphinxarg.ext",
"sphinx_markdown_tables",
# 'recommonmark',
"sphinx_rtd_theme",
"myst_parser",
]
myst_enable_extensions = [
"colon_fence",
"deflist",
"dollarmath",
"html_image",
]
myst_heading_anchors = 2
myst_highlight_code_blocks = True
myst_update_mathjax = False
templates_path = ["_templates"]
source_suffix = [".rst", ".md"]
pygments_style = "sphinx"
html_theme = "sphinx_rtd_theme"
docs/m2met2/images/buhui.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/m2met2/images/chenzhuo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 166 KiB

docs/m2met2/images/dataset_details.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 232 KiB

docs/m2met2/images/kong.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

docs/m2met2/images/lxie.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/m2met2/images/meeting_room.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 610 KiB

docs/m2met2/images/qian.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

docs/m2met2/images/sa_asr_arch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 991 KiB

docs/m2met2/images/task_diff.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 748 KiB

docs/m2met2/images/wujian.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

docs/m2met2/images/zhijie.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/m2met2/images/zsl.JPG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 MiB

docs/m2met2/index.rst
+26
View File
@@ -0,0 +1,26 @@
.. m2met2 documentation master file, created by
sphinx-quickstart on Tue Apr 11 14:18:55 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
ASRU 2023 MULTI-CHANNEL MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0 (M2MeT2.0)
==================================================================================
Building on the success of the M2MeT challenge, we are delighted to propose the M2MeT2.0 challenge as a special session at ASRU2023.
To advance the current state-of-the-art in multi-talker automatic speech recognition, the M2MeT2.0 challenge proposes a speaker-attributed ASR task, comprising two sub-tracks: fixed and open training conditions.
To facilitate reproducible research, we provide a comprehensive overview of the dataset, challenge rules, evaluation metrics, and baseline systems.
Now the new test set contains about 10 hours audio is available. You can download from `here <https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/AliMeeting/openlr/Test_2023_Ali.tar.gz>`_
.. toctree::
:maxdepth: 1
:caption: Contents:
./Introduction
./Dataset
./Track_setting_and_evaluation
./Baseline
./Rules
./Challenge_result
./Organizers
./Contact
docs/m2met2/make.bat
+35
View File
@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/make.bat
+35
View File
@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/migration_from_whisper.md
+96
View File
@@ -0,0 +1,96 @@
# Migrate from Whisper or Cloud ASR to FunASR
Use this guide when you already have a Whisper, OpenAI/Cloud ASR, or custom speech pipeline and want to decide whether FunASR is worth switching to. The goal is not to prove a benchmark with one sample file; it is to compare quality, speed, cost, and deployment fit on audio that looks like your real workload.
## When FunASR is a good fit
FunASR is usually worth evaluating when you need one or more of these properties:
- Private or self-hosted transcription where audio should stay inside your environment.
- High-throughput long-form transcription for meetings, archives, media, or call recordings.
- Speaker-aware transcripts with VAD, punctuation, timestamps, and diarization in one pipeline.
- An OpenAI-compatible audio endpoint for agents, Dify, LangChain, AutoGen, or internal apps.
- Streaming ASR or live captions with WebSocket/runtime service support.
- CPU-viable smoke tests before moving to GPU deployment.
Stay on your current pipeline if you need a managed service with no operations work, a vendor SLA, or a language/domain that your own benchmark shows FunASR does not handle well enough yet.
## Fast evaluation plan
1. Pick 20-50 representative audio files. Include short clips, long recordings, noisy samples, different speakers, and the languages or dialects you care about.
2. Run your current Whisper or cloud ASR pipeline exactly as you use it in production. Save transcripts, latency, cost, and failure cases.
3. Run FunASR locally with the README quick start, or use the [migration benchmark example](../examples/migration/) to measure a representative audio folder. Then choose a deployment path from the [deployment matrix](./deployment_matrix.md).
4. Compare output with human review or your normal WER/CER process. Do not compare only one clean demo file.
5. Run the OpenAI-compatible API smoke test if your application already uses OpenAI-style clients.
6. Record warmup time, model download time, device, GPU/CPU type, batch size, and audio duration separately from steady-state throughput.
## Feature mapping
| Existing workflow | FunASR path | What to validate |
|---|---|---|
| Whisper file transcription | [README quick start](../README.md#quick-start) and [model selection guide](./model_selection.md) with SenseVoice, Paraformer, or Fun-ASR-Nano | Transcript quality, timestamps, speed, model download, CPU/GPU behavior. |
| Whisper plus pyannote | `spk_model="cam++"` with VAD and punctuation | Speaker labels, speaker changes, overlapping speech, long silences. |
| OpenAI audio API or cloud batch ASR | [OpenAI-compatible API example](../examples/openai_api/) | `/v1/audio/transcriptions`, response format, client compatibility, upload limits. |
| Dify/LangChain/AutoGen agent audio | [Client recipes](../examples/openai_api/CLIENTS.md) or [MCP server](../examples/mcp_server/) | Tool latency, file handling, auth boundary, error reporting. |
| Live captions or call-center streaming | [Runtime service docs](../runtime/readme.md) | Chunking, endpointing, reconnects, backpressure, partial/final result behavior. |
| Subtitle generation | [Subtitle example](../examples/subtitle/) | Segment readability, line length, speaker labels, SRT/VTT compatibility. |
| Offline archive processing | [Batch ASR example](../examples/batch_asr_improved.py) | Manifest handling, retries, progress logs, throughput, failed-file recovery. |
## Minimal local comparison
Install FunASR and run the same file you used for your baseline:
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # use "cpu" for a portable smoke test
)
result = model.generate(input="sample.wav")
print(result)
```
For an API-style comparison:
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
If you want a repeatable folder-level benchmark, run [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) to produce `results.jsonl` and `summary.md` for your own audio set. For a container smoke test, start from `examples/openai_api/docker-compose.yml` and verify it with `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh`.
## Quality and speed checklist
Track these fields for both the old pipeline and FunASR:
- Audio duration, language, domain, sample rate, channel count, and speaker count.
- Model name, model version, FunASR version, Python/PyTorch/CUDA versions, and Docker image tag if used.
- Hardware, device mode, batch size, streaming chunk size, and whether warmup/model download is excluded.
- WER/CER or human review notes for names, numbers, punctuation, diarization, timestamps, and domain terms.
- Latency, throughput, GPU/CPU memory, cost per hour of audio, and failed-file rate.
- Operational requirements: authentication, upload limits, TLS, logs, monitoring, retries, and retention rules.
## Rollout checklist
- Keep the old pipeline available until FunASR passes your representative benchmark.
- Start with an internal endpoint or batch job before exposing a public API.
- Add request IDs and log audio duration, model, device, latency, and error type for every request.
- Pin the model alias and deployment command in your runbook.
- Test noisy audio, silence, overlapping speakers, long files, non-UTF-8 filenames, and network interruptions.
- Open a [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) with your command, logs, model, device, and sample characteristics if you hit a blocker.
## Share the result
If FunASR replaces or complements your existing ASR stack, consider opening a [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) or [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md). Migration reports with hardware, speed, quality notes, and deployment details help new users choose the right path faster.
docs/migration_from_whisper_zh.md
+96
View File
@@ -0,0 +1,96 @@
# 从 Whisper 或云端 ASR 迁移到 FunASR
当你已经有 Whisper、OpenAI/云端 ASR 或自研语音流水线，并想判断是否值得切到 FunASR 时，可以按这个指南评估。目标不是用一个样例音频证明结论，而是在真实业务音频上比较质量、速度、成本和部署可行性。
## 什么时候值得评估 FunASR
如果你有以下需求，FunASR 通常值得优先测试：
- 音频需要留在私有环境内，不能上传到外部云服务。
- 会议、归档、媒体、客服录音等长音频需要高吞吐转写。
- 希望一条流水线内完成 VAD、标点、时间戳和说话人分离。
- 需要 OpenAI 兼容音频接口，接入 Agent、Dify、LangChain、AutoGen 或内部应用。
- 需要 WebSocket/runtime 服务支撑流式识别或实时字幕。
- 希望先用 CPU 做可复现 smoke test，再迁移到 GPU 服务。
如果你更需要完全托管服务、厂商 SLA，或你的自有评测显示目标语言/领域还不够好，可以暂时保留现有方案。
## 快速评估计划
1. 选择 20-50 条有代表性的音频，覆盖短音频、长录音、噪声、多说话人、目标语言和方言。
2. 按生产方式运行当前 Whisper 或云端 ASR，保存转写结果、延迟、成本和失败样例。
3. 用 README 快速开始跑 FunASR，也可以用 [迁移评测示例](../examples/migration/) 测量一组代表性音频。然后根据 [部署选型表](./deployment_matrix_zh.md) 选择服务路径。
4. 用人工审阅或现有 WER/CER 流程比较结果，不要只看一个干净 demo 音频。
5. 如果应用已经使用 OpenAI 风格客户端，运行 OpenAI 兼容 API smoke test。
6. 分开记录 warmup、模型下载、设备、GPU/CPU 型号、batch size、音频时长和稳定吞吐。
## 功能映射
| 现有流程 | FunASR 路径 | 需要验证什么 |
|---|---|---|
| Whisper 文件转写 | 使用 [README 快速开始](../README_zh.md#快速开始) 和 [模型选择指南](./model_selection_zh.md)，在 SenseVoice、Paraformer 或 Fun-ASR-Nano 中选型 | 转写质量、时间戳、速度、模型下载、CPU/GPU 行为。 |
| Whisper + pyannote | VAD、标点和 `spk_model="cam++"` | 说话人标签、换人位置、重叠说话、长静音。 |
| OpenAI 音频 API 或云端批量 ASR | [OpenAI 兼容 API 示例](../examples/openai_api/README_zh.md) | `/v1/audio/transcriptions`、响应格式、客户端兼容性、上传限制。 |
| Dify/LangChain/AutoGen Agent 音频 | [客户端配方](../examples/openai_api/CLIENTS.md) 或 [MCP 服务](../examples/mcp_server/) | 工具延迟、文件处理、鉴权边界、错误返回。 |
| 实时字幕或客服流式识别 | [Runtime 服务文档](../runtime/readme_cn.md) | 分块、断句、重连、背压、中间/最终结果行为。 |
| 字幕生成 | [字幕示例](../examples/subtitle/) | 分段可读性、行长、说话人标签、SRT/VTT 兼容性。 |
| 离线归档处理 | [批处理示例](../examples/batch_asr_improved.py) | manifest、重试、进度日志、吞吐、失败文件恢复。 |
## 最小本地对比
安装 FunASR，并用你基线评测里的同一条音频运行：
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 便携 smoke test 可改成 "cpu"
)
result = model.generate(input="sample.wav")
print(result)
```
如果要按 API 服务方式对比：
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
如果要对自己的音频目录做可复现评测，可以运行 [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) 生成 `results.jsonl` 和 `summary.md`。如果需要容器 smoke test，可以从 `examples/openai_api/docker-compose.yml` 开始，并用 `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh` 验证。
## 质量与速度检查清单
对旧流水线和 FunASR 都记录这些字段：
- 音频时长、语言、领域、采样率、声道数和说话人数。
- 模型名、模型版本、FunASR 版本、Python/PyTorch/CUDA 版本，以及 Docker 镜像 tag。
- 硬件、设备模式、batch size、流式 chunk size，以及是否排除 warmup/模型下载时间。
- WER/CER 或人工审阅记录：姓名、数字、标点、说话人分离、时间戳、领域词。
- 延迟、吞吐、GPU/CPU 内存、每小时音频成本、失败文件比例。
- 运维要求：鉴权、上传限制、TLS、日志、监控、重试和数据留存规则。
## 上线检查清单
- 在代表性评测通过前，保留旧流水线作为回退。
- 先做内部 endpoint 或离线批处理，再对外暴露 API。
- 为每个请求记录 request id、音频时长、模型、设备、延迟和错误类型。
- 在 runbook 中固定模型别名和部署命令。
- 测试噪声、静音、多人重叠、长文件、非 UTF-8 文件名和网络中断。
- 遇到阻塞时，通过 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) 提交命令、日志、模型、设备和样本特征。
## 分享迁移结果
如果 FunASR 替代或补充了你的现有 ASR 栈，欢迎提交 [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) 或 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)。包含硬件、速度、质量记录和部署细节的迁移报告，能帮助新用户更快选型。
docs/model_selection.md
+99
View File
@@ -0,0 +1,99 @@
# FunASR Model Selection Guide
Use this guide when you are choosing a first model, comparing FunASR with Whisper or a cloud ASR provider, or deciding which model alias to expose through the OpenAI-compatible API.
## Fast default path
If you have a GPU, start with the flagship **Fun-ASR-Nano** — an LLM-based ASR model (SenseVoice encoder + a Qwen3 decoder) covering 31 languages, with the strongest accuracy on hard cases, context, and proper nouns:
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="meeting.wav")
print(result[0]["text"])
```
On CPU, or when you want multilingual + emotion/event tags and speaker-aware meeting transcripts in one fast non-autoregressive pass, use **SenseVoice-Small**:
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # use "cpu" for a portable smoke test
)
result = model.generate(input="meeting.wav")
```
Switch to Paraformer when your workload is Mandarin-only and you want character-level timestamps or hotwords.
## Decision table
| Need | Start with | Why | Next doc |
|---|---|---|---|
| Fast multilingual private transcription | SenseVoice-Small | Strong default with ASR, emotion tags, audio event tags, and CPU viability. | [README quick start](../README.md#quick-start) |
| Mandarin production ASR | Paraformer-Large | Mature Chinese ASR path with VAD and punctuation. | [Tutorial](./tutorial/README.md) |
| English-only route in the OpenAI API example | `paraformer-en` alias | Smaller English route for API compatibility checks. | [OpenAI API example](../examples/openai_api/) |
| LLM-based ASR or 31-language experiments | Fun-ASR-Nano | LLM-based model path; use vLLM when decoder throughput matters. | [vLLM guide](./vllm_guide.md) |
| Live captions or call-center streams | Runtime WebSocket service | Designed for long-lived streaming sessions and partial results. | [Runtime service docs](../runtime/readme.md) |
| Batch archive processing | SenseVoice-Small or Paraformer-Large | Stable offline transcription path; caller owns manifests, retries, and logs. | [Batch ASR example](../examples/batch_asr_improved.py) |
| Migration from Whisper/cloud ASR | SenseVoice-Small first, then benchmark alternatives | Gives a strong baseline before deeper model-specific tuning. | [Migration guide](./migration_from_whisper.md) |
## OpenAI-compatible API aliases
The `examples/openai_api` server exposes short aliases so application teams do not need to know model repository IDs:
| Alias | Underlying path | Use when |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | You want the default private speech API with multilingual ASR, event tags, and good CPU/GPU behavior. |
| `paraformer` | `paraformer-zh` with VAD and punctuation | You want a Mandarin-oriented production route. |
| `paraformer-en` | `paraformer-en` with VAD | You want a compact English route in OpenAI-style clients. |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | You are evaluating LLM-based ASR, 31-language coverage, or vLLM acceleration. |
For Ascend NPU deployments, treat `fun-asr-nano` separately from SenseVoice / Paraformer. The Fun-ASR-Nano PyTorch `AutoModel` path has community compatibility evidence on 310P3 after the NPU autocast fix, but it was much slower than CPU in that smoke test; `AutoModelVLLM` still depends on vLLM-Ascend operator support and has hit Qwen3 rotary / `TransData` failures. Use CUDA/vLLM, standard PyTorch CPU/GPU, or GGUF runtime for production unless you are actively validating an Ascend backend.
Check the live service before wiring clients:
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
For SDK, JavaScript, workflow, Postman, OpenAPI, Docker, and Kubernetes paths, start from the [OpenAI API example](../examples/openai_api/).
## Runtime choice by workload
| Workload | Runtime path | Notes |
|---|---|---|
| Notebook or one-off evaluation | Python `AutoModel` | Shortest path for install, model download, and output-shape checks. |
| Internal HTTP service | OpenAI-compatible API | Reuse OpenAI-style clients, Dify, n8n, LangChain, AutoGen, and HTTP nodes. |
| Repeatable local container demo | Docker Compose API | CPU-first smoke test; adapt the image before using CUDA. |
| Internal cluster service | Kubernetes API template | Private `ClusterIP`, persistent model cache, `/health` probes, and port-forward smoke test. |
| Live audio | Runtime WebSocket service | Validate chunk size, VAD, endpointing, reconnects, and client backpressure with real audio. |
| LLM-based ASR throughput | vLLM path for Fun-ASR-Nano | vLLM accelerates autoregressive decoding; it does not apply to non-autoregressive Paraformer. |
See the [deployment matrix](./deployment_matrix.md) when you are choosing between these paths.
## Benchmark before committing
Do not choose a model from a single clean demo file. Use a small representative set first:
- 20-50 audio files that cover short clips, long meetings, silence, noise, overlapping speakers, domain vocabulary, and target languages.
- Record model name, model revision, FunASR version, device, CPU/GPU type, CUDA/PyTorch version, runtime path, batch size, and whether warmup/model download time is excluded.
- Track quality with your normal WER/CER or human review process, not only transcript readability.
- Track latency, throughput, memory, failures, and upload size limits together.
- Keep at least one public sample for smoke tests and at least one private realistic sample for deployment validation.
For migration work, use the [migration benchmark example](../examples/migration/) and the [migration guide](./migration_from_whisper.md).
## Practical recommendations
- With a GPU, default to Fun-ASR-Nano — the flagship LLM-based model (31 languages), strongest on hard, contextual, and proper-noun-heavy audio.
- On CPU, or for multilingual + emotion workloads, use SenseVoice-Small (fast non-autoregressive, CPU-viable).
- Use Paraformer when your production traffic is primarily Mandarin and you want timestamps or hotwords.
- Use the streaming runtime when partial results and long-lived connections matter more than a single final transcript.
- Keep model aliases stable in production runbooks so benchmark results and bug reports are reproducible.
- Open a [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) with model, device, command, logs, audio duration, and runtime path when you get stuck.
docs/model_selection_ja.md
+62
View File
@@ -0,0 +1,62 @@
# FunASR モデル選択ガイド
初めて FunASR を試すとき、Whisper やクラウド ASR から移行するとき、または OpenAI 互換 API で公開するモデル alias を決めるときに使ってください。
## 迷ったらここから
まずは **SenseVoice-Small** から始めるのがおすすめです。
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 手元の smoke test では "cpu" でも可
)
result = model.generate(input="meeting.wav")
```
デモ、プライベート API、多言語文字起こし、話者付き会議録、Agent 音声入力の最初の選択肢として使いやすいモデルです。中国語本番精度、ストリーミング遅延、LLM-based ASR 評価など明確な要件が出たときだけ切り替えてください。
## 判断表
| やりたいこと | 最初に試すもの | 理由 | 次に読むもの |
|---|---|---|---|
| 高速な多言語プライベート文字起こし | SenseVoice-Small | ASR、感情タグ、音声イベントタグ、CPU/GPU の扱いやすさがそろった標準ルート。 | [README quick start](../README_ja.md#クイックスタート) |
| 中国語中心の本番 ASR | Paraformer-Large | VAD と句読点復元を組み合わせた成熟した中国語 ASR ルート。 | [Tutorial](./tutorial/README.md) |
| OpenAI API 例の英語ルート | `paraformer-en` alias | OpenAI-style client で互換性を確認しやすい軽量な英語ルート。 | [OpenAI API example](../examples/openai_api/README_ja.md) |
| LLM-based ASR や 31 言語の評価 | Fun-ASR-Nano | LLM-based モデル。decoder throughput が重要なら vLLM を使います。 | [vLLM guide](./vllm_guide.md) |
| ライブ字幕やコールセンターストリーム | Runtime WebSocket service | 長時間接続、部分結果、エンドポイント検出に向いたランタイム。 | [Runtime service docs](../runtime/readme.md) |
| Whisper / cloud ASR からの移行 | SenseVoice-Small で baseline を作り、必要に応じて比較 | まず強い標準ルートで評価してから、用途別に詰めるのが安全です。 | [Migration guide](./migration_from_whisper.md) |
## OpenAI 互換 API alias
`examples/openai_api` server は短い alias を提供します。アプリケーション側はモデル repository ID を知らなくても利用できます。
| Alias | 中身 | 使う場面 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 多言語 ASR、イベントタグ、CPU/GPU 両対応の標準プライベート音声 API。 |
| `paraformer` | `paraformer-zh` + VAD + punctuation | 中国語中心の本番ルート。 |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI-style client の英語互換性チェック。 |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | LLM-based ASR、31 言語、vLLM acceleration の評価。 |
接続前にサービスを確認します。
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK、JavaScript、workflow、Postman、OpenAPI、Docker、Kubernetes は [OpenAI API example](../examples/openai_api/README_ja.md) から始めてください。
## ベンチマークしてから決める
きれいな demo 音声 1 つだけでモデルを決めないでください。まず小さな代表セットで確認します。
- 短いクリップ、長い会議、無音、ノイズ、話者重なり、専門用語、対象言語を含む 20-50 ファイルを用意します。
- model name、model revision、FunASR version、device、CPU/GPU、CUDA/PyTorch、runtime path、batch size、download/warmup の扱いを記録します。
- 読みやすさだけでなく、通常使う WER/CER または人手レビューで品質を見ます。
- latency、throughput、memory、failure、upload size limit をまとめて比較します。
- 困ったときは model、device、command、logs、audio duration、runtime path を添えて [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) を開いてください。
docs/model_selection_ko.md
+62
View File
@@ -0,0 +1,62 @@
# FunASR 모델 선택 가이드
처음 FunASR을 사용할 때, Whisper나 클라우드 ASR에서 전환할 때, 또는 OpenAI 호환 API에서 노출할 model alias를 정할 때 참고하세요.
## 고민된다면 여기서 시작
먼저 **SenseVoice-Small**을 추천합니다.
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 간단한 smoke test는 "cpu"도 가능
)
result = model.generate(input="meeting.wav")
```
데모, 프라이빗 API, 다국어 전사, 화자 포함 회의록, Agent 음성 입력의 첫 선택지로 사용하기 좋습니다. 중국어 프로덕션 정확도, 스트리밍 지연 시간, LLM-based ASR 평가처럼 명확한 요구가 있을 때만 다른 경로로 전환하세요.
## 결정 표
| 필요 | 먼저 시도할 것 | 이유 | 다음 문서 |
|---|---|---|---|
| 빠른 다국어 프라이빗 전사 | SenseVoice-Small | ASR, 감정 태그, 음성 이벤트 태그, CPU/GPU 사용성이 균형 잡힌 기본 경로입니다. | [README quick start](../README_ko.md#빠른-시작) |
| 중국어 중심 프로덕션 ASR | Paraformer-Large | VAD와 문장부호 복원을 함께 쓰는 성숙한 중국어 ASR 경로입니다. | [Tutorial](./tutorial/README.md) |
| OpenAI API 예제의 영어 경로 | `paraformer-en` alias | OpenAI-style client에서 호환성을 확인하기 쉬운 가벼운 영어 경로입니다. | [OpenAI API example](../examples/openai_api/README_ko.md) |
| LLM-based ASR 또는 31개 언어 평가 | Fun-ASR-Nano | LLM-based 모델입니다. decoder throughput이 중요하면 vLLM을 사용합니다. | [vLLM guide](./vllm_guide.md) |
| 라이브 자막 또는 콜센터 스트림 | Runtime WebSocket service | 장시간 연결, 부분 결과, endpointing에 맞춘 런타임입니다. | [Runtime service docs](../runtime/readme.md) |
| Whisper / cloud ASR에서 전환 | SenseVoice-Small로 baseline을 만들고 필요하면 비교 | 강한 기본 경로로 먼저 평가한 뒤 용도별로 조정하는 편이 안전합니다. | [Migration guide](./migration_from_whisper.md) |
## OpenAI 호환 API alias
`examples/openai_api` server는 짧은 alias를 제공합니다. 애플리케이션 팀은 모델 repository ID를 몰라도 사용할 수 있습니다.
| Alias | 내부 경로 | 사용 시점 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 다국어 ASR, 이벤트 태그, CPU/GPU 동작이 균형 잡힌 기본 프라이빗 음성 API. |
| `paraformer` | `paraformer-zh` + VAD + punctuation | 중국어 중심 프로덕션 경로. |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI-style client의 영어 호환성 확인. |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | LLM-based ASR, 31개 언어, vLLM acceleration 평가. |
클라이언트를 연결하기 전에 서비스를 확인하세요.
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK, JavaScript, workflow, Postman, OpenAPI, Docker, Kubernetes는 [OpenAI API example](../examples/openai_api/README_ko.md)에서 시작하세요.
## 벤치마크 후 결정하기
깨끗한 demo 오디오 하나만 보고 모델을 정하지 마세요. 먼저 작은 대표 세트로 확인합니다.
- 짧은 클립, 긴 회의, 무음, 잡음, 화자 겹침, 도메인 용어, 대상 언어를 포함하는 20-50개 파일을 준비합니다.
- model name, model revision, FunASR version, device, CPU/GPU, CUDA/PyTorch, runtime path, batch size, download/warmup 처리 여부를 기록합니다.
- 읽기 쉬움만 보지 말고, 평소 사용하는 WER/CER 또는 사람 리뷰로 품질을 확인합니다.
- latency, throughput, memory, failure, upload size limit을 함께 비교합니다.
- 막히면 model, device, command, logs, audio duration, runtime path를 포함해 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)를 열어 주세요.
docs/model_selection_zh.md
+99
View File
@@ -0,0 +1,99 @@
# FunASR 模型选择指南
当你第一次选择模型、评估是否从 Whisper 或云端 ASR 迁移，或者准备通过 OpenAI 兼容 API 暴露模型别名时，可以先看这份指南。
## 默认快速路径
如果有 GPU，先从旗舰 **Fun-ASR-Nano** 开始 —— 基于 LLM 的识别模型（SenseVoice 编码器 + Qwen3 解码），覆盖 31 语种，在难例、上下文和专名上精度最强：
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="meeting.wav")
print(result[0]["text"])
```
在 CPU 上，或当你想要多语种 + 情感/事件标签、带说话人信息的会议转写一次完成时，用 **SenseVoice-Small**：
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 便携 smoke test 可改为 "cpu"
)
result = model.generate(input="meeting.wav")
```
当你的场景是纯中文、需要字级时间戳或热词时，切换到 Paraformer。
## 决策表
| 需求 | 优先尝试 | 原因 | 下一步文档 |
|---|---|---|---|
| 快速多语种私有转写 | SenseVoice-Small | 兼顾 ASR、情感标签、音频事件标签和 CPU 可用性。 | [README 快速开始](../README_zh.md#快速开始) |
| 中文生产 ASR | Paraformer-Large | 成熟中文 ASR 路径，可组合 VAD 和标点。 | [教程](./tutorial/README_zh.md) |
| OpenAI API 示例中的英文路由 | `paraformer-en` alias | 适合在 OpenAI 风格客户端里验证较轻量英文路径。 | [OpenAI API 示例](../examples/openai_api/README_zh.md) |
| LLM-based ASR 或 31 语种实验 | Fun-ASR-Nano | LLM-based 模型路径；解码吞吐敏感时配合 vLLM。 | [vLLM 指南](./vllm_guide.md) |
| 实时字幕或客服流式音频 | Runtime WebSocket 服务 | 面向长连接流式会话和中间结果。 | [Runtime 服务文档](../runtime/readme_cn.md) |
| 录音归档批处理 | SenseVoice-Small 或 Paraformer-Large | 稳定离线转写路径；调用方负责 manifest、重试和日志。 | [批处理示例](../examples/batch_asr_improved.py) |
| 从 Whisper/云端 ASR 迁移 | 先用 SenseVoice-Small，再 benchmark 其他模型 | 先建立强基线，再做模型专项调优。 | [迁移指南](./migration_from_whisper_zh.md) |
## OpenAI 兼容 API 别名
`examples/openai_api` 服务提供短别名，应用团队不需要了解具体模型仓库 ID：
| Alias | 底层路径 | 适合场景 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 默认私有语音 API，多语种 ASR、事件标签和 CPU/GPU 行为较均衡。 |
| `paraformer` | `paraformer-zh` + VAD + 标点 | 中文生产流量优先尝试。 |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI 风格客户端里的英文轻量路由。 |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | 评估 LLM-based ASR、31 语种覆盖或 vLLM 加速。 |
如果部署目标是昇腾 NPU，请把 `fun-asr-nano` 和 SenseVoice / Paraformer 分开看。Fun-ASR-Nano 的 PyTorch `AutoModel` 路径在修复 NPU autocast 后已有 310P3 社区兼容性 smoke 结果，但该测试明显慢于 CPU；`AutoModelVLLM` 仍依赖 vLLM-Ascend 算子支持，并已遇到 Qwen3 rotary / `TransData` 失败。生产部署优先使用 CUDA/vLLM、标准 PyTorch CPU/GPU 或 GGUF runtime，除非你正在主动验证 Ascend 后端。
接入客户端前先检查在线服务：
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK、JavaScript、工作流、Postman、OpenAPI、Docker 和 Kubernetes 路径可从 [OpenAI API 示例](../examples/openai_api/README_zh.md) 开始。
## 按工作负载选择运行路径
| 工作负载 | 运行路径 | 说明 |
|---|---|---|
| Notebook 或一次性评估 | Python `AutoModel` | 验证安装、模型下载和输出结构的最短路径。 |
| 内部 HTTP 服务 | OpenAI 兼容 API | 复用 OpenAI 风格客户端、Dify、n8n、LangChain、AutoGen 和 HTTP 节点。 |
| 可复现本地容器 demo | Docker Compose API | CPU-first smoke test；使用 CUDA 前先适配镜像。 |
| 集群内私有服务 | Kubernetes API 模板 | 私有 `ClusterIP`、持久化模型缓存、`/health` probes 和 port-forward smoke test。 |
| 实时音频 | Runtime WebSocket 服务 | 用真实音频验证 chunk size、VAD、断句、重连和客户端背压。 |
| LLM-based ASR 吞吐 | Fun-ASR-Nano 的 vLLM 路径 | vLLM 加速自回归解码；不适用于非自回归 Paraformer。 |
选择部署方式时可以参考 [部署选型表](./deployment_matrix_zh.md)。
## 上线前先 benchmark
不要只用一个干净 demo 文件选型。先准备一个小而有代表性的集合：
- 20-50 条音频，覆盖短音频、长会议、静音、噪声、多人重叠、领域词汇和目标语言。
- 记录模型名、模型版本、FunASR 版本、设备、CPU/GPU 型号、CUDA/PyTorch 版本、运行路径、batch size，以及是否排除 warmup/模型下载时间。
- 使用你已有的 WER/CER 流程或人工审阅，不要只看转写文本是否“读起来还行”。
- 同时记录延迟、吞吐、内存、失败样例和上传大小限制。
- 保留至少一个公开样例用于 smoke test，也保留至少一个真实私有样本用于部署验证。
迁移场景可以使用 [迁移评测示例](../examples/migration/) 和 [迁移指南](./migration_from_whisper_zh.md)。
## 实用建议
- demo、私有 API、Agent 语音输入和多语种场景优先试 SenseVoice-Small。
- 中文生产流量优先试 Paraformer，尤其是希望走成熟非自回归 ASR 路径时。
- 明确需要 LLM-based 模型路径或 vLLM 加速实验时，再试 Fun-ASR-Nano。
- 需要中间结果和长连接时，优先使用 streaming runtime，而不是普通 HTTP 转写接口。
- 生产 runbook 中固定模型 alias，保证 benchmark 和问题复现可追踪。
- 遇到阻塞时，用 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) 提供模型、设备、命令、日志、音频时长和运行路径。
docs/reference/FQA.md
+236
View File
@@ -0,0 +1,236 @@
# FAQ and Troubleshooting
This page collects the questions that most often block new FunASR users. Start here before opening an issue.
## Which install command should I use?
For most users:
```bash
pip install -U funasr
```
For the newest examples, server CLI, or unreleased fixes:
```bash
git clone https://github.com/modelscope/FunASR.git
cd FunASR
pip install -e ./
```
For the OpenAI-compatible API server, install the web runtime dependencies too:
```bash
pip install funasr fastapi uvicorn python-multipart
```
## Which Python and PyTorch versions are recommended?
Use Python 3.8 or later and install a PyTorch/torchaudio pair that matches your CUDA runtime. If CUDA is not configured, start with CPU to verify the workflow first:
```bash
funasr-server --model sensevoice --device cpu
```
After the CPU smoke test works, switch to CUDA:
```bash
funasr-server --model sensevoice --device cuda
```
## Model download is slow or fails. What should I check?
FunASR models are available from ModelScope and Hugging Face. Choose the hub that is fastest in your network environment, and make sure the machine can reach it before debugging model code.
Common checks:
```bash
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
python -c "import funasr; print(funasr.__version__)"
```
If a model is already downloaded on another machine, use the local model path instead of the remote model name.
## `funasr-server` says FastAPI or multipart packages are missing
Install the server dependencies:
```bash
pip install fastapi uvicorn python-multipart
```
Then start again:
```bash
funasr-server --model sensevoice --device cuda
```
## Port 8000 is already in use
Start the service on another port:
```bash
funasr-server --model sensevoice --device cuda --port 9000
```
Then point clients to the new base URL:
```bash
curl http://localhost:9000/health
```
## How do I verify the OpenAI-compatible API quickly?
Start the server:
```bash
funasr-server --model sensevoice --device cuda
```
In another terminal:
```bash
curl -L https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav -o sample.wav
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
The response should include `text`. With `verbose_json`, supported models may also return segment-level information.
## How do I run the OpenAI-compatible API with Docker Compose?
Use the example Compose setup when you want a reproducible local smoke test before wiring the API into a product or agent workflow:
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
Then verify it from another terminal:
```bash
BASE_URL=http://localhost:8000 bash smoke_test.sh
```
The example container defaults to `FUNASR_DEVICE=cpu` so it can start on machines without NVIDIA Container Toolkit. For CUDA, first adapt the image to use CUDA-capable PyTorch/FunASR dependencies, then set `FUNASR_DEVICE=cuda`.
See also:
- [OpenAI API example](../../examples/openai_api/)
- [Client recipes](../../examples/openai_api/CLIENTS.md)
- [Deployment matrix](../deployment_matrix.md)
## Docker starts but `/health` or transcription fails
Check the container logs first:
```bash
docker compose logs -f funasr-api
```
Common causes:
- The first startup is still downloading or loading the model.
- Port 8000 is already in use; set `FUNASR_HOST_PORT=9000` in `.env` and use `BASE_URL=http://localhost:9000`.
- The container is running in CPU mode but `.env` or the command expects CUDA.
- CUDA is requested but the image does not include CUDA-capable PyTorch/FunASR dependencies.
- The model cache volume is empty or corrupted; retry after removing the `funasr-cache` Docker volume.
- The uploaded audio file is too large for the machine; verify with the public sample before testing long recordings.
When opening a Deployment Help issue, include your `.env` values without secrets, the `docker compose` command, container logs, model alias, device, and audio duration.
## Long audio is slow, split incorrectly, or runs out of memory
Use VAD segmentation for long audio and tune segment length for your hardware:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
vad_kwargs={"max_single_segment_time": 30000},
device="cuda",
)
result = model.generate(input="long_meeting.wav", batch_size_s=300)
```
If memory is limited, reduce `batch_size_s`, use CPU for verification, or split very long recordings before batch processing.
## Speaker diarization has no speaker labels
Use a model pipeline that includes both VAD and speaker models:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
device="cuda",
)
result = model.generate(input="meeting.wav")
```
Then inspect `result[0]["sentence_info"]`. Each sentence should include fields such as `text`, `start`, `end`, and `spk` when diarization is available.
## Can I use a speaker model other than cam++?
Yes. `spk_model` can be any FunASR `AutoModel` speaker embedding model, including a ModelScope model ID or a local model path. For example, ERes2NetV2 can be used directly:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="iic/speech_eres2netv2_sv_zh-cn_16k-common",
device="cuda",
)
result = model.generate(input="meeting.wav")
```
For a fully custom speaker model, make sure the model can be loaded by FunASR `AutoModel` and that its `inference()` method returns `spk_embedding`; the user-facing call still goes through `AutoModel.generate()`. The diarization post-processing and clustering steps use those embeddings to assign speaker labels. If your diarization model emits speaker segments directly instead of embeddings, add an adapter layer that converts its output to the current speaker-label structure.
The tutorial has a longer ERes2NetV2 example: [Speaker Verification / Diarization](../tutorial/README.md#speaker-verification--diarization-eres2netv2).
## The same command works on CPU but fails on CUDA
This usually points to a CUDA, driver, PyTorch, or GPU memory mismatch. Include these checks in your issue:
```bash
nvidia-smi
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
python -c "import torchaudio; print(torchaudio.__version__)"
```
Try a smaller model or lower batch size to rule out GPU memory pressure.
## What information should I include in an issue?
Please include:
- OS and Python version
- FunASR version and install method (`pip`, source, Docker)
- PyTorch, torchaudio, CUDA, and GPU information
- Exact command or minimal Python snippet
- Full traceback or server logs
- Model name and hub (`modelscope`, `hf`, or local path)
- Audio duration, sample rate, format, language, speaker count, and whether the audio can be shared
## Existing ModelScope pipeline examples
- [VAD model with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/236)
- [Punctuation model with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/238)
- [Paraformer streaming with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/241)
- [VAD + ASR + punctuation with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/278)
- [VAD + ASR + punctuation + NNLM with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/134)
- [Timestamp prediction with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/246)
- [Switch online/offline decoding for UniASR](https://github.com/modelscope/FunASR/discussions/151)
docs/reference/application.md
+5
View File
@@ -0,0 +1,5 @@
## Audio Cut
## Realtime Speech Recognition
## Audio Chat
docs/reference/build_task.md
+125
View File
@@ -0,0 +1,125 @@
# Build custom tasks
FunASR is similar to ESPNet, which applies `Task` as the general interface ti achieve the training and inference of models. Each `Task` is a class inherited from `AbsTask` and its corresponding code can be seen in `funasr/tasks/abs_task.py`. The main functions of `AbsTask` are shown as follows:
```python
class AbsTask(ABC):
@classmethod
def add_task_arguments(cls, parser: argparse.ArgumentParser):
pass
@classmethod
def build_preprocess_fn(cls, args, train):
(...)
@classmethod
def build_collate_fn(cls, args: argparse.Namespace):
(...)
@classmethod
def build_model(cls, args):
(...)
@classmethod
def main(cls, args):
(...)
```
- add_task_arguments：Add parameters required by a specified `Task`
- build_preprocess_fn：定义如何处理对样本进行预处理 define how to preprocess samples
- build_collate_fn：define how to combine multiple samples into a `batch`
- build_model：define the model
- main：training interface, starting training through `Task.main()`
Next, we take the speech recognition as an example to introduce how to define a new `Task`. For the corresponding code, please see `ASRTask` in `funasr/tasks/asr.py`. The procedure of defining a new `Task` is actually the procedure of redefining the above functions according to the requirements of the specified `Task`.
- add_task_arguments
```python
@classmethod
def add_task_arguments(cls, parser: argparse.ArgumentParser):
group = parser.add_argument_group(description="Task related")
group.add_argument(
"--token_list",
type=str_or_none,
default=None,
help="A text mapping int-id to token",
)
(...)
```
For speech recognition tasks, specific parameters required include `token_list`, etc. According to the specific requirements of different tasks, users can define corresponding parameters in this function.
- build_preprocess_fn
```python
@classmethod
def build_preprocess_fn(cls, args, train):
if args.use_preprocessor:
retval = CommonPreprocessor(
train=train,
token_type=args.token_type,
token_list=args.token_list,
bpemodel=args.bpemodel,
non_linguistic_symbols=args.non_linguistic_symbols,
text_cleaner=args.cleaner,
...
)
else:
retval = None
return retval
```
This function defines how to preprocess samples. Specifically, the input of speech recognition tasks includes speech and text. For speech, functions such as (optional) adding noise and reverberation to the speech are supported. For text, functions such as (optional) processing text according to bpe and mapping text to `tokenid` are supported. Users can choose the preprocessing operation that needs to be performed on the sample. For the detail implementation, please refer to `CommonPreprocessor`.
- build_collate_fn
```python
@classmethod
def build_collate_fn(cls, args, train):
return CommonCollateFn(float_pad_value=0.0, int_pad_value=-1)
```
This function defines how to combine multiple samples into a `batch`. For speech recognition tasks, `padding` is employed to obtain equal-length data from different speech and text. Specifically, we set `0.0` as the default padding value for speech and `-1` as the default padding value for text. Users can define different `batch` operations here. For the detail implementation, please refer to `CommonCollateFn`.
- build_model
```python
@classmethod
def build_model(cls, args, train):
with open(args.token_list, encoding="utf-8") as f:
token_list = [line.rstrip() for line in f]
vocab_size = len(token_list)
frontend = frontend_class(**args.frontend_conf)
specaug = specaug_class(**args.specaug_conf)
normalize = normalize_class(**args.normalize_conf)
preencoder = preencoder_class(**args.preencoder_conf)
encoder = encoder_class(input_size=input_size, **args.encoder_conf)
postencoder = postencoder_class(input_size=encoder_output_size, **args.postencoder_conf)
decoder = decoder_class(vocab_size=vocab_size, encoder_output_size=encoder_output_size, **args.decoder_conf)
ctc = CTC(odim=vocab_size, encoder_output_size=encoder_output_size, **args.ctc_conf)
model = model_class(
vocab_size=vocab_size,
frontend=frontend,
specaug=specaug,
normalize=normalize,
preencoder=preencoder,
encoder=encoder,
postencoder=postencoder,
decoder=decoder,
ctc=ctc,
token_list=token_list,
**args.model_conf,
)
return model
```
This function defines the detail of the model. For different speech recognition models, the same speech recognition `Task` can usually be shared and the remaining thing needed to be done is to define a specific model in this function. For example, a speech recognition model with a standard encoder-decoder structure has been shown above. Specifically, it first defines each module of the model, including encoder, decoder, etc. and then combine these modules together to generate a complete model. In FunASR, the model needs to inherit `FunASRModel` and the corresponding code can be seen in `funasr/train/abs_espnet_model.py`. The main function needed to be implemented is the `forward` function.
Next, we take `SANMEncoder` as an example to introduce how to use a custom encoder as a part of the model when defining the specified model and the corresponding code can be seen in `funasr/models/encoder/sanm_encoder.py`. For a custom encoder, in addition to inheriting the common encoder class `AbsEncoder`, it is also necessary to define the `forward` function to achieve the forward computation of the `encoder`. After defining the `encoder`, it should also be registered in the `Task`. The corresponding code example can be seen as below:
```python
encoder_choices = ClassChoices(
"encoder",
classes=dict(
conformer=ConformerEncoder,
transformer=TransformerEncoder,
rnn=RNNEncoder,
sanm=SANMEncoder,
sanm_chunk_opt=SANMEncoderChunkOpt,
data2vec_encoder=Data2VecEncoder,
mfcca_enc=MFCCAEncoder,
),
type_check=AbsEncoder,
default="rnn",
)
```
In this code, `sanm=SANMEncoder` takes the newly defined `SANMEncoder` as an optional choice of the `encoder`. Once the user specifies the `encoder` as `sanm` in the configuration file, the `SANMEncoder` will be correspondingly employed as the `encoder` module of the model.
docs/reference/papers.md
+38
View File
@@ -0,0 +1,38 @@
# Papers
FunASR have implemented the following paper code
### Speech Recognition
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://arxiv.org/abs/2305.11013), INTERSPEECH 2023
- [BAT: Boundary aware transducer for memory-efficient and low-latency ASR](https://arxiv.org/abs/2305.11571), INTERSPEECH 2023
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://arxiv.org/abs/2206.08317), INTERSPEECH 2022
- [E-branchformer: Branchformer with enhanced merging for speech recognition](https://arxiv.org/abs/2210.00077), SLT 2022
- [Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding](https://proceedings.mlr.press/v162/peng22a.html?ref=https://githubhelp.com), ICML 2022
- [Universal ASR: Unifying Streaming and Non-Streaming ASR Using a Single Encoder-Decoder Model](https://arxiv.org/abs/2010.14099), arXiv preprint arXiv:2010.14099, 2020
- [San-m: Memory equipped self-attention for end-to-end speech recognition](https://arxiv.org/pdf/2006.01713), INTERSPEECH 2020
- [Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition](https://arxiv.org/abs/2006.01712), INTERSPEECH 2020
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://arxiv.org/abs/2005.08100), INTERSPEECH 2020
- [Sequence-to-sequence learning with Transducers](https://arxiv.org/pdf/1211.3711.pdf), NIPS 2016
### Multi-talker Speech Recognition
- [MFCCA:Multi-Frame Cross-Channel attention for multi-speaker ASR in Multi-party meeting scenario](https://arxiv.org/abs/2210.05265), ICASSP 2022
### Voice Activity Detection
- [Deep-FSMN for Large Vocabulary Continuous Speech Recognition](https://arxiv.org/abs/1803.05030), ICASSP 2018
### Punctuation Restoration
- [CT-Transformer: Controllable time-delay transformer for real-time punctuation prediction and disfluency detection](https://arxiv.org/pdf/2003.01309.pdf), ICASSP 2018
### Language Models
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762), NEURIPS 2017
### Speaker Verification
- [X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION](https://www.danielpovey.com/files/2018_icassp_xvectors.pdf), ICASSP 2018
### Speaker diarization
- [Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis](https://arxiv.org/abs/2211.10243), EMNLP 2022
- [TOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization](https://arxiv.org/abs/2303.05397), ICASSP 2023
### Timestamp Prediction
- [Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model](https://arxiv.org/abs/2301.12343), arXiv:2301.12343
docs/tutorial/README.md
+627
View File
@@ -0,0 +1,627 @@
([简体中文](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README_zh.md)|English)
FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE). Below, we list some representative models. For a comprehensive list, please refer to our [Model Zoo](https://github.com/modelscope/FunASR/tree/main/model_zoo).
<div align="center">
<h4>
<a href="#Inference"> Model Inference </a>
｜<a href="#Training"> Model Training and Testing </a>
｜<a href="#Export"> Model Export and Testing </a>
｜<a href="#new-model-registration-tutorial"> New Model Registration Tutorial </a>
</h4>
</div>
<a name="Inference"></a>
## Model Inference
### Quick Start
No local setup? Run the [Colab quickstart](../../examples/colab/) first, then move to the command line when you are ready.
For command-line invocation:
```shell
funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++input=asr_example_zh.wav
```
For python code invocation (recommended):
```python
from funasr import AutoModel
model = AutoModel(model="paraformer-zh")
res = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav")
print(res)
```
### API Description
#### AutoModel Definition
```python
model = AutoModel(model=[str], device=[str], ncpu=[int], output_dir=[str], batch_size=[int], hub=[str], **kwargs)
```
- `model`(str): model name in the [Model Repository](https://github.com/modelscope/FunASR/tree/main/model_zoo), or a model path on local disk.
- `device`(str): `cuda:0` (default gpu0) for using GPU for inference, specify `cpu` for using CPU. `mps`: Mac computers with M-series chips use MPS for inference. `xpu`: Uses Intel GPU for inference.
- `ncpu`(int): `4` (default), sets the number of threads for CPU internal operations.
- `output_dir`(str): `None` (default), set this to specify the output path for the results.
- `batch_size`(int): `1` (default), the number of samples per batch during decoding.
- `hub`(str)：`ms` (default) to download models from ModelScope. Use `hf` to download models from Hugging Face.
- `**kwargs`(dict): Any parameters found in config.yaml can be directly specified here, for instance, the maximum segmentation length in the vad model max_single_segment_time=6000 (milliseconds).
#### AutoModel Inference
```python
res = model.generate(input=[str], output_dir=[str])
```
- `input`: The input to be decoded, which could be:
- A wav file path, e.g., asr_example.wav
- A pcm file path, e.g., asr_example.pcm, in this case, specify the audio sampling rate fs (default is 16000)
- An audio byte stream, e.g., byte data from a microphone
- A wav.scp, a Kaldi-style wav list (wav_id \t wav_path), for example:
```text
asr_example1 ./audios/asr_example1.wav
asr_example2 ./audios/asr_example2.wav
```
When using wav.scp as input, you must set output_dir to save the output results.
- Audio samples, `e.g.`: `audio, rate = soundfile.read("asr_example_zh.wav")`, data type is numpy.ndarray. Supports batch inputs, type is list：
```[audio_sample1, audio_sample2, ..., audio_sampleN]```
- fbank input, supports batch grouping. Shape is [batch, frames, dim], type is torch.Tensor.
- `output_dir`: None (default), if set, specifies the output path for the results.
- `**kwargs`(dict): Inference parameters related to the model, for example,`beam_size=10`，`decoding_ctc_weight=0.1`.
### More Usage Introduction
#### Speech Recognition (Non-streaming)
```python
from funasr import AutoModel
# paraformer-zh is a multi-functional asr model
# use vad, punc, spk or not as you need
model = AutoModel(model="paraformer-zh",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 60000},
punc_model="ct-punc",
# spk_model="cam++"
)
wav_file = f"{model.model_path}/example/asr_example.wav"
res = model.generate(input=wav_file, batch_size_s=300, batch_size_threshold_s=60, hotword='魔搭')
print(res)
```
#### Postprocess hotword correction
Model-level `hotword` / `hotwords` boosts a small set of terms during decoding. For large vocabularies (for example thousands of stock names), apply text-level correction after recognition:
```python
from funasr import AutoModel
model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc")
res = model.generate(
input="asr_example.wav",
postprocess_hotwords={
"科大迅飞": "科大讯飞",
"东方财富": "东方财富",
},
postprocess_hotword_threshold=0.85,
return_postprocess_hotword_matches=True,
)
print(res[0]["text"])
print(res[0].get("postprocess_hotword_matches"))
```
You can also pass `postprocess_hotword_file` with one target word per line, or an explicit mapping such as `wrong=>right`. Fuzzy matching requires optional `pypinyin` and `rapidfuzz`; explicit mappings work without them.
Notes:
- Typically, the input duration for models is limited to under 30 seconds. However, when combined with `vad_model`, support for audio input of any length is enabled, not limited to the paraformer model—any audio input model can be used.
- Parameters related to model can be directly specified in the definition of AutoModel; parameters related to `vad_model` can be set through `vad_kwargs`, which is a dict; similar parameters include `punc_kwargs` and `spk_kwargs`.
- `max_single_segment_time`: Denotes the maximum audio segmentation length for `vad_model`, measured in milliseconds (ms).
- `batch_size_s` represents the use of dynamic batching, where the total audio duration within a batch is measured in seconds (s).
- `batch_size_threshold_s`: Indicates that when the duration of an audio segment post-VAD segmentation exceeds the batch_size_threshold_s threshold, the batch size is set to 1, measured in seconds (s).
Recommendations:
When you input long audio and encounter Out Of Memory (OOM) issues, since memory usage tends to increase quadratically with audio length, consider the following three scenarios:
a) At the beginning of inference, memory usage primarily depends on `batch_size_s`. Appropriately reducing this value can decrease memory usage.
b) During the middle of inference, when encountering long audio segments cut by VAD and the total token count is less than `batch_size_s`, yet still facing OOM, you can appropriately reduce `batch_size_threshold_s`. If the threshold is exceeded, the batch size is forced to 1.
c) Towards the end of inference, if long audio segments cut by VAD have a total token count less than `batch_size_s` and exceed the `threshold` batch_size_threshold_s, forcing the batch size to 1 and still facing OOM, you may reduce `max_single_segment_time` to shorten the VAD audio segment length.
#### Speech Recognition (Fun-ASR-Nano)
```python
from funasr import AutoModel
model = AutoModel(
model="FunAudioLLM/Fun-ASR-Nano-2512",
trust_remote_code=True,
remote_code="./model.py",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 30000},
device="cuda:0",
hub="hf",
)
res = model.generate(input="audio.wav", cache={}, batch_size=1, language="中文")
print(res[0]["text"])
print(res[0]["timestamps"]) # character-level timestamps
```
Notes:
- Fun-ASR-Nano is trained on tens of millions of hours of data, supporting 31 languages including Chinese dialects.
- Supports hotwords: `hotwords=["keyword1", "keyword2"]`
- Supports speaker diarization: add `spk_model="cam++"` and `punc_model="ct-punc"` to get `sentence_info` with speaker labels.
- Requires: `pip install tiktoken huggingface_hub`
#### Speech Recognition (Streaming)
```python
from funasr import AutoModel
chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention
decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention
model = AutoModel(model="paraformer-zh-streaming")
import soundfile
import os
wav_file = os.path.join(model.model_path, "example/asr_example.wav")
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = chunk_size[1] * 960 # 600ms
cache = {}
total_chunk_num = int((len(speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
is_final = i == total_chunk_num - 1
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
print(res)
```
Note: `chunk_size` is the configuration for streaming latency.` [0,10,5]` indicates that the real-time display granularity is `10*60=600ms`, and the lookahead information is `5*60=300ms`. Each inference input is `600ms` (sample points are `16000*0.6=960`), and the output is the corresponding text. For the last speech segment input, `is_final=True` needs to be set to force the output of the last word.
#### Voice Activity Detection (Non-Streaming)
```python
from funasr import AutoModel
model = AutoModel(model="fsmn-vad")
wav_file = f"{model.model_path}/example/vad_example.wav"
res = model.generate(input=wav_file)
print(res)
```
Note: The output format of the VAD model is: `[[beg1, end1], [beg2, end2], ..., [begN, endN]]`, where `begN/endN` indicates the starting/ending point of the `N-th` valid audio segment, measured in milliseconds.
#### Voice Activity Detection (Streaming)
```python
from funasr import AutoModel
chunk_size = 200 # ms
model = AutoModel(model="fsmn-vad")
import soundfile
wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = int(chunk_size * sample_rate / 1000)
cache = {}
total_chunk_num = int((len(speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
is_final = i == total_chunk_num - 1
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size)
if len(res[0]["value"]):
print(res)
```
Note: The output format for the streaming VAD model can be one of four scenarios:
- `[[beg1, end1], [beg2, end2], .., [begN, endN]]`：The same as the offline VAD output result mentioned above.
- `[[beg, -1]]`：Indicates that only a starting point has been detected.
- `[[-1, end]]`：Indicates that only an ending point has been detected.
- `[]`：Indicates that neither a starting point nor an ending point has been detected.
The output is measured in milliseconds and represents the absolute time from the starting point.
#### Punctuation Restoration
```python
from funasr import AutoModel
model = AutoModel(model="ct-punc")
res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
print(res)
```
#### Timestamp Prediction
```python
from funasr import AutoModel
model = AutoModel(model="fa-zh")
wav_file = f"{model.model_path}/example/asr_example.wav"
text_file = f"{model.model_path}/example/text.txt"
res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
print(res)
```
More examples ref to [docs](https://github.com/modelscope/FunASR/tree/main/examples/industrial_data_pretraining)
#### Speaker Verification / Diarization (ERes2NetV2)
```python
from funasr import AutoModel
# Standalone speaker embedding extraction
model = AutoModel(model="iic/speech_eres2netv2_sv_zh-cn_16k-common", device="cuda:0")
res = model.generate(input="audio.wav")
embedding = res[0]["spk_embedding"] # shape: [1, 192]
```
```python
from funasr import AutoModel
# ASR with speaker diarization
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 60000},
punc_model="ct-punc",
spk_model="iic/speech_eres2netv2_sv_zh-cn_16k-common",
device="cuda:0",
)
res = model.generate(input="meeting.wav", batch_size_s=300)
for sentence in res:
print(f"[Speaker {sentence['spk']}] {sentence['text']}")
```
Notes: `spk_model` can also be `"cam++"` (CAM++ model). ERes2NetV2 provides improved short-duration speaker feature extraction.
Custom `spk_model` values can be ModelScope model ids or local model paths. The model must load through FunASR `AutoModel` and return `spk_embedding` from its `inference()` method; users still call `AutoModel.generate()`. Downstream speaker clustering uses those embeddings to assign `spk` labels.
#### Multi-language ASR (Qwen3-ASR)
```python
from funasr import AutoModel
# Qwen3-ASR supports 52 languages with auto language detection
# hub="hf" for HuggingFace, default hub="ms" for ModelScope
model = AutoModel(model="Qwen/Qwen3-ASR-1.7B", hub="hf", device="cuda:0")
# With forced language
res = model.generate(input="audio_zh.wav", language="Chinese")
print(res[0]["text"])
# Auto language detection
res = model.generate(input="audio.wav")
print(res[0]["text"], res[0].get("language", ""))
```
Notes: Use `pip install -U "qwen-asr==0.0.6" "transformers==4.57.6" accelerate`. `qwen-asr==0.0.6` pins `transformers==4.57.6`; newer incompatible Transformers builds may fail with nested `thinker_config` errors. Supports 0.6B and 1.7B model sizes.
#### Multi-language ASR (GLM-ASR)
```python
from funasr import AutoModel
# GLM-ASR-Nano supports 17 languages, optimized for dialects and low-volume speech
# hub="hf" for HuggingFace, default hub="ms" for ModelScope
model = AutoModel(model="zai-org/GLM-ASR-Nano-2512", hub="hf", device="cuda:0")
res = model.generate(input="audio.wav")
print(res[0]["text"])
# ModelScope (Chinese users)
model = AutoModel(model="ZhipuAI/GLM-ASR-Nano-2512", hub="ms", device="cuda:0")
```
Notes: Requires `transformers>=5.0.0` (install from source: `pip install git+https://github.com/huggingface/transformers`).
<a name="Training"></a>
## Model Training and Testing
### Quick Start
Execute via command line (for quick testing, not recommended):
```shell
funasr-train ++model=paraformer-zh ++train_data_set_list=data/list/train.jsonl ++valid_data_set_list=data/list/val.jsonl ++output_dir="./outputs" &> log.txt &
```
Execute with Python code (supports multi-node and multi-GPU, recommended):
```shell
cd examples/industrial_data_pretraining/paraformer
bash finetune.sh
# "log_file: ./outputs/log.txt"
```
Full code ref to [finetune.sh](https://github.com/modelscope/FunASR/blob/main/examples/industrial_data_pretraining/paraformer/finetune.sh)
### Detailed Parameter Description:
```shell
funasr/bin/train_ds.py \
++model="${model_name_or_model_dir}" \
++train_data_set_list="${train_data}" \
++valid_data_set_list="${val_data}" \
++dataset_conf.batch_size=20000 \
++dataset_conf.batch_type="token" \
++dataset_conf.num_workers=4 \
++train_conf.max_epoch=50 \
++train_conf.log_interval=1 \
++train_conf.resume=false \
++train_conf.validate_interval=2000 \
++train_conf.save_checkpoint_interval=2000 \
++train_conf.keep_nbest_models=20 \
++train_conf.avg_nbest_model=10 \
++optim_conf.lr=0.0002 \
++output_dir="${output_dir}" &> ${log_file}
```
- `model`（str）: The name of the model (the ID in the model repository), at which point the script will automatically download the model to local storage; alternatively, the path to a model already downloaded locally.
- `train_data_set_list`（str）: The path to the training data, typically in jsonl format, for specific details refer to [examples](https://github.com/modelscope/FunASR/blob/main/data/list).
- `valid_data_set_list`（str）：The path to the validation data, also generally in jsonl format, for specific details refer to examples](https://github.com/modelscope/FunASR/blob/main/data/list).
- `dataset_conf.batch_type`（str）：example (default), the type of batch. example means batches are formed with a fixed number of batch_size samples; length or token means dynamic batching, with total length or number of tokens of the batch equalling batch_size.
- `dataset_conf.batch_size`（int）：Used in conjunction with batch_type. When batch_type=example, it represents the number of samples; when batch_type=length, it represents the length of the samples, measured in fbank frames (1 frame = 10 ms) or the number of text tokens.
- `train_conf.max_epoch`（int）：The total number of epochs for training.
- `train_conf.log_interval`（int）：The number of steps between logging.
- `train_conf.resume`（int）：Whether to enable checkpoint resuming for training.
- `train_conf.validate_interval`（int）：The interval in steps to run validation tests during training.
- `train_conf.save_checkpoint_interval`（int）：The interval in steps for saving the model during training.
- `train_conf.keep_nbest_models`（int）：The maximum number of model parameters to retain, sorted by validation set accuracy, from highest to lowest.
- `train_conf.avg_nbest_model`（int）：Average over the top n models with the highest accuracy.
- `optim_conf.lr`（float）：The learning rate.
- `output_dir`（str）：The path for saving the model.
- `**kwargs`(dict): Any parameters in config.yaml can be specified directly here, for example, to filter out audio longer than 20s: dataset_conf.max_token_length=2000, measured in fbank frames (1 frame = 10 ms) or the number of text tokens.
#### Multi-GPU Training
##### Single-Machine Multi-GPU Training
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 1 --nproc_per_node ${gpu_num} \
../../../funasr/bin/train_ds.py ${train_args}
```
--nnodes represents the total number of participating nodes, while --nproc_per_node indicates the number of processes running on each node.
##### Multi-Machine Multi-GPU Training
On the master node, assuming the IP is 192.168.1.1 and the port is 12345, and you're using 2 GPUs, you would run the following command:
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 2 --node_rank 0 --nproc_per_node ${gpu_num} --master_addr=192.168.1.1 --master_port=12345 \
../../../funasr/bin/train_ds.py ${train_args}
```
On the worker node (assuming the IP is 192.168.1.2), you need to ensure that the MASTER_ADDR and MASTER_PORT environment variables are set to match those of the master node, and then run the same command:
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 2 --node_rank 1 --nproc_per_node ${gpu_num} --master_addr=192.168.1.1 --master_port=12345 \
../../../funasr/bin/train_ds.py ${train_args}
```
--nnodes indicates the total number of nodes participating in the training, --node_rank represents the ID of the current node, and --nproc_per_node specifies the number of processes running on each node (usually corresponds to the number of GPUs).
#### Data prepare
`jsonl` ref to（[demo](https://github.com/modelscope/FunASR/blob/main/data/list)）.
The instruction scp2jsonl can be used to generate from wav.scp and text.txt. The preparation process for wav.scp and text.txt is as follows:
`train_text.txt`
```bash
ID0012W0013 当客户风险承受能力评估依据发生变化时
ID0012W0014 所有只要处理 data 不管你是做 machine learning 做 deep learning
ID0012W0015 he tried to think how it could be
```
`train_wav.scp`
```bash
BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
ID0012W0015 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav
```
`Command`
```shell
# generate train.jsonl and val.jsonl from wav.scp and text.txt
scp2jsonl \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_out="../../../data/list/train.jsonl"
```
(Optional, not required) If you need to parse from jsonl back to wav.scp and text.txt, you can use the following command:
```shell
# generate wav.scp and text.txt from train.jsonl and val.jsonl
jsonl2scp \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_in="../../../data/list/train.jsonl"
```
#### Large-Scale Data Training
When dealing with large datasets (e.g., 50,000 hours or more), memory issues may arise, especially in multi-GPU experiments. To address this, split the JSONL files into slices, write them into a TXT file (one slice per line), and set `data_split_num`. For example:
```shell
train_data="/root/data/list/data.list"
funasr/bin/train_ds.py \
++train_data_set_list="${train_data}" \
++dataset_conf.data_split_num=256
```
**Details:**
- `data.list`: A plain text file listing the split JSONL files. For example, the content of `data.list` might be:
```bash
data/list/train.0.jsonl
data/list/train.1.jsonl
...
```
- `data_split_num`: Specifies the number of slice groups. For instance, if `data.list` contains 512 lines and `data_split_num=256`, the data will be divided into 256 groups, each containing 2 JSONL files. This ensures that only 2 JSONL files are loaded for training at a time, reducing memory usage during training. Note: Groups are created sequentially.
**Recommendation:**
If the dataset is extremely large and contains heterogeneous data types, perform **data balancing** during splitting to ensure uniformity across groups.
#### Training log
##### log.txt
```shell
tail log.txt
[2024-03-21 15:55:52,137][root][INFO] - train, rank: 3, epoch: 0/50, step: 6990/1, total step: 6990, (loss_avg_rank: 0.327), (loss_avg_epoch: 0.409), (ppl_avg_epoch: 1.506), (acc_avg_epoch: 0.795), (lr: 1.165e-04), [('loss_att', 0.259), ('acc', 0.825), ('loss_pre', 0.04), ('loss', 0.299), ('batch_size', 40)], {'data_load': '0.000', 'forward_time': '0.315', 'backward_time': '0.555', 'optim_time': '0.076', 'total_time': '0.947'}, GPU, memory: usage: 3.830 GB, peak: 18.357 GB, cache: 20.910 GB, cache_peak: 20.910 GB
[2024-03-21 15:55:52,139][root][INFO] - train, rank: 1, epoch: 0/50, step: 6990/1, total step: 6990, (loss_avg_rank: 0.334), (loss_avg_epoch: 0.409), (ppl_avg_epoch: 1.506), (acc_avg_epoch: 0.795), (lr: 1.165e-04), [('loss_att', 0.285), ('acc', 0.823), ('loss_pre', 0.046), ('loss', 0.331), ('batch_size', 36)], {'data_load': '0.000', 'forward_time': '0.334', 'backward_time': '0.536', 'optim_time': '0.077', 'total_time': '0.948'}, GPU, memory: usage: 3.943 GB, peak: 18.291 GB, cache: 19.619 GB, cache_peak: 19.619 GB
```
- `rank`：gpu id。
- `epoch`,`step`,`total step`：the current epoch, step, and total steps.
- `loss_avg_rank`：the average loss across all GPUs for the current step.
- `loss/ppl/acc_avg_epoch`：the overall average loss/perplexity/accuracy for the current epoch, up to the current step count. The last step of the epoch when it ends represents the total average loss/perplexity/accuracy for that epoch; it is recommended to use the accuracy metric.
- `lr`：the learning rate for the current step.
- `[('loss_att', 0.259), ('acc', 0.825), ('loss_pre', 0.04), ('loss', 0.299), ('batch_size', 40)]`：the specific data for the current GPU ID.
- `total_time`：the total time taken for a single step.
- `GPU, memory`：the model-used/peak memory and the model+cache-used/peak memory.
##### tensorboard
```bash
tensorboard --logdir /xxxx/FunASR/examples/industrial_data_pretraining/paraformer/outputs/log/tensorboard
```
http://localhost:6006/
### 训练后模型测试
#### With `configuration.json` file
Assuming the training model path is: ./model_dir, if a configuration.json file has been generated in this directory, you only need to change the model name to the model path in the above model inference method.
For example, for shell inference:
```shell
python -m funasr.bin.inference ++model="./model_dir" ++input=="${input}" ++output_dir="${output_dir}"
```
Python inference
```python
from funasr import AutoModel
model = AutoModel(model="./model_dir")
res = model.generate(input=wav_file)
print(res)
```
#### Without `configuration.json` file
If there is no configuration.json in the model path, you need to manually specify the exact configuration file path and the model path.
```shell
python -m funasr.bin.inference \
--config-path "${local_path}" \
--config-name "${config}" \
++init_param="${init_param}" \
++tokenizer_conf.token_list="${tokens}" \
++frontend_conf.cmvn_file="${cmvn_file}" \
++input="${input}" \
++output_dir="${output_dir}" \
++device="${device}"
```
Parameter Introduction
- `config-path`：This is the path to the config.yaml saved during the experiment, which can be found in the experiment's output directory.
- `config-name`：The name of the configuration file, usually config.yaml. It supports both YAML and JSON formats, for example config.json.
- `init_param`：The model parameters that need to be tested, usually model.pt. You can choose a specific model file as needed.
- `tokenizer_conf.token_list`：The path to the vocabulary file, which is normally specified in config.yaml. There is no need to manually specify it again unless the path in config.yaml is incorrect, in which case the correct path must be manually specified here.
- `frontend_conf.cmvn_file`：The CMVN (Cepstral Mean and Variance Normalization) file used when extracting fbank features from WAV files, which is usually specified in config.yaml. There is no need to manually specify it again unless the path in config.yaml is incorrect, in which case the correct path must be manually specified here.
Other parameters are the same as mentioned above. A complete [example](https://github.com/modelscope/FunASR/blob/main/examples/industrial_data_pretraining/paraformer/infer_from_local.sh) can be found here.
<a name="Export"></a>
## Export ONNX
### Command-line usage
```shell
funasr-export ++model=paraformer ++quantize=false ++device=cpu
```
### Python
```python
from funasr import AutoModel
model = AutoModel(model="paraformer", device="cpu")
res = model.export(quantize=False)
```
### optimize onnx
```shell
# pip3 install -U onnxslim
onnxslim model.onnx model.onnx
```
### Test ONNX
```python
# pip3 install -U funasr-onnx
from funasr_onnx import Paraformer
model_dir = "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"
model = Paraformer(model_dir, batch_size=1, quantize=True)
wav_path = ['~/.cache/modelscope/hub/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav']
result = model(wav_path)
print(result)
```
More examples ref to [demo](https://github.com/modelscope/FunASR/tree/main/runtime/python/onnxruntime)
<a name="new-model-registration-tutorial"></a>
## New Model Registration Tutorial
### Viewing the Registry
```plaintext
from funasr.register import tables
tables.print()
```
Supports viewing the registry of a specified type: `tables.print("model")`
### Registering Models
```python
from funasr.register import tables
@tables.register("model_classes", "SenseVoiceSmall")
class SenseVoiceSmall(nn.Module):
def __init__(*args, **kwargs):
...
def forward(
self,
**kwargs,
):
def inference(
self,
data_in,
data_lengths=None,
key: list = None,
tokenizer=None,
frontend=None,
**kwargs,
):
...
```
Add `@tables.register("model_classes","SenseVoiceSmall")` before the class name that needs to be registered to complete the registration. The class needs to implement the methods: __init__, forward, and inference.
Complete code: [https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L443](https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L443)
After registration, specify the newly registered model in config.yaml to define the model
```python
model: SenseVoiceSmall
model_conf:
...
```
[More detailed tutorial documents](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/Tables.md)
docs/tutorial/README_zh.md
+610
View File
@@ -0,0 +1,610 @@
(简体中文|[English](./README.md))
FunASR开源了大量在工业数据上预训练模型，您可以在 [模型许可协议](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE)下自由使用、复制、修改和分享FunASR模型，下面列举代表性的模型，更多模型请参考 [模型仓库](https://github.com/modelscope/FunASR/tree/main/model_zoo)。
<div align="center">
<h4>
<a href="#模型推理"> 模型推理 </a>
｜<a href="#模型训练与测试"> 模型训练与测试 </a>
｜<a href="#模型导出与测试"> 模型导出与测试 </a>
｜<a href="#新模型注册教程"> 新模型注册教程 </a>
</h4>
</div>
<a name="模型推理"></a>
## 模型推理
### 快速使用
不想先配置本地环境？可以先跑 [Colab 快速体验](../../examples/colab/README_zh.md)，确认模型和输出格式后再切到命令行。
命令行方式调用：
```shell
funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++input=asr_example_zh.wav
```
python代码调用（推荐）
```python
from funasr import AutoModel
model = AutoModel(model="paraformer-zh")
res = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav")
print(res)
```
### 接口说明
#### AutoModel 定义
```python
model = AutoModel(model=[str], device=[str], ncpu=[int], output_dir=[str], batch_size=[int], hub=[str], **kwargs)
```
- `model`(str): [模型仓库](https://github.com/modelscope/FunASR/tree/main/model_zoo) 中的模型名称，或本地磁盘中的模型路径
- `device`(str): `cuda:0`（默认gpu0），使用 GPU 进行推理，指定。如果为`cpu`，则使用 CPU 进行推理。`mps`：mac电脑M系列新品试用mps进行推理。`xpu`：使用英特尔gpu进行推理。
- `ncpu`(int): `4` （默认），设置用于 CPU 内部操作并行性的线程数
- `output_dir`(str): `None` （默认），如果设置，输出结果的输出路径
- `batch_size`(int): `1` （默认），解码时的批处理，样本个数
- `hub`(str)：`ms`（默认），从modelscope下载模型。如果为`hf`，从huggingface下载模型。
- `**kwargs`(dict): 所有在`config.yaml`中参数，均可以直接在此处指定，例如，vad模型中最大切割长度 `max_single_segment_time=6000` （毫秒）。
#### AutoModel 推理
```python
res = model.generate(input=[str], output_dir=[str])
```
- `input`: 要解码的输入，可以是：
- wav文件路径, 例如: asr_example.wav
- pcm文件路径, 例如: asr_example.pcm，此时需要指定音频采样率fs（默认为16000）
- 音频字节数流，例如：麦克风的字节数数据
- wav.scp，kaldi 样式的 wav 列表 (`wav_id \t wav_path`), 例如:
```text
asr_example1 ./audios/asr_example1.wav
asr_example2 ./audios/asr_example2.wav
```
在这种输入 `wav.scp` 的情况下，必须设置 `output_dir` 以保存输出结果
- 音频采样点，例如：`audio, rate = soundfile.read("asr_example_zh.wav")`, 数据类型为 numpy.ndarray。支持batch输入，类型为list：
```[audio_sample1, audio_sample2, ..., audio_sampleN]```
- fbank输入，支持组batch。shape为[batch, frames, dim]，类型为torch.Tensor，例如
- `output_dir`: None （默认），如果设置，输出结果的输出路径
- `**kwargs`(dict): 与模型相关的推理参数，例如，`beam_size=10`，`decoding_ctc_weight=0.1`。
### 更多用法介绍
#### 非实时语音识别
```python
from funasr import AutoModel
# paraformer-zh is a multi-functional asr model
# use vad, punc, spk or not as you need
model = AutoModel(model="paraformer-zh",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 60000},
punc_model="ct-punc",
# spk_model="cam++"
)
wav_file = f"{model.model_path}/example/asr_example.wav"
res = model.generate(input=wav_file, batch_size_s=300, batch_size_threshold_s=60, hotword='魔搭')
print(res)
```
#### ASR 后处理热词替换
模型层 `hotword` / `hotwords` 用于在解码阶段提升少量高优先级词的识别率；如果词表很大（例如几千个股票名），更适合在识别完成后做文本级纠错：
```python
from funasr import AutoModel
model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc")
res = model.generate(
input="asr_example.wav",
postprocess_hotwords={
"科大迅飞": "科大讯飞",
"东方财富": "东方财富",
},
postprocess_hotword_threshold=0.85,
return_postprocess_hotword_matches=True,
)
print(res[0]["text"])
print(res[0].get("postprocess_hotword_matches"))
```
也支持热词文件 `postprocess_hotword_file`，每行一个目标词，或一行一个显式映射（`错误词=>目标词`）。模糊匹配需要额外安装 `pypinyin` 和 `rapidfuzz`；未安装时仍可使用显式映射。
注意：
- 通常模型输入限制时长30s以下，组合`vad_model`后，支持任意时长音频输入，不局限于paraformer模型，所有音频输入模型均可以。
- `model`相关的参数可以直接在`AutoModel`定义中直接指定；与`vad_model`相关参数可以通过`vad_kwargs`来指定，类型为dict；类似的有`punc_kwargs`，`spk_kwargs`；
- `max_single_segment_time`: 表示`vad_model`最大切割音频时长, 单位是毫秒ms.
- `batch_size_s` 表示采用动态batch，batch中总音频时长，单位为秒s。
- `batch_size_threshold_s`: 表示`vad_model`切割后音频片段时长超过 `batch_size_threshold_s`阈值时，将batch_size数设置为1, 单位为秒s.
建议：当您输入为长音频，遇到OOM问题时，因为显存占用与音频时长呈平方关系增加，分为3种情况：
- a)推理起始阶段，显存主要取决于`batch_size_s`，适当减小该值，可以减少显存占用；
- b)推理中间阶段，遇到VAD切割的长音频片段，总token数小于`batch_size_s`，仍然出现OOM，可以适当减小`batch_size_threshold_s`，超过阈值，强制batch为1;
- c)推理快结束阶段，遇到VAD切割的长音频片段，总token数小于`batch_size_s`，且超过阈值`batch_size_threshold_s`，强制batch为1，仍然出现OOM，可以适当减小`max_single_segment_time`，使得VAD切割音频时长变短。
#### 实时语音识别
```python
from funasr import AutoModel
chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention
decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention
model = AutoModel(model="paraformer-zh-streaming")
import soundfile
import os
wav_file = os.path.join(model.model_path, "example/asr_example.wav")
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = chunk_size[1] * 960 # 600ms
cache = {}
total_chunk_num = int((len(speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
is_final = i == total_chunk_num - 1
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
print(res)
```
注：`chunk_size`为流式延时配置，`[0,10,5]`表示上屏实时出字粒度为`10*60=600ms`，未来信息为`5*60=300ms`。每次推理输入为`600ms`（采样点数为`16000*0.6=960`），输出为对应文字，最后一个语音片段输入需要设置`is_final=True`来强制输出最后一个字。
#### 语音端点检测（非实时）
```python
from funasr import AutoModel
model = AutoModel(model="fsmn-vad")
wav_file = f"{model.model_path}/example/vad_example.wav"
res = model.generate(input=wav_file)
print(res)
```
注：VAD模型输出格式为：`[[beg1, end1], [beg2, end2], .., [begN, endN]]`，其中`begN/endN`表示第`N`个有效音频片段的起始点/结束点，
单位为毫秒。
#### 语音端点检测（实时）
```python
from funasr import AutoModel
chunk_size = 200 # ms
model = AutoModel(model="fsmn-vad")
import soundfile
wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = int(chunk_size * sample_rate / 1000)
cache = {}
total_chunk_num = int((len(speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
is_final = i == total_chunk_num - 1
res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size)
if len(res[0]["value"]):
print(res)
```
注：流式VAD模型输出格式为4种情况：
- `[[beg1, end1], [beg2, end2], .., [begN, endN]]`：同上离线VAD输出结果。
- `[[beg, -1]]`：表示只检测到起始点。
- `[[-1, end]]`：表示只检测到结束点。
- `[]`：表示既没有检测到起始点，也没有检测到结束点
输出结果单位为毫秒，从起始点开始的绝对时间。
#### 标点恢复
```python
from funasr import AutoModel
model = AutoModel(model="ct-punc")
res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
print(res)
```
#### 时间戳预测
```python
from funasr import AutoModel
model = AutoModel(model="fa-zh")
wav_file = f"{model.model_path}/example/asr_example.wav"
text_file = f"{model.model_path}/example/text.txt"
res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
print(res)
```
更多（[示例](https://github.com/modelscope/FunASR/tree/main/examples/industrial_data_pretraining)）
#### 说话人确认/分割 (ERes2NetV2)
```python
from funasr import AutoModel
# 单独提取说话人 embedding
model = AutoModel(model="iic/speech_eres2netv2_sv_zh-cn_16k-common", device="cuda:0")
res = model.generate(input="audio.wav")
embedding = res[0]["spk_embedding"] # shape: [1, 192]
```
```python
from funasr import AutoModel
# ASR + 说话人分割
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 60000},
punc_model="ct-punc",
spk_model="iic/speech_eres2netv2_sv_zh-cn_16k-common",
device="cuda:0",
)
res = model.generate(input="meeting.wav", batch_size_s=300)
for sentence in res:
print(f"[说话人 {sentence['spk']}] {sentence['text']}")
```
注：`spk_model` 也可以使用 `"cam++"` (CAM++ 模型)。ERes2NetV2 在短时说话人特征提取方面有所改进。
自定义 `spk_model` 可以传 ModelScope 模型 ID 或本地模型路径；模型需要能通过 FunASR `AutoModel` 加载，并在自身 `inference()` 方法中返回 `spk_embedding`，用户侧仍调用 `AutoModel.generate()`。后续说话人聚类会使用这些 embedding 分配 `spk` 标签。
#### 多语言语音识别 (Qwen3-ASR)
```python
from funasr import AutoModel
# Qwen3-ASR 支持52种语言自动语言检测
# hub="hf" 从 HuggingFace 下载，默认 hub="ms" 从 ModelScope 下载
model = AutoModel(model="Qwen/Qwen3-ASR-1.7B", hub="hf", device="cuda:0")
# 指定语言
res = model.generate(input="audio_zh.wav", language="Chinese")
print(res[0]["text"])
# 自动语言检测
res = model.generate(input="audio.wav")
print(res[0]["text"], res[0].get("language", ""))
```
注：请使用 `pip install -U "qwen-asr==0.0.6" "transformers==4.57.6" accelerate`。`qwen-asr==0.0.6` 锁定 `transformers==4.57.6`；若环境中混入不兼容的新版 Transformers，可能触发嵌套 `thinker_config` 报错。支持 0.6B 和 1.7B 两种模型规格。
#### 多语言语音识别 (GLM-ASR)
```python
from funasr import AutoModel
# GLM-ASR-Nano 支持17种语言，方言及低音量语音优化
# hub="hf" 从 HuggingFace 下载，默认 hub="ms" 从 ModelScope 下载
model = AutoModel(model="zai-org/GLM-ASR-Nano-2512", hub="hf", device="cuda:0")
res = model.generate(input="audio.wav")
print(res[0]["text"])
# ModelScope（国内用户）
model = AutoModel(model="ZhipuAI/GLM-ASR-Nano-2512", hub="ms", device="cuda:0")
```
注：需要 `transformers>=5.0.0`（从源码安装：`pip install git+https://github.com/huggingface/transformers`）。
<a name="核心功能"></a>
## 模型训练与测试
### 快速开始
命令行执行（用于快速测试，不推荐）：
```shell
funasr-train ++model=paraformer-zh ++train_data_set_list=data/list/train.jsonl ++valid_data_set_list=data/list/val.jsonl ++output_dir="./outputs" &> log.txt &
```
python代码执行（可以多机多卡，推荐）
```shell
cd examples/industrial_data_pretraining/paraformer
bash finetune.sh
# "log_file: ./outputs/log.txt"
```
详细完整的脚本参考 [finetune.sh](https://github.com/modelscope/FunASR/blob/main/examples/industrial_data_pretraining/paraformer/finetune.sh)
### 详细参数介绍
```shell
funasr/bin/train_ds.py \
++model="${model_name_or_model_dir}" \
++train_data_set_list="${train_data}" \
++valid_data_set_list="${val_data}" \
++dataset_conf.batch_size=20000 \
++dataset_conf.batch_type="token" \
++dataset_conf.num_workers=4 \
++train_conf.max_epoch=50 \
++train_conf.log_interval=1 \
++train_conf.resume=false \
++train_conf.validate_interval=2000 \
++train_conf.save_checkpoint_interval=2000 \
++train_conf.keep_nbest_models=20 \
++train_conf.avg_nbest_model=10 \
++optim_conf.lr=0.0002 \
++output_dir="${output_dir}" &> ${log_file}
```
- `model`（str）：模型名字（模型仓库中的ID），此时脚本会自动下载模型到本地；或者本地已经下载好的模型路径。
- `train_data_set_list`（str）：训练数据路径，默认为jsonl格式，具体参考（[例子](https://github.com/modelscope/FunASR/blob/main/data/list)）。
- `valid_data_set_list`（str）：验证数据路径，默认为jsonl格式，具体参考（[例子](https://github.com/modelscope/FunASR/blob/main/data/list)）。
- `dataset_conf.batch_type`（str）：`example`（默认），batch的类型。`example`表示按照固定数目batch_size个样本组batch；`length` or `token` 表示动态组batch，batch总长度或者token数为batch_size。
- `dataset_conf.batch_size`（int）：与 `batch_type` 搭配使用，当 `batch_type=example` 时，表示样本个数；当 `batch_type=length` 时，表示样本中长度，单位为fbank帧数（1帧10ms）或者文字token个数。
- `train_conf.max_epoch`（int）：`100`（默认），训练总epoch数。
- `train_conf.log_interval`（int）：`50`（默认），打印日志间隔step数。
- `train_conf.resume`（int）：`True`（默认），是否开启断点重训。
- `train_conf.validate_interval`（int）：`5000`（默认），训练中做验证测试的间隔step数。
- `train_conf.save_checkpoint_interval`（int）：`5000`（默认），训练中模型保存间隔step数。
- `train_conf.avg_keep_nbest_models_type`（str）：`acc`（默认），保留nbest的标准为acc（越大越好）。`loss`表示，保留nbest的标准为loss（越小越好）。
- `train_conf.keep_nbest_models`（int）：`500`（默认），保留最大多少个模型参数，配合 `avg_keep_nbest_models_type` 按照验证集 acc/loss 保留最佳的n个模型，其他删除，节约存储空间。
- `train_conf.avg_nbest_model`（int）：`10`（默认），保留最大多少个模型参数，配合 `avg_keep_nbest_models_type` 按照验证集 acc/loss 对最佳的n个模型平均。
- `train_conf.accum_grad`（int）：`1`（默认），梯度累积功能。
- `train_conf.grad_clip`（float）：`10.0`（默认），梯度截断功能。
- `train_conf.use_fp16`（bool）：`False`（默认），开启fp16训练，加快训练速度。
- `optim_conf.lr`（float）：学习率。
- `output_dir`（str）：模型保存路径。
- `**kwargs`(dict): 所有在`config.yaml`中参数，均可以直接在此处指定，例如，过滤20s以上长音频：`dataset_conf.max_token_length=2000`，单位为音频fbank帧数（1帧10ms）或者文字token个数。
#### 多gpu训练
##### 单机多gpu训练
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 1 --nproc_per_node ${gpu_num} \
../../../funasr/bin/train_ds.py ${train_args}
```
--nnodes 表示参与的节点总数，--nproc_per_node 表示每个节点上运行的进程数
##### 多机多gpu训练
在主节点上，假设IP为192.168.1.1，端口为12345，使用的是2个GPU，则运行如下命令：
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 2 --node_rank 0 --nproc_per_node ${gpu_num} --master_addr 192.168.1.1 --master_port 12345 \
../../../funasr/bin/train_ds.py ${train_args}
```
在从节点上（假设IP为192.168.1.2），你需要确保MASTER_ADDR和MASTER_PORT环境变量与主节点设置的一致，并运行同样的命令：
```shell
export CUDA_VISIBLE_DEVICES="0,1"
gpu_num=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
torchrun --nnodes 2 --node_rank 1 --nproc_per_node ${gpu_num} --master_addr 192.168.1.1 --master_port 12345 \
../../../funasr/bin/train_ds.py ${train_args}
```
--nnodes 表示参与的节点总数，--node_rank 表示当前节点id，--nproc_per_node 表示每个节点上运行的进程数（通常为gpu个数）
#### 准备数据
`jsonl`格式可以参考（[例子](https://github.com/modelscope/FunASR/blob/main/data/list)）。
可以用指令 `scp2jsonl` 从wav.scp与text.txt生成。wav.scp与text.txt准备过程如下：
`train_text.txt`
左边为数据唯一ID，需与`train_wav.scp`中的`ID`一一对应
右边为音频文件标注文本，格式如下：
```bash
ID0012W0013 当客户风险承受能力评估依据发生变化时
ID0012W0014 所有只要处理 data 不管你是做 machine learning 做 deep learning
ID0012W0015 he tried to think how it could be
```
`train_wav.scp`
左边为数据唯一ID，需与`train_text.txt`中的`ID`一一对应
右边为音频文件的路径，格式如下
```bash
BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
ID0012W0015 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav
```
`生成指令`
```shell
# generate train.jsonl and val.jsonl from wav.scp and text.txt
scp2jsonl \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_out="../../../data/list/train.jsonl"
```
（可选，非必需）如果需要从jsonl解析成wav.scp与text.txt，可以使用指令：
```shell
# generate wav.scp and text.txt from train.jsonl and val.jsonl
jsonl2scp \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_in="../../../data/list/train.jsonl"
```
#### 大数据训练
如果数据量很大，例如5万小时以上，这时候容易遇到内存不足的问题，特别是多gpu实验，这时候需要对jsonl文件进行切分成slice，然后写到txt里面，一行一个slice，然后设置`data_split_num`，例如：
```shell
train_data="/root/data/list/data.list"
funasr/bin/train_ds.py \
++train_data_set_list="${train_data}" \
++dataset_conf.data_split_num=256
```
其中：
`data.list`：为纯文本，内容是切割后的jsonl文件，例如，`data.list`的内容为：
```bash
data/list/train.0.jsonl
data/list/train.1.jsonl
...
```
`data_split_num`：表示切分slice分组个数，例如，data.list中共512行，data_split_num=256，表示分成256组，每组有2个jsonl文件，这样每次只load 2个jsonl数据进行训练，从而降低训练过程中内存使用。注意是按照顺序分组。
如果是，非常大，并且数据类型差异比较大，建议切分时候进行数据均衡。
#### 查看训练日志
##### 查看实验log
```shell
tail log.txt
[2024-03-21 15:55:52,137][root][INFO] - train, rank: 3, epoch: 0/50, step: 6990/1, total step: 6990, (loss_avg_rank: 0.327), (loss_avg_epoch: 0.409), (ppl_avg_epoch: 1.506), (acc_avg_epoch: 0.795), (lr: 1.165e-04), [('loss_att', 0.259), ('acc', 0.825), ('loss_pre', 0.04), ('loss', 0.299), ('batch_size', 40)], {'data_load': '0.000', 'forward_time': '0.315', 'backward_time': '0.555', 'optim_time': '0.076', 'total_time': '0.947'}, GPU, memory: usage: 3.830 GB, peak: 18.357 GB, cache: 20.910 GB, cache_peak: 20.910 GB
[2024-03-21 15:55:52,139][root][INFO] - train, rank: 1, epoch: 0/50, step: 6990/1, total step: 6990, (loss_avg_rank: 0.334), (loss_avg_epoch: 0.409), (ppl_avg_epoch: 1.506), (acc_avg_epoch: 0.795), (lr: 1.165e-04), [('loss_att', 0.285), ('acc', 0.823), ('loss_pre', 0.046), ('loss', 0.331), ('batch_size', 36)], {'data_load': '0.000', 'forward_time': '0.334', 'backward_time': '0.536', 'optim_time': '0.077', 'total_time': '0.948'}, GPU, memory: usage: 3.943 GB, peak: 18.291 GB, cache: 19.619 GB, cache_peak: 19.619 GB
```
指标解释：
- `rank`：表示gpu id。
- `epoch`,`step`,`total step`：表示当前epoch，step，总step。
- `loss_avg_rank`：表示当前step，所有gpu平均loss。
- `loss/ppl/acc_avg_epoch`：表示当前epoch周期，截止当前step数时，总平均loss/ppl/acc。epoch结束时的最后一个step表示epoch总平均loss/ppl/acc，推荐使用acc指标。
- `lr`：当前step的学习率。
- `[('loss_att', 0.259), ('acc', 0.825), ('loss_pre', 0.04), ('loss', 0.299), ('batch_size', 40)]`：表示当前gpu id的具体数据。
- `total_time`：表示单个step总耗时。
- `GPU, memory`：分别表示，模型使用/峰值显存，模型+缓存使用/峰值显存。
##### tensorboard可视化
```bash
tensorboard --logdir /xxxx/FunASR/examples/industrial_data_pretraining/paraformer/outputs/log/tensorboard
```
浏览器中打开：http://localhost:6006/
### 训练后模型测试
#### 有configuration.json
假定，训练模型路径为：./model_dir，如果该目录下有生成configuration.json，只需要将 [上述模型推理方法](https://github.com/modelscope/FunASR/blob/main/examples/README_zh.md#%E6%A8%A1%E5%9E%8B%E6%8E%A8%E7%90%86) 中模型名字修改为模型路径即可
例如：
从shell推理
```shell
python -m funasr.bin.inference ++model="./model_dir" ++input=="${input}" ++output_dir="${output_dir}"
```
从python推理
```python
from funasr import AutoModel
model = AutoModel(model="./model_dir")
res = model.generate(input=wav_file)
print(res)
```
#### 无configuration.json时
如果模型路径中无configuration.json时，需要手动指定具体配置文件路径与模型路径
```shell
python -m funasr.bin.inference \
--config-path "${local_path}" \
--config-name "${config}" \
++init_param="${init_param}" \
++tokenizer_conf.token_list="${tokens}" \
++frontend_conf.cmvn_file="${cmvn_file}" \
++input="${input}" \
++output_dir="${output_dir}" \
++device="${device}"
```
参数介绍
- `config-path`：为实验中保存的 `config.yaml`，可以从实验输出目录中查找。
- `config-name`：配置文件名，一般为 `config.yaml`，支持yaml格式与json格式，例如 `config.json`
- `init_param`：需要测试的模型参数，一般为`model.pt`，可以自己选择具体的模型文件
- `tokenizer_conf.token_list`：词表文件路径，一般在 `config.yaml` 有指定，无需再手动指定，当 `config.yaml` 中路径不正确时，需要在此处手动指定。
- `frontend_conf.cmvn_file`：wav提取fbank中用到的cmvn文件，一般在 `config.yaml` 有指定，无需再手动指定，当 `config.yaml` 中路径不正确时，需要在此处手动指定。
其他参数同上，完整 [示例](https://github.com/modelscope/FunASR/blob/main/examples/industrial_data_pretraining/paraformer/infer_from_local.sh)
<a name="模型导出与测试"></a>
## 模型导出与测试
### 从命令行导出
```shell
funasr-export ++model=paraformer ++quantize=false
```
### 从Python导出
```python
from funasr import AutoModel
model = AutoModel(model="paraformer")
res = model.export(quantize=False)
```
### 优化onnx
```shell
# pip3 install -U onnxslim
onnxslim model.onnx model.onnx
```
### 测试ONNX
```python
# pip3 install -U funasr-onnx
from funasr_onnx import Paraformer
model_dir = "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"
model = Paraformer(model_dir, batch_size=1, quantize=True)
wav_path = ['~/.cache/modelscope/hub/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav']
result = model(wav_path)
print(result)
```
更多例子请参考 [样例](https://github.com/modelscope/FunASR/tree/main/runtime/python/onnxruntime)
<a name="新模型注册教程"></a>
## 新模型注册教程
### 查看注册表
```plaintext
from funasr.register import tables
tables.print()
```
支持查看指定类型的注册表：\`tables.print("model")\`
### 注册模型
```python
from funasr.register import tables
@tables.register("model_classes", "SenseVoiceSmall")
class SenseVoiceSmall(nn.Module):
def __init__(*args, **kwargs):
...
def forward(
self,
**kwargs,
):
def inference(
self,
data_in,
data_lengths=None,
key: list = None,
tokenizer=None,
frontend=None,
**kwargs,
):
...
```
在需要注册的类名前加上 `@tables.register("model_classes","SenseVoiceSmall")`，即可完成注册，类需要实现有：__init__，forward，inference方法。
完整代码：[https://github.com/modelscope/FunASR/blob/main/funasr/models/sense\_voice/model.py#L443](https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L443)
注册完成后，在config.yaml中指定新注册模型，即可实现对模型的定义
```python
model: SenseVoiceSmall
model_conf:
...
```
[关于注册更多详细教程文档](https://github.com/modelscope/FunASR/blob/main/docs/tutorial/Tables_zh.md)
docs/tutorial/Tables.md
+363
View File
@@ -0,0 +1,363 @@
# FunASR-1.x.x Registration  New Model Tutorial
([简体中文](./Tables_zh.md)|English)
The original intention of the funasr-1.x.x version is to make model integration easier. The core feature is the registry and AutoModel:
* The introduction of the registry enables the development of building blocks to access the model, compatible with a variety of tasks;
* The newly designed AutoModel interface unifies modelscope, huggingface, and funasr inference and training interfaces, and supports free download of repositories;
* Support model export, demo-level service deployment, and industrial-level multi-concurrent service deployment;
* Unify academic and industrial model inference training scripts;
# Quick to get started
## AutoModel usage
### SenseVoiceSmall 模型
Input any length of voice, the output as the voice content corresponding to the text, the text has punctuation broken sentences, support Chinese, English, Japanese, Guangdong, Korean and 5 Chinese languages. \[Word-level timestamp and speaker identity\] will be supported later.
```python
from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 30000},
device="cuda:0",
)
res = model.generate(
input=f"{model.model_path}/example/en.mp3",
language="auto", # "zn", "en", "yue", "ja", "ko", "nospeech"
use_itn=True,
batch_size_s=60,
)
text = rich_transcription_postprocess(res[0]["text"])
print(text) #👏Senior staff, Priipal Doris Jackson, Wakefield faculty, and, of course, my fellow classmates.I am honored to have been chosen to speak before my classmates, as well as the students across America today.
```
## API documentation
#### Definition of AutoModel
```plaintext
Model = AutoModel(model=[str], device=[str], ncpu=[int], output_dir=[str], batch_size= [int], hub=[str], **quargs)
```
* `model`(str): [Model Warehouse](https://github.com/modelscope/FunASR/tree/main/model_zoo)The model name in, or the model path in the local disk
* `device`(str): `cuda:0`(Default gpu0), using GPU for inference, specified. If`cpu`Then the CPU is used for inference
* `ncpu`(int): `4`(Default), set the number of threads used for CPU internal operation parallelism
* `output_dir`(str): `None`(Default) If set, the output path of the output result
* `batch_size`(int): `1`(Default), batch processing during decoding, number of samples
* `hub`(str)：`ms`(Default) to download the model from modelscope. If`hf`To download the model from huggingface.
* `**kwargs`(dict): All in`config.yaml`Parameters, which can be specified directly here, for example, the maximum cut length in the vad model.`max_single_segment_time=6000`(Milliseconds).
#### AutoModel reasoning
```plaintext
Res = model.generate(input=[str], output_dir=[str])
```
* * wav file path, for example: asr\_example.wav
* pcm file path, for example: asr\_example.pcm, you need to specify the audio sampling rate fs (default is 16000)
* Audio byte stream, for example: microphone byte data
* wav.scp,kaldi-style wav list (`wav_id \t wav_path`), for example:
```plaintext
Asr_example1./audios/asr_example1.wav
Asr_example2./audios/asr_example2.wav
```
In this input
* Audio sampling points, for example:`audio, rate = soundfile.read("asr_example_zh.wav")`Is numpy.ndarray. batch input is supported. The type is list:`[audio_sample1, audio_sample2, ..., audio_sampleN]`
* fbank input, support group batch. shape is \[batch, frames, dim\], type is torch.Tensor, for example
* `output_dir`: None (default), if set, the output path of the output result
* `**kwargs`(dict): Model-related inference parameters, e.g,`beam_size=10`,`decoding_ctc_weight=0.1`.
Detailed documentation link:[https://github.com/modelscope/FunASR/blob/main/examples/README\_zh.md](https://github.com/modelscope/FunASR/blob/main/examples/README_zh.md)
# Registry Details
Take the SenseVoiceSmall model as an example, explain how to register a new model, model link:
**modelscope：**[https://www.modelscope.cn/models/iic/SenseVoiceSmall/files](https://www.modelscope.cn/models/iic/SenseVoiceSmall/files)
**huggingface：**[https://huggingface.co/FunAudioLLM/SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)
## Model Resource Catalog
![image.png](https://alidocs.oss-cn-zhangjiakou.aliyuncs.com/res/8oLl9y628rBNlapY/img/cab7f215-787f-4407-885a-14dc89ae9e02.png)
Configuration File: config.yaml
```yaml
encoder: SenseVoiceEncoderSmall
encoder_conf:
output_size: 512
attention_heads: 4
linear_units: 2048
num_blocks: 50
tp_blocks: 20
dropout_rate: 0.1
positional_dropout_rate: 0.1
attention_dropout_rate: 0.1
input_layer: pe
pos_enc_class: SinusoidalPositionEncoder
normalize_before: true
kernel_size: 11
sanm_shfit: 0
selfattention_layer_type: sanm
model: SenseVoiceSmall
model_conf:
length_normalized_loss: true
sos: 1
eos: 2
ignore_id: -1
tokenizer: SentencepiecesTokenizer
tokenizer_conf:
bpemodel: null
unk_symbol: <unk>
split_with_space: true
frontend: WavFrontend
frontend_conf:
fs: 16000
window: hamming
n_mels: 80
frame_length: 25
frame_shift: 10
lfr_m: 7
lfr_n: 6
cmvn_file: null
dataset: SenseVoiceCTCDataset
dataset_conf:
index_ds: IndexDSJsonl
batch_sampler: EspnetStyleBatchSampler
data_split_num: 32
batch_type: token
batch_size: 14000
max_token_length: 2000
min_token_length: 60
max_source_length: 2000
min_source_length: 60
max_target_length: 200
min_target_length: 0
shuffle: true
num_workers: 4
sos: ${model_conf.sos}
eos: ${model_conf.eos}
IndexDSJsonl: IndexDSJsonl
retry: 20
train_conf:
accum_grad: 1
grad_clip: 5
max_epoch: 20
keep_nbest_models: 10
avg_nbest_model: 10
log_interval: 100
resume: true
validate_interval: 10000
save_checkpoint_interval: 10000
optim: adamw
optim_conf:
lr: 0.00002
Scheduler: warmuplr
Scheduler_conf:
Warmup_steps: 25000
```
Model parameters: model.pt
Path resolution: configuration.json (not required)
```json
{
"framework": "pytorch",
"task" : "auto-speech-recognition",
"model": {"type" : "funasr"},
"pipeline": {"type":"funasr-pipeline"},
"model_name_in_hub": {
"ms":"",
"hf":""},
"file_path_metas": {
"init_param":"model.pt",
"config":"config.yaml",
"tokenizer_conf": {"bpemodel": "chn_jpn_yue_eng_ko_spectok.bpe.model"},
"frontend_conf":{"cmvn_file": "am.mvn"}}
}
```
The function of configuration.json is to add the model root directory to the item in file\_path\_metas, so that the path can be correctly parsed. For example, assume that the model root directory is:/home/zhifu.gzf/init\_model/SenseVoiceSmall,The relevant path in config.yaml in the directory is replaced with the correct path (ignoring irrelevant configuration):
```yaml
init_param: /home/zhifu.gz F/init_model/sensevoicemail Mall/model.pt
tokenizer_conf:
bpemodel: /home/Zhifu.gzf/init_model/SenseVoiceSmall/chn_jpn_yue_eng_ko_spectok.bpe.model
frontend_conf:
cmvn_file: /home/zhifu.Gzf/init_model/SenseVoiceSmall/am.mvn
```
## Registry
![image](https://alidocs.oss-cn-zhangjiakou.aliyuncs.com/a/pDaAnLxn5IX2w9Y1/73da157edae94d78b68c8d30c8e085eb0521.png)
### View Registry
```plaintext
from funasr.register import tables
tables.print()
```
Support to view the specified type of Registry: 'tables.print("model")', currently funasr has registered model as shown in the figure above. The following categories are currently predefined:
```python
model_classes = {}
frontend_classes = {}
specaug_classes = {}
normalize_classes = {}
encoder_classes = {}
decoder_classes = {}
joint_network_classes = {}
predictor_classes = {}
stride_conv_classes = {}
tokenizer_classes = {}
dataloader_classes = {}
batch_sampler_classes = {}
dataset_classes = {}
index_ds_classes = {}
```
### Registration Model
```python
from funasr.register import tables
@tables.register("model_classes", "SenseVoiceSmall")
class SenseVoiceSmall(nn.Module):
def __init__(*args, **kwargs):
...
def forward(
self,
**kwargs,
):
def inference(
self,
data_in,
data_lengths=None,
key: list = None,
tokenizer=None,
frontend=None,
**kwargs,
):
...
```
Add @ tables.register("model\_classes", "SenseVoiceSmall") before the name of the class to be registered. The class needs to implement the following methods:\_\_init \_\_, forward, and inference.
register Usage:
```python
@ tables.register("registration classification", "registration name")
```
Among them, "registration classification" can be a predefined classification (see the figure above). If it is a new classification defined by oneself, the new classification will be automatically written into the registry classification. "registration name" means the name you want to register and can be used directly in the future.
Full code:[https://github.com/modelscope/FunASR/blob/main/funasr/models/sense\_voice/model.py#L443](https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L443)
After the registration is complete, specify the new registration model in config.yaml to define the model.
```python
model: SenseVoiceSmall
model_conf:
...
```
### Registration failed
If the registration model or method is not found, assert model\_class is not None, f'{kwargs\["model"\]} is not registered '. The principle of model registration is to import the model file,You can view the specific reason for the registration failure through import. For example, the preceding model file is funasr/models/sense\_voice/model.py:
```python
from funasr.models.sense_voice.model import *
```
## Principles of Registration
* Model: models are independent of each other. Each Model needs to create a new Model directory under funasr/models/. Do not use class inheritance method!!! Do not import from other model directories, and put everything you need into your own model directory!!! Do not modify the existing model code!!!
* dataset,frontend,tokenizer, if you can reuse the existing one, reuse it directly, if you cannot reuse it, please register a new one, modify it again, and do not modify the original one!!!
# Independent warehouse
It can exist as a stand-alone repository for code secrecy, or as a stand-alone open source. Based on the registration mechanism, you do not need to integrate it into funasr. You can also use funasr for inference, and you can also directly perform inference. finetune is also supported.
**Using AutoModel for inference**
```python
from funasr import AutoModel
# trust_remote_code:'True' means that the model code implementation is loaded from 'remote_code', 'remote_code' specifies the location of the 'model' specific code (for example,'model.py') in the current directory, supports absolute and relative paths, and network url.
model = AutoModel (
model="iic/SenseVoiceSmall ",
trust_remote_code=True
remote_code = "./model.py",
)
```
**Direct inference**
```python
from model import SenseVoiceSmall
m, kwargs = SenseVoiceSmall.from_pretrained(model="iic/SenseVoiceSmall")
m.eval()
res = m.inference(
data_in=f"{kwargs ['model_path']}/example/en.mp3",
language="auto", # "zh", "en", "yue", "ja", "ko", "nospeech"
use_itn=False,
ban_emo_unk=False,
**kwargs,
)
print(text)
```
Trim reference:[https://github.com/FunAudioLLM/SenseVoice/blob/main/finetune.sh](https://github.com/FunAudioLLM/SenseVoice/blob/main/finetune.sh)
docs/tutorial/Tables_zh.md
+363
View File
@@ -0,0 +1,363 @@
# FunASR-1.x.x 注册新模型教程
(简体中文|[English](./Tables.md))
funasr-1.x.x 版本的设计初衷是【**让模型集成更简单**】，核心feature为注册表与AutoModel：
* 注册表的引入，使得开发中可以用搭积木的方式接入模型，兼容多种task；
* 新设计的AutoModel接口，统一modelscope、huggingface与funasr推理与训练接口，支持自由选择下载仓库；
* 支持模型导出，demo级别服务部署，以及工业级别多并发服务部署；
* 统一学术与工业模型推理训练脚本；
# 快速上手
## 基于automodel用法
### SenseVoiceSmall模型
输入任意时长语音，输出为语音内容对应文字，文字具有标点断句，支持中英日粤韩5中语言。【字级别时间戳，以及说话人身份】后续会支持。
```python
from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
vad_kwargs={"max_single_segment_time": 30000},
device="cuda:0",
)
res = model.generate(
input=f"{model.model_path}/example/en.mp3",
language="auto", # "zn", "en", "yue", "ja", "ko", "nospeech"
use_itn=True,
batch_size_s=60,
)
text = rich_transcription_postprocess(res[0]["text"])
print(text) #👏Senior staff, Priipal Doris Jackson, Wakefield faculty, and, of course, my fellow classmates.I am honored to have been chosen to speak before my classmates, as well as the students across America today.
```
## API文档
#### AutoModel 定义
```plaintext
model = AutoModel(model=[str], device=[str], ncpu=[int], output_dir=[str], batch_size=[int], hub=[str], **kwargs)
```
* `model`(str): [模型仓库](https://github.com/modelscope/FunASR/tree/main/model_zoo) 中的模型名称，或本地磁盘中的模型路径
* `device`(str): `cuda:0`（默认gpu0），使用 GPU 进行推理，指定。如果为`cpu`，则使用 CPU 进行推理
* `ncpu`(int): `4` （默认），设置用于 CPU 内部操作并行性的线程数
* `output_dir`(str): `None` （默认），如果设置，输出结果的输出路径
* `batch_size`(int): `1` （默认），解码时的批处理，样本个数
* `hub`(str)：`ms`（默认），从modelscope下载模型。如果为`hf`，从huggingface下载模型。
* `**kwargs`(dict): 所有在`config.yaml`中参数，均可以直接在此处指定，例如，vad模型中最大切割长度 `max_single_segment_time=6000` （毫秒）。
#### AutoModel 推理
```plaintext
res = model.generate(input=[str], output_dir=[str])
```
* * wav文件路径, 例如: asr\_example.wav
* pcm文件路径, 例如: asr\_example.pcm，此时需要指定音频采样率fs（默认为16000）
* 音频字节数流，例如：麦克风的字节数数据
* wav.scp，kaldi 样式的 wav 列表 (`wav_id \t wav_path`), 例如:
```plaintext
asr_example1 ./audios/asr_example1.wav
asr_example2 ./audios/asr_example2.wav
```
在这种输入 
* 音频采样点，例如：`audio, rate = soundfile.read("asr_example_zh.wav")`, 数据类型为 numpy.ndarray。支持batch输入，类型为list： `[audio_sample1, audio_sample2, ..., audio_sampleN]`
* fbank输入，支持组batch。shape为\[batch, frames, dim\]，类型为torch.Tensor，例如
* `output_dir`: None （默认），如果设置，输出结果的输出路径
* `**kwargs`(dict): 与模型相关的推理参数，例如，`beam_size=10`，`decoding_ctc_weight=0.1`。
详细文档链接：[https://github.com/modelscope/FunASR/blob/main/examples/README\_zh.md](https://github.com/modelscope/FunASR/blob/main/examples/README_zh.md)
# 注册表详解
以SenseVoiceSmall模型为例，讲解如何注册新模型，模型链接：
**modelscope：**[https://www.modelscope.cn/models/iic/SenseVoiceSmall/files](https://www.modelscope.cn/models/iic/SenseVoiceSmall/files)
**huggingface：**[https://huggingface.co/FunAudioLLM/SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)
## 模型资源目录
![image.png](https://alidocs.oss-cn-zhangjiakou.aliyuncs.com/res/8oLl9y628rBNlapY/img/cab7f215-787f-4407-885a-14dc89ae9e02.png)
**配置文件**：config.yaml
```yaml
encoder: SenseVoiceEncoderSmall
encoder_conf:
output_size: 512
attention_heads: 4
linear_units: 2048
num_blocks: 50
tp_blocks: 20
dropout_rate: 0.1
positional_dropout_rate: 0.1
attention_dropout_rate: 0.1
input_layer: pe
pos_enc_class: SinusoidalPositionEncoder
normalize_before: true
kernel_size: 11
sanm_shfit: 0
selfattention_layer_type: sanm
model: SenseVoiceSmall
model_conf:
length_normalized_loss: true
sos: 1
eos: 2
ignore_id: -1
tokenizer: SentencepiecesTokenizer
tokenizer_conf:
bpemodel: null
unk_symbol: <unk>
split_with_space: true
frontend: WavFrontend
frontend_conf:
fs: 16000
window: hamming
n_mels: 80
frame_length: 25
frame_shift: 10
lfr_m: 7
lfr_n: 6
cmvn_file: null
dataset: SenseVoiceCTCDataset
dataset_conf:
index_ds: IndexDSJsonl
batch_sampler: EspnetStyleBatchSampler
data_split_num: 32
batch_type: token
batch_size: 14000
max_token_length: 2000
min_token_length: 60
max_source_length: 2000
min_source_length: 60
max_target_length: 200
min_target_length: 0
shuffle: true
num_workers: 4
sos: ${model_conf.sos}
eos: ${model_conf.eos}
IndexDSJsonl: IndexDSJsonl
retry: 20
train_conf:
accum_grad: 1
grad_clip: 5
max_epoch: 20
keep_nbest_models: 10
avg_nbest_model: 10
log_interval: 100
resume: true
validate_interval: 10000
save_checkpoint_interval: 10000
optim: adamw
optim_conf:
lr: 0.00002
scheduler: warmuplr
scheduler_conf:
warmup_steps: 25000
```
**模型参数**：model.pt
**路径解析**：configuration.json（非必需）
```json
{
"framework": "pytorch",
"task" : "auto-speech-recognition",
"model": {"type" : "funasr"},
"pipeline": {"type":"funasr-pipeline"},
"model_name_in_hub": {
"ms":"",
"hf":""},
"file_path_metas": {
"init_param":"model.pt",
"config":"config.yaml",
"tokenizer_conf": {"bpemodel": "chn_jpn_yue_eng_ko_spectok.bpe.model"},
"frontend_conf":{"cmvn_file": "am.mvn"}}
}
```
configuration.json的作用是给file\_path\_metas中的item拼接上模型根目录，以便于路径能够被正确的解析，以上为例，假设模型根目录为：/home/zhifu.gzf/init\_model/SenseVoiceSmall，目录中config.yaml中的相关路径被替换成了正确的路径（忽略无关配置）：
```yaml
init_param: /home/zhifu.gzf/init_model/SenseVoiceSmall/model.pt
tokenizer_conf:
bpemodel: /home/zhifu.gzf/init_model/SenseVoiceSmall/chn_jpn_yue_eng_ko_spectok.bpe.model
frontend_conf:
cmvn_file: /home/zhifu.gzf/init_model/SenseVoiceSmall/am.mvn
```
## 注册表
![image](https://alidocs.oss-cn-zhangjiakou.aliyuncs.com/a/6Ea1DxkZVte8y0g2/c92059e82c38493988fbc8c032d3f5380521.png)
### 查看注册表
```plaintext
from funasr.register import tables
tables.print()
```
支持查看指定类型的注册表：\`tables.print("model")\`，目前funasr已经注册模型如上图所示。目前预先定义了如下几个分类：
```python
model_classes = {}
frontend_classes = {}
specaug_classes = {}
normalize_classes = {}
encoder_classes = {}
decoder_classes = {}
joint_network_classes = {}
predictor_classes = {}
stride_conv_classes = {}
tokenizer_classes = {}
dataloader_classes = {}
batch_sampler_classes = {}
dataset_classes = {}
index_ds_classes = {}
```
### 注册模型
```python
from funasr.register import tables
@tables.register("model_classes", "SenseVoiceSmall")
class SenseVoiceSmall(nn.Module):
def __init__(*args, **kwargs):
...
def forward(
self,
**kwargs,
):
def inference(
self,
data_in,
data_lengths=None,
key: list = None,
tokenizer=None,
frontend=None,
**kwargs,
):
...
```
在需要注册的类名前加上 @tables.register("model\_classes", "SenseVoiceSmall")，即可完成注册，类需要实现有：\_\_init\_\_，forward，inference方法。
register用法：
```python
@tables.register("注册分类", "注册名")
```
其中，"注册分类"可以是预先定义好的分类（见上面图），如果是自己定义的新分类，会自动将新分类写进注册表分类中，"注册名"即希望注册名字，后续可以直接来使用。
完整代码：[https://github.com/modelscope/FunASR/blob/main/funasr/models/sense\_voice/model.py#L443](https://github.com/modelscope/FunASR/blob/main/funasr/models/sense_voice/model.py#L443)
注册完成后，在config.yaml中指定新注册模型，即可实现对模型的定义
```python
model: SenseVoiceSmall
model_conf:
...
```
### 注册失败
如果出现找不到注册模型或发方法，assert model\_class is not None, f'{kwargs\["model"\]} is not registered'。模型注册的原理是，import 模型文件，可以通过import来查看具体注册失败原因，例如，上述模型文件为，funasr/models/sense\_voice/model.py：
```python
from funasr.models.sense_voice.model import *
```
## 注册原则
* Model：模型之间互相独立，每一个模型，都需要在funasr/models/下面新建一个模型目录，不要采用类的继承方法！！！不要从其他模型目录中import，所有需要用到的都单独放到自己的模型目录中！！！不要修改现有的模型代码！！！
* dataset，frontend，tokenizer，如果能复用现有的，直接复用，如果不能复用，请注册一个新的，再修改，不要修改原来的！！！
# 独立仓库
可以作为独立仓库存在，用于代码保密，或者独立开源。基于注册机制，无需集成到funasr中，使用funasr进行推理，也可以直接进行推理，同样支持finetune
**使用AutoModel进行推理**
```python
from funasr import AutoModel
# trust_remote_code：`True` 表示 model 代码实现从 `remote_code` 处加载，`remote_code` 指定 `model` 具体代码的位置（例如，当前目录下的 `model.py`），支持绝对路径与相对路径，以及网络 url。
model = AutoModel(
model="iic/SenseVoiceSmall",
trust_remote_code=True,
remote_code="./model.py",
)
```
**直接进行推理**
```python
from model import SenseVoiceSmall
m, kwargs = SenseVoiceSmall.from_pretrained(model="iic/SenseVoiceSmall")
m.eval()
res = m.inference(
data_in=f"{kwargs ['model_path']}/example/en.mp3",
language="auto", # "zh", "en", "yue", "ja", "ko", "nospeech"
use_itn=False,
ban_emo_unk=False,
**kwargs,
)
print(text)
```
微调参考：[https://github.com/FunAudioLLM/SenseVoice/blob/main/finetune.sh](https://github.com/FunAudioLLM/SenseVoice/blob/main/finetune.sh)
docs/use_case_showcase.md
+94
View File
@@ -0,0 +1,94 @@
# FunASR Use-case Showcase
FunASR is useful far beyond a single offline transcription command. This page collects the fastest paths for developers who want to evaluate, deploy, or integrate speech understanding in real products.
## Choose the right path
| Goal | Start here | Why it matters |
|---|---|---|
| Try FunASR in a browser | [Colab quickstart](../examples/colab/) | Run a public sample and upload your own audio before setting up a local environment. |
| Transcribe one file locally | [README quick start](../README.md#quick-start) and [model selection guide](./model_selection.md) | Verify install, model choice, and model download in minutes. |
| Compare accuracy and speed | [Benchmark report](https://modelscope.github.io/FunASR/benchmark.html) | Reproduce the 184-file long-audio benchmark before choosing a model. |
| Migrate from Whisper/cloud ASR | [Migration guide](./migration_from_whisper.md) | Map existing pipelines to FunASR, benchmark representative audio, and plan a safe rollout. |
| Build a private speech API | [OpenAI-compatible API example](../examples/openai_api/), [Gradio browser demo](../examples/openai_api/GRADIO.md), [client recipes](../examples/openai_api/CLIENTS.md), [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md), and [workflow recipes](../examples/openai_api/WORKFLOWS.md) | Reuse LangChain, Dify, n8n, AutoGen, and other OpenAI-style clients without sending audio to a cloud ASR provider. |
| Add speech input to agents | [MCP server](../examples/mcp_server/) and [voice input](../examples/voice_input/) | Connect local ASR to Claude, Cursor, and desktop agent workflows. |
| Choose a deployment path | [Deployment matrix](./deployment_matrix.md) | Compare Python API, OpenAI API, Docker Compose, Kubernetes, WebSocket, vLLM, MCP, batch, subtitles, and Triton. |
| Serve streaming ASR | [Runtime service docs](../runtime/readme.md) | Run WebSocket or service-mode ASR for live captioning and call-center style workloads. |
| Accelerate LLM-based ASR | [vLLM guide](./vllm_guide.md) | Use tensor parallel decoding and streaming service support for Fun-ASR-Nano. |
| Generate subtitles | [Subtitle example](../examples/subtitle/) | Turn long audio or video into subtitle files for media workflows. |
| Process many recordings | [Batch ASR example](../examples/batch_asr_improved.py) | Build repeatable offline jobs for archives, meetings, and datasets. |
## Production-oriented recipes
### Private transcription API
Use this path when an application already speaks OpenAI-style APIs or when audio cannot leave your environment.
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
```
```bash
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
Recommended next steps:
- Run the [OpenAI-compatible API smoke test](../examples/openai_api/smoke_test.sh) or the cross-platform [Python smoke test](../examples/openai_api/smoke_test.py).
- For browser upload or microphone demos, start from the [Gradio browser demo](../examples/openai_api/GRADIO.md).
- For Node.js or Next.js services, start from the [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md).
- For cluster services, start from the [Kubernetes deployment template](../examples/openai_api/kubernetes/).
- Add authentication and network controls at your service boundary; start from the [security and gateway guide](../examples/openai_api/SECURITY.md).
- Record model name, device, driver, and audio duration in bug reports and benchmarks.
### Agent speech input
Use this path when you want to talk to coding agents, internal assistants, or workflow tools.
- Start with the [MCP server example](../examples/mcp_server/) for Claude/Cursor-style tools.
- Use the [voice input example](../examples/voice_input/) for desktop speech input experiments.
- Keep latency visible: log audio duration, processing time, and selected model for each request.
### Streaming and call-center workloads
Use this path when partial results and low perceived latency matter more than a single final transcript.
- Start from the [runtime service docs](../runtime/readme.md).
- Pair ASR with VAD, punctuation, and speaker diarization when the transcript needs to be readable by humans.
- Validate with realistic audio: background noise, long silence, overlapping speakers, and different microphone quality.
### Benchmark before migrating from Whisper
Use this path when deciding whether FunASR is a good replacement for Whisper or a cloud ASR provider.
- Follow the [migration guide](./migration_from_whisper.md) to map features and benchmark representative audio.
- Read the [public benchmark report](https://modelscope.github.io/FunASR/benchmark.html).
- Benchmark your own sample set before migration; include both short clips and long-form recordings.
- Track cost and throughput together: GPU speed, CPU viability, model download size, and deployment complexity.
## Model selection hints
For a deeper comparison of SenseVoice, Paraformer, Fun-ASR-Nano, streaming runtime, and OpenAI API aliases, use the [model selection guide](./model_selection.md).
| Need | Good first choice | Notes |
|---|---|---|
| Fast multilingual transcription | SenseVoice-Small | Strong default for local demos and private APIs. |
| Mandarin production ASR | Paraformer-Large | Mature choice for Chinese speech recognition. |
| LLM-based ASR experiments | Fun-ASR-Nano | Pair with the [vLLM guide](./vllm_guide.md) when throughput matters. |
| Speaker-aware transcripts | SenseVoice or Paraformer with `spk_model="cam++"` | Useful for meetings, interviews, and customer calls. |
| Live audio | Runtime WebSocket service | Validate chunking, VAD, and endpointing with real traffic. |
## Share your result
If FunASR works well in your project, consider opening a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md), [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md), or GitHub Discussion with:
- Use case and deployment mode.
- Model, device, and processing speed.
- Audio domain, language, and rough duration.
- A public demo, screenshot, benchmark summary, or integration link when available.
Concrete usage reports help new users choose the right path and help maintainers prioritize the next round of docs and examples.
docs/use_case_showcase_zh.md
+94
View File
@@ -0,0 +1,94 @@
# FunASR 场景速览
FunASR 不只是一个离线转写命令。这个页面把常见的评测、部署、Agent 集成和生产场景整理到一起，方便新用户快速找到入口。
## 选择合适路径
| 目标 | 从这里开始 | 为什么重要 |
|---|---|---|
| 在浏览器里试用 FunASR | [Colab 快速体验](../examples/colab/README_zh.md) | 配置本地环境前，先跑公开样例并上传自己的音频。 |
| 本地转写一个文件 | [README 快速开始](../README_zh.md#快速开始) 和 [模型选择指南](./model_selection_zh.md) | 几分钟内验证安装、模型选择、模型下载和首次推理。 |
| 对比准确率和速度 | [性能评测报告](https://modelscope.github.io/FunASR/zh/benchmark.html) | 选型前先查看 184 条长音频评测结果。 |
| 从 Whisper/云端 ASR 迁移 | [迁移指南](./migration_from_whisper_zh.md) | 将现有流水线映射到 FunASR，用代表性音频评测并规划安全上线。 |
| 搭建私有语音 API | [OpenAI 兼容 API 示例](../examples/openai_api/README_zh.md)、[Gradio 浏览器 Demo](../examples/openai_api/GRADIO_zh.md)、[客户端配方](../examples/openai_api/CLIENTS.md)、[JavaScript/TypeScript 配方](../examples/openai_api/JAVASCRIPT_zh.md) 和 [工作流配方](../examples/openai_api/WORKFLOWS_zh.md) | 复用 LangChain、Dify、n8n、AutoGen 等 OpenAI 风格客户端，音频不出内网。 |
| 给 Agent 增加语音输入 | [MCP 服务](../examples/mcp_server/) 和 [语音输入示例](../examples/voice_input/) | 将本地 ASR 接入 Claude、Cursor 和桌面 Agent 工作流。 |
| 选择部署路径 | [部署选型表](./deployment_matrix_zh.md) | 对比 Python API、OpenAI API、Docker Compose、Kubernetes、WebSocket、vLLM、MCP、批处理、字幕和 Triton。 |
| 部署流式 ASR | [Runtime 服务文档](../runtime/readme_cn.md) | 面向实时字幕、客服、会议等低延迟场景。 |
| 加速 LLM-based ASR | [vLLM 指南](./vllm_guide.md) | 为 Fun-ASR-Nano 使用 tensor parallel 解码和流式服务能力。 |
| 生成字幕 | [字幕示例](../examples/subtitle/) | 将长音频或视频转成字幕文件。 |
| 批量处理录音 | [批处理示例](../examples/batch_asr_improved.py) | 为录音归档、会议纪要、数据集处理搭建可重复流水线。 |
## 面向生产的配方
### 私有转写 API
当应用已经兼容 OpenAI 风格接口，或音频不能离开私有环境时，优先使用这个路径。
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
```
```bash
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
建议下一步：
- 运行 [OpenAI 兼容 API smoke test](../examples/openai_api/smoke_test.sh) 或跨平台 [Python smoke test](../examples/openai_api/smoke_test.py)。
- 浏览器上传或麦克风 demo 可从 [Gradio 浏览器 Demo](../examples/openai_api/GRADIO_zh.md) 开始。
- Node.js 或 Next.js 服务可从 [JavaScript/TypeScript 配方](../examples/openai_api/JAVASCRIPT_zh.md) 开始。
- 集群内服务可从 [Kubernetes 部署模板](../examples/openai_api/kubernetes/README_zh.md) 开始。
- 在服务边界增加鉴权、限流和网络访问控制；可从 [安全与网关指南](../examples/openai_api/SECURITY_zh.md) 开始。
- 记录模型、设备、驱动、音频时长和处理耗时，便于复现问题和 benchmark。
### Agent 语音输入
当你想把语音输入接到编码助手、内部助手或工作流工具时，使用这个路径。
- Claude/Cursor 类工具优先看 [MCP 服务示例](../examples/mcp_server/)。
- 桌面语音输入实验可以从 [voice input 示例](../examples/voice_input/) 开始。
- 保持延迟可见：每次请求记录音频时长、处理耗时和模型名称。
### 流式与客服场景
当你更关注低延迟和中间结果，而不是单次完整转写时，使用这个路径。
- 从 [Runtime 服务文档](../runtime/readme_cn.md) 开始。
- 需要给人阅读时，把 ASR 与 VAD、标点恢复、说话人分离一起使用。
- 用真实音频验证：背景噪声、长静音、多人重叠、不同麦克风质量。
### 从 Whisper 迁移前先评测
当你在评估是否用 FunASR 替代 Whisper 或云端 ASR 时，使用这个路径。
- 按 [迁移指南](./migration_from_whisper_zh.md) 映射功能并评测代表性音频。
- 阅读 [公开性能评测](https://modelscope.github.io/FunASR/zh/benchmark.html)。
- 用自己的样本集再测一次；同时包含短音频和长音频。
- 同时记录成本和吞吐：GPU 速度、CPU 可用性、模型下载体积、部署复杂度。
## 模型选择建议
如需更完整地比较 SenseVoice、Paraformer、Fun-ASR-Nano、streaming runtime 和 OpenAI API alias，请看 [模型选择指南](./model_selection_zh.md)。
| 需求 | 推荐先试 | 说明 |
|---|---|---|
| 快速多语种转写 | SenseVoice-Small | 本地 demo 和私有 API 的稳妥默认选择。 |
| 中文生产 ASR | Paraformer-Large | 中文语音识别的成熟选择。 |
| LLM-based ASR 实验 | Fun-ASR-Nano | 吞吐敏感时配合 [vLLM 指南](./vllm_guide.md)。 |
| 带说话人信息的转写 | SenseVoice 或 Paraformer + `spk_model="cam++"` | 适合会议、访谈、客服录音。 |
| 实时音频 | Runtime WebSocket 服务 | 用真实流量验证分块、VAD 和断句。 |
## 分享你的结果
如果 FunASR 在你的项目里效果不错，欢迎通过 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)、[Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) 或 GitHub Discussion 分享：
- 使用场景和部署方式。
- 模型、设备和处理速度。
- 音频领域、语言和大致时长。
- 可以公开的 demo、截图、benchmark 摘要或集成链接。
具体的使用反馈能帮助新用户更快选型，也能帮助维护者决定下一批文档和示例优先级。
docs/vllm_guide.md
+819
View File
@@ -0,0 +1,819 @@
# FunASR vLLM Inference Engine Guide
---
## Benchmark
**Test set**: 184 files, 11,541 seconds total. Models: Fun-ASR-Nano / GLM-ASR-Nano. See [Benchmark RTF and Reproducibility Notes](./benchmark/rtf_reproducibility.md) for the `RTFx` definition, timing scope checklist, and fields required for comparable reports.
| Model | Engine | VAD | RTFx | CER | Notes |
|-------|--------|-----|------|-----|-------|
| Fun-ASR-Nano | PyTorch | dynamic | 21 | 8.06% | Baseline |
| Fun-ASR-Nano | **vLLM batch** | dynamic | **340** | **8.20%** | 16x speedup |
| Fun-ASR-Nano | **Offline service (no SPK)** | dynamic | **102** | 8.14% | |
| Fun-ASR-Nano | **Offline service (+SPK)** | dynamic | **46** | 8.19% | SPK off by default |
| GLM-ASR-Nano | **vLLM batch** | fixed | **265** | 12.93% | No long-audio support |
> vLLM matches PyTorch CER exactly (delta < 0.2%) while achieving 16–340x speedup.
---
## Table of Contents
1. [Installation & Environment](#1-installation--environment)
2. [vLLM Engine Architecture](#2-vllm-engine-architecture)
3. [Offline SDK Inference](#3-offline-sdk-inference)
4. [Streaming SDK Inference](#4-streaming-sdk-inference)
5. [Offline Speech Recognition Service](#5-offline-speech-recognition-service)
6. [Streaming Speech Recognition Service](#6-streaming-speech-recognition-service)
7. [Dynamic VAD](#7-dynamic-vad)
8. [API Reference](#8-api-reference)
9. [FAQ](#9-faq)
---
## 1. Installation & Environment
Install vLLM first, choosing a version compatible with your NVIDIA driver's CUDA. vLLM pins and installs a matching torch / torchaudio / torchvision trio automatically, so do not install torch/torchaudio yourself — the three are ABI-locked, i.e. they must be the matching set built against each other (e.g. torch 2.10.0 ↔ torchaudio 2.10.0 ↔ torchvision 0.25.0).
```bash
# 1) Install vLLM first. Pick the version by the CUDA version shown in `nvidia-smi`
# (the driver's max CUDA), NOT the runtime CUDA. vLLM brings a matching torch/torchaudio/torchvision.
# driver CUDA 12.x -> pip install vllm==0.19.1 (ships torch 2.10 / cu128)
# driver CUDA >= 13 -> pip install vllm (latest; ships torch 2.11 / cu130)
pip install "vllm==0.19.1" # adjust to your driver CUDA; see note below
# 2) Then FunASR and the rest.
pip install funasr>=1.3.0
cd /path/to/FunASR && pip install -e .
```
**Hardware**: GPU ≥ 8 GB VRAM, CUDA ≥ 11.8. 16 GB+ recommended.
Why not pip install torch torchaudio? The torch/torchaudio/torchvision versions are determined by the vLLM release — each major vLLM version bumps them together (see vLLM's requirements/cuda.txt). Installing them by hand pulls the newest wheel, which may be built for a newer CUDA runtime than your driver supports; PyTorch then fails during CUDA initialization with The NVIDIA driver on your system is too old before FunASR even starts. Letting vLLM own the trio avoids this. If you still hit a driver-too-old error, install a vLLM version whose CUDA build matches the CUDA reported by nvidia-smi (e.g. vllm==0.19.1 for CUDA 12.x), or update the NVIDIA driver first.
---
## 2. vLLM Engine Architecture
### Overall Architecture
FunASR's vLLM integration splits the ASR model into two independently running components:
```
┌──────────────────────────────────────────────────────────────┐
│ FunASR + vLLM Inference Architecture │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────── PyTorch (single GPU) ───────────┐ │
│ │ │ │
│ │ Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor │
│ │ (fbank) (SenseVoice/ (Transformer/ │
│ │ Whisper) MLP) │
│ │ │ │
│ │ ▼ │
│ │ Audio Embeddings │
│ │ │ │
│ │ Text Prompt ──→ Tokenize ──→ Embed │
│ │ (system/user/ │ │
│ │ hotwords/language) │ │
│ │ ▼ │
│ │ [Concat Embeddings] │
│ └─────────────────────────────────┼─────────────┘ │
│ │ │
│ ▼ EmbedsPrompt │
│ ┌─────────────── vLLM Engine ────────────────────┐ │
│ │ │ │
│ │ PagedAttention + Continuous Batching │ │
│ │ KV Cache management + CUDA Graph │ │
│ │ Tensor Parallel (multi-GPU) │ │
│ │ │ │
│ │ Qwen3-0.6B / Llama-2B (LLM decoding) │ │
│ │ │ │
│ └────────────────────┬───────────────────────────┘ │
│ │ │
│ ▼ │
│ Generated Text │
│ │ │
│ ┌────────────────────┼──────────────────────────┐ │
│ │ (Optional) CTC Decoder ──→ Forced Alignment │ │
│ │ ──→ Character-level timestamps │ │
│ └───────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### Why vLLM?
| Feature | PyTorch generate() | vLLM |
|---------|-------------------|------|
| KV Cache management | Fixed allocation, wastes memory | PagedAttention, on-demand allocation |
| Batching | Manual padding required | Continuous Batching, automatic scheduling |
| CUDA optimization | None | CUDA Graph + operator fusion |
| Multi-GPU parallelism | Manual implementation | Tensor Parallel with one-line config |
| Throughput | RTFx ~20 | **RTFx 340+** |
### Supported Models
| Model | LLM component | Audio encoder | vLLM speedup |
|-------|--------------|---------------|-------------|
| **Fun-ASR-Nano** | Qwen3-0.6B | SenseVoice | ✓ 21.7x |
| **GLM-ASR-Nano** | Llama-2B | Whisper-like | ✓ 7.6x |
| LLMASR | Qwen/Vicuna | Whisper | ✓ |
| Paraformer | No LLM | — | ✗ Non-autoregressive |
| SenseVoice | No LLM | — | ✗ Encoder-decoder |
### Key Implementation Details
1. **Weight separation**: LLM weights are extracted from `model.pt` and converted to HuggingFace format for vLLM loading
2. **EmbedsPrompt**: a vLLM input mode that feeds **precomputed embedding vectors** (rather than the usual token IDs) directly as the prompt (enabled via `enable_prompt_embeds=True`). Fun-ASR-Nano requires it because the audio, after the adaptor, is a sequence of continuous vectors — not tokens — so the audio embeddings and text embeddings are concatenated along the sequence dimension and submitted to vLLM as a whole
3. **use_low_frame_rate**: Fun-ASR-Nano's adaptor output must be truncated to the correct token count via a formula (critical for consistency)
4. **Batch encode**: Multiple audio files pass through `extract_fbank` → `audio_encoder` → `audio_adaptor` in a single forward pass
5. **CTC timestamps**: Encoder output is retained; after text generation, forced alignment yields character-level timing
---
## 3. Offline SDK Inference
Best suited for large-scale audio transcription and offline batch processing. vLLM's batching capability provides the greatest advantage in this scenario.
### Design Principles
Offline SDK inference splits the ASR pipeline into two stages executed independently:
```
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 1: Audio Encoding (PyTorch, single GPU) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Audio file list ──→ Group (batch of 8) ──→ Frontend (Fbank) │
│ │ │ │
│ │ ▼ │
│ │ SenseVoice Encoder │
│ │ │ │
│ │ ▼ │
│ │ Audio Adaptor │
│ │ (dim transform + LFR trunc) │
│ │ │ │
│ └─── Shared text prompt encoding ────┐ ▼ │
│ (system/hotwords/language) │ audio_embeds │
│ │ │ │ │
│ ▼ │ ▼ │
│ prefix_emb ──→ [concat: prefix | audio | suffix] │
│ │ │
│ ▼ │
│ EmbedsPrompt (N samples) │
└──────────────────────────────────────────────────┼─────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Stage 2: LLM Decoding (vLLM, multi-GPU Tensor Parallel) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ EmbedsPrompt × N ──→ vLLM Continuous Batching │
│ (PagedAttention + CUDA Graph) │
│ │ │
│ ▼ │
│ Generated token_ids × N │
│ │ │
│ ▼ │
│ Decode + post-processing (strip special tokens) │
│ │ │
│ ▼ │
│ (Optional) CTC Forced Alignment → char timestamps│
└─────────────────────────────────────────────────────────────────────┘
```
**Key design decisions:**
1. **Weight separation**: On first run, weights with the `llm.*` prefix are extracted from `model.pt` and saved in HuggingFace safetensors format for vLLM (cached in the `Qwen3-0.6B-vllm/` directory)
2. **Embedding concatenation**: The text prompt is encoded through the LLM's `embed_tokens` layer into embeddings, then concatenated with the audio adaptor output along the sequence dimension: `[prefix_emb | audio_emb | suffix_emb]`, and submitted to vLLM as an `EmbedsPrompt`
3. **Low Frame Rate truncation**: Adaptor output must be truncated to the correct length using: `fake_token_len = ((((fbank_len - 3 + 2) // 2 - 3 + 2) // 2) - 1) // 2 + 1`, ensuring consistency with the PyTorch training pipeline
4. **Batch audio encoding**: Multiple audio files are grouped in batches of 8 through the encoder + adaptor forward pass, reducing GPU kernel launch overhead
5. **Shared text prompt**: When hotwords and language are identical within a batch, prefix_emb and suffix_emb are computed only once
6. **CTC timestamps**: Encoder output is preserved; after LLM text generation, forced alignment produces character-level timestamps
**Why faster than PyTorch generate()?**
| Dimension | PyTorch | vLLM |
|-----------|---------|------|
| KV Cache | Fixed pre-allocation (wastes memory) | PagedAttention on-demand allocation |
| Batching | Manual padding alignment | Continuous Batching auto-scheduling |
| CUDA | Sequential per-sample execution | CUDA Graph + operator fusion |
| Multi-GPU | Manual implementation | Tensor Parallel one-line config |
| Result | RTFx ~20 | **RTFx 340+** (16x speedup) |
### Universal Interface (Recommended)
```python
from funasr.auto.auto_model_vllm import AutoModelVLLM
model = AutoModelVLLM(
model="FunAudioLLM/Fun-ASR-Nano-2512",
hub="ms", # or "hf"
tensor_parallel_size=2, # multi-GPU parallel
gpu_memory_utilization=0.8,
)
results = model.generate(
["audio1.wav", "audio2.wav"],
language="中文",
hotwords=["张三", "北京"],
)
for r in results:
print(f"[{r['key']}] {r['text']}")
```
### Direct Interface
```python
from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
engine = FunASRNanoVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
tensor_parallel_size=4,
)
results = engine.generate(
inputs="wav.scp", # supports scp/jsonl/file lists
hotwords=["开放时间"],
language="中文",
max_new_tokens=512,
)
```
### Command Line
```bash
cd examples/industrial_data_pretraining/fun_asr_nano
# Single file
python demo_vllm.py --input audio.wav --language 中文
# Batch + multi-GPU
python demo_vllm.py --input wav.scp --tensor-parallel-size 4 --batch-size 32
# With hotwords + save results
python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.jsonl
```
---
## 4. Streaming SDK Inference
Processes audio in 720 ms chunks incrementally, outputting progressively stable recognition results. Suited for SDK-integrated real-time subtitle scenarios.
### Design Principles
```
Audio stream (720 ms chunks)
│ Cumulative re-encoding (each chunk covers all audio from the start)
▼
┌──────────────────────────┐
│ Stage 1: First 10 chunks │ ← No prev_text; batch generation
│ Identify stable output │
└──────────┬───────────────┘
▼
┌──────────────────────────┐
│ Stage 2: Subsequent │ ← Use stable output as prev_text
└──────────┬───────────────┘
▼
Each chunk: [fixed region (confirmed)] + [8-char unfixed (may change)]
```
### Usage
```python
from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
engine = FunASRNanoStreamingVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
chunk_ms=720,
rollback_chars=8,
)
for result in engine.streaming_generate("audio.wav", language="中文"):
if result["is_final"]:
print(f"Final: {result['text']}")
else:
print(f"[{result['audio_duration_ms']:.0f}ms] Confirmed: {result['fixed_text']}")
```
### Output Characteristics
| Accumulated audio | Output quality |
|-------------------|---------------|
| < 1.5 s | Empty or noise |
| 1.5–3.0 s | Partially correct |
| > 3.0 s | Accurate output |
> **Note: `repetition_penalty` cannot be used with EmbedsPrompt.** Here the prompt is a sequence of embedding vectors with no corresponding token IDs, whereas `repetition_penalty` needs the prompt's token IDs to down-weight already-seen tokens in the logits; applying it under EmbedsPrompt **indexes out of bounds and triggers a CUDA device-side assert**.
### Production API Stability Checklist
When wrapping `AutoModelVLLM` in a long-running API service, keep request state isolated and pin safe decoding defaults:
```python
common = dict(
language="auto",
temperature=0.0,
repetition_penalty=1.0,
max_new_tokens=200,
)
for _ in range(2):
results = model.generate(["vad_segment_01.wav", "vad_segment_02.wav"], **common)
print([r["text"] for r in results])
```
If the same audio is normal on the first request but repeats on the second request:
1. Run the minimal script above outside the API layer with the same VAD segments.
2. If the script is stable, check whether the API wrapper reuses per-request variables, previous VAD segment lists, previous `results`, or accumulated text across requests.
3. If the script also repeats, capture the exact `funasr`, `vllm`, and `torch` versions, plus the first and second outputs, before tuning any decoding parameter.
Do not increase `repetition_penalty` to suppress repeats on Fun-ASR-Nano vLLM. The prompt-embeds path should stay at the neutral value `1.0`.
---
## 5. Offline Speech Recognition Service
### 5.1 Service Architecture
```
Client serve_vllm.py
│ │
│── HTTP / OpenAI / WebSocket ─────────→│
│ │
│ ┌────┴────────────────────────┐
│ │ 1. Receive complete audio │
│ │ 2. Dynamic VAD (≤60 s/seg) │
│ │ 3. vLLM batch all segments │
│ │ 4. CTC timestamps (per-char)│
│ │ 5. Speaker diarization (opt)│
│ └────┬────────────────────────┘
│ │
│←── JSON result ───────────────────────│
```
**Characteristics**:
- Processes audio only after it arrives in full — ideal for file transcription
- Dynamic VAD preserves long segments (≤60 s), reducing boundary-cut losses
- Batch inference over all VAD segments maximizes throughput
- Automatically outputs character-level timestamps
- Speaker diarization is off by default; clients can enable it
### 5.2 Starting the Service
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_vllm.py \
--port 8899 \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--gpu-memory-utilization 0.5
```
> **About [`CUDA_VISIBLE_DEVICES`](https://docs.vllm.ai/en/v0.4.3/serving/env_vars.html)**: the `=0` in the examples is just an illustrative value ("use GPU 0"), **not a fixed requirement**. It selects which GPUs are visible to this process (indexed as in `nvidia-smi`), a single GPU machine does not need to set it.
>
> - **Single GPU**: small models like 0.6B / 1.7B can run several instances on one card — point multiple processes at the same GPU (e.g. all `=0`) sharing it via MPS, or split across cards with process A `=0`, B `=1` (see §6.7).
>
### 5.3 Protocol 1: HTTP REST — `POST /asr`
The most feature-complete interface, supporting speaker diarization, timestamps, and hotwords.
**Request**: `multipart/form-data`
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `file` | file | required | Audio file (wav/mp3/flac) |
| `language` | string | None | Language ("中文" / "English" / ...), None for auto |
| `hotwords` | string | "" | Hotwords, comma-separated |
| `spk` | bool | false | Enable speaker diarization |
| `timestamp` | bool | true | Output character-level timestamps |
**Response**:
```json
{
"text": "Full transcription text",
"segments": [
{
"text": "Segment text",
"start": 1.7,
"end": 14.8,
"speaker": "SPK0",
"words": [
{"word": "砸", "start": 2.02, "end": 2.08},
{"word": "了", "start": 2.26, "end": 2.32}
]
}
],
"duration": 227.4,
"processing_time": 3.422,
"rtf": 0.015
}
```
**Client examples**:
```bash
# cURL
curl -X POST http://localhost:8899/asr \
-F "file=@meeting.wav" -F "language=中文" -F "spk=true"
```
```python
# Python requests
import requests
resp = requests.post("http://localhost:8899/asr",
files={"file": open("audio.wav", "rb")},
data={"language": "中文", "spk": "true"})
result = resp.json()
```
```javascript
// JavaScript fetch
const form = new FormData();
form.append("file", audioBlob, "audio.wav");
form.append("language", "中文");
form.append("spk", "true");
const resp = await fetch("http://localhost:8899/asr", { method: "POST", body: form });
const result = await resp.json();
```
### 5.4 Protocol 2: OpenAI Whisper Compatible — `POST /v1/audio/transcriptions`
Compatible with the OpenAI Whisper API standard; works directly with the OpenAI SDK.
**Request**: `multipart/form-data`
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `file` | file | required | Audio file |
| `model` | string | "fun-asr-nano" | Model name (compatibility field) |
| `language` | string | None | Language |
| `response_format` | string | "json" | "json" / "text" / "verbose_json" |
| `timestamp_granularities` | string | "word" | "word" / "segment" |
| `spk` | bool | false | Speaker diarization (FunASR extension) |
**Response** (`verbose_json`):
```json
{
"task": "transcribe",
"language": "zh",
"duration": 5.17,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"segments": [
{
"id": 0, "start": 0.0, "end": 5.15,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"words": [{"word": "我", "start": 0.42, "end": 0.48}, ...]
}
]
}
```
**Client examples**:
```python
# OpenAI SDK (recommended)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8899/v1", api_key="none")
result = client.audio.transcriptions.create(
model="fun-asr-nano",
file=open("audio.wav", "rb"),
response_format="verbose_json",
)
print(result.text)
```
```bash
# cURL
curl -X POST http://localhost:8899/v1/audio/transcriptions \
-F "file=@audio.wav" -F "model=fun-asr-nano" -F "response_format=verbose_json"
```
### 5.5 Protocol 3: WebSocket — `ws://host:port/ws`
WebSocket interface for the offline service. Send complete audio, then receive results. Speaker clustering is performed automatically on STOP, and results include the `spk` field.
**Client → Server**:
| Message | Description |
|---------|-------------|
| `"START"` | Begin session |
| `"LANGUAGE:中文"` | Set language (optional) |
| `"HOTWORDS:word1,word2"` | Set hotwords (optional) |
| `[binary]` | PCM16 16 kHz mono audio data |
| `"STOP"` | End session; request recognition result |
**Server → Client**:
```json
{"event": "started"}
{"event": "language_set", "language": "中文"}
{"sentences": [{"text":"...","start":..,"end":..}], "is_final": true, "duration_ms": 5170}
{"event": "stopped"}
```
**Client example**:
```python
import asyncio, websockets, json, numpy as np, soundfile as sf
async def offline_ws(audio_path):
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:8899/ws") as ws:
await ws.send("START")
await ws.recv()
await ws.send("LANGUAGE:中文")
await ws.recv()
# Send complete audio
await ws.send(pcm.tobytes())
await ws.send("STOP")
# Receive result
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(offline_ws("audio.wav"))
```
---
## 6. Streaming Speech Recognition Service
### 6.1 Service Architecture
```
Client (microphone / audio stream) serve_realtime_ws.py
│ │
│── WebSocket PCM16 16 kHz ───────────→│
│ (~100 ms per frame, continuous) │
│ │
│ ┌────┴──────────────────────────┐
│ │ Real-time loop: │
│ │ ├─ Dynamic VAD (60 ms chunk) │
│ │ ├─ Endpoint → vLLM decode │
│ │ ├─ No endpoint → partial │
│ │ └─ Streaming SPK assignment │
│ └────┬──────────────────────────┘
│ │
│←── JSON real-time push ──────────────│
```
**Characteristics**:
- Audio arrives frame by frame; processing starts immediately
- Natural sentence segmentation based on VAD endpoints
- Confirmed segment text is locked and never changes; partial text updates in real time
- Optional streaming speaker assignment (`--enable-spk`) + global re-clustering on STOP
- First-word latency ~480 ms
### 6.2 Starting the Service
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 --hotword-file hotword_list
```
For multi-client or long continuous-speech workloads, start by bounding partial previews and lowering the refresh rate:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
Speaker diarization is disabled by default; add `--enable-spk` only when the `spk` field is required.
For long-lived microphone sessions behind Docker, nginx, or a cloud load
balancer, keep WebSocket ping/pong enabled and tune the timeout to be longer
than short network stalls:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--ws-ping-interval 20 --ws-ping-timeout 60
```
Set `--ws-ping-interval 0` only when an external gateway already owns
keepalive/reconnect policy.
### 6.3 WebSocket Protocol
**Connection**: `ws://host:10095`
**Client → Server**:
| Message | Format | Description |
|---------|--------|-------------|
| Start | `"START"` | Initialize session |
| Hotwords | `"HOTWORDS:word1,word2"` | Optional |
| Language | `"LANGUAGE:中文"` | Optional |
| Audio | `binary` | PCM16 16 kHz mono |
| End | `"STOP"` | Final decode; SPK re-clustering only when `--enable-spk` is enabled |
**Server → Client**:
```json
{"event": "started"}
{"sentences": [{"text":"你好","start":300,"end":1200}], "partial": "世界", "is_final": false}
{"sentences": [...], "is_final": true}
{"event": "stopped"}
```
**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `partial_start_ms` = where the current provisional `partial` begins, `is_final` = true after STOP. When `--enable-spk` is enabled, `sentences[]` also includes `spk`.
**Sequence diagram**:
```
Client Server
│── START ───────→│
│←─ started ──────│
│── [audio] ─────→│
│←─ {partial} ────│ #refer to 6.5
│── [audio] ─────→│
│←─ {sentences+partial} ─│ (VAD cut a sentence)
│── STOP ────────→│
│←─ {is_final:true} ────│
│←─ stopped ─────│
```
### 6.4 Client Usage
**Python CLI**:
```bash
python client_python.py --server ws://localhost:10095 --mic
python client_python.py --server ws://localhost:10095 --file audio.wav
```
**Realtime benchmark**:
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav --server ws://localhost:10095 --clients 4 \
--output-jsonl realtime_ws_4c.jsonl
```
For metric definitions and reporting fields, see [Realtime WebSocket Benchmark](./benchmark/realtime_ws_benchmark.md).
**Browser**: Open `client_mic.html`
**Custom Python**:
```python
import asyncio, websockets, numpy as np, json
async def stream(audio_path):
import soundfile as sf
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:10095") as ws:
await ws.send("START")
await ws.recv()
for i in range(0, len(pcm), 1600):
await ws.send(pcm[i:i+1600].tobytes())
await asyncio.sleep(0.05)
await ws.send("STOP")
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(stream("audio.wav"))
```
### 6.5 Partial preview mechanism and long-sentence behavior
**What partial is and how it's produced**
While the user is speaking, the streaming service periodically (default `decode_interval≈0.48s` in `serve_realtime_ws.py`) decodes "the current sentence from its start up to now," emitting **provisional text** (the `partial` field in the protocol, which may be overwritten by later refreshes), until VAD detects the sentence end and locks it into `sentences`. This lets the user see text as they speak.
> Note: `serve_vllm.py`'s `/ws` (§5) has **no partial** and only returns at sentence end; use `serve_realtime_ws.py` for live preview.
**Frontend rendering rule**
Treat `partial` as a replaceable preview, not as text to append. A good UI keeps locked text and preview text separate:
```js
const committed = data.sentences.map((s) => s.text).join("");
const preview = data.partial || "";
render(committed + preview);
```
If `partial_start_ms` moves forward because `--partial-window-sec` is active, the preview only describes the current bounded decode window. Replace the preview area on each message; append only VAD-locked `sentences` or the final `is_final=true` result.
**Principle: why each partial re-encodes the whole segment from the start**
Fun-ASR-Nano's acoustic encoder (SenseVoice) is a **full-context, non-streaming** encoder — each frame's representation depends on the context of the entire segment. When the sentence continues and the audio grows, the context of the earlier frames changes, so **the previously computed encoding no longer holds**. It therefore cannot cache history and encode only the new frames the way a streaming / causal encoder would; it must run the whole "start → now" segment through the encoder again.
**Resulting behavior: partial gets slower on long sentences (O(L²))**
Because each refresh re-encodes from the sentence start, the longer a sentence, the longer each partial's audio and the more refreshes occur — so **total encoding work grows quadratically with sentence length**. In practice a ~29 s continuous utterance is fully re-encoded a dozen-plus times, with single-pass encoder time climbing from tens to hundreds of milliseconds. (The §4 SDK streaming "each chunk contains all audio from the start to now" is the same mechanism; long files behave the same way.)
**Usage guidance**
- Normal conversational speech has natural pauses, so VAD splits it into relatively short utterances and each partial's cost is naturally bounded — **usually nothing to worry about**.
- Only **very long, pauseless continuous speech** (e.g. reading aloud) makes a single utterance keep growing and the partial preview progressively slower. `serve_realtime_ws.py` bounds provisional previews with `--partial-window-sec 15` by default; for multi-client or continuous-monologue load tests, try `8-10` and raise `--decode-interval` to `0.8-1.0`. This only affects provisional `partial`; VAD-locked sentences and STOP final output still run on the full audio.
### 6.6 Cost of speaker diarization (SPK) and how to enable it
`serve_realtime_ws.py` **does not load** the SPK model by default. It loads `--spk-model` (default `iic/speech_eres2netv2_sv_zh-cn_16k-common`) only when started with `--enable-spk`, then runs speaker assignment for each VAD-completed sentence during streaming. Note:
- **SPK is of limited effectiveness on Fun-ASR-Nano** (see #2944); most real-time ASR scenarios do not need speaker separation.
- **Streaming SPK is expensive and grows with the session**: each sentence re-clusters **all historical embeddings** (**O(N²)**, more expensive per sentence as the session grows) and **synchronously blocks the event loop**; the session also **re-clusters everything again** at the end, so the per-sentence clustering during streaming is overwritten by the final result — redundant as far as the final output is concerned. This is especially pronounced under long sessions + high concurrency.
- **Recommendation**: keep the default off for multi-client live ASR; if diarization is required, add `--enable-spk` and treat the final STOP-time labels as authoritative.
### 6.7 Production concurrency and multi-process deployment
`serve_realtime_ws.py` is a **single-asyncio-event-loop** service: both `decode()` (timed partial) and `add_audio()` (decode triggered at VAD sentence end) **synchronously block** the entire event loop — while any one connection is decoding, all others pause sending/receiving. Therefore:
- **The single-process concurrency ceiling comes from event-loop serialization, not GPU compute.** Under high concurrency GPU utilization stays low and the encoder runs at ~86× real time; mistaking this for insufficient GPU and adding cards or tensor parallelism yields little (TP only splits the LLM, not the standalone encoder).
- **The right way to scale (currently) = multiple independent processes on one card + CUDA MPS + nginx round-robin**: each process has its own GIL and CUDA context, sidestepping the single-loop serialization; MPS lets the processes truly share the GPU concurrently and fill the idle compute; nginx round-robins across the WebSocket backends. Beyond a single card's headroom, scale out horizontally (one instance per card + a load balancer).
- **vLLM is not always more efficient for small real-time streams than several PyTorch processes.** vLLM helps most when requests can be batched or when LLM token decoding dominates. The current real-time WebSocket path submits many small, synchronous per-connection decode calls through one event loop, so it may reserve much more memory while still leaving GPU utilization modest. For a small number of continuous microphone streams, several lighter PyTorch processes can be easier to pack on one card. For vLLM, benchmark with the real traffic shape and start with lower `--gpu-memory-utilization` plus multiple service processes instead of assuming one vLLM process should carry every stream.
- **Sustainable concurrency has no universal "supports N connections" number.** The ceiling is set not by the number of connections but by **how many are speaking at the same moment** — each speaking connection triggers a partial decode roughly once per second, all serialized on that single event loop. It mainly varies with: **① silence ratio** — in real turn-taking users spend most of the time listening, so far fewer are decoding simultaneously than are connected, whereas a continuous monologue keeps nearly every connection decoding; **② sentence length** — longer sentences make each partial encode more expensive (see 6.5's O(L²)), raising load at the same connection count. So the same "single L20 + multi-process + MPS" setup can sustain dozens of connections under turn-taking-like load but markedly fewer under long, pauseless speech. **Any "supports X connections" figure holds only for the traffic profile it was measured under** — benchmark with your own real traffic (sentence length, pauses, continuous or not) rather than treating someone else's number as your spec.
---
## 7. Dynamic VAD
fsmn-vad enables dynamic silence thresholds by default. Offline and streaming modes use different configurations.
| Accumulated duration | Offline (preserve long segs ≤60 s) | Streaming (balance latency) |
|---------------------|-----------------------------------|-----------------------------|
| ≤ 5 s | 2000 ms | 2000 ms |
| 5–10 s | 2000 ms | 1500 ms |
| 10–15 s | 1000 ms | 1000 ms |
| 15–20 s | 1000 ms | 800 ms |
| 20–30 s | 800 ms | 800 ms |
| 30–45 s | 600 ms | 400 ms |
| 45–60 s | 200–400 ms | 100 ms |
| > 60 s | 100 ms | 100 ms |
Offline mode favors longer segments to reduce boundary-cut losses; streaming mode tightens faster to reduce latency.
### Customization
```python
model.generate(input="audio.wav", silence_schedule=[(5000,1500), (20000,800), (float('inf'),300)])
```
> GLM-ASR does not support long-segment inference; pass `dynamic_silence=False` when using it.
---
## 8. API Reference
| Parameter | AutoModelVLLM | serve_vllm.py | serve_realtime_ws.py |
|-----------|--------------|---------------|---------------------|
| model | ✓ | --model | --model |
| gpu_memory_utilization | ✓ | --gpu-memory-utilization | --gpu-memory-utilization |
| tensor_parallel_size | ✓ | — | --tensor-parallel-size |
| max_model_len | ✓ | --max-model-len | --max-model-len |
| language | generate() param | API param | --language / LANGUAGE: |
| hotwords | generate() param | API param | --hotword-file / HOTWORDS: |
---
## 9. FAQ
**Q: Offline or streaming?**
Complete files → offline (high throughput). Microphone / live stream → streaming (low latency).
**Q: Can GLM-ASR use dynamic VAD?**
It does not support long-segment inference. Use `dynamic_silence=False`.
**Q: Performance impact of SPK?**
RTFx drops from 102 to 46. CER is unchanged. Disabled by default.
**Q: Entry points for custom development?**
Offline: `serve_vllm.process_audio()` / `FunASRNanoVLLM.generate()`
Streaming: `serve_realtime_ws.RealtimeASRSession`
**Q: Slow first startup?**
vLLM initialization takes 60–90 s (KV Cache + CUDA Graph warmup). Subsequent inferences are instant.
**Q: vLLM returns repeated punctuation such as `!!!!!!!!` but PyTorch/HF generate is normal. What should I check?**
This usually means the audio frontend and checkpoint can work, but the vLLM
prompt-embedding path or decoding parameters differ from the upstream runner.
Check these items before changing the model:
- Pass prompt embeddings to vLLM as float32:
`EmbedsPrompt(prompt_embeds=input_embeds.float())`.
- Use ASR-style deterministic decoding. The Fun-ASR-Nano vLLM path defaults to
`temperature=0.0`, `top_p=1.0`, and `skip_special_tokens=True`. In
prompt-embeds mode, keep `repetition_penalty` at the neutral `1.0` unless you
are using a token-prompt path; other values are normalized by FunASR's vLLM
helpers to avoid vLLM CUDA scatter errors.
- Verify that `model_dir` and `vllm_model_dir` are the matching Fun-ASR-Nano
pair. If clearing `vllm_model_dir` makes the same audio work through HF
generate, keep debugging the vLLM path rather than the audio file.
- Log vLLM `finish_reason`, generated token ids, prompt embedding dtype, and
prompt embedding shape for one failing sample. Repeated punctuation with
`finish_reason="length"` usually points to decode/prompt mismatch rather than
VAD or audio loading.
docs/vllm_guide_zh.md
+819
View File
@@ -0,0 +1,819 @@
# FunASR vLLM 推理引擎指南
---
## Benchmark
**测试集**：184 文件，11541 秒，Fun-ASR-Nano / GLM-ASR-Nano。RTFx 定义、计时口径和可复现字段请见 [Benchmark RTF and Reproducibility Notes](./benchmark/rtf_reproducibility.md)。
| 模型 | 引擎 | VAD | RTFx | CER | 备注 |
|------|------|-----|------|-----|------|
| Fun-ASR-Nano | PyTorch | dynamic | 21 | 8.06% | 基准 |
| Fun-ASR-Nano | **vLLM batch** | dynamic | **340** | **8.20%** | 16x 加速 |
| Fun-ASR-Nano | **离线服务 (no SPK)** | dynamic | **102** | 8.14% | |
| Fun-ASR-Nano | **离线服务 (+SPK)** | dynamic | **46** | 8.19% | SPK 默认关闭 |
| GLM-ASR-Nano | **vLLM batch** | fixed | **265** | 12.93% | 不支持长音频推理 |
> vLLM 与 PyTorch CER 完全一致（差 < 0.2%），速度提升 16-340x。
---
## 目录
1. [安装与环境](#1-安装与环境)
2. [vLLM 推理引擎架构](#2-vllm-推理引擎架构)
3. [离线 SDK 推理](#3-离线-sdk-推理)
4. [流式 SDK 推理](#4-流式-sdk-推理)
5. [离线语音识别服务](#5-离线语音识别服务)
6. [流式语音识别服务](#6-流式语音识别服务)
7. [动态 VAD](#7-动态-vad)
8. [API 参考](#8-api-参考)
9. [FAQ](#9-faq)
---
## 1. 安装与环境
先安装 vLLM,按 NVIDIA 驱动的 CUDA 版本选对应版本;vLLM 会自动钉定并安装匹配的 torch / torchaudio / torchvision 三件套,所以不要自己装 torch/torchaudio——三者 ABI 锁死,必须是互相编译匹配的同一组(如 torch 2.10.0 ↔ torchaudio 2.10.0 ↔ torchvision 0.25.0),只能随 vLLM 一起来。
```bash
# 1) 先装 vLLM。按 `nvidia-smi` 显示的 CUDA 版本(驱动支持的最高 CUDA,不是 runtime CUDA)选版本,
# vLLM 会带来匹配的 torch/torchaudio/torchvision。
# 驱动 CUDA 12.x -> pip install vllm==0.19.1 (附带 torch 2.10 / cu128)
# 驱动 CUDA >= 13 -> pip install vllm (最新版;附带 torch 2.11 / cu130)
pip install "vllm==0.19.1" # 按你的驱动 CUDA 调整;见下方说明
# 2) 再装 FunASR 与其余依赖。
pip install funasr>=1.3.0
pip install safetensors tiktoken websockets regex fastapi uvicorn python-multipart
cd /path/to/FunASR && pip install -e .
```
**硬件**：GPU ≥ 8GB VRAM，CUDA ≥ 11.8。推荐 16GB+。
为什么不要单独执行 `pip install torch torchaudio` ? torch/torchaudio/torchvision 的版本由 vLLM 版本决定—— 每个大版本会一起升级(见 vLLM 的 [requirements/cuda.txt](https://github.com/vllm-project/vllm/blob/main/requirements/cuda.txt))。手动安装会拉到最新 wheel,可能是为比你驱动更新的 CUDA runtime 编译的;PyTorch 会在 CUDA 初始化阶段、FunASR 启动前就报 The NVIDIA driver on your system is too old。让 vLLM 统一钉定这三件套即可避免。若仍遇到该错误,请安装其 CUDA 构建与 nvidia-smi 显示的 CUDA 匹配的 vLLM 版本(如 CUDA 12.x 用 vllm==0.19.1),或先升级 NVIDIA 驱动。
---
## 2. vLLM 推理引擎架构
### 整体架构
FunASR 的 vLLM 集成将 ASR 模型拆分为两部分独立运行：
```
┌──────────────────────────────────────────────────────────────┐
│ FunASR + vLLM 推理架构 │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────── PyTorch (单 GPU) ───────────────┐ │
│ │ │ │
│ │ Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor │
│ │ (fbank) (SenseVoice/ (Transformer/ │
│ │ Whisper) MLP) │
│ │ │ │
│ │ ▼ │
│ │ Audio Embeddings │
│ │ │ │
│ │ Text Prompt ──→ Tokenize ──→ Embed │
│ │ (system/user/ │ │
│ │ hotwords/language) │ │
│ │ ▼ │
│ │ [Concat Embeddings] │
│ └─────────────────────────────────┼─────────────┘ │
│ │ │
│ ▼ EmbedsPrompt │
│ ┌─────────────── vLLM Engine ────────────────────┐ │
│ │ │ │
│ │ PagedAttention + Continuous Batching │ │
│ │ KV Cache 管理 + CUDA Graph │ │
│ │ Tensor Parallel (多卡) │ │
│ │ │ │
│ │ Qwen3-0.6B / Llama-2B (LLM 解码) │ │
│ │ │ │
│ └────────────────────┬───────────────────────────┘ │
│ │ │
│ ▼ │
│ Generated Text │
│ │ │
│ ┌────────────────────┼──────────────────────────┐ │
│ │ (可选) CTC Decoder ──→ Forced Alignment │ │
│ │ ──→ 字级别时间戳 │ │
│ └───────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### 为什么用 vLLM？
| 特性 | PyTorch generate() | vLLM |
|------|-------------------|------|
| KV Cache 管理 | 固定分配，浪费显存 | PagedAttention，按需分配 |
| 批处理 | 需手动 padding | Continuous Batching，自动调度 |
| CUDA 优化 | 无 | CUDA Graph + 算子融合 |
| 多卡并行 | 手动实现 | Tensor Parallel 一行配置 |
| 吞吐量 | RTFx ~20 | **RTFx 340+** |
### 支持模型
| 模型 | LLM 部分 | audio encoder | vLLM 加速 |
|------|---------|---------------|-----------|
| **Fun-ASR-Nano** | Qwen3-0.6B | SenseVoice | ✓ 21.7x |
| **GLM-ASR-Nano** | Llama-2B | Whisper-like | ✓ 7.6x |
| LLMASR | Qwen/Vicuna | Whisper | ✓ |
| Paraformer | 无 LLM | — | ✗ 非自回归 |
| SenseVoice | 无 LLM | — | ✗ encoder-decoder |
### 关键实现细节
1. **权重分离**：从 `model.pt` 提取 LLM 权重，转为 HuggingFace 格式供 vLLM 加载
2. **EmbedsPrompt**：直接把**已算好的 embedding 向量**（而非通常的 token ID）作为 prompt 送入 vLLM（开关 `enable_prompt_embeds=True`）。Fun-ASR-Nano 必须用它，因为音频经 adaptor 得到的是连续向量、不是 token，需把音频 embedding 与文本 embedding 在序列维拼接后整体送入 vLLM
3. **use_low_frame_rate**：Fun-ASR-Nano 的 adaptor 输出需按公式截断到正确 token 数（一致性关键）
4. **batch encode**：多条音频通过 `extract_fbank` → `audio_encoder` → `audio_adaptor` 一次前向
5. **CTC 时间戳**：保留 encoder_out，生成文本后做 forced alignment 得到字级别时间
---
## 3. 离线 SDK 推理
适用于大规模音频转写、离线批量处理。vLLM 的批处理能力在此场景优势最大。
### 设计原理
离线 SDK 推理将 ASR 流水线拆分为两阶段独立执行：
```
┌─────────────────────────────────────────────────────────────────────┐
│ 阶段 1: 音频编码（PyTorch, 单 GPU） │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 音频文件列表 ──→ 分组（每 8 条）──→ Frontend(Fbank) │
│ │ │ │
│ │ ▼ │
│ │ SenseVoice Encoder │
│ │ │ │
│ │ ▼ │
│ │ Audio Adaptor │
│ │ (dim 转换 + low_frame_rate 截断) │
│ │ │ │
│ └─── 共享文本 prompt 预编码 ─────┐ ▼ │
│ (system/hotwords/language) │ audio_embeds │
│ │ │ │ │
│ ▼ │ ▼ │
│ prefix_emb ──→ [concat: prefix | audio | suffix] │
│ │ │
│ ▼ │
│ EmbedsPrompt（N 条） │
└──────────────────────────────────────────────┼──────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 阶段 2: LLM 解码（vLLM, 多 GPU Tensor Parallel） │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ EmbedsPrompt × N ──→ vLLM Continuous Batching │
│ (PagedAttention + CUDA Graph) │
│ │ │
│ ▼ │
│ Generated token_ids × N │
│ │ │
│ ▼ │
│ Decode + 后处理（去特殊标记、清洗） │
│ │ │
│ ▼ │
│ (可选) CTC Forced Alignment → 字级别时间戳 │
└─────────────────────────────────────────────────────────────────────┘
```
**关键设计决策：**
1. **权重分离**：首次运行时从 `model.pt` 提取 `llm.*` 前缀的权重，保存为 HuggingFace safetensors 格式供 vLLM 加载（缓存到 `Qwen3-0.6B-vllm/` 目录）
2. **Embedding 拼接**：文本 prompt 通过 LLM 的 `embed_tokens` 层编码为 embedding，与音频 adaptor 输出在序列维度拼接：`[prefix_emb | audio_emb | suffix_emb]`，以 `EmbedsPrompt` 形式送入 vLLM
3. **Low Frame Rate 截断**：adaptor 输出需按公式 `fake_token_len = ((((fbank_len - 3 + 2) // 2 - 3 + 2) // 2) - 1) // 2 + 1` 截断到正确长度，确保与 PyTorch 训练时一致
4. **批量音频编码**：多条音频按 batch_size=8 分组通过 encoder + adaptor 前向，减少 GPU kernel launch 开销
5. **文本 prompt 共享**：同一批次内 hotwords/language 相同时，prefix_emb 和 suffix_emb 只计算一次
6. **CTC 时间戳**：保留 encoder_out，LLM 生成文本后做 forced alignment 得到字级别时间
**为什么比 PyTorch generate() 快？**
| 维度 | PyTorch | vLLM |
|------|---------|------|
| KV Cache | 固定预分配（浪费显存） | PagedAttention 按需分配 |
| 批处理 | 需手动 padding 对齐 | Continuous Batching 自动调度 |
| CUDA | 逐 sample 串行 | CUDA Graph + 算子融合 |
| 多卡 | 需手动实现 | Tensor Parallel 一行配置 |
| 结果 | RTFx ~20 | **RTFx 340+**（16倍加速） |
### 通用接口（推荐）
```python
from funasr.auto.auto_model_vllm import AutoModelVLLM
model = AutoModelVLLM(
model="FunAudioLLM/Fun-ASR-Nano-2512",
hub="ms", # 或 "hf"
tensor_parallel_size=2, # 多卡并行
gpu_memory_utilization=0.8,
)
results = model.generate(
["audio1.wav", "audio2.wav"],
language="中文",
hotwords=["张三", "北京"],
)
for r in results:
print(f"[{r['key']}] {r['text']}")
```
### 直接接口
```python
from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
engine = FunASRNanoVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
tensor_parallel_size=4,
)
results = engine.generate(
inputs="wav.scp", # 支持 scp/jsonl/文件列表
hotwords=["开放时间"],
language="中文",
max_new_tokens=512,
)
```
### 命令行
```bash
cd examples/industrial_data_pretraining/fun_asr_nano
# 单文件
python demo_vllm.py --input audio.wav --language 中文
# 批量 + 多卡
python demo_vllm.py --input wav.scp --tensor-parallel-size 4 --batch-size 32
# 带热词 + 保存结果
python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.jsonl
```
---
## 4. 流式 SDK 推理
将音频按 720ms chunk 逐步处理，输出逐步稳定的识别结果。适用于 SDK 集成实时字幕场景。
### 设计原理
```
音频流（720ms chunks）
│ 累积重编码（每个 chunk 包含从头到当前的全部音频）
▼
┌──────────────────────┐
│ Stage 1: 前 10 chunk │ ← 无 prev_text，批量生成
│ 找到稳定输出 │
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Stage 2: 后续 chunk │ ← 用稳定输出作 prev_text
└──────────┬───────────┘
▼
每个 chunk: [fixed 区域（确认）] + [8字 unfixed（可能变）]
```
### 用法
```python
from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
engine = FunASRNanoStreamingVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
chunk_ms=720,
rollback_chars=8,
)
for result in engine.streaming_generate("audio.wav", language="中文"):
if result["is_final"]:
print(f"最终: {result['text']}")
else:
print(f"[{result['audio_duration_ms']:.0f}ms] 确认: {result['fixed_text']}")
```
**注意：EmbedsPrompt 下不能用 `repetition_penalty`。** 此时 prompt 是 embedding 向量、没有对应的 token ID，而 `repetition_penalty` 要靠 prompt 的 token ID 在 logits 上给已出现的词降分；用在 EmbedsPrompt 上会**索引越界、触发 CUDA device-side assert**。
### 生产 API 稳定性清单
把 `AutoModelVLLM` 封装成长驻 API 服务时，请隔离每次请求的状态，并固定安全的解码默认值：
```python
common = dict(
language="auto",
temperature=0.0,
repetition_penalty=1.0,
max_new_tokens=200,
)
for _ in range(2):
results = model.generate(["vad_segment_01.wav", "vad_segment_02.wav"], **common)
print([r["text"] for r in results])
```
如果同一个音频第一次请求正常、第二次请求开始重复：
1. 先把 API 层拿掉，用相同 VAD 分段跑上面的最小脚本。
2. 如果最小脚本稳定，优先检查 API 封装是否复用了请求级变量、上一轮 VAD 分段列表、上一轮 `results` 或累积文本。
3. 如果最小脚本也重复，再记录完整的 `funasr`、`vllm`、`torch` 版本，以及第一次和第二次输出文本，再调整其它解码参数。
不要通过调大 `repetition_penalty` 来压制 Fun-ASR-Nano vLLM 重复输出；prompt-embeds 路径应保持中性值 `1.0`。
### 输出特性
| 累积音频 | 输出质量 |
|---------|---------|
| < 1.5s | 空或噪声 |
| 1.5-3.0s | 部分正确 |
| > 3.0s | 准确输出 |
---
## 5. 离线语音识别服务
### 5.1 服务架构
```
客户端 serve_vllm.py
│ │
│── HTTP/OpenAI/WebSocket ──────────────→│
│ │
│ ┌────┴────────────────────────┐
│ │ 1. 接收完整音频文件 │
│ │ 2. 动态 VAD 分段（≤60s/段） │
│ │ 3. vLLM batch 推理所有段 │
│ │ 4. CTC 时间戳（逐字） │
│ │ 5. 说话人分离（可选） │
│ └────┬────────────────────────┘
│ │
│←── JSON 结果 ─────────────────────────│
```
**特点**：
- 音频完整到达后处理，适合文件转写
- 动态 VAD 保留长段（≤60s），减少边界切割损失
- batch 推理所有 VAD 段，吞吐量高
- 自动输出字级别时间戳
- SPK 说话人分离默认关闭，客户端可开启
### 5.2 启动服务
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_vllm.py \
--port 8899 \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--gpu-memory-utilization 0.5
```
> **关于 `CUDA_VISIBLE_DEVICES`**：这是[vllm的一个环境变量](https://docs.vllm.ai/en/v0.4.3/serving/env_vars.html) ，示例中的 `=0` 只是"用第 0 张卡"的示例值，**不是固定写法**，它选择本进程可见的 GPU（编号同 `nvidia-smi`），单卡机器也不需要设置。
>
> **单卡多实例**：0.6B / 1.7B 这类小模型一张卡可起多个实例，多进程可都指向同一张卡（如都 `=0`）+ MPS 共享；分卡则进程 A `=0`、B `=1`（见 §6.7）。
### 5.3 协议一：HTTP REST — `POST /asr`
功能最全的接口，支持 SPK、时间戳、热词。
**请求**：`multipart/form-data`
| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
| `file` | file | 必填 | 音频文件（wav/mp3/flac） |
| `language` | string | None | 语种（"中文"/"English"/...），None 为自动 |
| `hotwords` | string | "" | 热词，逗号分隔 |
| `spk` | bool | false | 是否开启说话人分离 |
| `timestamp` | bool | true | 是否输出字级别时间戳 |
**响应**：
```json
{
"text": "完整识别文本",
"segments": [
{
"text": "段文本",
"start": 1.7,
"end": 14.8,
"speaker": "SPK0",
"words": [
{"word": "砸", "start": 2.02, "end": 2.08},
{"word": "了", "start": 2.26, "end": 2.32}
]
}
],
"duration": 227.4,
"processing_time": 3.422,
"rtf": 0.015
}
```
**客户端示例**：
```bash
# cURL
curl -X POST http://localhost:8899/asr \
-F "file=@meeting.wav" -F "language=中文" -F "spk=true"
```
```python
# Python requests
import requests
resp = requests.post("http://localhost:8899/asr",
files={"file": open("audio.wav", "rb")},
data={"language": "中文", "spk": "true"})
result = resp.json()
```
```javascript
// JavaScript fetch
const form = new FormData();
form.append("file", audioBlob, "audio.wav");
form.append("language", "中文");
form.append("spk", "true");
const resp = await fetch("http://localhost:8899/asr", { method: "POST", body: form });
const result = await resp.json();
```
### 5.4 协议二：OpenAI Whisper 兼容 — `POST /v1/audio/transcriptions`
兼容 OpenAI Whisper API 标准，可直接用 OpenAI SDK 接入。
**请求**：`multipart/form-data`
| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
| `file` | file | 必填 | 音频文件 |
| `model` | string | "fun-asr-nano" | 模型名（兼容字段） |
| `language` | string | None | 语种 |
| `response_format` | string | "json" | "json" / "text" / "verbose_json" |
| `timestamp_granularities` | string | "word" | "word" / "segment" |
| `spk` | bool | false | 说话人分离（FunASR 扩展字段） |
**响应**（`verbose_json`）：
```json
{
"task": "transcribe",
"language": "zh",
"duration": 5.17,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"segments": [
{
"id": 0, "start": 0.0, "end": 5.15,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"words": [{"word": "我", "start": 0.42, "end": 0.48}, ...]
}
]
}
```
**客户端示例**：
```python
# OpenAI SDK（推荐）
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8899/v1", api_key="none")
result = client.audio.transcriptions.create(
model="fun-asr-nano",
file=open("audio.wav", "rb"),
response_format="verbose_json",
)
print(result.text)
```
```bash
# cURL
curl -X POST http://localhost:8899/v1/audio/transcriptions \
-F "file=@audio.wav" -F "model=fun-asr-nano" -F "response_format=verbose_json"
```
### 5.5 协议三：WebSocket — `ws://host:port/ws`
离线服务的 WebSocket 接口，发送完整音频后获取结果。STOP 时自动进行说话人聚类，结果中包含 `spk` 字段。
**客户端 → 服务端**：
| 消息 | 说明 |
|------|------|
| `"START"` | 开始会话 |
| `"LANGUAGE:中文"` | 设置语种（可选） |
| `"HOTWORDS:词1,词2"` | 设置热词（可选） |
| `[binary]` | PCM16 16kHz mono 音频数据 |
| `"STOP"` | 结束，请求识别结果 |
**服务端 → 客户端**：
```json
{"event": "started"}
{"event": "language_set", "language": "中文"}
{"sentences": [{"text":"...","start":..,"end":..}], "is_final": true, "duration_ms": 5170}
{"event": "stopped"}
```
**客户端示例**：
```python
import asyncio, websockets, json, numpy as np, soundfile as sf
async def offline_ws(audio_path):
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:8899/ws") as ws:
await ws.send("START")
await ws.recv()
await ws.send("LANGUAGE:中文")
await ws.recv()
# 发送完整音频
await ws.send(pcm.tobytes())
await ws.send("STOP")
# 接收结果
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(offline_ws("audio.wav"))
```
---
## 6. 流式语音识别服务
### 6.1 服务架构
```
客户端（麦克风/音频流） serve_realtime_ws.py
│ │
│── WebSocket PCM16 16kHz ────────────→│
│ (每帧 ~100ms，持续发送) │
│ │
│ ┌────┴─────────────────────────┐
│ │ 实时循环： │
│ │ ├─ 动态 VAD（60ms chunk） │
│ │ ├─ 检测到端点 → vLLM 解码 │
│ │ ├─ 未结束 → partial 预览 │
│ │ └─ 说话人流式分配 │
│ └────┬─────────────────────────┘
│ │
│←── JSON 实时推送 ───────────────────│
```
**特点**：
- 音频逐帧到达，边收边处理
- 基于 VAD 端点自然分句
- 确认段文字锁定不变，partial 实时更新
- 可选流式说话人分配（`--enable-spk`）+ STOP 时全局重聚类
- 首字延迟 ~480ms
### 6.2 启动服务
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 --hotword-file 热词列表
```
多客户端或长时间连续语音场景，建议先限制 partial 预览窗口并适当降低刷新频率：
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
说话人分离默认关闭；只有确实需要 `spk` 字段时再加 `--enable-spk`。
如果麦克风长连接经过 Docker、nginx 或云负载均衡，建议保持 WebSocket
ping/pong 开启，并把 timeout 调到能覆盖短暂网络抖动：
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--ws-ping-interval 20 --ws-ping-timeout 60
```
只有在外部网关已经统一负责 keepalive / reconnect 策略时，才考虑设置
`--ws-ping-interval 0` 关闭服务端 ping。
### 6.3 WebSocket 协议
**连接**：`ws://host:10095`
**客户端 → 服务端**：
| 消息 | 格式 | 说明 |
|------|------|------|
| 开始 | `"START"` | 初始化 session |
| 热词 | `"HOTWORDS:词1,词2"` | 可选 |
| 语种 | `"LANGUAGE:中文"` | 可选 |
| 音频 | `binary` | PCM16 16kHz mono |
| 结束 | `"STOP"` | 最终解码；启用 `--enable-spk` 时会做 SPK 重聚类 |
**服务端 → 客户端**：
```json
{"event": "started"}
{"sentences": [{"text":"你好","start":300,"end":1200}], "partial": "世界", "is_final": false}
{"sentences": [...], "is_final": true}
{"event": "stopped"}
```
**字段**：`sentences[]` = 已锁定句子，`partial` = 当前正在说的临时文本（可能变化），`partial_start_ms` = 当前 `partial` 对应音频窗口的起点，`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
**时序**：
```
Client Server
│── START ───────→│
│←─ started ──────│
│── [audio] ─────→│
│←─ {partial} ────│ # partial 的原理是注意事项见 6.5
│── [audio] ─────→│
│←─ {sentences+partial} ─│ (VAD 切了一句)
│── STOP ────────→│
│←─ {is_final:true} ────│
│←─ stopped ─────│
```
### 6.4 客户端调用
**Python CLI**：
```bash
python client_python.py --server ws://localhost:10095 --mic
python client_python.py --server ws://localhost:10095 --file audio.wav
```
**实时压测**：
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav --server ws://localhost:10095 --clients 4 \
--output-jsonl realtime_ws_4c.jsonl
```
指标定义和报告字段见 [Realtime WebSocket Benchmark](./benchmark/realtime_ws_benchmark.md)。
**浏览器**：打开 `client_mic.html`
**自定义 Python**：
```python
import asyncio, websockets, numpy as np, json
async def stream(audio_path):
import soundfile as sf
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:10095") as ws:
await ws.send("START")
await ws.recv()
for i in range(0, len(pcm), 1600):
await ws.send(pcm[i:i+1600].tobytes())
await asyncio.sleep(0.05)
await ws.send("STOP")
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(stream("audio.wav"))
```
### 6.5 partial 预览机制与长句特性
**partial 是什么、怎么产生的**
流式服务在用户说话过程中会周期性地（`serve_realtime_ws.py` 默认 `decode_interval≈0.48s`）对"当前这句话从句首到现在"的音频解码一次，输出**临时文字**（即协议里的 `partial` 字段，可被后续刷新覆盖），直到 VAD 判定句尾才锁定进 `sentences`。这让用户边说边看到字。
> 注：`serve_vllm.py`（§5）的 `/ws` **没有 partial**、只在句尾返回；要实时预览请用 `serve_realtime_ws.py`。
**前端渲染原则**
`partial` 只能当作“可替换预览”，不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开：
```js
const committed = data.sentences.map((s) => s.text).join("");
const preview = data.partial || "";
render(committed + preview);
```
如果启用了 `--partial-window-sec`，`partial_start_ms` 可能随着窗口向前滑动；这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域，只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。
**原理：为什么每次 partial 都从句首整段重编**
Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时，先前那些帧的上下文随之改变，**之前算出的编码不再成立**，因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧"，只能把"句首→当前"的整段重新过一遍编码器。
**由此带来的特性：长句的 partial 会越来越慢（O(L²)）**
正因每次都从句首重编，一句话越长，单次 partial 要编的音频越长、刷新次数也越多——**总编码量随句长二次增长**。实测一句约 29s 的连续发言会被完整重编十余次，单次 encoder 耗时从几十毫秒爬到数百毫秒。（§4 SDK 流式"每个 chunk 包含从头到当前的全部音频"是同一机制，长文件同理。）
**使用建议**
- 正常对话语音有自然停顿，VAD 会把它切成一句句较短的语音，每句 partial 的开销自然受限，**通常无需关注**。
- 只有**超长、不停顿的连续语音**（如长篇朗读）会让单句不断变长、partial 预览逐渐变慢。`serve_realtime_ws.py` 默认用 `--partial-window-sec 15` 限制临时预览窗口；多客户端或连续独白压测时可降到 `8-10`，并把 `--decode-interval` 提高到 `0.8-1.0`。这只影响临时 `partial`，VAD 锁定句和 STOP 最终结果仍走完整音频。
### 6.6 说话人分离（SPK）的代价与开关
`serve_realtime_ws.py` 默认**不加载** SPK 模型。只有启动时显式加 `--enable-spk`，才会加载 `--spk-model`（默认 `iic/speech_eres2netv2_sv_zh-cn_16k-common`）并在流式中对每个 VAD 完成句调用一次说话人分配。需要注意：
- **Fun-ASR-Nano 上 SPK 效果有限**（见 #2944），多数实时 ASR 场景并不需要说话人分离。
- **流式 SPK 代价高且随会话变长**：每句对**全部历史 embedding** 做一次全量重聚类（**O(N²)**，会话越长每句越贵），且**同步阻塞事件循环**；而会话结束时还会**全量重聚一遍**，流式期间每句的聚类结果会被最终结果覆盖——对最终输出而言属于重复计算。长会话 + 高并发下尤其明显。
- **建议**：多客户端实时转写优先保持默认关闭；确需 diarization 时再加 `--enable-spk`，并以 STOP 后的最终 `spk` 标签为准。
### 6.7 生产并发与多进程部署
`serve_realtime_ws.py` 是**单 asyncio 事件循环**服务：`decode()`（定时 partial）与 `add_audio()`（VAD 句尾触发解码）都**同步阻塞**整个事件循环——任一路在解码时，其余连接全部暂停收发。因此：
- **单进程并发墙来自事件循环串行，不是 GPU 算力**。
- **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**：每个进程有独立的 GIL 与 CUDA 上下文，绕开单循环串行；MPS 让多进程真正并发共享 GPU、填满空闲算力；nginx 在多个 WebSocket 后端间轮询。超过单卡余量后，再横向加卡（每卡一实例 + 负载均衡）。
- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度；而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码，无法自然形成大 batch，所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流，多个轻量 PyTorch 进程有时更容易在一张卡上排布；使用 vLLM 时请按真实话务压测，先配合较低的 `--gpu-memory-utilization` 和多进程服务，而不是假设一个 vLLM 进程就应该承载所有连接。
- **可持续并发没有通用的"支持 N 路"数字**：决定上限的不是在线连接数，而是**同一时刻有多少路正在"说话"**——每路只要在说，就每约 1 秒触发一次 partial 解码，全部串行在那个单事件循环上。它主要随两点变化：**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声，同时解码的路数远少于在线连接数；连续独白则几乎每路都在持续解码，负载高得多。**② 句长**——句子越长，单次 partial 的编码越贵（见 6.5 的 O(L²)），同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS"，在接近真实 turn-taking 的负载下可稳定支撑数十路，而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务（句长、停顿、是否连续说话）压测确定，别把别处测出的某个并发数当成自己的规格。
---
## 7. 动态 VAD
fsmn-vad 默认启用动态静音阈值。离线和流式使用不同配置。
| 累积时长 | 离线（保留长段 ≤60s） | 流式（平衡延迟） |
|---------|-------------------|----------------|
| ≤ 5s | 2000ms | 2000ms |
| 5-10s | 2000ms | 1500ms |
| 10-15s | 1000ms | 1000ms |
| 15-20s | 1000ms | 800ms |
| 20-30s | 800ms | 800ms |
| 30-45s | 600ms | 400ms |
| 45-60s | 200-400ms | 100ms |
| > 60s | 100ms | 100ms |
离线倾向保留长段减少边界损失；流式更快收紧以降低延迟。
### 自定义
```python
model.generate(input="audio.wav", silence_schedule=[(5000,1500), (20000,800), (float('inf'),300)])
```
> GLM-ASR 不支持长段，使用时传 `dynamic_silence=False`。
---
## 8. API 参考
| 参数 | AutoModelVLLM | serve_vllm.py | serve_realtime_ws.py |
|------|--------------|---------------|---------------------|
| model | ✓ | --model | --model |
| gpu_memory_utilization | ✓ | --gpu-memory-utilization | --gpu-memory-utilization |
| tensor_parallel_size | ✓ | — | --tensor-parallel-size |
| max_model_len | ✓ | --max-model-len | --max-model-len |
| language | generate() 参数 | API 参数 | --language / LANGUAGE: |
| hotwords | generate() 参数 | API 参数 | --hotword-file / HOTWORDS: |
---
## 9. FAQ
**Q: 离线还是流式？**
完整文件 → 离线（高吞吐）。麦克风/直播 → 流式（低延迟）。
**Q: GLM-ASR 用动态 VAD？**
不支持长段推理，用 `dynamic_silence=False`。
**Q: SPK 性能影响？**
RTFx 102 → 46。CER 不变。默认关闭。
**Q: 二次开发入口？**
离线：`serve_vllm.process_audio()` / `FunASRNanoVLLM.generate()`
流式：`serve_realtime_ws.RealtimeASRSession`
**Q: 首次慢？**
vLLM 初始化 60-90s，之后即时。
**Q: vLLM 输出连续标点（例如 `!!!!!!!!`），但 PyTorch/HF generate 正常，应该先查什么？**
这通常说明音频 frontend 和 checkpoint 本身能工作，但 vLLM prompt-embedding
路径或解码参数和 upstream runner 不一致。改模型前先检查这些项：
- 传给 vLLM 的 prompt embeddings 要显式转成 float32：
`EmbedsPrompt(prompt_embeds=input_embeds.float())`。
- 使用 ASR 更合适的确定性解码。Fun-ASR-Nano vLLM 路径默认使用
`temperature=0.0`、`top_p=1.0` 和 `skip_special_tokens=True`。在
prompt-embeds 模式下，`repetition_penalty` 保持中性的 `1.0`，除非你走的是
token prompt 路径；FunASR 的 vLLM helper 会把其他值归一化，避免 vLLM CUDA scatter
错误。
- 确认 `model_dir` 和 `vllm_model_dir` 是匹配的一组 Fun-ASR-Nano 模型。如果清空
`vllm_model_dir` 后同一音频走 HF generate 正常，就继续排查 vLLM 路径，而不是音频文件。
- 对一个失败样本记录 vLLM `finish_reason`、生成 token ids、prompt embedding dtype
和 shape。连续标点且 `finish_reason="length"` 时，通常更像解码/prompt 不匹配，而不是
VAD 或音频读取问题。
docs/vllm_guide_zh_v2.md
+781
View File
@@ -0,0 +1,781 @@
# FunASR vLLM 推理引擎指南
---
## Benchmark
**测试集**：184 文件，11541 秒，Fun-ASR-Nano / GLM-ASR-Nano。
| 模型 | 引擎 | VAD | RTFx | CER | 备注 |
|------|------|-----|------|-----|------|
| Fun-ASR-Nano | PyTorch | dynamic | 21 | 8.06% | 基准 |
| Fun-ASR-Nano | **vLLM batch** | dynamic | **340** | **8.20%** | 16x 加速 |
| Fun-ASR-Nano | **离线服务 (no SPK)** | dynamic | **102** | 8.14% | |
| Fun-ASR-Nano | **离线服务 (+SPK)** | dynamic | **46** | 8.19% | SPK 默认关闭 |
| GLM-ASR-Nano | **vLLM batch** | fixed | **265** | 12.93% | 不支持长音频推理 |
> vLLM 与 PyTorch CER 完全一致（差 < 0.2%），速度提升 16-340x。
---
## 目录
1. [安装与环境](#1-安装与环境)
2. [vLLM 推理引擎架构](#2-vllm-推理引擎架构)
3. [离线 SDK 推理](#3-离线-sdk-推理)
4. [流式 SDK 推理](#4-流式-sdk-推理)
5. [离线语音识别服务](#5-离线语音识别服务)
6. [流式语音识别服务](#6-流式语音识别服务)
7. [动态 VAD](#7-动态-vad)
8. [API 参考](#8-api-参考)
9. [FAQ](#9-faq)
---
## 1. 安装与环境
先安装 vLLM,按 NVIDIA 驱动的 CUDA 版本选对应版本;vLLM 会自动钉定并安装匹配的 torch / torchaudio / torchvision 三件套,所以不要自己装 torch/torchaudio——三者 ABI 锁死,必须是互相编译匹配的同一组(如 torch 2.10.0 ↔ torchaudio 2.10.0 ↔ torchvision 0.25.0),只能随 vLLM 一起来。
```bash
# 1) 先装 vLLM。按 `nvidia-smi` 显示的 CUDA 版本(驱动支持的最高 CUDA,不是 runtime CUDA)选版本,
# vLLM 会带来匹配的 torch/torchaudio/torchvision。
# 驱动 CUDA 12.x -> pip install vllm==0.19.1 (附带 torch 2.10 / cu128)
# 驱动 CUDA >= 13 -> pip install vllm (最新版;附带 torch 2.11 / cu130)
pip install "vllm==0.19.1" # 按你的驱动 CUDA 调整;见下方说明
# 2) 再装 FunASR 与其余依赖。
pip install funasr>=1.3.0
pip install safetensors tiktoken websockets regex fastapi uvicorn python-multipart
cd /path/to/FunASR && pip install -e .
```
**硬件**：GPU ≥ 8GB VRAM，CUDA ≥ 11.8。推荐 16GB+。
为什么不要单独执行 `pip install torch torchaudio` ? torch/torchaudio/torchvision 的版本由 vLLM 版本决定—— 每个大版本会一起升级(见 vLLM 的 [requirements/cuda.txt](https://github.com/vllm-project/vllm/blob/main/requirements/cuda.txt))。手动安装会拉到最新 wheel,可能是为比你驱动更新的 CUDA runtime 编译的;PyTorch 会在 CUDA 初始化阶段、FunASR 启动前就报 The NVIDIA driver on your system is too old。让 vLLM 统一钉定这三件套即可避免。若仍遇到该错误,请安装其 CUDA 构建与 nvidia-smi 显示的 CUDA 匹配的 vLLM 版本(如 CUDA 12.x 用 vllm==0.19.1),或先升级 NVIDIA 驱动。
---
## 2. vLLM 推理引擎架构
### 整体架构
FunASR 的 vLLM 集成将 ASR 模型拆分为两部分独立运行：
```
┌──────────────────────────────────────────────────────────────┐
│ FunASR + vLLM 推理架构 │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────── PyTorch (单 GPU) ───────────────┐ │
│ │ │ │
│ │ Audio ──→ Frontend ──→ Audio Encoder ──→ Adaptor │
│ │ (fbank) (SenseVoice/ (Transformer/ │
│ │ Whisper) MLP) │
│ │ │ │
│ │ ▼ │
│ │ Audio Embeddings │
│ │ │ │
│ │ Text Prompt ──→ Tokenize ──→ Embed │
│ │ (system/user/ │ │
│ │ hotwords/language) │ │
│ │ ▼ │
│ │ [Concat Embeddings] │
│ └─────────────────────────────────┼─────────────┘ │
│ │ │
│ ▼ EmbedsPrompt │
│ ┌─────────────── vLLM Engine ────────────────────┐ │
│ │ │ │
│ │ PagedAttention + Continuous Batching │ │
│ │ KV Cache 管理 + CUDA Graph │ │
│ │ Tensor Parallel (多卡) │ │
│ │ │ │
│ │ Qwen3-0.6B / Llama-2B (LLM 解码) │ │
│ │ │ │
│ └────────────────────┬───────────────────────────┘ │
│ │ │
│ ▼ │
│ Generated Text │
│ │ │
│ ┌────────────────────┼──────────────────────────┐ │
│ │ (可选) CTC Decoder ──→ Forced Alignment │ │
│ │ ──→ 字级别时间戳 │ │
│ └───────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
### 为什么用 vLLM？
| 特性 | PyTorch generate() | vLLM |
|------|-------------------|------|
| KV Cache 管理 | 固定分配，浪费显存 | PagedAttention，按需分配 |
| 批处理 | 需手动 padding | Continuous Batching，自动调度 |
| CUDA 优化 | 无 | CUDA Graph + 算子融合 |
| 多卡并行 | 手动实现 | Tensor Parallel 一行配置 |
| 吞吐量 | RTFx ~20 | **RTFx 340+** |
### 支持模型
| 模型 | LLM 部分 | audio encoder | vLLM 加速 |
|------|---------|---------------|-----------|
| **Fun-ASR-Nano** | Qwen3-0.6B | SenseVoice | ✓ 21.7x |
| **GLM-ASR-Nano** | Llama-2B | Whisper-like | ✓ 7.6x |
| LLMASR | Qwen/Vicuna | Whisper | ✓ |
| Paraformer | 无 LLM | — | ✗ 非自回归 |
| SenseVoice | 无 LLM | — | ✗ encoder-decoder |
### 关键实现细节
1. **权重分离**：从 `model.pt` 提取 LLM 权重，转为 HuggingFace 格式供 vLLM 加载
2. **EmbedsPrompt**：直接把**已算好的 embedding 向量**（而非通常的 token ID）作为 prompt 送入 vLLM（开关 `enable_prompt_embeds=True`）。Fun-ASR-Nano 必须用它，因为音频经 adaptor 得到的是连续向量、不是 token，需把音频 embedding 与文本 embedding 在序列维拼接后整体送入 vLLM
3. **use_low_frame_rate**：Fun-ASR-Nano 的 adaptor 输出需按公式截断到正确 token 数（一致性关键）
4. **batch encode**：多条音频通过 `extract_fbank` → `audio_encoder` → `audio_adaptor` 一次前向
5. **CTC 时间戳**：保留 encoder_out，生成文本后做 forced alignment 得到字级别时间
---
## 3. 离线 SDK 推理
适用于大规模音频转写、离线批量处理。vLLM 的批处理能力在此场景优势最大。
### 设计原理
离线 SDK 推理将 ASR 流水线拆分为两阶段独立执行：
```
┌─────────────────────────────────────────────────────────────────────┐
│ 阶段 1: 音频编码（PyTorch, 单 GPU） │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ 音频文件列表 ──→ 分组（每 8 条）──→ Frontend(Fbank) │
│ │ │ │
│ │ ▼ │
│ │ SenseVoice Encoder │
│ │ │ │
│ │ ▼ │
│ │ Audio Adaptor │
│ │ (dim 转换 + low_frame_rate 截断) │
│ │ │ │
│ └─── 共享文本 prompt 预编码 ─────┐ ▼ │
│ (system/hotwords/language) │ audio_embeds │
│ │ │ │ │
│ ▼ │ ▼ │
│ prefix_emb ──→ [concat: prefix | audio | suffix] │
│ │ │
│ ▼ │
│ EmbedsPrompt（N 条） │
└──────────────────────────────────────────────┼──────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 阶段 2: LLM 解码（vLLM, 多 GPU Tensor Parallel） │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ EmbedsPrompt × N ──→ vLLM Continuous Batching │
│ (PagedAttention + CUDA Graph) │
│ │ │
│ ▼ │
│ Generated token_ids × N │
│ │ │
│ ▼ │
│ Decode + 后处理（去特殊标记、清洗） │
│ │ │
│ ▼ │
│ (可选) CTC Forced Alignment → 字级别时间戳 │
└─────────────────────────────────────────────────────────────────────┘
```
**关键设计决策：**
1. **权重分离**：首次运行时从 `model.pt` 提取 `llm.*` 前缀的权重，保存为 HuggingFace safetensors 格式供 vLLM 加载（缓存到 `Qwen3-0.6B-vllm/` 目录）
2. **Embedding 拼接**：文本 prompt 通过 LLM 的 `embed_tokens` 层编码为 embedding，与音频 adaptor 输出在序列维度拼接：`[prefix_emb | audio_emb | suffix_emb]`，以 `EmbedsPrompt` 形式送入 vLLM
3. **Low Frame Rate 截断**：adaptor 输出需按公式 `fake_token_len = ((((fbank_len - 3 + 2) // 2 - 3 + 2) // 2) - 1) // 2 + 1` 截断到正确长度，确保与 PyTorch 训练时一致
4. **批量音频编码**：多条音频按 batch_size=8 分组通过 encoder + adaptor 前向，减少 GPU kernel launch 开销
5. **文本 prompt 共享**：同一批次内 hotwords/language 相同时，prefix_emb 和 suffix_emb 只计算一次
6. **CTC 时间戳**：保留 encoder_out，LLM 生成文本后做 forced alignment 得到字级别时间
**为什么比 PyTorch generate() 快？**
| 维度 | PyTorch | vLLM |
|------|---------|------|
| KV Cache | 固定预分配（浪费显存） | PagedAttention 按需分配 |
| 批处理 | 需手动 padding 对齐 | Continuous Batching 自动调度 |
| CUDA | 逐 sample 串行 | CUDA Graph + 算子融合 |
| 多卡 | 需手动实现 | Tensor Parallel 一行配置 |
| 结果 | RTFx ~20 | **RTFx 340+**（16倍加速） |
### 通用接口（推荐）
```python
from funasr.auto.auto_model_vllm import AutoModelVLLM
model = AutoModelVLLM(
model="FunAudioLLM/Fun-ASR-Nano-2512",
hub="ms", # 或 "hf"
tensor_parallel_size=2, # 多卡并行
gpu_memory_utilization=0.8,
)
results = model.generate(
["audio1.wav", "audio2.wav"],
language="中文",
hotwords=["张三", "北京"],
)
for r in results:
print(f"[{r['key']}] {r['text']}")
```
### 直接接口
```python
from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
engine = FunASRNanoVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
tensor_parallel_size=4,
)
results = engine.generate(
inputs="wav.scp", # 支持 scp/jsonl/文件列表
hotwords=["开放时间"],
language="中文",
max_new_tokens=512,
)
```
### 命令行
```bash
cd examples/industrial_data_pretraining/fun_asr_nano
# 单文件
python demo_vllm.py --input audio.wav --language 中文
# 批量 + 多卡
python demo_vllm.py --input wav.scp --tensor-parallel-size 4 --batch-size 32
# 带热词 + 保存结果
python demo_vllm.py --input audio.wav --hotwords 张三 北京 --output results.jsonl
```
---
## 4. 流式 SDK 推理
将音频按 720ms chunk 逐步处理，输出逐步稳定的识别结果。适用于 SDK 集成实时字幕场景。
### 设计原理
```
音频流（720ms chunks）
│ 累积重编码（每个 chunk 包含从头到当前的全部音频）
▼
┌──────────────────────┐
│ Stage 1: 前 10 chunk │ ← 无 prev_text，批量生成
│ 找到稳定输出 │
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Stage 2: 后续 chunk │ ← 用稳定输出作 prev_text
└──────────┬───────────┘
▼
每个 chunk: [fixed 区域（确认）] + [8字 unfixed（可能变）]
```
### 用法
```python
from funasr.models.fun_asr_nano.inference_vllm_streaming import FunASRNanoStreamingVLLM
engine = FunASRNanoStreamingVLLM.from_pretrained(
model="FunAudioLLM/Fun-ASR-Nano-2512",
chunk_ms=720,
rollback_chars=8,
)
for result in engine.streaming_generate("audio.wav", language="中文"):
if result["is_final"]:
print(f"最终: {result['text']}")
else:
print(f"[{result['audio_duration_ms']:.0f}ms] 确认: {result['fixed_text']}")
```
**注意：EmbedsPrompt 下不能用 `repetition_penalty`。** 此时 prompt 是 embedding 向量、没有对应的 token ID，而 `repetition_penalty` 要靠 prompt 的 token ID 在 logits 上给已出现的词降分；用在 EmbedsPrompt 上会**索引越界、触发 CUDA device-side assert**。
### 生产 API 稳定性清单
把 `AutoModelVLLM` 封装成长驻 API 服务时，请隔离每次请求的状态，并固定安全的解码默认值：
```python
common = dict(
language="auto",
temperature=0.0,
repetition_penalty=1.0,
max_new_tokens=200,
)
for _ in range(2):
results = model.generate(["vad_segment_01.wav", "vad_segment_02.wav"], **common)
print([r["text"] for r in results])
```
如果同一个音频第一次请求正常、第二次请求开始重复：
1. 先把 API 层拿掉，用相同 VAD 分段跑上面的最小脚本。
2. 如果最小脚本稳定，优先检查 API 封装是否复用了请求级变量、上一轮 VAD 分段列表、上一轮 `results` 或累积文本。
3. 如果最小脚本也重复，再记录完整的 `funasr`、`vllm`、`torch` 版本，以及第一次和第二次输出文本，再调整其它解码参数。
不要通过调大 `repetition_penalty` 来压制 Fun-ASR-Nano vLLM 重复输出；prompt-embeds 路径应保持中性值 `1.0`。
### 输出特性
| 累积音频 | 输出质量 |
|---------|---------|
| < 1.5s | 空或噪声 |
| 1.5-3.0s | 部分正确 |
| > 3.0s | 准确输出 |
---
## 5. 离线语音识别服务
### 5.1 服务架构
```
客户端 serve_vllm.py
│ │
│── HTTP/OpenAI/WebSocket ──────────────→│
│ │
│ ┌────┴────────────────────────┐
│ │ 1. 接收完整音频文件 │
│ │ 2. 动态 VAD 分段（≤60s/段） │
│ │ 3. vLLM batch 推理所有段 │
│ │ 4. CTC 时间戳（逐字） │
│ │ 5. 说话人分离（可选） │
│ └────┬────────────────────────┘
│ │
│←── JSON 结果 ─────────────────────────│
```
**特点**：
- 音频完整到达后处理，适合文件转写
- 动态 VAD 保留长段（≤60s），减少边界切割损失
- batch 推理所有 VAD 段，吞吐量高
- 自动输出字级别时间戳
- SPK 说话人分离默认关闭，客户端可开启
### 5.2 启动服务
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_vllm.py \
--port 8899 \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--gpu-memory-utilization 0.5
```
> **关于 `CUDA_VISIBLE_DEVICES`**：这是[vllm的一个环境变量](https://docs.vllm.ai/en/v0.4.3/serving/env_vars.html) ，示例中的 `=0` 只是"用第 0 张卡"的示例值，**不是固定写法**，它选择本进程可见的 GPU（编号同 `nvidia-smi`），单卡机器也不需要设置。
>
> **单卡多实例**：0.6B / 1.7B 这类小模型一张卡可起多个实例，多进程可都指向同一张卡（如都 `=0`）+ MPS 共享；分卡则进程 A `=0`、B `=1`（见 §6.7）。
### 5.3 协议一：HTTP REST — `POST /asr`
功能最全的接口，支持 SPK、时间戳、热词。
**请求**：`multipart/form-data`
| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
| `file` | file | 必填 | 音频文件（wav/mp3/flac） |
| `language` | string | None | 语种（"中文"/"English"/...），None 为自动 |
| `hotwords` | string | "" | 热词，逗号分隔 |
| `spk` | bool | false | 是否开启说话人分离 |
| `timestamp` | bool | true | 是否输出字级别时间戳 |
**响应**：
```json
{
"text": "完整识别文本",
"segments": [
{
"text": "段文本",
"start": 1.7,
"end": 14.8,
"speaker": "SPK0",
"words": [
{"word": "砸", "start": 2.02, "end": 2.08},
{"word": "了", "start": 2.26, "end": 2.32}
]
}
],
"duration": 227.4,
"processing_time": 3.422,
"rtf": 0.015
}
```
**客户端示例**：
```bash
# cURL
curl -X POST http://localhost:8899/asr \
-F "file=@meeting.wav" -F "language=中文" -F "spk=true"
```
```python
# Python requests
import requests
resp = requests.post("http://localhost:8899/asr",
files={"file": open("audio.wav", "rb")},
data={"language": "中文", "spk": "true"})
result = resp.json()
```
```javascript
// JavaScript fetch
const form = new FormData();
form.append("file", audioBlob, "audio.wav");
form.append("language", "中文");
form.append("spk", "true");
const resp = await fetch("http://localhost:8899/asr", { method: "POST", body: form });
const result = await resp.json();
```
### 5.4 协议二：OpenAI Whisper 兼容 — `POST /v1/audio/transcriptions`
兼容 OpenAI Whisper API 标准，可直接用 OpenAI SDK 接入。
**请求**：`multipart/form-data`
| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
| `file` | file | 必填 | 音频文件 |
| `model` | string | "fun-asr-nano" | 模型名（兼容字段） |
| `language` | string | None | 语种 |
| `response_format` | string | "json" | "json" / "text" / "verbose_json" |
| `timestamp_granularities` | string | "word" | "word" / "segment" |
| `spk` | bool | false | 说话人分离（FunASR 扩展字段） |
**响应**（`verbose_json`）：
```json
{
"task": "transcribe",
"language": "zh",
"duration": 5.17,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"segments": [
{
"id": 0, "start": 0.0, "end": 5.15,
"text": "我一直没有照顾孩子，但是我想要抚养权。",
"words": [{"word": "我", "start": 0.42, "end": 0.48}, ...]
}
]
}
```
**客户端示例**：
```python
# OpenAI SDK（推荐）
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8899/v1", api_key="none")
result = client.audio.transcriptions.create(
model="fun-asr-nano",
file=open("audio.wav", "rb"),
response_format="verbose_json",
)
print(result.text)
```
```bash
# cURL
curl -X POST http://localhost:8899/v1/audio/transcriptions \
-F "file=@audio.wav" -F "model=fun-asr-nano" -F "response_format=verbose_json"
```
### 5.5 协议三：WebSocket — `ws://host:port/ws`
离线服务的 WebSocket 接口，发送完整音频后获取结果。STOP 时自动进行说话人聚类，结果中包含 `spk` 字段。
**客户端 → 服务端**：
| 消息 | 说明 |
|------|------|
| `"START"` | 开始会话 |
| `"LANGUAGE:中文"` | 设置语种（可选） |
| `"HOTWORDS:词1,词2"` | 设置热词（可选） |
| `[binary]` | PCM16 16kHz mono 音频数据 |
| `"STOP"` | 结束，请求识别结果 |
**服务端 → 客户端**：
```json
{"event": "started"}
{"event": "language_set", "language": "中文"}
{"sentences": [{"text":"...","start":..,"end":..}], "is_final": true, "duration_ms": 5170}
{"event": "stopped"}
```
**客户端示例**：
```python
import asyncio, websockets, json, numpy as np, soundfile as sf
async def offline_ws(audio_path):
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:8899/ws") as ws:
await ws.send("START")
await ws.recv()
await ws.send("LANGUAGE:中文")
await ws.recv()
# 发送完整音频
await ws.send(pcm.tobytes())
await ws.send("STOP")
# 接收结果
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(offline_ws("audio.wav"))
```
---
## 6. 流式语音识别服务
### 6.1 服务架构
```
客户端（麦克风/音频流） serve_realtime_ws.py
│ │
│── WebSocket PCM16 16kHz ────────────→│
│ (每帧 ~100ms，持续发送) │
│ │
│ ┌────┴─────────────────────────┐
│ │ 实时循环： │
│ │ ├─ 动态 VAD（60ms chunk） │
│ │ ├─ 检测到端点 → vLLM 解码 │
│ │ ├─ 未结束 → partial 预览 │
│ │ └─ 说话人流式分配 │
│ └────┬─────────────────────────┘
│ │
│←── JSON 实时推送 ───────────────────│
```
**特点**：
- 音频逐帧到达，边收边处理
- 基于 VAD 端点自然分句
- 确认段文字锁定不变，partial 实时更新
- 可选流式说话人分配（`--enable-spk`）+ STOP 时全局重聚类
- 首字延迟 ~480ms
### 6.2 启动服务
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 --hotword-file 热词列表
```
多客户端或长时间连续语音场景，建议先限制 partial 预览窗口并适当降低刷新频率：
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
说话人分离默认关闭；只有确实需要 `spk` 字段时再加 `--enable-spk`。
### 6.3 WebSocket 协议
**连接**：`ws://host:10095`
**客户端 → 服务端**：
| 消息 | 格式 | 说明 |
|------|------|------|
| 开始 | `"START"` | 初始化 session |
| 热词 | `"HOTWORDS:词1,词2"` | 可选 |
| 语种 | `"LANGUAGE:中文"` | 可选 |
| 音频 | `binary` | PCM16 16kHz mono |
| 结束 | `"STOP"` | 最终解码；启用 `--enable-spk` 时会做 SPK 重聚类 |
**服务端 → 客户端**：
```json
{"event": "started"}
{"sentences": [{"text":"你好","start":300,"end":1200}], "partial": "世界", "is_final": false}
{"sentences": [...], "is_final": true}
{"event": "stopped"}
```
**字段**：`sentences[]` = 已锁定句子，`partial` = 当前正在说的临时文本（可能变化），`partial_start_ms` = 当前 `partial` 对应音频窗口的起点，`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
**时序**：
```
Client Server
│── START ───────→│
│←─ started ──────│
│── [audio] ─────→│
│←─ {partial} ────│ # partial 的原理是注意事项见 6.5
│── [audio] ─────→│
│←─ {sentences+partial} ─│ (VAD 切了一句)
│── STOP ────────→│
│←─ {is_final:true} ────│
│←─ stopped ─────│
```
### 6.4 客户端调用
**Python CLI**：
```bash
python client_python.py --server ws://localhost:10095 --mic
python client_python.py --server ws://localhost:10095 --file audio.wav
```
**浏览器**：打开 `client_mic.html`
**自定义 Python**：
```python
import asyncio, websockets, numpy as np, json
async def stream(audio_path):
import soundfile as sf
audio, sr = sf.read(audio_path)
pcm = (audio * 32768).astype(np.int16)
async with websockets.connect("ws://localhost:10095") as ws:
await ws.send("START")
await ws.recv()
for i in range(0, len(pcm), 1600):
await ws.send(pcm[i:i+1600].tobytes())
await asyncio.sleep(0.05)
await ws.send("STOP")
async for msg in ws:
data = json.loads(msg)
if data.get("is_final"):
for s in data["sentences"]:
print(f"[{s['start']/1000:.1f}s] {s['text']}")
break
asyncio.run(stream("audio.wav"))
```
### 6.5 partial 预览机制与长句特性
**partial 是什么、怎么产生的**
流式服务在用户说话过程中会周期性地（`serve_realtime_ws.py` 默认 `decode_interval≈0.48s`）对"当前这句话从句首到现在"的音频解码一次，输出**临时文字**（即协议里的 `partial` 字段，可被后续刷新覆盖），直到 VAD 判定句尾才锁定进 `sentences`。这让用户边说边看到字。
> 注：`serve_vllm.py`（§5）的 `/ws` **没有 partial**、只在句尾返回；要实时预览请用 `serve_realtime_ws.py`。
**前端渲染原则**
`partial` 只能当作“可替换预览”，不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开：
```js
const committed = data.sentences.map((s) => s.text).join("");
const preview = data.partial || "";
render(committed + preview);
```
如果启用了 `--partial-window-sec`，`partial_start_ms` 可能随着窗口向前滑动；这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域，只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。
**原理：为什么每次 partial 都从句首整段重编**
Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时，先前那些帧的上下文随之改变，**之前算出的编码不再成立**，因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧"，只能把"句首→当前"的整段重新过一遍编码器。
**由此带来的特性：长句的 partial 会越来越慢（O(L²)）**
正因每次都从句首重编，一句话越长，单次 partial 要编的音频越长、刷新次数也越多——**总编码量随句长二次增长**。实测一句约 29s 的连续发言会被完整重编十余次，单次 encoder 耗时从几十毫秒爬到数百毫秒。（§4 SDK 流式"每个 chunk 包含从头到当前的全部音频"是同一机制，长文件同理。）
**使用建议**
- 正常对话语音有自然停顿，VAD 会把它切成一句句较短的语音，每句 partial 的开销自然受限，**通常无需关注**。
- 只有**超长、不停顿的连续语音**（如长篇朗读）会让单句不断变长、partial 预览逐渐变慢。`serve_realtime_ws.py` 默认用 `--partial-window-sec 15` 限制临时预览窗口；多客户端或连续独白压测时可降到 `8-10`，并把 `--decode-interval` 提高到 `0.8-1.0`。这只影响临时 `partial`，VAD 锁定句和 STOP 最终结果仍走完整音频。
### 6.6 说话人分离（SPK）的代价与开关
`serve_realtime_ws.py` 默认**不加载** SPK 模型。只有启动时显式加 `--enable-spk`，才会加载 `--spk-model`（默认 `iic/speech_eres2netv2_sv_zh-cn_16k-common`）并在流式中对每个 VAD 完成句调用一次说话人分配。需要注意：
- **Fun-ASR-Nano 上 SPK 效果有限**（见 #2944），多数实时 ASR 场景并不需要说话人分离。
- **流式 SPK 代价高且随会话变长**：每句对**全部历史 embedding** 做一次全量重聚类（**O(N²)**，会话越长每句越贵），且**同步阻塞事件循环**；而会话结束时还会**全量重聚一遍**，流式期间每句的聚类结果会被最终结果覆盖——对最终输出而言属于重复计算。长会话 + 高并发下尤其明显。
- **建议**：多客户端实时转写优先保持默认关闭；确需 diarization 时再加 `--enable-spk`，并以 STOP 后的最终 `spk` 标签为准。
### 6.7 生产并发与多进程部署
`serve_realtime_ws.py` 是**单 asyncio 事件循环**服务：`decode()`（定时 partial）与 `add_audio()`（VAD 句尾触发解码）都**同步阻塞**整个事件循环——任一路在解码时，其余连接全部暂停收发。因此：
- **单进程并发墙来自事件循环串行，不是 GPU 算力**。
- **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**：每个进程有独立的 GIL 与 CUDA 上下文，绕开单循环串行；MPS 让多进程真正并发共享 GPU、填满空闲算力；nginx 在多个 WebSocket 后端间轮询。超过单卡余量后，再横向加卡（每卡一实例 + 负载均衡）。
- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度；而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码，无法自然形成大 batch，所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流，多个轻量 PyTorch 进程有时更容易在一张卡上排布；使用 vLLM 时请按真实话务压测，先配合较低的 `--gpu-memory-utilization` 和多进程服务，而不是假设一个 vLLM 进程就应该承载所有连接。
- **可持续并发没有通用的"支持 N 路"数字**：决定上限的不是在线连接数，而是**同一时刻有多少路正在"说话"**——每路只要在说，就每约 1 秒触发一次 partial 解码，全部串行在那个单事件循环上。它主要随两点变化：**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声，同时解码的路数远少于在线连接数；连续独白则几乎每路都在持续解码，负载高得多。**② 句长**——句子越长，单次 partial 的编码越贵（见 6.5 的 O(L²)），同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS"，在接近真实 turn-taking 的负载下可稳定支撑数十路，而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务（句长、停顿、是否连续说话）压测确定，别把别处测出的某个并发数当成自己的规格。
---
## 7. 动态 VAD
fsmn-vad 默认启用动态静音阈值。离线和流式使用不同配置。
| 累积时长 | 离线（保留长段 ≤60s） | 流式（平衡延迟） |
|---------|-------------------|----------------|
| ≤ 5s | 2000ms | 2000ms |
| 5-10s | 2000ms | 1500ms |
| 10-15s | 1000ms | 1000ms |
| 15-20s | 1000ms | 800ms |
| 20-30s | 800ms | 800ms |
| 30-45s | 600ms | 400ms |
| 45-60s | 200-400ms | 100ms |
| > 60s | 100ms | 100ms |
离线倾向保留长段减少边界损失；流式更快收紧以降低延迟。
### 自定义
```python
model.generate(input="audio.wav", silence_schedule=[(5000,1500), (20000,800), (float('inf'),300)])
```
> GLM-ASR 不支持长段，使用时传 `dynamic_silence=False`。
---
## 8. API 参考
| 参数 | AutoModelVLLM | serve_vllm.py | serve_realtime_ws.py |
|------|--------------|---------------|---------------------|
| model | ✓ | --model | --model |
| gpu_memory_utilization | ✓ | --gpu-memory-utilization | --gpu-memory-utilization |
| tensor_parallel_size | ✓ | — | --tensor-parallel-size |
| max_model_len | ✓ | --max-model-len | --max-model-len |
| language | generate() 参数 | API 参数 | --language / LANGUAGE: |
| hotwords | generate() 参数 | API 参数 | --hotword-file / HOTWORDS: |
---
## 9. FAQ
**Q: 离线还是流式？**
完整文件 → 离线（高吞吐）。麦克风/直播 → 流式（低延迟）。
**Q: GLM-ASR 用动态 VAD？**
不支持长段推理，用 `dynamic_silence=False`。
**Q: SPK 性能影响？**
RTFx 102 → 46。CER 不变。默认关闭。
**Q: 二次开发入口？**
离线：`serve_vllm.process_audio()` / `FunASRNanoVLLM.generate()`
流式：`serve_realtime_ws.RealtimeASRSession`
**Q: 首次慢？**
vLLM 初始化 60-90s，之后即时。