wehub/modelscope--funasr
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:10 +08:00
commit c397331b1e
3684 changed files with 990993 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/ISSUE_TEMPLATE/ask_questions.md
+41
View File
@@ -0,0 +1,41 @@
---
name: ❓ Questions/Help
about: Ask for help after checking existing issues and docs
labels: 'question, needs triage'
---
Notice: In order to resolve issues efficiently, please follow the template.
（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节。）
## Before asking
1. Search existing issues: https://github.com/modelscope/FunASR/issues
2. Search the docs: https://modelscope.github.io/FunASR/
3. Check the README quick start and deployment section.
## Question
<!-- What are you trying to do, and where are you blocked? -->
## Code or command
```bash
```
## What have you tried?
<!-- Include links to docs or examples you followed, plus the exact error/result. -->
## Environment
- OS:
- Python version:
- FunASR version:
- ModelScope version:
- PyTorch / torchaudio version:
- Install method (`pip`, source, Docker):
- Device (`cuda`, `cpu`, `mps`):
- GPU model:
- CUDA/cuDNN version:
- Docker image tag, if used:
.github/ISSUE_TEMPLATE/bug_report.md
+64
View File
@@ -0,0 +1,64 @@
---
name: 🐛 Bug Report
about: Submit a reproducible bug report to help us improve FunASR
labels: 'bug, needs triage'
---
Notice: In order to resolve issues efficiently, please follow the template and include reproducible details.
（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充可复现细节。）
## 🐛 Bug
<!-- A clear and concise description of the bug. -->
## To Reproduce
Steps to reproduce the behavior. Always include the exact command you ran.
1. Install with: `...`
2. Run: `...`
3. See error: `...`
## Code sample
<!-- Paste the smallest code sample that still reproduces the issue. -->
```python
```
## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
## Error logs
<!-- Paste the full traceback or relevant logs. Remove private tokens and paths. -->
```text
```
## Environment
- OS:
- Python version:
- FunASR version:
- ModelScope version:
- PyTorch / torchaudio version:
- Install method (`pip`, source, Docker):
- Device (`cuda`, `cpu`, `mps`):
- GPU model:
- CUDA/cuDNN version:
- Docker image tag, if used:
## Audio details
If the audio cannot be shared, please describe:
- Duration:
- Sample rate:
- Format:
- Language/dialect:
- Speaker count:
- Background noise/music:
.github/ISSUE_TEMPLATE/config.yaml
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: FunASR documentation
url: https://modelscope.github.io/FunASR/
about: Read installation, tutorial, deployment, benchmark, and API docs.
- name: GitHub Discussions
url: https://github.com/modelscope/FunASR/discussions
about: Share ideas, showcase usage, and ask open-ended community questions.
- name: Hugging Face models
url: https://huggingface.co/funasr
about: Browse FunASR checkpoints on Hugging Face.
.github/ISSUE_TEMPLATE/deployment_help.md
+45
View File
@@ -0,0 +1,45 @@
---
name: 🚀 Deployment Help
about: Get help with API server, WebSocket, Docker, vLLM, Triton, Android, or agent integration
labels: 'deployment, needs triage'
---
## Deployment target
- [ ] OpenAI-compatible API server
- [ ] WebSocket streaming service
- [ ] Docker runtime
- [ ] vLLM acceleration
- [ ] Triton
- [ ] Android / mobile
- [ ] Browser / Web UI
- [ ] MCP / agent integration
- [ ] Other:
## Goal
<!-- What user-facing service or workflow are you trying to deploy? -->
## Command or config
```bash
```
## Error or unexpected behavior
```text
```
## Environment
- OS:
- Python version:
- FunASR version:
- Install method (`pip`, source, Docker):
- Device (`cuda`, `cpu`, `mps`):
- GPU model and memory:
- CUDA/cuDNN version:
- Docker image tag, if used:
- Network endpoint / port:
.github/ISSUE_TEMPLATE/error_docs.md
+15
View File
@@ -0,0 +1,15 @@
---
name: 📚 Documentation/Typos
about: Report an issue related to documentation or a typo
labels: 'documentation, needs triage'
---
## 📚 Documentation
For typos and doc fixes, please go ahead and:
1. Create an issue.
2. Fix the typo.
3. Submit a PR.
Thanks!
.github/ISSUE_TEMPLATE/feature_request.md
+27
View File
@@ -0,0 +1,27 @@
---
name: ✨ Feature Request
about: Suggest a model, language, runtime, deployment, benchmark, or documentation improvement
labels: 'enhancement, needs triage'
---
## Use case
<!-- What are you trying to build, deploy, evaluate, or research with FunASR? -->
## Proposed solution
<!-- Describe the desired model/API/runtime/doc/benchmark behavior. -->
## Alternatives or workarounds
<!-- What workaround are you using today? What other tools did you consider? -->
## Expected impact
<!-- Who benefits: new users, production deployers, researchers, agent builders, educators, etc.? -->
## Willing to contribute?
- [ ] I can submit a PR
- [ ] I can help test
- [ ] I can provide data, logs, or benchmark results
.github/ISSUE_TEMPLATE/migration_benchmark.md
+71
View File
@@ -0,0 +1,71 @@
---
name: 📊 Migration Benchmark Report
about: Share FunASR results when comparing against Whisper, OpenAI audio APIs, or cloud ASR
labels: 'benchmark, showcase, needs triage'
---
Thanks for benchmarking FunASR on your own audio. Migration reports help new users decide whether FunASR fits their language, domain, hardware, and deployment constraints.
If you need help debugging a failure, please use Bug Report or Deployment Help instead.
## Summary
<!-- One or two sentences: what did you compare, and what was the headline result? -->
## Baseline
- Baseline ASR (`Whisper`, `Whisper large-v3`, cloud provider, internal system, other):
- Baseline runtime or API:
- Baseline hardware or pricing tier:
## FunASR setup
- FunASR version:
- Model(s):
- Runtime path (`Python API`, `funasr-server`, `OpenAI API`, `Docker`, `WebSocket`, `vLLM`, other):
- Device (`cuda`, `cpu`, `mps`):
- GPU / CPU:
- CUDA / PyTorch versions:
- Command or script used:
```bash
```
## Audio set
- Number of files:
- Total audio duration:
- Language(s) / dialect(s):
- Domain (`meeting`, `call-center`, `subtitle`, `lecture`, `media`, `noisy field audio`, other):
- Speaker count range:
- Sample rate / format:
- Can any sample be shared publicly? yes/no
## Results
<!-- Paste the aggregate section from examples/migration/benchmark_funasr.py summary.md, or summarize your own measurement. -->
```text
```
## Quality notes
<!-- Share WER/CER if available, or human-review notes about names, numbers, punctuation, timestamps, speaker labels, and domain terms. -->
## Operational notes
- Model download / warmup time:
- Steady-state throughput:
- Memory usage:
- Failed files or error rate:
- Deployment blockers:
## Links or artifacts
<!-- Public repo, demo, blog, benchmark sheet, screenshot, anonymized transcript snippet, or architecture diagram. Do not include private audio, credentials, or customer data. -->
## What should FunASR improve next?
<!-- Missing docs, rough edges, model gaps, deployment friction, benchmark needs, etc. -->
.github/ISSUE_TEMPLATE/showcase.md
+56
View File
@@ -0,0 +1,56 @@
---
name: 🎤 Showcase / Production Story
about: Share how you use FunASR in a demo, product, benchmark, research project, or internal workflow
labels: 'showcase, needs triage'
---
Thanks for sharing your FunASR use case. Concrete usage reports help new users choose the right path and help maintainers prioritize docs, examples, models, and deployment work.
If this is a support question, please use Bug Report or Deployment Help instead.
## Summary
<!-- What did you build or evaluate with FunASR? One or two sentences is enough. -->
## Scenario
- [ ] Local transcription demo
- [ ] Private OpenAI-compatible speech API
- [ ] Agent / MCP / voice-input workflow
- [ ] Streaming ASR / live captions
- [ ] Subtitle generation
- [ ] Batch transcription / archive processing
- [ ] Benchmark or migration from Whisper/cloud ASR
- [ ] Research / education
- [ ] Production deployment
- [ ] Other:
## Deployment details
- FunASR version:
- Model(s):
- Device (`cuda`, `cpu`, `mps`):
- GPU / CPU:
- Runtime path (`Python API`, `funasr-server`, `WebSocket`, `Docker`, `vLLM`, `MCP`, other):
- Audio language/domain:
- Approximate audio duration or traffic:
## Results
<!-- Share speed, accuracy/CER/WER, latency, cost, CPU/GPU usage, or qualitative results if available. -->
```text
```
## Links or screenshots
<!-- Public demo, article, repo, benchmark report, screenshot, or architecture diagram. Do not include private audio, credentials, or customer data. -->
## What was easy?
<!-- Which docs, examples, or APIs helped most? -->
## What should be improved?
<!-- Missing docs, rough edges, model gaps, deployment friction, benchmark needs, etc. -->
.github/PULL_REQUEST_TEMPLATE.md
+28
View File
@@ -0,0 +1,28 @@
## Summary
<!-- What changed and why? -->
## Type of change
- [ ] Bug fix
- [ ] Documentation
- [ ] Example or demo
- [ ] Runtime or deployment
- [ ] Benchmark or evaluation
- [ ] Model/training change
## Validation
<!-- Commands run, logs, screenshots, benchmark results, or why validation is not applicable. -->
- [ ] `python -m compileall funasr examples tests`
- [ ] Docs or links checked
- [ ] Runtime/deployment command tested
## User impact
<!-- Who benefits from this change? New users, production deployers, researchers, agent builders, etc. -->
## Notes for reviewers
<!-- Compatibility notes, risks, follow-up work, or review focus. -->
.github/workflows/build-llamacpp-binaries.yml
+135
View File
@@ -0,0 +1,135 @@
name: Build llama.cpp runtime binaries
# Cross-platform prebuilt binaries for the FunASR llama.cpp / GGUF runtime,
# like whisper.cpp's whisper-bin-*. Push a `runtime-llamacpp-v*` tag to publish
# a GitHub Release with the binaries; or run manually (workflow_dispatch) to test
# the build without creating a release.
on:
workflow_dispatch:
push:
tags:
- 'runtime-llamacpp-v*'
permissions:
contents: write
jobs:
build:
name: build-${{ matrix.name }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
name: linux-x64
cmake_flags: >-
-DGGML_NATIVE=OFF
-DGGML_AVX=OFF
-DGGML_AVX2=OFF
-DGGML_AVX_VNNI=OFF
-DGGML_AVX512=OFF
-DGGML_AVX512_VBMI=OFF
-DGGML_AVX512_VNNI=OFF
-DGGML_AVX512_BF16=OFF
-DGGML_FMA=OFF
-DGGML_F16C=OFF
-DGGML_BMI2=OFF
- os: ubuntu-latest
name: linux-x64-avx2
cmake_flags: >-
-DGGML_NATIVE=OFF
-DGGML_AVX=ON
-DGGML_AVX2=ON
-DGGML_AVX_VNNI=OFF
-DGGML_AVX512=OFF
-DGGML_AVX512_VBMI=OFF
-DGGML_AVX512_VNNI=OFF
-DGGML_AVX512_BF16=OFF
-DGGML_FMA=ON
-DGGML_F16C=ON
-DGGML_BMI2=ON
- os: ubuntu-24.04-arm
name: linux-arm64
cmake_flags: -DGGML_NATIVE=OFF
- os: macos-latest
name: macos-arm64
cmake_flags: -DGGML_NATIVE=OFF
- os: windows-latest
name: windows-x64
cmake_flags: >-
-DGGML_NATIVE=OFF
-DGGML_AVX=OFF
-DGGML_AVX2=OFF
-DGGML_AVX_VNNI=OFF
-DGGML_AVX512=OFF
-DGGML_AVX512_VBMI=OFF
-DGGML_AVX512_VNNI=OFF
-DGGML_AVX512_BF16=OFF
-DGGML_FMA=OFF
-DGGML_F16C=OFF
-DGGML_BMI2=OFF
- os: windows-latest
name: windows-x64-avx2
cmake_flags: >-
-DGGML_NATIVE=OFF
-DGGML_AVX=ON
-DGGML_AVX2=ON
-DGGML_AVX_VNNI=OFF
-DGGML_AVX512=OFF
-DGGML_AVX512_VBMI=OFF
-DGGML_AVX512_VNNI=OFF
-DGGML_AVX512_BF16=OFF
-DGGML_FMA=ON
-DGGML_F16C=ON
-DGGML_BMI2=ON
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
working-directory: runtime/llama.cpp
steps:
- uses: actions/checkout@v4
- name: Configure and build
run: |
# Release binaries must be portable across user machines, not tuned to
# the ephemeral CI runner CPU. A native ggml build may emit AVX512 or
# AVX-VNNI instructions and then crash with SIGILL on ordinary CPUs.
# Keep the default x64 package conservative, and publish explicit
# x64-avx2 assets for machines that support AVX2/FMA/F16C/BMI2.
cmake -B build -DCMAKE_BUILD_TYPE=Release ${{ matrix.cmake_flags }}
cmake --build build --config Release -j 2
- name: Package
run: |
mkdir -p pkg
cp build/bin/llama-funasr-* pkg/ 2>/dev/null || true
cp build/bin/Release/llama-funasr-* pkg/ 2>/dev/null || true
cp README.md download-funasr-model.sh pkg/ 2>/dev/null || true
echo "--- packaged binaries ---"; ls -la pkg
if [ "${{ runner.os }}" = "Windows" ]; then
7z a "funasr-llamacpp-${{ matrix.name }}.zip" ./pkg/*
else
tar czf "funasr-llamacpp-${{ matrix.name }}.tar.gz" -C pkg .
fi
- uses: actions/upload-artifact@v4
with:
name: funasr-llamacpp-${{ matrix.name }}
path: runtime/llama.cpp/funasr-llamacpp-${{ matrix.name }}.*
if-no-files-found: error
release:
needs: build
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v4
with:
path: dist
merge-multiple: true
- name: Create GitHub Release
env:
GH_TOKEN: ${{ github.token }}
run: |
gh release create "${{ github.ref_name }}" dist/* \
--repo "${{ github.repository }}" \
--title "FunASR llama.cpp runtime ${{ github.ref_name }}" \
--notes "Prebuilt self-contained binaries for the FunASR llama.cpp / GGUF runtime — SenseVoice, Paraformer and Fun-ASR-Nano with built-in FSMN-VAD (a whisper.cpp-style on-device ASR, strong on Chinese). Get a model with \`bash download-funasr-model.sh <sensevoice|paraformer|nano>\`, then run \`llama-funasr-cli\` / \`llama-funasr-sensevoice\` / \`llama-funasr-paraformer\`. Use the default x64 asset for maximum CPU compatibility; use the x64-avx2 asset on CPUs with AVX2/FMA/F16C/BMI2 for higher throughput. No Python, no build. Docs: runtime/llama.cpp/README.md"
.github/workflows/release-on-tag.yml
+21
View File
@@ -0,0 +1,21 @@
name: Release on tag
on:
push:
tags:
- "v*"
permissions:
contents: write
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Create GitHub Release with auto-generated notes
env:
GH_TOKEN: ${{ github.token }}
run: gh release create "${{ github.ref_name }}" --generate-notes --verify-tag --title "${{ github.ref_name }}"
.github/workflows/update-api-docs.yml
+33
View File
@@ -0,0 +1,33 @@
name: Update API Documentation
on:
push:
branches: [main]
paths:
- 'funasr/**/*.py'
- 'scripts/gen_api_docs.py'
- 'training.html'
- '.github/workflows/update-api-docs.yml'
jobs:
build-api-docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Generate API docs
run: python3 scripts/gen_api_docs.py
- name: Deploy to gh-pages
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./gh-pages-output
publish_branch: gh-pages
keep_files: true
.pre-commit-config.yaml
+6
View File
@@ -0,0 +1,6 @@
repos:
- repo: https://github.com/psf/black
rev: 24.4.0
hooks:
- id: black
args: ['--line-length=100'] # 示例参数，black默认使用4个空格缩进
Acknowledge.md
+10
View File
@@ -0,0 +1,10 @@
## Acknowledge
1. We borrowed a lot of code from [Kaldi](http://kaldi-asr.org/) for data preparation.
2. We borrowed a lot of code from [ESPnet](https://github.com/espnet/espnet). FunASR follows up the training and finetuning pipelines of ESPnet.
3. We referred [Wenet](https://github.com/wenet-e2e/wenet) for building dataloader for large scale data training.
4. We acknowledge [ChinaTelecom](https://github.com/zhuzizyf/damo-fsmn-vad-infer-httpserver) for contributing the VAD runtime.
5. We acknowledge [RapidAI](https://github.com/RapidAI) for contributing the Paraformer and CT_Transformer-punc runtime.
6. We acknowledge [AiHealthx](http://www.aihealthx.com/) for contributing the websocket service and html5.
7. We acknowledge [XVERSE](http://www.xverse.cn/index.html) for contributing the grpc service.
8. We acknowledge [blt](https://github.com/bltcn) for develop and deploy website.
CONTRIBUTING.md
+55
View File
@@ -0,0 +1,55 @@
# Contributing to FunASR
Thanks for helping improve FunASR. We especially welcome contributions that make the first successful transcription faster, improve deployment reliability, or make benchmarks easier to reproduce.
## High-impact areas
- **Quick start reliability:** installation notes, CPU/GPU/MPS compatibility, dependency fixes, and runnable examples.
- **Deployment recipes:** OpenAI-compatible API, WebSocket streaming, Docker, vLLM, Triton, Android, browser, and agent integration.
- **Benchmarks:** reproducible speed, WER/CER, memory, and hardware comparison scripts.
- **Model examples:** multilingual ASR, speaker diarization, punctuation, VAD, emotion recognition, hotwords, timestamps, and fine-tuning.
- **Documentation:** shorter paths from README to working code, clearer troubleshooting, and verified links.
## Development setup
```bash
git clone https://github.com/modelscope/FunASR.git
cd FunASR
python -m venv .venv
source .venv/bin/activate
pip install -e ./
```
For docs work:
```bash
pip install -U "funasr[docs]"
cd docs
make html
```
## Before opening a pull request
Run the checks that match your change. For Python-only changes, start with:
```bash
python -m compileall funasr examples tests
```
For docs-only changes, preview the Markdown or generated HTML and verify relative links. For runtime changes, include the exact command, image tag, device, and endpoint you validated.
## Pull request checklist
- The PR has a focused scope and explains the user-facing value.
- New examples include command lines and the expected output shape.
- Bug fixes include reproduction steps or a short failure-mode explanation.
- Deployment docs list hardware, OS, Python/CUDA versions, and network endpoints.
- Large model, dataset, audio, or video files are not committed directly.
## Issue reports
Please use the templates and include environment details, exact commands, logs, and whether the audio can be shared. If audio is private, describe duration, sample rate, language, speaker count, format, and noise level.
## Maintainer focus for 20k+ stars
When reviewing changes, prioritize work that helps new users reach a good result in under five minutes, helps teams deploy FunASR privately, or gives external writers a clear story to share.
Contribution.md
+17
View File
@@ -0,0 +1,17 @@
# Contributing to FunASR
Thanks for taking the time to contribute to FunASR.
The canonical contribution guide is now [CONTRIBUTING.md](./CONTRIBUTING.md). It covers the development setup, high-impact contribution areas, validation expectations, pull request checklist, and issue-reporting guidance.
## Quick links
- [Bug reports](https://github.com/modelscope/FunASR/issues/new?template=bug_report.md)
- [Questions and help](https://github.com/modelscope/FunASR/issues/new?template=ask_questions.md)
- [Documentation fixes](https://github.com/modelscope/FunASR/issues/new?template=error_docs.md)
- [Feature requests](https://github.com/modelscope/FunASR/issues/new?template=feature_request.md)
- [Pull requests](https://github.com/modelscope/FunASR/pulls)
## Maintainer priorities
The most valuable contributions help new users reach a successful transcription quickly, make private deployment easier, or make benchmark results easier to reproduce.
LICENSE
+20
View File
@@ -0,0 +1,20 @@
MIT License
Copyright (c) 2025 FunASR
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
MODEL_LICENSE
+107
View File
@@ -0,0 +1,107 @@
FunASR Model Open Source License Agreement
Version: 1.1
Copyright (C) [2023-2028] [Alibaba Group]. All rights reserved.
Thank you for choosing the FunASR open-source model. The FunASR open-source model includes a range of free and open industrial models for you to use, modify, share, and learn from.
To ensure better community collaboration, we have established the following agreement, and we hope you will read and comply with its terms.
Definitions
In this agreement, [FunASR Software] refers to FunASR open-source model weights and their derivatives, including finetuned models; [You] refers to individuals or organizations using, modifying, sharing, and learning from [FunASR Software].
2 License and Restrictions
2.1 License
You are free to use, copy, modify, and share [FunASR Software] under the terms of this agreement.
2.2 Restrictions
When using, copying, modifying, and sharing [FunASR Software], you must attribute the source and author information and retain relevant model names in [FunASR Software].
3 Responsibility and Risk
[FunASR Software] is provided for reference and learning purposes only, and Alibaba Group assumes no responsibility for any direct or indirect losses resulting from your use or modification of [FunASR Software]. You should assume all risks associated with using and modifying [FunASR Software].
4 Community Conduct Guidelines
4.1 Encouraged Behavior
The community welcomes developers and users to engage in discussions about [FunASR Software]. Participants are encouraged to interact in a friendly, polite, and respectful manner to foster constructive discussion and collaboration.
4.2 Prohibited Behavior
Individual or organizational users shall not engage in unjustified denigration, malicious smearing, or baseless insults against [FunASR Software]. Such behavior is considered a violation of the spirit of community cooperation. If a user is found to be engaging in the prohibited behavior mentioned above, it will be considered an automatic forfeiture of all licenses under this agreement.
5 Termination
If you violate any terms of this agreement, your license will automatically terminate, and you must cease using, copying, modifying, and sharing [FunASR Software].
6 Revisions
This agreement may be updated and revised occasionally. The revised agreement will be published in the official repository of [FunASR Software] and will take effect automatically. Continuing to use, copy, modify, and share [FunASR Software] indicates your acceptance of the revised agreement.
7 Miscellaneous
This agreement is governed by the laws of [Country/Region]. If any provision is deemed illegal, invalid, or unenforceable, that provision shall be considered severed from this agreement, and the remaining provisions shall continue to be valid and binding.
If you have any questions or comments regarding this agreement, please contact us.
Copyright © [2023-2028] [Alibaba Group]. All rights reserved.
FunASR 模型开源协议
版本号：1.1
版权所有 (C) [2023-2028] [阿里巴巴集团]。保留所有权利。
感谢您选择 FunASR 开源模型。FunASR 开源模型包含一系列免费且开源的工业模型，让大家可以使用、修改、分享和学习该模型。
为了保证更好的社区合作，我们制定了以下协议，希望您仔细阅读并遵守本协议。
1 定义
本协议中，[FunASR 软件]指 FunASR 开源模型权重及其衍生品，包括 Finetune 后的模型；[您]指使用、修改、分享和学习[FunASR 软件]的个人或组织。
2 许可和限制
2.1 许可
您可以在遵守本协议的前提下，自由地使用、复制、修改和分享[FunASR 软件]。
2.2 限制
您在使用、复制、修改和分享[FunASR 软件]时，必须注明出处以及作者信息，并保留[FunASR 软件]中相关模型名称。
3 责任和风险承担
[FunASR 软件]仅作为参考和学习使用，不对您使用或修改[FunASR 软件]造成的任何直接或间接损失承担任何责任。您对[FunASR 软件]的使用和修改应该自行承担风险。
4 社区行为准则
4.1 欢迎交流
社区欢迎开发者与用户对[FunASR 软件]进行交流讨论。交流中请注意保持友好、礼貌和文明，以促进建设性的讨论和合作。
4.2 禁止行为
个人或组织用户不得对[FunASR 软件]进行无端诋毁、恶意抹黑或凭空谩骂。此类行为被视为违反社区合作精神。如被认定从事上述禁止行为，将视为自动放弃本协议下的所有许可。
5 终止
如果您违反本协议的任何条款，您的许可将自动终止，您必须停止使用、复制、修改和分享[FunASR 软件]。
6 修订
本协议可能会不时更新和修订。修订后的协议将在[FunASR 软件]官方仓库发布，并自动生效。如果您继续使用、复制、修改和分享[FunASR 软件]，即表示您同意修订后的协议。
7 其他规定
本协议受到[国家/地区] 的法律管辖。如果任何条款被裁定为不合法、无效或无法执行，则该条款应被视为从本协议中删除，而其余条款应继续有效并具有约束力。
如果您对本协议有任何问题或意见，请联系我们。
版权所有© [2023-2028] [阿里巴巴集团]。保留所有权利。
MinMo_gitlab
Symlink
+1
View File
@@ -0,0 +1 @@
../MinMo_gitlab
README.md
+317
View File
@@ -0,0 +1,317 @@
([简体中文](./README_zh.md)|English|[日本語](./README_ja.md)|[한국어](./README_ko.md))
<p align="center">
<a href="https://github.com/modelscope/FunASR"><img src="https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210" alt="FunASR"></a>
</p>
<p align="center">
<strong>Industrial speech recognition. Up to 340x realtime, 26x faster than Whisper. 50+ languages.</strong><br>
<em>Speaker diarization · Emotion detection · Streaming · One API call</em>
</p>
<p align="center">
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/v/funasr" alt="PyPI"></a>
<a href="https://github.com/modelscope/FunASR"><img src="https://img.shields.io/github/stars/modelscope/FunASR?style=social" alt="Stars"></a>
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/dm/funasr" alt="Downloads"></a>
<a href="https://modelscope.github.io/FunASR/"><img src="https://img.shields.io/badge/docs-online-blue" alt="Docs"></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/10479" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10479" alt="modelscope%2FFunASR | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<a href="#quick-start">Quick Start</a> · <a href="./examples/colab/">Colab</a> · <a href="#benchmark">Benchmark</a> · <a href="./docs/model_selection.md">Model selection</a> · <a href="./docs/migration_from_whisper.md">Migration guide</a> · <a href="./docs/use_case_showcase.md">Use cases</a> · <a href="./docs/deployment_matrix.md">Deployment matrix</a> · <a href="#model-zoo">Models</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent Integration</a> · <a href="https://modelscope.github.io/FunASR/">Docs</a> · <a href="./CONTRIBUTING.md">Contribute</a>
</p>
---
## Quick Start
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/modelscope/FunASR/blob/main/examples/colab/funasr_quickstart.ipynb)
No local setup? Open the [Colab quickstart](./examples/colab/) to transcribe a public sample or upload your own audio in a browser.
```bash
pip install torch torchaudio
pip install funasr
```
**Flagship model — Fun-ASR-Nano** (LLM-ASR, 31 languages; the default recommendation, needs a GPU):
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav")
print(result[0]["text"])
# 欢迎大家来体验达摩院推出的语音识别模型。
```
On CPU (or for multilingual + emotion in one pass), use **SenseVoice** — which also returns speaker diarization and timestamps:
```python
from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess
model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", spk_model="cam++", device="cuda") # use device="cpu" if you don't have a GPU
result = model.generate(
input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
batch_size_s=300,
)
# One call returns VAD segments with speaker id + timestamps — render them however you like:
for seg in result[0]["sentence_info"]:
print(f"[{seg['start']/1000:.1f}s] Speaker {seg['spk']}: {rich_transcription_postprocess(seg['sentence'])}")
```
**Output** — structured text with speaker labels, timestamps, and punctuation:
```
[0.6s] Speaker 0: 欢迎大家来体验达摩院推出的语音识别模型
```
That's it. **One model, one call** — VAD segmentation, speech recognition, punctuation, speaker diarization all happen automatically.
### Scale & deploy the flagship
At scale, accelerate Fun-ASR-Nano with vLLM (batch processing):
```python
from funasr.auto.auto_model_vllm import AutoModelVLLM
model = AutoModelVLLM(model="FunAudioLLM/Fun-ASR-Nano-2512", tensor_parallel_size=1)
results = model.generate(["audio1.wav", "audio2.wav"], language="auto")
```
> **Deploy as API server:** `funasr-server --device cuda` → OpenAI-compatible endpoint at localhost:8000
>
> **Use with AI agents:** [MCP Server](examples/mcp_server/) for Claude/Cursor · [OpenAI API](examples/openai_api/) for LangChain/Dify/AutoGen
### Why FunASR?
Whisper is a single model; **FunASR is a toolkit** — you pick the right model per job: **Fun-ASR-Nano** (flagship LLM-ASR, GPU, 340x realtime with vLLM, 31 languages), **SenseVoice** (CPU-friendly, + emotion & audio events), **Paraformer** (low-latency streaming). The table shows what the toolkit delivers vs one Whisper model — each capability is labelled with the model that provides it:
| | FunASR (toolkit) | Whisper | Cloud APIs |
|---|---|---|---|
| Top speed | **340x realtime** (Fun-ASR-Nano + vLLM) | 13x realtime | ~1x realtime |
| Speaker ID | ✅ Built-in | ❌ Needs pyannote | ✅ Extra cost |
| Emotion | ✅ via SenseVoice | ❌ | ❌ |
| Languages | 50+ (Qwen3-ASR 52, Nano 31) | 57 | Varies |
| Streaming | ✅ WebSocket (Paraformer) | ❌ | ✅ |
| CPU viable | ✅ 17x realtime (SenseVoice) | ❌ Too slow | N/A |
| Self-hosted | ✅ MIT license | ✅ MIT license | ❌ Cloud only |
| Cost | Free | Free | $0.006/min+ |
Trying FunASR for the first time? Use the [Colab quickstart](./examples/colab/) before setting up a local environment. Choosing a first model? Start with the [model selection guide](./docs/model_selection.md). Planning a switch from Whisper or a cloud ASR provider? Use the [migration guide](./docs/migration_from_whisper.md) and [benchmark example](./examples/migration/) to test representative audio, map features, and roll out safely.
---
## Installation
```bash
pip install funasr
```
<details><summary>From source / Requirements</summary>
```bash
git clone https://github.com/modelscope/FunASR.git && cd FunASR
pip install -e ./
```
Requirements: Python ≥ 3.8. Install PyTorch + torchaudio first ([pytorch.org](https://pytorch.org/get-started/locally/)), then `pip install funasr`.
</details>
---
## Model Zoo
| Model | Task | Languages | Params | Links |
|-------|------|-----------|--------|-------|
| **Fun-ASR-Nano** | ASR + timestamps | 31 languages | 800M | [⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) |
| **SenseVoiceSmall** | ASR + emotion + events | zh/en/ja/ko/yue | 234M | [⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) |
| **Paraformer-zh** | ASR + timestamps | zh/en | 220M | [⭐](https://www.modelscope.cn/models/iic/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) |
| Paraformer-zh-streaming | Streaming ASR | zh/en | 220M | [⭐](https://modelscope.cn/models/iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) |
| Qwen3-ASR | ASR, 52 languages | multilingual | 1.7B | [usage](examples/industrial_data_pretraining/qwen3_asr) |
| GLM-ASR-Nano | ASR, 17 languages | multilingual | 1.5B | [usage](examples/industrial_data_pretraining/glm_asr) |
| Whisper-large-v3 | ASR + translation | multilingual | 1550M | [usage](examples/industrial_data_pretraining/whisper) |
| Whisper-large-v3-turbo | ASR + translation | multilingual | 809M | [usage](examples/industrial_data_pretraining/whisper) |
| ct-punc | Punctuation | zh/en | 290M | [⭐](https://modelscope.cn/models/iic/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) |
| fsmn-vad | VAD | zh/en | 0.4M | [⭐](https://modelscope.cn/models/iic/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) |
| cam++ | Speaker diarization | — | 7.2M | [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) |
| emotion2vec+large | Emotion recognition | — | 300M | [⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) |
---
## Usage
> Full examples with parameter docs: [Tutorial →](https://modelscope.github.io/FunASR/tutorial.html)
```python
from funasr import AutoModel
# Chinese production (VAD + ASR + punctuation + speaker)
model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", spk_model="cam++", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav", hotword="关键词 20")
# Streaming real-time (feed audio chunk by chunk)
import soundfile as sf
model = AutoModel(model="paraformer-zh-streaming", device="cuda")
audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz mono
chunk_size = [0, 10, 5] # 600 ms chunks
chunk_stride = chunk_size[1] * 960
cache = {}
n_chunks = (len(audio) - 1) // chunk_stride + 1
for i in range(n_chunks):
chunk = audio[i * chunk_stride : (i + 1) * chunk_stride]
res = model.generate(input=chunk, cache=cache, is_final=(i == n_chunks - 1),
chunk_size=chunk_size, encoder_chunk_look_back=4, decoder_chunk_look_back=1)
if res[0]["text"]:
print(res[0]["text"], end="", flush=True)
# Emotion recognition
model = AutoModel(model="emotion2vec_plus_large", device="cuda")
result = model.generate(input="audio.wav", granularity="utterance")
```
### CLI (Agent-Friendly)
```bash
# Transcribe audio (simplest)
funasr audio.wav
# JSON output (for AI agents)
funasr audio.wav --output-format json
# SRT subtitles
funasr audio.wav --output-format srt --output-dir ./subs
# Speaker diarization + timestamps
funasr audio.wav --spk --timestamps -f json
# Choose model and language
funasr audio.wav --model paraformer --language zh
# Batch transcribe
funasr *.wav --output-format srt --output-dir ./output
```
Available models: `sensevoice` (default), `paraformer`, `paraformer-en`, `fun-asr-nano`
---
## Deploy
```bash
# OpenAI-compatible API (recommended)
pip install torch torchaudio
pip install funasr vllm fastapi uvicorn python-multipart
funasr-server --device cuda
# → POST /v1/audio/transcriptions at localhost:8000
```
Verify it with a public sample:
```bash
curl -L https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav -o sample.wav
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
```bash
# Docker streaming service
docker pull registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.12
```
### CPU / Edge — llama.cpp / GGUF (no GPU, no Python)
Run **SenseVoice / Paraformer / Fun-ASR-Nano** as a **single self-contained binary** on CPU and edge devices — this is to FunASR what [whisper.cpp](https://github.com/ggml-org/whisper.cpp) is to Whisper, but with **~3× lower CER than whisper.cpp on Chinese**. Built-in FSMN-VAD, no Python at runtime.
```bash
# 1) Grab a prebuilt binary from Releases (Linux / macOS / Windows), then:
bash download-funasr-model.sh sensevoice ./gguf # or: paraformer | nano
llama-funasr-sensevoice -m ./gguf/SenseVoiceSmall-f16.gguf --vad ./gguf/fsmn-vad.gguf -a audio.wav
# → 欢迎大家来体验达摩院推出的语音识别模型
```
**Prebuilt binaries:** [Releases](../../releases) · **Download & quickstart:** [funasr.com/llama-cpp](https://www.funasr.com/llama-cpp.html) · **GGUF models:** [Hugging Face](https://huggingface.co/FunAudioLLM) · **Docs & benchmarks:** [runtime/llama.cpp/](./runtime/llama.cpp/)
[OpenAI API example →](./examples/openai_api/) · [Gradio demo →](./examples/openai_api/GRADIO.md) · [Client recipes →](./examples/openai_api/CLIENTS.md) · [JavaScript/TypeScript recipes →](./examples/openai_api/JAVASCRIPT.md) · [Kubernetes template →](./examples/openai_api/kubernetes/) · [Workflow recipes →](./examples/openai_api/WORKFLOWS.md) · [Postman collection →](./examples/openai_api/POSTMAN.md) · [OpenAPI spec →](./examples/openai_api/OPENAPI.md) · [Security guide →](./examples/openai_api/SECURITY.md) · [Deployment matrix →](./docs/deployment_matrix.md) · [Deployment docs →](./runtime/readme.md) · [Agent integration →](https://modelscope.github.io/FunASR/agent.html)
---
## Benchmark
> 184 long-form audio files (192 min). [Full report →](https://modelscope.github.io/FunASR/benchmark.html) · [RTFx and reproducibility notes →](./docs/benchmark/rtf_reproducibility.md)
| Model | Chinese CER ↓ | GPU Speed | CPU Speed | vs Whisper-large-v3 |
|-------|------|-----------|-----------|-------------------|
| **Fun-ASR-Nano** (vLLM) | **8.20%** | **340x** realtime | — | 🚀 **26x faster** |
| **SenseVoice-Small** | **7.81%** | **170x** realtime | **17x** realtime | 🚀 **13x faster** |
| **Paraformer-Large** | 10.18% | **120x** realtime | **15x** realtime | 🚀 **9x faster** |
| Whisper-large-v3-turbo | 21.71% | 46x realtime | ❌ | 3.4x faster |
| Whisper-large-v3 | 20.02% | 13x realtime | ❌ | baseline |
> **Key takeaway:** FunASR models run on CPU faster than Whisper runs on GPU.
---
## What's new
- 2026/06/20: **llama.cpp / GGUF runtime** — run SenseVoice / Paraformer / Fun-ASR-Nano on CPU & edge as a single self-contained binary (a whisper.cpp-style alternative), built-in FSMN-VAD, no Python at runtime. Prebuilt binaries for Linux / macOS / Windows + **q8 quantized models (~half the size, same accuracy)**. [runtime/llama.cpp/](./runtime/llama.cpp/) · [Releases](../../releases)
- 2026/06/21: **v1.3.12** on PyPI — rolling fixes (qwen3-asr language codes, glm_asr, vLLM repetition_penalty). `pip install --upgrade funasr`
- 2026/05/24: **vLLM Inference Engine** — 2-3x faster LLM decoding for Fun-ASR-Nano. Streaming WebSocket service with VAD + Speaker Diarization. [Guide →](docs/vllm_guide.md) · [Realtime WS tuning →](docs/vllm_guide.md#67-production-concurrency-and-multi-process-deployment) · [API stability checklist →](docs/vllm_guide.md#production-api-stability-checklist)
- 2026/05/24: **Dynamic VAD** — adaptive silence threshold (default on). Short sentences stay intact, long segments get auto-split. [Details →](docs/vllm_guide.md#附录dynamicstreamingvad)
- 2026/05/24: **v1.3.3** — `funasr-server` CLI, OpenAI-compatible API, MCP Server for AI agents. `pip install --upgrade funasr`
- 2026/05/20: Added Qwen3-ASR (0.6B/1.7B) — 52 languages, auto detection. [usage](examples/industrial_data_pretraining/qwen3_asr)
- 2026/05/20: Added GLM-ASR-Nano (1.5B) — 17 languages, dialect support. [usage](examples/industrial_data_pretraining/glm_asr)
- 2026/05/19: Fun-ASR-Nano and SenseVoice now support speaker diarization.
- 2025/12/15: [Fun-ASR-Nano-2512](https://github.com/FunAudioLLM/Fun-ASR) — 31 languages, tens of millions of hours training.
<details><summary>Older</summary>
- 2024/10/10: Whisper-large-v3-turbo support added.
- 2024/07/04: [SenseVoice](https://github.com/FunAudioLLM/SenseVoice) — ASR + emotion + audio events.
- 2024/01/30: FunASR 1.0 released.
</details>
---
## Community
| | |
|---|---|
| 📖 [Documentation](https://modelscope.github.io/FunASR/) | 🐛 [Issues](https://github.com/modelscope/FunASR/issues) |
| 💬 [Discussions](https://github.com/modelscope/FunASR/discussions) | 🤗 [HuggingFace](https://huggingface.co/funasr) |
| 🤝 [Contributing](./CONTRIBUTING.md) | 🌐 [funasr.com](https://www.funasr.com) |
| 🧩 [Community projects](./docs/community_projects.md) | 💡 [Use-case showcase](./docs/use_case_showcase.md) |
## Star History
<a href="https://star-history.com/#modelscope/FunASR&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date" width="600" />
</picture>
</a>
## License
[MIT License](./LICENSE)
## Citations
```bibtex
@inproceedings{gao2023funasr,
author={Zhifu Gao and others},
title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit},
booktitle={INTERSPEECH},
year={2023}
}
```
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`modelscope/FunASR`
- 原始仓库：https://github.com/modelscope/FunASR
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
README_ja.md
+152
View File
@@ -0,0 +1,152 @@
([English](./README.md)|[简体中文](./README_zh.md)|日本語|[한국어](./README_ko.md))
<p align="center">
<a href="https://github.com/modelscope/FunASR"><img src="https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210" alt="FunASR"></a>
</p>
<p align="center">
<strong>産業グレードの音声認識。最大340倍リアルタイム、Whisperより26倍高速。50以上の言語に対応。</strong><br>
<em>話者分離 · 感情認識 · ストリーミング · ワンコールで完結</em>
</p>
<p align="center">
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/v/funasr" alt="PyPI"></a>
<a href="https://github.com/modelscope/FunASR"><img src="https://img.shields.io/github/stars/modelscope/FunASR?style=social" alt="Stars"></a>
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/dm/funasr" alt="Downloads"></a>
<a href="https://modelscope.github.io/FunASR/"><img src="https://img.shields.io/badge/ドキュメント-オンライン-blue" alt="Docs"></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/10479" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10479" alt="modelscope%2FFunASR | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<a href="#クイックスタート">クイックスタート</a> · <a href="./examples/colab/README_ja.md">Colab</a> · <a href="./docs/model_selection_ja.md">モデル選択</a> · <a href="#ベンチマーク">ベンチマーク</a> · <a href="./docs/migration_from_whisper.md">Migration guide</a> · <a href="./docs/use_case_showcase.md">Use cases</a> · <a href="./docs/deployment_matrix_ja.md">Deployment matrix</a> · <a href="#モデル一覧">モデル一覧</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent連携</a> · <a href="https://modelscope.github.io/FunASR/">ドキュメント</a>
</p>
---
<a name="クイックスタート"></a>
## クイックスタート
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", spk_model="cam++", device="cuda")
result = model.generate(input="meeting.wav")
```
**出力** — 話者ラベル・タイムスタンプ・句読点付きの構造化テキスト：
```
[00:00.4 → 00:03.8] 話者0: Q3の計画について話し合いましょう。
[00:04.2 → 00:07.1] 話者1: いいですね。3つのポイントがあります。
[00:07.5 → 00:12.3] 話者0: どうぞ。あと30分あります。
```
1つのモデル、1回の呼び出し — VADセグメンテーション、音声認識、句読点復元、話者分離がすべて自動で実行されます。
初めて使う場合は [Colab クイックスタート](./examples/colab/README_ja.md) から試せます。どのモデルを選ぶか迷う場合は [モデル選択ガイド](./docs/model_selection_ja.md) を参照してください。
> **APIサーバーとしてデプロイ：** `funasr-server --device cuda` → localhost:8000でOpenAI互換エンドポイント
>
> **AIエージェント連携：** [MCPサーバー](examples/mcp_server/) Claude/Cursor対応 · [OpenAI API](examples/openai_api/) LangChain/Dify/AutoGen対応
### なぜFunASRを選ぶのか？
Whisper は単一モデルですが、**FunASR はツールキット**です——用途に応じてモデルを選べます：**Fun-ASR-Nano**（フラッグシップ LLM-ASR、GPU が必要、vLLM で 340 倍リアルタイム、31 言語）、**SenseVoice**（CPU に優しく、感情・音声イベントも）、**Paraformer**（低遅延ストリーミング）。下の表は、単一の Whisper モデルに対してツールキットが提供できるものです——各機能にはそれを提供するモデルを併記しています：
| | FunASR（ツールキット） | Whisper | クラウドAPI |
|---|---|---|---|
| 最高速度 | **340倍リアルタイム**（Fun-ASR-Nano + vLLM） | 13倍リアルタイム | 〜1倍リアルタイム |
| 話者認識 | ✅ 内蔵 | ❌ pyannoteが必要 | ✅ 追加料金 |
| 感情認識 | ✅ SenseVoice による | ❌ | ❌ |
| 言語数 | 50以上（Qwen3-ASR 52、Nano 31） | 57 | サービスにより異なる |
| ストリーミング | ✅ WebSocket（Paraformer） | ❌ | ✅ |
| CPU対応 | ✅ 17倍リアルタイム（SenseVoice） | ❌ 遅すぎる | 該当なし |
| セルフホスト | ✅ MITライセンス | ✅ MITライセンス | ❌ クラウドのみ |
| コスト | 無料 | 無料 | $0.006/分〜 |
---
<a name="ベンチマーク"></a>
## ベンチマーク
> 184件の長時間音声（計192分）。[詳細レポート →](https://modelscope.github.io/FunASR/benchmark.html)
| モデル | 中国語 CER ↓ | GPU速度 | CPU速度 | Whisper-large-v3比 |
|--------|------|---------|---------|-------------------|
| **Fun-ASR-Nano**（vLLM） | **8.20%** | **340倍**リアルタイム | — | 🚀 **26倍高速** |
| **SenseVoice-Small** | **7.81%** | **170倍**リアルタイム | **17倍**リアルタイム | 🚀 **13倍高速** |
| **Paraformer-Large** | 10.18% | **120倍**リアルタイム | **15倍**リアルタイム | 🚀 **9倍高速** |
| Whisper-large-v3-turbo | 21.71% | 46倍リアルタイム | ❌ | 3.4倍高速 |
| Whisper-large-v3 | 20.02% | 13倍リアルタイム | ❌ | ベースライン |
> **ポイント：** FunASRのCPU速度は、WhisperのGPU速度より速い。
---
## 最新情報
- 2026/05/24：**v1.3.3** — `funasr-server` CLI、OpenAI互換API、MCPサーバー。`pip install --upgrade funasr`
- 2026/05/20：Qwen3-ASR (0.6B/1.7B) 追加 — 52言語対応。
- 2026/05/20：GLM-ASR-Nano (1.5B) 追加 — 17言語、方言対応。
- 2025/12/15：[Fun-ASR-Nano-2512](https://github.com/FunAudioLLM/Fun-ASR) — 31言語対応。
---
## インストール
```bash
pip install funasr
```
要件：Python ≥ 3.8、PyTorch ≥ 1.13、torchaudio
---
<a name="モデル一覧"></a>
## モデル一覧
| モデル | タスク | 言語 | パラメータ | リンク |
|--------|--------|------|-----------|--------|
| **Fun-ASR-Nano** | 認識 + タイムスタンプ | 31言語 | 800M | [⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) |
| **SenseVoiceSmall** | 認識 + 感情 + イベント | 中/英/日/韓/粤 | 234M | [⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) |
| **Paraformer-zh** | 認識 + タイムスタンプ | 中/英 | 220M | [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) |
| Qwen3-ASR | 認識、52言語 | 多言語 | 1.7B | [使用法](examples/industrial_data_pretraining/qwen3_asr) |
| GLM-ASR-Nano | 認識、17言語 | 多言語 | 1.5B | [使用法](examples/industrial_data_pretraining/glm_asr) |
| Whisper-large-v3-turbo | 認識 + 翻訳 | 多言語 | 809M | [使用法](examples/industrial_data_pretraining/whisper) |
---
## デプロイ
```bash
# OpenAI互換API（推奨）
pip install funasr fastapi uvicorn python-multipart
funasr-server --device cuda
# Dockerストリーミングサービス
docker pull registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.12
```
[Colab quickstart →](./examples/colab/README_ja.md) · [OpenAI API example →](./examples/openai_api/README_ja.md) · [Client recipes →](./examples/openai_api/CLIENTS.md) · [Workflow recipes →](./examples/openai_api/WORKFLOWS.md) · [Postman collection →](./examples/openai_api/POSTMAN.md) · [OpenAPI spec →](./examples/openai_api/OPENAPI.md) · [Security guide →](./examples/openai_api/SECURITY.md) · [Deployment matrix →](./docs/deployment_matrix_ja.md) · [デプロイドキュメント →](./runtime/readme.md) · [Agent連携 →](https://modelscope.github.io/FunASR/agent.html)
---
## コミュニティ
| | |
|---|---|
| 📖 [ドキュメント](https://modelscope.github.io/FunASR/) | 🐛 [Issues](https://github.com/modelscope/FunASR/issues) |
| 💬 [Discussions](https://github.com/modelscope/FunASR/discussions) | 🤗 [HuggingFace](https://huggingface.co/funasr) |
## ライセンス
[MIT License](./LICENSE)
README_ko.md
+152
View File
@@ -0,0 +1,152 @@
([English](./README.md)|[简体中文](./README_zh.md)|[日本語](./README_ja.md)|한국어)
<p align="center">
<a href="https://github.com/modelscope/FunASR"><img src="https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210" alt="FunASR"></a>
</p>
<p align="center">
<strong>산업용 음성인식. 최대 340배 실시간, Whisper보다 26배 빠름. 50개 이상 언어 지원.</strong><br>
<em>화자 분리 · 감정 인식 · 스트리밍 · 한 번의 호출로 해결</em>
</p>
<p align="center">
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/v/funasr" alt="PyPI"></a>
<a href="https://github.com/modelscope/FunASR"><img src="https://img.shields.io/github/stars/modelscope/FunASR?style=social" alt="Stars"></a>
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/dm/funasr" alt="Downloads"></a>
<a href="https://modelscope.github.io/FunASR/"><img src="https://img.shields.io/badge/문서-온라인-blue" alt="Docs"></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/10479" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10479" alt="modelscope%2FFunASR | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<a href="#빠른-시작">빠른 시작</a> · <a href="./examples/colab/README_ko.md">Colab</a> · <a href="./docs/model_selection_ko.md">모델 선택</a> · <a href="#벤치마크">벤치마크</a> · <a href="./docs/migration_from_whisper.md">Migration guide</a> · <a href="./docs/use_case_showcase.md">Use cases</a> · <a href="./docs/deployment_matrix_ko.md">Deployment matrix</a> · <a href="#모델-목록">모델 목록</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent 연동</a> · <a href="https://modelscope.github.io/FunASR/">문서</a>
</p>
---
<a name="빠른-시작"></a>
## 빠른 시작
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", spk_model="cam++", device="cuda")
result = model.generate(input="meeting.wav")
```
**출력** — 화자 라벨, 타임스탬프, 구두점이 포함된 구조화된 텍스트:
```
[00:00.4 → 00:03.8] 화자0: Q3 계획에 대해 논의하겠습니다.
[00:04.2 → 00:07.1] 화자1: 좋습니다. 세 가지 포인트가 있습니다.
[00:07.5 → 00:12.3] 화자0: 말씀하세요. 30분 남았습니다.
```
하나의 모델, 한 번의 호출 — VAD 분할, 음성인식, 구두점 복원, 화자 분리가 모두 자동으로 수행됩니다.
처음 사용한다면 [Colab 빠른 시작](./examples/colab/README_ko.md)으로 먼저 확인할 수 있습니다. 어떤 모델을 선택할지 고민된다면 [모델 선택 가이드](./docs/model_selection_ko.md)를 참고하세요.
> **API 서버로 배포:** `funasr-server --device cuda` → localhost:8000에서 OpenAI 호환 엔드포인트
>
> **AI Agent 연동:** [MCP 서버](examples/mcp_server/) Claude/Cursor 지원 · [OpenAI API](examples/openai_api/) LangChain/Dify/AutoGen 지원
### 왜 FunASR인가?
Whisper는 단일 모델이지만, **FunASR는 툴킷**입니다 — 용도에 맞는 모델을 고르세요: **Fun-ASR-Nano**(플래그십 LLM-ASR, GPU 필요, vLLM로 340배 실시간, 31개 언어), **SenseVoice**(CPU 친화적, 감정·오디오 이벤트 포함), **Paraformer**(저지연 스트리밍). 아래 표는 단일 Whisper 모델 대비 툴킷이 제공하는 것이며, 각 기능에는 이를 제공하는 모델을 표기했습니다:
| | FunASR(툴킷) | Whisper | 클라우드 API |
|---|---|---|---|
| 최고 속도 | **340배 실시간**(Fun-ASR-Nano + vLLM) | 13배 실시간 | ~1배 실시간 |
| 화자 인식 | ✅ 내장 | ❌ pyannote 필요 | ✅ 추가 비용 |
| 감정 인식 | ✅ SenseVoice 제공 | ❌ | ❌ |
| 언어 수 | 50개 이상(Qwen3-ASR 52, Nano 31) | 57개 | 서비스마다 다름 |
| 스트리밍 | ✅ WebSocket(Paraformer) | ❌ | ✅ |
| CPU 사용 | ✅ 17배 실시간(SenseVoice) | ❌ 너무 느림 | 해당 없음 |
| 자체 호스팅 | ✅ MIT 라이선스 | ✅ MIT 라이선스 | ❌ 클라우드만 |
| 비용 | 무료 | 무료 | $0.006/분~ |
---
<a name="벤치마크"></a>
## 벤치마크
> 184개 장시간 오디오(총 192분). [상세 보고서 →](https://modelscope.github.io/FunASR/benchmark.html)
| 모델 | 중국어 CER ↓ | GPU 속도 | CPU 속도 | Whisper-large-v3 대비 |
|------|------|----------|----------|---------------------|
| **Fun-ASR-Nano**(vLLM) | **8.20%** | **340배** 실시간 | — | 🚀 **26배 빠름** |
| **SenseVoice-Small** | **7.81%** | **170배** 실시간 | **17배** 실시간 | 🚀 **13배 빠름** |
| **Paraformer-Large** | 10.18% | **120배** 실시간 | **15배** 실시간 | 🚀 **9배 빠름** |
| Whisper-large-v3-turbo | 21.71% | 46배 실시간 | ❌ | 3.4배 빠름 |
| Whisper-large-v3 | 20.02% | 13배 실시간 | ❌ | 기준선 |
> **핵심:** FunASR의 CPU 속도가 Whisper의 GPU 속도보다 빠릅니다.
---
## 최신 소식
- 2026/05/24: **v1.3.3** — `funasr-server` CLI, OpenAI 호환 API, MCP 서버. `pip install --upgrade funasr`
- 2026/05/20: Qwen3-ASR (0.6B/1.7B) 추가 — 52개 언어 지원.
- 2026/05/20: GLM-ASR-Nano (1.5B) 추가 — 17개 언어, 방언 지원.
- 2025/12/15: [Fun-ASR-Nano-2512](https://github.com/FunAudioLLM/Fun-ASR) — 31개 언어 지원.
---
## 설치
```bash
pip install funasr
```
요구사항: Python ≥ 3.8, PyTorch ≥ 1.13, torchaudio
---
<a name="모델-목록"></a>
## 모델 목록
| 모델 | 작업 | 언어 | 파라미터 | 링크 |
|------|------|------|---------|------|
| **Fun-ASR-Nano** | 인식 + 타임스탬프 | 31개 언어 | 800M | [⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) |
| **SenseVoiceSmall** | 인식 + 감정 + 이벤트 | 중/영/일/한/광둥어 | 234M | [⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) |
| **Paraformer-zh** | 인식 + 타임스탬프 | 중/영 | 220M | [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) |
| Qwen3-ASR | 인식, 52개 언어 | 다국어 | 1.7B | [사용법](examples/industrial_data_pretraining/qwen3_asr) |
| GLM-ASR-Nano | 인식, 17개 언어 | 다국어 | 1.5B | [사용법](examples/industrial_data_pretraining/glm_asr) |
| Whisper-large-v3-turbo | 인식 + 번역 | 다국어 | 809M | [사용법](examples/industrial_data_pretraining/whisper) |
---
## 배포
```bash
# OpenAI 호환 API (권장)
pip install funasr fastapi uvicorn python-multipart
funasr-server --device cuda
# Docker 스트리밍 서비스
docker pull registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.12
```
[Colab quickstart →](./examples/colab/README_ko.md) · [OpenAI API example →](./examples/openai_api/README_ko.md) · [Client recipes →](./examples/openai_api/CLIENTS.md) · [Workflow recipes →](./examples/openai_api/WORKFLOWS.md) · [Postman collection →](./examples/openai_api/POSTMAN.md) · [OpenAPI spec →](./examples/openai_api/OPENAPI.md) · [Security guide →](./examples/openai_api/SECURITY.md) · [Deployment matrix →](./docs/deployment_matrix_ko.md) · [배포 문서 →](./runtime/readme.md) · [Agent 연동 →](https://modelscope.github.io/FunASR/agent.html)
---
## 커뮤니티
| | |
|---|---|
| 📖 [문서](https://modelscope.github.io/FunASR/) | 🐛 [Issues](https://github.com/modelscope/FunASR/issues) |
| 💬 [Discussions](https://github.com/modelscope/FunASR/discussions) | 🤗 [HuggingFace](https://huggingface.co/funasr) |
## 라이선스
[MIT License](./LICENSE)
README_zh.md
+302
View File
@@ -0,0 +1,302 @@
([English](./README.md)|简体中文|[日本語](./README_ja.md)|[한국어](./README_ko.md))
<p align="center">
<a href="https://github.com/modelscope/FunASR"><img src="https://svg-banners.vercel.app/api?type=origin&text1=FunASR🤠&text2=💖%20A%20Fundamental%20End-to-End%20Speech%20Recognition%20Toolkit&width=800&height=210" alt="FunASR"></a>
</p>
<p align="center">
<strong>工业级语音识别。最高 340 倍实时，比 Whisper 快 26 倍。支持 50+ 语言。</strong><br>
<em>说话人分离 · 情感识别 · 流式转写 · 一次调用搞定</em>
</p>
<p align="center">
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/v/funasr" alt="PyPI"></a>
<a href="https://github.com/modelscope/FunASR"><img src="https://img.shields.io/github/stars/modelscope/FunASR?style=social" alt="Stars"></a>
<a href="https://pypi.org/project/funasr/"><img src="https://img.shields.io/pypi/dm/funasr" alt="Downloads"></a>
<a href="https://modelscope.github.io/FunASR/zh/"><img src="https://img.shields.io/badge/文档-在线-blue" alt="Docs"></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/10479" target="_blank"><img src="https://trendshift.io/api/badge/repositories/10479" alt="modelscope%2FFunASR | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<a href="#快速开始">快速开始</a> · <a href="./examples/colab/README_zh.md">Colab</a> · <a href="#性能评测">性能评测</a> · <a href="./docs/model_selection_zh.md">模型选择</a> · <a href="./docs/migration_from_whisper_zh.md">迁移指南</a> · <a href="./docs/use_case_showcase_zh.md">场景速览</a> · <a href="./docs/deployment_matrix_zh.md">部署选型</a> · <a href="#模型列表">模型列表</a> · <a href="https://modelscope.github.io/FunASR/agent.html">Agent 集成</a> · <a href="https://modelscope.github.io/FunASR/zh/">文档</a> · <a href="./CONTRIBUTING.md">贡献</a>
</p>
---
## 快速开始
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/modelscope/FunASR/blob/main/examples/colab/funasr_quickstart.ipynb)
不想先配置本地环境？可以打开 [Colab 快速体验](./examples/colab/README_zh.md) 在浏览器里转写公开样例或上传自己的音频。
```bash
pip install funasr
```
```python
from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess
model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", spk_model="cam++", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav")
# 一次调用即返回带说话人 id 和时间戳的 VAD 分段，可自由渲染：
for seg in result[0]["sentence_info"]:
print(f"[{seg['start']/1000:.1f}s] 说话人{seg['spk']}: {rich_transcription_postprocess(seg['sentence'])}")
```
**输出** — 带说话人标签、时间戳和标点的结构化文本：
```
[0.6s] 说话人0: 欢迎大家来体验达摩院推出的语音识别模型
```
一个模型、一次调用 — VAD 分段、语音识别、标点恢复、说话人分离全部自动完成。
### LLM 语音识别：Fun-ASR-Nano
追求更高精度、支持 31 种语言（含中文方言），使用 [Fun-ASR-Nano](https://github.com/FunAudioLLM/Fun-ASR) — SenseVoice 编码器 + Qwen3-0.6B 解码器的 LLM-based ASR：
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", vad_model="fsmn-vad", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav")
```
使用 vLLM 加速（批量处理快 16 倍）：
```python
from funasr.auto.auto_model_vllm import AutoModelVLLM
model = AutoModelVLLM(model="FunAudioLLM/Fun-ASR-Nano-2512", tensor_parallel_size=1)
results = model.generate(["audio1.wav", "audio2.wav"], language="auto")
```
> **部署为 API 服务：** `funasr-server --device cuda` → 本地 OpenAI 兼容接口 localhost:8000
>
> **接入 AI Agent：** [MCP 服务](examples/mcp_server/) 支持 Claude/Cursor · [OpenAI API](examples/openai_api/README_zh.md) 支持 LangChain/Dify/AutoGen
### 为什么选 FunASR？
Whisper 是单个模型，**FunASR 是一个工具箱**——按场景挑模型：**Fun-ASR-Nano**（旗舰 LLM-ASR，需 GPU，vLLM 下 340 倍实时，31 种语言）、**SenseVoice**（CPU 友好，额外带情感与音频事件）、**Paraformer**（低延迟流式）。下表是工具箱相比单个 Whisper 模型能提供什么——每项能力都标注了由哪个模型提供：
| | FunASR（工具箱） | Whisper | 云端 API |
|---|---|---|---|
| 最高速度 | **340 倍实时**（Fun-ASR-Nano + vLLM） | 13 倍实时 | ~1 倍实时 |
| 说话人识别 | ✅ 内置 | ❌ 需要 pyannote | ✅ 额外付费 |
| 情感识别 | ✅ 由 SenseVoice 提供 | ❌ | ❌ |
| 语言数 | 50+（Qwen3-ASR 52、Nano 31） | 57 | 因服务而异 |
| 流式识别 | ✅ WebSocket（Paraformer） | ❌ | ✅ |
| CPU 可用 | ✅ 17 倍实时（SenseVoice） | ❌ 太慢 | 不适用 |
| 私有部署 | ✅ MIT 开源 | ✅ MIT 开源 | ❌ 仅云端 |
| 费用 | 免费 | 免费 | ¥0.04/分钟起 |
第一次试用 FunASR？可以先跑 [Colab 快速体验](./examples/colab/README_zh.md)，再配置本地环境。还不确定先用哪个模型？先看 [模型选择指南](./docs/model_selection_zh.md)。计划从 Whisper 或云端 ASR 切换？请按 [迁移指南](./docs/migration_from_whisper_zh.md) 和 [评测示例](./examples/migration/) 用代表性音频评测、映射功能并安全上线。
---
<a name="性能评测"></a>
## 性能评测
> 184 条长音频（共 192 分钟）。[完整报告 →](https://modelscope.github.io/FunASR/zh/benchmark.html)
| 模型 | 中文 CER ↓ | GPU 速度 | CPU 速度 | 对比 Whisper-large-v3 |
|------|------|----------|----------|---------------------|
| **Fun-ASR-Nano**（vLLM） | **8.20%** | **340 倍**实时 | — | 🚀 **快 26 倍** |
| **SenseVoice-Small** | **7.81%** | **170 倍**实时 | **17 倍**实时 | 🚀 **快 13 倍** |
| **Paraformer-Large** | 10.18% | **120 倍**实时 | **15 倍**实时 | 🚀 **快 9 倍** |
| Whisper-large-v3-turbo | 21.71% | 46 倍实时 | ❌ | 快 3.4 倍 |
| Whisper-large-v3 | 20.02% | 13 倍实时 | ❌ | 基准 |
> **一句话：** FunASR 在 CPU 上的速度，比 Whisper 在 GPU 上还快。
---
## 最新动态
- 2026/05/24：**vLLM 推理引擎** — Fun-ASR-Nano 解码加速 2-3 倍。支持流式 WebSocket 服务（VAD + 说话人分离 + 热词）。[文档 →](docs/vllm_guide_zh.md) · [实时 WS 调优 →](docs/vllm_guide_zh.md#67-生产并发与多进程部署) · [API 稳定性清单 →](docs/vllm_guide_zh.md#生产-api-稳定性清单)
- 2026/05/24：**动态 VAD** — 自适应静音阈值（默认开启），短句不切碎、长句自动切分。[详情 →](docs/vllm_guide_zh.md#7-动态-vad)
- 2026/05/24：**v1.3.3** — `funasr-server` 命令行工具、OpenAI 兼容 API、MCP 服务。`pip install --upgrade funasr`
- 2026/05/20：新增 Qwen3-ASR (0.6B/1.7B)，52 种语言自动检测。[使用方法](examples/industrial_data_pretraining/qwen3_asr)
- 2026/05/20：新增 GLM-ASR-Nano (1.5B)，17 种语言，方言优化。[使用方法](examples/industrial_data_pretraining/glm_asr)
- 2026/05/19：Fun-ASR-Nano 和 SenseVoice 支持说话人分离。
- 2025/12/15：[Fun-ASR-Nano-2512](https://github.com/FunAudioLLM/Fun-ASR) 上线。
<details><summary>更早</summary>
- 2024/10/10：支持 Whisper-large-v3-turbo。
- 2024/07/04：[SenseVoice](https://github.com/FunAudioLLM/SenseVoice) 发布。
- 2024/01/30：FunASR 1.0 发布。
</details>
---
## 安装
```bash
pip install funasr
```
<details><summary>从源码安装</summary>
```bash
git clone https://github.com/modelscope/FunASR.git && cd FunASR
pip install -e ./
```
环境要求：Python ≥ 3.8、PyTorch ≥ 1.13、torchaudio
</details>
---
<a name="模型列表"></a>
## 模型列表
| 模型 | 任务 | 语言 | 参数量 | 链接 |
|------|------|------|--------|------|
| **Fun-ASR-Nano** | 识别 + 时间戳 | 31 种语言 | 800M | [⭐](https://www.modelscope.cn/models/FunAudioLLM/Fun-ASR-Nano-2512) [🤗](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) |
| **SenseVoiceSmall** | 识别 + 情感 + 事件 | 中/英/日/韩/粤 | 234M | [⭐](https://www.modelscope.cn/models/iic/SenseVoiceSmall) [🤗](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) |
| **Paraformer-zh** | 识别 + 时间戳 | 中/英 | 220M | [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗](https://huggingface.co/funasr/paraformer-zh) |
| Paraformer-zh-streaming | 流式识别 | 中/英 | 220M | [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗](https://huggingface.co/funasr/paraformer-zh-streaming) |
| Qwen3-ASR | 识别，52 种语言 | 多语言 | 1.7B | [使用](examples/industrial_data_pretraining/qwen3_asr) |
| GLM-ASR-Nano | 识别，17 种语言 | 多语言 | 1.5B | [使用](examples/industrial_data_pretraining/glm_asr) |
| Whisper-large-v3 | 识别 + 翻译 | 多语言 | 1550M | [使用](examples/industrial_data_pretraining/whisper) |
| Whisper-large-v3-turbo | 识别 + 翻译 | 多语言 | 809M | [使用](examples/industrial_data_pretraining/whisper) |
| ct-punc | 标点恢复 | 中/英 | 290M | [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗](https://huggingface.co/funasr/ct-punc) |
| fsmn-vad | 语音检测 | 中/英 | 0.4M | [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) |
| cam++ | 说话人分离 | — | 7.2M | [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) |
| emotion2vec+large | 情感识别 | — | 300M | [⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) |
---
## 使用示例
> 完整参数文档：[教程 →](https://modelscope.github.io/FunASR/zh/tutorial.html)
```python
from funasr import AutoModel
# 中文生产级（VAD + 识别 + 标点 + 说话人）
model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", spk_model="cam++", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav", hotword="关键词 20")
# 31 种语言 + 时间戳
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", hub="hf", trust_remote_code=True,
vad_model="fsmn-vad", vad_kwargs={"max_single_segment_time": 30000}, device="cuda")
result = model.generate(input="audio.wav", batch_size=1)
# 流式实时识别(逐块喂音频)
import soundfile as sf
model = AutoModel(model="paraformer-zh-streaming", device="cuda")
audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz 单声道
chunk_size = [0, 10, 5] # 每块 600ms
chunk_stride = chunk_size[1] * 960
cache = {}
n_chunks = (len(audio) - 1) // chunk_stride + 1
for i in range(n_chunks):
chunk = audio[i * chunk_stride : (i + 1) * chunk_stride]
res = model.generate(input=chunk, cache=cache, is_final=(i == n_chunks - 1),
chunk_size=chunk_size, encoder_chunk_look_back=4, decoder_chunk_look_back=1)
if res[0]["text"]:
print(res[0]["text"], end="", flush=True)
# 情感识别
model = AutoModel(model="emotion2vec_plus_large", device="cuda")
result = model.generate(input="audio.wav", granularity="utterance")
```
### 命令行工具（Agent 友好）
```bash
# 转写音频（最简用法）
funasr audio.wav
# JSON 输出（适合 AI Agent 调用）
funasr audio.wav --output-format json
# 生成 SRT 字幕
funasr audio.wav --output-format srt --output-dir ./subs
# 说话人分离 + 时间戳
funasr audio.wav --spk --timestamps -f json
# 指定模型和语言
funasr audio.wav --model paraformer --language zh
# 批量转写
funasr *.wav --output-format srt --output-dir ./output
```
可用模型：`sensevoice`（默认）、`paraformer`、`paraformer-en`、`fun-asr-nano`
---
## 部署
```bash
# OpenAI 兼容 API（推荐）
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
# → POST /v1/audio/transcriptions，地址 localhost:8000
```
使用公开样例音频验证服务：
```bash
curl -L https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav -o sample.wav
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
```bash
# Docker 流式服务
docker pull registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.12
```
> **CPU / 边缘部署(无需 GPU、无需 Python):** 用 **llama.cpp / GGUF** 跑 Fun-ASR-Nano / SenseVoice / Paraformer —— 单个自包含二进制,对标 whisper.cpp。详见 [runtime/llama.cpp/](./runtime/llama.cpp/)。
[OpenAI API 示例 →](./examples/openai_api/README_zh.md) · [Gradio Demo →](./examples/openai_api/GRADIO_zh.md) · [客户端配方 →](./examples/openai_api/CLIENTS.md) · [JavaScript/TypeScript 配方 →](./examples/openai_api/JAVASCRIPT_zh.md) · [Kubernetes 模板 →](./examples/openai_api/kubernetes/README_zh.md) · [工作流配方 →](./examples/openai_api/WORKFLOWS_zh.md) · [Postman 集合 →](./examples/openai_api/POSTMAN_zh.md) · [OpenAPI 规范 →](./examples/openai_api/OPENAPI_zh.md) · [安全指南 →](./examples/openai_api/SECURITY_zh.md) · [部署选型 →](./docs/deployment_matrix_zh.md) · [部署文档 →](./runtime/readme_cn.md) · [Agent 集成 →](https://modelscope.github.io/FunASR/agent.html)
---
## 社区
| | |
|---|---|
| 📖 [文档](https://modelscope.github.io/FunASR/zh/) | 🐛 [问题反馈](https://github.com/modelscope/FunASR/issues) |
| 💬 [讨论](https://github.com/modelscope/FunASR/discussions) | 🤗 [HuggingFace](https://huggingface.co/funasr) |
| 🤝 [贡献指南](./CONTRIBUTING.md) | 📈 [20k 增长计划](./docs/community_growth_20k.md) |
## Star 趋势
<a href="https://star-history.com/#modelscope/FunASR&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=modelscope/FunASR&type=Date" width="600" />
</picture>
</a>
## 许可证
[MIT License](./LICENSE)
## 引用
```bibtex
@inproceedings{gao2023funasr,
author={Zhifu Gao and others},
title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit},
booktitle={INTERSPEECH},
year={2023}
}
```
SECURITY.md
+11
View File
@@ -0,0 +1,11 @@
# Security Policy
## Reporting a Vulnerability
If you believe you have discovered a security vulnerability, please report it to us through the following portal:
👉[Report Security Issue](https://yundun.console.aliyun.com/?p=xznew#/taskmanagement/tasks/detail/153)
> **Note:** This channel is strictly for reporting security-related issues. Non-security vulnerabilities or general bug reports will not be addressed here.
We sincerely appreciate your responsible disclosure and your contribution to helping us keep our project secure.
benchmark_vllm.py
+224
View File
@@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""FunASR vLLM Benchmark: unified speed + CER comparison for all supported models.
Supports: Fun-ASR-Nano, GLM-ASR-Nano (and any model via AutoModelVLLM).
Usage:
# Fun-ASR-Nano
CUDA_VISIBLE_DEVICES=0 python benchmark_vllm.py \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--audio-dir /path/to/benchmark_audio \
--label-json /path/to/benchmark_testset.json
# GLM-ASR-Nano
CUDA_VISIBLE_DEVICES=0 python benchmark_vllm.py \
--model zai-org/GLM-ASR-Nano-2512 \
--audio-dir /path/to/benchmark_audio \
--label-json /path/to/benchmark_testset.json
# Quick test (first N files)
CUDA_VISIBLE_DEVICES=0 python benchmark_vllm.py \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--audio-dir /path/to/benchmark_audio \
--label-json /path/to/benchmark_testset.json \
--max-files 20
# Skip PyTorch (only test vLLM)
CUDA_VISIBLE_DEVICES=0 python benchmark_vllm.py \
--model FunAudioLLM/Fun-ASR-Nano-2512 \
--skip-pytorch \
--audio-dir /path/to/benchmark_audio \
--label-json /path/to/benchmark_testset.json
"""
import argparse
import json
import os
import re
import time
import kaldialign
import numpy as np
import soundfile as sf
import torch
def normalize_zh(text):
text = re.sub(r'[^\w一-鿿]', '', text)
return text.upper()
def compute_cer(refs, hyps):
total_ref = 0
total_errs = 0
for ref, hyp in zip(refs, hyps):
r = list(normalize_zh(ref))
h = list(normalize_zh(hyp))
total_ref += len(r)
ali = kaldialign.align(r, h, '*')
total_errs += sum(1 for a, b in ali if a != b)
return total_errs / total_ref * 100 if total_ref > 0 else 0
def vad_segment(files, device="cuda:0"):
from funasr import AutoModel
vad_model = AutoModel(model="fsmn-vad", device=device, disable_update=True)
all_segments = []
for fi, wav_path in enumerate(files):
audio, sr = sf.read(wav_path)
if audio.ndim > 1:
audio = audio[:, 0]
audio = audio.astype(np.float32)
res = vad_model.generate(input=wav_path, dynamic_silence=False)
for seg in res[0]["value"]:
s0 = int(seg[0] * sr / 1000)
s1 = int(seg[1] * sr / 1000)
seg_audio = audio[s0:s1]
if len(seg_audio) > sr * 0.5:
all_segments.append((fi, seg_audio))
return all_segments
def concat_results(all_segments, seg_texts, n_files):
file_texts = {}
for (fi, _), text in zip(all_segments, seg_texts):
file_texts.setdefault(fi, []).append(text)
return ["".join(file_texts.get(fi, [])) for fi in range(n_files)]
def run_pytorch(model_name, seg_files, device="cuda:0"):
from funasr import AutoModel
kwargs = {"model": model_name, "device": device, "disable_update": True}
if "Fun-ASR-Nano" in model_name:
kwargs["trust_remote_code"] = True
kwargs["remote_code"] = os.path.join(
os.path.dirname(__file__),
"examples/industrial_data_pretraining/fun_asr_nano/model.py"
)
model = AutoModel(**kwargs)
model.generate(input=seg_files[0]) # warmup
t0 = time.perf_counter()
texts = []
for f in seg_files:
res = model.generate(input=f)
texts.append(res[0]["text"])
t1 = time.perf_counter()
return t1 - t0, texts
def run_vllm(model_name, seg_files, device="cuda:0", hub="ms"):
if "Fun-ASR-Nano" in model_name:
from funasr.models.fun_asr_nano.inference_vllm import FunASRNanoVLLM
engine = FunASRNanoVLLM.from_pretrained(
model=model_name, hub=hub, device=device, dtype="bf16",
max_model_len=4096, gpu_memory_utilization=0.5)
engine.generate(inputs=[seg_files[0]], language="中文") # warmup
t0 = time.perf_counter()
results = engine.generate(inputs=seg_files, language="中文", max_new_tokens=500)
t1 = time.perf_counter()
texts = [r["text"] for r in results]
elif "GLM-ASR" in model_name:
from funasr.models.glm_asr.inference_vllm import GLMASRVLLMEngine
engine = GLMASRVLLMEngine.from_pretrained(
model=model_name, hub=hub, device=device, dtype="bf16",
gpu_memory_utilization=0.4, max_model_len=4096)
engine.generate(inputs=[seg_files[0]]) # warmup
t0 = time.perf_counter()
results = engine.generate(inputs=seg_files, max_new_tokens=500)
t1 = time.perf_counter()
texts = [r["text"] for r in results]
else:
from funasr.auto.auto_model_vllm import AutoModelVLLM
engine = AutoModelVLLM(model=model_name, hub=hub, device=device)
engine.generate(inputs=[seg_files[0]])
t0 = time.perf_counter()
results = engine.generate(inputs=seg_files, max_new_tokens=500)
t1 = time.perf_counter()
texts = [r["text"] for r in results]
return t1 - t0, texts
if __name__ == '__main__':
parser = argparse.ArgumentParser(description="FunASR vLLM Benchmark")
parser.add_argument("--model", type=str, required=True, help="Model name or path")
parser.add_argument("--hub", type=str, default="ms", choices=["ms", "hf"])
parser.add_argument("--audio-dir", type=str, required=True)
parser.add_argument("--label-json", type=str, required=True)
parser.add_argument("--device", type=str, default="cuda:0")
parser.add_argument("--max-files", type=int, default=0)
parser.add_argument("--skip-pytorch", action="store_true")
args = parser.parse_args()
with open(args.label_json) as f:
dataset = json.load(f)
files = []
refs = []
for item in dataset:
wav_path = os.path.join(args.audio_dir, f"{item['id']:03d}.wav")
if os.path.exists(wav_path):
files.append(wav_path)
refs.append(item["ref"])
if args.max_files > 0:
files = files[:args.max_files]
refs = refs[:args.max_files]
total_audio = sum(sf.info(f).duration for f in files)
print(f"{'='*60}")
print(f"FunASR vLLM Benchmark")
print(f"{'='*60}")
print(f"Model: {args.model}")
print(f"Dataset: {len(files)} files, {total_audio:.0f}s audio")
# VAD
print(f"\n>>> VAD pre-segmenting...")
all_segments = vad_segment(files, device=args.device)
print(f" {len(all_segments)} segments")
os.makedirs("/tmp/benchmark_vllm_segs", exist_ok=True)
seg_files = []
for i, (fi, audio) in enumerate(all_segments):
path = f"/tmp/benchmark_vllm_segs/{i:04d}.wav"
sf.write(path, audio, 16000)
seg_files.append(path)
# PyTorch
cer_pt = None
pt_time = None
if not args.skip_pytorch:
print(f"\n>>> PyTorch native...")
pt_time, pt_seg_texts = run_pytorch(args.model, seg_files, device=args.device)
pt_texts = concat_results(all_segments, pt_seg_texts, len(files))
cer_pt = compute_cer(refs, pt_texts)
print(f" Time: {pt_time:.1f}s | RTFx: {total_audio/pt_time:.1f} | CER: {cer_pt:.2f}%")
del torch.cuda.memory_allocated
torch.cuda.empty_cache()
# vLLM
print(f"\n>>> vLLM...")
vllm_time, vllm_seg_texts = run_vllm(args.model, seg_files, device=args.device, hub=args.hub)
vllm_texts = concat_results(all_segments, vllm_seg_texts, len(files))
cer_vllm = compute_cer(refs, vllm_texts)
print(f" Time: {vllm_time:.1f}s | RTFx: {total_audio/vllm_time:.1f} | CER: {cer_vllm:.2f}%")
# Summary
print(f"\n{'='*60}")
print(f"RESULTS")
print(f"{'-'*60}")
print(f"{'Method':<20} {'Time':<10} {'RTFx':<10} {'CER'}")
print(f"{'-'*60}")
if not args.skip_pytorch:
print(f"{'PyTorch':<20} {pt_time:<10.1f} {total_audio/pt_time:<10.1f} {cer_pt:.2f}%")
print(f"{'vLLM':<20} {vllm_time:<10.1f} {total_audio/vllm_time:<10.1f} {cer_vllm:.2f}%")
if not args.skip_pytorch:
print(f"{'-'*60}")
speedup = (total_audio/vllm_time) / (total_audio/pt_time)
print(f"Speedup: {speedup:.1f}x | CER diff: {cer_vllm - cer_pt:+.2f}%")
print(f"{'='*60}")
benchmarks/benchmark_pipeline_cer.md
+216
View File
@@ -0,0 +1,216 @@
# Leaderboard IO
## Configuration
### Data set:
[Aishell1](https://www.openslr.org/33/): dev, test
[Aishell2](https://www.aishelltech.com/aishell_2): dev_ios, test_ios, test_android, test_mic
[WenetSpeech](https://github.com/wenet-e2e/WenetSpeech): dev, test_meeting, test_net
### Tools
#### [Install Requirements](https://alibaba-damo-academy.github.io/FunASR/en/installation/installation.html#installation)
Install ModelScope and FunASR from pip
```shell
pip install -U modelscope funasr
# For the users in China, you could install with the command:
#pip install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
Or install FunASR from source code
```shell
git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip install -e ./
# For the users in China, you could install with the command:
# pip install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
#### Recipe
##### [Test CER](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_pipeline/asr_pipeline.html#inference-with-multi-thread-cpus-or-multi-gpus)
set the `model`, `data_dir` and `output_dir` in `infer.sh`.
```shell
cd egs_modelscope/asr/TEMPLATE
bash infer.sh
```
## Benchmark CER
### Chinese Dataset
<table border="1">
<tr align="center">
<td style="border: 1px solid">Model</td>
<td style="border: 1px solid">Offline/Online</td>
<td colspan="2" style="border: 1px solid">Aishell1</td>
<td colspan="4" style="border: 1px solid">Aishell2</td>
<td colspan="3" style="border: 1px solid">WenetSpeech</td>
</tr>
<tr align="center">
<td style="border: 1px solid"></td>
<td style="border: 1px solid"></td>
<td style="border: 1px solid">dev</td>
<td style="border: 1px solid">test</td>
<td style="border: 1px solid">dev_ios</td>
<td style="border: 1px solid">test_ios</td>
<td style="border: 1px solid">test_android</td>
<td style="border: 1px solid">test_mic</td>
<td style="border: 1px solid">dev</td>
<td style="border: 1px solid">test_meeting</td>
<td style="border: 1px solid">test_net</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary">Paraformer-large</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">1.76</td>
<td style="border: 1px solid">1.94</td>
<td style="border: 1px solid">2.79</td>
<td style="border: 1px solid">2.84</td>
<td style="border: 1px solid">3.08</td>
<td style="border: 1px solid">3.03</td>
<td style="border: 1px solid">3.43</td>
<td style="border: 1px solid">7.01</td>
<td style="border: 1px solid">6.66</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary">Paraformer-large-long</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">1.80</td>
<td style="border: 1px solid">2.10</td>
<td style="border: 1px solid">2.78</td>
<td style="border: 1px solid">2.87</td>
<td style="border: 1px solid">3.12</td>
<td style="border: 1px solid">3.11</td>
<td style="border: 1px solid">3.44</td>
<td style="border: 1px solid">13.28</td>
<td style="border: 1px solid">7.08</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/summary">Paraformer-large-contextual</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">1.76</td>
<td style="border: 1px solid">2.02</td>
<td style="border: 1px solid">2.73</td>
<td style="border: 1px solid">2.85</td>
<td style="border: 1px solid">2.98</td>
<td style="border: 1px solid">2.95</td>
<td style="border: 1px solid">3.42</td>
<td style="border: 1px solid">7.16</td>
<td style="border: 1px solid">6.72</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary">Paraformer-large-online</a> </td>
<td style="border: 1px solid">Online</td>
<td style="border: 1px solid">2.37</td>
<td style="border: 1px solid">3.34</td>
<td style="border: 1px solid">4.04</td>
<td style="border: 1px solid">3.86</td>
<td style="border: 1px solid">4.38</td>
<td style="border: 1px solid">4.21</td>
<td style="border: 1px solid">4.55</td>
<td style="border: 1px solid">10.64</td>
<td style="border: 1px solid">7.78</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://modelscope.cn/models/damo/speech_paraformer_asr_nat-zh-cn-16k-common-vocab8358-tensorflow1/summary">Paraformer</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">3.24</td>
<td style="border: 1px solid">3.69</td>
<td style="border: 1px solid">4.58</td>
<td style="border: 1px solid">4.63</td>
<td style="border: 1px solid">4.83</td>
<td style="border: 1px solid">4.71</td>
<td style="border: 1px solid">4.19</td>
<td style="border: 1px solid">8.32</td>
<td style="border: 1px solid">9.19</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://modelscope.cn/models/damo/speech_UniASR_asr_2pass-zh-cn-16k-common-vocab8358-tensorflow1-online/summary">UniASR</a> </td>
<td style="border: 1px solid">Online</td>
<td style="border: 1px solid">3.34</td>
<td style="border: 1px solid">3.99</td>
<td style="border: 1px solid">4.62</td>
<td style="border: 1px solid">4.52</td>
<td style="border: 1px solid">4.77</td>
<td style="border: 1px solid">4.73</td>
<td style="border: 1px solid">4.51</td>
<td style="border: 1px solid">10.63</td>
<td style="border: 1px solid">9.70</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://modelscope.cn/models/damo/speech_UniASR-large_asr_2pass-zh-cn-16k-common-vocab8358-tensorflow1-offline/summary">UniASR-large</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">2.93</td>
<td style="border: 1px solid">3.48</td>
<td style="border: 1px solid">3.95</td>
<td style="border: 1px solid">3.87</td>
<td style="border: 1px solid">4.11</td>
<td style="border: 1px solid">4.11</td>
<td style="border: 1px solid">4.16</td>
<td style="border: 1px solid">10.09</td>
<td style="border: 1px solid">8.69</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer_asr_nat-aishell1-pytorch/summary">Paraformer-aishell</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">4.88</td>
<td style="border: 1px solid">5.43</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://modelscope.cn/models/damo/speech_paraformerbert_asr_nat-zh-cn-16k-aishell1-vocab4234-pytorch/summary">ParaformerBert-aishell</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">6.14</td>
<td style="border: 1px solid">7.01</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformer_asr_nat-zh-cn-16k-aishell2-vocab5212-pytorch/summary">Paraformer-aishell2</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">5.82</td>
<td style="border: 1px solid">6.30</td>
<td style="border: 1px solid">6.60</td>
<td style="border: 1px solid">5.83</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
</tr>
<tr align="center">
<td style="border: 1px solid"> <a href="https://www.modelscope.cn/models/damo/speech_paraformerbert_asr_nat-zh-cn-16k-aishell2-vocab5212-pytorch/summary">ParaformerBert-aishell2</a> </td>
<td style="border: 1px solid">Offline</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">4.95</td>
<td style="border: 1px solid">5.45</td>
<td style="border: 1px solid">5.59</td>
<td style="border: 1px solid">5.83</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
<td style="border: 1px solid">-</td>
</tr>
</table>
### English Dataset
data/list/train.jsonl
+4
View File
@@ -0,0 +1,4 @@
{"key": "BAC009S0764W0121", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav", "source_len": 90, "target": "甚至出现交易几乎停滞的情况", "target_len": 13}
{"key": "BAC009S0916W0489", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav", "source_len": 90, "target": "湖北一公司以员工名义贷款数十员工负债千万", "target_len": 20}
{"key": "asr_example_cn_en", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav", "source_len": 91, "target": "所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些也许对", "target_len": 19}
{"key": "ID0012W0014", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav", "source_len": 88, "target": "he tried to think how it could be", "target_len": 8}
data/list/train_emo.txt
+4
View File
@@ -0,0 +1,4 @@
BAC009S0764W0121 <|NEUTRAL|>
BAC009S0916W0489 <|NEUTRAL|>
asr_example_cn_en <|NEUTRAL|>
ID0012W0014 <|NEUTRAL|>
data/list/train_event.txt
+4
View File
@@ -0,0 +1,4 @@
BAC009S0764W0121 <|Speech|>
BAC009S0916W0489 <|Speech|>
asr_example_cn_en <|Speech|>
ID0012W0014 <|Speech|>
data/list/train_text.txt
+4
View File
@@ -0,0 +1,4 @@
BAC009S0764W0121 甚至出现交易几乎停滞的情况
BAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万
asr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些也许对
ID0012W0014 he tried to think how it could be
data/list/train_text_language.txt
+4
View File
@@ -0,0 +1,4 @@
BAC009S0764W0121 <|zh|>
BAC009S0916W0489 <|zh|>
asr_example_cn_en <|zh|>
ID0012W0014 <|en|>
data/list/train_wav.scp
+4
View File
@@ -0,0 +1,4 @@
BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
asr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav
ID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav
data/list/val.jsonl
+2
View File
@@ -0,0 +1,2 @@
{"key": "ID0012W0013", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav", "source_len": 88, "target": "欢迎大家来体验达摩院推出的语音识别模型", "target_len": 19}
{"key": "ID0012W0014", "source": "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav", "source_len": 88, "target": "he tried to think how it could be", "target_len": 8}
data/list/val_text.txt
+2
View File
@@ -0,0 +1,2 @@
ID0012W0013 欢迎大家来体验达摩院推出的语音识别模型
ID0012W0014 he tried to think how it could be
data/list/val_wav.scp
+2
View File
@@ -0,0 +1,2 @@
ID0012W0013 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav
ID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav
docs/Makefile
+21
View File
@@ -0,0 +1,21 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = FunASR
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/README.md
+19
View File
@@ -0,0 +1,19 @@
# FunASR document generation
## Generate HTML
For convenience, we provide users with the ability to generate local HTML manually.
First, you should install the following packages, which is required for building HTML:
```sh
pip3 install -U "funasr[docs]"
```
Then you can generate HTML manually.
```sh
cd docs
make html
```
The generated files are all contained in the "FunASR/docs/_build" directory. You can access the FunASR documentation by simply opening the "html/index.html" file in your browser from this directory.
docs/benchmark/realtime_ws_benchmark.md
+92
View File
@@ -0,0 +1,92 @@
# Realtime WebSocket Benchmark
Use this benchmark when you need to measure the client-observable behavior of
`examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py` under
real streaming traffic. Offline `RTFx` and realtime service latency are
different metrics: this page focuses on first update latency, final latency
after `STOP`, response lag, and multi-client behavior.
The benchmark client accepts only 16 kHz mono PCM16 WAV input. Keeping the input
format strict removes resampling and file decoding from the measurement.
## Start the Service
For long continuous speech or multiple browser clients, start with a bounded
partial window and a moderate partial refresh interval:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
Speaker diarization is disabled by default. Add `--enable-spk` only when the
`spk` field is required, and report that setting with the benchmark result.
## Run a Single Realtime Replay
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 1 \
--output-jsonl realtime_ws_1c.jsonl
```
With pacing enabled, the client sends audio at realtime speed using 100 ms
frames. This is the closest mode to a microphone or browser stream.
## Run Concurrent Replays
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 8 \
--loops 3 \
--chunk-ms 100 \
--language 中文 \
--output-jsonl realtime_ws_8c.jsonl
```
Use a representative audio file. A long, pauseless monologue creates a very
different load shape from turn-taking meetings, because nearly every client is
speaking and triggering partial decodes at the same time.
For an unpaced stress test, add `--no-pace`. Treat that result as a throughput
stress signal, not as user-facing realtime latency.
## Metrics
| Metric | Meaning |
|--------|---------|
| `aggregate_audio_per_wall` | Total input audio seconds across all clients divided by benchmark wall time |
| `first_update_ms_p50/p95` | Time from first audio frame to first result message with `sentences`, `partial`, or `is_final` |
| `final_after_stop_ms_p50/p95` | Time from sending `STOP` to receiving the final result |
| `client_response_lag_ms_p95_max` | Largest per-client p95 of non-final `(client receive time - audio start) - server duration_ms`; useful mainly in paced mode for preview/partial lag |
| `partial_messages` | Count of non-final result messages with a non-empty `partial` |
| `final_messages` | Count of final result messages |
| `errors` | Connection, timeout, protocol, or client-side validation errors |
The script can observe only client-side timing and fields returned by the
server. If you are debugging service internals, collect server logs separately
for queue wait, VAD time, ASR decode time, speaker diarization time, GPU memory,
and GPU utilization.
## Report Template
When publishing a realtime WebSocket benchmark or issue report, include:
| Category | What to record |
|----------|----------------|
| Data | Audio duration, sample rate, language/domain, silence ratio or speaking pattern, and whether the same file was looped |
| Load | `--clients`, `--loops`, `--chunk-ms`, paced or `--no-pace`, and total benchmark wall time |
| Service | `serve_realtime_ws.py` command, `--partial-window-sec`, `--decode-interval`, `--enable-spk`, language, and hotwords |
| Hardware | GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions, CPU model, and available RAM |
| Software | `funasr`, PyTorch, torchaudio, vLLM, Python, OS, and container image if any |
| Output | Summary line, JSONL artifact, server logs, and any failed client IDs |
Do not reuse an offline `RTFx` number as a concurrency claim. For realtime
service sizing, benchmark with the actual traffic shape, especially sentence
length, pause distribution, simultaneous speakers, and whether speaker
diarization is enabled.
docs/benchmark/rtf_reproducibility.md
+107
View File
@@ -0,0 +1,107 @@
# Benchmark RTF and Reproducibility Notes
Use this page when comparing FunASR with Whisper, a cloud ASR provider, a Rust
runtime, or another self-hosted engine. Speed numbers are only useful when the
timing scope, data, model, runtime, and hardware are reported together.
## RTF and RTFx
FunASR benchmark tables usually report throughput as `RTFx`, or "times
realtime":
```text
RTF = processing_time_seconds / input_audio_seconds
RTFx = input_audio_seconds / processing_time_seconds
= 1 / RTF
```
For example, an `RTFx` value of `340` means 340 seconds of input audio are
processed in about 1 second, under that benchmark's data, runtime, batching, and
hardware setup. On the public vLLM table, the 184-file set has 11,541 seconds of
audio, so `340x` corresponds to roughly 34 seconds of measured processing time
for the whole set if the same scope is used:
```text
11541 / 340 = 33.94 seconds
```
Do not compare an offline batch `RTFx` result with streaming first-token latency
or end-to-end product latency. They measure different things.
For realtime WebSocket service sizing, use the
[Realtime WebSocket Benchmark](./realtime_ws_benchmark.md) instead.
## Current Public vLLM Benchmark Scope
The vLLM guide currently reports the following public scope for the Fun-ASR-Nano
and GLM-ASR-Nano table:
| Field | Value |
|-------|-------|
| Audio set | 184 long-form files |
| Total audio duration | 11,541 seconds, about 192 minutes |
| Models | Fun-ASR-Nano and GLM-ASR-Nano |
| Reported metric | CER and `RTFx` throughput |
| Fun-ASR-Nano vLLM batch result | `RTFx 340`, CER `8.20%` |
| Fun-ASR-Nano PyTorch baseline | `RTFx 21`, CER `8.06%` |
| Fun-ASR-Nano offline service without speaker diarization | `RTFx 102`, CER `8.14%` |
| Fun-ASR-Nano offline service with speaker diarization | `RTFx 46`, CER `8.19%` |
The table describes offline throughput on the stated long-form set. It should
not be read as a guarantee for every GPU, batch shape, language mix, streaming
chunk size, or service deployment.
The main website benchmark page is a separate public table for the broader ASR
comparison. It reports 184 long-form Chinese audio files, 11,539 seconds total,
and an NVIDIA H100 80GB HBM3 GPU. Keep the two tables separate when citing
numbers: the website table documents the general ASR benchmark, while the vLLM
guide table documents the Fun-ASR-Nano / GLM-ASR-Nano vLLM throughput rows.
## Required Fields for Reproducible Benchmark Claims
When publishing a FunASR benchmark, include these fields with the number:
| Category | What to record |
|----------|----------------|
| Data | File count, total audio duration, language/domain, sample rate, mono/stereo handling, and whether test files are public |
| Model | Model ID, checkpoint source, model revision or commit, language setting, hotwords, and text normalization |
| Runtime | Python SDK, ONNX, C++, vLLM, llama.cpp/GGUF, API server, or another path |
| Hardware | CPU model and thread count, GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions |
| Software | `funasr`, PyTorch, torchaudio, vLLM, ONNX Runtime, CUDA, Python, and operating system versions |
| Pipeline | VAD, punctuation, speaker diarization, ITN, timestamps, and post-processing on/off |
| Batching | Batch size, `batch_size_s`, concurrent requests, tensor parallel size, chunk size, VAD segment policy |
| Timing scope | Whether timing includes model download, cold start, warmup, file I/O, audio decoding/resampling, VAD, post-processing, and result serialization |
| Quality | CER/WER method, reference normalization, ignored tokens, and failed-file handling |
For official README or website numbers, include the fields above or link to a
report that includes them.
## Suggested Timing Protocol
1. Put all input audio in a manifest or directory and compute total duration
before running ASR.
2. Warm the model once if the published number is intended to represent steady
state throughput. If you include cold start, say so explicitly.
3. Time exactly one scope: model-only, pipeline-only, or end-to-end service.
4. Run the same scope at least three times and report median plus min/max.
5. Keep transcript output, failed-file list, and timing JSON/CSV with the run.
For migration or product evaluation, start from
[`examples/migration/benchmark_funasr.py`](../../examples/migration/benchmark_funasr.py).
It writes per-file timing and a Markdown summary for your own audio set. The
same reporting fields above also apply when you use vLLM, ONNX, C++, GGUF, or a
custom runtime instead of the migration example.
## Comparing with a Rust or Other Custom Runtime
For a fair engine-to-engine comparison:
- use the same audio files and total duration;
- resample and downmix with the same policy;
- keep VAD, punctuation, speaker diarization, and timestamps either all on or
all off;
- compare both speed and quality, because a faster decode path can change CER;
- report `RTFx` and raw processing seconds, not only a relative speedup.
If you can share your result publicly, open a Migration Benchmark Report issue
with the fields above. That makes the comparison useful to other users and gives
maintainers enough context to reproduce or improve the path.
docs/blog_whisper_vs_funasr_zh.md
+110
View File
@@ -0,0 +1,110 @@
# FunASR 实测：比 Whisper 快 30 倍的开源语音识别，还自带说话人分离
> 本文适合正在使用 OpenAI Whisper 做语音转写、但遇到速度慢或中文效果差问题的开发者。
## 背景
OpenAI Whisper 是目前最流行的开源 ASR 模型，但在生产环境中经常遇到几个痛点：
1. **速度慢** — Whisper-large-v3 在 A100 上只有 13x 实时，处理 1 小时音频需要 ~5 分钟
2. **中文方言差** — 对粤语、四川话、上海话等方言识别率很低
3. **缺少说话人分离** — 需要额外集成 pyannote，增加复杂度
4. **没有流式能力** — 无法做实时字幕
这些正是 [FunASR](https://github.com/modelscope/FunASR) 解决的问题。
## 性能对比
测试条件：184 个中文会议录音，总计 192 分钟，NVIDIA A100。
| 模型 | GPU 速度 | CPU 速度 | 说话人 | 流式 |
|------|---------|---------|--------|------|
| **FunASR SenseVoice-Small** | **170x** 实时 | **17x** 实时 | ✅ | ❌ |
| **FunASR Fun-ASR-Nano (vLLM)** | **340x** 实时 | — | ✅ | ✅ |
| Whisper-large-v3-turbo | 46x 实时 | ❌ | ❌ | ❌ |
| Whisper-large-v3 | 13x 实时 | ❌ | ❌ | ❌ |
**关键发现：FunASR 在 CPU 上比 Whisper 在 GPU 上还快。**
## 最大区别：一站式 vs 拼装
用 Whisper 搭建一个完整的转写系统，你需要：
```
Whisper (ASR) + pyannote (说话人) + silero-vad (VAD) +
deepmultilingualpunctuation (标点) + 自己写逻辑拼接
```
用 FunASR，**一次调用搞定**：
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
)
result = model.generate(input="meeting.wav")
# 输出：带说话人、时间戳、标点的结构化结果
```
## OpenAI 兼容 API：零改动迁移
如果你已经在用 OpenAI 的 Whisper API，迁移到 FunASR 不需要改一行客户端代码：
```bash
# 一行启动本地 ASR 服务
pip install torch torchaudio
pip install funasr vllm fastapi uvicorn python-multipart
funasr-server --device cuda
```
```python
# 客户端代码完全不变
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
result = client.audio.transcriptions.create(
model="fun-asr-nano",
file=open("meeting.wav", "rb")
)
```
## 中文方言支持
这是 Whisper 生态完全没有的：
- **7 大方言**：吴语、粤语、闽语、客家话、赣语、湘语、晋语
- **26 种地域口音**：河南、四川、广东、湖北、云南等
- **歌词识别**：音乐背景下的人声转写
## 什么时候选 FunASR？
| 场景 | 推荐 |
|------|------|
| 中文会议转写 | FunASR（方言+说话人） |
| 实时字幕/直播 | FunASR（流式 WebSocket） |
| 批量文件处理 | FunASR（vLLM 340x 加速） |
| 私有化部署 | FunASR（MIT 开源，无费用） |
| 纯英文、少量使用 | Whisper 也可以 |
## 开始使用
```bash
pip install torch torchaudio
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(model="iic/SenseVoiceSmall", device="cuda")
result = model.generate(input="audio.wav")
print(result[0]["text"])
```
更多信息：
- GitHub：https://github.com/modelscope/FunASR（16K+ stars）
- 官网：https://www.funasr.com
- 完整评测：https://modelscope.github.io/FunASR/benchmark.html
- 从 Whisper 迁移指南：https://github.com/modelscope/FunASR/blob/main/docs/migration_from_whisper.md
docs/cli.md
+117
View File
@@ -0,0 +1,117 @@
# Command-Line Interface
FunASR provides an agent-friendly CLI for speech recognition from the terminal. Designed for AI agents (Claude Code, Codex, Cursor), shell scripts, and automation pipelines.
## Installation
```bash
pip install funasr
```
## Basic Usage
```bash
# Transcribe audio (simplest)
funasr audio.wav
# Specify model
funasr audio.wav --model paraformer
# JSON output (structured, parseable)
funasr audio.wav --output-format json
# SRT subtitles
funasr audio.wav --output-format srt --output-dir ./subs
```
## Options
| Option | Short | Default | Description |
|--------|-------|---------|-------------|
| `--model` | `-m` | sensevoice | Model: sensevoice, paraformer, paraformer-en, fun-asr-nano |
| `--hub` | `-H` | ms | Model hub: ms (ModelScope) or hf (Hugging Face) |
| `--language` | `-l` | auto | Language: zh, en, ja, ko, yue, auto |
| `--device` | | auto | Device: cuda:0, cpu |
| `--output-format` | `-f` | text | Output: text, json, srt, tsv |
| `--output-dir` | `-o` | stdout | Write output files to directory |
| `--timestamps` | | off | Include word-level timestamps |
| `--spk` | | off | Enable speaker diarization |
| `--hotwords` | | none | Comma-separated hotwords |
| `--verbose` | `-v` | off | Show loading/timing info on stderr |
## Output Formats
### text (default)
Plain transcription text, one result per file. Best for piping:
```bash
funasr audio.wav | wc -w
```
### json
Structured output for programmatic use:
```json
{
"text": "欢迎大家来体验达摩院推出的语音识别模型",
"segments": [
{"start": 0, "end": 5540, "text": "欢迎大家来体验达摩院推出的语音识别模型"}
],
"file": "audio.wav",
"model": "sensevoice",
"language": "auto",
"duration_s": 0.29
}
```
### srt
SubRip subtitle format:
```
1
00:00:00,000 --> 00:00:05,540
欢迎大家来体验达摩院推出的语音识别模型
```
### tsv
Tab-separated values (start/end in seconds):
```
start end text
0.000 5.540 欢迎大家来体验达摩院推出的语音识别模型
```
## Advanced Examples
```bash
# Speaker diarization + JSON
funasr meeting.wav --spk --timestamps -f json
# Batch transcribe all WAV files
funasr *.wav --output-format srt --output-dir ./output
# Chinese with hotwords
funasr audio.wav --model paraformer --language zh --hotwords "FunASR,达摩院"
# Pipe to jq for processing
funasr audio.wav -f json | jq '.text'
# Load models from Hugging Face instead of ModelScope
funasr audio.wav --hub hf --model fun-asr-nano
# Use with AI agents
result=$(funasr audio.wav -f json)
echo "$result" | jq -r '.text'
```
## Models
| Model | Languages | Speed | Best for |
|-------|-----------|-------|----------|
| sensevoice | 50+ | ~70ms/10s | General use, multilingual |
| paraformer | zh + mixed | ~60ms/10s | Chinese production (with punctuation) |
| paraformer-en | en | ~60ms/10s | English |
| fun-asr-nano | 31 | varies | Encoder+LLM, complex audio |
## Legacy CLI
The original Hydra-based CLI is available as `funasr-hydra`:
```bash
funasr-hydra ++model=paraformer-zh ++input=audio.wav
```
docs/community_growth_20k.md
+373
View File
@@ -0,0 +1,373 @@
# FunASR Ecosystem Growth Plan: +20,000 GitHub Stars
Baseline on 2026-06-27: the FunASR ecosystem repositories had 31,224 combined GitHub stars. The next target is earning 20,000 additional stars across the ecosystem by 2026-09-30, reaching 51,224+ combined stars through better onboarding, stronger proof, and repeatable launches.
This plan focuses on useful adoption work rather than vanity marketing: if more users can install, evaluate, deploy, and share FunASR successfully, stars follow naturally.
## Repository scope
| Repository | Role in the ecosystem | Growth surface |
|---|---|---|
| `modelscope/FunASR` | Core toolkit, Python package, runtime services, deployment docs | First-run success, production deployment, issue/PR operations |
| `FunAudioLLM/Fun-ASR` | Fun-ASR-Nano model family and LLM-ASR identity | Model cards, Transformers integration, benchmarks, vLLM/GGUF stories |
| `FunAudioLLM/SenseVoice` | CPU-friendly multilingual ASR with emotion and audio events | Quick demos, edge use cases, app integrations |
| `modelscope/FunClip` | Video clipping and transcription workflow | Creator workflows, local GUI/API recipes, showcase stories |
## North-star metrics
- GitHub stars: +20,000 across the four-repository ecosystem by 2026-09-30
- Monthly PyPI downloads: sustained growth after each release
- README quick-start success: first transcription in under 5 minutes
- Deployment success: API server, WebSocket, Docker, and vLLM examples verified on fresh machines
- Community throughput: faster issue triage, more external PRs, more user showcases
## Audience segments
| Audience | What they need | Repository surface that should serve them |
|---|---|---|
| ASR developers | Fast local transcript, model choice, Python API | README quick start, tutorial, model zoo |
| Production teams | Private deployment, streaming, API compatibility, Docker | runtime docs, OpenAI API example, vLLM guide |
| Agent builders | Speech input to Claude/Cursor/LangChain/Dify/AutoGen | MCP server and OpenAI API examples |
| Researchers | Benchmarks, reproducibility, citations, model details | benchmark report, docs, model cards |
| Chinese users | Chinese docs, ModelScope path, deployment in China | README_zh, docs/zh, ModelScope links |
| International users | English docs, Hugging Face path, comparison with Whisper | README, HF models, benchmark story |
## Current campaign snapshot
As of 2026-07-08 07:32 UTC, the ecosystem has 35,109 combined GitHub stars, or 3,885 additional stars since the 31,224 baseline. The remaining gap to the +20,000 target is 16,115 stars by 2026-09-30, which requires roughly 192 stars/day across the remaining 84 days.
| Repository | Stars | Forks | Open issues | Open PRs | Last push |
|---|---:|---:|---:|---:|---|
| `modelscope/FunASR` | 19,041 | 1,912 | 2 | 0 | 2026-07-08 |
| `FunAudioLLM/Fun-ASR` | 1,363 | 134 | 0 | 0 | 2026-07-07 |
| `FunAudioLLM/SenseVoice` | 8,807 | 789 | 0 | 0 | 2026-07-07 |
| `modelscope/FunClip` | 5,898 | 709 | 0 | 0 | 2026-07-07 |
Keep this snapshot fresh during weekly planning. The ecosystem mode also reports the remaining gap, days left to 2026-09-30, and the required daily average:
```bash
python scripts/collect_growth_metrics.py --ecosystem
python scripts/collect_growth_metrics.py --ecosystem --format json
```
## Traffic funnel snapshot
The highest owned conversion surfaces are the repository overviews and Chinese READMEs. Use these surfaces for model-selection clarity, deployment proof, and official ecosystem routing before adding new long-form pages.
14-day GitHub traffic as of 2026-07-07 18:46 UTC:
| Repository | Views | Unique visitors | Clones | Unique cloners | Highest-traffic owned paths |
|---|---:|---:|---:|---:|---|
| `modelscope/FunASR` | 40,588 | 9,650 | 12,635 | 2,144 | overview: 7,243 uniques; `README_zh.md`: 2,521; releases: 469 |
| `FunAudioLLM/Fun-ASR` | 6,173 | 1,926 | 1,023 | 817 | overview: 1,584 uniques; `README_zh.md`: 581; `docs/vllm_guide.md`: 148 |
| `FunAudioLLM/SenseVoice` | 8,917 | 3,477 | 1,028 | 620 | overview: 2,802 uniques; `README_zh.md`: 923; releases: 163 |
| `modelscope/FunClip` | 2,764 | 1,366 | 622 | 455 | overview: 1,187 uniques; `README_zh.md`: 327; UI screenshots: 203 combined |
Top referrer pattern:
- Search and GitHub dominate all four repos, so SEO snippets, GitHub topics, and README first-screen clarity matter more than low-frequency social posts.
- `funasr.com`, `modelscope.github.io`, Hugging Face, ChatGPT, and `ai-bot.cn` are meaningful secondary referrers; keep model cards and homepage links synchronized with README claims.
- FunClip has unusually visible image traffic, so screenshot freshness and UI clarity can influence creator conversion.
## Workstream 1: Convert first-time visitors
- Keep the README first screen focused on speed, supported tasks, one-call usage, and links to docs.
- Keep the benchmark table close to the quick start, with exact benchmark scope and full report link.
- Maintain a short model-selection table: SenseVoice, Paraformer, Fun-ASR-Nano, Qwen3-ASR, GLM-ASR, streaming, punctuation, VAD, speaker, emotion.
- Add visible links to deployment, agent integration, contribution, and discussions.
- Avoid burying install and first inference behind long research history.
## Workstream 2: Make deployment shareable
- Verify `funasr-server --device cuda` on a clean GPU machine and publish the exact command/output.
- Add curl examples for `/v1/audio/transcriptions` and WebSocket streaming.
- Keep Docker image tags explicit and document CPU/GPU differences.
- Add a small deployment decision table: Python API vs OpenAI API vs WebSocket vs Docker vs vLLM vs Triton.
- Turn common deployment failures into FAQ entries.
## Workstream 3: Launch content that earns stars
Use one release note or article per clear story:
- "FunASR is 170x realtime and CPU viable for long audio."
- "Self-hosted OpenAI-compatible speech transcription with one command."
- "Streaming ASR with VAD, punctuation, and speaker diarization."
- "MCP speech input for Claude/Cursor and agent workflows."
- "vLLM acceleration for Fun-ASR-Nano."
- "Chinese dialect, accent, meeting, and industrial ASR examples."
For each launch, prepare:
- GitHub release notes with GIF/screenshot or terminal output
- README update and docs link
- Hugging Face and ModelScope model-card update
- Homepage update on `modelscope.github.io/FunASR`
- Short posts for developer communities
- A pinned GitHub discussion for feedback
## Workstream 4: Community operations
- Use issue templates to collect reproducible environment, audio, and deployment details.
- Triage issues weekly with labels: bug, question, documentation, deployment, enhancement, good first issue.
- Convert repeated support answers into docs within 48 hours.
- Keep 5-10 `good first issue` tasks ready for contributors.
- Thank external contributors in release notes.
Daily maintainer loop:
- Check open issues and PRs in `modelscope/FunASR`, `FunAudioLLM/Fun-ASR`, `FunAudioLLM/SenseVoice`, and `modelscope/FunClip`.
- Prioritize blockers that stop a first transcription, an API server launch, a WebSocket deployment, or a model download.
- Keep external ecosystem PRs unblocked, especially integrations in high-visibility AI, agent, workflow, ASR, and video repositories.
- Turn repeated answers into docs or examples before closing the loop.
- Keep actionable issues labelled as `good first issue`, `help wanted`, or `ready for PR` so contributors can help without guessing.
Use the issue snapshot before each maintainer pass so waiting-on-reporter, waiting-on-contributor, and maintainer-owned items stay separate:
```bash
python scripts/collect_growth_metrics.py --issues
python scripts/collect_growth_metrics.py --issues --format json
```
## Workstream 5: External proof
- Publish reproducible benchmark scripts and raw configuration for the 184-file benchmark.
- Encourage users to share production stories, dashboards, public demos, or benchmark notes through the showcase issue template.
- Add third-party integrations and community projects to README when they are maintained and runnable.
- Keep citations and paper links visible for researchers.
- Track and unblock integrations in Hugging Face Transformers, inference runtimes, agent frameworks, workflow tools, video tools, and speech-recognition libraries.
## External integration tracker
High-visibility external integrations can create more qualified traffic than a one-off announcement because they put FunASR in the workflow users already trust. Track them as an operations queue, not as a comment-ping list.
Refresh this table before weekly planning and after any reviewer or CI change:
```bash
python scripts/collect_growth_metrics.py --integrations
python scripts/collect_growth_metrics.py --integrations --format json
```
Set `GITHUB_TOKEN` when running this from CI or a shared network so GitHub's public API rate limit does not hide the real PR state.
### External adoption issue queue
High-star feature requests and roadmap issues are earlier in the funnel than PRs: they shape whether upstream projects expose a clean local STT contract before any code branch exists. Track them separately from merge-ready PRs and comment only when a maintainer asks for implementation detail, test scope, or provider behavior.
| Adoption issue | Growth reason | Current maintainer action |
|---|---|---|
| `openclaw/openclaw#35835` audio file support in the read tool | Extends a high-visibility agent read path from images to audio, with a natural fallback slot for private OpenAI-compatible STT when the active model cannot consume native audio | Keep native audio attachments separate from transcription fallback: audio-capable models should receive original media, while text-only models can use an explicit `POST /v1/audio/transcriptions` adapter returning transcript plus metadata. FunASR/SenseVoice are useful self-hosted regression backends for that fallback contract. |
| `openclaw/openclaw#40078` silent preflight transcription in mention-gated channels | Turns voice-note history capture into a high-visibility STT workflow where private transcription can enrich context without triggering assistant replies | Track the proposed `preflightTranscribe` gate: server-side STT should run only after channel/attachment checks, write transcript provenance into history, keep response generation mention-gated, and use an OpenAI-compatible transcription backend so hosted and local FunASR/SenseVoice services share the same contract. |
| `openclaw/openclaw#45508` self-hosted STT/TTS provider support in WebChat | Routes OpenClaw WebChat users toward private OpenAI-compatible STT providers instead of browser-only speech APIs | A batch push-to-talk path has been scoped: browser records a blob, the Gateway calls the configured `tools.media.audio` / OpenAI-compatible transcription backend, and streaming can remain a follow-up. Watch for product/API decisions before proposing code. |
| `openclaw/openclaw#46661` custom ASR server configuration | Connects a 382k-star OpenClaw native voice-input request to self-hosted/private ASR, giving FunASR, SenseVoice, and Qwen3-ASR a clean OpenAI-compatible STT provider path | LauraGPT posted the provider-contract note at https://github.com/openclaw/openclaw/issues/46661#issuecomment-4909572545. Track maintainer decisions on platform default behavior, explicit credential isolation for custom endpoints, multipart `{base_url}/audio/transcriptions`, minimum `{ "text": "..." }` responses, optional segment/timestamp metadata, timeout messaging, and fallback tests before proposing code. |
| `openclaw/openclaw#87140` pluggable macOS Push-to-Talk STT backend | Opens a native desktop path for FunASR/SenseVoice as local/private ASR behind OpenClaw's existing PTT UX | Keep Apple Speech as the default and watch for a batch-only provider boundary with explicit capability flags (`batch`, `streaming`, `languages`, `local_only`, `requires_network`). |
| `openclaw/openclaw#98027` editable Control UI dictation | Turns a 382k-star agent UI's ordinary prompt composer into a local/private STT adoption path where FunASR or SenseVoice can sit behind the existing Gateway/media-audio provider boundary | PR is open with `check-docs` failure and missing real microphone -> Gateway -> editable draft proof, so do not call it merge-ready. LauraGPT posted the downstream ASR contract checklist at https://github.com/openclaw/openclaw/pull/98027#issuecomment-4911740942; track whether proof covers local/private transcription provider dispatch, no auto-send, redacted errors, and temp-audio cleanup on success/failure. |
| `openclaw/openclaw#102028` QQBot speech transcription timeout | Hardens a very high-visibility voice-message transcription path so stalled hosted or self-hosted STT providers do not block inbound QQBot attachments indefinitely | PR is open and behind main while checks continue; LauraGPT posted a non-blocking self-hosted STT contract note at https://github.com/openclaw/openclaw/pull/102028#issuecomment-4911671891. Track whether the timeout remains configurable through the public provider/audio model path and whether timeout errors stay distinguishable from auth/model/route errors for slow FunASR/SenseVoice endpoints. |
| `NousResearch/hermes-agent#56989` local STT docs and hardening | Helps self-hosted messaging gateways document fully local voice-note transcription for private deployments | Track whether docs define a backend-neutral `local_command` and local HTTP contract; FunASR/SenseVoice should fit as wrappers or OpenAI-compatible `/v1/audio/transcriptions` services. |
| `NousResearch/hermes-agent#16185` Telegram voice-message STT provider precedence | Turns Telegram voice notes in a high-visibility messaging agent into a direct local/private STT adoption path, where FunASR/SenseVoice can fit through either `local_command` wrappers or OpenAI-compatible transcription servers | Fresh user report says removing the OpenAI STT block made local STT work, suggesting provider precedence/config resolution rather than model failure. LauraGPT posted the backend-neutral provider-boundary checklist at https://github.com/NousResearch/hermes-agent/issues/16185#issuecomment-4911698678; track for a Telegram adapter or gateway fix that logs the selected provider and prevents silent OpenAI fallback. |
| `NousResearch/hermes-agent#60624` Windows ffmpeg discovery for Discord voice | Removes a Windows setup blocker in a high-visibility Discord voice transcription/TTS path before local/private STT providers such as FunASR/SenseVoice can work reliably | PR `NousResearch/hermes-agent#60627` adds a shared ffmpeg resolver for `FFMPEG_PATH`, PATH, and winget installs. LauraGPT linked the PR and validation back to the user report on 2026-07-08 at `https://github.com/NousResearch/hermes-agent/issues/60624#issuecomment-4911355349`; now monitor checks and maintainer feedback. |
| `anomalyco/opencode#33300` desktop voice input | Puts local speech input in a high-visibility coding-agent desktop app where privacy-sensitive users may avoid browser dictation | Watch for an app-side transcription provider abstraction that inserts transcripts into the prompt draft and keeps provider credentials out of the renderer. |
| `ggml-org/llama.cpp#24375` ASR model routing for WebUI mic input | Lets Qwen3-ASR / Fun-ASR-Nano act as the dedicated transcription model for a separate text-only chat model | Track the proposed `/v1/models` input-modality discovery and explicit transcription-model selector; avoid duplicate comments while a contributor is working on the UI behavior. |
| `Tencent/ncnn#6790` Qwen3-ASR ncnn port and multi-platform deployment | Opens a 23k-star mobile/edge inference path for Qwen3-ASR-style local transcription, where successful conversion can feed Fun-ASR-Nano/Qwen3-ASR users looking for lightweight on-device ASR | This is a 2026 Rhino-Bird activity issue, so LauraGPT posted a non-claim technical validation note rather than trying to own the task: https://github.com/Tencent/ncnn/issues/6790#issuecomment-4912104458. Watch for a public `Qwen3-ASR-ncnn` repo or PR, then verify PyTorch parity on fixed preprocessing, normalized CJK text, module-level encoder/projector/decoder drift, and Linux/Windows smoke commands. |
| `Wei-Shaw/sub2api#3754` OpenAI audio endpoint passthrough | Adds a Chinese gateway path for OpenAI-compatible STT/TTS clients and self-hosted FunASR/SenseVoice endpoints | Watch for standard multipart `POST /v1/audio/transcriptions` passthrough, language/model forwarding, and a CJK regression that preserves `language=zh`. |
| `elizaOS/eliza#14807` audio PII redaction pipeline | Makes timed ASR spans part of an agent safety workflow where local CJK/private transcription can be useful | Keep FunASR/SenseVoice positioned as optional verifier backends; avoid coupling the primary redaction path to one ASR engine. |
| `modelcontextprotocol/servers#4299` FunASR speech-to-text MCP server | Keeps FunASR visible in the canonical MCP server discussion for local speech tools used by Claude/Cursor-style clients | The compact upstream server now exists in `examples/mcp_server` with `transcribe_audio`, Dockerfile, `glama.json`, and stdio smoke evidence. LauraGPT posted the upstream-status update at https://github.com/modelcontextprotocol/servers/issues/4299#issuecomment-4901476584; next action is to track whether MCP maintainers prefer an external community-server reference instead of vendoring. |
| `crewAIInc/crewAI#5983` FunASR for voice-enabled agents | Routes a 55k-star multi-agent framework toward a provider-neutral voice command path where FunASR/SenseVoice can be a local OpenAI-compatible transcription backend | LauraGPT revived the stale broad request with a concrete `voice.stt.*` config, multipart `/v1/audio/transcriptions` contract, mock endpoint test shape, and agent-command handoff at https://github.com/crewAIInc/crewAI/issues/5983#issuecomment-4901293351. Watch for maintainer direction before opening a code PR. |
| `Significant-Gravitas/AutoGPT#13347` FunASR as an open-source STT backend | Keeps the 185k-star AutoGPT voice-input discussion anchored on a generic OpenAI-compatible transcription contract rather than a heavyweight FunASR-only dependency | LauraGPT first mapped the existing hard-coded Copilot Whisper route, then opened draft PR `Significant-Gravitas/AutoGPT#13500` and linked it back at https://github.com/Significant-Gravitas/AutoGPT/issues/13347#issuecomment-4908286807. Track whether maintainers prefer env-based `/v1/audio/transcriptions` configurability, a workflow block, or a separate self-hosted transcript pipeline. |
| `chatchat-space/Langchain-Chatchat#5479` FunASR/SenseVoice voice input | Puts FunASR in a 38k-star Chinese RAG/Agent app where local audio upload can feed existing Chat and knowledge-base flows | LauraGPT proposed an OpenAI-compatible ASR endpoint slice, including `asr.base_url`, multipart request shape, minimal `{ "text": "..." }` response, and mocked CI endpoint at https://github.com/chatchat-space/Langchain-Chatchat/issues/5479#issuecomment-4901293565. Monitor stale handling and maintainer appetite for a first audio-upload transcription path. |
| `royshil/obs-localvocal#314` SenseVoice/Paraformer engine option | Opens an OBS live-captioning path where SenseVoice/Paraformer can be evaluated through a cleaner ASR-engine boundary instead of a Whisper-only runtime | Maintainer is interested but capacity-limited until mid/late August. LauraGPT mapped current Whisper coupling points and suggested a facade-only first refactor at https://github.com/royshil/obs-localvocal/issues/314#issuecomment-4909662013; wait for a refactor branch or review request before proposing Sherpa-ONNX/FunASR runtime code. |
| `SevaSk/ecoute#203` FunASR alternative ASR backend | Places FunASR/SenseVoice in a local realtime transcription app where users already compare Whisper-compatible and local ASR backends | LauraGPT recommended a two-step path at https://github.com/SevaSk/ecoute/issues/203#issuecomment-4906862462: first expose an OpenAI-compatible local STT endpoint with `stt_base_url`, optional key, model, and language; later add a native backend only with explicit streaming, language, timestamp, and batch-window capabilities. Monitor for maintainer appetite or an implementation PR before posting again. |
| `521xueweihan/HelloGitHub#3296` FunASR Chinese open-source recommendation | Gives FunASR a high-visibility Chinese developer discovery path in a long-running open-source recommendation channel | LauraGPT posted a 2026-07-08 editor-ready refresh at `https://github.com/521xueweihan/HelloGitHub/issues/3296#issuecomment-4910800381` with current ecosystem star counts, category fit, quick-start shape, OpenAI-compatible endpoint value, and the FunClip related-project angle; wait for editor triage without duplicate bumps. |
| `duixcom/Duix-Avatar#605` FunASR/SenseVoice ASR backend option | Puts FunASR/SenseVoice into a 13k-star digital-human project where low-latency ASR plus emotion/audio-event tags can drive avatar interaction | Maintainers acknowledged the current-contract suggestion and said they noted it for subsequent version iterations. Track for a Duix-side version/update invite, then verify the existing `duix-avatar-asr` `/v1/preprocess_and_tran` contract before pushing any parallel OpenAI-compatible endpoint path. |
| `sgl-project/sglang-omni#924` ASR support roadmap | Coordinates Fun-ASR-Nano serving tasks across cache behavior, batching, evaluation, and benchmark parity in a serving runtime used by ASR/GPU deployment contributors | LauraGPT posted FunASR-side guidance on full-context encoder cache scope, per-sample speech embedding lengths, batching checks, and CJK evaluation coverage; monitor F-PR follow-ups and avoid duplicate roadmap comments. |
| `pollinations/pollinations#5321` next-model voting for STT | Places FunASR/SenseVoice/Fun-ASR-Nano in a high-traffic open inference model request funnel used by app builders who want hosted or OpenAI-compatible transcription models | LauraGPT posted one exact model request on 2026-07-07 under the TTS & STT voting thread; avoid duplicate comments and monitor for maintainer model-selection signals or requests for endpoint/spec details. |
| `cjpais/Handy#1626` Qwen3-ASR Intel Mac AMD Metal fallback | Reduces a visible local-dictation failure mode in a 25k-star desktop app, keeping Qwen3-ASR / SenseVoice users from abandoning the model after a backend-specific crash | Maintainer confirmed `transcribe.cpp` will avoid Metal by default on non-Apple-Silicon Macs in the next release. Watch the release, then verify Intel macOS + AMD Metal + Qwen3-ASR routes to CPU fallback while SenseVoice keeps its working path. |
| `Zackriya-Solutions/meetily#567` Apple Voice Memos drag-to-import | Routes a 20k-star privacy-first meeting assistant toward imported-audio transcription workflows where local STT providers can be evaluated | Treat this as macOS import normalization, not an ASR swap: inspect Voice Memos drag payloads, normalize them to a local `.m4a` path plus title/start-time/duration metadata, reuse the existing import/transcription provider pipeline, and keep FunASR/SenseVoice as later optional providers behind that boundary. GitHub returned `403 Blocked` on the first public comment attempt, so monitor without retrying until there is new maintainer activity or account policy changes. |
| `altic-dev/FluidVoice#547` remote transcription backend | Gives lightweight or Intel Macs a path to consume a LAN/GPU STT server instead of running every model locally | Relevant to FunASR/SenseVoice remote endpoints, but GitHub returned `403 Blocked` on both the first comment attempt and a 2026-07-08 retry after new maintainer/user activity. Maintainers currently framed near-term work as easier local server startup plus future Windows/Linux GPU support, not a remote consumer endpoint; do not retry again unless account policy or repository write policy changes. |
| `freestyle-voice/freestyle#415` custom OpenAI-compatible STT base URL | Routes a voice-dictation app's batch STT provider toward LAN/self-hosted FunASR/SenseVoice endpoints without forcing users through hosted OpenAI URLs | Author accepted LauraGPT's compatibility scope at https://github.com/freestyle-voice/freestyle/issues/415#issuecomment-4911863210: strip trailing `/v1` or slash before appending `/v1`, keep the first pass batch-only on the existing `openai` provider, reuse the current API key/model flow, and test both bare-host and `/v1` base URLs. Wait for the implementation PR before adding more comments. |
| Integration PR | Growth reason | Current maintainer action |
|---|---|---|
| `1c7/chinese-independent-developer#1065` FunClip listing link refresh | Keeps FunClip's existing entry in a 49k-star Chinese independent-developer directory pointed at the current official GitHub repository instead of the old organization homepage | PR updates the accepted FunClip listing's "更多介绍" link from `github.com/alibaba-damo-academy` to `github.com/modelscope/FunClip`; monitor for maintainer acceptance without extra pings. |
| `Kedreamix/Linly-Talker#151` FunASR link refresh | Keeps a 3.3k-star digital-avatar conversational system's ASR docs pointed at the current official FunASR repository, so users comparing Whisper and FunASR land on maintained setup docs | PR updates the root README and `ASR/README.md` references from `github.com/alibaba-damo-academy/FunASR` to `github.com/modelscope/FunASR`; monitor for maintainer acceptance without extra pings. |
| `huggingface/transformers#46180` Fun-ASR-Nano model support | Makes Fun-ASR-Nano usable through the default HF API surface and model docs | Current blocker is an unrelated LightOnOCR shared-cache failure in `tests_processors`; the account cannot rerun Hugging Face Actions jobs, so wait for a maintainer rerun and avoid duplicate comments unless new CI evidence appears. |
| `huggingface/transformers#47111` Qwen3-ASR hotword and language parsing fixes | Keeps the high-visibility Transformers Qwen3-ASR implementation aligned with upstream hotword/context behavior, language hints, and processor training paths used by downstream Fun-ASR-Nano/Qwen3-ASR adopters | The Qwen3-ASR slow job was explicitly rerun and reported no PR-specific failures on head `caf37300fc3d3333c3d7c2162a92a886e424eb1c`; the aggregate CI recap still has unrelated suite failures. Monitor for reviewer requests around hotword serialization, language parsing, and qwen3_asr processor tests before adding any FunASR-side comment. |
| `sgl-project/sglang-omni#898` Fun-ASR serving support | Exposes Fun-ASR-Nano through a high-visibility serving runtime for ASR benchmarks and GPU deployment | The contributor-owned branch is currently dirty; LauraGPT already posted a concrete ASR benchmark rename/docs conflict recipe, and the author replied that conflicts will be resolved during merging. Wait for contributor or maintainer conflict resolution before adding more comments. |
| `vllm-project/vllm#42478` Qwen3-ASR streaming postprocessing | Improves the upstream Qwen3-ASR streaming path used by OpenAI-compatible transcription clients, so downstream apps can consume clean SSE transcript deltas without model-specific `language ...<asr_text>` cleanup | LauraGPT posted downstream validation guidance on 2026-07-08. CI is currently blocked only by vLLM's pre-run gate: the PR needs a maintainer `ready`/`verified` label or an author history of 4+ merged PRs (current log found 2), so wait for maintainer review rather than adding code-change comments; still watch for a no-space CJK regression. |
| `vllm-project/vllm#47729` MOSS-Transcribe-Diarize support | Expands vLLM's OpenAI-compatible `/v1/audio/transcriptions` route for long-form ASR with timestamps and speaker labels, creating another high-visibility comparison point for Qwen3-ASR / Fun-ASR-Nano-style serving behavior | PR is open and `mergeable=true` with `mergeable_state=blocked`; LauraGPT posted a non-blocking downstream ASR contract note at https://github.com/vllm-project/vllm/pull/47729#issuecomment-4912065897. Watch that `response_format=json` keeps the plain `{ "text": "..." }` shape stable, any future `segments` remain additive, CJK long-form audio stays covered, and the new model registration does not alter existing Qwen3-ASR request/response behavior. |
| `tenstorrent/tt-metal#49104` Qwen3-ASR Blackhole/P150 bringup | Opens a hardware-accelerated Qwen3-ASR path with an OpenAI-compatible `/v1/audio/transcriptions` server for Tenstorrent users evaluating speech workloads | PR is open, approved, and process-blocked rather than waiting on FunASR feedback; LauraGPT posted a non-blocking API-contract note at https://github.com/tenstorrent/tt-metal/pull/49104#issuecomment-4911526665. Track merge/release docs so downstream FunASR users can compare TT, CPU, vLLM, and H100 timing scopes correctly. |
| `ray-project/ray#64053` Ray Serve FunASR ASR example | Puts FunASR in production serving docs for teams already using Ray | Monitor review, answer questions quickly, and keep the example command aligned with the current OpenAI-compatible API behavior. |
| `huggingface/optimum-intel#1801` OpenVINO support | Helps CPU and edge users evaluate Fun-ASR on Intel hardware | Current failed jobs are unrelated OpenVINO matrix failures in Pix2Struct, image-text quantization/export, and tiny-random T5; LauraGPT's direct mirror PR `#1856` was closed after maintainers said they will address the cleanup in a preliminary PR, so wait for that maintainer-side update before adding more comments. |
| `huggingface/speech-to-speech#319` SenseVoice STT handler | Adds SenseVoice/FunASR to local open-source voice-agent pipelines where low-latency STT is a core comparison point | Keep lint/import fixes on the PR head, explain optional `speech-to-speech[sensevoice]` install behavior, and answer handler-scope review quickly. |
| `OpenBMB/VoxCPM#349` Windows CUDA installer with SenseVoice fallback | Puts SenseVoice into a 32k-star multilingual TTS app's Windows install and first-run ASR path as the fallback when local Parakeet/CUDA is unavailable | Current head `9f34141cc81e04028c4e62ac652f2a66dd453dfa` is dirty only in `app.py`; LauraGPT posted the conflict recipe and light validation at https://github.com/OpenBMB/VoxCPM/pull/349#issuecomment-4905293176. Wait for author rebase or maintainer action without duplicate comments. |
| `livekit/agents#6176` FunASR/SenseVoice realtime STT plugin | Opens a path into LiveKit's realtime voice-agent ecosystem where local STT is evaluated alongside hosted providers | CI and CLA are green; avoid duplicate pings and monitor for maintainer review on plugin scope, package metadata, or optional dependency expectations. |
| `datajuicer/data-juicer#938` HumanVBench audio/video operators | Places FunASR/SenseVoice-style speech understanding into a 6k-star data processing toolkit used to evaluate human-centric video and multimodal datasets | Current PR is open and mergeable but process-blocked while unit-test jobs are still waiting; LauraGPT posted the FunASR/SenseVoice dependency and validation follow-up at https://github.com/datajuicer/data-juicer/pull/938#issuecomment-4905235851. Wait for CI or maintainer feedback before adding more comments. |
| `mahimairaja/voiceai#16` FunASR and SenseVoice STT resource listing | Places FunASR/SenseVoice in a curated voice-AI resource map for builders comparing STT providers and local speech stacks | Maintainer approved. A pre-existing Stars badge target caused `link-check` to fail; pushed `16c451d` to retarget the badge to the repo homepage, verified `lychee 0.23.0` with the CI args (`480 Total`, `0 Errors`), and posted the green-status note after GitHub reported combined status `success`. Wait for maintainer merge without further pings. |
| `punkpeye/awesome-mcp-servers#7153` FunASR MCP server listing | Exposes FunASR to a high-star MCP discovery list used by agent-tool builders looking for local speech tools | Glama manifest now points at the current `server.json` schema and declares the maintainer for verification; the MCP Dockerfile also carries the official Registry OCI ownership label. Next step is maintainer OAuth/Glama ingestion or a public OCI image plus matching MCP Registry `server.json`, then update the external PR only after a valid score badge or maintainer feedback. |
| `zts212653/clowder-ai#1083` Qwen3-ASR service unification | Puts Qwen3-ASR into a user-facing local STT service slot by making it a `whisper-stt` backend variant instead of a separate, easier-to-misconfigure service | Head `28067864e313cd03dd3d6f4ce0a72ec5b5026b47` reports green CI after fixes for Rosetta hardware detection, Qwen3-ASR install/server dispatch, async backend locking, temp WAV cleanup, and stale setup docs; author status update: https://github.com/zts212653/clowder-ai/pull/1083#issuecomment-4910797452. Wait for maintainer re-review without duplicate comments. |
| `run-llama/llama_index#21958` FunASR endpoint reader | Puts FunASR behind a LlamaIndex reader for OpenAI-compatible transcription endpoints used in RAG and agent pipelines | LauraGPT rechecked head `07a8599deaebe5e7a559d62174e7a872870c2f7e` at https://github.com/run-llama/llama_index/pull/21958#issuecomment-4905345545; author acknowledged the shared-reader pytest caveat at https://github.com/run-llama/llama_index/pull/21958#issuecomment-4905384536. Keep the endpoint contract clear and avoid forcing local `funasr` dependencies into the main package. |
| `run-llama/llama_index#21996` local FunASR reader | Gives LlamaIndex users a local SenseVoice/FunASR reader for private transcription workflows | Keep optional dependencies isolated, verify the reader does not affect default installs, and watch for maintainer guidance on package extras. |
| `mem0ai/mem0#5571` optional FunASR transcription helper | Adds local FunASR transcription to a high-star memory layer used by agent builders | Keep FunASR optional and make examples clear about local model downloads; the current failing Vercel preview is a Mem0 team authorization gate, not a FunASR code failure, so wait for maintainer-side preview access or review. |
| `Significant-Gravitas/AutoGPT#13500` configurable transcription endpoints | Opens a path from AutoGPT's high-visibility agent platform to local FunASR/SenseVoice gateways through the existing OpenAI-compatible transcription route | GitHub Actions are green after the typed `HeadersInit` fix; remaining blockers are the author-controlled CLA, AutoGPT-team Vercel preview authorization, and maintainer review, not FunASR-side code. |
| `harry0703/MoneyPrinterTurbo#1006` shareable video-generation presets | Puts subtitle, language, voice, and provider settings in a 96k-star short-video generator's reusable preset schema, creating the right safety boundary for later local/offline subtitle ASR or FunClip handoff workflows | LauraGPT posted a non-blocking preset-contract note at https://github.com/harry0703/MoneyPrinterTurbo/pull/1006#issuecomment-4912521164: export only stable non-secret choices such as subtitle provider, language, subtitle style, TTS/voice selection, and local/offline capability; keep API keys, bearer tokens, signed URLs, temporary media paths, cached model files, and unsanitized endpoint URLs out of shareable presets. Track review without asking this preset PR to add a FunASR/SenseVoice engine. |
| `harry0703/MoneyPrinterTurbo#911` OpenVINO local subtitle provider | Adds a hardware-accelerated local transcription provider to a 96k-star short-video generator's subtitle pipeline, creating the same `audio -> SRT/timestamped segments` seam that later FunASR/SenseVoice or FunClip workflows can reuse | LauraGPT posted a non-blocking provider-boundary note at https://github.com/harry0703/MoneyPrinterTurbo/pull/911#issuecomment-4912751845. Track whether the PR keeps `subtitle_provider` generic, handles missing OpenVINO packages/model directories before subtitle correction, preserves valid UTF-8 SRT and CJK punctuation, and avoids another settings migration for future `funasr` / `sensevoice` providers. |
| `maximhq/bifrost#5020` diarized OpenAI transcription compatibility | Keeps a high-visibility LLM gateway's OpenAI-compatible `/v1/audio/transcriptions` path safe for diarized STT responses, including downstream clients that compare hosted models with self-hosted FunASR/SenseVoice endpoints | PR is open with green checks and bot review confidence; LauraGPT posted a non-blocking downstream contract note at https://github.com/maximhq/bifrost/pull/5020#issuecomment-4911586742. The fix covers string diarized segment IDs, explicit empty `segments: []`, multipart transcription params, and speaker passthrough; watch merge/release feedback before adding more comments. |
| `getpaseo/paseo#313` FunASR streaming STT provider | Adds a standalone FunASR/SenseVoice server and streaming dictation provider to a 10k-star agent orchestration app | Current head `19d9270f1b55d5a0d59288ff2c4a4e707bd60c9a` is dirty against main; LauraGPT triage at https://github.com/getpaseo/paseo/pull/313#issuecomment-4905606566 documents the conflict set and a `needsFunASR` provider-wiring bug to fix during rebase. Wait for author rebase or maintainer guidance without duplicate pings. |
| `getpaseo/paseo#1634` SenseVoice local STT model support | Adds SenseVoice to a local voice/dictation app with a visible offline STT model catalog and speech CLI path | Current PR is mergeable at head `4cc65ac6fbc9735ac487648bbbd02ee487050f31`; posted focused local validation at https://github.com/getpaseo/paseo/pull/1634#issuecomment-4905550193 covering server deps build, format check, 3 speech Vitest files with 10 tests, and server build. Wait for maintainer review without duplicate pings. |
| `infiniflow/ragflow#16473` FunASR / SenseVoice STT provider | Adds FunASR to a high-star RAG workflow product where local STT can become a visible configuration choice | Track duplicate/conflict cleanup, keep provider naming consistent, and verify that skipped CI is expected rather than a hidden regression. |
| `mudler/LocalAI#10090` FunASR backend | Exposes FunASR through a high-star local AI engine used by self-hosters and edge deployments | Upstream DCO is green; the fork-side `tests` failure is a CodeQL/SARIF upload permission gate on a fork, not a FunASR backend regression. Local CPU smoke validation remains `backend/python/funasr/test.py` with four passing tests, so wait for maintainer-side review or rerun without duplicate pings. |
| `agno-agi/agno#8501` FunASR transcription tool | Places FunASR in a high-star agent platform as a local multilingual transcription tool | Keep issue linkage and formatting gates green, avoid heavy default dependencies, and respond to review scope requests quickly. |
| `GetStream/Vision-Agents#606` FunASR STT plugin | Adds SenseVoice/FunASR to multimodal voice and vision agent examples | Keep package include paths and tests aligned with upstream conventions, then watch review threads for optional dependency or documentation requests. |
| `TEN-framework/ten-framework#2191` FunASR ASR extension | Places SenseVoice/FunASR in a realtime multimodal agent framework with extension discovery | The failing `claude-review` action is a fork-permission gate, not a code failure; wait for maintainer-side review or rerun, and respond only to concrete code/doc feedback. |
| `activepieces/activepieces#13985` FunASR speech recognition piece | Gives no-code workflow users a direct FunASR speech recognition action | Replacement PR for conflicted `#13450` is open and mergeable at head `f9d22ee03c58199c7236e2f1008f083d6f80b4a2`; Greptile marks the additive community piece safe to merge and normal checks are green/skipped. Latest LauraGPT recheck: https://github.com/activepieces/activepieces/pull/13985#issuecomment-4904635178. Remaining blocker is the `license/cla` status, which is author/account-controlled and should not be signed by the operator. |
| `omnigent-ai/omnigent#2093` server-side streaming dictation | Opens a model-agnostic dictation backend in a multi-agent desktop/mobile harness where FunASR/SenseVoice can later fit behind the same local worker or OpenAI-compatible fallback contract | LauraGPT posted the non-blocking FunASR/SenseVoice backend-contract follow-up at https://github.com/omnigent-ai/omnigent/pull/2093#issuecomment-4907140875. Current PR is open and mergeable but blocked by `npm test`, pre-commit, and maintainer approval checks; wait for concrete review or CI logs before more comments. |
| `speaches-ai/speaches#658` FunASR transcription backend | Adds SenseVoice/Paraformer to an OpenAI-compatible local speech API used by self-hosted voice stacks | Current `unstable` state is a fork-workflow approval gate with no check-runs/logs on the head commit; keep the local validation note current and respond only if real CI logs or maintainer review appear. |
| `Uberi/speech_recognition#903` FunASR recognizer | Exposes FunASR through a widely known Python speech-recognition wrapper | Current `unstable` state is a fork-workflow approval gate with no check-runs/logs on the head commit; keep the recognizer optional and lightweight, and be ready with a minimal install/import smoke test if maintainers ask for scope reduction. |
| `fighting41love/funNLP#478` SenseVoice and FunClip speech-section listing | Adds the newer FunASR ecosystem projects to a high-star Chinese NLP resource list with strong domestic discovery value | PR is clean and mergeable; avoid duplicate pings unless maintainers ask for category, wording, or link changes. |
| `josephmisiti/awesome-machine-learning#1339` FunASR Python speech-recognition listing | Places FunASR in a broad high-star machine-learning discovery list where users compare Python ASR toolkits | PR is open but `mergeable_state=blocked`; recheck only for maintainer feedback or a new failed status, and keep the entry description aligned with the core toolkit rather than every model variant. |
| `crownpku/Awesome-Chinese-NLP#32` Speech Recognition and Audio section | Creates a Chinese-NLP discovery path that can route users to FunASR, SenseVoice, Fun-ASR-Nano, and FunClip together | PR is clean and mergeable; monitor without status bumps unless maintainers request section naming or Chinese wording changes. |
| `mahseema/awesome-ai-tools#1403` FunASR tool listing | Complements the FunClip listing in the same AI-tools catalog with the core ASR toolkit | PR is clean and mergeable; keep FunASR and FunClip descriptions distinct so one does not look like a duplicate of the other. |
| `INTERMT/Awesome-PyTorch-Chinese#5` FunASR Chinese PyTorch listing | Adds FunASR to a Chinese PyTorch resource list used by developers looking for local speech models and toolkits | PR is clean and mergeable; monitor for maintainer style feedback and keep the link pointed at the GitHub repo rather than a transient model page. |
| `krzjoa/awesome-python-data-science#99` FunASR computer-audition listing | Adds FunASR to a Python data-science discovery list where speech/audio tooling is compared with broader ML stacks | PR is clean and mergeable; maintain a short toolkit-focused description and avoid duplicate comments unless maintainers ask for category changes. |
| `zzw922cn/awesome-speech-recognition-speech-synthesis-papers#27` FunAudioLLM paper listing | Places Paraformer, FunASR, SenseVoice, and Fun-ASR-Nano papers in a speech-recognition/synthesis paper list used by researchers | PR is clean and mergeable; keep paper links stable and be ready to adjust ordering or paper metadata if requested. |
| `Osmantic/ODS#1639` SenseVoice/FunASR STT backend | Adds a direct local STT backend to an app surface where users can choose open speech engines | PR is mergeable but blocked by review/process state; wait for maintainer feedback and keep any follow-up focused on optional dependency and endpoint behavior. |
| `faroit/awesome-python-scientific-audio#85` FunASR scientific-audio listing | Exposes FunASR to Python users browsing scientific audio and speech-processing toolkits | PR is clean and mergeable; monitor for taxonomy or wording feedback without status bumps. |
| `joewongjc/type4me#207` Qwen3-only local ASR mode | Helps a local dictation app use Qwen3-ASR final transcription without keeping SenseVoice/VAD preview models resident | Keep as draft until a full Xcode/XCTest run or real Qwen3-only ASR smoke test validates the runtime path; do not request review yet. |
| `ga642381/speech-trident#31` SenseVoice model-list entry | Adds SenseVoice to a speech/audio language-model catalog that routes model researchers toward FunASR ecosystem components | PR is clean and mergeable; monitor for citation or category feedback. |
| `vinta/awesome-python#3246` SenseVoice listing | Places SenseVoice in the largest Python discovery list as a practical local speech-recognition option | PR is blocked by maintainer/review gate rather than a known code failure; watch for category or project-quality feedback before changing scope. |
| `RVC-Boss/GPT-SoVITS#2801` Fun-ASR-Nano fallback fix | Improves a high-star voice generation workflow by making Fun-ASR-Nano setup errors fall back cleanly instead of blocking users | PR is clean and mergeable; monitor for maintainer questions and keep the fix narrowly scoped to registration/fallback behavior. |
| `jobbole/awesome-python-cn#141` FunASR Chinese Python listing | Adds FunASR to a long-lived Chinese Python resource list used by domestic developers | PR is clean and mergeable; avoid duplicate comments unless maintainers ask for ordering or Chinese wording changes. |
| `yuekaizhang/Fun-ASR-vllm#21` vLLM prompt embedding dtype fix | Keeps a community Fun-ASR-vLLM GPU inference path stable for users trying Fun-ASR-Nano with vLLM | PR is clean and has a validation note. Monitor for maintainer feedback, keep the change limited to float32 prompt embeddings, and avoid extra status bumps unless runtime questions appear. |
| `openvino-agent/optimum-intel#5` FunASR OpenVINO review cleanup | Preserves an OpenVINO review-structure cleanup branch while the upstream optimum-intel path is maintainer-owned | Keep as a reference branch for review cleanup and do not ping externally unless maintainers ask for the generated-file or modularization details. |
| `EmulationAI/awesome-large-audio-models#19` FunAudioLLM paper listing | Adds Paraformer, FunASR, SenseVoice, and Fun-ASR-Nano papers to a large-audio-models discovery list | README-only PR with validation posted; wait for maintainer review and be ready to adjust paper ordering or citation text. |
| `ddlBoJack/Awesome-Speech-Language-Model#6` Fun-ASR-Nano listing | Routes speech language model readers toward Fun-ASR-Nano as an audio-LLM component | Validation was posted on 2026-07-07; PR is clean with no exposed checks, so monitor without duplicate pings. |
| `LqNoob/Neural-Codec-and-Speech-Language-Models#4` Fun-ASR-Nano and SenseVoice listing | Adds the FunAudioLLM speech understanding models to a neural codec and speech-language-model resource list | Validation was posted on 2026-07-07; wait for maintainer review and keep any follow-up focused on model categorization. |
| `PyTorchKR/oss-landscape#688` FunASR OSS landscape entry | Adds FunASR to a Korean PyTorch and OSS discovery map that can route regional users to the toolkit | PR is README-only, clean, and has no check contexts; avoid low-signal comments unless maintainers ask for metadata or category changes. |
| `metame-ai/awesome-audio-plaza#10` FunASR, SenseVoice, and FunClip ASR listing | Presents the full FunASR ecosystem together in an audio tools plaza used by speech and creator-tool readers | Prior pings are already present; monitor for maintainer feedback and keep future replies limited to link or category changes. |
| `ChristosChristofidis/awesome-deep-learning#317` FunASR framework listing | Adds FunASR to a 28k-star deep-learning discovery list where users compare frameworks and toolkits | Validation was refreshed on 2026-07-07; wait for maintainer review without another status bump. |
| `Hannibal046/Awesome-LLM#623` Fun-ASR-Nano Alibaba-section listing | Routes a large LLM audience to Fun-ASR-Nano as a speech understanding model in the Alibaba ecosystem | Validation was refreshed on 2026-07-07; wait for maintainer review and adjust section placement only if requested. |
| `AiHubCN/Awesome-Chinese-LLM#103` SenseVoice and Fun-ASR-Nano Chinese LLM listing | Adds FunAudioLLM speech models to a high-star Chinese LLM catalog with strong domestic discovery value | Validation was refreshed on 2026-07-07; PR is currently blocked by maintainer review/process state rather than a code failure. |
| `pluja/awesome-privacy#836` FunASR privacy-focused STT listing | Puts fully self-hosted FunASR in front of privacy-conscious users looking for local speech-to-text models | Validation was refreshed on 2026-07-07; wait for maintainer review and keep privacy wording concise. |
| `BradyFU/Awesome-Multimodal-Large-Language-Models#280` Fun-ASR-Nano MLLM listing | Adds Fun-ASR-Nano to a multimodal-LLM discovery list where speech understanding models are compared | Validation was refreshed on 2026-07-07; monitor for maintainer category feedback. |
| `mahmoud/awesome-python-applications#227` FunASR audio application listing | Adds FunASR to a broad Python applications list used by developers looking for usable audio tooling | Validation was refreshed on 2026-07-07; wait for maintainer review without extra pings. |
| `bharathgs/Awesome-pytorch-list#164` FunASR PyTorch speech listing | Adds FunASR to a PyTorch NLP and speech processing list with a broad ML practitioner audience | Validation was refreshed on 2026-07-07; PR is clean and should only need maintainer review. |
| `WangRongsheng/awesome-LLM-resources#162` FunASR and SenseVoice resources | Adds FunASR/SenseVoice to a Chinese LLM resources list where multimodal and speech tool readers compare projects | Validation was refreshed on 2026-07-07; wait for maintainer review and keep future replies focused on link stability. |
| `steven2358/awesome-generative-ai#821` FunASR speech-to-text listing | Routes generative-AI readers looking for STT tools toward the core FunASR repo | PR is clean and mergeable; monitor for section placement feedback. |
| `owainlewis/awesome-artificial-intelligence#243` FunASR audio listing | Adds another broad AI-discovery path for users comparing open-source audio tools | PR is clean and mergeable; keep the description concise and avoid status pings. |
| `ai4s-research/awesome-ai-for-science#69` FunASR science toolkit listing | Creates a discovery path from scientific AI tooling lists to FunASR for transcription and field-recording workflows | Validate the link, keep the description technically accurate, and avoid extra comments unless maintainers ask for category or wording changes. |
| `lukasmasuch/best-of-ml-python#455` FunASR project listing | Adds FunASR to a high-star ranked Python ML discovery list that is refreshed weekly | The PR is mergeable and already has prior pings; keep monitoring without adding more comments unless new maintainer feedback or validation evidence appears. |
| `tensorchord/Awesome-LLMOps#533` FunASR audio foundation model listing | Adds FunASR to an LLMOps discovery list where users look for model-serving and operations-ready audio foundation models | Current PR is README-only, clean, DCO-passing, and has fresh FunASR-side link validation; wait for maintainer review without duplicate pings. |
| `rafska/awesome-local-llm#118` FunASR local CPU ASR listing | Exposes the FunASR llama.cpp/GGUF path to local-LLM users looking for fully local CPU speech recognition | Current PR is README-only, clean, and has fresh link validation; monitor for maintainer feedback without adding another status comment. |
| `krzemienski/awesome-video#102` FunClip AI video tools listing | Adds FunClip to a curated streaming/video tools list where video engineers and creator-tool builders discover AI-assisted clipping and subtitle generation workflows | Sourcery's rerun passes on head `f90be00`; the PR is README-only, mergeable clean, and uses owner/repo-style link text `[modelscope/FunClip]`. LauraGPT posted a concise ready/validation note on 2026-07-08 at `https://github.com/krzemienski/awesome-video/pull/102#issuecomment-4911378040`; now wait for maintainer review without duplicate pings. |
| `CopilotKit/CopilotKit#5863` voice runtime transcription docs | Puts self-hosted OpenAI-compatible ASR behind a 35k-star agent UI voice surface by clarifying that custom `TranscriptionService` implementations can proxy CopilotKit `/transcribe` requests to local or LAN STT endpoints | LauraGPT posted a 2026-07-08 downstream compatibility note at `https://github.com/CopilotKit/CopilotKit/pull/5863#issuecomment-4911282937`, suggesting an optional self-hosted ASR callout for FunASR, SenseVoice, and Qwen3-ASR-style `/v1/audio/transcriptions` deployments. Wait for maintainer/doc-author feedback; do not turn this into a blocking review. |
| `manaflow-ai/cmux#7314` provider-neutral iOS voice backend | Puts a 23k-star AI terminal's iOS Voice Mode and composer dictation behind a recognition-backend seam where local/on-device Apple or Parakeet engines can later coexist with LAN/self-hosted FunASR/SenseVoice/Qwen3-ASR transcription gateways | LauraGPT posted a non-blocking backend-contract note at https://github.com/manaflow-ai/cmux/pull/7314#issuecomment-4912323678: keep engine capability metadata provider-neutral, send exactly one final transcript to `mobile.voice.input`, preserve fail-closed target checks outside the ASR engine, sanitize provider errors, and keep downloaded model state separate from any future remote endpoint settings. Track review without asking this PR to add another engine. |
| `QwenLM/qwen-code#6516` VoiceButton i18n and provider-error boundary | Makes voice dictation states and retry affordances translatable in the same high-visibility Qwen Web Shell surface, which matters for non-English users running local FunASR/SenseVoice STT backends | LauraGPT posted the local/private STT note at https://github.com/QwenLM/qwen-code/pull/6516#issuecomment-4911909587. Latest head `79f3294e651211de4095947eec35e33d94da5c7a` adds locale-cache invalidation after the earlier missed placeholder fix; Ubuntu Node 22 tests and coverage comment passed, while `review-pr` is still in progress. Reviewer `wenshao` downgraded to non-blocking comment at https://github.com/QwenLM/qwen-code/pull/6516#issuecomment-4912381983 and suggested i18n interpolation/timeline tests plus sentinel cleanup. Monitor checks/review; avoid another FunASR comment unless provider-error or voice insertion tests are requested. |
| `agent-of-empires/agent-of-empires#2585` composer action extension point | Adds a provider-agnostic composer action API in an agent UI, creating the right plugin boundary for local FunASR/SenseVoice dictation without hard-coding STT into core | Maintainer agreed with LauraGPT's provider-neutral boundary at https://github.com/agent-of-empires/agent-of-empires/pull/2585#issuecomment-4911979551 and asked the author to rebase. Keep STT-provider code out of this PR; after the extension point is clean, propose or review a plugin that calls an OpenAI-compatible FunASR/SenseVoice `/v1/audio/transcriptions` endpoint. |
| `tmoroney/auto-subs#652` Whisper download failure with SenseVoice fallback | Protects a live subtitle workflow after AutoSubs added SenseVoice: a Chinese-video user is blocked by `large-v3-turbo` whisper.cpp model download failure, not by audio normalization or ASR execution | LauraGPT posted safe triage at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4911843184: avoid unverified ZIP/bypass scripts, keep the Whisper fix in the model-manager/cache path, and use the built-in SenseVoice path from #629 plus the app translation route as the immediate `lang=zh` fallback. A new user asked for 1:1 help, so LauraGPT moved support back into the public issue at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4912472340; after they asked which option is SenseVoice, LauraGPT identified the UI label/model id at https://github.com/tmoroney/auto-subs/issues/652#issuecomment-4912653724. Track whether they can see `SenseVoice` / `sense-voice`, whether their build includes the newer engine, and whether Whisper remains isolated to download/cache handling. |
| `OpenWhispr/openwhispr#769` custom endpoint transcription contract | Keeps a privacy-first dictation app's BYOK/custom STT route compatible with OpenAI-style multipart transcription endpoints, the same wire shape used by self-hosted FunASR/SenseVoice gateways | LauraGPT posted the regression-contract checklist at https://github.com/OpenWhispr/openwhispr/issues/769#issuecomment-4909852583 after a user confirmed the OpenRouter fix branch worked. Track for a main/release-side fix that preserves `file`, `model`, optional `language`, optional `response_format=verbose_json`, exact `/audio/transcriptions` base-url joining, safe `audio/webm` handling, and provider-specific JSON/base64 only when required. |
| `mahseema/awesome-ai-tools#1689` FunClip video-tool listing | Adds FunClip to a high-visibility AI tools list where video creators discover clipping and subtitle workflows | Validate the FunClip link, keep the description aligned with local transcription/SRT/AI clipping capabilities, and avoid status pings after evidence is posted. |
Completed external wins:
| Integration | Growth reason | Result |
|---|---|---|
| `xinnan-tech/xiaozhi-esp32-server#3255` configurable FunASR language | Improves a high-star ESP32 voice-agent backend by letting users pin SenseVoice/FunASR language for short utterances | Merged on 2026-06-30 after adding the multi-module management-console database migration requested by maintainers. |
| `tmoroney/auto-subs#629` SenseVoice engine for subtitle workflows | Adds SenseVoice to an on-device subtitle generation app used by video editors and creators | Merged on 2026-06-30 after adding the declarative model manifest, SenseVoice/Canary/Cohere engines, real CTC timestamps, native translation routing, and robustness fixes. |
| `pipecat-ai/pipecat#4844` FunASR local STT service | Puts SenseVoice/FunASR into realtime voice agent pipelines used by builders comparing local STT backends | Merged on 2026-07-02 after adding the local STT service, optional dependency handling, docs, and validation coverage. |
| `liusongxiang/Large-Audio-Models#26` FunAudioLLM audio-language-model listing | Adds FunAudioLLM to a focused audio language model catalog where researchers compare speech and audio LLM projects | Merged on 2026-07-03 with a README-only FunAudioLLM entry. |
| `yuekaizhang/Fun-ASR-vllm#20` deterministic vLLM sampling for ASR | Improves a community Fun-ASR-vLLM inference path by making ASR generation deterministic enough for repeatable evaluation and demos | Merged on 2026-07-07 after switching the vLLM sampling path to deterministic decoding for ASR. Follow-up `#21` continues the prompt-embedding dtype cleanup. |
| `xorbitsai/inference#5140` Fun-ASR-Nano Xinference dependency fix | Unblocks Fun-ASR-Nano deployment through Xinference by moving model specs off an old FunASR git pin that predates `FunASRNano` registration | Merged on 2026-07-07 as `4a625fafa55827ee932f1e2cfcd362b9a7b4aabe`; LauraGPT synced FunASR deployment docs in https://github.com/modelscope/FunASR/pull/3117 and posted the merge follow-up at https://github.com/xorbitsai/inference/pull/5140#issuecomment-4901893926. Track Xinference release notes so FunASR docs can point users at a fixed build. |
| `debpalash/OmniVoice-Studio#1003` OpenAI-compatible ASR backend | Gives an 8k-star local voice/video studio a generic `POST /v1/audio/transcriptions` path that can point at self-hosted FunASR, SenseVoice, or Qwen3-ASR servers before direct Transformers support lands | Merged by maintainer `debpalash` on 2026-07-08 as `5a7d9cc05cda45682b11b5ece35190a5c39bbb6f` after LauraGPT's timeout, fallback-scope, key-clear, and error-sanitization review note. Watch release/user feedback for real FunASR/SenseVoice endpoint smoke results; do not post another follow-up unless those hardening items resurface. |
| `OpenWhispr/openwhispr#1093` transcript-retention fix | Strengthens a privacy-first dictation and meeting transcription app so local/private STT providers do not lose genuine microphone speech when echo evidence is audio-only | Merged on 2026-07-08 as `4d6f2d2e8b4602feb72f1493aff1f8f12de9e3f3` after LauraGPT highlighted the final-text retention contract; watch release feedback and future OpenAI-compatible/local provider work without posting duplicate follow-ups. |
| `elizaOS/eliza#15426` streaming ASR live transcript contract | Lands a high-star agent OS voice UX path where segment preview and final batch transcript semantics can support local/private FunASR or SenseVoice-style ASR providers | Merged on 2026-07-08 as `e8d780cd31d1b5e8a870c20e370da0010ce8eb2d` after LauraGPT called out the final-transcript safety contract at https://github.com/elizaOS/eliza/pull/15426#issuecomment-4911110675. Watch follow-up provider work and keep any FunASR bridge aligned with the final-only fallback contract. |
| `QwenLM/qwen-code#6510` pane-scoped voice dictation restore | Restores voice dictation in a 25k-star Qwen coding-agent Web Shell split-view, creating a visible local/private STT surface when the daemon advertises `voice_transcribe` | Merged on 2026-07-08 as `d8dc8043d6cfd4d7605f83b8a4838fee9dacdffa` after LauraGPT's pane-scoped local/private ASR boundary note at https://github.com/QwenLM/qwen-code/pull/6510#issuecomment-4911877569 and maintainer confirmation at https://github.com/QwenLM/qwen-code/pull/6510#issuecomment-4912022943: capability-gated mic visibility, editable transcript draft without auto-submit, pane-specific routing, and sanitized provider errors for FunASR/SenseVoice/OpenAI-compatible STT backends. |
| `chatmcp/mcpso#1` MCP server directory submission | Routes MCP builders browsing mcp.so toward FunASR's local speech-recognition MCP server | Listed on mcp.so on 2026-07-08 at https://mcp.so/server/mcp-server-funasr/radial-hks after LauraGPT submitted the server in the public directory thread. The MCP README now links the live directory page so Claude/Cursor-style users can discover `transcribe_audio` without reading the growth tracker. |
Operating rules:
- Comment on external PRs only when there is new evidence, a reviewer question, or a concrete unblock; avoid low-signal status bumps.
- Keep each integration's local verification command, CI link, and latest reviewer decision in the weekly notes.
- Promote an integration in FunASR README or release notes only after it is merged or has a maintained, runnable branch.
- When a third-party PR stalls, look for a smaller docs/example PR in that upstream instead of forcing a large runtime integration.
## Tracking cadence
Use the lightweight metrics script before and after major README, homepage, release, or demo updates:
```bash
python scripts/collect_growth_metrics.py
python scripts/collect_growth_metrics.py --format json
```
The script captures GitHub stars, forks, watchers, open issues, open pull requests, latest push time, and the current PyPI version using public APIs. Set `GITHUB_TOKEN` when running it from CI or a shared network to avoid public GitHub API rate limits. Paste the Markdown output into launch notes, weekly community updates, or release retrospectives so the 20k-star effort stays measurable.
## 30-day execution checklist
### Week 1: Repository conversion
- [ ] Verify README quick start on clean CPU and GPU environments.
- [ ] Verify all README links and docs links.
- [ ] Add or refresh CONTRIBUTING, PR template, and issue templates.
- [ ] Add a short FAQ for install/deployment failures.
- [ ] Confirm GitHub repo description and topics mention ASR, speech-recognition, streaming, diarization, OpenAI-compatible API, MCP, and vLLM.
### Week 2: Deployment proof
- [ ] Publish API server curl examples.
- [ ] Publish WebSocket streaming examples.
- [ ] Validate Docker CPU and GPU images.
- [ ] Validate vLLM guide on one GPU server.
- [ ] Record concise terminal outputs for release notes.
### Week 3: Launch package
- [ ] Create a GitHub release focused on one sharp value proposition.
- [ ] Update PyPI release description if needed.
- [ ] Update Hugging Face organization/model cards.
- [ ] Update ModelScope model cards and demos.
- [ ] Update homepage hero and docs entry points.
### Week 4: Distribution and feedback
- [ ] Share the release in relevant developer communities.
- [ ] Pin a GitHub discussion for the release.
- [ ] Triage all new issues within 48 hours.
- [ ] Convert top 3 support questions into docs.
- [ ] Track stars, PyPI downloads, issue volume, and docs traffic.
## Release-note template
````markdown
## Why this release matters
FunASR now lets you ...
## Try it in 60 seconds
```bash
pip install -U funasr
funasr-server --device cuda
```
## Highlights
- ...
- ...
- ...
## Verified environments
- GPU:
- CPU:
- Docker:
## Links
- Quick start:
- Deployment guide:
- Benchmark:
- Discussion:
````
## 中文摘要
目标不是单纯“求 Star”，而是让更多用户能更快完成安装、转写、部署和分享。当前增长目标按 FunASR / Fun-ASR / SenseVoice / FunClip 四仓生态统一跟踪：以 2026-06-27 的 31,224 combined stars 为基线，到 2026-09-30 累计新增 20,000 stars。最有效的抓手是：首屏转化、可复现 benchmark、OpenAI 兼容 API/流式/vLLM/Docker 部署闭环、HF/ModelScope/官网同步更新、高星生态集成，以及高效的 issue/PR 社区运营。
docs/community_projects.md
+40
View File
@@ -0,0 +1,40 @@
# FunASR Community Projects
This page collects maintained community integrations and experiments around FunASR models. These projects are useful for users exploring deployment paths beyond the official Python runtime, but they are not official FunASR implementations unless explicitly noted.
## VAD runtimes
| Project | What it provides | Notes |
|---|---|---|
| [vad-burn](https://github.com/di-osc/vad-burn) | FSMN VAD inference in pure Rust with Python bindings. It supports offline VAD, streaming VAD, CPU-only inference, and automatic download of the default `iic/speech_fsmn_vad_zh-cn-16k-common-pytorch` model from ModelScope. | Community project from [#3106](https://github.com/modelscope/FunASR/issues/3106). Useful for Rust services, CLI tools, desktop apps, or Python pipelines that need FSMN VAD without depending on the original Python runtime. |
`vad-burn` reports the following benchmark on `assets/vad_example.wav` (16 kHz mono PCM, 70.47 seconds) on a MacBook Pro M1 with the Burn Flex CPU backend:
| Mode | Avg time | RTF | Speedup |
|---|---:|---:|---:|
| FSMN VAD offline | 73.631 ms | 0.001045 | 957.08x |
| FSMN VAD streaming, 600 ms chunks | 198.425 ms | 0.002816 | 355.15x |
Minimal Python usage from the project:
```python
from vad_burn import FsmnVadModel, VadOptions
vad = FsmnVadModel.from_modelscope()
segments = vad.detect(samples, 16000, VadOptions())
stream = vad.new_stream(VadOptions())
for chunk in chunks:
segments = stream.push(chunk, 16000)
final_segments = stream.finish()
```
## Add your project
If you maintain a FunASR integration, open a [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md) with:
- repository link and maintenance status
- supported FunASR model or runtime path
- install and minimal usage instructions
- benchmark or validation details when available
- a note about whether the project is official, community-maintained, or experimental
docs/conf.py
+67
View File
@@ -0,0 +1,67 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = "FunASR"
copyright = "2022, Speech Lab, Alibaba Group"
author = "Speech Lab, Alibaba Group"
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"nbsphinx",
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.viewcode",
"sphinx.ext.mathjax",
"sphinx.ext.todo",
# "sphinxarg.ext",
"sphinx_markdown_tables",
"recommonmark",
"sphinx_rtd_theme",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
source_suffix = [".rst", ".md"]
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "sphinx_rtd_theme"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
docs/deployment_matrix.md
+111
View File
@@ -0,0 +1,111 @@
# FunASR Deployment Matrix
Use this page to choose the shortest deployment path for a product, demo, benchmark, or internal workflow. Start with the smallest surface that satisfies the job, then move to heavier runtimes only when throughput, latency, or integration requirements demand it.
## Quick decision table
| Path | Best for | Start here | Operational notes |
|---|---|---|---|
| Colab notebook | Browser smoke tests, first evaluation, shareable demos | [Colab quickstart](../examples/colab/) | No local setup; first run downloads model files, GPU runtime is faster. |
| Python API | Notebooks, offline jobs, first model evaluation | [README quick start](../README.md#quick-start) | Lowest ceremony; caller owns batching, retries, and files. |
| OpenAI-compatible API | Private speech API, agents, Dify/LangChain/AutoGen-style clients | [OpenAI API example](../examples/openai_api/) | Easiest integration for apps that already support OpenAI audio APIs. |
| Xinference | Teams that already operate Xinference model serving | [Xinference repository](https://github.com/xorbitsai/inference) | Use a Xinference version containing [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140) so Fun-ASR-Nano uses packaged `funasr~=1.3.0` instead of the old pinned git commit. |
| Docker Compose API | Reproducible local smoke test or small internal service | [OpenAI API Docker docs](../examples/openai_api/#docker-deployment) | CPU by default; adapt the image before using CUDA in containers. |
| Kubernetes API | Internal speech API for cluster services | [Kubernetes template](../examples/openai_api/kubernetes/) | Starts as private `ClusterIP`; add auth, TLS, network policy, and GPU scheduling before broader exposure. |
| Runtime WebSocket service | Live captions, meetings, call-center streams | [Runtime service docs](../runtime/readme.md) | Use when partial results, endpointing, or long-lived audio streams matter. |
| ONNX/C++ runtime | High-concurrency CPU services or embedded realtime ASR | [ONNX runtime docs](../runtime/onnxruntime/readme.md) | Keep this path when latency/concurrency is already proven; add text post-processing for fixed business terms before moving to GPU LLMs. |
| vLLM acceleration | Higher-throughput LLM-based ASR with Fun-ASR-Nano | [vLLM guide](./vllm_guide.md) | Use for LLM decoder throughput; does not apply to non-autoregressive Paraformer. |
| MCP server | Claude/Cursor/desktop agent speech tools | [MCP example](../examples/mcp_server/) | Good when the ASR result should be exposed as a local tool. |
| Subtitle generator | SRT/VTT from long audio or video | [Subtitle example](../examples/subtitle/) | Use verbose segments and speaker labels when readability matters. |
| Batch ASR script | Archives, meetings, datasets, repeated offline runs | [Batch example](../examples/batch_asr_improved.py) | Add queueing, manifests, and retry logs for production use. |
| Triton runtime | Specialized high-performance serving | [Triton runtime docs](../runtime/triton_gpu/README.md) | Heavier setup; choose when your team already operates Triton/GPU serving. |
## Common choices
### I want to try FunASR in five minutes
Use the [Colab quickstart](../examples/colab/) when you want a browser-only smoke test, or use the Python API from the README for local work. It is the shortest route for validating installation, model download, device selection, and basic output shape. If you are unsure which model to start with, use the [model selection guide](./model_selection.md).
### I want a local replacement for cloud transcription
Use the OpenAI-compatible API. It exposes `/v1/audio/transcriptions`, `/v1/models`, `/health`, and Swagger docs. Start with `sensevoice`, run `examples/openai_api/smoke_test.sh` or `examples/openai_api/smoke_test.py`, then connect existing SDK or HTTP clients using [client recipes](../examples/openai_api/CLIENTS.md) and [JavaScript/TypeScript recipes](../examples/openai_api/JAVASCRIPT.md). For browser upload or microphone demos, use the [Gradio browser demo](../examples/openai_api/GRADIO.md). For Dify, n8n, HTTP nodes, or webhook workers, follow the [workflow recipes](../examples/openai_api/WORKFLOWS.md). For API gateways, developer portals, and schema-driven imports, use the [OpenAPI spec](../examples/openai_api/OPENAPI.md). Before sharing the service, review the [security and gateway guide](../examples/openai_api/SECURITY.md).
### I already run Xinference
Use Xinference when your stack already standardizes on its model registry,
virtualenv isolation, and serving lifecycle. Make sure your Xinference build
includes [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140);
that update moved the Fun-ASR-Nano model specs from an old FunASR git SHA to the
packaged `funasr~=1.3.0` dependency line. For first-time FunASR evaluation or
agent-oriented OpenAI-compatible transcription, start with the native FunASR
OpenAI API example above instead.
### I want a repeatable container demo
Use `examples/openai_api/docker-compose.yml` for a CPU-mode smoke test:
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
Keep CPU mode until you have a CUDA-capable PyTorch/FunASR image. After that, set `FUNASR_DEVICE=cuda` and verify with the same smoke test. Use `python examples/openai_api/smoke_test.py --base-url http://localhost:8000` on systems without bash/curl.
### I want an internal Kubernetes service
Use the [Kubernetes template](../examples/openai_api/kubernetes/) for a private `ClusterIP` OpenAI-compatible API with persistent model cache, `/health` probes, and a port-forward smoke-test path. Keep the CPU default until you have a CUDA-capable image and cluster GPU scheduling in place.
### I need streaming or live captioning
Use the runtime WebSocket service. Validate chunk size, VAD, endpointing, punctuation, speaker diarization, reconnect behavior, and client backpressure with real audio before production rollout.
### I already have high-concurrency ONNX and need hotwords
Do not move an entire working CPU ONNX/C++ realtime stack to a GPU LLM only
because the current path lacks hotwords. First decide what "hotword" means in
your product:
- For fixed company names, product names, or common misrecognitions, keep the
ONNX path and add deterministic text post-processing on final results. This
preserves proven CPU concurrency and is easier to audit.
- For true decoder-time biasing, pronunciation ambiguity, or multilingual
accuracy gaps, test a GPU path such as Fun-ASR-Nano or Qwen3-ASR, but size it
against your target first-token latency, final latency, audio chunk length,
VAD policy, and simultaneous speaking sessions.
- If both are needed, use the GPU model only where it wins on quality or
language coverage, and keep ONNX for the high-volume traffic that already
meets latency and accuracy requirements.
When opening a deployment issue, include your current ONNX concurrency, CPU
cores, target terms, acceptable end-to-end latency, GPU model, model command,
and whether hotwords are post-recognition corrections or decoder-time biasing.
### I need more LLM-based ASR throughput
Use the vLLM path for Fun-ASR-Nano. Benchmark with your own audio distribution and watch GPU memory, tensor parallel size, first-token latency, and warmup time.
### I want to run Fun-ASR-Nano on Ascend NPU
Fun-ASR-Nano's LLM-based path is documented and validated for CUDA/vLLM, standard PyTorch CPU/GPU runs, and CPU/edge GGUF runtimes. Ascend NPU (`torch_npu`) is still not an official production runtime for this model. Do not assume that a SenseVoice or Paraformer NPU deployment means Fun-ASR-Nano will also work, because Nano also exercises the Qwen decoder, `inputs_embeds`, and backend-specific autocast/operator paths.
A community smoke test in [#3034](https://github.com/modelscope/FunASR/issues/3034) confirmed that the PyTorch `AutoModel(..., device="npu:*")` path can now enter the NPU backend after the autocast device fix, and produced the expected transcript on a 310P3 with `torch_npu 2.5.1` / CANN 8.5.1. That run was much slower than CPU (`rtf_avg` about 124 on NPU vs about 1.9 on CPU), so treat it as compatibility evidence, not a performance recommendation. The same report showed `AutoModelVLLM` on `vllm_ascend 0.9.2rc1` failing inside Qwen3 rotary embedding / `TransData` operator support; debug that path as a vLLM-Ascend runtime/operator issue and capture logs with `ASCEND_LAUNCH_BLOCKING=1`.
If you are adapting this backend, keep the first PR narrow: start with `torch.bfloat16`, capture `torch` / `torch_npu` / CANN / driver / NPU model, separate PyTorch `AutoModel` from `AutoModelVLLM`, and attach the minimal command plus full stack trace.
## Readiness checklist
- Pick a model alias and pin it in deployment notes.
- Record FunASR version, model version, device, CUDA/PyTorch version, Docker image tag, and command line.
- Run a short public smoke sample and at least one realistic private sample.
- Log audio duration, model, device, latency, response format, and error type for every request.
- Add upload-size limits, authentication, TLS, and rate limits before exposing an API outside a trusted network; use the [security and gateway guide](../examples/openai_api/SECURITY.md) to plan the boundary.
- For hotword or correction requirements, record whether the change is
deterministic post-processing or decoder-time biasing, then benchmark quality
and latency before replacing an existing runtime.
- For streaming, test silence, noise, overlapping speakers, long sessions, reconnects, and slow clients.
- For benchmark claims, include input duration, hardware, batch size, model, runtime path, and whether model download/warmup time is excluded.
## When to open an issue
Use [Deployment Help](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) for runtime, Docker, vLLM, Triton, Android, browser, or agent integration problems. Include your deployment path, exact command/config, logs, model, device, and audio characteristics.
docs/deployment_matrix_ja.md
+53
View File
@@ -0,0 +1,53 @@
# FunASR デプロイ選択マトリクス
プロダクト、デモ、ベンチマーク、社内ワークフローに合わせて最短のデプロイ経路を選ぶためのガイドです。まずは要件を満たす最小構成から始め、throughput、latency、integration 要件が明確になったら重い runtime に移行してください。
## クイック判断表
| Path | 向いている用途 | 最初に読むもの | 運用メモ |
|---|---|---|---|
| Colab notebook | ブラウザ smoke test、初回評価、共有 demo | [Colab クイックスタート](../examples/colab/README_ja.md) | ローカル環境不要。初回はモデルをダウンロードし、GPU runtime の方が高速です。 |
| Python API | Notebook、offline job、最初の model evaluation | [README quick start](../README_ja.md#クイックスタート) | 最小構成。batching、retry、file 管理は呼び出し側で扱います。 |
| OpenAI 互換 API | Private speech API、Agent、Dify/LangChain/AutoGen style clients | [OpenAI API example](../examples/openai_api/README_ja.md) | OpenAI audio API に対応した既存 app に最も接続しやすい経路です。 |
| Docker Compose API | 再現可能な local smoke test、小さな internal service | [OpenAI API Docker docs](../examples/openai_api/README_ja.md#docker-デプロイ) | デフォルトは CPU。CUDA を使う前に CUDA-capable image へ調整してください。 |
| Kubernetes API | Cluster service 向け internal speech API | [Kubernetes template](../examples/openai_api/kubernetes/) | private `ClusterIP` から開始。公開範囲を広げる前に auth、TLS、network policy、GPU scheduling を追加します。 |
| Runtime WebSocket service | Live captions、meeting、call-center stream | [Runtime service docs](../runtime/readme.md) | partial result、endpointing、long-lived audio stream が重要な場合に使います。 |
| vLLM acceleration | Fun-ASR-Nano の LLM-based ASR throughput 向上 | [vLLM guide](./vllm_guide.md) | LLM decoder throughput 向け。non-autoregressive Paraformer には適用しません。 |
| MCP server | Claude/Cursor/desktop agent の speech tool | [MCP example](../examples/mcp_server/) | ASR 結果を local tool として Agent に渡したい場合に便利です。 |
| Subtitle generator | 長時間 audio/video から SRT/VTT 作成 | [Subtitle example](../examples/subtitle/) | readability が重要な場合は verbose segment と speaker label を使います。 |
| Batch ASR script | Archive、meeting、dataset、繰り返し offline run | [Batch example](../examples/batch_asr_improved.py) | production では queue、manifest、retry log を追加してください。 |
## よくある選択
### 5分で FunASR を試したい
ブラウザだけで試すなら [Colab クイックスタート](../examples/colab/README_ja.md) を使います。ローカルで作業する場合は README の Python API から始めます。どのモデルを使うか迷う場合は [モデル選択ガイド](./model_selection_ja.md) を参照してください。
### Cloud transcription の local replacement が欲しい
OpenAI 互換 API を使います。`/v1/audio/transcriptions`、`/v1/models`、`/health`、Swagger docs を提供します。まず `sensevoice` で smoke test を実行し、既存 SDK や HTTP client を [OpenAI API example](../examples/openai_api/README_ja.md) に合わせて接続してください。
### 再現可能な container demo が欲しい
`examples/openai_api/docker-compose.yml` を CPU mode の smoke test として使います。
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
CUDA を使う場合は CUDA-capable PyTorch/FunASR image を作成してから `FUNASR_DEVICE=cuda` に変更し、同じ smoke test で確認します。
### Streaming または live captioning が必要
Runtime WebSocket service を使います。本番投入前に chunk size、VAD、endpointing、punctuation、speaker diarization、reconnect、client backpressure を実音声で検証してください。
## Readiness checklist
- model alias を決め、deployment note に固定します。
- FunASR version、model version、device、CUDA/PyTorch version、Docker image tag、command line を記録します。
- public smoke sample と realistic private sample を少なくとも 1 つずつ実行します。
- request ごとに audio duration、model、device、latency、response format、error type をログ化します。
- trusted network の外へ API を出す前に upload-size limit、authentication、TLS、rate limit を入れます。[Security guide](../examples/openai_api/SECURITY.md) も確認してください。
- 詰まったら deployment path、command/config、logs、model、device、audio characteristics を添えて [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) を開いてください。
docs/deployment_matrix_ko.md
+53
View File
@@ -0,0 +1,53 @@
# FunASR 배포 선택 매트릭스
제품, 데모, 벤치마크, 내부 워크플로에 맞는 가장 짧은 배포 경로를 고르기 위한 가이드입니다. 먼저 요구를 만족하는 최소 구성에서 시작하고, throughput, latency, integration 요구가 명확해질 때 더 무거운 runtime으로 이동하세요.
## 빠른 결정 표
| Path | 적합한 용도 | 시작 문서 | 운영 메모 |
|---|---|---|---|
| Colab notebook | 브라우저 smoke test, 첫 평가, 공유 가능한 demo | [Colab 빠른 시작](../examples/colab/README_ko.md) | 로컬 환경이 필요 없습니다. 첫 실행은 모델을 다운로드하며 GPU runtime이 더 빠릅니다. |
| Python API | Notebook, offline job, 첫 model evaluation | [README quick start](../README_ko.md#빠른-시작) | 가장 단순한 경로입니다. batching, retry, file 관리는 호출 측에서 담당합니다. |
| OpenAI 호환 API | Private speech API, Agent, Dify/LangChain/AutoGen style clients | [OpenAI API example](../examples/openai_api/README_ko.md) | OpenAI audio API를 이미 지원하는 앱에 가장 쉽게 연결됩니다. |
| Docker Compose API | 재현 가능한 local smoke test, 작은 internal service | [OpenAI API Docker docs](../examples/openai_api/README_ko.md#docker-배포) | 기본은 CPU입니다. CUDA를 쓰기 전에 CUDA-capable image로 조정하세요. |
| Kubernetes API | Cluster service용 internal speech API | [Kubernetes template](../examples/openai_api/kubernetes/) | private `ClusterIP`부터 시작합니다. 범위를 넓히기 전에 auth, TLS, network policy, GPU scheduling을 추가하세요. |
| Runtime WebSocket service | Live captions, meeting, call-center stream | [Runtime service docs](../runtime/readme.md) | partial result, endpointing, long-lived audio stream이 중요할 때 사용합니다. |
| vLLM acceleration | Fun-ASR-Nano의 LLM-based ASR throughput 향상 | [vLLM guide](./vllm_guide.md) | LLM decoder throughput용입니다. non-autoregressive Paraformer에는 적용되지 않습니다. |
| MCP server | Claude/Cursor/desktop agent speech tool | [MCP example](../examples/mcp_server/) | ASR 결과를 local tool로 Agent에 전달할 때 유용합니다. |
| Subtitle generator | 긴 audio/video에서 SRT/VTT 생성 | [Subtitle example](../examples/subtitle/) | readability가 중요하면 verbose segment와 speaker label을 사용합니다. |
| Batch ASR script | Archive, meeting, dataset, 반복 offline run | [Batch example](../examples/batch_asr_improved.py) | production에서는 queue, manifest, retry log를 추가하세요. |
## 자주 쓰는 선택
### 5분 안에 FunASR을 시험하고 싶다
브라우저만으로 확인하려면 [Colab 빠른 시작](../examples/colab/README_ko.md)을 사용하세요. 로컬에서 작업하려면 README의 Python API부터 시작합니다. 어떤 모델을 고를지 고민된다면 [모델 선택 가이드](./model_selection_ko.md)를 참고하세요.
### Cloud transcription의 local replacement가 필요하다
OpenAI 호환 API를 사용하세요. `/v1/audio/transcriptions`, `/v1/models`, `/health`, Swagger docs를 제공합니다. 먼저 `sensevoice`로 smoke test를 실행하고 기존 SDK나 HTTP client를 [OpenAI API example](../examples/openai_api/README_ko.md)에 맞춰 연결하세요.
### 재현 가능한 container demo가 필요하다
`examples/openai_api/docker-compose.yml`을 CPU mode smoke test로 사용합니다.
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
CUDA를 사용하려면 CUDA-capable PyTorch/FunASR image를 만든 뒤 `FUNASR_DEVICE=cuda`로 바꾸고 같은 smoke test로 확인하세요.
### Streaming 또는 live captioning이 필요하다
Runtime WebSocket service를 사용하세요. production 전에 chunk size, VAD, endpointing, punctuation, speaker diarization, reconnect, client backpressure를 실제 오디오로 검증하세요.
## Readiness checklist
- model alias를 정하고 deployment note에 고정합니다.
- FunASR version, model version, device, CUDA/PyTorch version, Docker image tag, command line을 기록합니다.
- public smoke sample과 realistic private sample을 최소 1개씩 실행합니다.
- request마다 audio duration, model, device, latency, response format, error type을 로깅합니다.
- trusted network 밖으로 API를 노출하기 전에 upload-size limit, authentication, TLS, rate limit을 넣습니다. [Security guide](../examples/openai_api/SECURITY.md)도 확인하세요.
- 막히면 deployment path, command/config, logs, model, device, audio characteristics를 포함해 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)를 열어 주세요.
docs/deployment_matrix_zh.md
+92
View File
@@ -0,0 +1,92 @@
# FunASR 部署选型表
这个页面帮助你为产品、demo、benchmark 或内部工作流选择最短部署路径。先选择能满足目标的最小方案，只有在吞吐、延迟或集成方式有明确要求时，再切换到更重的运行时。
## 快速决策表
| 路径 | 适合场景 | 从这里开始 | 运维提示 |
|---|---|---|---|
| Colab Notebook | 浏览器 smoke test、首次评估、可分享 demo | [Colab 快速体验](../examples/colab/README_zh.md) | 不需要本地环境；首次运行会下载模型，GPU runtime 更快。 |
| Python API | Notebook、离线任务、首次模型评测 | [README 快速开始](../README_zh.md#快速开始) | 最简单；调用方自己负责批处理、重试和文件管理。 |
| OpenAI 兼容 API | 私有语音 API、Agent、Dify/LangChain/AutoGen 风格客户端 | [OpenAI API 示例](../examples/openai_api/README_zh.md) | 已支持 OpenAI audio API 的应用最容易接入。 |
| Xinference | 已经使用 Xinference 统一管理模型服务的团队 | [Xinference 仓库](https://github.com/xorbitsai/inference) | 使用包含 [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140) 的版本或 commit，确保 Fun-ASR-Nano 使用打包发布的 `funasr~=1.3.0`，而不是旧的 git commit pin。 |
| Docker Compose API | 可复现本地 smoke test 或小型内部服务 | [OpenAI API Docker 文档](../examples/openai_api/README_zh.md) | 默认 CPU；容器里使用 CUDA 前需要先适配 CUDA-capable 镜像。 |
| Kubernetes API | 集群内私有语音 API | [Kubernetes 模板](../examples/openai_api/kubernetes/README_zh.md) | 默认私有 `ClusterIP`；对外开放前补齐鉴权、TLS、网络策略和 GPU 调度。 |
| Runtime WebSocket 服务 | 实时字幕、会议、客服流式音频 | [Runtime 服务文档](../runtime/readme_cn.md) | 需要中间结果、断句或长连接音频流时选择。 |
| ONNX/C++ Runtime | 高并发 CPU 服务或嵌入式实时 ASR | [ONNX Runtime 文档](../runtime/onnxruntime/readme.md) | 如果延迟和并发已经验证，不要轻易替换；固定业务词优先做文本后处理。 |
| vLLM 加速 | Fun-ASR-Nano 等 LLM-based ASR 高吞吐 | [vLLM 指南](./vllm_guide.md) | 适合 LLM 解码吞吐；不适用于非自回归 Paraformer。 |
| MCP 服务 | Claude/Cursor/桌面 Agent 语音工具 | [MCP 示例](../examples/mcp_server/) | 适合把 ASR 结果暴露成一个本地工具。 |
| 字幕生成 | 从长音频或视频生成 SRT/VTT | [字幕示例](../examples/subtitle/) | 需要可读性时使用 verbose segments 和说话人标签。 |
| 批处理脚本 | 录音归档、会议纪要、数据集处理 | [批处理示例](../examples/batch_asr_improved.py) | 生产使用时建议增加队列、manifest 和重试日志。 |
| Triton Runtime | 专门的高性能推理服务 | [Triton 文档](../runtime/triton_gpu/README.md) | 配置更重；适合已经在运维 Triton/GPU serving 的团队。 |
## 常见选择
### 我想五分钟内试跑 FunASR
如果只想在浏览器里 smoke test，可以先用 [Colab 快速体验](../examples/colab/README_zh.md)；本地工作再使用 README 里的 Python API。它是验证安装、模型下载、设备选择和基础输出格式的最短路径。如果还不确定先用哪个模型，请看 [模型选择指南](./model_selection_zh.md)。
### 我想替代云端转写服务
使用 OpenAI 兼容 API。它提供 `/v1/audio/transcriptions`、`/v1/models`、`/health` 和 Swagger docs。先用 `sensevoice` 跑通 `examples/openai_api/smoke_test.sh` 或 `examples/openai_api/smoke_test.py`，再根据 [客户端配方](../examples/openai_api/CLIENTS.md) 和 [JavaScript/TypeScript 配方](../examples/openai_api/JAVASCRIPT_zh.md) 接入 SDK 或 HTTP 客户端。浏览器上传或麦克风 demo 可使用 [Gradio 浏览器 Demo](../examples/openai_api/GRADIO_zh.md)。Dify、n8n、HTTP 节点或 webhook worker 可参考 [工作流配方](../examples/openai_api/WORKFLOWS_zh.md)。API 网关、开发者门户或按 schema 导入时可使用 [OpenAPI 规范](../examples/openai_api/OPENAPI_zh.md)。跨团队共享服务前，请先阅读 [安全与网关指南](../examples/openai_api/SECURITY_zh.md)。
### 我已经在使用 Xinference
如果你的系统已经用 Xinference 管理模型注册、virtualenv 隔离和服务生命周期，可以选择 Xinference 路径。请确认使用的 Xinference 版本或 commit 包含 [xorbitsai/inference#5140](https://github.com/xorbitsai/inference/pull/5140)；该更新把 Fun-ASR-Nano model spec 从旧的 FunASR git SHA 改为打包发布的 `funasr~=1.3.0` 依赖。首次评估 FunASR，或需要面向 Agent 的 OpenAI 兼容转写接口时，仍建议先从上面的 FunASR 原生 OpenAI API 示例开始。
### 我想要可复现的容器 demo
使用 `examples/openai_api/docker-compose.yml` 跑 CPU smoke test：
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
在没有 CUDA-capable PyTorch/FunASR 镜像前保持 CPU 模式。准备好 CUDA 镜像后，再设置 `FUNASR_DEVICE=cuda` 并用同一个 smoke test 验证。没有 bash/curl 时可运行 `python examples/openai_api/smoke_test.py --base-url http://localhost:8000`。
### 我想部署集群内服务
使用 [Kubernetes 模板](../examples/openai_api/kubernetes/README_zh.md) 部署私有 `ClusterIP` OpenAI 兼容 API，包含持久化模型缓存、`/health` probes 和 port-forward smoke test 路径。在没有 CUDA-capable 镜像和集群 GPU 调度前，请保持默认 CPU 模式。
### 我需要流式识别或实时字幕
使用 Runtime WebSocket 服务。上线前请用真实音频验证 chunk size、VAD、断句、标点、说话人分离、重连行为和客户端背压。
### 我已有高并发 ONNX，但需要热词
不要只因为当前 CPU ONNX/C++ 实时链路缺少热词，就把整条已经跑稳的链路迁到 GPU 大模型。先确认产品里的“热词”到底是哪一种：
- 如果是公司名、产品名、专有名词或固定误识别纠错，优先保留 ONNX 链路，在最终文本上做确定性后处理。这样能保留已经验证过的 CPU 并发能力，也更容易审计。
- 如果必须在解码阶段影响识别，或者确实存在多语言/语义理解上的质量缺口，再测试 Fun-ASR-Nano、Qwen3-ASR 等 GPU 路径；但要按首包延迟、尾包延迟、音频切片长度、VAD 策略和同时说话路数重新压测。
- 如果两类需求都有，只在 GPU 模型质量或语言覆盖明显更好的流量上使用 GPU，其他高吞吐稳定流量继续走 ONNX。
开部署 issue 时，请附上当前 ONNX 并发、CPU 核数、目标热词、可接受端到端延迟、GPU 型号、模型启动命令，以及热词是“识别后纠错”还是“解码阶段偏置”。
### 我需要更高的 LLM-based ASR 吞吐
Fun-ASR-Nano 走 vLLM 路径。请用自己的音频分布做 benchmark，并关注 GPU 显存、tensor parallel size、首 token 延迟和 warmup 时间。
### 我想在昇腾 NPU 上跑 Fun-ASR-Nano
Fun-ASR-Nano 的 LLM-based 路径目前主要按 CUDA/vLLM、标准 PyTorch CPU/GPU，以及 CPU/边缘 GGUF runtime 记录和验证；Ascend NPU（`torch_npu`）仍不是这个模型的官方生产运行时。不要因为 SenseVoice 或 Paraformer 能在 NPU 上跑，就默认 Fun-ASR-Nano 也能直接跑通，因为 Nano 还会经过 Qwen 解码器、`inputs_embeds`，以及后端相关的 autocast / 算子路径。
[#3034](https://github.com/modelscope/FunASR/issues/3034) 里的社区复测表明：修复 autocast device 之后，PyTorch `AutoModel(..., device="npu:*")` 路径已经能进入 NPU 后端，并在 310P3、`torch_npu 2.5.1`、CANN 8.5.1 环境里产出正确文本。但该 smoke run 明显慢于 CPU（NPU `rtf_avg` 约 124，CPU 约 1.9），所以这只能作为兼容性证据，不是性能推荐。同一报告里，`AutoModelVLLM` + `vllm_ascend 0.9.2rc1` 仍在 Qwen3 rotary embedding / `TransData` 算子支持处失败；这条路径应按 vLLM-Ascend runtime / 算子兼容问题继续排查，并建议带 `ASCEND_LAUNCH_BLOCKING=1` 复现日志。
如果你正在适配这个后端，请保持第一版范围很小：先从 `torch.bfloat16` 开始，记录 `torch` / `torch_npu` / CANN / 驱动 / NPU 型号，把 PyTorch `AutoModel` 和 `AutoModelVLLM` 分开验证，并在 PR 或 deployment issue 里附上最小命令和完整 stack trace。
## 上线检查清单
- 选择模型 alias，并写入部署说明。
- 记录 FunASR 版本、模型版本、设备、CUDA/PyTorch 版本、Docker 镜像 tag 和启动命令。
- 跑一个公开短音频 smoke sample，再跑至少一个真实私有样本。
- 每次请求记录音频时长、模型、设备、延迟、响应格式和错误类型。
- API 暴露到可信网络外之前，增加上传大小限制、鉴权、TLS 和限流；可用 [安全与网关指南](../examples/openai_api/SECURITY_zh.md) 规划边界。
- 热词或纠错需求要先记录它是确定性后处理，还是解码阶段偏置；替换已有 runtime 前同时 benchmark 质量和延迟。
- 流式场景需要测试静音、噪声、多人重叠、长连接、重连和慢客户端。
- 发布 benchmark 结论时，说明输入时长、硬件、batch size、模型、运行路径，以及是否排除模型下载和 warmup 时间。
## 什么时候开 issue
Runtime、Docker、vLLM、Triton、Android、浏览器或 Agent 集成问题，请使用 [Deployment Help](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)。请附上部署路径、完整命令/config、日志、模型、设备和音频特征。
docs/images/China_Telecom.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

docs/images/DeepScience.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.9 KiB

docs/images/RapidAI.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

docs/images/XVERSE.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

docs/images/aihealthx.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/images/alibaba.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

docs/images/damo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

docs/images/dingding.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

docs/images/funasr_logo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 624 KiB

docs/images/funasr_overview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

docs/images/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 653 KiB

docs/images/nwpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

docs/images/wechat.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 187 KiB

docs/index.rst
+133
View File
@@ -0,0 +1,133 @@
.. Funasr documentation master file, created by
sphinx-quickstart on Tues Dec 6 19:05:00 2022.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
FunASR: A Fundamental End-to-End Speech Recognition Toolkit
============================================================
.. image:: ./images/funasr_logo.jpg
FunASR hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on `ModelScope <https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition>`_, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！
Overview
============================================================
.. image:: ./images/funasr_overview.png
.. toctree::
:maxdepth: 1
:caption: Installation
./installation/installation.md
./installation/docker.md
.. toctree::
:maxdepth: 5
:caption: Quick Start
./funasr/quick_start.md
.. toctree::
:maxdepth: 1
:caption: Academic Egs
./academic_recipe/asr_recipe.md
./academic_recipe/punc_recipe.md
./academic_recipe/vad_recipe.md
./academic_recipe/sv_recipe.md
./academic_recipe/sd_recipe.md
.. toctree::
:maxdepth: 1
:caption: ModelScope Egs
./modelscope_pipeline/quick_start.md
./egs_modelscope/asr/TEMPLATE/README.md
./egs_modelscope/vad/TEMPLATE/README.md
./egs_modelscope/punctuation/TEMPLATE/README.md
./egs_modelscope/tp/TEMPLATE/README.md
./modelscope_pipeline/sv_pipeline.md
./modelscope_pipeline/sd_pipeline.md
./modelscope_pipeline/itn_pipeline.md
.. toctree::
:maxdepth: 1
:caption: Huggingface Egs
Undo
.. toctree::
:maxdepth: 1
:caption: Model Zoo
./model_zoo/modelscope_models.md
./model_zoo/huggingface_models.md
.. toctree::
:maxdepth: 1
:caption: Runtime and Service
./deployment_matrix.md
./deployment_matrix_zh.md
./runtime/readme.md
./runtime/docs/SDK_tutorial_online.md
./runtime/docs/SDK_tutorial.md
./runtime/html5/readme.md
.. toctree::
:maxdepth: 1
:caption: Benchmark and Leaderboard
./benchmark/rtf_reproducibility.md
./benchmark/realtime_ws_benchmark.md
./benchmark/benchmark_onnx.md
./benchmark/benchmark_onnx_cpp.md
./benchmark/benchmark_libtorch.md
./benchmark/benchmark_pipeline_cer.md
.. toctree::
:maxdepth: 1
:caption: Funasr Library
./reference/build_task.md
.. toctree::
:maxdepth: 1
:caption: Papers
./reference/papers.md
.. toctree::
:maxdepth: 1
:caption: Application
./model_selection.md
./model_selection_zh.md
./migration_from_whisper.md
./migration_from_whisper_zh.md
./cli.md
./use_case_showcase.md
./use_case_showcase_zh.md
./community_projects.md
./reference/application.md
.. toctree::
:maxdepth: 1
:caption: FAQ and Troubleshooting
./reference/FQA.md
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
docs/installation/docker.md
+72
View File
@@ -0,0 +1,72 @@
([简体中文](./docker_zh.md)|English)
# Docker
## Install Docker
### Ubuntu
```shell
curl -fsSL https://test.docker.com -o test-docker.sh
sudo sh test-docker.sh
```
### Debian
```shell
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
### CentOS
```shell
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
```
### MacOS
```shell
brew install --cask --appdir=/Applications docker
```
### Windows
Ref to [docs](https://docs.docker.com/desktop/install/windows-install/)
## Start Docker
```shell
sudo systemctl start docker
```
## Download image
### Image Hub
#### CPU
`registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-cpu-0.4.1`
#### GPU
`registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch1.11.0-tf1.15.5-1.8.1`
### Pull Image
```shell
sudo docker pull <image-name>:<tag>
```
### Check Image
```shell
sudo docker images
```
## Run Docker
```shell
# cpu
sudo docker run -itd --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
# gpu
sudo docker run -itd --gpus all --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
sudo docker exec -it funasr /bin/bash
```
## Stop Docker
```shell
exit
sudo docker ps
sudo docker stop funasr
```
docs/installation/docker_zh.md
+72
View File
@@ -0,0 +1,72 @@
(简体中文|[English](./docker.md))
# Docker
## 安装Docker
### Ubuntu
```shell
curl -fsSL https://test.docker.com -o test-docker.sh
sudo sh test-docker.sh
```
### Debian
```shell
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
### CentOS
```shell
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
```
### MacOS
```shell
brew install --cask --appdir=/Applications docker
```
### Windows
请参考[文档](https://docs.docker.com/desktop/install/windows-install/)
## 启动Docker
```shell
sudo systemctl start docker
```
## 下载Docker镜像
### 镜像仓库
#### CPU
`registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-cpu-0.4.1`
#### GPU
`registry.cn-hangzhou.aliyuncs.com/modelscope-repo/modelscope:ubuntu20.04-py38-torch1.11.0-tf1.15.5-1.8.1`
### 拉取镜像
```shell
sudo docker pull <image-name>:<tag>
```
### 查看镜像
```shell
sudo docker images
```
## 运行Docker
```shell
# cpu
sudo docker run -itd --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
# gpu
sudo docker run -itd --gpus all --name funasr -v <local_dir:dir_in_docker> <image-name>:<tag> /bin/bash
sudo docker exec -it funasr /bin/bash
```
## 停止Docker
```shell
exit
sudo docker ps
sudo docker stop funasr
```
docs/installation/installation.md
Executable
+74
View File
@@ -0,0 +1,74 @@
([简体中文](./installation_zh.md)|English)
<p align="left">
<a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Python->=3.8,<=3.13-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>
</p>
## Installation
### Install Conda (Optional):
#### Linux
```sh
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Mac
```sh
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
# For M1 chip
# wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX*
source ~/.zashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Windows
Ref to [docs](https://docs.conda.io/en/latest/miniconda.html#windows-installers)
### Install Pytorch (version >= 1.11.0):
```sh
pip3 install torch torchaudio
```
If there exists CUDAs in your environments, you should install the pytorch with the version matching the CUDA. The matching list could be found in [docs](https://pytorch.org/get-started/previous-versions/).
### Install funasr
#### Install from pip
```shell
pip3 install -U funasr
# For the users in China, you could install with the command:
# pip3 install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
#### Or install from source code
``` sh
git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip3 install -e ./
# For the users in China, you could install with the command:
# pip3 install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### Install modelscope (Optional)
If you want to use the pretrained models in ModelScope, you should install the modelscope:
```shell
pip3 install -U modelscope
# For the users in China, you could install with the command:
# pip3 install -U modelscope -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### FQA
- For installation on MAC M1 chip, the following error may happen:
- - _cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
```shell
pip uninstall cffi pycparser
ARCHFLAGS="-arch arm64" pip install cffi pycparser --compile --no-cache-dir
```
docs/installation/installation_zh.md
Executable
+75
View File
@@ -0,0 +1,75 @@
(简体中文|[English](./installation.md))
<p align="left">
<a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Python->=3.8,<=3.13-aff.svg"></a>
<a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>
</p>
## 安装
### 安装Conda（可选）：
#### Linux
```sh
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Mac
```sh
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
# For M1 chip
# wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX*
source ~/.zashrc
conda create -n funasr python=3.8
conda activate funasr
```
#### Windows
Ref to [docs](https://docs.conda.io/en/latest/miniconda.html#windows-installers)
### 安装Pytorch（版本 >= 1.11.0）：
```sh
pip3 install torch torchaudio
```
如果您的环境中存在CUDAs，则应安装与CUDA匹配版本的pytorch，匹配列表可在文档中找到（[文档](https://pytorch.org/get-started/previous-versions/)）。
### 安装funasr
#### 从pip安装
```shell
pip3 install -U funasr
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
#### 或者从源代码安装
``` sh
git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip3 install -e ./
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### 安装modelscope（可选）
如果您想要使用ModelScope中的预训练模型，则应安装modelscope:
```shell
pip3 install -U modelscope
# 对于中国大陆用户，可以使用以下命令进行安装：
# pip3 install -U modelscope -i https://mirror.sjtu.edu.cn/pypi/web/simple
```
### 常见问题解答
- 在MAC M1芯片上安装时，可能会出现以下错误：
- - _cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
```shell
pip uninstall cffi pycparser
ARCHFLAGS="-arch arm64" pip install cffi pycparser --compile --no-cache-dir
```
docs/m2met2/Baseline.md
+38
View File
@@ -0,0 +1,38 @@
# Baseline
## Overview
We will release an E2E SA-ASR baseline conducted on [FunASR](https://github.com/modelscope/FunASR) at the time according to the timeline. The model architecture is shown in Figure 3. The SpeakerEncoder is initialized with a pre-trained speaker verification model from ModelScope. This speaker verification model is also be used to extract the speaker embedding in the speaker profile.
![model archietecture](images/sa_asr_arch.png)
## Quick start
To run the baseline, first you need to install FunASR and ModelScope. ([installation](https://github.com/modelscope/FunASR#installation))
There are two startup scripts, `run.sh` for training and evaluating on the old eval and test sets, and `run_m2met_2023_infer.sh` for inference on the new test set of the Multi-Channel Multi-Party Meeting Transcription 2.0 ([M2MeT2.0](https://alibaba-damo-academy.github.io/FunASR/m2met2/index.html)) Challenge.
Before running `run.sh`, you must manually download and unpack the [AliMeeting](http://www.openslr.org/119/) corpus and place it in the `./dataset` directory:
```shell
dataset
|—— Eval_Ali_far
|—— Eval_Ali_near
|—— Test_Ali_far
|—— Test_Ali_near
|—— Train_Ali_far
|—— Train_Ali_near
```
Before running `run_m2met_2023_infer.sh`, you need to place the new test set `Test_2023_Ali_far` (to be released after the challenge starts) in the `./dataset` directory, which contains only raw audios. Then put the given `wav.scp`, `wav_raw.scp`, `segments`, `utt2spk` and `spk2utt` in the `./data/Test_2023_Ali_far` directory.
```shell
data/Test_2023_Ali_far
|—— wav.scp
|—— wav_raw.scp
|—— segments
|—— utt2spk
|—— spk2utt
```
For more details you can see [here](https://github.com/modelscope/FunASR/blob/main/egs/alimeeting/sa-asr/README.md)
## Baseline results
The results of the baseline system are shown in Table 3. The speaker profile adopts the oracle speaker embedding during training. However, due to the lack of oracle speaker label during evaluation, the speaker profile provided by an additional spectral clustering is used. Meanwhile, the results of using the oracle speaker profile on Eval and Test Set are also provided to show the impact of speaker profile accuracy.
| |SI-CER(%) |cpCER(%) |
|:---------------|:------------:|----------:|
|oracle profile |32.72 |42.92 |
|cluster profile|32.73 |49.37 |
docs/m2met2/Challenge_result.md
+14
View File
@@ -0,0 +1,14 @@
# Challenge Result
The following table shows the final results of the competition, where Sub-track1 represents the sub-track under fixed training condition and Sub-track 2 represents the sub-track under the open training condition. All result in this table is cp-CER (%). The rankings in the table are the combined rankings of the two sub-tracks as all teams' submissions met the requirements of the sub-track under fixed training condition.
| Rank &nbsp; &nbsp; | Team Name &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | Sub-track1 &nbsp; &nbsp; | Sub-track2 &nbsp; &nbsp; | paper |
|------|----------------------|------------|------------|------------------------|
| 1 | Ximalaya Speech Team | 11.27 | 11.27 | |
| 2 | 小马达 | 18.64 | 18.64 | |
| 3 | AIzyzx | 22.83 | 22.83 | |
| 4 | AsrSpeeder | / | 23.51 | |
| 5 | zyxlhz | 24.82 | 24.82 | |
| 6 | CMCAI | 26.11 | / | |
| 7 | Volcspeech | 34.21 | 34.21 | |
| 8 | 鉴往知来 | 40.14 | 40.14 | |
| 9 | baseline | 41.55 | 41.55 | |
| 10 | DAICT | 41.64 | | |
docs/m2met2/Contact.md
+4
View File
@@ -0,0 +1,4 @@
# Contact
If you have any questions about M2MeT2.0 challenge, please contact us by
- email: [m2met.alimeeting@gmail.com](mailto:m2met.alimeeting@gmail.com)
docs/m2met2/Dataset.md
+26
View File
@@ -0,0 +1,26 @@
# Datasets
## Overview of training data
In the fixed training condition, the training dataset is restricted to three publicly available corpora, namely, AliMeeting, AISHELL-4, and CN-Celeb. To evaluate the performance of the models trained on these datasets, we will release a new Test set called Test-2023 for scoring and ranking. We will describe the AliMeeting dataset and the Test-2023 set in detail.
## Detail of AliMeeting corpus
AliMeeting contains 118.75 hours of speech data in total. The dataset is divided into 104.75 hours for training (Train), 4 hours for evaluation (Eval) and 10 hours as test set (Test) for scoring and ranking. Specifically, the Train, Eval and Test sets contain 212, 8 and 20 sessions, respectively. Each session consists of a 15 to 30-minute discussion by a group of participants. The total number of participants in Train, Eval and Test sets is 456, 25 and 60, respectively, with balanced gender coverage.
The dataset is collected in 13 meeting venues, which are categorized into three types: small, medium, and large rooms with sizes ranging from 8 m$^{2}$ to 55 m$^{2}$. Different rooms give us a variety of acoustic properties and layouts. The detailed parameters of each meeting venue will be released together with the Train data. The type of wall material of the meeting venues covers cement, glass, etc. Other furnishings in meeting venues include sofa, TV, blackboard, fan, air conditioner, plants, etc. During recording, the participants of the meeting sit around the microphone array which is placed on the table and conduct a natural conversation. The microphone-speaker distance ranges from 0.3 m to 5.0 m. All participants are native Chinese speakers speaking Mandarin without strong accents. During the meeting, various kinds of indoor noise including but not limited to clicking, keyboard, door opening/closing, fan, bubble noise, etc., are made naturally. For both Train and Eval sets, the participants are required to remain in the same position during recording. There is no speaker overlap between the Train and Eval set. An example of the recording venue from the Train set is shown in Fig 1.
![meeting room](images/meeting_room.png)
The number of participants within one meeting session ranges from 2 to 4. To ensure the coverage of different overlap ratios, we select various meeting topics during recording, including medical treatment, education, business, organization management, industrial production and other daily routine meetings. The average speech overlap ratio of Train, Eval and Test sets are 42.27\%, 34.76\% and 42.8\%, respectively. More details of AliMeeting are shown in Table 1. A detailed overlap ratio distribution of meeting sessions with different numbers of speakers in the Train, Eval and Test set is shown in Table 2.
![dataset detail](images/dataset_details.png)
The Test-2023 set consists of 20 sessions that were recorded in an identical acoustic setting to that of the AliMeeting corpus. Each meeting session in the Test-2023 dataset comprises between 2 and 4 participants, thereby sharing a similar configuration with the AliMeeting test set.
We also record the near-field signal of each participant using a headset microphone and ensure that only the participant's own speech is recorded and transcribed. It is worth noting that the far-field audio recorded by the microphone array and the near-field audio recorded by the headset microphone will be synchronized to a common timeline range.
All transcriptions of the speech data are prepared in TextGrid format for each session, which contains the information of the session duration, speaker information (number of speaker, speaker-id, gender, etc.), the total number of segments of each speaker, the timestamp and transcription of each segment, etc.
## Get the data
The three dataset for training mentioned above can be downloaded at [OpenSLR](https://openslr.org/resources.php). The participants can download via the following links. Particularly, in the baseline we provide convenient data preparation scripts for AliMeeting corpus.
- [AliMeeting](https://openslr.org/119/)
- [AISHELL-4](https://openslr.org/111/)
- [CN-Celeb](https://openslr.org/82/)
Now, the new test set is available [here](https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/AliMeeting/openlr/Test_2023_Ali.tar.gz)
docs/m2met2/Introduction.md
+28
View File
@@ -0,0 +1,28 @@
# Introduction
## Call for participation
Automatic speech recognition (ASR) and speaker diarization have made significant strides in recent years, resulting in a surge of speech technology applications across various domains. However, meetings present unique challenges to speech technologies due to their complex acoustic conditions and diverse speaking styles, including overlapping speech, variable numbers of speakers, far-field signals in large conference rooms, and environmental noise and reverberation.
Over the years, several challenges have been organized to advance the development of meeting transcription, including the Rich Transcription evaluation and Computational Hearing in Multisource Environments (CHIME) challenges. The latest iteration of the CHIME challenge has a particular focus on distant automatic speech recognition and developing systems that can generalize across various array topologies and application scenarios. However, while progress has been made in English meeting transcription, language differences remain a significant barrier to achieving comparable results in non-English languages, such as Mandarin. The Multimodal Information Based Speech Processing (MISP) and Multi-Channel Multi-Party Meeting Transcription (M2MeT) challenges have been instrumental in advancing Mandarin meeting transcription. The MISP challenge seeks to address the problem of audio-visual distant multi-microphone signal processing in everyday home environments, while the M2MeT challenge focuses on tackling the speech overlap issue in offline meeting rooms.
The ICASSP2022 M2MeT challenge focuses on meeting scenarios, and it comprises two main tasks: speaker diarization and multi-speaker automatic speech recognition. The former involves identifying who spoke when in the meeting, while the latter aims to transcribe speech from multiple speakers simultaneously, which poses significant technical difficulties due to overlapping speech and acoustic interferences.
Building on the success of the previous M2MeT challenge, we are excited to propose the M2MeT2.0 challenge as an ASRU 2023 challenge special session. In the original M2MeT challenge, the evaluation metric was speaker-independent, which meant that the transcription could be determined, but not the corresponding speaker. To address this limitation and further advance the current multi-talker ASR system towards practicality, the M2MeT2.0 challenge proposes the speaker-attributed ASR task with two sub-tracks: fixed and open training conditions. The speaker-attribute automatic speech recognition (ASR) task aims to tackle the practical and challenging problem of identifying "who spoke what at when". To facilitate reproducible research in this field, we offer a comprehensive overview of the dataset, rules, evaluation metrics, and baseline systems. Furthermore, we will release a carefully curated test set, comprising approximately 10 hours of audio, according to the timeline. The new test set is designed to enable researchers to validate and compare their models' performance and advance the state of the art in this area.
## Timeline(AOE Time)
- $ April~29, 2023: $ Challenge and registration open.
- $ May~11, 2023: $ Baseline release.
- $ May~22, 2023: $ Registration deadline, the due date for participants to join the Challenge.
- $ June~16, 2023: $ Test data release and leaderboard open.
- $ June~20, 2023: $ Final submission deadline and leaderboar close.
- $ June~26, 2023: $ Evaluation result and ranking release.
- $ July~3, 2023: $ Deadline for paper submission.
- $ July~10, 2023: $ Deadline for final paper submission.
- $ December~12\ to\ 16, 2023: $ ASRU Workshop and Challenge Session.
## Guidelines
Interested participants, whether from academia or industry, must register for the challenge by completing the Google form below. The deadline for registration is May 22, 2023.
[M2MeT2.0 Registration](https://docs.google.com/forms/d/e/1FAIpQLSf77T9vAl7Ym-u5g8gXu18SBofoWRaFShBo26Ym0-HDxHW9PQ/viewform?usp=sf_link)
Within three working days, the challenge organizer will send email invitations to eligible teams to participate in the challenge. All qualified teams are required to adhere to the challenge rules, which will be published on the challenge page. Prior to the ranking release time, each participant must submit a system description document detailing their approach and methods. The organizer will select the top ranking submissions to be included in the ASRU2023 Proceedings.
docs/m2met2/Makefile
+20
View File
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/m2met2/Organizers.md
+48
View File
@@ -0,0 +1,48 @@
# Organizers
***Lei Xie, Professor, AISHELL foundation, China***
Email: [lxie@nwpu.edu.cn](mailto:lxie@nwpu.edu.cn)
<img src="images/lxie.jpeg" alt="lxie" width="20%">
***Kong Aik Lee, Senior Scientist at Institute for Infocomm Research, A\*Star, Singapore***
Email: [kongaik.lee@ieee.org](mailto:kongaik.lee@ieee.org)
<img src="images/kong.png" alt="kong" width="20%">
***Zhijie Yan, Principal Engineer at Alibaba, China***
Email: [zhijie.yzj@alibaba-inc.com](mailto:zhijie.yzj@alibaba-inc.com)
<img src="images/zhijie.jpg" alt="zhijie" width="20%">
***Shiliang Zhang, Senior Engineer at Alibaba, China***
Email: [sly.zsl@alibaba-inc.com](mailto:sly.zsl@alibaba-inc.com)
<img src="images/zsl.JPG" alt="zsl" width="20%">
***Yanmin Qian, Professor, Shanghai Jiao Tong University, China***
Email: [yanminqian@sjtu.edu.cn](mailto:yanminqian@sjtu.edu.cn)
<img src="images/qian.jpeg" alt="qian" width="20%">
***Zhuo Chen, Applied Scientist in Microsoft, USA***
Email: [zhuc@microsoft.com](mailto:zhuc@microsoft.com)
<img src="images/chenzhuo.jpg" alt="chenzhuo" width="20%">
***Jian Wu, Applied Scientist in Microsoft, USA***
Email: [wujian@microsoft.com](mailto:wujian@microsoft.com)
<img src="images/wujian.jpg" alt="wujian" width="20%">
***Hui Bu, CEO, AISHELL foundation, China***
Email: [buhui@aishelldata.com](mailto:buhui@aishelldata.com)
<img src="images/buhui.jpeg" alt="buhui" width="20%">
docs/m2met2/Rules.md
+14
View File
@@ -0,0 +1,14 @@
# Rules
All participants should adhere to the following rules to be eligible for the challenge.
- Data augmentation is allowed on the original training dataset, including, but not limited to, adding noise or reverberation, speed perturbation and tone change.
- Participants are permitted to use the Eval set for model training, but it is not allowed to use the Test set for this purpose. Instead, the Test set should only be utilized for parameter tuning and model selection. Any use of the Test-2023 dataset that violates these rules is strictly prohibited, including but not limited to the use of the Test set for fine-tuning or training the model.
- If the cpCER of the two systems on the Test dataset are the same, the system with lower computation complexity will be judged as the superior one.
- If the forced alignment is used to obtain the frame-level classification label, the forced alignment model must be trained on the basis of the data allowed by the corresponding sub-track.
- Shallow fusion is allowed to the end-to-end approaches, e.g., LAS, RNNT and Transformer, but the training data of the shallow fusion language model can only come from the transcripts of the allowed training dataset.
- The right of final interpretation belongs to the organizer. In case of special circumstances, the organizer will coordinate the interpretation.
docs/m2met2/Track_setting_and_evaluation.md
+17
View File
@@ -0,0 +1,17 @@
# Track & Evaluation
## Speaker-Attributed ASR
The speaker-attributed ASR task poses a unique challenge of transcribing speech from multiple speakers and assigning a speaker label to the transcription. Figure 2 illustrates the difference between the speaker-attributed ASR task and the multi-speaker ASR task. This track allows for the use of the AliMeeting, Aishell4, and Cn-Celeb datasets as constrained data sources during both training and evaluation. The AliMeeting dataset, which was used in the M2MeT challenge, includes Train, Eval, and Test sets. Additionally, a new Test-2023 set, consisting of approximately 10 hours of meeting data recorded in an identical acoustic setting as the AliMeeting corpus, will be released soon for challenge scoring and ranking. It's worth noting that the organizers will not provide the near-field audio, transcriptions, or oracle timestamps of the Test-2023 set. Instead, segments containing multiple speakers will be provided, which can be obtained using a simple voice activity detection (VAD) model.
![task difference](images/task_diff.png)
## Evaluation metric
The accuracy of a speaker-attributed ASR system is evaluated using the concatenated minimum permutation character error rate (cpCER) metric. The calculation of cpCER involves three steps. Firstly, the reference and hypothesis transcriptions from each speaker in a session are concatenated in chronological order. Secondly, the character error rate (CER) is calculated between the concatenated reference and hypothesis transcriptions, and this process is repeated for all possible speaker permutations. Finally, the permutation with the lowest CER is selected as the cpCER for that session. TThe CER is obtained by dividing the total number of insertions (Ins), substitutions (Sub), and deletions(Del) of characters required to transform the ASR output into the reference transcript by the total number of characters in the reference transcript. Specifically, CER is calculated by:
$$\text{CER} = \frac {\mathcal N_{\text{Ins}} + \mathcal N_{\text{Sub}} + \mathcal N_{\text{Del}} }{\mathcal N_{\text{Total}}} \times 100\%,$$
where $\mathcal N_{\text{Ins}}$, $\mathcal N_{\text{Sub}}$, $\mathcal N_{\text{Del}}$ are the character number of the three errors, and $\mathcal N_{\text{Total}}$ is the total number of characters.
## Sub-track arrangement
### Sub-track I (Fixed Training Condition):
Participants are required to use only the fixed-constrained data (i.e., AliMeeting, Aishell-4, and CN-Celeb) for system development. The usage of any additional data is strictly prohibited. However, participants can use open-source pre-trained models from third-party sources, such as [Hugging Face](https://huggingface.co/models) and [ModelScope](https://www.modelscope.cn/models), provided that the list of utilized models is clearly stated in the final system description paper.
### Sub-track II (Open Training Condition):
Participants are allowed to use any publicly available data set, privately recorded data, and manual simulation to build their systems in addition to the fixed-constrained data. They can also utilize any open-source pre-trained models, but it is mandatory to provide a clear list of the data and models used in the final system description paper. If manually simulated data is used, a detailed description of the data simulation scheme must be provided.
docs/m2met2/conf.py
+44
View File
@@ -0,0 +1,44 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = "MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0"
copyright = "2023, Speech Lab, Alibaba Group; ASLP Group, Northwestern Polytechnical University"
author = "Speech Lab, Alibaba Group; Audio, Speech and Language Processing Group, Northwestern Polytechnical University"
extensions = [
"nbsphinx",
"sphinx.ext.autodoc",
"sphinx.ext.napoleon",
"sphinx.ext.viewcode",
"sphinx.ext.mathjax",
"sphinx.ext.todo",
# "sphinxarg.ext",
"sphinx_markdown_tables",
# 'recommonmark',
"sphinx_rtd_theme",
"myst_parser",
]
myst_enable_extensions = [
"colon_fence",
"deflist",
"dollarmath",
"html_image",
]
myst_heading_anchors = 2
myst_highlight_code_blocks = True
myst_update_mathjax = False
templates_path = ["_templates"]
source_suffix = [".rst", ".md"]
pygments_style = "sphinx"
html_theme = "sphinx_rtd_theme"
docs/m2met2/images/buhui.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/m2met2/images/chenzhuo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 166 KiB

docs/m2met2/images/dataset_details.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 232 KiB

docs/m2met2/images/kong.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

docs/m2met2/images/lxie.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/m2met2/images/meeting_room.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 610 KiB

docs/m2met2/images/qian.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

docs/m2met2/images/sa_asr_arch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 991 KiB

docs/m2met2/images/task_diff.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 748 KiB

docs/m2met2/images/wujian.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

docs/m2met2/images/zhijie.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/m2met2/images/zsl.JPG
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 MiB

docs/m2met2/index.rst
+26
View File
@@ -0,0 +1,26 @@
.. m2met2 documentation master file, created by
sphinx-quickstart on Tue Apr 11 14:18:55 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
ASRU 2023 MULTI-CHANNEL MULTI-PARTY MEETING TRANSCRIPTION CHALLENGE 2.0 (M2MeT2.0)
==================================================================================
Building on the success of the M2MeT challenge, we are delighted to propose the M2MeT2.0 challenge as a special session at ASRU2023.
To advance the current state-of-the-art in multi-talker automatic speech recognition, the M2MeT2.0 challenge proposes a speaker-attributed ASR task, comprising two sub-tracks: fixed and open training conditions.
To facilitate reproducible research, we provide a comprehensive overview of the dataset, challenge rules, evaluation metrics, and baseline systems.
Now the new test set contains about 10 hours audio is available. You can download from `here <https://speech-lab-share-data.oss-cn-shanghai.aliyuncs.com/AliMeeting/openlr/Test_2023_Ali.tar.gz>`_
.. toctree::
:maxdepth: 1
:caption: Contents:
./Introduction
./Dataset
./Track_setting_and_evaluation
./Baseline
./Rules
./Challenge_result
./Organizers
./Contact
docs/m2met2/make.bat
+35
View File
@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/make.bat
+35
View File
@@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
docs/migration_from_whisper.md
+96
View File
@@ -0,0 +1,96 @@
# Migrate from Whisper or Cloud ASR to FunASR
Use this guide when you already have a Whisper, OpenAI/Cloud ASR, or custom speech pipeline and want to decide whether FunASR is worth switching to. The goal is not to prove a benchmark with one sample file; it is to compare quality, speed, cost, and deployment fit on audio that looks like your real workload.
## When FunASR is a good fit
FunASR is usually worth evaluating when you need one or more of these properties:
- Private or self-hosted transcription where audio should stay inside your environment.
- High-throughput long-form transcription for meetings, archives, media, or call recordings.
- Speaker-aware transcripts with VAD, punctuation, timestamps, and diarization in one pipeline.
- An OpenAI-compatible audio endpoint for agents, Dify, LangChain, AutoGen, or internal apps.
- Streaming ASR or live captions with WebSocket/runtime service support.
- CPU-viable smoke tests before moving to GPU deployment.
Stay on your current pipeline if you need a managed service with no operations work, a vendor SLA, or a language/domain that your own benchmark shows FunASR does not handle well enough yet.
## Fast evaluation plan
1. Pick 20-50 representative audio files. Include short clips, long recordings, noisy samples, different speakers, and the languages or dialects you care about.
2. Run your current Whisper or cloud ASR pipeline exactly as you use it in production. Save transcripts, latency, cost, and failure cases.
3. Run FunASR locally with the README quick start, or use the [migration benchmark example](../examples/migration/) to measure a representative audio folder. Then choose a deployment path from the [deployment matrix](./deployment_matrix.md).
4. Compare output with human review or your normal WER/CER process. Do not compare only one clean demo file.
5. Run the OpenAI-compatible API smoke test if your application already uses OpenAI-style clients.
6. Record warmup time, model download time, device, GPU/CPU type, batch size, and audio duration separately from steady-state throughput.
## Feature mapping
| Existing workflow | FunASR path | What to validate |
|---|---|---|
| Whisper file transcription | [README quick start](../README.md#quick-start) and [model selection guide](./model_selection.md) with SenseVoice, Paraformer, or Fun-ASR-Nano | Transcript quality, timestamps, speed, model download, CPU/GPU behavior. |
| Whisper plus pyannote | `spk_model="cam++"` with VAD and punctuation | Speaker labels, speaker changes, overlapping speech, long silences. |
| OpenAI audio API or cloud batch ASR | [OpenAI-compatible API example](../examples/openai_api/) | `/v1/audio/transcriptions`, response format, client compatibility, upload limits. |
| Dify/LangChain/AutoGen agent audio | [Client recipes](../examples/openai_api/CLIENTS.md) or [MCP server](../examples/mcp_server/) | Tool latency, file handling, auth boundary, error reporting. |
| Live captions or call-center streaming | [Runtime service docs](../runtime/readme.md) | Chunking, endpointing, reconnects, backpressure, partial/final result behavior. |
| Subtitle generation | [Subtitle example](../examples/subtitle/) | Segment readability, line length, speaker labels, SRT/VTT compatibility. |
| Offline archive processing | [Batch ASR example](../examples/batch_asr_improved.py) | Manifest handling, retries, progress logs, throughput, failed-file recovery. |
## Minimal local comparison
Install FunASR and run the same file you used for your baseline:
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # use "cpu" for a portable smoke test
)
result = model.generate(input="sample.wav")
print(result)
```
For an API-style comparison:
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
If you want a repeatable folder-level benchmark, run [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) to produce `results.jsonl` and `summary.md` for your own audio set. For a container smoke test, start from `examples/openai_api/docker-compose.yml` and verify it with `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh`.
## Quality and speed checklist
Track these fields for both the old pipeline and FunASR:
- Audio duration, language, domain, sample rate, channel count, and speaker count.
- Model name, model version, FunASR version, Python/PyTorch/CUDA versions, and Docker image tag if used.
- Hardware, device mode, batch size, streaming chunk size, and whether warmup/model download is excluded.
- WER/CER or human review notes for names, numbers, punctuation, diarization, timestamps, and domain terms.
- Latency, throughput, GPU/CPU memory, cost per hour of audio, and failed-file rate.
- Operational requirements: authentication, upload limits, TLS, logs, monitoring, retries, and retention rules.
## Rollout checklist
- Keep the old pipeline available until FunASR passes your representative benchmark.
- Start with an internal endpoint or batch job before exposing a public API.
- Add request IDs and log audio duration, model, device, latency, and error type for every request.
- Pin the model alias and deployment command in your runbook.
- Test noisy audio, silence, overlapping speakers, long files, non-UTF-8 filenames, and network interruptions.
- Open a [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) with your command, logs, model, device, and sample characteristics if you hit a blocker.
## Share the result
If FunASR replaces or complements your existing ASR stack, consider opening a [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) or [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md). Migration reports with hardware, speed, quality notes, and deployment details help new users choose the right path faster.
docs/migration_from_whisper_zh.md
+96
View File
@@ -0,0 +1,96 @@
# 从 Whisper 或云端 ASR 迁移到 FunASR
当你已经有 Whisper、OpenAI/云端 ASR 或自研语音流水线，并想判断是否值得切到 FunASR 时，可以按这个指南评估。目标不是用一个样例音频证明结论，而是在真实业务音频上比较质量、速度、成本和部署可行性。
## 什么时候值得评估 FunASR
如果你有以下需求，FunASR 通常值得优先测试：
- 音频需要留在私有环境内，不能上传到外部云服务。
- 会议、归档、媒体、客服录音等长音频需要高吞吐转写。
- 希望一条流水线内完成 VAD、标点、时间戳和说话人分离。
- 需要 OpenAI 兼容音频接口，接入 Agent、Dify、LangChain、AutoGen 或内部应用。
- 需要 WebSocket/runtime 服务支撑流式识别或实时字幕。
- 希望先用 CPU 做可复现 smoke test，再迁移到 GPU 服务。
如果你更需要完全托管服务、厂商 SLA，或你的自有评测显示目标语言/领域还不够好，可以暂时保留现有方案。
## 快速评估计划
1. 选择 20-50 条有代表性的音频，覆盖短音频、长录音、噪声、多说话人、目标语言和方言。
2. 按生产方式运行当前 Whisper 或云端 ASR，保存转写结果、延迟、成本和失败样例。
3. 用 README 快速开始跑 FunASR，也可以用 [迁移评测示例](../examples/migration/) 测量一组代表性音频。然后根据 [部署选型表](./deployment_matrix_zh.md) 选择服务路径。
4. 用人工审阅或现有 WER/CER 流程比较结果，不要只看一个干净 demo 音频。
5. 如果应用已经使用 OpenAI 风格客户端，运行 OpenAI 兼容 API smoke test。
6. 分开记录 warmup、模型下载、设备、GPU/CPU 型号、batch size、音频时长和稳定吞吐。
## 功能映射
| 现有流程 | FunASR 路径 | 需要验证什么 |
|---|---|---|
| Whisper 文件转写 | 使用 [README 快速开始](../README_zh.md#快速开始) 和 [模型选择指南](./model_selection_zh.md)，在 SenseVoice、Paraformer 或 Fun-ASR-Nano 中选型 | 转写质量、时间戳、速度、模型下载、CPU/GPU 行为。 |
| Whisper + pyannote | VAD、标点和 `spk_model="cam++"` | 说话人标签、换人位置、重叠说话、长静音。 |
| OpenAI 音频 API 或云端批量 ASR | [OpenAI 兼容 API 示例](../examples/openai_api/README_zh.md) | `/v1/audio/transcriptions`、响应格式、客户端兼容性、上传限制。 |
| Dify/LangChain/AutoGen Agent 音频 | [客户端配方](../examples/openai_api/CLIENTS.md) 或 [MCP 服务](../examples/mcp_server/) | 工具延迟、文件处理、鉴权边界、错误返回。 |
| 实时字幕或客服流式识别 | [Runtime 服务文档](../runtime/readme_cn.md) | 分块、断句、重连、背压、中间/最终结果行为。 |
| 字幕生成 | [字幕示例](../examples/subtitle/) | 分段可读性、行长、说话人标签、SRT/VTT 兼容性。 |
| 离线归档处理 | [批处理示例](../examples/batch_asr_improved.py) | manifest、重试、进度日志、吞吐、失败文件恢复。 |
## 最小本地对比
安装 FunASR，并用你基线评测里的同一条音频运行：
```bash
pip install funasr
```
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 便携 smoke test 可改成 "cpu"
)
result = model.generate(input="sample.wav")
print(result)
```
如果要按 API 服务方式对比：
```bash
pip install funasr fastapi uvicorn python-multipart
funasr-server --model sensevoice --device cuda
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
如果要对自己的音频目录做可复现评测，可以运行 [`examples/migration/benchmark_funasr.py`](../examples/migration/benchmark_funasr.py) 生成 `results.jsonl` 和 `summary.md`。如果需要容器 smoke test，可以从 `examples/openai_api/docker-compose.yml` 开始，并用 `BASE_URL=http://localhost:8000 bash examples/openai_api/smoke_test.sh` 验证。
## 质量与速度检查清单
对旧流水线和 FunASR 都记录这些字段：
- 音频时长、语言、领域、采样率、声道数和说话人数。
- 模型名、模型版本、FunASR 版本、Python/PyTorch/CUDA 版本，以及 Docker 镜像 tag。
- 硬件、设备模式、batch size、流式 chunk size，以及是否排除 warmup/模型下载时间。
- WER/CER 或人工审阅记录：姓名、数字、标点、说话人分离、时间戳、领域词。
- 延迟、吞吐、GPU/CPU 内存、每小时音频成本、失败文件比例。
- 运维要求：鉴权、上传限制、TLS、日志、监控、重试和数据留存规则。
## 上线检查清单
- 在代表性评测通过前，保留旧流水线作为回退。
- 先做内部 endpoint 或离线批处理，再对外暴露 API。
- 为每个请求记录 request id、音频时长、模型、设备、延迟和错误类型。
- 在 runbook 中固定模型别名和部署命令。
- 测试噪声、静音、多人重叠、长文件、非 UTF-8 文件名和网络中断。
- 遇到阻塞时，通过 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) 提交命令、日志、模型、设备和样本特征。
## 分享迁移结果
如果 FunASR 替代或补充了你的现有 ASR 栈，欢迎提交 [Migration Benchmark Report](https://github.com/modelscope/FunASR/issues/new?template=migration_benchmark.md) 或 [showcase issue](https://github.com/modelscope/FunASR/issues/new?template=showcase.md)。包含硬件、速度、质量记录和部署细节的迁移报告，能帮助新用户更快选型。
docs/model_selection.md
+99
View File
@@ -0,0 +1,99 @@
# FunASR Model Selection Guide
Use this guide when you are choosing a first model, comparing FunASR with Whisper or a cloud ASR provider, or deciding which model alias to expose through the OpenAI-compatible API.
## Fast default path
If you have a GPU, start with the flagship **Fun-ASR-Nano** — an LLM-based ASR model (SenseVoice encoder + a Qwen3 decoder) covering 31 languages, with the strongest accuracy on hard cases, context, and proper nouns:
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="meeting.wav")
print(result[0]["text"])
```
On CPU, or when you want multilingual + emotion/event tags and speaker-aware meeting transcripts in one fast non-autoregressive pass, use **SenseVoice-Small**:
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # use "cpu" for a portable smoke test
)
result = model.generate(input="meeting.wav")
```
Switch to Paraformer when your workload is Mandarin-only and you want character-level timestamps or hotwords.
## Decision table
| Need | Start with | Why | Next doc |
|---|---|---|---|
| Fast multilingual private transcription | SenseVoice-Small | Strong default with ASR, emotion tags, audio event tags, and CPU viability. | [README quick start](../README.md#quick-start) |
| Mandarin production ASR | Paraformer-Large | Mature Chinese ASR path with VAD and punctuation. | [Tutorial](./tutorial/README.md) |
| English-only route in the OpenAI API example | `paraformer-en` alias | Smaller English route for API compatibility checks. | [OpenAI API example](../examples/openai_api/) |
| LLM-based ASR or 31-language experiments | Fun-ASR-Nano | LLM-based model path; use vLLM when decoder throughput matters. | [vLLM guide](./vllm_guide.md) |
| Live captions or call-center streams | Runtime WebSocket service | Designed for long-lived streaming sessions and partial results. | [Runtime service docs](../runtime/readme.md) |
| Batch archive processing | SenseVoice-Small or Paraformer-Large | Stable offline transcription path; caller owns manifests, retries, and logs. | [Batch ASR example](../examples/batch_asr_improved.py) |
| Migration from Whisper/cloud ASR | SenseVoice-Small first, then benchmark alternatives | Gives a strong baseline before deeper model-specific tuning. | [Migration guide](./migration_from_whisper.md) |
## OpenAI-compatible API aliases
The `examples/openai_api` server exposes short aliases so application teams do not need to know model repository IDs:
| Alias | Underlying path | Use when |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | You want the default private speech API with multilingual ASR, event tags, and good CPU/GPU behavior. |
| `paraformer` | `paraformer-zh` with VAD and punctuation | You want a Mandarin-oriented production route. |
| `paraformer-en` | `paraformer-en` with VAD | You want a compact English route in OpenAI-style clients. |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | You are evaluating LLM-based ASR, 31-language coverage, or vLLM acceleration. |
For Ascend NPU deployments, treat `fun-asr-nano` separately from SenseVoice / Paraformer. The Fun-ASR-Nano PyTorch `AutoModel` path has community compatibility evidence on 310P3 after the NPU autocast fix, but it was much slower than CPU in that smoke test; `AutoModelVLLM` still depends on vLLM-Ascend operator support and has hit Qwen3 rotary / `TransData` failures. Use CUDA/vLLM, standard PyTorch CPU/GPU, or GGUF runtime for production unless you are actively validating an Ascend backend.
Check the live service before wiring clients:
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
For SDK, JavaScript, workflow, Postman, OpenAPI, Docker, and Kubernetes paths, start from the [OpenAI API example](../examples/openai_api/).
## Runtime choice by workload
| Workload | Runtime path | Notes |
|---|---|---|
| Notebook or one-off evaluation | Python `AutoModel` | Shortest path for install, model download, and output-shape checks. |
| Internal HTTP service | OpenAI-compatible API | Reuse OpenAI-style clients, Dify, n8n, LangChain, AutoGen, and HTTP nodes. |
| Repeatable local container demo | Docker Compose API | CPU-first smoke test; adapt the image before using CUDA. |
| Internal cluster service | Kubernetes API template | Private `ClusterIP`, persistent model cache, `/health` probes, and port-forward smoke test. |
| Live audio | Runtime WebSocket service | Validate chunk size, VAD, endpointing, reconnects, and client backpressure with real audio. |
| LLM-based ASR throughput | vLLM path for Fun-ASR-Nano | vLLM accelerates autoregressive decoding; it does not apply to non-autoregressive Paraformer. |
See the [deployment matrix](./deployment_matrix.md) when you are choosing between these paths.
## Benchmark before committing
Do not choose a model from a single clean demo file. Use a small representative set first:
- 20-50 audio files that cover short clips, long meetings, silence, noise, overlapping speakers, domain vocabulary, and target languages.
- Record model name, model revision, FunASR version, device, CPU/GPU type, CUDA/PyTorch version, runtime path, batch size, and whether warmup/model download time is excluded.
- Track quality with your normal WER/CER or human review process, not only transcript readability.
- Track latency, throughput, memory, failures, and upload size limits together.
- Keep at least one public sample for smoke tests and at least one private realistic sample for deployment validation.
For migration work, use the [migration benchmark example](../examples/migration/) and the [migration guide](./migration_from_whisper.md).
## Practical recommendations
- With a GPU, default to Fun-ASR-Nano — the flagship LLM-based model (31 languages), strongest on hard, contextual, and proper-noun-heavy audio.
- On CPU, or for multilingual + emotion workloads, use SenseVoice-Small (fast non-autoregressive, CPU-viable).
- Use Paraformer when your production traffic is primarily Mandarin and you want timestamps or hotwords.
- Use the streaming runtime when partial results and long-lived connections matter more than a single final transcript.
- Keep model aliases stable in production runbooks so benchmark results and bug reports are reproducible.
- Open a [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) with model, device, command, logs, audio duration, and runtime path when you get stuck.
docs/model_selection_ja.md
+62
View File
@@ -0,0 +1,62 @@
# FunASR モデル選択ガイド
初めて FunASR を試すとき、Whisper やクラウド ASR から移行するとき、または OpenAI 互換 API で公開するモデル alias を決めるときに使ってください。
## 迷ったらここから
まずは **SenseVoice-Small** から始めるのがおすすめです。
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 手元の smoke test では "cpu" でも可
)
result = model.generate(input="meeting.wav")
```
デモ、プライベート API、多言語文字起こし、話者付き会議録、Agent 音声入力の最初の選択肢として使いやすいモデルです。中国語本番精度、ストリーミング遅延、LLM-based ASR 評価など明確な要件が出たときだけ切り替えてください。
## 判断表
| やりたいこと | 最初に試すもの | 理由 | 次に読むもの |
|---|---|---|---|
| 高速な多言語プライベート文字起こし | SenseVoice-Small | ASR、感情タグ、音声イベントタグ、CPU/GPU の扱いやすさがそろった標準ルート。 | [README quick start](../README_ja.md#クイックスタート) |
| 中国語中心の本番 ASR | Paraformer-Large | VAD と句読点復元を組み合わせた成熟した中国語 ASR ルート。 | [Tutorial](./tutorial/README.md) |
| OpenAI API 例の英語ルート | `paraformer-en` alias | OpenAI-style client で互換性を確認しやすい軽量な英語ルート。 | [OpenAI API example](../examples/openai_api/README_ja.md) |
| LLM-based ASR や 31 言語の評価 | Fun-ASR-Nano | LLM-based モデル。decoder throughput が重要なら vLLM を使います。 | [vLLM guide](./vllm_guide.md) |
| ライブ字幕やコールセンターストリーム | Runtime WebSocket service | 長時間接続、部分結果、エンドポイント検出に向いたランタイム。 | [Runtime service docs](../runtime/readme.md) |
| Whisper / cloud ASR からの移行 | SenseVoice-Small で baseline を作り、必要に応じて比較 | まず強い標準ルートで評価してから、用途別に詰めるのが安全です。 | [Migration guide](./migration_from_whisper.md) |
## OpenAI 互換 API alias
`examples/openai_api` server は短い alias を提供します。アプリケーション側はモデル repository ID を知らなくても利用できます。
| Alias | 中身 | 使う場面 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 多言語 ASR、イベントタグ、CPU/GPU 両対応の標準プライベート音声 API。 |
| `paraformer` | `paraformer-zh` + VAD + punctuation | 中国語中心の本番ルート。 |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI-style client の英語互換性チェック。 |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | LLM-based ASR、31 言語、vLLM acceleration の評価。 |
接続前にサービスを確認します。
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK、JavaScript、workflow、Postman、OpenAPI、Docker、Kubernetes は [OpenAI API example](../examples/openai_api/README_ja.md) から始めてください。
## ベンチマークしてから決める
きれいな demo 音声 1 つだけでモデルを決めないでください。まず小さな代表セットで確認します。
- 短いクリップ、長い会議、無音、ノイズ、話者重なり、専門用語、対象言語を含む 20-50 ファイルを用意します。
- model name、model revision、FunASR version、device、CPU/GPU、CUDA/PyTorch、runtime path、batch size、download/warmup の扱いを記録します。
- 読みやすさだけでなく、通常使う WER/CER または人手レビューで品質を見ます。
- latency、throughput、memory、failure、upload size limit をまとめて比較します。
- 困ったときは model、device、command、logs、audio duration、runtime path を添えて [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) を開いてください。
docs/model_selection_ko.md
+62
View File
@@ -0,0 +1,62 @@
# FunASR 모델 선택 가이드
처음 FunASR을 사용할 때, Whisper나 클라우드 ASR에서 전환할 때, 또는 OpenAI 호환 API에서 노출할 model alias를 정할 때 참고하세요.
## 고민된다면 여기서 시작
먼저 **SenseVoice-Small**을 추천합니다.
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 간단한 smoke test는 "cpu"도 가능
)
result = model.generate(input="meeting.wav")
```
데모, 프라이빗 API, 다국어 전사, 화자 포함 회의록, Agent 음성 입력의 첫 선택지로 사용하기 좋습니다. 중국어 프로덕션 정확도, 스트리밍 지연 시간, LLM-based ASR 평가처럼 명확한 요구가 있을 때만 다른 경로로 전환하세요.
## 결정 표
| 필요 | 먼저 시도할 것 | 이유 | 다음 문서 |
|---|---|---|---|
| 빠른 다국어 프라이빗 전사 | SenseVoice-Small | ASR, 감정 태그, 음성 이벤트 태그, CPU/GPU 사용성이 균형 잡힌 기본 경로입니다. | [README quick start](../README_ko.md#빠른-시작) |
| 중국어 중심 프로덕션 ASR | Paraformer-Large | VAD와 문장부호 복원을 함께 쓰는 성숙한 중국어 ASR 경로입니다. | [Tutorial](./tutorial/README.md) |
| OpenAI API 예제의 영어 경로 | `paraformer-en` alias | OpenAI-style client에서 호환성을 확인하기 쉬운 가벼운 영어 경로입니다. | [OpenAI API example](../examples/openai_api/README_ko.md) |
| LLM-based ASR 또는 31개 언어 평가 | Fun-ASR-Nano | LLM-based 모델입니다. decoder throughput이 중요하면 vLLM을 사용합니다. | [vLLM guide](./vllm_guide.md) |
| 라이브 자막 또는 콜센터 스트림 | Runtime WebSocket service | 장시간 연결, 부분 결과, endpointing에 맞춘 런타임입니다. | [Runtime service docs](../runtime/readme.md) |
| Whisper / cloud ASR에서 전환 | SenseVoice-Small로 baseline을 만들고 필요하면 비교 | 강한 기본 경로로 먼저 평가한 뒤 용도별로 조정하는 편이 안전합니다. | [Migration guide](./migration_from_whisper.md) |
## OpenAI 호환 API alias
`examples/openai_api` server는 짧은 alias를 제공합니다. 애플리케이션 팀은 모델 repository ID를 몰라도 사용할 수 있습니다.
| Alias | 내부 경로 | 사용 시점 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 다국어 ASR, 이벤트 태그, CPU/GPU 동작이 균형 잡힌 기본 프라이빗 음성 API. |
| `paraformer` | `paraformer-zh` + VAD + punctuation | 중국어 중심 프로덕션 경로. |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI-style client의 영어 호환성 확인. |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | LLM-based ASR, 31개 언어, vLLM acceleration 평가. |
클라이언트를 연결하기 전에 서비스를 확인하세요.
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK, JavaScript, workflow, Postman, OpenAPI, Docker, Kubernetes는 [OpenAI API example](../examples/openai_api/README_ko.md)에서 시작하세요.
## 벤치마크 후 결정하기
깨끗한 demo 오디오 하나만 보고 모델을 정하지 마세요. 먼저 작은 대표 세트로 확인합니다.
- 짧은 클립, 긴 회의, 무음, 잡음, 화자 겹침, 도메인 용어, 대상 언어를 포함하는 20-50개 파일을 준비합니다.
- model name, model revision, FunASR version, device, CPU/GPU, CUDA/PyTorch, runtime path, batch size, download/warmup 처리 여부를 기록합니다.
- 읽기 쉬움만 보지 말고, 평소 사용하는 WER/CER 또는 사람 리뷰로 품질을 확인합니다.
- latency, throughput, memory, failure, upload size limit을 함께 비교합니다.
- 막히면 model, device, command, logs, audio duration, runtime path를 포함해 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md)를 열어 주세요.
docs/model_selection_zh.md
+99
View File
@@ -0,0 +1,99 @@
# FunASR 模型选择指南
当你第一次选择模型、评估是否从 Whisper 或云端 ASR 迁移，或者准备通过 OpenAI 兼容 API 暴露模型别名时，可以先看这份指南。
## 默认快速路径
如果有 GPU，先从旗舰 **Fun-ASR-Nano** 开始 —— 基于 LLM 的识别模型（SenseVoice 编码器 + Qwen3 解码），覆盖 31 语种，在难例、上下文和专名上精度最强：
```python
from funasr import AutoModel
model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="meeting.wav")
print(result[0]["text"])
```
在 CPU 上，或当你想要多语种 + 情感/事件标签、带说话人信息的会议转写一次完成时，用 **SenseVoice-Small**：
```python
from funasr import AutoModel
model = AutoModel(
model="iic/SenseVoiceSmall",
vad_model="fsmn-vad",
spk_model="cam++",
device="cuda", # 便携 smoke test 可改为 "cpu"
)
result = model.generate(input="meeting.wav")
```
当你的场景是纯中文、需要字级时间戳或热词时，切换到 Paraformer。
## 决策表
| 需求 | 优先尝试 | 原因 | 下一步文档 |
|---|---|---|---|
| 快速多语种私有转写 | SenseVoice-Small | 兼顾 ASR、情感标签、音频事件标签和 CPU 可用性。 | [README 快速开始](../README_zh.md#快速开始) |
| 中文生产 ASR | Paraformer-Large | 成熟中文 ASR 路径，可组合 VAD 和标点。 | [教程](./tutorial/README_zh.md) |
| OpenAI API 示例中的英文路由 | `paraformer-en` alias | 适合在 OpenAI 风格客户端里验证较轻量英文路径。 | [OpenAI API 示例](../examples/openai_api/README_zh.md) |
| LLM-based ASR 或 31 语种实验 | Fun-ASR-Nano | LLM-based 模型路径；解码吞吐敏感时配合 vLLM。 | [vLLM 指南](./vllm_guide.md) |
| 实时字幕或客服流式音频 | Runtime WebSocket 服务 | 面向长连接流式会话和中间结果。 | [Runtime 服务文档](../runtime/readme_cn.md) |
| 录音归档批处理 | SenseVoice-Small 或 Paraformer-Large | 稳定离线转写路径；调用方负责 manifest、重试和日志。 | [批处理示例](../examples/batch_asr_improved.py) |
| 从 Whisper/云端 ASR 迁移 | 先用 SenseVoice-Small，再 benchmark 其他模型 | 先建立强基线，再做模型专项调优。 | [迁移指南](./migration_from_whisper_zh.md) |
## OpenAI 兼容 API 别名
`examples/openai_api` 服务提供短别名，应用团队不需要了解具体模型仓库 ID：
| Alias | 底层路径 | 适合场景 |
|---|---|---|
| `sensevoice` | `iic/SenseVoiceSmall` | 默认私有语音 API，多语种 ASR、事件标签和 CPU/GPU 行为较均衡。 |
| `paraformer` | `paraformer-zh` + VAD + 标点 | 中文生产流量优先尝试。 |
| `paraformer-en` | `paraformer-en` + VAD | OpenAI 风格客户端里的英文轻量路由。 |
| `fun-asr-nano` | `FunAudioLLM/Fun-ASR-Nano-2512` | 评估 LLM-based ASR、31 语种覆盖或 vLLM 加速。 |
如果部署目标是昇腾 NPU，请把 `fun-asr-nano` 和 SenseVoice / Paraformer 分开看。Fun-ASR-Nano 的 PyTorch `AutoModel` 路径在修复 NPU autocast 后已有 310P3 社区兼容性 smoke 结果，但该测试明显慢于 CPU；`AutoModelVLLM` 仍依赖 vLLM-Ascend 算子支持，并已遇到 Qwen3 rotary / `TransData` 失败。生产部署优先使用 CUDA/vLLM、标准 PyTorch CPU/GPU 或 GGUF runtime，除非你正在主动验证 Ascend 后端。
接入客户端前先检查在线服务：
```bash
curl http://localhost:8000/v1/models
python examples/openai_api/smoke_test.py --base-url http://localhost:8000 --model sensevoice
```
SDK、JavaScript、工作流、Postman、OpenAPI、Docker 和 Kubernetes 路径可从 [OpenAI API 示例](../examples/openai_api/README_zh.md) 开始。
## 按工作负载选择运行路径
| 工作负载 | 运行路径 | 说明 |
|---|---|---|
| Notebook 或一次性评估 | Python `AutoModel` | 验证安装、模型下载和输出结构的最短路径。 |
| 内部 HTTP 服务 | OpenAI 兼容 API | 复用 OpenAI 风格客户端、Dify、n8n、LangChain、AutoGen 和 HTTP 节点。 |
| 可复现本地容器 demo | Docker Compose API | CPU-first smoke test；使用 CUDA 前先适配镜像。 |
| 集群内私有服务 | Kubernetes API 模板 | 私有 `ClusterIP`、持久化模型缓存、`/health` probes 和 port-forward smoke test。 |
| 实时音频 | Runtime WebSocket 服务 | 用真实音频验证 chunk size、VAD、断句、重连和客户端背压。 |
| LLM-based ASR 吞吐 | Fun-ASR-Nano 的 vLLM 路径 | vLLM 加速自回归解码；不适用于非自回归 Paraformer。 |
选择部署方式时可以参考 [部署选型表](./deployment_matrix_zh.md)。
## 上线前先 benchmark
不要只用一个干净 demo 文件选型。先准备一个小而有代表性的集合：
- 20-50 条音频，覆盖短音频、长会议、静音、噪声、多人重叠、领域词汇和目标语言。
- 记录模型名、模型版本、FunASR 版本、设备、CPU/GPU 型号、CUDA/PyTorch 版本、运行路径、batch size，以及是否排除 warmup/模型下载时间。
- 使用你已有的 WER/CER 流程或人工审阅，不要只看转写文本是否“读起来还行”。
- 同时记录延迟、吞吐、内存、失败样例和上传大小限制。
- 保留至少一个公开样例用于 smoke test，也保留至少一个真实私有样本用于部署验证。
迁移场景可以使用 [迁移评测示例](../examples/migration/) 和 [迁移指南](./migration_from_whisper_zh.md)。
## 实用建议
- demo、私有 API、Agent 语音输入和多语种场景优先试 SenseVoice-Small。
- 中文生产流量优先试 Paraformer，尤其是希望走成熟非自回归 ASR 路径时。
- 明确需要 LLM-based 模型路径或 vLLM 加速实验时，再试 Fun-ASR-Nano。
- 需要中间结果和长连接时，优先使用 streaming runtime，而不是普通 HTTP 转写接口。
- 生产 runbook 中固定模型 alias，保证 benchmark 和问题复现可追踪。
- 遇到阻塞时，用 [Deployment Help issue](https://github.com/modelscope/FunASR/issues/new?template=deployment_help.md) 提供模型、设备、命令、日志、音频时长和运行路径。
docs/reference/FQA.md
+236
View File
@@ -0,0 +1,236 @@
# FAQ and Troubleshooting
This page collects the questions that most often block new FunASR users. Start here before opening an issue.
## Which install command should I use?
For most users:
```bash
pip install -U funasr
```
For the newest examples, server CLI, or unreleased fixes:
```bash
git clone https://github.com/modelscope/FunASR.git
cd FunASR
pip install -e ./
```
For the OpenAI-compatible API server, install the web runtime dependencies too:
```bash
pip install funasr fastapi uvicorn python-multipart
```
## Which Python and PyTorch versions are recommended?
Use Python 3.8 or later and install a PyTorch/torchaudio pair that matches your CUDA runtime. If CUDA is not configured, start with CPU to verify the workflow first:
```bash
funasr-server --model sensevoice --device cpu
```
After the CPU smoke test works, switch to CUDA:
```bash
funasr-server --model sensevoice --device cuda
```
## Model download is slow or fails. What should I check?
FunASR models are available from ModelScope and Hugging Face. Choose the hub that is fastest in your network environment, and make sure the machine can reach it before debugging model code.
Common checks:
```bash
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
python -c "import funasr; print(funasr.__version__)"
```
If a model is already downloaded on another machine, use the local model path instead of the remote model name.
## `funasr-server` says FastAPI or multipart packages are missing
Install the server dependencies:
```bash
pip install fastapi uvicorn python-multipart
```
Then start again:
```bash
funasr-server --model sensevoice --device cuda
```
## Port 8000 is already in use
Start the service on another port:
```bash
funasr-server --model sensevoice --device cuda --port 9000
```
Then point clients to the new base URL:
```bash
curl http://localhost:9000/health
```
## How do I verify the OpenAI-compatible API quickly?
Start the server:
```bash
funasr-server --model sensevoice --device cuda
```
In another terminal:
```bash
curl -L https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav -o sample.wav
curl http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav \
-F model=sensevoice \
-F response_format=verbose_json
```
The response should include `text`. With `verbose_json`, supported models may also return segment-level information.
## How do I run the OpenAI-compatible API with Docker Compose?
Use the example Compose setup when you want a reproducible local smoke test before wiring the API into a product or agent workflow:
```bash
cd examples/openai_api
cp .env.example .env
docker compose up --build
```
Then verify it from another terminal:
```bash
BASE_URL=http://localhost:8000 bash smoke_test.sh
```
The example container defaults to `FUNASR_DEVICE=cpu` so it can start on machines without NVIDIA Container Toolkit. For CUDA, first adapt the image to use CUDA-capable PyTorch/FunASR dependencies, then set `FUNASR_DEVICE=cuda`.
See also:
- [OpenAI API example](../../examples/openai_api/)
- [Client recipes](../../examples/openai_api/CLIENTS.md)
- [Deployment matrix](../deployment_matrix.md)
## Docker starts but `/health` or transcription fails
Check the container logs first:
```bash
docker compose logs -f funasr-api
```
Common causes:
- The first startup is still downloading or loading the model.
- Port 8000 is already in use; set `FUNASR_HOST_PORT=9000` in `.env` and use `BASE_URL=http://localhost:9000`.
- The container is running in CPU mode but `.env` or the command expects CUDA.
- CUDA is requested but the image does not include CUDA-capable PyTorch/FunASR dependencies.
- The model cache volume is empty or corrupted; retry after removing the `funasr-cache` Docker volume.
- The uploaded audio file is too large for the machine; verify with the public sample before testing long recordings.
When opening a Deployment Help issue, include your `.env` values without secrets, the `docker compose` command, container logs, model alias, device, and audio duration.
## Long audio is slow, split incorrectly, or runs out of memory
Use VAD segmentation for long audio and tune segment length for your hardware:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
vad_kwargs={"max_single_segment_time": 30000},
device="cuda",
)
result = model.generate(input="long_meeting.wav", batch_size_s=300)
```
If memory is limited, reduce `batch_size_s`, use CPU for verification, or split very long recordings before batch processing.
## Speaker diarization has no speaker labels
Use a model pipeline that includes both VAD and speaker models:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
device="cuda",
)
result = model.generate(input="meeting.wav")
```
Then inspect `result[0]["sentence_info"]`. Each sentence should include fields such as `text`, `start`, `end`, and `spk` when diarization is available.
## Can I use a speaker model other than cam++?
Yes. `spk_model` can be any FunASR `AutoModel` speaker embedding model, including a ModelScope model ID or a local model path. For example, ERes2NetV2 can be used directly:
```python
from funasr import AutoModel
model = AutoModel(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="iic/speech_eres2netv2_sv_zh-cn_16k-common",
device="cuda",
)
result = model.generate(input="meeting.wav")
```
For a fully custom speaker model, make sure the model can be loaded by FunASR `AutoModel` and that its `inference()` method returns `spk_embedding`; the user-facing call still goes through `AutoModel.generate()`. The diarization post-processing and clustering steps use those embeddings to assign speaker labels. If your diarization model emits speaker segments directly instead of embeddings, add an adapter layer that converts its output to the current speaker-label structure.
The tutorial has a longer ERes2NetV2 example: [Speaker Verification / Diarization](../tutorial/README.md#speaker-verification--diarization-eres2netv2).
## The same command works on CPU but fails on CUDA
This usually points to a CUDA, driver, PyTorch, or GPU memory mismatch. Include these checks in your issue:
```bash
nvidia-smi
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
python -c "import torchaudio; print(torchaudio.__version__)"
```
Try a smaller model or lower batch size to rule out GPU memory pressure.
## What information should I include in an issue?
Please include:
- OS and Python version
- FunASR version and install method (`pip`, source, Docker)
- PyTorch, torchaudio, CUDA, and GPU information
- Exact command or minimal Python snippet
- Full traceback or server logs
- Model name and hub (`modelscope`, `hf`, or local path)
- Audio duration, sample rate, format, language, speaker count, and whether the audio can be shared
## Existing ModelScope pipeline examples
- [VAD model with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/236)
- [Punctuation model with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/238)
- [Paraformer streaming with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/241)
- [VAD + ASR + punctuation with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/278)
- [VAD + ASR + punctuation + NNLM with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/134)
- [Timestamp prediction with ModelScope pipeline](https://github.com/modelscope/FunASR/discussions/246)
- [Switch online/offline decoding for UniASR](https://github.com/modelscope/FunASR/discussions/151)
docs/reference/application.md
+5
View File
@@ -0,0 +1,5 @@
## Audio Cut
## Realtime Speech Recognition
## Audio Chat

Some files were not shown because too many files have changed in this diff Show More