wehub/modelscope--funasr
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:10 +08:00
commit c397331b1e
3684 changed files with 990993 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/benchmark/realtime_ws_benchmark.md
+92
View File
@@ -0,0 +1,92 @@
# Realtime WebSocket Benchmark
Use this benchmark when you need to measure the client-observable behavior of
`examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py` under
real streaming traffic. Offline `RTFx` and realtime service latency are
different metrics: this page focuses on first update latency, final latency
after `STOP`, response lag, and multi-client behavior.
The benchmark client accepts only 16 kHz mono PCM16 WAV input. Keeping the input
format strict removes resampling and file decoding from the measurement.
## Start the Service
For long continuous speech or multiple browser clients, start with a bounded
partial window and a moderate partial refresh interval:
```bash
CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_realtime_ws.py \
--port 10095 --language 中文 \
--partial-window-sec 8 --decode-interval 0.8
```
Speaker diarization is disabled by default. Add `--enable-spk` only when the
`spk` field is required, and report that setting with the benchmark result.
## Run a Single Realtime Replay
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 1 \
--output-jsonl realtime_ws_1c.jsonl
```
With pacing enabled, the client sends audio at realtime speed using 100 ms
frames. This is the closest mode to a microphone or browser stream.
## Run Concurrent Replays
```bash
python examples/industrial_data_pretraining/fun_asr_nano/realtime_ws_benchmark.py \
audio_16k_mono_pcm16.wav \
--server ws://localhost:10095 \
--clients 8 \
--loops 3 \
--chunk-ms 100 \
--language 中文 \
--output-jsonl realtime_ws_8c.jsonl
```
Use a representative audio file. A long, pauseless monologue creates a very
different load shape from turn-taking meetings, because nearly every client is
speaking and triggering partial decodes at the same time.
For an unpaced stress test, add `--no-pace`. Treat that result as a throughput
stress signal, not as user-facing realtime latency.
## Metrics
| Metric | Meaning |
|--------|---------|
| `aggregate_audio_per_wall` | Total input audio seconds across all clients divided by benchmark wall time |
| `first_update_ms_p50/p95` | Time from first audio frame to first result message with `sentences`, `partial`, or `is_final` |
| `final_after_stop_ms_p50/p95` | Time from sending `STOP` to receiving the final result |
| `client_response_lag_ms_p95_max` | Largest per-client p95 of non-final `(client receive time - audio start) - server duration_ms`; useful mainly in paced mode for preview/partial lag |
| `partial_messages` | Count of non-final result messages with a non-empty `partial` |
| `final_messages` | Count of final result messages |
| `errors` | Connection, timeout, protocol, or client-side validation errors |
The script can observe only client-side timing and fields returned by the
server. If you are debugging service internals, collect server logs separately
for queue wait, VAD time, ASR decode time, speaker diarization time, GPU memory,
and GPU utilization.
## Report Template
When publishing a realtime WebSocket benchmark or issue report, include:
| Category | What to record |
|----------|----------------|
| Data | Audio duration, sample rate, language/domain, silence ratio or speaking pattern, and whether the same file was looped |
| Load | `--clients`, `--loops`, `--chunk-ms`, paced or `--no-pace`, and total benchmark wall time |
| Service | `serve_realtime_ws.py` command, `--partial-window-sec`, `--decode-interval`, `--enable-spk`, language, and hotwords |
| Hardware | GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions, CPU model, and available RAM |
| Software | `funasr`, PyTorch, torchaudio, vLLM, Python, OS, and container image if any |
| Output | Summary line, JSONL artifact, server logs, and any failed client IDs |
Do not reuse an offline `RTFx` number as a concurrency claim. For realtime
service sizing, benchmark with the actual traffic shape, especially sentence
length, pause distribution, simultaneous speakers, and whether speaker
diarization is enabled.
docs/benchmark/rtf_reproducibility.md
+107
View File
@@ -0,0 +1,107 @@
# Benchmark RTF and Reproducibility Notes
Use this page when comparing FunASR with Whisper, a cloud ASR provider, a Rust
runtime, or another self-hosted engine. Speed numbers are only useful when the
timing scope, data, model, runtime, and hardware are reported together.
## RTF and RTFx
FunASR benchmark tables usually report throughput as `RTFx`, or "times
realtime":
```text
RTF = processing_time_seconds / input_audio_seconds
RTFx = input_audio_seconds / processing_time_seconds
= 1 / RTF
```
For example, an `RTFx` value of `340` means 340 seconds of input audio are
processed in about 1 second, under that benchmark's data, runtime, batching, and
hardware setup. On the public vLLM table, the 184-file set has 11,541 seconds of
audio, so `340x` corresponds to roughly 34 seconds of measured processing time
for the whole set if the same scope is used:
```text
11541 / 340 = 33.94 seconds
```
Do not compare an offline batch `RTFx` result with streaming first-token latency
or end-to-end product latency. They measure different things.
For realtime WebSocket service sizing, use the
[Realtime WebSocket Benchmark](./realtime_ws_benchmark.md) instead.
## Current Public vLLM Benchmark Scope
The vLLM guide currently reports the following public scope for the Fun-ASR-Nano
and GLM-ASR-Nano table:
| Field | Value |
|-------|-------|
| Audio set | 184 long-form files |
| Total audio duration | 11,541 seconds, about 192 minutes |
| Models | Fun-ASR-Nano and GLM-ASR-Nano |
| Reported metric | CER and `RTFx` throughput |
| Fun-ASR-Nano vLLM batch result | `RTFx 340`, CER `8.20%` |
| Fun-ASR-Nano PyTorch baseline | `RTFx 21`, CER `8.06%` |
| Fun-ASR-Nano offline service without speaker diarization | `RTFx 102`, CER `8.14%` |
| Fun-ASR-Nano offline service with speaker diarization | `RTFx 46`, CER `8.19%` |
The table describes offline throughput on the stated long-form set. It should
not be read as a guarantee for every GPU, batch shape, language mix, streaming
chunk size, or service deployment.
The main website benchmark page is a separate public table for the broader ASR
comparison. It reports 184 long-form Chinese audio files, 11,539 seconds total,
and an NVIDIA H100 80GB HBM3 GPU. Keep the two tables separate when citing
numbers: the website table documents the general ASR benchmark, while the vLLM
guide table documents the Fun-ASR-Nano / GLM-ASR-Nano vLLM throughput rows.
## Required Fields for Reproducible Benchmark Claims
When publishing a FunASR benchmark, include these fields with the number:
| Category | What to record |
|----------|----------------|
| Data | File count, total audio duration, language/domain, sample rate, mono/stereo handling, and whether test files are public |
| Model | Model ID, checkpoint source, model revision or commit, language setting, hotwords, and text normalization |
| Runtime | Python SDK, ONNX, C++, vLLM, llama.cpp/GGUF, API server, or another path |
| Hardware | CPU model and thread count, GPU/NPU model, GPU count, memory, driver, CUDA/CANN/runtime versions |
| Software | `funasr`, PyTorch, torchaudio, vLLM, ONNX Runtime, CUDA, Python, and operating system versions |
| Pipeline | VAD, punctuation, speaker diarization, ITN, timestamps, and post-processing on/off |
| Batching | Batch size, `batch_size_s`, concurrent requests, tensor parallel size, chunk size, VAD segment policy |
| Timing scope | Whether timing includes model download, cold start, warmup, file I/O, audio decoding/resampling, VAD, post-processing, and result serialization |
| Quality | CER/WER method, reference normalization, ignored tokens, and failed-file handling |
For official README or website numbers, include the fields above or link to a
report that includes them.
## Suggested Timing Protocol
1. Put all input audio in a manifest or directory and compute total duration
before running ASR.
2. Warm the model once if the published number is intended to represent steady
state throughput. If you include cold start, say so explicitly.
3. Time exactly one scope: model-only, pipeline-only, or end-to-end service.
4. Run the same scope at least three times and report median plus min/max.
5. Keep transcript output, failed-file list, and timing JSON/CSV with the run.
For migration or product evaluation, start from
[`examples/migration/benchmark_funasr.py`](../../examples/migration/benchmark_funasr.py).
It writes per-file timing and a Markdown summary for your own audio set. The
same reporting fields above also apply when you use vLLM, ONNX, C++, GGUF, or a
custom runtime instead of the migration example.
## Comparing with a Rust or Other Custom Runtime
For a fair engine-to-engine comparison:
- use the same audio files and total duration;
- resample and downmix with the same policy;
- keep VAD, punctuation, speaker diarization, and timestamps either all on or
all off;
- compare both speed and quality, because a faster decode path can change CER;
- report `RTFx` and raw processing seconds, not only a relative speedup.
If you can share your result publicly, open a Migration Benchmark Report issue
with the fields above. That makes the comparison useful to other users and gives
maintainers enough context to reproduce or improve the path.