wehub/microsoft--vibevoice
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:05:25 +08:00
commit b0897668a9
102 changed files with 20314 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/setup_gradio_demo.md
+196
View File
@@ -0,0 +1,196 @@
# VibeVoice Gradio Demo Setup Guide
End-to-end instructions to deploy the VibeVoice ASR server and launch the Gradio web demo.
## Prerequisites
- CUDA-capable GPU(s)
- Docker with GPU support (`nvidia-docker`)
- VibeVoice repository cloned locally
```bash
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice
```
---
## Step 1 — Start the ASR Server
Launch a Docker container running the vLLM ASR server. The launcher script handles everything automatically (system deps, pip install, model download, tokenizer generation, server start).
### Single GPU (default)
```bash
docker run -d --gpus '"device=0"' --name vibevoice-asr-demo \
--ipc=host \
-p 6001:6001 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --port 6001"
```
### Multi-GPU with Data Parallel (load balancing)
Run 4 independent replicas, one per GPU. vLLM distributes requests automatically:
```bash
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-asr-demo \
--ipc=host \
-p 6001:6001 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --port 6001 --dp 4"
```
> **Tip**: Use `--dp N` for N-way data parallel (throughput scaling). Use `--tp N` for tensor parallel (large models). See `docs/vibevoice-vllm-asr.md` for details.
### Check Logs
```bash
docker logs -f vibevoice-asr-demo
```
Wait until you see `Application startup complete.` — this means the server is ready.
---
## Step 2 — Verify the Server
```bash
# Check the model is loaded
curl http://localhost:6001/v1/models
```
Expected output:
```json
{
"data": [{ "id": "vibevoice", ... }]
}
```
### Quick Test with Audio File
```bash
docker exec -it vibevoice-asr-demo \
python3 /app/vllm_plugin/tests/test_api.py /app/en-Alice_woman.wav \
--url http://localhost:6001
```
---
## Step 3 — Launch the Gradio Demo
### Install tmux inside the container (to keep it running)
```bash
docker exec vibevoice-asr-demo apt-get install -y tmux
```
### Start Gradio in tmux
```bash
docker exec vibevoice-asr-demo bash -c \
"PYTHONUNBUFFERED=1 tmux new-session -d -s gradio \
'PYTHONUNBUFFERED=1 python3 /app/vllm_plugin/scripts/gradio_asr_demo_api_video.py \
--api_url http://localhost:6001 --share \
2>&1 | tee /tmp/gradio.log'"
```
### Get the Share Link
Wait ~20 seconds, then:
```bash
docker exec vibevoice-asr-demo cat /tmp/gradio.log
```
You should see:
```
✅ Connected to API: http://localhost:6001 | Model: vibevoice
🚀 Starting VibeVoice ASR Demo
* Running on local URL: http://0.0.0.0:7860
* Running on public URL: https://xxxxxx.gradio.live
```
The `gradio.live` link is publicly accessible (valid for 1 week).
### Gradio Options
| Flag | Description | Default |
|------|-------------|---------|
| `--api_url URL` | vLLM server URL | `http://localhost:8000` |
| `--share` | Create a public Gradio link | off |
| `--port PORT` | Local Gradio port | `7860` |
| `--cloudflared` | Use Cloudflare tunnel instead of Gradio share | off |
| `--max_video_size MB` | Max upload video size | `50` |
---
## Managing the Service
### Stop Gradio (keep ASR server running)
```bash
docker exec vibevoice-asr-demo tmux kill-session -t gradio
```
### Restart Gradio
Re-run the tmux command from Step 3.
### Stop Everything
```bash
docker stop vibevoice-asr-demo
docker rm vibevoice-asr-demo
```
---
## Example: Full Setup on GPU 0 with Port 6001
```bash
# 1. Start server
docker run -d --gpus '"device=0"' --name vibevoice-asr-demo \
--ipc=host -p 6001:6001 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app -w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --port 6001"
# 2. Wait for startup (~2 min), then verify
docker logs -f vibevoice-asr-demo # wait for "Application startup complete."
curl http://localhost:6001/v1/models
# 3. Install tmux and launch Gradio
docker exec vibevoice-asr-demo apt-get install -y tmux
docker exec vibevoice-asr-demo bash -c \
"PYTHONUNBUFFERED=1 tmux new-session -d -s gradio \
'PYTHONUNBUFFERED=1 python3 /app/vllm_plugin/scripts/gradio_asr_demo_api_video.py \
--api_url http://localhost:6001 --share \
2>&1 | tee /tmp/gradio.log'"
# 4. Get the public link
sleep 20 && docker exec vibevoice-asr-demo cat /tmp/gradio.log
```
## Troubleshooting
| Issue | Fix |
|-------|-----|
| `CUDA out of memory` | Use a different GPU (`device=X`) or reduce `--gpu-memory-utilization 0.7` in `start_server.py` |
| Gradio log is empty | Wait longer (~30s); Gradio buffers output. Use `PYTHONUNBUFFERED=1` as shown above |
| `Port already in use` | Pick a different port or stop the existing container: `docker stop <name> && docker rm <name>` |
| Share link shows "No interface" | Gradio is still loading. Wait for `Application startup complete` in the log |
| `tmux: command not found` | Run `docker exec <container> apt-get install -y tmux` first |
docs/vibevoice-asr.md
+134
View File
@@ -0,0 +1,134 @@
# VibeVoice-ASR
[![Hugging Face](https://img.shields.io/badge/HuggingFace-Collection-orange?logo=huggingface)](https://huggingface.co/microsoft/VibeVoice-ASR)
[![Live Playground](https://img.shields.io/badge/Live-Playground-green?logo=gradio)](https://aka.ms/vibevoice-asr)
**VibeVoice-ASR** is a unified speech-to-text model designed to handle **60-minute long-form audio** in a single pass, generating structured transcriptions containing **Who (Speaker), When (Timestamps), and What (Content)**, with support for **Customized Hotwords** and over **50 languages**.
**Model:** [VibeVoice-ASR-7B](https://huggingface.co/microsoft/VibeVoice-ASR)<br>
**Demo:** [VibeVoice-ASR-Demo](https://aka.ms/vibevoice-asr)<br>
**Report:** [VibeVoice-ASR-Report](https://arxiv.org/pdf/2601.18184)<br>
**Finetuning:** [finetune-guide](../finetuning-asr/README.md)<br>
**vLLM:** [vLLM-asr](./vibevoice-vllm-asr.md)<br>
**Transformers:** [VibeVoice-ASR-HF](https://huggingface.co/microsoft/VibeVoice-ASR-HF)<br>
## 🔥 Key Features
- **🕒 60-minute Single-Pass Processing**:
Unlike conventional ASR models that slice audio into short chunks (often losing global context), VibeVoice ASR accepts up to **60 minutes** of continuous audio input within 64K token length. This ensures consistent speaker tracking and semantic coherence across the entire hour.
- **👤 Customized Hotwords**:
Users can provide customized hotwords (e.g., specific names, technical terms, or background info) to guide the recognition process, significantly improving accuracy on domain-specific content.
- **📝 Rich Transcription (Who, When, What)**:
The model jointly performs ASR, diarization, and timestamping, producing a structured output that indicates *who* said *what* and *when*.
- **🌍 Multilingual & Code-Switching Support**:
It supports over 50 languages, requires no explicit language setting, and natively handles code-switching within and across utterances. See the [Language distribution](#language-distribution).
## 🏗️ Model Architecture
<p align="center">
<img src="../Figures/VibeVoice_ASR_archi.png" alt="VibeVoice ASR Architecture" width="80%">
</p>
# Demo
<div align="center" id="vibevoice-asr">
https://github.com/user-attachments/assets/acde5602-dc17-4314-9e3b-c630bc84aefa
</div>
## Evaluation
<p align="center">
<img src="../Figures/DER.jpg" alt="DER" width="50%"><br>
<img src="../Figures/cpWER.jpg" alt="cpWER" width="50%"><br>
<img src="../Figures/tcpWER.jpg" alt="tcpWER" width="50%">
</p>
## Installation
We recommend using NVIDIA Deep Learning Container to manage the CUDA environment.
1. Launch docker
```bash
# NVIDIA PyTorch Container 24.07 ~ 25.12 verified.
# Previous versions are also compatible.
sudo docker run --privileged --net=host --ipc=host --ulimit memlock=-1:-1 --ulimit stack=-1:-1 --gpus all --rm -it nvcr.io/nvidia/pytorch:25.12-py3
## If flash attention is not included in your docker environment, you need to install it manually
## Refer to https://github.com/Dao-AILab/flash-attention for installation instructions
# pip install flash-attn --no-build-isolation
```
2. Install from github
```bash
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice
pip install -e .
```
## Usages
### Usage 1: Launch Gradio demo
```bash
apt update && apt install ffmpeg -y # for demo
python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR --share
```
### Usage 2: Inference from files directly
```bash
python demo/vibevoice_asr_inference_from_file.py --model_path microsoft/VibeVoice-ASR --audio_files [add an audio path here]
```
## Finetuning
LoRA (Low-Rank Adaptation) fine-tuning is supported. See [Finetuning](../finetuning-asr/README.md) for detailed guide.
## Results
### Multilingual
| Dataset | Language | DER | cpWER | tcpWER | WER |
|----------------|-----------|------|-------|--------|------|
| MLC-Challenge | English | 4.28 | 11.48 | 13.02 | 7.99 |
| MLC-Challenge | French | 3.80 | 18.80 | 19.64 | 15.21 |
| MLC-Challenge | German | 1.04 | 17.10 | 17.26 | 16.30 |
| MLC-Challenge | Italian | 2.08 | 15.76 | 15.91 | 13.91 |
| MLC-Challenge | Japanese | 0.82 | 15.33 | 15.41 | 14.69 |
| MLC-Challenge | Korean | 4.52 | 15.35 | 16.07 | 9.65 |
| MLC-Challenge | Portuguese| 7.98 | 29.91 | 31.65 | 21.54 |
| MLC-Challenge | Russian | 0.90 | 12.94 | 12.98 | 12.40 |
| MLC-Challenge | Spanish | 2.67 | 10.51 | 11.71 | 8.04 |
| MLC-Challenge | Thai | 4.09 | 14.91 | 15.57 | 13.61 |
| MLC-Challenge | Vietnamese| 0.16 | 14.57 | 14.57 | 14.43 |
---
| Dataset | Language | DER | cpWER | tcpWER | WER |
|----------------|-----------|------|-------|--------|------|
| AISHELL-4 | Chinese | 6.77 | 24.99 | 25.35 | 21.40 |
| AMI-IHM | English | 11.92| 20.41 | 20.82 | 18.81 |
| AMI-SDM | English | 13.43| 28.82 | 29.80 | 24.65 |
| AliMeeting | Chinese | 10.92| 29.33 | 29.51 | 27.40 |
| MLC-Challenge | Average | 3.42 | 14.81 | 15.66 | 12.07|
## Language Distribution
<p align="center">
<img src="../Figures/language_distribution_horizontal.png" alt="Language Distribution" width="80%">
</p>
## 📄 License
This project is licensed under the [MIT License](../LICENSE).
docs/vibevoice-realtime-0.5b.md
+142
View File
@@ -0,0 +1,142 @@
<div align="center">
## 🎙️ VibeVoice-Realtime: Real-time Long‑Form Text‑to‑Speech with Streaming Input
[![Hugging Face](https://img.shields.io/badge/HuggingFace-Collection-orange?logo=huggingface)](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B)
[![Colab](https://img.shields.io/badge/Run-Colab-orange?logo=googlecolab)](https://colab.research.google.com/github/microsoft/VibeVoice/blob/main/demo/vibevoice_realtime_colab.ipynb)
</div>
VibeVoice-Realtime is a **lightweight real‑time** text-to-speech model supporting **streaming text input** and **robust long-form speech generation**. It can be used to build real-time TTS services, narrate live data streams, and let different LLMs start speaking from their very first tokens (plug in your preferred model) long before a full answer is generated. It produces initial audible speech in **~200 milliseconds** (hardware dependent).
**Model:** [VibeVoice Realtime 0.5B (on Hugging Face)](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B)<br>
**Colab:** [VibeVoice Realtime Colab (Jupyter Notebook)](https://colab.research.google.com/github/microsoft/VibeVoice/blob/main/demo/vibevoice_realtime_colab.ipynb)<br>
> Note (multilingual exploration): Although the model is primarily built for English, we found that it still exhibits a certain level of multilingual capability—and even performs reasonably well in some languages. We provide nine additional languages (German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, and Spanish) for users to explore. These multilingual behaviors have not been extensively tested; use with caution and share observations.
The model uses an interleaved, windowed design: it incrementally encodes incoming text chunks while, in parallel, continuing diffusion-based acoustic latent generation from prior context. Unlike the full multi-speaker long-form variants, this streaming model removes the semantic tokenizer and relies solely on an efficient acoustic tokenizer operating at an ultra-low frame rate (7.5 Hz).
<div align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="../Figures/VibeVoice_logo_white.png">
<img src="../Figures/VibeVoice_Realtime.png" alt="VibeVoice Realtime Overview" width="800" />
</picture>
<br>
<em>Overview of VibeVoice Realtime Model.</em>
</div>
Key features:
- Parameter size: 0.5B (deployment-friendly)
- Real-time TTS (~200 milliseconds first audible latency)
- Streaming text input
- Robust long-form speech generation
- 8k context window( ~10 minutes audio generation)
This real-time variant supports only a single speaker. For multi‑speaker conversational speech generation, please use other VibeVoice models (long‑form multi‑speaker variants). The model is currently intended for English speech only; other languages may produce unpredictable results.
To mitigate deepfake risks and ensure low latency for the first speech chunk, voice prompts are provided in an embedded format. For users requiring voice customization, please reach out to our team. We will also be expanding the range of available speakers.
### 📋 TODO
- [√] Add more voices (expand available speakers/voice timbres)
- [ ] Implement streaming text input function to feed new tokens while audio is still being generated
- [ ] Merge models into official HuggingFace's `transformers` repository
### 🎵 Demo Examples
<div align="center" id="generated-example-audio-vibevoice-realtime">
https://github.com/user-attachments/assets/9aa8ab3c-681d-4a02-b9ea-3f54ffd180b2
</div>
## Results
The model achieves satisfactory performance on short-sentence benchmarks, while the model is more focused on long‑form speech generation.
### Zero-shot TTS performance on LibriSpeech test-clean set
| Model | WER (%) ↓ | Speaker Similarity ↑ |
|:--------------------|:---------:|:----------------:|
| VALL-E 2 | 2.40 | 0.643 |
| Voicebox | 1.90 | 0.662 |
| MELLE | 2.10 | 0.625 |
| **VibeVoice-Realtime-0.5B** | 2.00 | 0.695 |
### Zero-shot TTS performance on SEED test-en set
| Model | WER (%) ↓ | Speaker Similarity ↑ |
|:--------------------|:---------:|:----------------:|
| MaskGCT | 2.62 | 0.714 |
| Seed-TTS | 2.25 | 0.762 |
| FireRedTTS | 3.82 | 0.460 |
| SparkTTS | 1.98 | 0.584 |
| CosyVoice2 | 2.57 | 0.652 |
| **VibeVoice-Realtime-0.5B** | 2.05 | 0.633 |
## Installation
We recommend using NVIDIA Deep Learning Container to manage the CUDA environment.
1. Launch docker
```bash
# NVIDIA PyTorch Container 24.07 / 24.10 / 24.12 verified.
# Later versions are also compatible.
sudo docker run --privileged --net=host --ipc=host --ulimit memlock=-1:-1 --ulimit stack=-1:-1 --gpus all --rm -it nvcr.io/nvidia/pytorch:24.07-py3
## If flash attention is not included in your docker environment, you need to install it manually
## Refer to https://github.com/Dao-AILab/flash-attention for installation instructions
# pip install flash-attn --no-build-isolation
```
2. Install from github
```bash
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice/
pip install -e .[streamingtts]
```
## Usages
### Usage 1: Launch real-time websocket demo
Note: NVIDIA T4 / Mac M4 Pro achieve real-time performance in our tests; other devices with weaker inference capability may require further testing and speed optimizations.
Due to network latency, the time when audio playback is heard may exceed the ~300 ms first speech chunk generation latency.
```bash
python demo/vibevoice_realtime_demo.py --model_path microsoft/VibeVoice-Realtime-0.5B
```
Tip: Just try it on [Colab](https://colab.research.google.com/github/microsoft/VibeVoice/blob/main/demo/vibevoice_realtime_colab.ipynb).
### Usage 2: Inference from files directly
```bash
# We provide some example scripts under demo/text_examples/ for demo
python demo/realtime_model_inference_from_file.py --model_path microsoft/VibeVoice-Realtime-0.5B --txt_path demo/text_examples/1p_vibevoice.txt --speaker_name Carter
```
### [Optional] More experimental voices
Download additional experimental multi-lingual speakers before launching demo or inference from files.
```bash
bash demo/download_experimental_voices.sh
```
## Risks and limitations
While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 0.5b in this release).
Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content.
English only: Transcripts in languages other than English may result in unexpected audio outputs.
Non-Speech Audio: The model focuses solely on speech synthesis and does not handle background noise, music, or other sound effects.
Code, formulas, and special symbols: The model does not currently support reading code, mathematical formulas, or uncommon symbols. Please pre‑process input text to remove or normalize such content to avoid unpredictable results.
Very short inputs: When the input text is extremely short (three words or fewer), the model’s stability may degrade.
We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.
docs/vibevoice-tts.md
+147
View File
@@ -0,0 +1,147 @@
# VibeVoice-TTS
[![Hugging Face](https://img.shields.io/badge/HuggingFace-Collection-orange?logo=huggingface)](https://huggingface.co/microsoft/VibeVoice-1.5B)
[![Technical Report](https://img.shields.io/badge/Technical-Report-red?logo=arxiv)](https://arxiv.org/pdf/2508.19205)
**VibeVoice-TTS** is a **long-form**, **multi-speaker** text-to-speech model designed to generate **expressive conversational audio** such as podcasts from text. It can synthesize speech up to **90 minutes** long with up to **4 distinct speakers**, surpassing the typical 1–2 speaker limits of many prior models.
**Model:** [VibeVoice-1.5B](https://huggingface.co/microsoft/VibeVoice-1.5B)<br>
**Report:** [Technical Report](https://arxiv.org/pdf/2508.19205)<br>
<div align="center">
| Model | Context Length | Generation Length | Weight |
|-------|----------------|-------------------|--------|
| VibeVoice-1.5B | 64K | ~90 min | [HF link](https://huggingface.co/microsoft/VibeVoice-1.5B) |
| VibeVoice-Large | 32K | ~45 min | Disabled |
</div>
## 🔥 Key Features
- **⏱️ 90-minute Long-form Generation**:
Synthesizes conversational/single-speaker speech up to **90 minutes** in a single pass, maintaining speaker consistency and semantic coherence throughout.
- **👥 Multi-speaker Support**:
Supports up to **4 distinct speakers** in a single conversation, with natural turn-taking and speaker consistency across long dialogues.
- **🎭 Expressive Speech**:
Generates expressive, natural-sounding speech that captures conversational dynamics and emotional nuances.
- **🌐 Multi-lingual Support**:
Supports English, Chinese and other languages.
## 🏗️ Model Architecture
VibeVoice-TTS employs a [next-token diffusion](https://arxiv.org/pdf/2508.19205) framework that combines:
- **Large Language Model (LLM)**: Based on Qwen2.5, understands textual context and dialogue flow
- **Continuous Speech Tokenizers**: Acoustic and Semantic tokenizers operating at an ultra-low frame rate of **7.5 Hz**, efficiently preserving audio fidelity while boosting computational efficiency
- **Diffusion Head**: Generates high-fidelity acoustic details through diffusion-based generation
<div align="center">
<img src="../Figures/VibeVoice.jpg" alt="VibeVoice Overview" width="80%">
</div>
## 🎵 Demo Examples
**English**
<div align="center">
https://github.com/user-attachments/assets/0967027c-141e-4909-bec8-091558b1b784
</div>
**Chinese**
<div align="center">
https://github.com/user-attachments/assets/322280b7-3093-4c67-86e3-10be4746c88f
</div>
**Cross-Lingual**
<div align="center">
https://github.com/user-attachments/assets/838d8ad9-a201-4dde-bb45-8cd3f59ce722
</div>
**Spontaneous Singing**
<div align="center">
https://github.com/user-attachments/assets/6f27a8a5-0c60-4f57-87f3-7dea2e11c730
</div>
**Long Conversation with 4 people**
<div align="center">
https://github.com/user-attachments/assets/a357c4b6-9768-495c-a576-1618f6275727
</div>
For more examples, see the [Project Page](https://microsoft.github.io/VibeVoice).
## Installation and Usage
Disabled due to widespread misuse.
## Results
The model achieves state-of-the-art performance on long-form multi-speaker speech generation tasks. For detailed evaluation results, please refer to the [paper](https://arxiv.org/pdf/2508.19205).
<div align="center">
<img src="../Figures/VibeVoice-TTS-results.jpg" alt="VibeVoice Results" width="80%">
</div>
## 🚨 Tips
We observed users may encounter occasional instability when synthesizing Chinese speech. We recommend:
- Using English punctuation even for Chinese text, preferably only commas and periods.
- Using the Large model variant, which is considerably more stable.
- If you find that the generated voice speaks too fast, try chunking your text into multiple turns with the same speaker label.
We'd like to thank [PsiPi](https://huggingface.co/PsiPi) for sharing an interesting way for emotion control. Details can be found via [discussion12](https://huggingface.co/microsoft/VibeVoice-1.5B/discussions/12).
## FAQ
#### Q1: Is this a pretrained model?
**A:** Yes, it's a pretrained model without any post-training or benchmark-specific optimizations. In a way, this makes VibeVoice very versatile and fun to use.
#### Q2: Randomly trigger Sounds / Music / BGM.
**A:** As you can see from our demo page, the background music or sounds are spontaneous. This means we can't directly control whether they are generated or not. The model is content-aware, and these sounds are triggered based on the input text and the chosen voice prompt.
Here are a few things we've noticed:
* If the voice prompt you use contains background music, the generated speech is more likely to have it as well. (The Large model is quite stable and effective at this—give it a try on the demo!)
* If the voice prompt is clean (no BGM), but the input text includes introductory words or phrases like "Welcome to," "Hello," or "However," background music might still appear.
* Speaker voice related, using "Alice" results in random BGM than others (fixed).
* In other scenarios, the Large model is more stable and has a lower probability of generating unexpected background music.
In fact, we intentionally decided not to denoise our training data because we think it's an interesting feature for BGM to show up at just the right moment. You can think of it as a little easter egg we left for you.
#### Q3: Text normalization?
**A:** We don't perform any text normalization during training or inference. Our philosophy is that a large language model should be able to handle complex user inputs on its own. However, due to the nature of the training data, you might still run into some corner cases.
#### Q4: Singing Capability.
**A:** Our training data **doesn't contain any music data**. The ability to sing is an emergent capability of the model (which is why it might sound off-key, even on a famous song like 'See You Again'). (The Large model is more likely to exhibit this than the 1.5B).
#### Q5: Some Chinese pronunciation errors.
**A:** The volume of Chinese data in our training set is significantly smaller than the English data. Additionally, certain special characters (e.g., Chinese quotation marks) may occasionally cause pronunciation issues.
#### Q6: Instability of cross-lingual transfer.
**A:** The model does exhibit strong cross-lingual transfer capabilities, including the preservation of accents, but its performance can be unstable. This is an emergent ability of the model that we have not specifically optimized. It's possible that a satisfactory result can be achieved through repeated sampling.
## Risks and Limitations
While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content.
English and Chinese only: Transcripts in languages other than English or Chinese may result in unexpected audio outputs.
Non-Speech Audio: The model focuses solely on speech synthesis and does not handle background noise, music, or other sound effects.
Overlapping Speech: The current model does not explicitly model or generate overlapping speech segments in conversations.
We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.
docs/vibevoice-vllm-asr.md
+189
View File
@@ -0,0 +1,189 @@
# VibeVoice vLLM ASR Deployment
<a href="https://huggingface.co/microsoft/VibeVoice-ASR"><img alt="Huggingface" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-VibeVoice--ASR-blue"></a>
Deploy VibeVoice ASR model as a high-performance API service using [vLLM](https://github.com/vllm-project/vllm). This plugin provides OpenAI-compatible API endpoints for speech-to-text transcription with streaming support.
## 🔥 Key Features
- **🚀 High-Performance Serving**: Optimized for high-throughput ASR inference with vLLM's continuous batching
- **📡 OpenAI-Compatible API**: Standard `/v1/chat/completions` endpoint with streaming support
- **🎵 Long Audio Support**: Process up to 60+ minutes of audio in a single request
- **🔌 Plugin Architecture**: No vLLM source code modification required - just install and run
- **⚡ Data Parallel (DP)**: Run independent model replicas across multiple GPUs with automatic load balancing behind a single port
## 🛠️ Installation
Using Official vLLM Docker Image (Recommended)
1. Clone the repository
```bash
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice
```
2. Launch the server (background mode)
```bash
docker run -d --gpus all --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py"
```
## ⚡ Multi-GPU Deployment
The launcher supports two types of GPU parallelism via `--tp` and `--dp` flags:
| Flag | Name | What it does |
|------|------|-------------|
| `--tp N` | Tensor Parallel | Splits **one model** across N GPUs (for models too large for a single GPU) |
| `--dp N` | Data Parallel | Runs **N independent replicas**, one per GPU, with automatic load balancing behind a single port |
### Data Parallel (Recommended for scaling throughput)
Run N independent replicas on N GPUs with automatic load balancing behind a single port.
When `--dp N` is specified (N > 1), the launcher automatically starts N independent vLLM
processes behind an nginx reverse proxy (2×N workers) for optimal throughput:
```bash
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 4"
```
Run on all 8 GPUs:
```bash
docker run -d --gpus all --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 8"
```
### Tensor Parallel
Split a single model across 2 GPUs (useful if GPU memory is limited):
```bash
docker run -d --gpus '"device=0,1"' --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --tp 2"
```
### Hybrid (DP × TP)
Combine both — e.g., 2 replicas, each split across 2 GPUs (4 GPUs total):
```bash
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 2 --tp 2"
```
> **Note**: Total GPUs required = `dp × tp`. Make sure to expose enough GPU devices in the Docker `--gpus` flag.
3. View logs
```bash
docker logs -f vibevoice-vllm
```
> **Note**:
> - The `-d` flag runs the container in background (detached mode)
> - Use `docker stop vibevoice-vllm` to stop the service
> - The model will be downloaded to HuggingFace cache (`~/.cache/huggingface`) inside the container
## 🚀 Usages
### Test the API
Once the vLLM server is running, test it with the provided script:
```bash
# Basic transcription
docker exec -it vibevoice-vllm python3 vllm_plugin/tests/test_api.py /app/audio.wav
# With hotwords for better recognition of specific terms
docker exec -it vibevoice-vllm python3 vllm_plugin/tests/test_api.py /app/audio.wav --hotwords "Microsoft,VibeVoice"
```
```bash
# With auto-recovery from repetition loops (for long audio)
docker exec -it vibevoice-vllm python3 vllm_plugin/tests/test_api_auto_recover.py /app/audio.wav
# Auto-recover with hotwords
docker exec -it vibevoice-vllm python3 vllm_plugin/tests/test_api_auto_recover.py /app/audio.wav --hotwords "Microsoft,VibeVoice"
```
> **Note**:
> - The audio/video file must be inside the mounted directory (`/app` in the container). Copy your files to the VibeVoice folder before testing.
> - Hotwords help improve recognition of domain-specific terms like proper nouns, technical terms, and speaker names.
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` | Maximum FFmpeg processes for audio decoding | `64` |
| `PYTORCH_ALLOC_CONF` | PyTorch memory allocator config | `expandable_segments:True` |
## 📊 Performance Tips
1. **GPU Memory**: Use `--gpu-memory-utilization 0.9` for maximum throughput if you have dedicated GPU
2. **Batch Size**: Increase `--max-num-seqs` for higher concurrency (requires more GPU memory)
3. **FFmpeg Concurrency**: Tune `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` based on CPU cores
## 🚨 Troubleshooting
### Common Issues
1. **"CUDA out of memory"**
- Reduce `--gpu-memory-utilization`
- Reduce `--max-num-seqs`
- Use smaller `--max-model-len`
2. **"Audio decoding failed"**
- Ensure FFmpeg is installed: `ffmpeg -version`
- Check audio file format is supported
3. **"Model not found"**
- Ensure model path contains `config.json` and model weights
- Generate tokenizer files if missing
4. **"Plugin not loaded"**
- Verify installation: `pip show vibevoice`
- Check entry point: `pip show -f vibevoice | grep entry`