From 7717cb18ca7447a32d50619f126d46354d271af4 Mon Sep 17 00:00:00 2001 From: wehub-resource-sync Date: Mon, 13 Jul 2026 10:27:39 +0000 Subject: [PATCH] docs: make Chinese README the default --- README.md | 703 +++++++++++++++++++++++++++--------------------------- 1 file changed, 351 insertions(+), 352 deletions(-) diff --git a/README.md b/README.md index 8b20199..fb2b750 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,23 @@ -

VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning

+ +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/OpenBMB/VoxCPM) · [上游 README](https://github.com/OpenBMB/VoxCPM/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 + +

VoxCPM2：基于连续表征的多语言语音合成、创意音色设计与高保真声音克隆

- English | 中文 + English | 中文

Project Page Live Playground - Documentation + Documentation Hugging Face ModelScope DemoPage - VoxCPM2 Technical Report +

@@ -23,10 +29,10 @@

- 👋 Join our community for discussion and support! + 👋 欢迎加入社区，参与讨论与交流！
- Feishu + 飞书群  |  @@ -34,67 +40,69 @@

-VoxCPM is a **tokenizer-free** Text-to-Speech system that directly generates continuous speech representations via an end-to-end **diffusion autoregressive architecture**, bypassing discrete tokenization to achieve highly natural and expressive synthesis. +VoxCPM 是一个**无离散音频分词器**（Tokenizer-Free）的语音合成系统，通过端到端的**扩散自回归架构**直接生成连续语音表征，绕过对音频的离散编码步骤，实现高度自然且富有表现力的语音合成。 -**VoxCPM2** is the latest major release — a **2B** parameter model trained on **over 2 million hours** of multilingual speech data, now supporting **30 languages**, **Voice Design**, **Controllable Voice Cloning**, and **48kHz** studio-quality audio output. Built on a [MiniCPM-4](https://github.com/OpenBMB/MiniCPM) backbone. +**VoxCPM2** 是最新的版本 — 基于 [MiniCPM-4](https://github.com/OpenBMB/MiniCPM) 基座构建，总计 **20亿** 参数，在超过 **200万小时** 的多语种音频数据上训练，支持 **30种全球语言+9种中文方言**、**音色设计**、**可控声音克隆**，原生输出 **48kHz** 高质量音频。 -### ✨ Highlights +### ✨ 核心特性 -- 🌍 **30-Language Multilingual** — Input text in any of the 30 supported languages and synthesize directly, no language tag needed -- 🎨 **Voice Design** — Create a brand-new voice from a natural-language description alone (gender, age, tone, emotion, pace …), no reference audio required -- 🎛️ **Controllable Cloning** — Clone any voice from a short reference clip, with optional style guidance to steer emotion, pace, and expression while preserving the original timbre -- 🎙️ **Ultimate Cloning** — Reproduce every vocal nuance: provide both reference audio and its transcript, and the model continues seamlessly from the reference, faithfully preserving every vocal detail — timbre, rhythm, emotion, and style (same as VoxCPM1.5) -- 🔊 **48kHz High-Quality Audio** — Accepts 16kHz reference audio and directly outputs 48kHz studio-quality audio via AudioVAE V2's asymmetric encode/decode design, with built-in super-resolution — no external upsampler needed -- 🧠 **Context-Aware Synthesis** — Automatically infers appropriate prosody and expressiveness from text content -- ⚡ **Real-Time Streaming** — RTF as low as ~0.3 on NVIDIA RTX 4090, and ~0.13 accelerated by [Nano-vLLM](https://github.com/a710128/nanovllm-voxcpm) or [vLLM-Omni](https://github.com/vllm-project/vllm-omni) — official vLLM omni-modal serving for VoxCPM2 with PagedAttention and an OpenAI-compatible API -- 📜 **Fully Open-Source & Commercial-Ready** — Weights and code released under the [Apache-2.0](LICENSE) license, free for commercial use +- 🌍 **30种语言语音合成** — 直接输入原始文本即可合成（支持语言详见下文），无需额外语言标签 +- 🎨 **音色设计** — 用自然语言描述（性别、年龄、音色、情绪、语速……）凭空创建全新音色，无需参考音频 +- 🎛️ **可控声音克隆** — 从参考音频片段克隆任意声音，可叠加风格指令控制情绪、语速和表现力，同时保持原始音色 +- 🎙️ **极致克隆** — 提供参考音频及其文本内容，模型接着参考音频进行无缝续写，从而精准还原声音细节特征（与 VoxCPM1.5 一致） +- 🔊 **48kHz 高质量音频** — 输入 16kHz 参考音频，通过 AudioVAE V2 的非对称编解码设计直接输出 48kHz 高质量音频，内置超分能力 +- 🧠 **语境感知合成** — 根据文本内容自动推断合适的韵律和表现力 +- ⚡ **实时流式合成** — 在 NVIDIA RTX 4090 上 RTF 低至 ~0.3，通过 [Nano-vLLM](https://github.com/a710128/nanovllm-voxcpm) 或 [vLLM-Omni](https://github.com/vllm-project/vllm-omni)（官方 vLLM 全模态服务，原生支持 VoxCPM2，提供 PagedAttention 与 OpenAI 兼容 API）加速后可达 ~0.13 +- 📜 **完全开源，商用就绪** — 权重和代码基于 [Apache-2.0](LICENSE) 协议发布，免费商用 -**🌍 Supported Languages (30)** -Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, Vietnamese +🌍 支持的语言（30种） +
+阿拉伯语、缅甸语、中文、丹麦语、荷兰语、英语、芬兰语、法语、德语、希腊语、希伯来语、印地语、印尼语、意大利语、日语、高棉语、韩语、老挝语、马来语、挪威语、波兰语、葡萄牙语、俄语、西班牙语、斯瓦希里语、瑞典语、菲律宾语、泰语、土耳其语、越南语 -Chinese Dialect: 四川话, 粤语, 吴语, 东北话, 河南话, 陕西话, 山东话, 天津话, 闽南话 +中国方言：四川话、粤语、吴语、东北话、河南话、陕西话、山东话、天津话、闽南话 -### News -- **[2026.04]** 🔥 We release **VoxCPM2** — 2B, 30 languages, Voice Design & Controllable Voice Cloning, 48kHz audio output! [Weights](https://huggingface.co/openbmb/VoxCPM2) | [Docs](https://voxcpm.readthedocs.io/en/latest/) | [Playground](https://huggingface.co/spaces/OpenBMB/VoxCPM-Demo) | [Technical Report](https://arxiv.org/abs/2606.06928) -- **[2025.12]** 🎉 Open-source **VoxCPM1.5** [weights](https://huggingface.co/openbmb/VoxCPM1.5) with SFT & LoRA fine-tuning. (**🏆 #1 GitHub Trending**) -- **[2025.09]** 🔥 Release VoxCPM [Technical Report](https://arxiv.org/abs/2509.24650). -- **[2025.09]** 🎉 Open-source **VoxCPM-0.5B** [weights](https://huggingface.co/openbmb/VoxCPM-0.5B) (**🏆 #1 HuggingFace Trending**) +### 最新动态 + +* **[2026.04]** 🔥 发布 **VoxCPM2** — 20亿参数，30种语言，音色设计与可控声音克隆，48kHz 音频输出！[模型权重](https://huggingface.co/openbmb/VoxCPM2) | [使用文档](https://voxcpm.readthedocs.io/zh-cn/latest/) | [在线体验](https://huggingface.co/spaces/OpenBMB/VoxCPM-Demo) | [官网体验](https://voxcpm.modelbest.cn/) (适用国内访问) | [技术报告](https://arxiv.org/abs/2606.06928) +* **[2025.12]** 🎉 开源 **VoxCPM1.5** [模型权重](https://huggingface.co/openbmb/VoxCPM1.5)，支持 SFT 和 LoRA 微调。(**🏆 GitHub Trending #1**) +* **[2025.09]** 🔥 发布 VoxCPM [技术报告](https://arxiv.org/abs/2509.24650)。 +* **[2025.09]** 🎉 开源 **VoxCPM-0.5B** [模型权重](https://huggingface.co/openbmb/VoxCPM-0.5B) (**🏆 HuggingFace Trending #1**) --- -## Contents +## 目录 -- [Quick Start](#-quick-start) - - [Installation](#installation) +- [快速开始](#-快速开始) + - [安装](#安装) - [Python API](#python-api) - - [CLI Usage](#cli-usage) + - [命令行使用](#命令行使用) - [Web Demo](#web-demo) - - [Production Deployment](#-production-deployment-nano-vllm) - - [On-Device Inference (llama.cpp-omni)](#-on-device-inference-llamacpp-omni) -- [Models & Versions](#-models--versions) -- [Performance](#-performance) -- [Fine-tuning](#%EF%B8%8F-fine-tuning) -- [Documentation](#-documentation) -- [Ecosystem & Community](#-ecosystem--community) -- [Risks and Limitations](#%EF%B8%8F-risks-and-limitations) -- [Citation](#-citation) + - [生产部署](#-生产部署nano-vllm) + - [端侧推理（llama.cpp-omni）](#-端侧推理llamacpp-omni) +- [模型与版本](#-模型与版本) +- [性能评测](#-性能评测) +- [微调](#%EF%B8%8F-微调) +- [文档](#-文档) +- [生态与社区](#-生态与社区) +- [风险与局限性](#%EF%B8%8F-风险与局限性) +- [引用](#-引用) --- -## 🚀 Quick Start +## 🚀 快速开始 -### Installation +### 安装 ```sh pip install voxcpm ``` -> **Requirements:** Python ≥ 3.10 (<3.13), PyTorch ≥ 2.5.0, CUDA ≥ 12.0. See [Quick Start Docs](https://voxcpm.readthedocs.io/en/latest/quickstart.html) for details. +> **环境要求：** Python ≥ 3.10 (<3.13)，PyTorch ≥ 2.5.0，CUDA ≥ 12.0。详见 [快速开始文档](https://voxcpm.readthedocs.io/zh-cn/latest/quickstart.html)。 ### Python API -#### 🗣️ Text-to-Speech +#### 🗣️ 文本转语音 ```python from voxcpm import VoxCPM @@ -106,16 +114,16 @@ model = VoxCPM.from_pretrained( ) wav = model.generate( - text="VoxCPM2 is the current recommended release for realistic multilingual speech synthesis.", + text="VoxCPM2 是目前推荐使用的多语言语音合成版本。", cfg_value=2.0, inference_timesteps=10, seed=42, ) sf.write("demo.wav", wav, model.tts_model.sample_rate) -print("saved: demo.wav") +print("已保存: demo.wav") ``` -If you prefer downloading from ModelScope first, you can use: +如果你希望先从 ModelScope 下载模型到本地（适用于国内网络访问），可以使用： ```bash pip install modelscope @@ -123,14 +131,14 @@ pip install modelscope ```python from modelscope import snapshot_download -snapshot_download("OpenBMB/VoxCPM2", local_dir='./pretrained_models/VoxCPM2') # specify the local directory to save the model +snapshot_download("OpenBMB/VoxCPM2", local_dir='./pretrained_models/VoxCPM2') # 指定模型保存的本地路径 from voxcpm import VoxCPM import soundfile as sf -model = VoxCPM.from_pretrained("./pretrained_models/VoxCPM2", load_denoiser=False) +model = VoxCPM.from_pretrained('./pretrained_models/VoxCPM2', load_denoiser=False) wav = model.generate( - text="VoxCPM2 is the current recommended release for realistic multilingual speech synthesis.", + text="VoxCPM2 是目前推荐使用的多语言语音合成版本。", cfg_value=2.0, inference_timesteps=10, seed=42, @@ -138,13 +146,13 @@ wav = model.generate( sf.write("demo.wav", wav, model.tts_model.sample_rate) ``` -#### 🎨 Voice Design +#### 🎨 音色设计 -Create a voice from a natural-language description — no reference audio needed. **Format:** put the description in parentheses at the start of `text`(e.g. `"(your voice description)The text to synthesize."`): +用自然语言描述创建全新音色，无需参考音频。**格式：** 在 `text` 开头用括号写入音色描述（如 `"(音色描述)要合成的文本。"`）： ```python wav = model.generate( - text="(A young woman, gentle and sweet voice)Hello, welcome to VoxCPM2!", + text="(年轻女性，声音温柔甜美)你好，欢迎使用VoxCPM2！", cfg_value=2.0, inference_timesteps=10, seed=42, @@ -152,19 +160,19 @@ wav = model.generate( sf.write("voice_design.wav", wav, model.tts_model.sample_rate) ``` -#### 🎛️ Controllable Voice Cloning +#### 🎛️ 可控声音克隆 -Upload a reference audio. The model clones the timbre, and you can still use control instructions to adjust speed, emotion, or style. +上传一段参考音频，模型克隆其音色，同时可以使用控制指令调节语速、情绪或风格。 ```python wav = model.generate( - text="This is a cloned voice generated by VoxCPM2.", + text="这是VoxCPM2生成的克隆语音。", reference_wav_path="path/to/voice.wav", ) sf.write("clone.wav", wav, model.tts_model.sample_rate) wav = model.generate( - text="(slightly faster, cheerful tone)This is a cloned voice with style control.", + text="(稍快一点，欢快的语气)这是带风格控制的克隆语音。", reference_wav_path="path/to/voice.wav", cfg_value=2.0, inference_timesteps=10, @@ -173,78 +181,78 @@ wav = model.generate( sf.write("controllable_clone.wav", wav, model.tts_model.sample_rate) ``` -#### 🎙️ Ultimate Cloning +#### 🎙️ 极致克隆 -Provide both the reference audio and its exact transcript for audio-continuation-based cloning with every vocal nuance reproduced. For maximum cloning similarity, pass the same reference clip to both `reference_wav_path` and `prompt_wav_path` as shown below: +提供参考音频及其精确文本转录，实现基于音频续写的高保真克隆。为获得最高克隆相似度，可将同一音频同时传给 `reference_wav_path` 和 `prompt_wav_path`： ```python wav = model.generate( - text="This is an ultimate cloning demonstration using VoxCPM2.", + text="这是使用VoxCPM2的极致克隆演示。", prompt_wav_path="path/to/voice.wav", - prompt_text="The transcript of the reference audio.", - reference_wav_path="path/to/voice.wav", # optional, for better simliarity + prompt_text="参考音频的文本转录。", + reference_wav_path="path/to/voice.wav", # 可选，提升相似度 ) sf.write("hifi_clone.wav", wav, model.tts_model.sample_rate) ``` -**🔄 Streaming API** +
+🔄 流式 API ```python import numpy as np chunks = [] for chunk in model.generate_streaming( - text="Streaming text to speech is easy with VoxCPM!", + text="使用VoxCPM进行流式语音合成非常简单！", ): chunks.append(chunk) wav = np.concatenate(chunks) sf.write("streaming.wav", wav, model.tts_model.sample_rate) ``` +
- - -### CLI Usage +### 命令行使用 ```bash -# Voice design (no reference audio needed) +# 音色设计（无需参考音频） voxcpm design \ - --text "VoxCPM2 brings studio-quality multilingual speech synthesis." \ + --text "VoxCPM2带来全新语音合成体验。" \ --output out.wav -# Controllable voice cloning with style control +# 可控声音克隆（带风格控制） voxcpm design \ - --text "VoxCPM2 brings studio-quality multilingual speech synthesis." \ - --control "Young female voice, warm and gentle, slightly smiling" \ + --text "VoxCPM2带来全新语音合成体验。" \ + --control "年轻女声，温暖温柔，略带微笑" \ --seed 42 \ --output out.wav -# Voice cloning (reference audio) +# 声音克隆（参考音频） voxcpm clone \ - --text "This is a voice cloning demo." \ + --text "这是一个声音克隆的演示。" \ --reference-audio path/to/voice.wav \ --output out.wav -# Ultimate cloning (prompt audio + transcript) +# 极致克隆（提示音频 + 转录文本） voxcpm clone \ - --text "This is a voice cloning demo." \ + --text "这是一个声音克隆的演示。" \ --prompt-audio path/to/voice.wav \ - --prompt-text "reference transcript" \ - --reference-audio path/to/voice.wav \ # optional, for better simliarity + --prompt-text "参考音频转录文本" \ + --reference-audio path/to/voice.wav \ --output out.wav -# Batch processing +# 批量处理 voxcpm batch --input examples/input.txt --output-dir outs -# Optional post-generation timestamps with stable-ts +# 可选的生成后时间戳对齐（基于 stable-ts） pip install "voxcpm[timestamps]" voxcpm design \ - --text "VoxCPM2 brings studio-quality multilingual speech synthesis." \ + --text "VoxCPM2带来全新语音合成体验。" \ --output out.wav \ --timestamps \ --timestamp-level word \ - --timestamp-language en + --timestamp-language zh -# Character timestamps are best-effort and are derived from word alignment +# 字级时间戳是 best-effort，会基于词级对齐结果拆分 voxcpm design \ --text "欢迎使用 VoxCPM2。" \ --output out.wav \ @@ -252,27 +260,27 @@ voxcpm design \ --timestamp-level char \ --timestamp-language zh -# Help +# 帮助 voxcpm --help ``` ### Web Demo ```bash -python app.py --port 8808 # then open in browser: http://localhost:8808 +python app.py --port 8808 # 然后在浏览器打开 http://localhost:8808 ``` -Use `--device` to choose the runtime device: +使用 `--device` 选择运行设备： ```bash python app.py --device auto ``` -Supported values are `auto`, `cpu`, `mps`, `cuda`, and `cuda:N`. On Apple Silicon Macs, `auto` uses MPS when available. +支持的取值包括 `auto`、`cpu`、`mps`、`cuda` 和 `cuda:N`。在 Apple Silicon Mac 上，`auto` 会在可用时使用 MPS。 -### 🚢 Production Deployment (Nano-vLLM) +### 🚢 生产部署（Nano-vLLM） -For high-throughput serving, use **[Nano-vLLM-VoxCPM](https://github.com/a710128/nanovllm-voxcpm)** — a dedicated inference engine built on Nano-vLLM with concurrent request support and an async API. +如需高吞吐量部署，使用 [**Nano-vLLM-VoxCPM**](https://github.com/a710128/nanovllm-voxcpm) — 基于 Nano-vLLM 构建的专用推理引擎，支持并发请求和异步 API。 ```bash pip install nano-vllm-voxcpm @@ -283,46 +291,46 @@ from nanovllm_voxcpm import VoxCPM import numpy as np, soundfile as sf server = VoxCPM.from_pretrained(model="/path/to/VoxCPM", devices=[0]) -chunks = list(server.generate(target_text="Hello from VoxCPM!")) +chunks = list(server.generate(target_text="你好，我来自VoxCPM！")) sf.write("out.wav", np.concatenate(chunks), 48000) server.stop() ``` -> **RTF as low as ~0.13 on NVIDIA RTX 4090** (vs ~0.3 with the standard PyTorch implementation), with support for batched concurrent requests and a FastAPI HTTP server. See the [Nano-vLLM-VoxCPM repo](https://github.com/a710128/nanovllm-voxcpm) for deployment details. +> **在 NVIDIA RTX 4090 上 RTF 低至 ~0.13**（标准 PyTorch 实现约 ~0.3），支持批量并发请求和 FastAPI HTTP 服务。详见 [Nano-vLLM-VoxCPM 仓库](https://github.com/a710128/nanovllm-voxcpm)。 -### 🏭 Production Serving (vLLM-Omni) +### 🏭 生产环境部署（vLLM-Omni） -For production multi-tenant deployments, use **[vLLM-Omni](https://github.com/vllm-project/vllm-omni)** — the official vLLM project's omni-modal extension with native **VoxCPM2** support. PagedAttention KV cache, continuous batching, and a drop-in **OpenAI-compatible** `/v1/audio/speech` endpoint. +如需生产级多租户部署，使用 [**vLLM-Omni**](https://github.com/vllm-project/vllm-omni) — 官方 vLLM 项目的全模态扩展，原生支持 **VoxCPM2**。具备 PagedAttention KV 缓存、连续批处理，以及与 OpenAI 完全兼容的 `/v1/audio/speech` 接口。 ```bash -# Install from source (latest main — vllm-omni is rapidly evolving) +# 从源码安装（最新 main 分支 —— vllm-omni 正在快速迭代） uv pip install vllm==0.19.0 --torch-backend=auto git clone https://github.com/vllm-project/vllm-omni.git && cd vllm-omni uv pip install -e . ``` -See the [vLLM-Omni installation guide](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/) for other platforms (ROCm, XPU, MUSA, NPU) and Docker images. +其他平台（ROCm、XPU、MUSA、NPU）与 Docker 镜像请参考 [vLLM-Omni 安装文档](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/)。 ```bash -# Launch an OpenAI-compatible TTS server (--omni enables omni-modal serving) +# 启动 OpenAI 兼容的 TTS 服务（--omni 启用全模态服务） vllm serve openbmb/VoxCPM2 --omni --port 8000 -# Call it from any OpenAI client +# 任意 OpenAI 客户端均可调用 curl http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ - -d '{"model":"openbmb/VoxCPM2","input":"Hello from VoxCPM2 on vLLM-Omni!","voice":"default"}' \ + -d '{"model":"openbmb/VoxCPM2","input":"你好，欢迎使用 VoxCPM2 on vLLM-Omni！","voice":"default"}' \ --output out.wav ``` -> Built on the upstream vLLM scheduler, with batched concurrent requests, streaming chunk delivery, and multi-GPU deployment out of the box. See the [VoxCPM2 example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm2) for full deployment recipes. +> 基于上游 vLLM 调度器构建，开箱即用支持批量并发、流式分块输出和多 GPU 部署。完整示例见 [VoxCPM2 部署样例](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm2)。 -### 📱 On-Device Inference (llama.cpp-omni) +### 📱 端侧推理（llama.cpp-omni） -For on-device / edge deployment without Python, use **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** — a high-performance C++ inference engine built on llama.cpp, with native VoxCPM2 GGUF support on **CPU / Metal / CUDA / Vulkan**. +如需在端侧/消费级硬件上无 Python 运行，使用 **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** — 基于 llama.cpp 的高性能 C++ 推理引擎，原生支持 VoxCPM2 GGUF，可在 **CPU / Metal / CUDA / Vulkan** 上运行。 -**1. Download GGUF weights** from [HuggingFace](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) | [ModelScope](https://modelscope.cn/models/DennisHuang/VoxCPM2-GGUF) — you need one **BaseLM** (F16 or Q8_0) + the **Acoustic** file. Q8_0 halves the download with negligible quality loss. +**1. 下载 GGUF 权重**：从 [HF下载](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) | [ModelScope](https://modelscope.cn/models/DennisHuang/VoxCPM2-GGUF)，需要一个 **BaseLM**（F16 或 Q8_0）+ **Acoustic** 文件。Q8_0 体积减半，质量损失可忽略。 -**2. Build** +**2. 编译** ```bash git clone https://github.com/tc-mb/llama.cpp-omni.git && cd llama.cpp-omni @@ -330,330 +338,317 @@ cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --target voxcpm2-cli -j ``` -> CMake auto-detects Metal (macOS) or CUDA (Linux with NVIDIA GPU). +> CMake 会自动检测并启用 Metal（macOS）或 CUDA（Linux + NVIDIA GPU）。 -**3. Run** +**3. 运行** ```bash -# Basic TTS +# 基础 TTS ./build/bin/voxcpm2-cli \ - -t "Hello, this is VoxCPM2 running through llama.cpp-omni." \ + -t "你好，我是通过 llama.cpp-omni 运行的 VoxCPM2。" \ -o output.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf -# Voice cloning (reference audio) +# 声音克隆（参考音频） ./build/bin/voxcpm2-cli \ - -t "Cloned voice." -r speaker.wav -o clone.wav \ + -t "克隆的声音。" -r speaker.wav -o clone.wav \ VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf -# Ultimate cloning (reference audio + transcript) +# 精准克隆（参考音频 + 转写文本） ./build/bin/voxcpm2-cli \ - -t "Target text." --prompt-wav speaker.wav --prompt-text "transcript of speaker.wav" \ + -t "目标文本。" --prompt-wav speaker.wav --prompt-text "参考音频的转写文本" \ -o clone.wav VoxCPM2-BaseLM-Q8_0.gguf VoxCPM2-Acoustic-F16.gguf ``` -> **RTF ~1.76 (Q8_0) on Apple M4 Pro / Metal.** Key flags: `--cfg` (guidance scale), `--timesteps` (CFM steps), `--seed`, `--temperature`, `--stream`. See the [llama.cpp-omni repo](https://github.com/tc-mb/llama.cpp-omni) and [GGUF weights page](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) for full details. +> **在 Apple M4 Pro / Metal 上 RTF ~1.76（Q8_0）。** 主要参数：`--cfg`（引导尺度）、`--timesteps`（CFM 步数）、`--seed`、`--temperature`、`--stream`。详见 [llama.cpp-omni 仓库](https://github.com/tc-mb/llama.cpp-omni) 和 [GGUF 权重页面](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF)。 -> **Full parameter reference, multi-scenario examples, and voice cloning tips →** [Quick Start Guide](https://voxcpm.readthedocs.io/en/latest/quickstart.html) | [Usage Guide](https://voxcpm.readthedocs.io/en/latest/usage_guide.html) | [Cookbook](https://voxcpm.readthedocs.io/en/latest/cookbook.html) +> **完整参数说明、多场景示例与声音克隆技巧 →** [快速开始指南](https://voxcpm.readthedocs.io/zh-cn/latest/quickstart.html) | [使用指南](https://voxcpm.readthedocs.io/zh-cn/latest/usage_guide.html) | [Cookbook](https://voxcpm.readthedocs.io/zh-cn/latest/cookbook.html) --- -## 📦 Models & Versions +## 📦 模型与版本 +| | **VoxCPM2** | **VoxCPM1.5** | **VoxCPM-0.5B** | +|---|:---:|:---:|:---:| +| **状态** | 🟢 最新版本 | 稳定版 | 旧版 | +| **主模型参数量** | 2B | 0.6B | 0.5B | +| **音频采样率** | 48kHz | 44.1kHz | 16kHz | +| **LM处理码率** | 6.25Hz | 6.25Hz | 12.5Hz | +| **语言支持数量** | 30 | 2（中文、英文） | 2（中文、英文） | +| **克隆模式** | 隔离参考音频（无需文本） & 音频续写 | 仅音频续写 | 仅音频续写 | +| **音色设计** | ✅ | — | — | +| **可控声音克隆** | ✅ | — | — | +| **SFT / LoRA** | ✅ | ✅ | ✅ | +| **RTF (RTX 4090)** | ~0.30 | ~0.15 | ~0.17 | +| **RTF Nano-VLLM (RTX 4090)** | ~0.13 | ~0.08 | ~0.10 | +| **显存占用** | ~8 GB | ~6 GB | ~5 GB | +| **模型权重** | [🤗 HF](https://huggingface.co/openbmb/VoxCPM2) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM2) | [🤗 HF](https://huggingface.co/openbmb/VoxCPM1.5) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM1.5) | [🤗 HF](https://huggingface.co/openbmb/VoxCPM-0.5B) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM-0.5B) | +| **技术报告** | [arXiv](https://arxiv.org/abs/2606.06928) | — | [arXiv](https://arxiv.org/abs/2509.24650) [ICLR 2026](https://openreview.net/forum?id=h5KLpGoqzC) | +| **Demo 页面** | [音频示例](https://openbmb.github.io/voxcpm2-demopage) | — | [音频示例](https://openbmb.github.io/VoxCPM-demopage) | -| | **VoxCPM2** | **VoxCPM1.5** | **VoxCPM-0.5B** | -| ------------------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -| **Status** | 🟢 Latest | Stable | Legacy | -| **Backbone Parameters** | 2B | 0.6B | 0.5B | -| **Audio Sample Rate** | 48kHz | 44.1kHz | 16kHz | -| **LM Token Rate** | 6.25Hz | 6.25Hz | 12.5Hz | -| **Languages** | 30 | 2 (zh, en) | 2 (zh, en) | -| **Cloning Mode** | Isolated Reference & Continuation | Continuation only | Continuation only | -| **Voice Design** | ✅ | — | — | -| **Controllable Voice Cloning** | ✅ | — | — | -| **SFT / LoRA** | ✅ | ✅ | ✅ | -| **RTF (RTX 4090)** | ~0.30 | ~0.15 | ~0.17 | -| **RTF in Nano-VLLM (RTX 4090)** | ~0.13 | ~0.08 | ~0.10 | -| **VRAM** | ~8 GB | ~6 GB | ~5 GB | -| **Weights** | [🤗 HF](https://huggingface.co/openbmb/VoxCPM2) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM2) | [🤗 HF](https://huggingface.co/openbmb/VoxCPM1.5) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM1.5) | [🤗 HF](https://huggingface.co/openbmb/VoxCPM-0.5B) / [MS](https://modelscope.cn/models/OpenBMB/VoxCPM-0.5B) | -| **Technical Report** | [arXiv](https://arxiv.org/abs/2606.06928) | — | [arXiv](https://arxiv.org/abs/2509.24650) [ICLR 2026](https://openreview.net/forum?id=h5KLpGoqzC) | -| **Demo Page** | [Audio Samples](https://openbmb.github.io/voxcpm2-demopage) | — | [Audio Samples](https://openbmb.github.io/VoxCPM-demopage) | - - -VoxCPM2 is built on a **tokenizer-free, diffusion autoregressive** paradigm. The model operates entirely in the latent space of **AudioVAE V2**, following a four-stage pipeline: **LocEnc → TSLM → RALM → LocDiT**, enabling rich expressiveness and 48kHz native audio output. +VoxCPM2 采用**连续音频表征、扩散自回归**范式，模型在 **AudioVAE** 的连续隐空间中通过四阶段处理：**LocEnc → TSLM → RALM → LocDiT**，实现丰富的表现力语音合成和 48kHz 原生音频输出。
- VoxCPM2 Model Architecture + VoxCPM2 模型架构
-> For full architectural details, VoxCPM2-specific upgrades, and a model comparison table, see the [Architecture Design](https://voxcpm.readthedocs.io/en/latest/models/architecture.html). +> 完整架构细节、VoxCPM2 升级内容和模型对比表见 [架构设计文档](https://voxcpm.readthedocs.io/zh-cn/latest/models/architecture.html)。 --- -## 📊 Performance +## 📊 性能评测 -VoxCPM2 achieves state-of-the-art or comparable results on public zero-shot and controllable TTS benchmarks. +VoxCPM2 在公开的零样本和可控 TTS 基准测试中取得了 SOTA 或可比的结果。 ### Seed-TTS-eval -**Seed-TTS-eval WER(⬇)&SIM(⬆) Results (click to expand)** - - -| Model | Parameters | Open-Source | test-EN | | test-ZH | | test-Hard | | -| ----------------- | ---------- | ----------- | ------- | ------ | ------- | ------ | --------- | ------ | -| | | | WER/%⬇ | SIM/%⬆ | CER/%⬇ | SIM/%⬆ | CER/%⬇ | SIM/%⬆ | -| MegaTTS3 | 0.5B | ❌ | 2.79 | 77.1 | 1.52 | 79.0 | - | - | -| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 | - | - | -| CosyVoice3 | 0.5B | ❌ | 2.02 | 71.8 | 1.16 | 78.0 | 6.08 | 75.8 | -| CosyVoice3 | 1.5B | ❌ | 2.22 | 72.0 | 1.12 | 78.1 | 5.83 | 75.8 | -| Seed-TTS | - | ❌ | 2.25 | 76.2 | 1.12 | 79.6 | 7.59 | 77.6 | -| MiniMax-Speech | - | ❌ | 1.65 | 69.2 | 0.83 | 78.3 | - | - | -| F5-TTS | 0.3B | ✅ | 2.00 | 67.0 | 1.53 | 76.0 | 8.67 | 71.3 | -| MaskGCT | 1B | ✅ | 2.62 | 71.7 | 2.27 | 77.4 | - | - | -| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 | 11.75 | 70.9 | -| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 | 6.83 | 72.4 | -| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66.0 | - | - | -| FireRedTTS | 0.5B | ✅ | 3.82 | 46.0 | 1.51 | 63.5 | 17.45 | 62.1 | -| FireRedTTS-2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 | - | - | -| Qwen2.5-Omni | 7B | ✅ | 2.72 | 63.2 | 1.70 | 75.2 | 7.97 | 74.7 | -| Qwen3-Omni | 30B-A3B | ✅ | 1.39 | - | 1.07 | - | - | - | -| OpenAudio-s1-mini | 0.5B | ✅ | 1.94 | 55.0 | 1.18 | 68.5 | 23.37 | 64.3 | -| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 | 7.12 | 75.5 | -| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 | - | - | -| HiggsAudio-v2 | 3B | ✅ | 2.44 | 67.7 | 1.50 | 74.0 | 55.07 | 65.6 | -| VoxCPM-0.5B | 0.6B | ✅ | 1.85 | 72.9 | 0.93 | 77.2 | 8.87 | 73.0 | -| VoxCPM1.5 | 0.8B | ✅ | 2.12 | 71.4 | 1.18 | 77.0 | 7.74 | 73.1 | -| MOSS-TTS | | ✅ | 1.85 | 73.4 | 1.20 | 78.8 | - | - | -| Qwen3-TTS | 1.7B | ✅ | 1.23 | 71.7 | 1.22 | 77.0 | 6.76 | 74.8 | -| FishAudio S2 | 4B | ✅ | 0.99 | - | 0.54 | - | 5.99 | - | -| LongCat-Audio-DiT | 3.5B | ✅ | 1.50 | 78.6 | 1.09 | 81.8 | 6.04 | 79.7 | -| **VoxCPM2** | 2B | ✅ | 1.84 | 75.3 | 0.97 | 79.5 | 8.13 | 75.3 | - - - - -### CV3-eval - -**CV3-eval Multilingual WER/CER(⬇) Results (click to expand)** - - -| Model | zh | en | hard-zh | hard-en | ja | ko | de | es | fr | it | ru | -| --------------- | ---- | ---- | ------- | ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -| CosyVoice2 | 4.08 | 6.32 | 12.58 | 11.96 | 9.13 | 19.7 | - | - | - | - | - | -| CosyVoice3-1.5B | 3.91 | 4.99 | 9.77 | 10.55 | 7.57 | 5.69 | 6.43 | 4.47 | 11.8 | 10.5 | 6.64 | -| Fish Audio S2 | 2.65 | 2.43 | 9.10 | 4.40 | 3.96 | 2.76 | 2.22 | 2.00 | 6.26 | 2.04 | 2.78 | -| **VoxCPM2** | 3.65 | 5.00 | 8.55 | 8.48 | 5.96 | 5.69 | 4.77 | 3.80 | 9.85 | 4.25 | 5.21 | +
+Seed-TTS-eval WER(⬇)&SIM(⬆) 结果（点击展开） + +| Model | Parameters | Open-Source | test-EN | | test-ZH | | test-Hard | | +|------|------|------|:------------:|:--:|:------------:|:--:|:-------------:|:--:| +| | | | WER/%⬇ | SIM/%⬆| CER/%⬇| SIM/%⬆ | CER/%⬇ | SIM/%⬆ | +| MegaTTS3 | 0.5B | ❌ | 2.79 | 77.1 | 1.52 | 79.0 | - | - | +| DiTAR | 0.6B | ❌ | 1.69 | 73.5 | 1.02 | 75.3 | - | - | +| CosyVoice3 | 0.5B | ❌ | 2.02 | 71.8 | 1.16 | 78.0 | 6.08 | 75.8 | +| CosyVoice3 | 1.5B | ❌ | 2.22 | 72.0 | 1.12 | 78.1 | 5.83 | 75.8 | +| Seed-TTS | - | ❌ | 2.25 | 76.2 | 1.12 | 79.6 | 7.59 | 77.6 | +| MiniMax-Speech | - | ❌ | 1.65 | 69.2 | 0.83 | 78.3 | - | - | +| F5-TTS | 0.3B | ✅ | 2.00 | 67.0 | 1.53 | 76.0 | 8.67 | 71.3 | +| MaskGCT | 1B | ✅ | 2.62 | 71.7 | 2.27 | 77.4 | - | - | +| CosyVoice | 0.3B | ✅ | 4.29 | 60.9 | 3.63 | 72.3 | 11.75 | 70.9 | +| CosyVoice2 | 0.5B | ✅ | 3.09 | 65.9 | 1.38 | 75.7 | 6.83 | 72.4 | +| SparkTTS | 0.5B | ✅ | 3.14 | 57.3 | 1.54 | 66.0 | - | - | +| FireRedTTS | 0.5B | ✅ | 3.82 | 46.0 | 1.51 | 63.5 | 17.45 | 62.1 | +| FireRedTTS-2 | 1.5B | ✅ | 1.95 | 66.5 | 1.14 | 73.6 | - | - | +| Qwen2.5-Omni | 7B | ✅ | 2.72 | 63.2 | 1.70 | 75.2 | 7.97 | 74.7 | +| Qwen3-Omni | 30B-A3B | ✅ | 1.39 | - | 1.07 | - | - | - | +| OpenAudio-s1-mini | 0.5B | ✅ | 1.94 | 55.0 | 1.18 | 68.5 | 23.37 | 64.3 | +| IndexTTS2 | 1.5B | ✅ | 2.23 | 70.6 | 1.03 | 76.5 | 7.12 | 75.5 | +| VibeVoice | 1.5B | ✅ | 3.04 | 68.9 | 1.16 | 74.4 | - | - | +| HiggsAudio-v2 | 3B | ✅ | 2.44 | 67.7 | 1.50 | 74.0 | 55.07 | 65.6 | +| VoxCPM-0.5B | 0.6B | ✅ | 1.85 | 72.9 | 0.93 | 77.2 | 8.87 | 73.0 | +| VoxCPM1.5 | 0.8B | ✅ | 2.12 | 71.4 | 1.18 | 77.0 | 7.74 | 73.1 | +| MOSS-TTS | | ✅ | 1.85 | 73.4 | 1.20 | 78.8 | - | - | +| Qwen3-TTS | 1.7B | ✅ | 1.23 | 71.7 | 1.22 | 77.0 | 6.76 | 74.8 | +| FishAudio S2 | 4B | ✅ | 0.99 | - | 0.54 | - | 5.99 | - | +| LongCat-Audio-DiT | 3.5B | ✅ | 1.50 | 78.6 | 1.09 | 81.8 | 6.04 | 79.7 | +| **VoxCPM2** | 2B | ✅ | 1.84 | 75.3 | 0.97| 79.5| 8.13 | 75.3 | +
+### CV3-eval +
+CV3-eval 多语言 WER/CER(⬇) 结果（点击展开） +| Model | zh | en | hard-zh | hard-en | ja | ko | de | es | fr | it | ru | +|-------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| +| CosyVoice2 | 4.08 | 6.32 | 12.58| 11.96| 9.13 | 19.7 |- | - | - | - | - | +| CosyVoice3-1.5B | 3.91 | 4.99 | 9.77 | 10.55 | 7.57 | 5.69 | 6.43 | 4.47 | 11.8 | 10.5 | 6.64 | +| Fish Audio S2 | 2.65 | 2.43 | 9.10 | 4.40 | 3.96 | 2.76 | 2.22 | 2.00 | 6.26 | 2.04 | 2.78 | +| **VoxCPM2** | 3.65 | 5.00 | 8.55 | 8.48 | 5.96 | 5.69 | 4.77 | 3.80 | 9.85 | 4.25 | 5.21 | +
### MiniMax-Multilingual-Test -**Minimax-MLS-test WER(⬇) Results (click to expand)** +
+Minimax-MLS-test WER(⬇) 结果（点击展开） +| Language | Minimax | ElevenLabs | Qwen3-TTS | FishAudio S2 | **VoxCPM2** | +|----------|:-------:|:----------:|:--------------------:|:------------:|:-----------:| +| Arabic | **1.665** | 1.666 | – | 3.500 | 13.046 | +| Cantonese | 34.111 | 51.513 | – | **30.670** | 38.584 | +| Chinese | 2.252 | 16.026 | 0.928 | **0.730** | 1.136 | +| Czech | 3.875 | **2.108** | – | 2.840 | 24.132 | +| Dutch | 1.143 | **0.803** | – | 0.990 | 0.913 | +| English | 2.164 | 2.339 | **0.934** | 1.620 | 2.289 | +| Finnish | 4.666 | 2.964 | – | 3.330 | **2.632** | +| French | 4.099 | 5.216 | **2.858** | 3.050 | 4.534 | +| German | 1.906 | 0.572 | 1.235 | **0.550** | 0.679 | +| Greek | 2.016 | **0.991** | – | 5.740 | 2.844 | +| Hindi | 6.962 | **5.827** | – | 14.640 | 19.699 | +| Indonesian | 1.237 | **1.059** | – | 1.460 | 1.084 | +| Italian | 1.543 | 1.743 | **0.948** | 1.270 | 1.563 | +| Japanese | 3.519 | 10.646 | 3.823 | **2.760** | 4.628 | +| Korean | 1.747 | 1.865 | 1.755 | **1.180** | 1.962 | +| Polish | 1.415 | **0.766** | – | 1.260 | 1.141 | +| Portuguese | 1.877 | 1.331 | 1.526 | **1.140** | 1.938 | +| Romanian | 2.878 | **1.347** | – | 10.740 | 21.577 | +| Russian | 4.281 | 3.878 | 3.212 | **2.400** | 3.634 | +| Spanish | 1.029 | 1.084 | 1.126 | **0.910** | 1.438 | +| Thai | 2.701 | 73.936 | – | 4.230 | 2.961 | +| Turkish | 1.52 | 0.699 | – | 0.870 | 0.817 | +| Ukrainian | 1.082 | **0.997** | – | 2.300 | 6.316 | +| Vietnamese | **0.88** | 73.415 | – | 7.410 | 3.307 | -| Language | Minimax | ElevenLabs | Qwen3-TTS | FishAudio S2 | **VoxCPM2** | -| ---------- | --------- | ---------- | --------- | ------------ | ----------- | -| Arabic | **1.665** | 1.666 | – | 3.500 | 13.046 | -| Cantonese | 34.111 | 51.513 | – | **30.670** | 38.584 | -| Chinese | 2.252 | 16.026 | 0.928 | **0.730** | 1.136 | -| Czech | 3.875 | **2.108** | – | 2.840 | 24.132 | -| Dutch | 1.143 | **0.803** | – | 0.990 | 0.913 | -| English | 2.164 | 2.339 | **0.934** | 1.620 | 2.289 | -| Finnish | 4.666 | 2.964 | – | 3.330 | **2.632** | -| French | 4.099 | 5.216 | **2.858** | 3.050 | 4.534 | -| German | 1.906 | 0.572 | 1.235 | **0.550** | 0.679 | -| Greek | 2.016 | **0.991** | – | 5.740 | 2.844 | -| Hindi | 6.962 | **5.827** | – | 14.640 | 19.699 | -| Indonesian | 1.237 | **1.059** | – | 1.460 | 1.084 | -| Italian | 1.543 | 1.743 | **0.948** | 1.270 | 1.563 | -| Japanese | 3.519 | 10.646 | 3.823 | **2.760** | 4.628 | -| Korean | 1.747 | 1.865 | 1.755 | **1.180** | 1.962 | -| Polish | 1.415 | **0.766** | – | 1.260 | 1.141 | -| Portuguese | 1.877 | 1.331 | 1.526 | **1.140** | 1.938 | -| Romanian | 2.878 | **1.347** | – | 10.740 | 21.577 | -| Russian | 4.281 | 3.878 | 3.212 | **2.400** | 3.634 | -| Spanish | 1.029 | 1.084 | 1.126 | **0.910** | 1.438 | -| Thai | 2.701 | 73.936 | – | 4.230 | 2.961 | -| Turkish | 1.52 | 0.699 | – | 0.870 | 0.817 | -| Ukrainian | 1.082 | **0.997** | – | 2.300 | 6.316 | -| Vietnamese | **0.88** | 73.415 | – | 7.410 | 3.307 | - - - - -**Minimax-MLS-test SIM(⬆) Results (click to expand)** - - -| Language | Minimax | ElevenLabs | Qwen3-TTS | FishAudio S2 | **VoxCPM2** | -| ---------- | -------- | ---------- | --------- | ------------ | ----------- | -| Arabic | 73.6 | 70.6 | – | 75.0 | **79.1** | -| Cantonese | 77.8 | 67.0 | – | 80.5 | **83.5** | -| Chinese | 78.0 | 67.7 | 79.9 | 81.6 | **82.5** | -| Czech | 79.6 | 68.5 | – | **79.8** | 78.3 | -| Dutch | 73.8 | 68.0 | – | 73.0 | **80.8** | -| English | 75.6 | 61.3 | 77.5 | 79.7 | **85.4** | -| Finnish | 83.5 | 75.9 | – | 81.9 | **89.0** | -| French | 62.8 | 53.5 | 62.8 | 69.8 | **73.5** | -| German | 73.3 | 61.4 | 77.5 | 76.7 | **80.3** | -| Greek | 82.6 | 73.3 | – | 79.5 | **86.0** | -| Hindi | 81.8 | 73.0 | – | 82.1 | **85.6** | -| Indonesian | 72.9 | 66.0 | – | 76.3 | **80.0** | -| Italian | 69.9 | 57.9 | 81.7 | 74.7 | **78.0** | -| Japanese | 77.6 | 73.8 | 78.8 | 79.6 | **82.8** | -| Korean | 77.6 | 70.0 | 79.9 | 81.7 | **83.3** | -| Polish | 80.2 | 72.9 | – | 81.9 | **88.4** | -| Portuguese | 80.5 | 71.1 | 81.7 | 78.1 | **83.7** | -| Romanian | **80.9** | 69.9 | – | 73.3 | 79.7 | -| Russian | 76.1 | 67.6 | 79.2 | 79.0 | **81.1** | -| Spanish | 76.2 | 61.5 | 81.4 | 77.6 | **83.1** | -| Thai | 80.0 | 58.8 | – | 78.6 | **84.0** | -| Turkish | 77.9 | 59.6 | – | 83.5 | **87.1** | -| Ukrainian | 73.0 | 64.7 | – | 74.7 | **79.8** | -| Vietnamese | 74.3 | 36.9 | – | 74.0 | **80.6** | +
+
+Minimax-MLS-test SIM(⬆) 结果（点击展开） +| Language | Minimax | ElevenLabs | Qwen3-TTS | FishAudio S2 | **VoxCPM2** | +|----------|:-------:|:----------:|:--------------------:|:------------:|:-----------:| +| Arabic | 73.6 | 70.6 | – | 75.0 | **79.1** | +| Cantonese | 77.8 | 67.0 | – | 80.5 | **83.5** | +| Chinese | 78.0 | 67.7 | 79.9 | 81.6 | **82.5** | +| Czech | 79.6 | 68.5 | – | **79.8** | 78.3 | +| Dutch | 73.8 | 68.0 | – | 73.0 | **80.8** | +| English | 75.6 | 61.3 | 77.5 | 79.7 | **85.4** | +| Finnish | 83.5 | 75.9 | – | 81.9 | **89.0** | +| French | 62.8 | 53.5 | 62.8 | 69.8 | **73.5** | +| German | 73.3 | 61.4 | 77.5 | 76.7 | **80.3** | +| Greek | 82.6 | 73.3 | – | 79.5 | **86.0** | +| Hindi | 81.8 | 73.0 | – | 82.1 | **85.6** | +| Indonesian | 72.9 | 66.0 | – | 76.3 | **80.0** | +| Italian | 69.9 | 57.9 | 81.7 | 74.7 | **78.0** | +| Japanese | 77.6 | 73.8 | 78.8 | 79.6 | **82.8** | +| Korean | 77.6 | 70.0 | 79.9 | 81.7 | **83.3** | +| Polish | 80.2 | 72.9 | – | 81.9 | **88.4** | +| Portuguese | 80.5 | 71.1 | 81.7 | 78.1 | **83.7** | +| Romanian | **80.9** | 69.9 | – | 73.3 | 79.7 | +| Russian | 76.1 | 67.6 | 79.2 | 79.0 | **81.1** | +| Spanish | 76.2 | 61.5 | 81.4 | 77.6 | **83.1** | +| Thai | 80.0 | 58.8 | – | 78.6 | **84.0** | +| Turkish | 77.9 | 59.6 | – | 83.5 | **87.1** | +| Ukrainian | 73.0 | 64.7 | – | 74.7 | **79.8** | +| Vietnamese | 74.3 | 36.9 | – | 74.0 | **80.6** | +
### Internal 30-Language ASR Benchmark -We additionally run an internal multilingual intelligibility benchmark with **30 languages × 500 samples**. ASR transcription is evaluated via **Gemini 3.1 Flash Lite API**. +我们额外进行了内部多语言可懂度评测：**30 语种 × 500 样本**，ASR 转写评估使用 **Gemini 3.1 Flash Lite API**。 -**Internal 30-Language ASR Benchmark (click to expand)** - - -| Language | Metric | VoxCPM2 | Fish S2-Pro | -| ---------------------- | ------ | --------- | ----------- | -| ar (Arabic) | CER | 1.23% | 0.30% | -| da (Danish) | WER | 2.70% | 3.52% | -| de (German) | WER | 0.96% | 0.64% | -| el (Greek) | WER | 3.17% | 4.61% | -| en (English) | WER | 0.42% | 1.03% | -| es (Spanish) | WER | 1.33% | 0.64% | -| fi (Finnish) | WER | 2.24% | 2.80% | -| fr (French) | WER | 2.16% | 2.34% | -| he (Hebrew) | CER | 2.98% | 15.27% | -| hi (Hindi) | CER | 0.79% | 0.91% | -| id (Indonesian) | WER | 1.36% | 1.68% | -| it (Italian) | WER | 1.65% | 1.08% | -| ja (Japanese) | CER | 2.40% | 1.82% | -| km (Khmer) | CER | 2.05% | 75.15% | -| ko (Korean) | CER | 0.95% | 0.29% | -| lo (Lao) | CER | 1.90% | 87.40% | -| ms (Malay) | WER | 1.75% | 1.41% | -| my (Burmese) | CER | 1.42% | 85.27% | -| nl (Dutch) | WER | 1.25% | 1.68% | -| no (Norwegian) | WER | 2.49% | 3.76% | -| pl (Polish) | WER | 1.90% | 1.65% | -| pt (Portuguese) | WER | 1.48% | 1.49% | -| ru (Russian) | WER | 0.90% | 0.86% | -| sv (Swedish) | WER | 2.22% | 2.63% | -| sw (Swahili) | CER | 1.07% | 2.02% | -| th (Thai) | CER | 0.94% | 1.92% | -| tl (Tagalog) | WER | 2.63% | 4.00% | -| tr (Turkish) | WER | 1.65% | 1.65% | -| vi (Vietnamese) | WER | 1.56% | 5.56% | -| zh (Chinese) | CER | 0.92% | 1.02% | -| Average (30 languages) | | **1.68%** | - | +
+内部30语种评测集ASR结果（点击展开） +| 语言 | 指标 | VoxCPM2 | Fish S2-Pro | +|---|---:|---:|---:| +| ar (阿拉伯语) | CER | 1.23% | 0.30% | +| da (丹麦语) | WER | 2.70% | 3.52% | +| de (德语) | WER | 0.96% | 0.64% | +| el (希腊语) | WER | 3.17% | 4.61% | +| en (英语) | WER | 0.42% | 1.03% | +| es (西班牙语) | WER | 1.33% | 0.64% | +| fi (芬兰语) | WER | 2.24% | 2.80% | +| fr (法语) | WER | 2.16% | 2.34% | +| he (希伯来语) | CER | 2.98% | 15.27% | +| hi (印地语) | CER | 0.79% | 0.91% | +| id (印尼语) | WER | 1.36% | 1.68% | +| it (意大利语) | WER | 1.65% | 1.08% | +| ja (日语) | CER | 2.40% | 1.82% | +| km (高棉语) | CER | 2.05% | 75.15% | +| ko (韩语) | CER | 0.95% | 0.29% | +| lo (老挝语) | CER | 1.90% | 87.40% | +| ms (马来语) | WER | 1.75% | 1.41% | +| my (缅甸语) | CER | 1.42% | 85.27% | +| nl (荷兰语) | WER | 1.25% | 1.68% | +| no (挪威语) | WER | 2.49% | 3.76% | +| pl (波兰语) | WER | 1.90% | 1.65% | +| pt (葡萄牙语) | WER | 1.48% | 1.49% | +| ru (俄语) | WER | 0.90% | 0.86% | +| sv (瑞典语) | WER | 2.22% | 2.63% | +| sw (斯瓦希里语) | CER | 1.07% | 2.02% | +| th (泰语) | CER | 0.94% | 1.92% | +| tl (菲律宾语) | WER | 2.63% | 4.00% | +| tr (土耳其语) | WER | 1.65% | 1.65% | +| vi (越南语) | WER | 1.56% | 5.56% | +| zh (中文) | CER | 0.92% | 1.02% | +| 平均（30 语种） | | **1.68%** | - | +
### InstructTTSEval -**Instruction-Guided Voice Design Results (click to expand)** - - -| Model | InstructTTSEval-ZH | | | InstructTTSEval-EN | | | -| ---------------------- | ------------------ | -------- | -------- | ------------------ | -------- | -------- | -| | APS⬆ | DSD⬆ | RP⬆ | APS⬆ | DSD⬆ | RP⬆ | -| Hume | – | – | – | 83.0 | 75.3 | 54.3 | -| VoxInstruct | 47.5 | 52.3 | 42.6 | 54.9 | 57.0 | 39.3 | -| Parler-tts-mini | – | – | – | 63.4 | 48.7 | 28.6 | -| Parler-tts-large | – | – | – | 60.0 | 45.9 | 31.2 | -| PromptTTS | – | – | – | 64.3 | 47.2 | 31.4 | -| PromptStyle | – | – | – | 57.4 | 46.4 | 30.9 | -| VoiceSculptor | 75.7 | 64.7 | 61.5 | – | – | – | -| Mimo-Audio-7B-Instruct | 75.7 | 74.3 | 61.5 | 80.6 | 77.6 | 59.5 | -| Qwen3TTS-12Hz-1.7B-VD | **85.2** | **81.1** | **65.1** | 82.9 | 82.4 | 68.4 | -| **VoxCPM2** | **85.2** | 71.5 | 60.8 | **84.2** | **83.2** | **71.4** | - +
+指令驱动音色设计结果 (点击展开) +| Model | InstructTTSEval-ZH | | | InstructTTSEval-EN | | | +|-------|:---:|:----:|:----:|:----:|:----:|:----:| +| | APS⬆| DSD⬆ | RP⬆| APS⬆ | DSD⬆ | RP⬆ | +| Hume | – | – | – | 83.0 | 75.3 | 54.3 | +| VoxInstruct | 47.5 | 52.3 | 42.6 | 54.9 | 57.0 | 39.3 | +| Parler-tts-mini | – | – | – | 63.4 | 48.7 | 28.6 | +| Parler-tts-large | – | – | – | 60.0 | 45.9 | 31.2 | +| PromptTTS | – | – | – | 64.3 | 47.2 | 31.4 | +| PromptStyle | – | – | – | 57.4 | 46.4 | 30.9 | +| VoiceSculptor | 75.7 | 64.7 | 61.5 | – | – | – | +| Mimo-Audio-7B-Instruct | 75.7 | 74.3 | 61.5 | 80.6 | 77.6 | 59.5 | +| Qwen3TTS-12Hz-1.7B-VD | **85.2** | **81.1** | **65.1** | 82.9 | 82.4 | 68.4 | +| **VoxCPM2** | **85.2** | 71.5 | 60.8 | **84.2** | **83.2** | **71.4** | +
--- -## ⚙️ Fine-tuning +## ⚙️ 微调 -VoxCPM supports both **full fine-tuning (SFT)** and **LoRA fine-tuning**. With as little as **5–10 minutes** of audio, you can adapt to a specific speaker, language, or domain. +VoxCPM 支持**全参数微调（SFT）** 和 **LoRA 微调**。仅需 **5-10分钟** 的音频数据，即可适配特定说话人、语言或领域。 ```bash -# LoRA fine-tuning (parameter-efficient, recommended) +# LoRA 微调（参数高效，推荐） python scripts/train_voxcpm_finetune.py \ --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml -# Full fine-tuning +# 全参数微调 python scripts/train_voxcpm_finetune.py \ --config_path conf/voxcpm_v2/voxcpm_finetune_all.yaml -# WebUI for training & inference -python lora_ft_webui.py # then open http://localhost:7860 +# WebUI 训练与推理 +python lora_ft_webui.py # 然后打开 http://localhost:7860 ``` -> **Full guide →** [Fine-tuning Guide](https://voxcpm.readthedocs.io/en/latest/finetuning/finetune.html) (data preparation, configuration, training, LoRA hot-swapping, FAQ) +> **完整指南 →** [微调文档](https://voxcpm.readthedocs.io/zh-cn/latest/finetuning/finetune.html)（数据准备、配置、训练、LoRA 热切换、常见问题） --- -## 📚 Documentation +## 📚 文档 -Full documentation: **[voxcpm.readthedocs.io](https://voxcpm.readthedocs.io/en/latest/)** - - -| Topic | Link | -| -------------------------- | ------------------------------------------------------------------------------------- | -| Quick Start & Installation | [Quick Start](https://voxcpm.readthedocs.io/en/latest/quickstart.html) | -| Usage Guide & Cookbook | [User Guide](https://voxcpm.readthedocs.io/en/latest/usage_guide.html) | -| VoxCPM Series | [Models](https://voxcpm.readthedocs.io/en/latest/models/version_history.html) | -| Fine-tuning (SFT & LoRA) | [Fine-tuning Guide](https://voxcpm.readthedocs.io/en/latest/finetuning/finetune.html) | -| FAQ & Troubleshooting | [FAQ](https://voxcpm.readthedocs.io/en/latest/faq.html) | +完整文档：**[voxcpm.readthedocs.io](https://voxcpm.readthedocs.io/zh-cn/latest/)** +| 主题 | 链接 | +|---|---| +| 快速开始与安装 | [快速开始](https://voxcpm.readthedocs.io/zh-cn/latest/quickstart.html) | +| 使用指南与 Cookbook | [使用指南](https://voxcpm.readthedocs.io/zh-cn/latest/usage_guide.html) | +| VoxCPM 系列模型 | [模型列表](https://voxcpm.readthedocs.io/zh-cn/latest/models/version_history.html) | +| 微调（SFT & LoRA） | [微调指南](https://voxcpm.readthedocs.io/zh-cn/latest/finetuning/finetune.html) | +| 常见问题 | [FAQ](https://voxcpm.readthedocs.io/zh-cn/latest/faq.html) | --- -## 🌟 Ecosystem & Community +## 🌟 生态与社区 +| 项目 | 说明 | +|---|---| +| [**Nano-vLLM**](https://github.com/a710128/nanovllm-voxcpm) | 高吞吐快速 GPU 推理引擎 | +| [**vLLM-Omni**](https://github.com/vllm-project/vllm-omni) | 官方 vLLM 全模态服务（原生支持 VoxCPM2）— PagedAttention、OpenAI 兼容 API | +| [**llama.cpp-omni**](https://github.com/tc-mb/llama.cpp-omni) | 全双工全模态推理引擎 — VoxCPM2 GGUF，支持 CPU / Metal / CUDA / Vulkan | +| [**VoxCPM.cpp**](https://github.com/bluryar/VoxCPM.cpp) | GGML/GGUF：CPU、CUDA、Vulkan 推理 | +| [**audio.cpp**](https://github.com/0xShug0/audio.cpp) | 基于 ggml 的统一 C++ 推理框架 — CPU/CUDA/Vulkan/Metal，CLI 与服务端，无需 Python | +| [**VoxCPM-ONNX**](https://github.com/bluryar/VoxCPM-ONNX) | ONNX 导出，支持 CPU 推理 | +| [**VoxCPMANE**](https://github.com/0seba/VoxCPMANE) | Apple Neural Engine 后端 | +| [**voxcpm_rs**](https://github.com/madushan1000/voxcpm_rs) | Rust 重新实现 | +| [**ComfyUI-VoxCPM**](https://github.com/wildminder/ComfyUI-VoxCPM) | ComfyUI 节点工作流 | +| [**ComfyUI_RH_VoxCPM**](https://github.com/HM-RunningHub/ComfyUI_RH_VoxCPM) | 面向 VoxCPM 2 的功能更完整的 ComfyUI 工作流，支持多说话人、LoRA 和自动 ASR | +| [**ComfyUI-VoxCPMTTS**](https://github.com/1038lab/ComfyUI-VoxCPMTTS) | ComfyUI TTS 扩展 | +| [**TTS WebUI**](https://github.com/rsxdalv/tts_webui_extension.vox_cpm) | 浏览器端 TTS 扩展 | -| Project | Description | -| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | -| **[Nano-vLLM](https://github.com/a710128/nanovllm-voxcpm)** | High-throughput and Fast GPU serving | -| **[vLLM-Omni](https://github.com/vllm-project/vllm-omni)** | Official vLLM omni-modal serving for VoxCPM2 — PagedAttention, OpenAI-compatible API | -| **[llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni)** | Full-duplex omni inference engine — VoxCPM2 GGUF on CPU / Metal / CUDA / Vulkan | -| **[VoxCPM.cpp](https://github.com/bluryar/VoxCPM.cpp)** | GGML/GGUF: CPU, CUDA, Vulkan inference | -| **[audio.cpp](https://github.com/0xShug0/audio.cpp)** | ggml-based unified C++ inference framework — CPU/CUDA/Vulkan/Metal, CLI & server, no Python | -| **[VoxCPM-ONNX](https://github.com/bluryar/VoxCPM-ONNX)** | ONNX export for CPU inference | -| **[VoxCPMANE](https://github.com/0seba/VoxCPMANE)** | Apple Neural Engine backend | -| **[voxcpm_rs](https://github.com/madushan1000/voxcpm_rs)** | Rust re-implementation | -| **[ComfyUI-VoxCPM](https://github.com/wildminder/ComfyUI-VoxCPM)** | ComfyUI node-based workflows | -| **[ComfyUI_RH_VoxCPM](https://github.com/HM-RunningHub/ComfyUI_RH_VoxCPM)** | Feature-complete ComfyUI workflow for VoxCPM 2 with multi-speaker generation, LoRA, and auto-ASR | -| **[ComfyUI-VoxCPMTTS](https://github.com/1038lab/ComfyUI-VoxCPMTTS)** | ComfyUI TTS extension | -| **[TTS WebUI](https://github.com/rsxdalv/tts_webui_extension.vox_cpm)** | Browser-based TTS extension | - - -> See the full [Ecosystem](https://voxcpm.readthedocs.io/en/latest/) in the docs. Community projects are not officially maintained by OpenBMB. Built something cool? [Open an issue or PR](https://github.com/OpenBMB/VoxCPM/issues) to add it! +> 完整生态见[文档](https://voxcpm.readthedocs.io/zh-cn/latest/)。社区项目非 OpenBMB 官方维护。做了什么有趣的东西？[提 Issue 或 PR](https://github.com/OpenBMB/VoxCPM/issues) 把它加进来！ --- -## ⚠️ Risks and Limitations +## ⚠️ 风险与局限性 -- **Potential for Misuse:** VoxCPM's voice cloning can generate highly realistic synthetic speech. It is **strictly forbidden** to use VoxCPM for impersonation, fraud, or disinformation. We strongly recommend clearly marking any AI-generated content. -- **Controllable Generation Stability:** Voice Design and Controllable Voice Cloning results can vary between runs — you may try to generate 1~3 times to obtain the desired voice or style. We are actively working on improving controllability consistency. -- **Language Coverage:** VoxCPM2 officially supports 30 languages. For languages not on the list, you are welcome to test directly or try fine-tuning on your own data. We plan to expand language coverage in future releases. -- **Usage:** This model is released under the Apache-2.0 license. For production deployments, we recommend conducting thorough testing and safety evaluation tailored to your use case. +- **滥用风险：** VoxCPM 的声音克隆能力可生成高度逼真的合成语音。**严禁**将 VoxCPM 用于冒充他人、欺诈或虚假信息传播。我们强烈建议对所有 AI 生成的内容进行明确标注。 +- **可控生成稳定性：** 音色设计和可控声音克隆的结果可能因生成次数而异 — 建议尝试生成 1~3 次以获得理想的音色或风格。我们正在积极提升可控性的一致性。 +- **语言覆盖：** VoxCPM2 官方支持 30 种语言。对于未列入的语言，欢迎直接测试或使用自有数据进行微调。我们计划在未来版本中扩展语言覆盖。 +- **使用说明：** 本模型基于 Apache-2.0 协议发布。用于生产部署时，我们建议针对具体场景进行充分的测试和安全评估。 --- -## 📖 Citation +## 📖 引用 -If you find VoxCPM helpful, please consider citing our work and starring ⭐ the repository! +如果 VoxCPM 对您有帮助，请考虑引用我们的工作并为仓库加星 ⭐！ ```bib @article{zhou2026voxcpm2, @@ -671,22 +666,26 @@ If you find VoxCPM helpful, please consider citing our work and starring ⭐ the } ``` -## 📄 License +## 📄 许可证 -VoxCPM model weights and code are open-sourced under the [Apache-2.0](LICENSE) license. +VoxCPM 模型权重和代码基于 [Apache-2.0](LICENSE) 协议开源。 -## 🙏 Acknowledgments +## 🙏 致谢 -- [DiTAR](https://arxiv.org/abs/2502.03930) for the diffusion autoregressive backbone -- [MiniCPM-4](https://github.com/OpenBMB/MiniCPM) for the language model foundation -- [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) for the Flow Matching-based LocDiT implementation -- [DAC](https://github.com/descriptinc/descript-audio-codec) for the Audio VAE backbone -- Our community users for trying VoxCPM, reporting issues, sharing ideas, and contributing—your support helps the project keep getting better +- [DiTAR](https://arxiv.org/abs/2502.03930) 扩散自回归骨干架构 +- [MiniCPM-4](https://github.com/OpenBMB/MiniCPM) 语言模型基座 +- [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) 基于 Flow Matching 的 LocDiT 实现 +- [DAC](https://github.com/descriptinc/descript-audio-codec) Audio VAE 骨干 +- 感谢所有社区用户试用 VoxCPM、反馈问题、分享想法和贡献——你们的支持让项目持续进步 -## Institutions +## 机构 -[ModelBest](https://modelbest.cn/)     [THUHCSI](https://github.com/thuhcsi) +

+ 面壁智能 +     + 清华大学人机交互实验室 +

-## ⭐ Star History +## ⭐ Star 历史 -[Star History Chart](https://star-history.com/#OpenBMB/VoxCPM&Date) \ No newline at end of file +[![Star History Chart](https://api.star-history.com/svg?repos=OpenBMB/VoxCPM&type=Date)](https://star-history.com/#OpenBMB/VoxCPM&Date)