Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 wehub-resource-sync
|
e50ac961fb | ||
|
wehub-resource-sync
|
2ea16835a1 | ||
|
wehub-resource-sync
|
8492c4cc85 |
+297
@@ -0,0 +1,297 @@
|
||||
[](https://github.com/SYSTRAN/faster-whisper/actions?query=workflow%3ACI) [](https://badge.fury.io/py/faster-whisper)
|
||||
|
||||
# Faster Whisper transcription with CTranslate2
|
||||
|
||||
**faster-whisper** is a reimplementation of OpenAI's Whisper model using [CTranslate2](https://github.com/OpenNMT/CTranslate2/), which is a fast inference engine for Transformer models.
|
||||
|
||||
This implementation is up to 4 times faster than [openai/whisper](https://github.com/openai/whisper) for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.
|
||||
|
||||
## Benchmark
|
||||
|
||||
### Whisper
|
||||
|
||||
For reference, here's the time and memory usage that are required to transcribe [**13 minutes**](https://www.youtube.com/watch?v=0u7tTptBo9I) of audio using different implementations:
|
||||
|
||||
* [openai/whisper](https://github.com/openai/whisper)@[v20240930](https://github.com/openai/whisper/tree/v20240930)
|
||||
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)@[v1.7.2](https://github.com/ggerganov/whisper.cpp/tree/v1.7.2)
|
||||
* [transformers](https://github.com/huggingface/transformers)@[v4.46.3](https://github.com/huggingface/transformers/tree/v4.46.3)
|
||||
* [faster-whisper](https://github.com/SYSTRAN/faster-whisper)@[v1.1.0](https://github.com/SYSTRAN/faster-whisper/tree/v1.1.0)
|
||||
|
||||
### Large-v2 model on GPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | VRAM Usage |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp16 | 5 | 2m23s | 4708MB |
|
||||
| whisper.cpp (Flash Attention) | fp16 | 5 | 1m05s | 4127MB |
|
||||
| transformers (SDPA)[^1] | fp16 | 5 | 1m52s | 4960MB |
|
||||
| faster-whisper | fp16 | 5 | 1m03s | 4525MB |
|
||||
| faster-whisper (`batch_size=8`) | fp16 | 5 | 17s | 6090MB |
|
||||
| faster-whisper | int8 | 5 | 59s | 2926MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 16s | 4500MB |
|
||||
|
||||
### distil-whisper-large-v3 model on GPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | YT Commons WER |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| transformers (SDPA) (`batch_size=16`) | fp16 | 5 | 46m12s | 14.801 |
|
||||
| faster-whisper (`batch_size=16`) | fp16 | 5 | 25m50s | 13.527 |
|
||||
|
||||
*GPU Benchmarks are Executed with CUDA 12.4 on a NVIDIA RTX 3070 Ti 8GB.*
|
||||
[^1]: transformers OOM for any batch size > 1
|
||||
|
||||
### Small model on CPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | RAM Usage |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp32 | 5 | 6m58s | 2335MB |
|
||||
| whisper.cpp | fp32 | 5 | 2m05s | 1049MB |
|
||||
| whisper.cpp (OpenVINO) | fp32 | 5 | 1m45s | 1642MB |
|
||||
| faster-whisper | fp32 | 5 | 2m37s | 2257MB |
|
||||
| faster-whisper (`batch_size=8`) | fp32 | 5 | 1m06s | 4230MB |
|
||||
| faster-whisper | int8 | 5 | 1m42s | 1477MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 51s | 3608MB |
|
||||
|
||||
*Executed with 8 threads on an Intel Core i7-12700K.*
|
||||
|
||||
|
||||
## Requirements
|
||||
|
||||
* Python 3.9 or greater
|
||||
|
||||
Unlike openai-whisper, FFmpeg does **not** need to be installed on the system. The audio is decoded with the Python library [PyAV](https://github.com/PyAV-Org/PyAV) which bundles the FFmpeg libraries in its package.
|
||||
|
||||
### GPU
|
||||
|
||||
GPU execution requires the following NVIDIA libraries to be installed:
|
||||
|
||||
* [cuBLAS for CUDA 12](https://developer.nvidia.com/cublas)
|
||||
* [cuDNN 9 for CUDA 12](https://developer.nvidia.com/cudnn)
|
||||
|
||||
**Note**: The latest versions of `ctranslate2` only support CUDA 12 and cuDNN 9. For CUDA 11 and cuDNN 8, the current workaround is downgrading to the `3.24.0` version of `ctranslate2`, for CUDA 12 and cuDNN 8, downgrade to the `4.4.0` version of `ctranslate2`, (This can be done with `pip install --force-reinstall ctranslate2==4.4.0` or specifying the version in a `requirements.txt`).
|
||||
|
||||
There are multiple ways to install the NVIDIA libraries mentioned above. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
|
||||
|
||||
**Note:** For all these methods below, keep in mind the above note regarding CUDA versions. Depending on your setup, you may need to install the _CUDA 11_ versions of libraries that correspond to the CUDA 12 libraries listed in the instructions below.
|
||||
|
||||
#### Use Docker
|
||||
|
||||
The libraries (cuBLAS, cuDNN) are installed in this official NVIDIA CUDA Docker images: `nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`.
|
||||
|
||||
#### Install with `pip` (Linux only)
|
||||
|
||||
On Linux these libraries can be installed with `pip`. Note that `LD_LIBRARY_PATH` must be set before launching Python.
|
||||
|
||||
```bash
|
||||
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
|
||||
export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
|
||||
```
|
||||
|
||||
#### Download the libraries from Purfview's repository (Windows & Linux)
|
||||
|
||||
Purfview's [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) provides the required NVIDIA libraries for Windows & Linux in a [single archive](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). Decompress the archive and place the libraries in a directory included in the `PATH`.
|
||||
|
||||
</details>
|
||||
|
||||
## Installation
|
||||
|
||||
The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/):
|
||||
|
||||
```bash
|
||||
pip install faster-whisper
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
|
||||
### Install the master branch
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"
|
||||
```
|
||||
|
||||
### Install a specific commit
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Usage
|
||||
|
||||
### Faster-whisper
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
model_size = "large-v3"
|
||||
|
||||
# Run on GPU with FP16
|
||||
model = WhisperModel(model_size, device="cuda", compute_type="float16")
|
||||
|
||||
# or run on GPU with INT8
|
||||
# model = WhisperModel(model_size, device="cuda", compute_type="int8_float16")
|
||||
# or run on CPU with INT8
|
||||
# model = WhisperModel(model_size, device="cpu", compute_type="int8")
|
||||
|
||||
segments, info = model.transcribe("audio.mp3", beam_size=5)
|
||||
|
||||
print("Detected language '%s' with probability %f" % (info.language, info.language_probability))
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
**Warning:** `segments` is a *generator* so the transcription only starts when you iterate over it. The transcription can be run to completion by gathering the segments in a list or a `for` loop:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3")
|
||||
segments = list(segments) # The transcription will actually run here.
|
||||
```
|
||||
|
||||
### Batched Transcription
|
||||
The following code snippet illustrates how to run batched transcription on an example audio file. `BatchedInferencePipeline.transcribe` is a drop-in replacement for `WhisperModel.transcribe`
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel, BatchedInferencePipeline
|
||||
|
||||
model = WhisperModel("turbo", device="cuda", compute_type="float16")
|
||||
batched_model = BatchedInferencePipeline(model=model)
|
||||
segments, info = batched_model.transcribe("audio.mp3", batch_size=16)
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
### Faster Distil-Whisper
|
||||
|
||||
The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet
|
||||
demonstrates how to run inference with distil-large-v3 on a specified audio file:
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
model_size = "distil-large-v3"
|
||||
|
||||
model = WhisperModel(model_size, device="cuda", compute_type="float16")
|
||||
segments, info = model.transcribe("audio.mp3", beam_size=5, language="en", condition_on_previous_text=False)
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
For more information about the distil-large-v3 model, refer to the original [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
|
||||
### Word-level timestamps
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", word_timestamps=True)
|
||||
|
||||
for segment in segments:
|
||||
for word in segment.words:
|
||||
print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))
|
||||
```
|
||||
|
||||
### VAD filter
|
||||
|
||||
The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad) model to filter out parts of the audio without speech:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", vad_filter=True)
|
||||
```
|
||||
|
||||
The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe(
|
||||
"audio.mp3",
|
||||
vad_filter=True,
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
)
|
||||
```
|
||||
Vad filter is enabled by default for batched transcription.
|
||||
|
||||
### Logging
|
||||
|
||||
The library logging level can be configured like this:
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig()
|
||||
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
|
||||
```
|
||||
|
||||
### Going further
|
||||
|
||||
See more model and transcription options in the [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
|
||||
|
||||
## Community integrations
|
||||
|
||||
Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!
|
||||
|
||||
|
||||
* [speaches](https://github.com/speaches-ai/speaches) is an OpenAI compatible server using `faster-whisper`. It's easily deployable with Docker, works with OpenAI SDKs/CLI, supports streaming, and live transcription.
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) Standalone CLI executables of faster-whisper for Windows, Linux & macOS.
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into `.lrc` files in the desired language using OpenAI-GPT.
|
||||
* [wscribe](https://github.com/geekodour/wscribe) is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with [wscribe-editor](https://github.com/geekodour/wscribe-editor)
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows ([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)) and Linux.
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) is a simple but reliable voice transcriber that provides a user-friendly interface.
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) is open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages.
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi is a very simple script that provides an API backend compatible with OpenAI, HomeAssistant, and Konele (Android voice typing) formats.
|
||||
|
||||
## Model conversion
|
||||
|
||||
When loading a model from its size such as `WhisperModel("large-v3")`, the corresponding CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/Systran).
|
||||
|
||||
We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.
|
||||
|
||||
For example the command below converts the [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) and saves the weights in FP16:
|
||||
|
||||
```bash
|
||||
pip install transformers[torch]>=4.23
|
||||
|
||||
ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-large-v3-ct2
|
||||
--copy_files tokenizer.json preprocessor_config.json --quantization float16
|
||||
```
|
||||
|
||||
* The option `--model` accepts a model name on the Hub or a path to a model directory.
|
||||
* If the option `--copy_files tokenizer.json` is not used, the tokenizer configuration is automatically downloaded when the model is loaded later.
|
||||
|
||||
Models can also be converted from the code. See the [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
|
||||
### Load a converted model
|
||||
|
||||
1. Directly load the model from a local directory:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) and load it from its name:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
## Comparing performance against other implementations
|
||||
|
||||
If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:
|
||||
|
||||
* Verify that the same transcription options are used, especially the same beam size. For example in openai/whisper, `model.transcribe` uses a default beam size of 1 but here we use a default beam size of 5.
|
||||
* Transcription speed is closely affected by the number of words in the transcript, so ensure that other implementations have a similar WER (Word Error Rate) to this one.
|
||||
* When running on CPU, make sure to set the same number of threads. Many frameworks will read the environment variable `OMP_NUM_THREADS`, which can be set when running your script:
|
||||
|
||||
```bash
|
||||
OMP_NUM_THREADS=4 python3 my_script.py
|
||||
```
|
||||
@@ -1,25 +1,31 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/SYSTRAN/faster-whisper) · [上游 README](https://github.com/SYSTRAN/faster-whisper/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
[](https://github.com/SYSTRAN/faster-whisper/actions?query=workflow%3ACI) [](https://badge.fury.io/py/faster-whisper)
|
||||
|
||||
# Faster Whisper transcription with CTranslate2
|
||||
# 使用 CTranslate2 的 Faster Whisper 转录
|
||||
|
||||
**faster-whisper** is a reimplementation of OpenAI's Whisper model using [CTranslate2](https://github.com/OpenNMT/CTranslate2/), which is a fast inference engine for Transformer models.
|
||||
**faster-whisper** 是使用 [CTranslate2](https://github.com/OpenNMT/CTranslate2/), 对 OpenAI Whisper 模型的重新实现，CTranslate2 是一个用于 Transformer 模型的快速推理引擎。
|
||||
|
||||
This implementation is up to 4 times faster than [openai/whisper](https://github.com/openai/whisper) for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.
|
||||
在相同精度下，该实现的运行速度最高可达 [openai/whisper](https://github.com/openai/whisper) 的 4 倍，同时占用更少内存。通过在 CPU 和 GPU 上使用 8 位量化，效率还可以进一步提升。
|
||||
|
||||
## Benchmark
|
||||
## 基准测试
|
||||
|
||||
### Whisper
|
||||
|
||||
For reference, here's the time and memory usage that are required to transcribe [**13 minutes**](https://www.youtube.com/watch?v=0u7tTptBo9I) of audio using different implementations:
|
||||
作为参考，以下是使用不同实现转录 [**13 分钟**](https://www.youtube.com/watch?v=0u7tTptBo9I) 音频所需的时间和内存占用：
|
||||
|
||||
* [openai/whisper](https://github.com/openai/whisper)@[v20240930](https://github.com/openai/whisper/tree/v20240930)
|
||||
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)@[v1.7.2](https://github.com/ggerganov/whisper.cpp/tree/v1.7.2)
|
||||
* [transformers](https://github.com/huggingface/transformers)@[v4.46.3](https://github.com/huggingface/transformers/tree/v4.46.3)
|
||||
* [faster-whisper](https://github.com/SYSTRAN/faster-whisper)@[v1.1.0](https://github.com/SYSTRAN/faster-whisper/tree/v1.1.0)
|
||||
|
||||
### Large-v2 model on GPU
|
||||
### GPU 上的 Large-v2 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | VRAM Usage |
|
||||
| 实现 | 精度 | Beam size | 时间 | VRAM 占用 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp16 | 5 | 2m23s | 4708MB |
|
||||
| whisper.cpp (Flash Attention) | fp16 | 5 | 1m05s | 4127MB |
|
||||
@@ -29,19 +35,19 @@ For reference, here's the time and memory usage that are required to transcribe
|
||||
| faster-whisper | int8 | 5 | 59s | 2926MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 16s | 4500MB |
|
||||
|
||||
### distil-whisper-large-v3 model on GPU
|
||||
### GPU 上的 distil-whisper-large-v3 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | YT Commons WER |
|
||||
| 实现 | 精度 | Beam size | 时间 | YT Commons WER |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| transformers (SDPA) (`batch_size=16`) | fp16 | 5 | 46m12s | 14.801 |
|
||||
| faster-whisper (`batch_size=16`) | fp16 | 5 | 25m50s | 13.527 |
|
||||
|
||||
*GPU Benchmarks are Executed with CUDA 12.4 on a NVIDIA RTX 3070 Ti 8GB.*
|
||||
[^1]: transformers OOM for any batch size > 1
|
||||
*GPU 基准测试在配备 8GB 显存的 NVIDIA RTX 3070 Ti 上，使用 CUDA 12.4 执行。*
|
||||
[^1]: 对于 batch size > 1 的任何情况，transformers 会出现 OOM
|
||||
|
||||
### Small model on CPU
|
||||
### CPU 上的 Small 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | RAM Usage |
|
||||
| 实现 | 精度 | Beam size | 时间 | RAM 占用 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp32 | 5 | 6m58s | 2335MB |
|
||||
| whisper.cpp | fp32 | 5 | 2m05s | 1049MB |
|
||||
@@ -51,39 +57,39 @@ For reference, here's the time and memory usage that are required to transcribe
|
||||
| faster-whisper | int8 | 5 | 1m42s | 1477MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 51s | 3608MB |
|
||||
|
||||
*Executed with 8 threads on an Intel Core i7-12700K.*
|
||||
*在 Intel Core i7-12700K 上使用 8 个线程执行。*
|
||||
|
||||
|
||||
## Requirements
|
||||
## 环境要求
|
||||
|
||||
* Python 3.9 or greater
|
||||
* Python 3.9 或更高版本
|
||||
|
||||
Unlike openai-whisper, FFmpeg does **not** need to be installed on the system. The audio is decoded with the Python library [PyAV](https://github.com/PyAV-Org/PyAV) which bundles the FFmpeg libraries in its package.
|
||||
与 openai-whisper 不同，系统**不**需要安装 FFmpeg。音频解码通过 Python 库 [PyAV](https://github.com/PyAV-Org/PyAV) 完成，该库在其软件包中捆绑了 FFmpeg 库。
|
||||
|
||||
### GPU
|
||||
|
||||
GPU execution requires the following NVIDIA libraries to be installed:
|
||||
GPU 执行需要安装以下 NVIDIA 库：
|
||||
|
||||
* [cuBLAS for CUDA 12](https://developer.nvidia.com/cublas)
|
||||
* [cuDNN 9 for CUDA 12](https://developer.nvidia.com/cudnn)
|
||||
|
||||
**Note**: The latest versions of `ctranslate2` only support CUDA 12 and cuDNN 9. For CUDA 11 and cuDNN 8, the current workaround is downgrading to the `3.24.0` version of `ctranslate2`, for CUDA 12 and cuDNN 8, downgrade to the `4.4.0` version of `ctranslate2`, (This can be done with `pip install --force-reinstall ctranslate2==4.4.0` or specifying the version in a `requirements.txt`).
|
||||
**注意**：最新版本的 `ctranslate2` 仅支持 CUDA 12 和 cuDNN 9。对于 CUDA 11 和 cuDNN 8，目前的变通方法是降级到 `ctranslate2` 的 `3.24.0` 版本；对于 CUDA 12 和 cuDNN 8，则降级到 `ctranslate2` 的 `4.4.0` 版本（可通过 `pip install --force-reinstall ctranslate2==4.4.0` 完成，或在 `requirements.txt` 中指定版本）。
|
||||
|
||||
There are multiple ways to install the NVIDIA libraries mentioned above. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.
|
||||
安装上述 NVIDIA 库有多种方式。推荐方式见官方 NVIDIA 文档，但我们也建议在下方采用其他安装方法。
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
<summary>其他安装方法（点击展开）</summary>
|
||||
|
||||
|
||||
**Note:** For all these methods below, keep in mind the above note regarding CUDA versions. Depending on your setup, you may need to install the _CUDA 11_ versions of libraries that correspond to the CUDA 12 libraries listed in the instructions below.
|
||||
**注意：** 对于以下所有方法，请记住上述关于 CUDA 版本的说明。根据你的环境，你可能需要安装与下方说明中列出的 CUDA 12 库相对应的 _CUDA 11_ 版本库。
|
||||
|
||||
#### Use Docker
|
||||
#### 使用 Docker
|
||||
|
||||
The libraries (cuBLAS, cuDNN) are installed in this official NVIDIA CUDA Docker images: `nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`.
|
||||
这些库（cuBLAS、cuDNN）已安装在官方 NVIDIA CUDA Docker 镜像中：`nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`。
|
||||
|
||||
#### Install with `pip` (Linux only)
|
||||
#### 使用 `pip` 安装（仅 Linux）
|
||||
|
||||
On Linux these libraries can be installed with `pip`. Note that `LD_LIBRARY_PATH` must be set before launching Python.
|
||||
在 Linux 上，可以使用 `pip` 安装这些库。请注意，在启动 Python 之前必须设置 `LD_LIBRARY_PATH`。
|
||||
|
||||
```bash
|
||||
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
@@ -91,30 +97,30 @@ pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
|
||||
```
|
||||
|
||||
#### Download the libraries from Purfview's repository (Windows & Linux)
|
||||
#### 从 Purfview 的仓库下载库（Windows 和 Linux）
|
||||
|
||||
Purfview's [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) provides the required NVIDIA libraries for Windows & Linux in a [single archive](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). Decompress the archive and place the libraries in a directory included in the `PATH`.
|
||||
Purfview 的 [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) 为 Windows 和 Linux 提供了所需的 NVIDIA 库，打包在[单个压缩包](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). 中。解压后将库放置在包含在 `PATH` 中的目录里。
|
||||
|
||||
</details>
|
||||
|
||||
## Installation
|
||||
## 安装
|
||||
|
||||
The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/):
|
||||
该模块可从 [PyPI](https://pypi.org/project/faster-whisper/): 安装：
|
||||
|
||||
```bash
|
||||
pip install faster-whisper
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
<summary>其他安装方法（点击展开）</summary>
|
||||
|
||||
### Install the master branch
|
||||
### 安装 master 分支
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"
|
||||
```
|
||||
|
||||
### Install a specific commit
|
||||
### 安装特定提交
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
|
||||
@@ -122,7 +128,7 @@ pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faste
|
||||
|
||||
</details>
|
||||
|
||||
## Usage
|
||||
## 用法
|
||||
|
||||
### Faster-whisper
|
||||
|
||||
@@ -147,15 +153,16 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
**Warning:** `segments` is a *generator* so the transcription only starts when you iterate over it. The transcription can be run to completion by gathering the segments in a list or a `for` loop:
|
||||
**警告：** `segments` 是一个*生成器*（generator），因此只有在你对其进行迭代时，转录才会开始。可以通过将片段收集到列表中，或使用 `for` 循环，来运行转录直至完成：
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3")
|
||||
segments = list(segments) # The transcription will actually run here.
|
||||
```
|
||||
|
||||
### Batched Transcription
|
||||
The following code snippet illustrates how to run batched transcription on an example audio file. `BatchedInferencePipeline.transcribe` is a drop-in replacement for `WhisperModel.transcribe`
|
||||
### 批量转录
|
||||
|
||||
以下代码片段演示了如何对示例音频文件运行批量转录。`BatchedInferencePipeline.transcribe` 可直接替代 `WhisperModel.transcribe`
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel, BatchedInferencePipeline
|
||||
@@ -170,9 +177,8 @@ for segment in segments:
|
||||
|
||||
### Faster Distil-Whisper
|
||||
|
||||
The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet
|
||||
demonstrates how to run inference with distil-large-v3 on a specified audio file:
|
||||
Distil-Whisper 检查点与 Faster-Whisper 软件包兼容。特别是，最新的 [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
检查点本质上就是为配合 Faster-Whisper 转录算法而设计的。以下代码片段演示了如何在指定音频文件上使用 distil-large-v3 运行推理：
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
@@ -186,9 +192,9 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
For more information about the distil-large-v3 model, refer to the original [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
有关 distil-large-v3 模型的更多信息，请参阅原始 [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
|
||||
### Word-level timestamps
|
||||
### 词级时间戳（Word-level timestamps）
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", word_timestamps=True)
|
||||
@@ -198,15 +204,15 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))
|
||||
```
|
||||
|
||||
### VAD filter
|
||||
### VAD 滤波器（VAD filter）
|
||||
|
||||
The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad) model to filter out parts of the audio without speech:
|
||||
该库集成了 [Silero VAD](https://github.com/snakers4/silero-vad) 模型，用于滤除音频中没有语音的部分：
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", vad_filter=True)
|
||||
```
|
||||
|
||||
The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:
|
||||
默认行为较为保守，仅移除长于 2 秒的静音。可在 [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). 中查看可用的 VAD 参数及其默认值。可通过字典参数 `vad_parameters` 进行自定义：
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe(
|
||||
@@ -215,11 +221,11 @@ segments, _ = model.transcribe(
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
)
|
||||
```
|
||||
Vad filter is enabled by default for batched transcription.
|
||||
批量转写默认启用 Vad 滤波器。
|
||||
|
||||
### Logging
|
||||
### 日志（Logging）
|
||||
|
||||
The library logging level can be configured like this:
|
||||
可按如下方式配置库的日志级别：
|
||||
|
||||
```python
|
||||
import logging
|
||||
@@ -228,37 +234,37 @@ logging.basicConfig()
|
||||
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
|
||||
```
|
||||
|
||||
### Going further
|
||||
### 进一步探索（Going further）
|
||||
|
||||
See more model and transcription options in the [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
|
||||
在 [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) 类实现中查看更多模型与转写选项。
|
||||
|
||||
## Community integrations
|
||||
## 社区集成（Community integrations）
|
||||
|
||||
Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!
|
||||
以下是使用 faster-whisper 的开源项目列表（非穷尽）。欢迎将你的项目添加到列表中！
|
||||
|
||||
|
||||
* [speaches](https://github.com/speaches-ai/speaches) is an OpenAI compatible server using `faster-whisper`. It's easily deployable with Docker, works with OpenAI SDKs/CLI, supports streaming, and live transcription.
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) Standalone CLI executables of faster-whisper for Windows, Linux & macOS.
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into `.lrc` files in the desired language using OpenAI-GPT.
|
||||
* [wscribe](https://github.com/geekodour/wscribe) is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with [wscribe-editor](https://github.com/geekodour/wscribe-editor)
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows ([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)) and Linux.
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) is a simple but reliable voice transcriber that provides a user-friendly interface.
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) is open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages.
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi is a very simple script that provides an API backend compatible with OpenAI, HomeAssistant, and Konele (Android voice typing) formats.
|
||||
* [speaches](https://github.com/speaches-ai/speaches) 是基于 `faster-whisper` 的 OpenAI 兼容服务器。它易于通过 Docker 部署，可与 OpenAI SDKs/CLI 配合使用，并支持流式传输与实时转写。
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) 是一款屡获殊荣的 Python 库，通过 wav2vec2 对齐提供说话人分离（speaker diarization）和精确的词级时间戳
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) 是基于 faster-whisper 的命令行客户端，并与 openai/whisper 的原始客户端兼容。
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) 是一款基于 faster-whisper 和 NVIDIA NeMo 的说话人分离工具。
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) faster-whisper 在 Windows、Linux 和 macOS 上的独立 CLI 可执行文件。
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) 提供基于 AzureML pipelines 实现的可扩展、模块化、端到端的多说话人语音转文本解决方案。
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) 是一款 Python 库，使用 faster-whisper 转写语音文件，并利用 OpenAI-GPT 将结果文本翻译/润色为所需语言的 `.lrc` 文件。
|
||||
* [wscribe](https://github.com/geekodour/wscribe) 是一款灵活的转录生成工具，支持 faster-whisper，可导出词级转录，导出的转录可使用 [wscribe-editor](https://github.com/geekodour/wscribe-editor) 进行编辑
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) 是格拉茨大学 BANDAS-Center 开发的 faster-whisper 图形用户界面实现，用于在 Windows（[Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2))）和 Linux 上进行转写与说话人分离。
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) 为离线类 Whisper 语音转文本模型实现实时模式，并以 faster-whisper 作为最推荐的后端。它实现了基于实际源复杂度的自适应延迟流式策略，并展示了当前最先进水平。
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) 是 OpenAI Whisper 的近实时实现，使用 faster-whisper 作为后端实时转写音频。
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) 是一款简单但可靠的语音转写器，提供用户友好的界面。
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) open dubbing 是一款 AI 配音系统，使用机器学习模型自动将音频对话翻译并同步到不同语言。
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi 是一个非常简单的脚本，提供兼容 OpenAI、HomeAssistant 和 Konele（Android 语音输入）格式的 API 后端。
|
||||
|
||||
## Model conversion
|
||||
## 模型转换（Model conversion）
|
||||
|
||||
When loading a model from its size such as `WhisperModel("large-v3")`, the corresponding CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/Systran).
|
||||
当按尺寸加载模型（例如 `WhisperModel("large-v3")`）时，相应的 CTranslate2 模型会自动从 [Hugging Face Hub](https://huggingface.co/Systran). 下载。
|
||||
|
||||
We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.
|
||||
我们还提供了一个脚本，用于转换任何与 Transformers 库兼容的 Whisper 模型。这些模型可以是原始 OpenAI 模型，也可以是用户微调后的模型。
|
||||
|
||||
For example the command below converts the [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) and saves the weights in FP16:
|
||||
例如，下面的命令会转换 [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) 并以 FP16 格式保存权重：
|
||||
|
||||
```bash
|
||||
pip install transformers[torch]>=4.23
|
||||
@@ -267,30 +273,30 @@ ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-
|
||||
--copy_files tokenizer.json preprocessor_config.json --quantization float16
|
||||
```
|
||||
|
||||
* The option `--model` accepts a model name on the Hub or a path to a model directory.
|
||||
* If the option `--copy_files tokenizer.json` is not used, the tokenizer configuration is automatically downloaded when the model is loaded later.
|
||||
* 选项 `--model` 接受 Hub 上的模型名称或模型目录路径。
|
||||
* 如果未使用选项 `--copy_files tokenizer.json`，则在后续加载模型时会自动下载 tokenizer 配置。
|
||||
|
||||
Models can also be converted from the code. See the [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
也可以从代码中进行模型转换。请参阅 [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
|
||||
### Load a converted model
|
||||
### 加载已转换的模型
|
||||
|
||||
1. Directly load the model from a local directory:
|
||||
1. 直接从本地目录加载模型：
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) and load it from its name:
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) 并按名称加载：
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
## Comparing performance against other implementations
|
||||
## 与其他实现的性能对比
|
||||
|
||||
If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:
|
||||
如果你正在将性能与其他 Whisper 实现进行对比，应确保在相似设置下运行对比。特别要注意：
|
||||
|
||||
* Verify that the same transcription options are used, especially the same beam size. For example in openai/whisper, `model.transcribe` uses a default beam size of 1 but here we use a default beam size of 5.
|
||||
* Transcription speed is closely affected by the number of words in the transcript, so ensure that other implementations have a similar WER (Word Error Rate) to this one.
|
||||
* When running on CPU, make sure to set the same number of threads. Many frameworks will read the environment variable `OMP_NUM_THREADS`, which can be set when running your script:
|
||||
* 确认使用相同的转写选项，尤其是相同的 beam size。例如在 openai/whisper 中，`model.transcribe` 默认 beam size 为 1，而此处默认 beam size 为 5。
|
||||
* 转写速度与转录文本中的词数密切相关，因此请确保其他实现与本实现的 WER（Word Error Rate，词错误率）相近。
|
||||
* 在 CPU 上运行时，请确保设置相同的线程数。许多框架会读取环境变量 `OMP_NUM_THREADS`，可在运行脚本时设置：
|
||||
|
||||
```bash
|
||||
OMP_NUM_THREADS=4 python3 my_script.py
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目：`SYSTRAN/faster-whisper`
|
||||
- 原始仓库：https://github.com/SYSTRAN/faster-whisper
|
||||
- 导入方式：上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
|
||||
Reference in New Issue
Block a user