Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e50ac961fb | |||
| 2ea16835a1 | |||
| 8492c4cc85 |
@@ -17,10 +17,10 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.10
|
||||
- name: Set up Python 3.9
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: 3.9
|
||||
|
||||
- name: Install module
|
||||
run: |
|
||||
@@ -47,10 +47,10 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.10
|
||||
- name: Set up Python 3.9
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: 3.9
|
||||
|
||||
- name: Install module
|
||||
run: |
|
||||
@@ -69,10 +69,10 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.10
|
||||
- name: Set up Python 3.9
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: 3.9
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
|
||||
+297
@@ -0,0 +1,297 @@
|
||||
[](https://github.com/SYSTRAN/faster-whisper/actions?query=workflow%3ACI) [](https://badge.fury.io/py/faster-whisper)
|
||||
|
||||
# Faster Whisper transcription with CTranslate2
|
||||
|
||||
**faster-whisper** is a reimplementation of OpenAI's Whisper model using [CTranslate2](https://github.com/OpenNMT/CTranslate2/), which is a fast inference engine for Transformer models.
|
||||
|
||||
This implementation is up to 4 times faster than [openai/whisper](https://github.com/openai/whisper) for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.
|
||||
|
||||
## Benchmark
|
||||
|
||||
### Whisper
|
||||
|
||||
For reference, here's the time and memory usage that are required to transcribe [**13 minutes**](https://www.youtube.com/watch?v=0u7tTptBo9I) of audio using different implementations:
|
||||
|
||||
* [openai/whisper](https://github.com/openai/whisper)@[v20240930](https://github.com/openai/whisper/tree/v20240930)
|
||||
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)@[v1.7.2](https://github.com/ggerganov/whisper.cpp/tree/v1.7.2)
|
||||
* [transformers](https://github.com/huggingface/transformers)@[v4.46.3](https://github.com/huggingface/transformers/tree/v4.46.3)
|
||||
* [faster-whisper](https://github.com/SYSTRAN/faster-whisper)@[v1.1.0](https://github.com/SYSTRAN/faster-whisper/tree/v1.1.0)
|
||||
|
||||
### Large-v2 model on GPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | VRAM Usage |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp16 | 5 | 2m23s | 4708MB |
|
||||
| whisper.cpp (Flash Attention) | fp16 | 5 | 1m05s | 4127MB |
|
||||
| transformers (SDPA)[^1] | fp16 | 5 | 1m52s | 4960MB |
|
||||
| faster-whisper | fp16 | 5 | 1m03s | 4525MB |
|
||||
| faster-whisper (`batch_size=8`) | fp16 | 5 | 17s | 6090MB |
|
||||
| faster-whisper | int8 | 5 | 59s | 2926MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 16s | 4500MB |
|
||||
|
||||
### distil-whisper-large-v3 model on GPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | YT Commons WER |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| transformers (SDPA) (`batch_size=16`) | fp16 | 5 | 46m12s | 14.801 |
|
||||
| faster-whisper (`batch_size=16`) | fp16 | 5 | 25m50s | 13.527 |
|
||||
|
||||
*GPU Benchmarks are Executed with CUDA 12.4 on a NVIDIA RTX 3070 Ti 8GB.*
|
||||
[^1]: transformers OOM for any batch size > 1
|
||||
|
||||
### Small model on CPU
|
||||
|
||||
| Implementation | Precision | Beam size | Time | RAM Usage |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp32 | 5 | 6m58s | 2335MB |
|
||||
| whisper.cpp | fp32 | 5 | 2m05s | 1049MB |
|
||||
| whisper.cpp (OpenVINO) | fp32 | 5 | 1m45s | 1642MB |
|
||||
| faster-whisper | fp32 | 5 | 2m37s | 2257MB |
|
||||
| faster-whisper (`batch_size=8`) | fp32 | 5 | 1m06s | 4230MB |
|
||||
| faster-whisper | int8 | 5 | 1m42s | 1477MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 51s | 3608MB |
|
||||
|
||||
*Executed with 8 threads on an Intel Core i7-12700K.*
|
||||
|
||||
|
||||
## Requirements
|
||||
|
||||
* Python 3.9 or greater
|
||||
|
||||
Unlike openai-whisper, FFmpeg does **not** need to be installed on the system. The audio is decoded with the Python library [PyAV](https://github.com/PyAV-Org/PyAV) which bundles the FFmpeg libraries in its package.
|
||||
|
||||
### GPU
|
||||
|
||||
GPU execution requires the following NVIDIA libraries to be installed:
|
||||
|
||||
* [cuBLAS for CUDA 12](https://developer.nvidia.com/cublas)
|
||||
* [cuDNN 9 for CUDA 12](https://developer.nvidia.com/cudnn)
|
||||
|
||||
**Note**: The latest versions of `ctranslate2` only support CUDA 12 and cuDNN 9. For CUDA 11 and cuDNN 8, the current workaround is downgrading to the `3.24.0` version of `ctranslate2`, for CUDA 12 and cuDNN 8, downgrade to the `4.4.0` version of `ctranslate2`, (This can be done with `pip install --force-reinstall ctranslate2==4.4.0` or specifying the version in a `requirements.txt`).
|
||||
|
||||
There are multiple ways to install the NVIDIA libraries mentioned above. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
|
||||
|
||||
**Note:** For all these methods below, keep in mind the above note regarding CUDA versions. Depending on your setup, you may need to install the _CUDA 11_ versions of libraries that correspond to the CUDA 12 libraries listed in the instructions below.
|
||||
|
||||
#### Use Docker
|
||||
|
||||
The libraries (cuBLAS, cuDNN) are installed in this official NVIDIA CUDA Docker images: `nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`.
|
||||
|
||||
#### Install with `pip` (Linux only)
|
||||
|
||||
On Linux these libraries can be installed with `pip`. Note that `LD_LIBRARY_PATH` must be set before launching Python.
|
||||
|
||||
```bash
|
||||
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
|
||||
export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
|
||||
```
|
||||
|
||||
#### Download the libraries from Purfview's repository (Windows & Linux)
|
||||
|
||||
Purfview's [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) provides the required NVIDIA libraries for Windows & Linux in a [single archive](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). Decompress the archive and place the libraries in a directory included in the `PATH`.
|
||||
|
||||
</details>
|
||||
|
||||
## Installation
|
||||
|
||||
The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/):
|
||||
|
||||
```bash
|
||||
pip install faster-whisper
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
|
||||
### Install the master branch
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"
|
||||
```
|
||||
|
||||
### Install a specific commit
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Usage
|
||||
|
||||
### Faster-whisper
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
model_size = "large-v3"
|
||||
|
||||
# Run on GPU with FP16
|
||||
model = WhisperModel(model_size, device="cuda", compute_type="float16")
|
||||
|
||||
# or run on GPU with INT8
|
||||
# model = WhisperModel(model_size, device="cuda", compute_type="int8_float16")
|
||||
# or run on CPU with INT8
|
||||
# model = WhisperModel(model_size, device="cpu", compute_type="int8")
|
||||
|
||||
segments, info = model.transcribe("audio.mp3", beam_size=5)
|
||||
|
||||
print("Detected language '%s' with probability %f" % (info.language, info.language_probability))
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
**Warning:** `segments` is a *generator* so the transcription only starts when you iterate over it. The transcription can be run to completion by gathering the segments in a list or a `for` loop:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3")
|
||||
segments = list(segments) # The transcription will actually run here.
|
||||
```
|
||||
|
||||
### Batched Transcription
|
||||
The following code snippet illustrates how to run batched transcription on an example audio file. `BatchedInferencePipeline.transcribe` is a drop-in replacement for `WhisperModel.transcribe`
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel, BatchedInferencePipeline
|
||||
|
||||
model = WhisperModel("turbo", device="cuda", compute_type="float16")
|
||||
batched_model = BatchedInferencePipeline(model=model)
|
||||
segments, info = batched_model.transcribe("audio.mp3", batch_size=16)
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
### Faster Distil-Whisper
|
||||
|
||||
The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet
|
||||
demonstrates how to run inference with distil-large-v3 on a specified audio file:
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
model_size = "distil-large-v3"
|
||||
|
||||
model = WhisperModel(model_size, device="cuda", compute_type="float16")
|
||||
segments, info = model.transcribe("audio.mp3", beam_size=5, language="en", condition_on_previous_text=False)
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
For more information about the distil-large-v3 model, refer to the original [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
|
||||
### Word-level timestamps
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", word_timestamps=True)
|
||||
|
||||
for segment in segments:
|
||||
for word in segment.words:
|
||||
print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))
|
||||
```
|
||||
|
||||
### VAD filter
|
||||
|
||||
The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad) model to filter out parts of the audio without speech:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", vad_filter=True)
|
||||
```
|
||||
|
||||
The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe(
|
||||
"audio.mp3",
|
||||
vad_filter=True,
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
)
|
||||
```
|
||||
Vad filter is enabled by default for batched transcription.
|
||||
|
||||
### Logging
|
||||
|
||||
The library logging level can be configured like this:
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig()
|
||||
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
|
||||
```
|
||||
|
||||
### Going further
|
||||
|
||||
See more model and transcription options in the [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
|
||||
|
||||
## Community integrations
|
||||
|
||||
Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!
|
||||
|
||||
|
||||
* [speaches](https://github.com/speaches-ai/speaches) is an OpenAI compatible server using `faster-whisper`. It's easily deployable with Docker, works with OpenAI SDKs/CLI, supports streaming, and live transcription.
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) Standalone CLI executables of faster-whisper for Windows, Linux & macOS.
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into `.lrc` files in the desired language using OpenAI-GPT.
|
||||
* [wscribe](https://github.com/geekodour/wscribe) is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with [wscribe-editor](https://github.com/geekodour/wscribe-editor)
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows ([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)) and Linux.
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) is a simple but reliable voice transcriber that provides a user-friendly interface.
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) is open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages.
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi is a very simple script that provides an API backend compatible with OpenAI, HomeAssistant, and Konele (Android voice typing) formats.
|
||||
|
||||
## Model conversion
|
||||
|
||||
When loading a model from its size such as `WhisperModel("large-v3")`, the corresponding CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/Systran).
|
||||
|
||||
We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.
|
||||
|
||||
For example the command below converts the [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) and saves the weights in FP16:
|
||||
|
||||
```bash
|
||||
pip install transformers[torch]>=4.23
|
||||
|
||||
ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-large-v3-ct2
|
||||
--copy_files tokenizer.json preprocessor_config.json --quantization float16
|
||||
```
|
||||
|
||||
* The option `--model` accepts a model name on the Hub or a path to a model directory.
|
||||
* If the option `--copy_files tokenizer.json` is not used, the tokenizer configuration is automatically downloaded when the model is loaded later.
|
||||
|
||||
Models can also be converted from the code. See the [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
|
||||
### Load a converted model
|
||||
|
||||
1. Directly load the model from a local directory:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) and load it from its name:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
## Comparing performance against other implementations
|
||||
|
||||
If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:
|
||||
|
||||
* Verify that the same transcription options are used, especially the same beam size. For example in openai/whisper, `model.transcribe` uses a default beam size of 1 but here we use a default beam size of 5.
|
||||
* Transcription speed is closely affected by the number of words in the transcript, so ensure that other implementations have a similar WER (Word Error Rate) to this one.
|
||||
* When running on CPU, make sure to set the same number of threads. Many frameworks will read the environment variable `OMP_NUM_THREADS`, which can be set when running your script:
|
||||
|
||||
```bash
|
||||
OMP_NUM_THREADS=4 python3 my_script.py
|
||||
```
|
||||
@@ -1,25 +1,31 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/SYSTRAN/faster-whisper) · [上游 README](https://github.com/SYSTRAN/faster-whisper/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
[](https://github.com/SYSTRAN/faster-whisper/actions?query=workflow%3ACI) [](https://badge.fury.io/py/faster-whisper)
|
||||
|
||||
# Faster Whisper transcription with CTranslate2
|
||||
# 使用 CTranslate2 的 Faster Whisper 转录
|
||||
|
||||
**faster-whisper** is a reimplementation of OpenAI's Whisper model using [CTranslate2](https://github.com/OpenNMT/CTranslate2/), which is a fast inference engine for Transformer models.
|
||||
**faster-whisper** 是使用 [CTranslate2](https://github.com/OpenNMT/CTranslate2/), 对 OpenAI Whisper 模型的重新实现,CTranslate2 是一个用于 Transformer 模型的快速推理引擎。
|
||||
|
||||
This implementation is up to 4 times faster than [openai/whisper](https://github.com/openai/whisper) for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU.
|
||||
在相同精度下,该实现的运行速度最高可达 [openai/whisper](https://github.com/openai/whisper) 的 4 倍,同时占用更少内存。通过在 CPU 和 GPU 上使用 8 位量化,效率还可以进一步提升。
|
||||
|
||||
## Benchmark
|
||||
## 基准测试
|
||||
|
||||
### Whisper
|
||||
|
||||
For reference, here's the time and memory usage that are required to transcribe [**13 minutes**](https://www.youtube.com/watch?v=0u7tTptBo9I) of audio using different implementations:
|
||||
作为参考,以下是使用不同实现转录 [**13 分钟**](https://www.youtube.com/watch?v=0u7tTptBo9I) 音频所需的时间和内存占用:
|
||||
|
||||
* [openai/whisper](https://github.com/openai/whisper)@[v20240930](https://github.com/openai/whisper/tree/v20240930)
|
||||
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)@[v1.7.2](https://github.com/ggerganov/whisper.cpp/tree/v1.7.2)
|
||||
* [transformers](https://github.com/huggingface/transformers)@[v4.46.3](https://github.com/huggingface/transformers/tree/v4.46.3)
|
||||
* [faster-whisper](https://github.com/SYSTRAN/faster-whisper)@[v1.1.0](https://github.com/SYSTRAN/faster-whisper/tree/v1.1.0)
|
||||
|
||||
### Large-v2 model on GPU
|
||||
### GPU 上的 Large-v2 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | VRAM Usage |
|
||||
| 实现 | 精度 | Beam size | 时间 | VRAM 占用 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp16 | 5 | 2m23s | 4708MB |
|
||||
| whisper.cpp (Flash Attention) | fp16 | 5 | 1m05s | 4127MB |
|
||||
@@ -29,19 +35,19 @@ For reference, here's the time and memory usage that are required to transcribe
|
||||
| faster-whisper | int8 | 5 | 59s | 2926MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 16s | 4500MB |
|
||||
|
||||
### distil-whisper-large-v3 model on GPU
|
||||
### GPU 上的 distil-whisper-large-v3 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | YT Commons WER |
|
||||
| 实现 | 精度 | Beam size | 时间 | YT Commons WER |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| transformers (SDPA) (`batch_size=16`) | fp16 | 5 | 46m12s | 14.801 |
|
||||
| faster-whisper (`batch_size=16`) | fp16 | 5 | 25m50s | 13.527 |
|
||||
|
||||
*GPU Benchmarks are Executed with CUDA 12.4 on a NVIDIA RTX 3070 Ti 8GB.*
|
||||
[^1]: transformers OOM for any batch size > 1
|
||||
*GPU 基准测试在配备 8GB 显存的 NVIDIA RTX 3070 Ti 上,使用 CUDA 12.4 执行。*
|
||||
[^1]: 对于 batch size > 1 的任何情况,transformers 会出现 OOM
|
||||
|
||||
### Small model on CPU
|
||||
### CPU 上的 Small 模型
|
||||
|
||||
| Implementation | Precision | Beam size | Time | RAM Usage |
|
||||
| 实现 | 精度 | Beam size | 时间 | RAM 占用 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| openai/whisper | fp32 | 5 | 6m58s | 2335MB |
|
||||
| whisper.cpp | fp32 | 5 | 2m05s | 1049MB |
|
||||
@@ -51,39 +57,39 @@ For reference, here's the time and memory usage that are required to transcribe
|
||||
| faster-whisper | int8 | 5 | 1m42s | 1477MB |
|
||||
| faster-whisper (`batch_size=8`) | int8 | 5 | 51s | 3608MB |
|
||||
|
||||
*Executed with 8 threads on an Intel Core i7-12700K.*
|
||||
*在 Intel Core i7-12700K 上使用 8 个线程执行。*
|
||||
|
||||
|
||||
## Requirements
|
||||
## 环境要求
|
||||
|
||||
* Python 3.10 or greater
|
||||
* Python 3.9 或更高版本
|
||||
|
||||
Unlike openai-whisper, FFmpeg does **not** need to be installed on the system. The audio is decoded with the Python library [PyAV](https://github.com/PyAV-Org/PyAV) which bundles the FFmpeg libraries in its package.
|
||||
与 openai-whisper 不同,系统**不**需要安装 FFmpeg。音频解码通过 Python 库 [PyAV](https://github.com/PyAV-Org/PyAV) 完成,该库在其软件包中捆绑了 FFmpeg 库。
|
||||
|
||||
### GPU
|
||||
|
||||
GPU execution requires the following NVIDIA libraries to be installed:
|
||||
GPU 执行需要安装以下 NVIDIA 库:
|
||||
|
||||
* [cuBLAS for CUDA 12](https://developer.nvidia.com/cublas)
|
||||
* [cuDNN 9 for CUDA 12](https://developer.nvidia.com/cudnn)
|
||||
|
||||
**Note**: The latest versions of `ctranslate2` only support CUDA 12 and cuDNN 9. For CUDA 11 and cuDNN 8, the current workaround is downgrading to the `3.24.0` version of `ctranslate2`, for CUDA 12 and cuDNN 8, downgrade to the `4.4.0` version of `ctranslate2`, (This can be done with `pip install --force-reinstall ctranslate2==4.4.0` or specifying the version in a `requirements.txt`).
|
||||
**注意**:最新版本的 `ctranslate2` 仅支持 CUDA 12 和 cuDNN 9。对于 CUDA 11 和 cuDNN 8,目前的变通方法是降级到 `ctranslate2` 的 `3.24.0` 版本;对于 CUDA 12 和 cuDNN 8,则降级到 `ctranslate2` 的 `4.4.0` 版本(可通过 `pip install --force-reinstall ctranslate2==4.4.0` 完成,或在 `requirements.txt` 中指定版本)。
|
||||
|
||||
There are multiple ways to install the NVIDIA libraries mentioned above. The recommended way is described in the official NVIDIA documentation, but we also suggest other installation methods below.
|
||||
安装上述 NVIDIA 库有多种方式。推荐方式见官方 NVIDIA 文档,但我们也建议在下方采用其他安装方法。
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
<summary>其他安装方法(点击展开)</summary>
|
||||
|
||||
|
||||
**Note:** For all these methods below, keep in mind the above note regarding CUDA versions. Depending on your setup, you may need to install the _CUDA 11_ versions of libraries that correspond to the CUDA 12 libraries listed in the instructions below.
|
||||
**注意:** 对于以下所有方法,请记住上述关于 CUDA 版本的说明。根据你的环境,你可能需要安装与下方说明中列出的 CUDA 12 库相对应的 _CUDA 11_ 版本库。
|
||||
|
||||
#### Use Docker
|
||||
#### 使用 Docker
|
||||
|
||||
The libraries (cuBLAS, cuDNN) are installed in this official NVIDIA CUDA Docker images: `nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`.
|
||||
这些库(cuBLAS、cuDNN)已安装在官方 NVIDIA CUDA Docker 镜像中:`nvidia/cuda:12.3.2-cudnn9-runtime-ubuntu22.04`。
|
||||
|
||||
#### Install with `pip` (Linux only)
|
||||
#### 使用 `pip` 安装(仅 Linux)
|
||||
|
||||
On Linux these libraries can be installed with `pip`. Note that `LD_LIBRARY_PATH` must be set before launching Python.
|
||||
在 Linux 上,可以使用 `pip` 安装这些库。请注意,在启动 Python 之前必须设置 `LD_LIBRARY_PATH`。
|
||||
|
||||
```bash
|
||||
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
@@ -91,30 +97,30 @@ pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
|
||||
export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
|
||||
```
|
||||
|
||||
#### Download the libraries from Purfview's repository (Windows & Linux)
|
||||
#### 从 Purfview 的仓库下载库(Windows 和 Linux)
|
||||
|
||||
Purfview's [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) provides the required NVIDIA libraries for Windows & Linux in a [single archive](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). Decompress the archive and place the libraries in a directory included in the `PATH`.
|
||||
Purfview 的 [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) 为 Windows 和 Linux 提供了所需的 NVIDIA 库,打包在[单个压缩包](https://github.com/Purfview/whisper-standalone-win/releases/tag/libs). 中。解压后将库放置在包含在 `PATH` 中的目录里。
|
||||
|
||||
</details>
|
||||
|
||||
## Installation
|
||||
## 安装
|
||||
|
||||
The module can be installed from [PyPI](https://pypi.org/project/faster-whisper/):
|
||||
该模块可从 [PyPI](https://pypi.org/project/faster-whisper/): 安装:
|
||||
|
||||
```bash
|
||||
pip install faster-whisper
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Other installation methods (click to expand)</summary>
|
||||
<summary>其他安装方法(点击展开)</summary>
|
||||
|
||||
### Install the master branch
|
||||
### 安装 master 分支
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/refs/heads/master.tar.gz"
|
||||
```
|
||||
|
||||
### Install a specific commit
|
||||
### 安装特定提交
|
||||
|
||||
```bash
|
||||
pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faster-whisper/archive/a4f1cc8f11433e454c3934442b5e1a4ed5e865c3.tar.gz"
|
||||
@@ -122,7 +128,7 @@ pip install --force-reinstall "faster-whisper @ https://github.com/SYSTRAN/faste
|
||||
|
||||
</details>
|
||||
|
||||
## Usage
|
||||
## 用法
|
||||
|
||||
### Faster-whisper
|
||||
|
||||
@@ -147,15 +153,16 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
**Warning:** `segments` is a *generator* so the transcription only starts when you iterate over it. The transcription can be run to completion by gathering the segments in a list or a `for` loop:
|
||||
**警告:** `segments` 是一个*生成器*(generator),因此只有在你对其进行迭代时,转录才会开始。可以通过将片段收集到列表中,或使用 `for` 循环,来运行转录直至完成:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3")
|
||||
segments = list(segments) # The transcription will actually run here.
|
||||
```
|
||||
|
||||
### Batched Transcription
|
||||
The following code snippet illustrates how to run batched transcription on an example audio file. `BatchedInferencePipeline.transcribe` is a drop-in replacement for `WhisperModel.transcribe`
|
||||
### 批量转录
|
||||
|
||||
以下代码片段演示了如何对示例音频文件运行批量转录。`BatchedInferencePipeline.transcribe` 可直接替代 `WhisperModel.transcribe`
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel, BatchedInferencePipeline
|
||||
@@ -170,9 +177,8 @@ for segment in segments:
|
||||
|
||||
### Faster Distil-Whisper
|
||||
|
||||
The Distil-Whisper checkpoints are compatible with the Faster-Whisper package. In particular, the latest [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
checkpoint is intrinsically designed to work with the Faster-Whisper transcription algorithm. The following code snippet
|
||||
demonstrates how to run inference with distil-large-v3 on a specified audio file:
|
||||
Distil-Whisper 检查点与 Faster-Whisper 软件包兼容。特别是,最新的 [distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)
|
||||
检查点本质上就是为配合 Faster-Whisper 转录算法而设计的。以下代码片段演示了如何在指定音频文件上使用 distil-large-v3 运行推理:
|
||||
|
||||
```python
|
||||
from faster_whisper import WhisperModel
|
||||
@@ -186,9 +192,9 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
For more information about the distil-large-v3 model, refer to the original [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
有关 distil-large-v3 模型的更多信息,请参阅原始 [model card](https://huggingface.co/distil-whisper/distil-large-v3).
|
||||
|
||||
### Word-level timestamps
|
||||
### 词级时间戳(Word-level timestamps)
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", word_timestamps=True)
|
||||
@@ -198,15 +204,15 @@ for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (word.start, word.end, word.word))
|
||||
```
|
||||
|
||||
### VAD filter
|
||||
### VAD 滤波器(VAD filter)
|
||||
|
||||
The library integrates the [Silero VAD](https://github.com/snakers4/silero-vad) model to filter out parts of the audio without speech:
|
||||
该库集成了 [Silero VAD](https://github.com/snakers4/silero-vad) 模型,用于滤除音频中没有语音的部分:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe("audio.mp3", vad_filter=True)
|
||||
```
|
||||
|
||||
The default behavior is conservative and only removes silence longer than 2 seconds. See the available VAD parameters and default values in the [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). They can be customized with the dictionary argument `vad_parameters`:
|
||||
默认行为较为保守,仅移除长于 2 秒的静音。可在 [source code](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/vad.py). 中查看可用的 VAD 参数及其默认值。可通过字典参数 `vad_parameters` 进行自定义:
|
||||
|
||||
```python
|
||||
segments, _ = model.transcribe(
|
||||
@@ -215,11 +221,11 @@ segments, _ = model.transcribe(
|
||||
vad_parameters=dict(min_silence_duration_ms=500),
|
||||
)
|
||||
```
|
||||
Vad filter is enabled by default for batched transcription.
|
||||
批量转写默认启用 Vad 滤波器。
|
||||
|
||||
### Logging
|
||||
### 日志(Logging)
|
||||
|
||||
The library logging level can be configured like this:
|
||||
可按如下方式配置库的日志级别:
|
||||
|
||||
```python
|
||||
import logging
|
||||
@@ -228,37 +234,37 @@ logging.basicConfig()
|
||||
logging.getLogger("faster_whisper").setLevel(logging.DEBUG)
|
||||
```
|
||||
|
||||
### Going further
|
||||
### 进一步探索(Going further)
|
||||
|
||||
See more model and transcription options in the [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) class implementation.
|
||||
在 [`WhisperModel`](https://github.com/SYSTRAN/faster-whisper/blob/master/faster_whisper/transcribe.py) 类实现中查看更多模型与转写选项。
|
||||
|
||||
## Community integrations
|
||||
## 社区集成(Community integrations)
|
||||
|
||||
Here is a non exhaustive list of open-source projects using faster-whisper. Feel free to add your project to the list!
|
||||
以下是使用 faster-whisper 的开源项目列表(非穷尽)。欢迎将你的项目添加到列表中!
|
||||
|
||||
|
||||
* [speaches](https://github.com/speaches-ai/speaches) is an OpenAI compatible server using `faster-whisper`. It's easily deployable with Docker, works with OpenAI SDKs/CLI, supports streaming, and live transcription.
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) is an award-winning Python library that offers speaker diarization and accurate word-level timestamps using wav2vec2 alignment
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) is a command line client based on faster-whisper and compatible with the original client from openai/whisper.
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) is a speaker diarization tool that is based on faster-whisper and NVIDIA NeMo.
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) Standalone CLI executables of faster-whisper for Windows, Linux & macOS.
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) provides a scalable, modular, end to end multi-speaker speech to text solution implemented using AzureML pipelines.
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) is a Python library that transcribes voice files using faster-whisper, and translates/polishes the resulting text into `.lrc` files in the desired language using OpenAI-GPT.
|
||||
* [wscribe](https://github.com/geekodour/wscribe) is a flexible transcript generation tool supporting faster-whisper, it can export word level transcript and the exported transcript then can be edited with [wscribe-editor](https://github.com/geekodour/wscribe-editor)
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) is a graphical user interface implementation of faster-whisper developed at the BANDAS-Center at the University of Graz for transcription and diarization in Windows ([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)) and Linux.
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) implements real-time mode for offline Whisper-like speech-to-text models with faster-whisper as the most recommended back-end. It implements a streaming policy with self-adaptive latency based on the actual source complexity, and demonstrates the state of the art.
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) is a nearly-live implementation of OpenAI's Whisper which uses faster-whisper as the backend to transcribe audio in real-time.
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) is a simple but reliable voice transcriber that provides a user-friendly interface.
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) is open dubbing is an AI dubbing system which uses machine learning models to automatically translate and synchronize audio dialogue into different languages.
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi is a very simple script that provides an API backend compatible with OpenAI, HomeAssistant, and Konele (Android voice typing) formats.
|
||||
* [speaches](https://github.com/speaches-ai/speaches) 是基于 `faster-whisper` 的 OpenAI 兼容服务器。它易于通过 Docker 部署,可与 OpenAI SDKs/CLI 配合使用,并支持流式传输与实时转写。
|
||||
* [WhisperX](https://github.com/m-bain/whisperX) 是一款屡获殊荣的 Python 库,通过 wav2vec2 对齐提供说话人分离(speaker diarization)和精确的词级时间戳
|
||||
* [whisper-ctranslate2](https://github.com/Softcatala/whisper-ctranslate2) 是基于 faster-whisper 的命令行客户端,并与 openai/whisper 的原始客户端兼容。
|
||||
* [whisper-diarize](https://github.com/MahmoudAshraf97/whisper-diarization) 是一款基于 faster-whisper 和 NVIDIA NeMo 的说话人分离工具。
|
||||
* [whisper-standalone-win](https://github.com/Purfview/whisper-standalone-win) faster-whisper 在 Windows、Linux 和 macOS 上的独立 CLI 可执行文件。
|
||||
* [asr-sd-pipeline](https://github.com/hedrergudene/asr-sd-pipeline) 提供基于 AzureML pipelines 实现的可扩展、模块化、端到端的多说话人语音转文本解决方案。
|
||||
* [Open-Lyrics](https://github.com/zh-plus/Open-Lyrics) 是一款 Python 库,使用 faster-whisper 转写语音文件,并利用 OpenAI-GPT 将结果文本翻译/润色为所需语言的 `.lrc` 文件。
|
||||
* [wscribe](https://github.com/geekodour/wscribe) 是一款灵活的转录生成工具,支持 faster-whisper,可导出词级转录,导出的转录可使用 [wscribe-editor](https://github.com/geekodour/wscribe-editor) 进行编辑
|
||||
* [aTrain](https://github.com/BANDAS-Center/aTrain) 是格拉茨大学 BANDAS-Center 开发的 faster-whisper 图形用户界面实现,用于在 Windows([Windows Store App](https://apps.microsoft.com/detail/atrain/9N15Q44SZNS2)))和 Linux 上进行转写与说话人分离。
|
||||
* [Whisper-Streaming](https://github.com/ufal/whisper_streaming) 为离线类 Whisper 语音转文本模型实现实时模式,并以 faster-whisper 作为最推荐的后端。它实现了基于实际源复杂度的自适应延迟流式策略,并展示了当前最先进水平。
|
||||
* [WhisperLive](https://github.com/collabora/WhisperLive) 是 OpenAI Whisper 的近实时实现,使用 faster-whisper 作为后端实时转写音频。
|
||||
* [Faster-Whisper-Transcriber](https://github.com/BBC-Esq/ctranslate2-faster-whisper-transcriber) 是一款简单但可靠的语音转写器,提供用户友好的界面。
|
||||
* [Open-dubbing](https://github.com/softcatala/open-dubbing) open dubbing 是一款 AI 配音系统,使用机器学习模型自动将音频对话翻译并同步到不同语言。
|
||||
* [Whisper-FastAPI](https://github.com/heimoshuiyu/whisper-fastapi) whisper-fastapi 是一个非常简单的脚本,提供兼容 OpenAI、HomeAssistant 和 Konele(Android 语音输入)格式的 API 后端。
|
||||
|
||||
## Model conversion
|
||||
## 模型转换(Model conversion)
|
||||
|
||||
When loading a model from its size such as `WhisperModel("large-v3")`, the corresponding CTranslate2 model is automatically downloaded from the [Hugging Face Hub](https://huggingface.co/Systran).
|
||||
当按尺寸加载模型(例如 `WhisperModel("large-v3")`)时,相应的 CTranslate2 模型会自动从 [Hugging Face Hub](https://huggingface.co/Systran). 下载。
|
||||
|
||||
We also provide a script to convert any Whisper models compatible with the Transformers library. They could be the original OpenAI models or user fine-tuned models.
|
||||
我们还提供了一个脚本,用于转换任何与 Transformers 库兼容的 Whisper 模型。这些模型可以是原始 OpenAI 模型,也可以是用户微调后的模型。
|
||||
|
||||
For example the command below converts the [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) and saves the weights in FP16:
|
||||
例如,下面的命令会转换 [original "large-v3" Whisper model](https://huggingface.co/openai/whisper-large-v3) 并以 FP16 格式保存权重:
|
||||
|
||||
```bash
|
||||
pip install transformers[torch]>=4.23
|
||||
@@ -267,30 +273,30 @@ ct2-transformers-converter --model openai/whisper-large-v3 --output_dir whisper-
|
||||
--copy_files tokenizer.json preprocessor_config.json --quantization float16
|
||||
```
|
||||
|
||||
* The option `--model` accepts a model name on the Hub or a path to a model directory.
|
||||
* If the option `--copy_files tokenizer.json` is not used, the tokenizer configuration is automatically downloaded when the model is loaded later.
|
||||
* 选项 `--model` 接受 Hub 上的模型名称或模型目录路径。
|
||||
* 如果未使用选项 `--copy_files tokenizer.json`,则在后续加载模型时会自动下载 tokenizer 配置。
|
||||
|
||||
Models can also be converted from the code. See the [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
也可以从代码中进行模型转换。请参阅 [conversion API](https://opennmt.net/CTranslate2/python/ctranslate2.converters.TransformersConverter.html).
|
||||
|
||||
### Load a converted model
|
||||
### 加载已转换的模型
|
||||
|
||||
1. Directly load the model from a local directory:
|
||||
1. 直接从本地目录加载模型:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) and load it from its name:
|
||||
2. [Upload your model to the Hugging Face Hub](https://huggingface.co/docs/transformers/model_sharing#upload-with-the-web-interface) 并按名称加载:
|
||||
```python
|
||||
model = faster_whisper.WhisperModel("username/whisper-large-v3-ct2")
|
||||
```
|
||||
|
||||
## Comparing performance against other implementations
|
||||
## 与其他实现的性能对比
|
||||
|
||||
If you are comparing the performance against other Whisper implementations, you should make sure to run the comparison with similar settings. In particular:
|
||||
如果你正在将性能与其他 Whisper 实现进行对比,应确保在相似设置下运行对比。特别要注意:
|
||||
|
||||
* Verify that the same transcription options are used, especially the same beam size. For example in openai/whisper, `model.transcribe` uses a default beam size of 1 but here we use a default beam size of 5.
|
||||
* Transcription speed is closely affected by the number of words in the transcript, so ensure that other implementations have a similar WER (Word Error Rate) to this one.
|
||||
* When running on CPU, make sure to set the same number of threads. Many frameworks will read the environment variable `OMP_NUM_THREADS`, which can be set when running your script:
|
||||
* 确认使用相同的转写选项,尤其是相同的 beam size。例如在 openai/whisper 中,`model.transcribe` 默认 beam size 为 1,而此处默认 beam size 为 5。
|
||||
* 转写速度与转录文本中的词数密切相关,因此请确保其他实现与本实现的 WER(Word Error Rate,词错误率)相近。
|
||||
* 在 CPU 上运行时,请确保设置相同的线程数。许多框架会读取环境变量 `OMP_NUM_THREADS`,可在运行脚本时设置:
|
||||
|
||||
```bash
|
||||
OMP_NUM_THREADS=4 python3 my_script.py
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目:`SYSTRAN/faster-whisper`
|
||||
- 原始仓库:https://github.com/SYSTRAN/faster-whisper
|
||||
- 导入方式:上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
|
||||
Binary file not shown.
@@ -418,23 +418,34 @@ class BatchedInferencePipeline:
|
||||
"Set 'vad_filter' to True or provide 'clip_timestamps'."
|
||||
)
|
||||
|
||||
clip_timestamps_provided = False
|
||||
audio_chunks, chunks_metadata = collect_chunks(
|
||||
audio, clip_timestamps, max_duration=chunk_length
|
||||
)
|
||||
|
||||
else:
|
||||
clip_timestamps_provided = True
|
||||
clip_timestamps = [
|
||||
{k: int(v * sampling_rate) for k, v in segment.items()}
|
||||
for segment in clip_timestamps
|
||||
]
|
||||
|
||||
audio_chunks, chunks_metadata = [], []
|
||||
for clip in clip_timestamps:
|
||||
for i, clip in enumerate(clip_timestamps):
|
||||
audio_chunks.append(audio[clip["start"] : clip["end"]])
|
||||
|
||||
clip_duration = (clip["end"] - clip["start"]) / sampling_rate
|
||||
if clip_duration > 30:
|
||||
self.model.logger.warning(
|
||||
"Segment %d is longer than 30 seconds, "
|
||||
"only the first 30 seconds will be transcribed",
|
||||
i,
|
||||
)
|
||||
|
||||
chunks_metadata.append(
|
||||
{
|
||||
"offset": clip["start"] / sampling_rate,
|
||||
"duration": (clip["end"] - clip["start"]) / sampling_rate,
|
||||
"duration": clip_duration,
|
||||
"segments": [clip],
|
||||
}
|
||||
)
|
||||
@@ -559,7 +570,10 @@ class BatchedInferencePipeline:
|
||||
options,
|
||||
log_progress,
|
||||
)
|
||||
segments = restore_speech_timestamps(segments, clip_timestamps, sampling_rate)
|
||||
if not clip_timestamps_provided:
|
||||
segments = restore_speech_timestamps(
|
||||
segments, clip_timestamps, sampling_rate
|
||||
)
|
||||
|
||||
return segments, info
|
||||
|
||||
|
||||
+1
-20
@@ -5,7 +5,6 @@ import re
|
||||
from typing import List, Optional, Union
|
||||
|
||||
import huggingface_hub
|
||||
import requests
|
||||
|
||||
from tqdm.auto import tqdm
|
||||
|
||||
@@ -106,7 +105,6 @@ def download_model(
|
||||
|
||||
if output_dir is not None:
|
||||
kwargs["local_dir"] = output_dir
|
||||
kwargs["local_dir_use_symlinks"] = False
|
||||
|
||||
if cache_dir is not None:
|
||||
kwargs["cache_dir"] = cache_dir
|
||||
@@ -114,24 +112,7 @@ def download_model(
|
||||
if use_auth_token is not None:
|
||||
kwargs["token"] = use_auth_token
|
||||
|
||||
try:
|
||||
return huggingface_hub.snapshot_download(repo_id, **kwargs)
|
||||
except (
|
||||
huggingface_hub.utils.HfHubHTTPError,
|
||||
requests.exceptions.ConnectionError,
|
||||
) as exception:
|
||||
logger = get_logger()
|
||||
logger.warning(
|
||||
"An error occured while synchronizing the model %s from the Hugging Face Hub:\n%s",
|
||||
repo_id,
|
||||
exception,
|
||||
)
|
||||
logger.warning(
|
||||
"Trying to load the model directly from the local cache, if it exists."
|
||||
)
|
||||
|
||||
kwargs["local_files_only"] = True
|
||||
return huggingface_hub.snapshot_download(repo_id, **kwargs)
|
||||
return huggingface_hub.snapshot_download(repo_id, **kwargs)
|
||||
|
||||
|
||||
def format_timestamp(
|
||||
|
||||
+58
-24
@@ -27,11 +27,15 @@ class VadOptions:
|
||||
min_speech_duration_ms: Final speech chunks shorter min_speech_duration_ms are thrown out.
|
||||
max_speech_duration_s: Maximum duration of speech chunks in seconds. Chunks longer
|
||||
than max_speech_duration_s will be split at the timestamp of the last silence that
|
||||
lasts more than 100ms (if any), to prevent aggressive cutting. Otherwise, they will be
|
||||
split aggressively just before max_speech_duration_s.
|
||||
lasts more than min_silence_at_max_speech (if any), to prevent aggressive cutting.
|
||||
Otherwise, they will be split aggressively just before max_speech_duration_s.
|
||||
min_silence_duration_ms: In the end of each speech chunk wait for min_silence_duration_ms
|
||||
before separating it
|
||||
speech_pad_ms: Final speech chunks are padded by speech_pad_ms each side
|
||||
min_silence_at_max_speech: Minimum silence duration in ms which is used to avoid abrupt cuts
|
||||
when max_speech_duration_s is reached.
|
||||
use_max_poss_sil_at_max_speech: Whether to use the maximum possible silence at
|
||||
max_speech_duration_s or not. If not, the last silence is used.
|
||||
"""
|
||||
|
||||
threshold: float = 0.5
|
||||
@@ -40,6 +44,8 @@ class VadOptions:
|
||||
max_speech_duration_s: float = float("inf")
|
||||
min_silence_duration_ms: int = 2000
|
||||
speech_pad_ms: int = 400
|
||||
min_silence_at_max_speech: int = 98
|
||||
use_max_poss_sil_at_max_speech: bool = True
|
||||
|
||||
|
||||
def get_speech_timestamps(
|
||||
@@ -69,6 +75,9 @@ def get_speech_timestamps(
|
||||
min_silence_duration_ms = vad_options.min_silence_duration_ms
|
||||
window_size_samples = 512
|
||||
speech_pad_ms = vad_options.speech_pad_ms
|
||||
min_silence_at_max_speech = vad_options.min_silence_at_max_speech
|
||||
use_max_poss_sil_at_max_speech = vad_options.use_max_poss_sil_at_max_speech
|
||||
|
||||
min_speech_samples = sampling_rate * min_speech_duration_ms / 1000
|
||||
speech_pad_samples = sampling_rate * speech_pad_ms / 1000
|
||||
max_speech_samples = (
|
||||
@@ -77,7 +86,7 @@ def get_speech_timestamps(
|
||||
- 2 * speech_pad_samples
|
||||
)
|
||||
min_silence_samples = sampling_rate * min_silence_duration_ms / 1000
|
||||
min_silence_samples_at_max_speech = sampling_rate * 98 / 1000
|
||||
min_silence_samples_at_max_speech = sampling_rate * min_silence_at_max_speech / 1000
|
||||
|
||||
audio_length_samples = len(audio)
|
||||
|
||||
@@ -91,6 +100,8 @@ def get_speech_timestamps(
|
||||
triggered = False
|
||||
speeches = []
|
||||
current_speech = {}
|
||||
possible_ends = []
|
||||
|
||||
if neg_threshold is None:
|
||||
neg_threshold = max(threshold - 0.15, 0.01)
|
||||
|
||||
@@ -100,45 +111,67 @@ def get_speech_timestamps(
|
||||
prev_end = next_start = 0
|
||||
|
||||
for i, speech_prob in enumerate(speech_probs):
|
||||
cur_sample = window_size_samples * i
|
||||
|
||||
if (speech_prob >= threshold) and temp_end:
|
||||
sil_dur = cur_sample - temp_end
|
||||
if sil_dur > min_silence_samples_at_max_speech:
|
||||
possible_ends.append((temp_end, sil_dur))
|
||||
temp_end = 0
|
||||
if next_start < prev_end:
|
||||
next_start = window_size_samples * i
|
||||
next_start = cur_sample
|
||||
|
||||
if (speech_prob >= threshold) and not triggered:
|
||||
triggered = True
|
||||
current_speech["start"] = window_size_samples * i
|
||||
current_speech["start"] = cur_sample
|
||||
continue
|
||||
|
||||
if (
|
||||
triggered
|
||||
and (window_size_samples * i) - current_speech["start"] > max_speech_samples
|
||||
):
|
||||
if prev_end:
|
||||
if triggered and (cur_sample - current_speech["start"] > max_speech_samples):
|
||||
if use_max_poss_sil_at_max_speech and possible_ends:
|
||||
prev_end, dur = max(possible_ends, key=lambda x: x[1])
|
||||
current_speech["end"] = prev_end
|
||||
speeches.append(current_speech)
|
||||
current_speech = {}
|
||||
# previously reached silence (< neg_thres) and is still not speech (< thres)
|
||||
if next_start < prev_end:
|
||||
triggered = False
|
||||
else:
|
||||
next_start = prev_end + dur
|
||||
|
||||
if next_start < prev_end + cur_sample:
|
||||
current_speech["start"] = next_start
|
||||
else:
|
||||
triggered = False
|
||||
prev_end = next_start = temp_end = 0
|
||||
possible_ends = []
|
||||
else:
|
||||
current_speech["end"] = window_size_samples * i
|
||||
speeches.append(current_speech)
|
||||
current_speech = {}
|
||||
prev_end = next_start = temp_end = 0
|
||||
triggered = False
|
||||
continue
|
||||
if prev_end:
|
||||
current_speech["end"] = prev_end
|
||||
speeches.append(current_speech)
|
||||
current_speech = {}
|
||||
if next_start < prev_end:
|
||||
triggered = False
|
||||
else:
|
||||
current_speech["start"] = next_start
|
||||
prev_end = next_start = temp_end = 0
|
||||
possible_ends = []
|
||||
else:
|
||||
current_speech["end"] = cur_sample
|
||||
speeches.append(current_speech)
|
||||
current_speech = {}
|
||||
prev_end = next_start = temp_end = 0
|
||||
triggered = False
|
||||
possible_ends = []
|
||||
continue
|
||||
|
||||
if (speech_prob < neg_threshold) and triggered:
|
||||
if not temp_end:
|
||||
temp_end = window_size_samples * i
|
||||
# condition to avoid cutting in very short silence
|
||||
if (window_size_samples * i) - temp_end > min_silence_samples_at_max_speech:
|
||||
temp_end = cur_sample
|
||||
sil_dur_now = cur_sample - temp_end
|
||||
|
||||
if (
|
||||
not use_max_poss_sil_at_max_speech
|
||||
and sil_dur_now > min_silence_samples_at_max_speech
|
||||
):
|
||||
prev_end = temp_end
|
||||
if (window_size_samples * i) - temp_end < min_silence_samples:
|
||||
|
||||
if sil_dur_now < min_silence_samples:
|
||||
continue
|
||||
else:
|
||||
current_speech["end"] = temp_end
|
||||
@@ -149,6 +182,7 @@ def get_speech_timestamps(
|
||||
current_speech = {}
|
||||
prev_end = next_start = temp_end = 0
|
||||
triggered = False
|
||||
possible_ends = []
|
||||
continue
|
||||
|
||||
if (
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Version information."""
|
||||
|
||||
__version__ = "1.2.0"
|
||||
__version__ = "1.2.1"
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
ctranslate2>=4.0,<5
|
||||
huggingface_hub>=0.13
|
||||
huggingface_hub>=0.23
|
||||
tokenizers>=0.13,<1
|
||||
onnxruntime>=1.14,<2
|
||||
av>=11
|
||||
tqdm
|
||||
tqdm
|
||||
|
||||
@@ -45,13 +45,13 @@ setup(
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3 :: Only",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||
],
|
||||
keywords="openai whisper speech ctranslate2 inference quantization transformer",
|
||||
python_requires=">=3.10",
|
||||
python_requires=">=3.9",
|
||||
install_requires=install_requires,
|
||||
extras_require={
|
||||
"conversion": conversion_requires,
|
||||
|
||||
@@ -245,7 +245,7 @@ def test_transcribe_signature():
|
||||
|
||||
|
||||
def test_monotonic_timestamps(physcisworks_path):
|
||||
model = WhisperModel("tiny")
|
||||
model = WhisperModel("base")
|
||||
pipeline = BatchedInferencePipeline(model=model)
|
||||
|
||||
segments, info = model.transcribe(physcisworks_path, word_timestamps=True)
|
||||
@@ -290,3 +290,26 @@ def test_cliptimestamps_segments(jfk_path):
|
||||
" And so my fellow Americans ask not what your country can do for you, "
|
||||
"ask what you can do for your country."
|
||||
)
|
||||
|
||||
|
||||
def test_cliptimestamps_timings(physcisworks_path):
|
||||
model = WhisperModel("tiny")
|
||||
pipeline = BatchedInferencePipeline(model=model)
|
||||
|
||||
audio = decode_audio(physcisworks_path)
|
||||
clip_timestamps = [{"start": 0.0, "end": 5.0}, {"start": 6.0, "end": 15.0}]
|
||||
transcripts = [
|
||||
" Now I want to return to the conservation of mechanical energy.",
|
||||
(
|
||||
" I have here a pendulum. I have an object that weighs 15 kilograms"
|
||||
" and I can lift it up one meter, which I have done now."
|
||||
),
|
||||
]
|
||||
segments, info = pipeline.transcribe(audio, clip_timestamps=clip_timestamps)
|
||||
segments = list(segments)
|
||||
|
||||
assert len(segments) == 2
|
||||
for segment, clip, transcript in zip(segments, clip_timestamps, transcripts):
|
||||
assert clip["start"] == segment.start
|
||||
assert clip["end"] == segment.end
|
||||
assert segment.text == transcript
|
||||
|
||||
Reference in New Issue
Block a user