chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
+15
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-audio
|
||||
slug: "/external-integrations-audio"
|
||||
description: "External integrations that enable working with audio in Haystack by transcribing files or converting text to audio."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable working with audio in Haystack by transcribing files or converting text to audio.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [AssemblyAI](https://haystack.deepset.ai/integrations/assemblyai) | Perform speech recognition, speaker diarization and summarization. |
|
||||
| [Elevenlabs](https://haystack.deepset.ai/integrations/elevenlabs) | Convert text to speech using ElevenLabs’ API. |
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "FunASRTranscriber"
|
||||
id: funasrtranscriber
|
||||
slug: "/funasrtranscriber"
|
||||
description: "Transcribe audio files to Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages."
|
||||
---
|
||||
|
||||
# FunASRTranscriber
|
||||
|
||||
Transcribe audio files to Haystack Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of audio file paths (`str` or `Path`) or `ByteStream` objects |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents, one per source, with transcript text in `content` |
|
||||
| **API reference** | [FunASR integration](/reference/integrations-funasr) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/funasr/src/haystack_integrations/components/audio/funasr/transcriber.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FunASRTranscriber` uses [FunASR](https://github.com/modelscope/FunASR), an open-source speech recognition toolkit from Alibaba DAMO Academy, to transcribe audio files into Haystack `Document` objects. It runs entirely locally — no API key required.
|
||||
|
||||
The default model is `iic/SenseVoiceSmall`, a multilingual model supporting 50+ languages that is 5–10x faster than Whisper. Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
|
||||
|
||||
The component accepts audio file paths (`str` or `Path`) as well as `ByteStream` objects. Call `warm_up()` before running in a pipeline to load the model into memory.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.funasr import FunASRTranscriber
|
||||
|
||||
transcriber = FunASRTranscriber()
|
||||
transcriber.warm_up()
|
||||
|
||||
result = transcriber.run(sources=["speech.wav"])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack_integrations.components.audio.funasr import FunASRTranscriber
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", FunASRTranscriber())
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": ["https://example.com/interview.wav"],
|
||||
},
|
||||
}
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "LocalWhisperTranscriber"
|
||||
id: localwhispertranscriber
|
||||
slug: "/localwhispertranscriber"
|
||||
description: "Use `LocalWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model using your local installation of Whisper."
|
||||
---
|
||||
|
||||
# LocalWhisperTranscriber
|
||||
|
||||
Use `LocalWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model using your local installation of Whisper.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of paths or binary streams that you want to transcribe |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Whisper](/reference/integrations-whisper) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/whisper |
|
||||
| **Package name** | `whisper-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The component also needs to know which Whisper model to work with. Specify this in the `model` parameter when initializing the component. All transcription is completed on the executing machine, and the audio is never sent to a third-party provider.
|
||||
|
||||
See other optional parameters you can specify in our [API documentation](/reference/integrations-whisper).
|
||||
|
||||
See the [Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper [GitHub repo](https://github.com/openai/whisper) for the supported audio formats and languages.
|
||||
|
||||
The `LocalWhisperTranscriber` is part of the `whisper-haystack` integration package. To work with it, install the package along with [Whisper](https://github.com/openai/whisper) (which also pulls in torch) using the following commands:
|
||||
|
||||
```python
|
||||
pip install whisper-haystack
|
||||
pip install -U openai-whisper
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here’s an example of how to use `LocalWhisperTranscriber` on its own:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from haystack_integrations.components.audio.whisper import LocalWhisperTranscriber
|
||||
|
||||
response = requests.get(
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
)
|
||||
with open("kennedy_speech.mp3", "wb") as file:
|
||||
file.write(response.content)
|
||||
|
||||
transcriber = LocalWhisperTranscriber(model="tiny")
|
||||
|
||||
transcription = transcriber.run(sources=["./kennedy_speech.mp3"])
|
||||
print(transcription["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The pipeline below fetches an audio file from a specified URL and transcribes it. It first retrieves the audio file using `LinkContentFetcher`, then transcribes the audio into text with `LocalWhisperTranscriber`, and finally outputs the transcription text.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import LocalWhisperTranscriber
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack import Pipeline
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", LocalWhisperTranscriber(model="tiny"))
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Multilingual RAG from a podcast with Whisper, Qdrant and Mistral](https://haystack.deepset.ai/cookbook/multilingual_rag_podcast)
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: "RemoteWhisperTranscriber"
|
||||
id: remotewhispertranscriber
|
||||
slug: "/remotewhispertranscriber"
|
||||
description: "Use `RemoteWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model."
|
||||
---
|
||||
|
||||
# RemoteWhisperTranscriber
|
||||
|
||||
Use `RemoteWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with an environment variable `OPENAI_API_KEY`. |
|
||||
| **Mandatory run variables** | `sources`: A list of paths or binary streams that you want to transcribe |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Whisper](/reference/integrations-whisper) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/whisper |
|
||||
| **Package name** | `whisper-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `RemoteWhisperTranscriber` is part of the `whisper-haystack` integration package. Install it with:
|
||||
|
||||
```python
|
||||
pip install whisper-haystack
|
||||
```
|
||||
|
||||
`RemoteWhisperTranscriber` works with OpenAI-compatible clients and isn't limited to just OpenAI as a provider. For example, [Groq](https://console.groq.com/docs/speech-text) offers a drop-in replacement that can be used as well. You can set the API key in one of two ways:
|
||||
|
||||
1. Through the `api_key` initialization parameter, where the key is resolved using [Secret API](../../concepts/secret-management.mdx).
|
||||
2. By setting it in the `OPENAI_API_KEY` environment variable, which the system will use to access the key.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
|
||||
transcriber = RemoteWhisperTranscriber()
|
||||
```
|
||||
|
||||
Additionally, the component requires the following parameters to work:
|
||||
|
||||
- `model` specifies the Whisper model.
|
||||
- `api_base_url` specifies the OpenAI base URL and defaults to `"https://api.openai.com/v1"`. If you are using Whisper provider other than OpenAI set this parameter according to provider's documentation.
|
||||
|
||||
See other optional parameters in our [API documentation](/reference/integrations-whisper).
|
||||
|
||||
See the [Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper [GitHub repo](https://github.com/openai/whisper) for the supported audio formats and languages.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here’s an example of how to use `RemoteWhisperTranscriber` to transcribe a local file:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
|
||||
response = requests.get(
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
)
|
||||
with open("kennedy_speech.mp3", "wb") as file:
|
||||
file.write(response.content)
|
||||
|
||||
transcriber = RemoteWhisperTranscriber()
|
||||
transcription = transcriber.run(sources=["./kennedy_speech.mp3"])
|
||||
|
||||
print(transcription["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The pipeline below fetches an audio file from a specified URL and transcribes it. It first retrieves the audio file using `LinkContentFetcher`, then transcribes the audio into text with `RemoteWhisperTranscriber`, and finally outputs the transcription text.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack import Pipeline
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", RemoteWhisperTranscriber())
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Multilingual RAG from a podcast with Whisper, Qdrant and Mistral](https://haystack.deepset.ai/cookbook/multilingual_rag_podcast)
|
||||
Reference in New Issue
Block a user