chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:22:28 +08:00
commit c56bef871b
9296 changed files with 1854228 additions and 0 deletions
@@ -0,0 +1,73 @@
---
title: "ArcadeDBDocumentStore"
id: arcadedbdocumentstore
slug: "/arcadedbdocumentstore"
---
# ArcadeDBDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [ArcadeDB](/reference/integrations-arcadedb) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/arcadedb |
</div>
ArcadeDB is a multi-model database that supports vector search via its LSM_VECTOR (HNSW) index. The `ArcadeDBDocumentStore` uses ArcadeDB's HTTP/JSON API for all operations—no special drivers required. It supports dense embedding retrieval and SQL-based metadata filtering.
For more information, see the [ArcadeDB documentation](https://docs.arcadedb.com/).
## Installation
Run ArcadeDB with Docker and update the password according to your setup:
```shell
docker run -d -p 2480:2480 \
-e JAVA_OPTS="-Darcadedb.server.rootPassword=arcadedb" \
arcadedata/arcadedb:latest
```
Install the Haystack integration:
```shell
pip install arcadedb-haystack
```
## Usage
Set credentials via environment variables (recommended) or pass them explicitly:
```shell
export ARCADEDB_USERNAME=root
export ARCADEDB_PASSWORD=arcadedb
```
Initialize the document store and write documents:
```python
from haystack import Document
from haystack_integrations.document_stores.arcadedb import ArcadeDBDocumentStore
document_store = ArcadeDBDocumentStore(
url="http://localhost:2480",
database="haystack",
embedding_dimension=768,
recreate_type=True,
)
document_store.write_documents([
Document(content="This is first", embedding=[0.0] * 768),
Document(content="This is second", embedding=[0.1, 0.2, 0.3] + [0.0] * 765),
])
print(document_store.count_documents())
```
To learn more about the initialization parameters, see the [API docs](/reference/integrations-arcadedb#arcadedbdocumentstore).
Documents without embeddings or with a different dimension are stored with a zero-padded vector so they can be written and filtered; use an [Embedder](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx) for real embeddings.
### Supported Retrievers
- [ArcadeDBEmbeddingRetriever](../pipeline-components/retrievers/arcadedbembeddingretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store by vector similarity (HNSW).
@@ -0,0 +1,82 @@
---
title: "AstraDocumentStore"
id: astradocumentstore
slug: "/astradocumentstore"
---
# AstraDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Astra](/reference/integrations-astra) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/astra |
</div>
DataStax Astra DB is a serverless vector database built on Apache Cassandra, and it supports vector-based search and auto-scaling. You can deploy it on AWS, GCP, or Azure and easily expand to one or more regions within those clouds for multi-region availability, low latency data access, data sovereignty, and to avoid cloud vendor lock-in. For more information, see the [DataStax documentation](https://docs.datastax.com/en/home/docs/index.html).
### Initialization
Once you have an AstraDB account and have created a database, install the `astra-haystack` integration:
```shell
pip install astra-haystack
```
From the configuration in AstraDBs web UI, you need the database ID and a generated token.
You will additionally need a collection name and a namespace. When you create the collection name, you also need to set the embedding dimensions and the similarity metric. The namespace organizes data in a database and is called a keyspace in Apache Cassandra.
Then, in Haystack, initialize an `AstraDocumentStore` object thats connected to the AstraDB instance, and write documents to it.
We strongly encourage passing authentication data through environment variables: make sure to populate the environment variables `ASTRA_DB_API_ENDPOINT` and `ASTRA_DB_APPLICATION_TOKEN` before running the following example.
```python
from haystack import Document
from haystack_integrations.document_stores.astra import AstraDocumentStore
document_store = AstraDocumentStore()
document_store.write_documents(
[Document(content="This is first"), Document(content="This is second")],
)
print(document_store.count_documents())
```
### Supported Retrievers
[AstraEmbeddingRetriever](../pipeline-components/retrievers/astraretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
### Indexing Warnings
When you create an Astra DB Document Store, you might see one of these warnings:
> Astra DB collection `...` is detected as having indexing turned on for all fields (either created manually or by older versions of this plugin). This implies stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
Or:
> Astra DB collection `...` is detected as having the following indexing policy: `{...}`. This does not match the requested indexing policy for this object: `{...}`. In particular, there may be stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
#### Why You See This Warning
The collection already exists and is configured to [index all fields for search](https://docs.datastax.com/en/astra-db-serverless/api-reference/collections.html#the-indexing-option), possibly because you created it earlier or an older plugin did. When Haystack tries to create the collection, it applies an indexing policy optimized for your intended use. This policy lets you store longer texts and avoids indexing fields you wont filter on, which also reduces write overhead.
#### Common Causes
1. You created the collection outside Haystack (for example, in the Astra UI or with AstraPys `Database.create_collection()`).
2. You created the collection with an older version of the plugin.
#### Impact
This is only a warning. Your application keeps running unless you try to store very long text fields. If you do, Astra DB returns an indexing error.
#### Solutions
- **Recommended:** _Drop and recreate the collection_ if you can repopulate it. Then rerun your Haystack application so it creates the collection with the optimized indexing policy.
- _Ignore the warning_ if youre sure you wont store very long text fields.
## Additional References
🧑‍🍳 Cookbook: [Using AstraDB as a data store in your Haystack pipelines](https://haystack.deepset.ai/cookbook/astradb_haystack_integration)
@@ -0,0 +1,70 @@
---
title: "AzureAISearchDocumentStore"
id: azureaisearchdocumentstore
slug: "/azureaisearchdocumentstore"
description: "A Document Store for storing and retrieval from Azure AI Search Index."
---
# AzureAISearchDocumentStore
A Document Store for storing and retrieval from Azure AI Search Index.
<div className="key-value-table">
| | |
| --- | --- |
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
</div>
[Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search) is an enterprise-ready search and retrieval system to build RAG-based applications on Azure, with native LLM integrations.
`AzureAISearchDocumentStore` supports semantic reranking and metadata/content filtering. The Document Store is useful for various tasks such as generating knowledge base insights (catalog or document search), information discovery (data exploration), RAG, and automation.
### Initialization
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
Once you have the subscription, install the `azure-ai-search-haystack` integration:
```python
pip install azure-ai-search-haystack
```
To use the `AzureAISearchDocumentStore`, you need to provide a search service endpoint as an `AZURE_AI_SEARCH_ENDPOINT` and an API key as `AZURE_AI_SEARCH_API_KEY` for authentication. If the API key is not provided, the `DefaultAzureCredential` will attempt to authenticate you through the browser.
During initialization the Document Store will either retrieve the existing search index for the given `index_name` or create a new one if it doesn't already exist. Note that one of the limitations of `AzureAISearchDocumentStore` is that the fields of the Azure search index cannot be modified through the API after creation. Therefore, any additional fields beyond the default ones must be provided as `metadata_fields` during the Document Store's initialization. However, if needed, [Azure AI portal](https://azure.microsoft.com/) can be used to modify the fields without deleting the index.
It is recommended to pass authentication data through `AZURE_AI_SEARCH_API_KEY` and `AZURE_AI_SEARCH_ENDPOINT` before running the following example.
```python
from haystack_integrations.document_stores.azure_ai_search import (
AzureAISearchDocumentStore,
)
from haystack import Document
document_store = AzureAISearchDocumentStore(index_name="haystack-docs")
document_store.write_documents(
[
Document(content="This is the first document."),
Document(content="This is the second document."),
],
)
print(document_store.count_documents())
```
:::info[Latency Notice]
Due to Azure search index latency, the document count returned in the example might be zero if executed immediately. To ensure accurate results, be mindful of this latency when retrieving documents from the search index.
:::
You can enable semantic reranking in `AzureAISearchDocumentStore` by providing [SemanticSearch](https://learn.microsoft.com/en-us/python/api/azure-search-documents/azure.search.documents.indexes.models.semanticsearch?view=azure-python) configuration in `index_creation_kwargs` during initialization and calling it from one of the Retrievers. For more information, refer to the [Azure AI tutorial](https://learn.microsoft.com/en-us/azure/search/search-get-started-semantic) on this feature.
### Supported Retrievers
The Haystack Azure AI Search integration includes three Retriever components. Each Retriever leverages the Azure AI Search API and you can select the one that best suits your pipeline:
- [`AzureAISearchEmbeddingRetriever`](../pipeline-components/retrievers/azureaisearchembeddingretriever.mdx): This Retriever accepts the embeddings of a single query as input and returns a list of matching documents. The query must be embedded beforehand, which can be done using an [Embedder](../pipeline-components/embedders.mdx) component.
- [`AzureAISearchBM25Retriever`](../pipeline-components/retrievers/azureaisearchbm25retriever.mdx): A keyword-based Retriever that retrieves documents matching a query from the Azure AI Search index.
- [`AzureAISearchHybridRetriever`](../pipeline-components/retrievers/azureaisearchhybridretriever.mdx): This Retriever combines embedding-based retrieval and keyword search to find matching documents in the search index to get more relevant results.
@@ -0,0 +1,97 @@
---
title: "ChromaDocumentStore"
id: chromadocumentstore
slug: "/chromadocumentstore"
---
# ChromaDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Chroma](/reference/integrations-chroma) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chroma |
</div>
[Chroma](https://docs.trychroma.com/) is an open source vector database capable of storing collections of documents along with their metadata, creating embeddings for documents and queries, and searching the collections filtering by document metadata or content. Additionally, Chroma supports multi-modal embedding functions.
Chroma can be used in-memory, as an embedded database, or in a client-server fashion. When running in-memory, Chroma can still keep its contents on disk across different sessions. This allows users to quickly put together prototypes using the in-memory version and later move to production, where the client-server version is deployed.
## Initialization
First, install the Chroma integration, which will install Haystack and Chroma if they are not already present. The following command is all you need to start:
```shell
pip install chroma-haystack
```
To store data in Chroma, create a `ChromaDocumentStore` instance and write documents with:
```python
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
from haystack import Document
document_store = ChromaDocumentStore()
document_store.write_documents(
[
Document(content="This is the first document."),
Document(content="This is the second document."),
],
)
print(document_store.count_documents())
```
In this case, since we didnt pass any embeddings along with our documents, Chroma will create them for us using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
### Connection Options
1. **In-Memory Mode (Local)**: Chroma can be set up as a local Document Store for fast and lightweight usage. You can use this option during development or small-scale experiments. Set up a local in-memory instance of `ChromaDocumentStore` like this:
```python
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
document_store = ChromaDocumentStore()
```
2. **Persistent Storage**: If you need to retain the documents between sessions, Chroma supports persistent storage by specifying a path to store data on disk:
```python
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
document_store = ChromaDocumentStore(persist_path="your_directory_path")
```
3. **Remote Connection**: You can connect to a remote Chroma database through HTTP. This is suitable for distributed setups where multiple clients might interact with the same remote Chroma instance.
Note that this option is incompatible with in-memory or persistent storage modes.
First, start a Chroma server:
```shell
chroma run --path /db_path
```
Or using docker:
```shell
docker run -p 8000:8000 chromadb/chroma
```
Then, initialize the Document Store with `host` and `port` parameters:
```python
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
document_store = ChromaDocumentStore(host="localhost", port="8000")
```
## Supported Retrievers
The Haystack Chroma integration comes with three Retriever components. They all rely on the Chroma [query API](https://docs.trychroma.com/reference/Collection#query), but they have different inputs and outputs so that you can pick the one that best fits your pipeline:
- [`ChromaQueryTextRetriever`](../pipeline-components/retrievers/chromaqueryretriever.mdx): This Retriever takes a plain-text query string in input and returns a list of matching documents. Chroma will create the embeddings for the query using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
- [`ChromaEmbeddingRetriever`](../pipeline-components/retrievers/chromaembeddingretriever.mdx): This Retriever takes the embeddings of a single query in input and returns a list of matching documents. The query needs to be embedded before being passed to this component. For example, you can use an [embedder](../pipeline-components/embedders.mdx) component.
## Additional References
🧑‍🍳 Cookbook: [Use Chroma for RAG and Indexing](https://haystack.deepset.ai/cookbook/chroma-indexing-and-rag-examples)
@@ -0,0 +1,67 @@
---
title: "ElasticsearchDocumentStore"
id: elasticsearch-document-store
slug: "/elasticsearch-document-store"
description: "Use an Elasticsearch database with Haystack."
---
# ElasticsearchDocumentStore
Use an Elasticsearch database with Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Elasticsearch](/reference/integrations-elasticsearch) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
</div>
ElasticsearchDocumentStore is excellent if you want to evaluate the performance of different retrieval options (dense vs. sparse) and aim for a smooth transition from PoC to production.
It features the approximate nearest neighbours (ANN) search.
### Initialization
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
If you have Docker set up, we recommend pulling the Docker image and running it.
```shell
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
```
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
```shell
docker compose up
```
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
```shell
pip install elasticsearch-haystack
```
Then, initialize an `ElasticsearchDocumentStore` object thats connected to the Elasticsearch instance and writes documents to it:
```python
from haystack_integrations.document_stores.elasticsearch import (
ElasticsearchDocumentStore,
)
from haystack import Document
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200")
document_store.write_documents(
[Document(content="This is first"), Document(content="This is second")],
)
print(document_store.count_documents())
```
### Supported Retrievers
[`ElasticsearchBM25Retriever`](../pipeline-components/retrievers/elasticsearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
[`ElasticsearchEmbeddingRetriever`](../pipeline-components/retrievers/elasticsearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
@@ -0,0 +1,152 @@
---
title: "FAISSDocumentStore"
id: faissdocumentstore
slug: "/faissdocumentstore"
---
# FAISSDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [FAISS](/reference/integrations-faiss) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/faiss |
</div>
`FAISSDocumentStore` is a local Document Store backed by [FAISS](https://github.com/facebookresearch/faiss) for vector similarity search.
It keeps vectors in a FAISS index and stores document data in memory, with optional persistence to disk.
`FAISSDocumentStore` is a good fit for local development and small to medium-sized datasets where you want a lightweight setup without running an external database service.
## Installation
Install the FAISS integration:
```shell
pip install faiss-haystack
```
## Initialization
Create a `FAISSDocumentStore` instance and write embedded documents:
```python
from haystack import Document
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
document_store = FAISSDocumentStore(
index_path="my_faiss_index", # Optional: enables persistence on disk
index_string="Flat",
embedding_dim=768,
)
document_store.write_documents(
[
Document(content="This is first", embedding=[0.1] * 768),
Document(content="This is second", embedding=[0.2] * 768),
],
policy=DuplicatePolicy.OVERWRITE,
)
print(document_store.count_documents())
# Persist index and metadata files (`.faiss` and `.json`)
document_store.save("my_faiss_index")
```
### Persistence
If you provide `index_path` when initializing `FAISSDocumentStore`, it tries to load existing persisted files (`.faiss` and `.json`) from that path.
You can also explicitly call:
- `save(index_path)` to write index and metadata to disk.
- `load(index_path)` to load them later.
Example of loading from a previously saved folder/path:
```python
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
# This loads `my_faiss_index.faiss` and `my_faiss_index.json` if they exist
document_store = FAISSDocumentStore(index_path="my_faiss_index")
# Alternatively, initialize first and then load explicitly
another_store = FAISSDocumentStore(embedding_dim=768)
another_store.load("my_faiss_index")
```
## Supported Retrievers
[`FAISSEmbeddingRetriever`](../pipeline-components/retrievers/faissembeddingretriever.mdx): Retrieves documents from `FAISSDocumentStore` based on query embeddings.
### Fixing OpenMP Runtime Conflicts on macOS
#### Symptoms
You may encounter one or both of the following errors at runtime:
```
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.
```
```
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
```
If setting `OMP_NUM_THREADS=1` prevents the crash, the root cause is **multiple OpenMP runtimes loaded simultaneously**. Each runtime maintains its own thread pool and thread-local storage (TLS). When two runtimes spin up worker threads at the same time, they corrupt each other's memory — causing segfaults at `N > 1` threads.
---
#### Diagnosis
First, find how many copies of `libomp.dylib` exist in your virtual environment:
```bash
find /path/to/your/.venv -name "libomp.dylib" 2>/dev/null
```
If you see more than one, e.g.:
```
.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib
.venv/lib/pythonX.Y/site-packages/sklearn/.dylibs/libomp.dylib
.venv/lib/pythonX.Y/site-packages/faiss/.dylibs/libomp.dylib
```
you need to consolidate them into a single runtime.
---
#### Fix
The solution is to pick one canonical `libomp.dylib` (torch's is a good choice) and replace all other copies with symlinks pointing to it.
For each duplicate, delete the copy and replace it with a symlink:
```bash
# Delete the duplicate
rm /path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
# Replace with a symlink to the canonical copy
ln -s /path/to/.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib \
/path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
```
Repeat for every duplicate found. Because these packages use `@loader_path`-relative references to load `libomp.dylib`, the symlink will be transparently resolved to the single canonical runtime at load time.
---
#### Verify
After applying the fix, confirm only one unique `libomp.dylib` is being referenced:
```bash
find /path/to/your/.venv -name "*.so" | xargs otool -L 2>/dev/null | grep libomp | sort -u
```
All entries should resolve to the same canonical path. You should now be able to run without `OMP_NUM_THREADS=1`.
@@ -0,0 +1,27 @@
---
title: "InMemoryDocumentStore"
id: inmemorydocumentstore
slug: "/inmemorydocumentstore"
---
# InMemoryDocumentStore
The `InMemoryDocumentStore` is a very simple document store with no extra services or dependencies.
It is great for experimenting with Haystack, however we do not recommend using it for production.
### Initialization
`InMemoryDocumentStore` requires no external setup. Simply use this code:
```python
from haystack.document_stores.in_memory import InMemoryDocumentStore
document_store = InMemoryDocumentStore()
```
### Supported Retrievers
[`InMemoryBM25Retriever`](../pipeline-components/retrievers/inmemorybm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from a temporary in-memory database.
[`InMemoryEmbeddingRetriever`](../pipeline-components/retrievers/inmemoryembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
@@ -0,0 +1,59 @@
---
title: "MongoDBAtlasDocumentStore"
id: mongodbatlasdocumentstore
slug: "/mongodbatlasdocumentstore"
---
# MongoDBAtlasDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [MongoDB Atlas](/reference/integrations-mongodb-atlas) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mongodb_atlas |
</div>
`MongoDBAtlasDocumentStore` can be used to manage documents using [MongoDB Atlas](https://www.mongodb.com/atlas), a multi-cloud database service by the same people who build MongoDB. Atlas simplifies deploying and managing your databases while offering the versatility you need to build resilient and performant global applications on the cloud providers of your choice. You can use MongoDB Atlas on cloud providers such as AWS, Azure, or Google Cloud, all without leaving Atlas' web UI.
MongoDB Atlas supports embeddings and can therefore be used for embedding retrieval.
## Installation
To use MongoDB Atlas with Haystack, install the integration first:
```shell
pip install mongodb-atlas-haystack
```
## Initialization
To use MongoDB Atlas with Haystack, you will need to create your MongoDB Atlas account: check the [MongoDB Atlas documentation](https://www.mongodb.com/docs/atlas/getting-started/) for help. You also need to [create a vector search index](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/#std-label-avs-create-index) and [a full-text search index](https://www.mongodb.com/docs/atlas/atlas-search/manage-indexes/#create-an-atlas-search-index) for the collection you plan to use.
Once you have your connection string, you should export it in an environment variable called `MONGO_CONNECTION_STRING`. It should look something like this:
```python
export MONGO_CONNECTION_STRING="mongodb+srv://<username>:<password>@<cluster_name>.gwkckbk.mongodb.net/?retryWrites=true&w=majority"
```
At this point, youre ready to initialize the store:
```python
from haystack_integrations.document_stores.mongodb_atlas import (
MongoDBAtlasDocumentStore,
)
## Initialize the document store
document_store = MongoDBAtlasDocumentStore(
database_name="haystack_test",
collection_name="test_collection",
vector_search_index="embedding_index",
full_text_search_index="search_index",
)
```
## Supported Retrievers
- [`MongoDBAtlasEmbeddingRetriever`](../pipeline-components/retrievers/mongodbatlasembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
- [`MongoDBAtlasFullTextRetriever`](../pipeline-components/retrievers/mongodbatlasfulltextretriever.mdx): A full-text search Retriever.
@@ -0,0 +1,76 @@
---
title: "OpenSearchDocumentStore"
id: opensearch-document-store
slug: "/opensearch-document-store"
description: "A Document Store for storing and retrieval from OpenSearch."
---
# OpenSearchDocumentStore
A Document Store for storing and retrieval from OpenSearch.
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [OpenSearch](/reference/integrations-opensearch) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
</div>
OpenSearch is a fully open source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/).
This Document Store is great if you want to evaluate the performance of different retrieval options (dense vs. sparse). Its compatible with the Amazon OpenSearch Service.
OpenSearch provides support for vector similarity comparisons and approximate nearest neighbors algorithms.
### Initialization
[Install](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) and run an OpenSearch instance.
If you have Docker set up, we recommend pulling the Docker image and running it.
```shell
docker pull opensearchproject/opensearch:2.11.0
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" opensearchproject/opensearch:2.11.0
```
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
```shell
docker compose up
```
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
```shell
pip install opensearch-haystack
```
Then, initialize an `OpenSearchDocumentStore` object thats connected to the OpenSearch instance and writes documents to it:
```python
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
from haystack import Document
document_store = OpenSearchDocumentStore(
hosts="http://localhost:9200",
use_ssl=True,
verify_certs=False,
http_auth=("admin", "admin"),
)
document_store.write_documents(
[Document(content="This is first"), Document(content="This is second")],
)
print(document_store.count_documents())
```
### Supported Retrievers
[`OpenSearchBM25Retriever`](../pipeline-components/retrievers/opensearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
[`OpenSearchEmbeddingRetriever`](../pipeline-components/retrievers/opensearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
## Additional References
🧑‍🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
@@ -0,0 +1,109 @@
---
title: "PgvectorDocumentStore"
id: pgvectordocumentstore
slug: "/pgvectordocumentstore"
---
# PgvectorDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Pgvector](/reference/integrations-pgvector) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector/ |
</div>
Pgvector is an extension for PostgreSQL that enhances its capabilities with vector similarity search. It builds upon the classic features of PostgreSQL, such as ACID compliance and point-in-time recovery, and introduces the ability to perform exact and approximate nearest neighbor search using vectors.
For more information, see the [pgvector repository](https://github.com/pgvector/pgvector).
Pgvector Document Store supports embedding retrieval and metadata filtering.
## Installation
To quickly set up a PostgreSQL database with pgvector, you can use Docker:
```shell
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
```
For more information on installing pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).
To use pgvector with Haystack, install the `pgvector-haystack` integration:
```shell
pip install pgvector-haystack
```
## Usage
### Connection String
Define the connection string to your PostgreSQL database in the `PG_CONN_STR` environment variable. Two formats are supported:
**URI format:**
```shell
export PG_CONN_STR="postgresql://USER:PASSWORD@HOST:PORT/DB_NAME"
```
**Keyword/value format:**
```shell
export PG_CONN_STR="host=HOST port=PORT dbname=DB_NAME user=USER password=PASSWORD"
```
:::caution[Special Characters in Connection URIs]
When using the URI format, special characters in the password must be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding). Otherwise, connection errors may occur. A password like `p=ssword` would cause the error `psycopg.OperationalError: [Errno -2] Name or service not known`.
For example, if your password is `p=ssword`, the connection string should be:
```shell
export PG_CONN_STR="postgresql://postgres:p%3Dssword@localhost:5432/postgres"
```
Alternatively, use the keyword/value format, which does not require percent-encoding:
```shell
export PG_CONN_STR="host=localhost port=5432 dbname=postgres user=postgres password=p=ssword"
```
:::
For more details, see the [PostgreSQL connection string documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).
## Initialization
Initialize a `PgvectorDocumentStore` object thats connected to the PostgreSQL database and writes documents to it:
```python
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
from haystack import Document
document_store = PgvectorDocumentStore(
embedding_dimension=768,
vector_function="cosine_similarity",
recreate_table=True,
search_strategy="hnsw",
)
document_store.write_documents(
[
Document(content="This is first", embedding=[0.1] * 768),
Document(content="This is second", embedding=[0.3] * 768),
],
)
print(document_store.count_documents())
```
To learn more about the initialization parameters, see our [API docs](/reference/integrations-pgvector#pgvectordocumentstore).
To properly compute embeddings for your documents, you can use a Document Embedder (for instance, the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx)).
### Supported Retrievers
- [`PgvectorEmbeddingRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
- [`PgvectorKeywordRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store.
@@ -0,0 +1,67 @@
---
title: "PineconeDocumentStore"
id: pinecone-document-store
slug: "/pinecone-document-store"
description: "Use a Pinecone vector database with Haystack."
---
# PineconeDocumentStore
Use a Pinecone vector database with Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Pinecone](/reference/integrations-pinecone) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pinecone |
</div>
[Pinecone](https://www.pinecone.io/) is a cloud-based vector database. It is fast and easy to use.
Unlike other solutions (such as Qdrant and Weaviate), it cant run locally on the user's machine but provides a generous free tier.
### Installation
You can simply install the Pinecone Haystack integration with:
```shell
pip install pinecone-haystack
```
### Initialization
- To use Pinecone as a Document Store in Haystack, sign up for a free Pinecone [account](https://app.pinecone.io/) and get your API key.
The Pinecone API key can be explicitly provided or automatically read from the environment variable `PINECONE_API_KEY` (recommended).
- In Haystack, each `PineconeDocumentStore` operates in a specific namespace of an index. If not provided, both index and namespace are `default`.
If the index already exists, the Document Store connects to it. Otherwise, it creates a new index.
- When creating a new index, you can provide a `spec` in the form of a dictionary. This allows choosing between serverless and pod deployment options and setting additional parameters. Refer to the [Pinecone documentation](https://docs.pinecone.io/reference/api/control-plane/create_index) for more details. If not provided, a default spec with serverless deployment in the `us-east-1` region will be used (compatible with the free tier).
- You can provide `dimension` and `metric`, but they are only taken into account if the Pinecone index does not already exist.
Then, you can use the Document Store like this:
```python
from haystack import Document
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore
## Make sure you have the PINECONE_API_KEY environment variable set
document_store = PineconeDocumentStore(
index="default",
namespace="default",
dimension=5,
metric="cosine",
spec={"serverless": {"region": "us-east-1", "cloud": "aws"}},
)
document_store.write_documents(
[
Document(content="This is first", embedding=[0.1] * 5),
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
],
)
print(document_store.count_documents())
```
### Supported Retrievers
[`PineconeEmbeddingRetriever`](../pipeline-components/retrievers/pineconedenseretriever.mdx): Retrieves documents from the `PineconeDocumentStore` based on their dense embeddings (vectors).
@@ -0,0 +1,103 @@
---
title: "QdrantDocumentStore"
id: qdrant-document-store
slug: "/qdrant-document-store"
description: "Use the Qdrant vector database with Haystack."
---
# QdrantDocumentStore
Use the Qdrant vector database with Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Qdrant](/reference/integrations-qdrant) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
</div>
Qdrant is a powerful high-performance, massive-scale vector database. The `QdrantDocumentStore` can be used with any Qdrant instance, in-memory, locally persisted, hosted, and the official Qdrant Cloud.
### Installation
You can simply install the Qdrant Haystack integration with:
```shell
pip install qdrant-haystack
```
### Initialization
The quickest way to use `QdrantDocumentStore` is to create an in-memory instance of it:
```python
from haystack.dataclasses.document import Document
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
document_store = QdrantDocumentStore(
":memory:",
recreate_index=True,
return_embedding=True,
wait_result_from_api=True,
)
document_store.write_documents(
[
Document(content="This is first", embedding=[0.0] * 768),
Document(content="This is second", embedding=[0.1] * 768),
],
)
print(document_store.count_documents())
```
:::warning[Collections Created Outside Haystack]
When you create a `QdrantDocumentStore` instance, Haystack takes care of setting up the collection. In general, you cannot use a Qdrant collection created without Haystack with Haystack. If you want to migrate your existing collection, see the sample script at https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/src/haystack_integrations/document_stores/qdrant/migrate_to_sparse.py.
:::
You can also connect directly to [Qdrant Cloud](https://cloud.qdrant.io/login) directly. Once you have your API key and your cluster URL from the Qdrant dashboard, you can connect like this:
```python
from haystack.dataclasses.document import Document
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
from haystack.utils import Secret
document_store = QdrantDocumentStore(
url="https://XXXXXXXXX.us-east4-0.gcp.cloud.qdrant.io:6333",
index="your_index_name",
embedding_dim=1024, # based on the embedding model
recreate_index=True, # enable only to recreate the index and not connect to the existing one
api_key=Secret.from_token("YOUR_TOKEN"),
)
document_store.write_documents(
[
Document(content="This is first", embedding=[0.0] * 5),
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
],
)
print(document_store.count_documents())
```
:::tip[More information]
You can find more ways to initialize and use QdrantDocumentStore on our [integration page](https://haystack.deepset.ai/integrations/qdrant-document-store).
:::
### Supported Retrievers
- [`QdrantEmbeddingRetriever`](../pipeline-components/retrievers/qdrantembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their dense embeddings (vectors).
- [`QdrantSparseEmbeddingRetriever`](../pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their sparse embeddings.
- [`QdrantHybridRetriever`](../pipeline-components/retrievers/qdranthybridretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on both dense and sparse embeddings.
:::note[Sparse Embedding Support]
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
:::
## Additional References
🧑‍🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
@@ -0,0 +1,150 @@
---
title: "ValkeyDocumentStore"
id: valkeydocumentstore
slug: "/valkeydocumentstore"
description: "Use a Valkey database with Haystack."
---
# ValkeyDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Valkey](/reference/integrations-valkey) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/valkey |
</div>
[Valkey](https://valkey.io/) is a high-performance, in-memory data structure store that you can use in Haystack pipelines with the `ValkeyDocumentStore`. Valkey operates in-memory by default for maximum performance, but can be configured with persistence options for data durability.
The `ValkeyDocumentStore` connects to a Valkey server with the search module running and supports vector similarity search for RAG and other retrieval use cases. For a detailed overview of all the available methods and settings, visit the [API Reference](/reference/integrations-valkey#valkeydocumentstore).
## Installation
You can install the Valkey Haystack integration with:
```shell
pip install valkey-haystack
```
## Initialization
To use Valkey as your data storage for Haystack pipelines, you need a Valkey server with the search module running. Initialize a `ValkeyDocumentStore` like this:
```python
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
document_store = ValkeyDocumentStore(
nodes_list=[("localhost", 6379)],
index_name="my_documents",
embedding_dim=768,
distance_metric="cosine"
)
```
### Running Valkey locally
For development and testing, you can start a Valkey server with Docker:
```shell
docker run -d -p 6379:6379 valkey/valkey-bundle:latest
```
Then connect with the same initialization code above, using `nodes_list=[("localhost", 6379)]`.
For more advanced configurations and clustering setups, refer to the [Valkey documentation](https://valkey.io/docs/).
## Writing documents
To write documents to your `ValkeyDocumentStore`, create an indexing pipeline or use the `write_documents()` method. You can use [Converters](../pipeline-components/converters.mdx), [PreProcessors](../pipeline-components/preprocessors.mdx), and other integrations to fetch and prepare data. Below is an example that indexes Markdown files into Valkey.
### Indexing pipeline
```python
from haystack import Pipeline
from haystack.components.converters import MarkdownToDocument
from haystack.components.writers import DocumentWriter
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
from haystack.components.preprocessors import DocumentSplitter
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
document_store = ValkeyDocumentStore(
nodes_list=[("localhost", 6379)],
index_name="my_documents",
embedding_dim=768,
distance_metric="cosine"
)
indexing = Pipeline()
indexing.add_component("converter", MarkdownToDocument())
indexing.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=2))
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("converter", "splitter")
indexing.connect("splitter", "embedder")
indexing.connect("embedder", "writer")
indexing.run({"converter": {"sources": ["filename.md"]}})
```
## Using Valkey in a RAG pipeline
Once documents are in your `ValkeyDocumentStore`, you can use [`ValkeyEmbeddingRetriever`](../pipeline-components/retrievers/valkeyembeddingretriever.mdx) to retrieve them. The following example builds a RAG pipeline with a custom prompt:
```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack.dataclasses import ChatMessage
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
from haystack_integrations.components.retrievers.valkey import ValkeyEmbeddingRetriever
document_store = ValkeyDocumentStore(
nodes_list=[("localhost", 6379)],
index_name="my_documents",
embedding_dim=768,
distance_metric="cosine"
)
prompt_template = [
ChatMessage.from_system("Answer the question based on the provided context. If the context does not include an answer, reply with 'I don't know'."),
ChatMessage.from_user(
"Query: {{query}}\n"
"Documents:\n{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
"Answer:",
),
]
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
query_pipeline.add_component("retriever", ValkeyEmbeddingRetriever(document_store=document_store))
query_pipeline.add_component("prompt_builder", ChatPromptBuilder(template=prompt_template, required_variables=["query", "documents"]))
query_pipeline.add_component("generator", OpenAIChatGenerator(api_key=Secret.from_token("YOUR_OPENAI_API_KEY"), model="gpt-4o"))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
query_pipeline.connect("prompt_builder.messages", "generator.messages")
query = "What is Valkey?"
results = query_pipeline.run(
{
"text_embedder": {"text": query},
"prompt_builder": {"query": query},
}
)
```
For more examples, see the [examples folder](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/valkey/examples) in the repository.
## Performance benefits
- **In-memory storage**: Fast read and write operations.
- **High throughput**: Handles many operations per second.
- **Low latency**: Minimal response times for document operations.
- **Scalability**: Supports clustering for horizontal scaling.
## Supported Retrievers
[`ValkeyEmbeddingRetriever`](../pipeline-components/retrievers/valkeyembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query from the `ValkeyDocumentStore`.
@@ -0,0 +1,155 @@
---
title: "WeaviateDocumentStore"
id: weaviatedocumentstore
slug: "/weaviatedocumentstore"
---
# WeaviateDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Weaviate](/reference/integrations-weaviate) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
</div>
Weaviate is a multi-purpose vector DB that can store both embeddings and data objects, making it a good choice for multi-modality.
The `WeaviateDocumentStore` can connect to any Weaviate instance, whether it's running on Weaviate Cloud Services, Kubernetes, or a local Docker container.
## Installation
You can simply install the Weaviate Haystack integration with:
```shell
pip install weaviate-haystack
```
## Initialization
### Weaviate Embedded
To use `WeaviateDocumentStore` as a temporary instance, initialize it as ["Embedded"](https://weaviate.io/developers/weaviate/installation/embedded):
```python
from haystack_integrations.document_stores.weaviate import WeaviateDocumentStore
from weaviate.embedded import EmbeddedOptions
document_store = WeaviateDocumentStore(embedded_options=EmbeddedOptions())
```
### Docker
You can use `WeaviateDocumentStore` in a local Docker container. This is what a minimal `docker-compose.yml` could look like:
```yaml
---
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: semitechnologies/weaviate:1.30.17
ports:
- 8080:8080
- 50051:50051
volumes:
- weaviate_data:/var/lib/weaviate
restart: 'no'
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ENABLE_MODULES: ''
CLUSTER_HOSTNAME: 'node1'
volumes:
weaviate_data:
...
```
:::warning
With this example, we explicitly enable access without authentication, so you don't need to set any username, password, or API key to connect to our local instance. That is strongly discouraged for production use. See the [authorization](#authorization) section for detailed information.
:::
Start your container with `docker compose up -d` and then initialize the Document Store with:
```python
from haystack_integrations.document_stores.weaviate.document_store import (
WeaviateDocumentStore,
)
from haystack import Document
document_store = WeaviateDocumentStore(url="http://localhost:8080")
document_store.write_documents(
[Document(content="This is first"), Document(content="This is second")],
)
print(document_store.count_documents())
```
### Weaviate Cloud Service
To use the [Weaviate managed cloud service](https://weaviate.io/developers/wcs), first, create your Weaviate cluster.
Then, initialize the `WeaviateDocumentStore` using the API Key and URL found in your [Weaviate account](https://console.weaviate.cloud/):
```python
from haystack_integrations.document_stores.weaviate import (
WeaviateDocumentStore,
AuthApiKey,
)
from haystack import Document
import os
os.environ["WEAVIATE_API_KEY"] = "YOUR-API-KEY"
auth_client_secret = AuthApiKey()
document_store = WeaviateDocumentStore(
url="YOUR-WEAVIATE-URL",
auth_client_secret=auth_client_secret,
)
```
## Authorization
We provide some utility classes in the `auth` package to handle authorization using different credentials. Every class stores distinct [secrets](../concepts/secret-management.mdx) and retrieves them from the environment variables when required.
The default environment variables for the classes are:
- **`AuthApiKey`**
- `WEAVIATE_API_KEY`
- **`AuthBearerToken`**
- `WEAVIATE_ACCESS_TOKEN`
- `WEAVIATE_REFRESH_TOKEN`
- **`AuthClientCredentials`**
- `WEAVIATE_CLIENT_SECRET`
- `WEAVIATE_SCOPE`
- **`AuthClientPassword`**
- `WEAVIATE_USERNAME`
- `WEAVIATE_PASSWORD`
- `WEAVIATE_SCOPE`
You can easily change environment variables if needed. In the following snippet, we instruct `AuthApiKey` to look for `MY_ENV_VAR`.
```python
from haystack_integrations.document_stores.weaviate.auth import AuthApiKey
from haystack.utils.auth import Secret
AuthApiKey(api_key=Secret.from_env_var("MY_ENV_VAR"))
```
## Supported Retrievers
[`WeaviateBM25Retriever`](../pipeline-components/retrievers/weaviatebm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
[`WeaviateEmbeddingRetriever`](../pipeline-components/retrievers/weaviateembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.