chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "AlloyDBDocumentStore"
|
||||
id: alloydbdocumentstore
|
||||
slug: "/alloydbdocumentstore"
|
||||
---
|
||||
|
||||
# AlloyDBDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [AlloyDB](/reference/integrations-alloydb) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/alloydb |
|
||||
|
||||
</div>
|
||||
|
||||
[AlloyDB](https://cloud.google.com/alloydb) is a fully managed, PostgreSQL-compatible database service on Google Cloud. The `AlloyDBDocumentStore` uses the [pgvector extension](https://cloud.google.com/alloydb/docs/ai/work-with-embeddings) to perform vector similarity search.
|
||||
|
||||
Connection is handled securely via the [AlloyDB Python Connector](https://github.com/GoogleCloudPlatform/alloydb-python-connector), which provides TLS encryption and IAM-based authorization without requiring manual SSL certificate management, firewall rules, or IP allowlisting.
|
||||
|
||||
The `AlloyDBDocumentStore` supports embedding retrieval, keyword retrieval, and metadata filtering.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `alloydb-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install alloydb-haystack
|
||||
```
|
||||
|
||||
To set up an AlloyDB cluster and instance, follow the [AlloyDB quickstart](https://cloud.google.com/alloydb/docs/quickstart).
|
||||
|
||||
## Usage
|
||||
|
||||
### Authentication
|
||||
|
||||
The `AlloyDBDocumentStore` uses [Secrets](../concepts/secret-management.mdx) and reads connection details from environment variables by default:
|
||||
|
||||
- `ALLOYDB_INSTANCE_URI`: the AlloyDB instance URI in the format `projects/PROJECT/locations/REGION/clusters/CLUSTER/instances/INSTANCE`.
|
||||
- `ALLOYDB_USER`: the database user. When using IAM database authentication, use the service account email (omitting `.gserviceaccount.com`) or the full IAM user email.
|
||||
- `ALLOYDB_PASSWORD`: the database password. Not required when `enable_iam_auth=True`.
|
||||
|
||||
```shell
|
||||
export ALLOYDB_INSTANCE_URI="projects/MY_PROJECT/locations/MY_REGION/clusters/MY_CLUSTER/instances/MY_INSTANCE"
|
||||
export ALLOYDB_USER="my-db-user"
|
||||
export ALLOYDB_PASSWORD="my-db-password"
|
||||
```
|
||||
|
||||
To authenticate with IAM instead of a password, set `enable_iam_auth=True` and grant the IAM principal the AlloyDB Client role. See the [AlloyDB IAM authentication documentation](https://cloud.google.com/alloydb/docs/manage-iam-authn) for details.
|
||||
|
||||
## Initialization
|
||||
|
||||
Initialize an `AlloyDBDocumentStore` and write Documents to it. Connection to AlloyDB is established lazily on first use, and the table that stores Haystack Documents is created automatically if it doesn't exist:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.alloydb import AlloyDBDocumentStore
|
||||
|
||||
document_store = AlloyDBDocumentStore(
|
||||
db="my-database",
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 768),
|
||||
Document(content="This is second", embedding=[0.3] * 768),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see our [API docs](/reference/integrations-alloydb#alloydbdocumentstore).
|
||||
|
||||
To compute embeddings for your Documents, you can use a Document Embedder, such as the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx).
|
||||
|
||||
### Search Strategy
|
||||
|
||||
The `AlloyDBDocumentStore` supports two search strategies for embedding retrieval:
|
||||
|
||||
- `"exact_nearest_neighbor"` (default): provides perfect recall but can be slow on large numbers of documents.
|
||||
- `"hnsw"`: an approximate nearest neighbor search strategy that trades off some accuracy for speed. Recommended for large numbers of documents.
|
||||
|
||||
When using `"hnsw"`, an index is created based on the `vector_function` you choose, so subsequent queries should keep using the same vector similarity function in order to take advantage of the index. You can tune index creation through `hnsw_index_creation_kwargs` (see the [pgvector documentation](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw)).
|
||||
|
||||
### Metadata Filtering
|
||||
|
||||
The `AlloyDBDocumentStore` fully supports comparison operators (`==`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `not in`, `like`, `not like`) and the logical operators `AND` and `OR`. The `like` and `not like` operators are PostgreSQL-specific extensions to the standard Haystack filter syntax and map to the SQL `LIKE` / `NOT LIKE` pattern-matching operators.
|
||||
|
||||
The `NOT` logical operator is **not** supported. Because every comparison operator already has a negated counterpart (`==`/`!=`, `in`/`not in`, `like`/`not like`), any filter expressible with `NOT` around a single condition can be rewritten by inverting the comparison operator instead. To negate a nested `AND`/`OR` group, apply De Morgan's laws — for example, `NOT (A AND B)` becomes `(NOT A) OR (NOT B)`, where each `NOT A` / `NOT B` is expressed via the inverted comparison.
|
||||
|
||||
For more details on filter syntax, refer to [Metadata Filtering](../concepts/metadata-filtering.mdx).
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`AlloyDBEmbeddingRetriever`](../pipeline-components/retrievers/alloydbembeddingretriever.mdx): An embedding-based Retriever that fetches Documents from the Document Store based on a query embedding.
|
||||
- [`AlloyDBKeywordRetriever`](../pipeline-components/retrievers/alloydbkeywordretriever.mdx): A keyword-based Retriever that fetches Documents matching a query using PostgreSQL full-text search.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: "ArcadeDBDocumentStore"
|
||||
id: arcadedbdocumentstore
|
||||
slug: "/arcadedbdocumentstore"
|
||||
---
|
||||
|
||||
# ArcadeDBDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [ArcadeDB](/reference/integrations-arcadedb) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/arcadedb |
|
||||
|
||||
</div>
|
||||
|
||||
ArcadeDB is a multi-model database that supports vector search via its LSM_VECTOR (HNSW) index. The `ArcadeDBDocumentStore` uses ArcadeDB's HTTP/JSON API for all operations—no special drivers required. It supports dense embedding retrieval and SQL-based metadata filtering.
|
||||
|
||||
For more information, see the [ArcadeDB documentation](https://docs.arcadedb.com/).
|
||||
|
||||
## Installation
|
||||
|
||||
Run ArcadeDB with Docker and update the password according to your setup:
|
||||
|
||||
```shell
|
||||
docker run -d -p 2480:2480 \
|
||||
-e JAVA_OPTS="-Darcadedb.server.rootPassword=arcadedb" \
|
||||
arcadedata/arcadedb:latest
|
||||
```
|
||||
|
||||
Install the Haystack integration:
|
||||
|
||||
```shell
|
||||
pip install arcadedb-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Set credentials via environment variables (recommended) or pass them explicitly:
|
||||
|
||||
```shell
|
||||
export ARCADEDB_USERNAME=root
|
||||
export ARCADEDB_PASSWORD=arcadedb
|
||||
```
|
||||
|
||||
Initialize the document store and write documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.arcadedb import ArcadeDBDocumentStore
|
||||
|
||||
document_store = ArcadeDBDocumentStore(
|
||||
url="http://localhost:2480",
|
||||
database="haystack",
|
||||
embedding_dimension=768,
|
||||
recreate_type=True,
|
||||
)
|
||||
|
||||
document_store.write_documents([
|
||||
Document(content="This is first", embedding=[0.0] * 768),
|
||||
Document(content="This is second", embedding=[0.1, 0.2, 0.3] + [0.0] * 765),
|
||||
])
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see the [API docs](/reference/integrations-arcadedb#arcadedbdocumentstore).
|
||||
|
||||
Documents without embeddings or with a different dimension are stored with a zero-padded vector so they can be written and filtered; use an [Embedder](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx) for real embeddings.
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [ArcadeDBEmbeddingRetriever](../pipeline-components/retrievers/arcadedbembeddingretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store by vector similarity (HNSW).
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "AstraDocumentStore"
|
||||
id: astradocumentstore
|
||||
slug: "/astradocumentstore"
|
||||
---
|
||||
|
||||
# AstraDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Astra](/reference/integrations-astra) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/astra |
|
||||
|
||||
</div>
|
||||
|
||||
DataStax Astra DB is a serverless vector database built on Apache Cassandra, and it supports vector-based search and auto-scaling. You can deploy it on AWS, GCP, or Azure and easily expand to one or more regions within those clouds for multi-region availability, low latency data access, data sovereignty, and to avoid cloud vendor lock-in. For more information, see the [DataStax documentation](https://docs.datastax.com/en/home/docs/index.html).
|
||||
|
||||
### Initialization
|
||||
|
||||
Once you have an AstraDB account and have created a database, install the `astra-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install astra-haystack
|
||||
```
|
||||
|
||||
From the configuration in AstraDB’s web UI, you need the database ID and a generated token.
|
||||
|
||||
You will additionally need a collection name and a namespace. When you create the collection name, you also need to set the embedding dimensions and the similarity metric. The namespace organizes data in a database and is called a keyspace in Apache Cassandra.
|
||||
|
||||
Then, in Haystack, initialize an `AstraDocumentStore` object that’s connected to the AstraDB instance, and write documents to it.
|
||||
|
||||
We strongly encourage passing authentication data through environment variables: make sure to populate the environment variables `ASTRA_DB_API_ENDPOINT` and `ASTRA_DB_APPLICATION_TOKEN` before running the following example.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.astra import AstraDocumentStore
|
||||
|
||||
document_store = AstraDocumentStore()
|
||||
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[AstraEmbeddingRetriever](../pipeline-components/retrievers/astraretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
|
||||
|
||||
### Indexing Warnings
|
||||
|
||||
When you create an Astra DB Document Store, you might see one of these warnings:
|
||||
|
||||
> Astra DB collection `...` is detected as having indexing turned on for all fields (either created manually or by older versions of this plugin). This implies stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
|
||||
|
||||
Or:
|
||||
|
||||
> Astra DB collection `...` is detected as having the following indexing policy: `{...}`. This does not match the requested indexing policy for this object: `{...}`. In particular, there may be stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
|
||||
|
||||
#### Why You See This Warning
|
||||
|
||||
The collection already exists and is configured to [index all fields for search](https://docs.datastax.com/en/astra-db-serverless/api-reference/collections.html#the-indexing-option), possibly because you created it earlier or an older plugin did. When Haystack tries to create the collection, it applies an indexing policy optimized for your intended use. This policy lets you store longer texts and avoids indexing fields you won’t filter on, which also reduces write overhead.
|
||||
|
||||
#### Common Causes
|
||||
|
||||
1. You created the collection outside Haystack (for example, in the Astra UI or with AstraPy’s `Database.create_collection()`).
|
||||
2. You created the collection with an older version of the plugin.
|
||||
|
||||
#### Impact
|
||||
|
||||
This is only a warning. Your application keeps running unless you try to store very long text fields. If you do, Astra DB returns an indexing error.
|
||||
|
||||
#### Solutions
|
||||
|
||||
- **Recommended:** _Drop and recreate the collection_ if you can repopulate it. Then rerun your Haystack application so it creates the collection with the optimized indexing policy.
|
||||
- _Ignore the warning_ if you’re sure you won’t store very long text fields.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Using AstraDB as a data store in your Haystack pipelines](https://haystack.deepset.ai/cookbook/astradb_haystack_integration)
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: "AzureAISearchDocumentStore"
|
||||
id: azureaisearchdocumentstore
|
||||
slug: "/azureaisearchdocumentstore"
|
||||
description: "A Document Store for storing and retrieval from Azure AI Search Index."
|
||||
---
|
||||
|
||||
# AzureAISearchDocumentStore
|
||||
|
||||
A Document Store for storing and retrieval from Azure AI Search Index.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
|
||||
|
||||
</div>
|
||||
|
||||
[Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search) is an enterprise-ready search and retrieval system to build RAG-based applications on Azure, with native LLM integrations.
|
||||
|
||||
`AzureAISearchDocumentStore` supports semantic reranking and metadata/content filtering. The Document Store is useful for various tasks such as generating knowledge base insights (catalog or document search), information discovery (data exploration), RAG, and automation.
|
||||
|
||||
### Initialization
|
||||
|
||||
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
|
||||
|
||||
Once you have the subscription, install the `azure-ai-search-haystack` integration:
|
||||
|
||||
```python
|
||||
pip install azure-ai-search-haystack
|
||||
```
|
||||
|
||||
To use the `AzureAISearchDocumentStore`, you need to provide a search service endpoint as an `AZURE_AI_SEARCH_ENDPOINT` and an API key as `AZURE_AI_SEARCH_API_KEY` for authentication. If the API key is not provided, the `DefaultAzureCredential` will attempt to authenticate you through the browser.
|
||||
|
||||
During initialization the Document Store will either retrieve the existing search index for the given `index_name` or create a new one if it doesn't already exist. Note that one of the limitations of `AzureAISearchDocumentStore` is that the fields of the Azure search index cannot be modified through the API after creation. Therefore, any additional fields beyond the default ones must be provided as `metadata_fields` during the Document Store's initialization. However, if needed, [Azure AI portal](https://azure.microsoft.com/) can be used to modify the fields without deleting the index.
|
||||
|
||||
It is recommended to pass authentication data through `AZURE_AI_SEARCH_API_KEY` and `AZURE_AI_SEARCH_ENDPOINT` before running the following example.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="haystack-docs")
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is the first document."),
|
||||
Document(content="This is the second document."),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::info[Latency Notice]
|
||||
|
||||
Due to Azure search index latency, the document count returned in the example might be zero if executed immediately. To ensure accurate results, be mindful of this latency when retrieving documents from the search index.
|
||||
:::
|
||||
|
||||
You can enable semantic reranking in `AzureAISearchDocumentStore` by providing [SemanticSearch](https://learn.microsoft.com/en-us/python/api/azure-search-documents/azure.search.documents.indexes.models.semanticsearch?view=azure-python) configuration in `index_creation_kwargs` during initialization and calling it from one of the Retrievers. For more information, refer to the [Azure AI tutorial](https://learn.microsoft.com/en-us/azure/search/search-get-started-semantic) on this feature.
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
The Haystack Azure AI Search integration includes three Retriever components. Each Retriever leverages the Azure AI Search API and you can select the one that best suits your pipeline:
|
||||
|
||||
- [`AzureAISearchEmbeddingRetriever`](../pipeline-components/retrievers/azureaisearchembeddingretriever.mdx): This Retriever accepts the embeddings of a single query as input and returns a list of matching documents. The query must be embedded beforehand, which can be done using an [Embedder](../pipeline-components/embedders.mdx) component.
|
||||
- [`AzureAISearchBM25Retriever`](../pipeline-components/retrievers/azureaisearchbm25retriever.mdx): A keyword-based Retriever that retrieves documents matching a query from the Azure AI Search index.
|
||||
- [`AzureAISearchHybridRetriever`](../pipeline-components/retrievers/azureaisearchhybridretriever.mdx): This Retriever combines embedding-based retrieval and keyword search to find matching documents in the search index to get more relevant results.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "ChromaDocumentStore"
|
||||
id: chromadocumentstore
|
||||
slug: "/chromadocumentstore"
|
||||
---
|
||||
|
||||
# ChromaDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Chroma](/reference/integrations-chroma) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chroma |
|
||||
|
||||
</div>
|
||||
|
||||
[Chroma](https://docs.trychroma.com/) is an open source vector database capable of storing collections of documents along with their metadata, creating embeddings for documents and queries, and searching the collections filtering by document metadata or content. Additionally, Chroma supports multi-modal embedding functions.
|
||||
|
||||
Chroma can be used in-memory, as an embedded database, or in a client-server fashion. When running in-memory, Chroma can still keep its contents on disk across different sessions. This allows users to quickly put together prototypes using the in-memory version and later move to production, where the client-server version is deployed.
|
||||
|
||||
## Initialization
|
||||
|
||||
First, install the Chroma integration, which will install Haystack and Chroma if they are not already present. The following command is all you need to start:
|
||||
|
||||
```shell
|
||||
pip install chroma-haystack
|
||||
```
|
||||
|
||||
To store data in Chroma, create a `ChromaDocumentStore` instance and write documents with:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is the first document."),
|
||||
Document(content="This is the second document."),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
In this case, since we didn’t pass any embeddings along with our documents, Chroma will create them for us using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
|
||||
|
||||
### Connection Options
|
||||
|
||||
1. **In-Memory Mode (Local)**: Chroma can be set up as a local Document Store for fast and lightweight usage. You can use this option during development or small-scale experiments. Set up a local in-memory instance of `ChromaDocumentStore` like this:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
```
|
||||
2. **Persistent Storage**: If you need to retain the documents between sessions, Chroma supports persistent storage by specifying a path to store data on disk:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore(persist_path="your_directory_path")
|
||||
```
|
||||
3. **Remote Connection**: You can connect to a remote Chroma database through HTTP. This is suitable for distributed setups where multiple clients might interact with the same remote Chroma instance.
|
||||
|
||||
Note that this option is incompatible with in-memory or persistent storage modes.
|
||||
|
||||
First, start a Chroma server:
|
||||
|
||||
```shell
|
||||
chroma run --path /db_path
|
||||
```
|
||||
|
||||
Or using docker:
|
||||
|
||||
```shell
|
||||
docker run -p 8000:8000 chromadb/chroma
|
||||
```
|
||||
|
||||
Then, initialize the Document Store with `host` and `port` parameters:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore(host="localhost", port="8000")
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
The Haystack Chroma integration comes with three Retriever components. They all rely on the Chroma [query API](https://docs.trychroma.com/reference/Collection#query), but they have different inputs and outputs so that you can pick the one that best fits your pipeline:
|
||||
|
||||
- [`ChromaQueryTextRetriever`](../pipeline-components/retrievers/chromaqueryretriever.mdx): This Retriever takes a plain-text query string in input and returns a list of matching documents. Chroma will create the embeddings for the query using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
|
||||
- [`ChromaEmbeddingRetriever`](../pipeline-components/retrievers/chromaembeddingretriever.mdx): This Retriever takes the embeddings of a single query in input and returns a list of matching documents. The query needs to be embedded before being passed to this component. For example, you can use an [embedder](../pipeline-components/embedders.mdx) component.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Use Chroma for RAG and Indexing](https://haystack.deepset.ai/cookbook/chroma-indexing-and-rag-examples)
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "ElasticsearchDocumentStore"
|
||||
id: elasticsearch-document-store
|
||||
slug: "/elasticsearch-document-store"
|
||||
description: "Use an Elasticsearch database with Haystack."
|
||||
---
|
||||
|
||||
# ElasticsearchDocumentStore
|
||||
|
||||
Use an Elasticsearch database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Elasticsearch](/reference/integrations-elasticsearch) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
|
||||
</div>
|
||||
|
||||
ElasticsearchDocumentStore is excellent if you want to evaluate the performance of different retrieval options (dense vs. sparse) and aim for a smooth transition from PoC to production.
|
||||
|
||||
It features the approximate nearest neighbours (ANN) search.
|
||||
|
||||
### Initialization
|
||||
|
||||
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
Then, initialize an `ElasticsearchDocumentStore` object that’s connected to the Elasticsearch instance and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200")
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`ElasticsearchBM25Retriever`](../pipeline-components/retrievers/elasticsearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`ElasticsearchEmbeddingRetriever`](../pipeline-components/retrievers/elasticsearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: "FAISSDocumentStore"
|
||||
id: faissdocumentstore
|
||||
slug: "/faissdocumentstore"
|
||||
---
|
||||
|
||||
# FAISSDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [FAISS](/reference/integrations-faiss) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/faiss |
|
||||
|
||||
</div>
|
||||
|
||||
`FAISSDocumentStore` is a local Document Store backed by [FAISS](https://github.com/facebookresearch/faiss) for vector similarity search.
|
||||
It keeps vectors in a FAISS index and stores document data in memory, with optional persistence to disk.
|
||||
|
||||
`FAISSDocumentStore` is a good fit for local development and small to medium-sized datasets where you want a lightweight setup without running an external database service.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the FAISS integration:
|
||||
|
||||
```shell
|
||||
pip install faiss-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
Create a `FAISSDocumentStore` instance and write embedded documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
|
||||
|
||||
document_store = FAISSDocumentStore(
|
||||
index_path="my_faiss_index", # Optional: enables persistence on disk
|
||||
index_string="Flat",
|
||||
embedding_dim=768,
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 768),
|
||||
Document(content="This is second", embedding=[0.2] * 768),
|
||||
],
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
print(document_store.count_documents())
|
||||
|
||||
# Persist index and metadata files (`.faiss` and `.json`)
|
||||
document_store.save("my_faiss_index")
|
||||
```
|
||||
|
||||
### Persistence
|
||||
|
||||
If you provide `index_path` when initializing `FAISSDocumentStore`, it tries to load existing persisted files (`.faiss` and `.json`) from that path.
|
||||
You can also explicitly call:
|
||||
|
||||
- `save(index_path)` to write index and metadata to disk.
|
||||
- `load(index_path)` to load them later.
|
||||
|
||||
Example of loading from a previously saved folder/path:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
|
||||
|
||||
# This loads `my_faiss_index.faiss` and `my_faiss_index.json` if they exist
|
||||
document_store = FAISSDocumentStore(index_path="my_faiss_index")
|
||||
|
||||
# Alternatively, initialize first and then load explicitly
|
||||
another_store = FAISSDocumentStore(embedding_dim=768)
|
||||
another_store.load("my_faiss_index")
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
[`FAISSEmbeddingRetriever`](../pipeline-components/retrievers/faissembeddingretriever.mdx): Retrieves documents from `FAISSDocumentStore` based on query embeddings.
|
||||
|
||||
|
||||
### Fixing OpenMP Runtime Conflicts on macOS
|
||||
|
||||
#### Symptoms
|
||||
|
||||
You may encounter one or both of the following errors at runtime:
|
||||
|
||||
```
|
||||
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
|
||||
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.
|
||||
```
|
||||
|
||||
```
|
||||
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
|
||||
```
|
||||
|
||||
If setting `OMP_NUM_THREADS=1` prevents the crash, the root cause is **multiple OpenMP runtimes loaded simultaneously**. Each runtime maintains its own thread pool and thread-local storage (TLS). When two runtimes spin up worker threads at the same time, they corrupt each other's memory — causing segfaults at `N > 1` threads.
|
||||
|
||||
---
|
||||
|
||||
#### Diagnosis
|
||||
|
||||
First, find how many copies of `libomp.dylib` exist in your virtual environment:
|
||||
|
||||
```bash
|
||||
find /path/to/your/.venv -name "libomp.dylib" 2>/dev/null
|
||||
```
|
||||
|
||||
If you see more than one, e.g.:
|
||||
|
||||
```
|
||||
.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib
|
||||
.venv/lib/pythonX.Y/site-packages/sklearn/.dylibs/libomp.dylib
|
||||
.venv/lib/pythonX.Y/site-packages/faiss/.dylibs/libomp.dylib
|
||||
```
|
||||
|
||||
you need to consolidate them into a single runtime.
|
||||
|
||||
---
|
||||
|
||||
#### Fix
|
||||
|
||||
The solution is to pick one canonical `libomp.dylib` (torch's is a good choice) and replace all other copies with symlinks pointing to it.
|
||||
|
||||
For each duplicate, delete the copy and replace it with a symlink:
|
||||
|
||||
```bash
|
||||
# Delete the duplicate
|
||||
rm /path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
|
||||
|
||||
# Replace with a symlink to the canonical copy
|
||||
ln -s /path/to/.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib \
|
||||
/path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
|
||||
```
|
||||
|
||||
Repeat for every duplicate found. Because these packages use `@loader_path`-relative references to load `libomp.dylib`, the symlink will be transparently resolved to the single canonical runtime at load time.
|
||||
|
||||
---
|
||||
|
||||
#### Verify
|
||||
|
||||
After applying the fix, confirm only one unique `libomp.dylib` is being referenced:
|
||||
|
||||
```bash
|
||||
find /path/to/your/.venv -name "*.so" | xargs otool -L 2>/dev/null | grep libomp | sort -u
|
||||
```
|
||||
|
||||
All entries should resolve to the same canonical path. You should now be able to run without `OMP_NUM_THREADS=1`.
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: "FalkorDBDocumentStore"
|
||||
id: falkordbdocumentstore
|
||||
slug: "/falkordbdocumentstore"
|
||||
description: "Use the FalkorDB graph database with Haystack for GraphRAG workloads."
|
||||
---
|
||||
|
||||
# FalkorDBDocumentStore
|
||||
|
||||
Use the FalkorDB graph database with Haystack for GraphRAG workloads.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [FalkorDB](/reference/integrations-falkordb) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/falkordb |
|
||||
|
||||
</div>
|
||||
|
||||
FalkorDB is a high-performance graph database optimized for GraphRAG workloads. The `FalkorDBDocumentStore` stores documents as graph nodes and supports native vector search — no APOC is required. Documents and their `meta` fields are stored flat on each node, and all bulk writes use `UNWIND` + `MERGE` for safe OpenCypher upserts.
|
||||
|
||||
For more information, see the [FalkorDB documentation](https://docs.falkordb.com/).
|
||||
|
||||
## Installation
|
||||
|
||||
Run FalkorDB with Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 6379:6379 falkordb/falkordb:latest
|
||||
```
|
||||
|
||||
Install the Haystack integration:
|
||||
|
||||
```shell
|
||||
pip install falkordb-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Initialize the document store and write documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
embedding_dim=768,
|
||||
recreate_graph=True,
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="There are over 7,000 languages spoken around the world today.",
|
||||
),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see the [API docs](/reference/integrations-falkordb#falkordbdocumentstore).
|
||||
|
||||
To compute real embeddings for your documents, use a Document Embedder such as the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx).
|
||||
|
||||
### Authentication
|
||||
|
||||
To connect to a password-protected FalkorDB instance, pass the password via `Secret`:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
password=Secret.from_env_var("FALKORDB_PASSWORD"),
|
||||
)
|
||||
```
|
||||
|
||||
### Similarity Functions
|
||||
|
||||
`FalkorDBDocumentStore` supports two similarity functions for vector search:
|
||||
|
||||
- `"cosine"` (default): cosine similarity, best for normalized embeddings.
|
||||
- `"euclidean"`: Euclidean distance, useful when embedding magnitude matters.
|
||||
|
||||
```python
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
embedding_dim=768,
|
||||
similarity="euclidean",
|
||||
)
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`FalkorDBEmbeddingRetriever`](../pipeline-components/retrievers/falkordbembeddingretriever.mdx): Retrieves documents from the `FalkorDBDocumentStore` based on vector similarity using FalkorDB's native vector index.
|
||||
- [`FalkorDBCypherRetriever`](../pipeline-components/retrievers/falkordbcypherretriever.mdx): Retrieves documents by executing arbitrary OpenCypher queries, enabling graph traversal and multi-hop queries for GraphRAG pipelines.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "InMemoryDocumentStore"
|
||||
id: inmemorydocumentstore
|
||||
slug: "/inmemorydocumentstore"
|
||||
---
|
||||
|
||||
# InMemoryDocumentStore
|
||||
|
||||
The `InMemoryDocumentStore` is a very simple document store with no extra services or dependencies.
|
||||
|
||||
It is great for experimenting with Haystack, however we do not recommend using it for production.
|
||||
|
||||
### Initialization
|
||||
|
||||
`InMemoryDocumentStore` requires no external setup. Simply use this code:
|
||||
|
||||
```python
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`InMemoryBM25Retriever`](../pipeline-components/retrievers/inmemorybm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from a temporary in-memory database.
|
||||
|
||||
[`InMemoryEmbeddingRetriever`](../pipeline-components/retrievers/inmemoryembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: "MongoDBAtlasDocumentStore"
|
||||
id: mongodbatlasdocumentstore
|
||||
slug: "/mongodbatlasdocumentstore"
|
||||
---
|
||||
|
||||
# MongoDBAtlasDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [MongoDB Atlas](/reference/integrations-mongodb-atlas) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mongodb_atlas |
|
||||
|
||||
</div>
|
||||
|
||||
`MongoDBAtlasDocumentStore` can be used to manage documents using [MongoDB Atlas](https://www.mongodb.com/atlas), a multi-cloud database service by the same people who build MongoDB. Atlas simplifies deploying and managing your databases while offering the versatility you need to build resilient and performant global applications on the cloud providers of your choice. You can use MongoDB Atlas on cloud providers such as AWS, Azure, or Google Cloud, all without leaving Atlas' web UI.
|
||||
|
||||
MongoDB Atlas supports embeddings and can therefore be used for embedding retrieval.
|
||||
|
||||
## Installation
|
||||
|
||||
To use MongoDB Atlas with Haystack, install the integration first:
|
||||
|
||||
```shell
|
||||
pip install mongodb-atlas-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
To use MongoDB Atlas with Haystack, you will need to create your MongoDB Atlas account: check the [MongoDB Atlas documentation](https://www.mongodb.com/docs/atlas/getting-started/) for help. You also need to [create a vector search index](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/#std-label-avs-create-index) and [a full-text search index](https://www.mongodb.com/docs/atlas/atlas-search/manage-indexes/#create-an-atlas-search-index) for the collection you plan to use.
|
||||
|
||||
Once you have your connection string, you should export it in an environment variable called `MONGO_CONNECTION_STRING`. It should look something like this:
|
||||
|
||||
```python
|
||||
export MONGO_CONNECTION_STRING="mongodb+srv://<username>:<password>@<cluster_name>.gwkckbk.mongodb.net/?retryWrites=true&w=majority"
|
||||
```
|
||||
|
||||
At this point, you’re ready to initialize the store:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
|
||||
# Initialize the document store
|
||||
document_store = MongoDBAtlasDocumentStore(
|
||||
database_name="haystack_test",
|
||||
collection_name="test_collection",
|
||||
vector_search_index="embedding_index",
|
||||
full_text_search_index="search_index",
|
||||
)
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
- [`MongoDBAtlasEmbeddingRetriever`](../pipeline-components/retrievers/mongodbatlasembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
- [`MongoDBAtlasFullTextRetriever`](../pipeline-components/retrievers/mongodbatlasfulltextretriever.mdx): A full-text search Retriever.
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "OpenSearchDocumentStore"
|
||||
id: opensearch-document-store
|
||||
slug: "/opensearch-document-store"
|
||||
description: "A Document Store for storing and retrieval from OpenSearch."
|
||||
---
|
||||
|
||||
# OpenSearchDocumentStore
|
||||
|
||||
A Document Store for storing and retrieval from OpenSearch.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
|
||||
</div>
|
||||
|
||||
OpenSearch is a fully open source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/).
|
||||
|
||||
This Document Store is great if you want to evaluate the performance of different retrieval options (dense vs. sparse). It’s compatible with the Amazon OpenSearch Service.
|
||||
|
||||
OpenSearch provides support for vector similarity comparisons and approximate nearest neighbors algorithms.
|
||||
|
||||
### Initialization
|
||||
|
||||
[Install](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) and run an OpenSearch instance.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull opensearchproject/opensearch:3.5.0
|
||||
docker run \
|
||||
-p 9200:9200 \
|
||||
-p 9600:9600 \
|
||||
-e "discovery.type=single-node" \
|
||||
-e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" \
|
||||
-e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=SecureHaystack*2026" \
|
||||
opensearchproject/opensearch:3.5.0
|
||||
```
|
||||
|
||||
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
Then, initialize an `OpenSearchDocumentStore` object that’s connected to the OpenSearch instance and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = OpenSearchDocumentStore(
|
||||
hosts="http://localhost:9200",
|
||||
use_ssl=True,
|
||||
verify_certs=False,
|
||||
http_auth=("admin", "SecureHaystack*2026"),
|
||||
)
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`OpenSearchBM25Retriever`](../pipeline-components/retrievers/opensearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`OpenSearchEmbeddingRetriever`](../pipeline-components/retrievers/opensearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "PgvectorDocumentStore"
|
||||
id: pgvectordocumentstore
|
||||
slug: "/pgvectordocumentstore"
|
||||
---
|
||||
|
||||
# PgvectorDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Pgvector](/reference/integrations-pgvector) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector/ |
|
||||
|
||||
</div>
|
||||
|
||||
Pgvector is an extension for PostgreSQL that enhances its capabilities with vector similarity search. It builds upon the classic features of PostgreSQL, such as ACID compliance and point-in-time recovery, and introduces the ability to perform exact and approximate nearest neighbor search using vectors.
|
||||
|
||||
For more information, see the [pgvector repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
Pgvector Document Store supports embedding retrieval and metadata filtering.
|
||||
|
||||
## Installation
|
||||
|
||||
To quickly set up a PostgreSQL database with pgvector, you can use Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
|
||||
```
|
||||
|
||||
For more information on installing pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
To use pgvector with Haystack, install the `pgvector-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install pgvector-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Connection String
|
||||
|
||||
Define the connection string to your PostgreSQL database in the `PG_CONN_STR` environment variable. Two formats are supported:
|
||||
|
||||
**URI format:**
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="postgresql://USER:PASSWORD@HOST:PORT/DB_NAME"
|
||||
```
|
||||
|
||||
**Keyword/value format:**
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="host=HOST port=PORT dbname=DB_NAME user=USER password=PASSWORD"
|
||||
```
|
||||
|
||||
:::caution[Special Characters in Connection URIs]
|
||||
|
||||
When using the URI format, special characters in the password must be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding). Otherwise, connection errors may occur. A password like `p=ssword` would cause the error `psycopg.OperationalError: [Errno -2] Name or service not known`.
|
||||
|
||||
For example, if your password is `p=ssword`, the connection string should be:
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="postgresql://postgres:p%3Dssword@localhost:5432/postgres"
|
||||
```
|
||||
|
||||
Alternatively, use the keyword/value format, which does not require percent-encoding:
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="host=localhost port=5432 dbname=postgres user=postgres password=p=ssword"
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
For more details, see the [PostgreSQL connection string documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).
|
||||
|
||||
## Initialization
|
||||
|
||||
Initialize a `PgvectorDocumentStore` object that’s connected to the PostgreSQL database and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = PgvectorDocumentStore(
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
search_strategy="hnsw",
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 768),
|
||||
Document(content="This is second", embedding=[0.3] * 768),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see our [API docs](/reference/integrations-pgvector#pgvectordocumentstore).
|
||||
|
||||
To properly compute embeddings for your documents, you can use a Document Embedder (for instance, the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx)).
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`PgvectorEmbeddingRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
|
||||
- [`PgvectorKeywordRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "PineconeDocumentStore"
|
||||
id: pinecone-document-store
|
||||
slug: "/pinecone-document-store"
|
||||
description: "Use a Pinecone vector database with Haystack."
|
||||
---
|
||||
|
||||
# PineconeDocumentStore
|
||||
|
||||
Use a Pinecone vector database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Pinecone](/reference/integrations-pinecone) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pinecone |
|
||||
|
||||
</div>
|
||||
|
||||
[Pinecone](https://www.pinecone.io/) is a cloud-based vector database. It is fast and easy to use.
|
||||
Unlike other solutions (such as Qdrant and Weaviate), it can’t run locally on the user's machine but provides a generous free tier.
|
||||
|
||||
### Installation
|
||||
|
||||
You can simply install the Pinecone Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install pinecone-haystack
|
||||
```
|
||||
|
||||
### Initialization
|
||||
|
||||
- To use Pinecone as a Document Store in Haystack, sign up for a free Pinecone [account](https://app.pinecone.io/) and get your API key.
|
||||
The Pinecone API key can be explicitly provided or automatically read from the environment variable `PINECONE_API_KEY` (recommended).
|
||||
- In Haystack, each `PineconeDocumentStore` operates in a specific namespace of an index. If not provided, both index and namespace are `default`.
|
||||
If the index already exists, the Document Store connects to it. Otherwise, it creates a new index.
|
||||
- When creating a new index, you can provide a `spec` in the form of a dictionary. This allows choosing between serverless and pod deployment options and setting additional parameters. Refer to the [Pinecone documentation](https://docs.pinecone.io/reference/api/control-plane/create_index) for more details. If not provided, a default spec with serverless deployment in the `us-east-1` region will be used (compatible with the free tier).
|
||||
- You can provide `dimension` and `metric`, but they are only taken into account if the Pinecone index does not already exist.
|
||||
|
||||
Then, you can use the Document Store like this:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore
|
||||
|
||||
# Make sure you have the PINECONE_API_KEY environment variable set
|
||||
document_store = PineconeDocumentStore(
|
||||
index="default",
|
||||
namespace="default",
|
||||
dimension=5,
|
||||
metric="cosine",
|
||||
spec={"serverless": {"region": "us-east-1", "cloud": "aws"}},
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 5),
|
||||
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`PineconeEmbeddingRetriever`](../pipeline-components/retrievers/pineconedenseretriever.mdx): Retrieves documents from the `PineconeDocumentStore` based on their dense embeddings (vectors).
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "QdrantDocumentStore"
|
||||
id: qdrant-document-store
|
||||
slug: "/qdrant-document-store"
|
||||
description: "Use the Qdrant vector database with Haystack."
|
||||
---
|
||||
|
||||
# QdrantDocumentStore
|
||||
|
||||
Use the Qdrant vector database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Qdrant](/reference/integrations-qdrant) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
|
||||
|
||||
</div>
|
||||
|
||||
Qdrant is a powerful high-performance, massive-scale vector database. The `QdrantDocumentStore` can be used with any Qdrant instance, in-memory, locally persisted, hosted, and the official Qdrant Cloud.
|
||||
|
||||
### Installation
|
||||
|
||||
You can simply install the Qdrant Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
### Initialization
|
||||
|
||||
The quickest way to use `QdrantDocumentStore` is to create an in-memory instance of it:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.document import Document
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
wait_result_from_api=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.0] * 768),
|
||||
Document(content="This is second", embedding=[0.1] * 768),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::warning[Collections Created Outside Haystack]
|
||||
|
||||
When you create a `QdrantDocumentStore` instance, Haystack takes care of setting up the collection. In general, you cannot use a Qdrant collection created without Haystack with Haystack. If you want to migrate your existing collection, see the sample script at https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/src/haystack_integrations/document_stores/qdrant/migrate_to_sparse.py.
|
||||
:::
|
||||
|
||||
You can also connect directly to [Qdrant Cloud](https://cloud.qdrant.io/login) directly. Once you have your API key and your cluster URL from the Qdrant dashboard, you can connect like this:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.document import Document
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.utils import Secret
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
url="https://XXXXXXXXX.us-east4-0.gcp.cloud.qdrant.io:6333",
|
||||
index="your_index_name",
|
||||
embedding_dim=1024, # based on the embedding model
|
||||
recreate_index=True, # enable only to recreate the index and not connect to the existing one
|
||||
api_key=Secret.from_token("YOUR_TOKEN"),
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.0] * 5),
|
||||
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::tip[More information]
|
||||
|
||||
You can find more ways to initialize and use QdrantDocumentStore on our [integration page](https://haystack.deepset.ai/integrations/qdrant-document-store).
|
||||
:::
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`QdrantEmbeddingRetriever`](../pipeline-components/retrievers/qdrantembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their dense embeddings (vectors).
|
||||
- [`QdrantSparseEmbeddingRetriever`](../pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their sparse embeddings.
|
||||
- [`QdrantHybridRetriever`](../pipeline-components/retrievers/qdranthybridretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on both dense and sparse embeddings.
|
||||
|
||||
:::note[Sparse Embedding Support]
|
||||
|
||||
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
|
||||
|
||||
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
|
||||
:::
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "ValkeyDocumentStore"
|
||||
id: valkeydocumentstore
|
||||
slug: "/valkeydocumentstore"
|
||||
description: "Use a Valkey database with Haystack."
|
||||
---
|
||||
|
||||
# ValkeyDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Valkey](/reference/integrations-valkey) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/valkey |
|
||||
|
||||
</div>
|
||||
|
||||
[Valkey](https://valkey.io/) is a high-performance, in-memory data structure store that you can use in Haystack pipelines with the `ValkeyDocumentStore`. Valkey operates in-memory by default for maximum performance, but can be configured with persistence options for data durability.
|
||||
|
||||
The `ValkeyDocumentStore` connects to a Valkey server with the search module running and supports vector similarity search for RAG and other retrieval use cases. For a detailed overview of all the available methods and settings, visit the [API Reference](/reference/integrations-valkey#valkeydocumentstore).
|
||||
|
||||
## Installation
|
||||
|
||||
You can install the Valkey Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install valkey-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
To use Valkey as your data storage for Haystack pipelines, you need a Valkey server with the search module running. Initialize a `ValkeyDocumentStore` like this:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
|
||||
|
||||
document_store = ValkeyDocumentStore(
|
||||
nodes_list=[("localhost", 6379)],
|
||||
index_name="my_documents",
|
||||
embedding_dim=768,
|
||||
distance_metric="cosine",
|
||||
)
|
||||
```
|
||||
|
||||
### Running Valkey locally
|
||||
|
||||
For development and testing, you can start a Valkey server with Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 6379:6379 valkey/valkey-bundle:latest
|
||||
```
|
||||
|
||||
Then connect with the same initialization code above, using `nodes_list=[("localhost", 6379)]`.
|
||||
|
||||
For more advanced configurations and clustering setups, refer to the [Valkey documentation](https://valkey.io/docs/).
|
||||
|
||||
## Writing documents
|
||||
|
||||
To write documents to your `ValkeyDocumentStore`, create an indexing pipeline or use the `write_documents()` method. You can use [Converters](../pipeline-components/converters.mdx), [PreProcessors](../pipeline-components/preprocessors.mdx), and other integrations to fetch and prepare data. Below is an example that indexes Markdown files into Valkey.
|
||||
|
||||
### Indexing pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import MarkdownToDocument
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
|
||||
|
||||
document_store = ValkeyDocumentStore(
|
||||
nodes_list=[("localhost", 6379)],
|
||||
index_name="my_documents",
|
||||
embedding_dim=768,
|
||||
distance_metric="cosine",
|
||||
)
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("converter", MarkdownToDocument())
|
||||
indexing.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=2),
|
||||
)
|
||||
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
|
||||
indexing.add_component("writer", DocumentWriter(document_store))
|
||||
indexing.connect("converter", "splitter")
|
||||
indexing.connect("splitter", "embedder")
|
||||
indexing.connect("embedder", "writer")
|
||||
|
||||
indexing.run({"converter": {"sources": ["filename.md"]}})
|
||||
```
|
||||
|
||||
## Using Valkey in a RAG pipeline
|
||||
|
||||
Once documents are in your `ValkeyDocumentStore`, you can use [`ValkeyEmbeddingRetriever`](../pipeline-components/retrievers/valkeyembeddingretriever.mdx) to retrieve them. The following example builds a RAG pipeline with a custom prompt:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.embedders import SentenceTransformersTextEmbedder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
|
||||
from haystack_integrations.components.retrievers.valkey import ValkeyEmbeddingRetriever
|
||||
|
||||
document_store = ValkeyDocumentStore(
|
||||
nodes_list=[("localhost", 6379)],
|
||||
index_name="my_documents",
|
||||
embedding_dim=768,
|
||||
distance_metric="cosine",
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system(
|
||||
"Answer the question based on the provided context. If the context does not include an answer, reply with 'I don't know'.",
|
||||
),
|
||||
ChatMessage.from_user(
|
||||
"Query: {{query}}\n"
|
||||
"Documents:\n{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
|
||||
"Answer:",
|
||||
),
|
||||
]
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
ValkeyEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables=["query", "documents"],
|
||||
),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"generator",
|
||||
OpenAIChatGenerator(
|
||||
api_key=Secret.from_token("YOUR_OPENAI_API_KEY"),
|
||||
model="gpt-4o",
|
||||
),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
|
||||
query_pipeline.connect("prompt_builder.prompt", "generator.messages")
|
||||
|
||||
query = "What is Valkey?"
|
||||
results = query_pipeline.run(
|
||||
{
|
||||
"text_embedder": {"text": query},
|
||||
"prompt_builder": {"query": query},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
For more examples, see the [examples folder](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/valkey/examples) in the repository.
|
||||
|
||||
## Performance benefits
|
||||
|
||||
- **In-memory storage**: Fast read and write operations.
|
||||
- **High throughput**: Handles many operations per second.
|
||||
- **Low latency**: Minimal response times for document operations.
|
||||
- **Scalability**: Supports clustering for horizontal scaling.
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
[`ValkeyEmbeddingRetriever`](../pipeline-components/retrievers/valkeyembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query from the `ValkeyDocumentStore`.
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: "VespaDocumentStore"
|
||||
id: vespadocumentstore
|
||||
slug: "/vespadocumentstore"
|
||||
---
|
||||
|
||||
# VespaDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Vespa](/reference/integrations-vespa) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/vespa |
|
||||
|
||||
</div>
|
||||
|
||||
[Vespa](https://vespa.ai/) is an open-source big data serving engine that supports structured, text, and vector search at scale. The `VespaDocumentStore` connects Haystack to an existing Vespa application through [pyvespa](https://vespa-engine.github.io/pyvespa/) and supports both lexical and dense vector retrieval as well as metadata filtering.
|
||||
|
||||
Unlike most other Haystack Document Stores, the `VespaDocumentStore` does **not** create or deploy the Vespa application or schema for you. You configure Vespa with the fields and rank profiles you need, deploy it (either self-hosted or on [Vespa Cloud](https://cloud.vespa.ai/)), and then point the Document Store at the running endpoint.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `vespa-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install vespa-haystack
|
||||
```
|
||||
|
||||
To run Vespa locally, see the [Vespa quick start](https://docs.vespa.ai/en/vespa-quick-start.html). To deploy a managed Vespa application, see [Vespa Cloud](https://cloud.vespa.ai/en/getting-started).
|
||||
|
||||
## Usage
|
||||
|
||||
### Prerequisites: Vespa Schema
|
||||
|
||||
Before using the `VespaDocumentStore`, you need a deployed Vespa application with a schema compatible with the fields you configure on the Document Store. By default, the integration expects:
|
||||
|
||||
- A text field named `content` for the Document body.
|
||||
- A tensor field named `embedding` for dense vectors (when using embedding retrieval).
|
||||
- A rank profile named `bm25` for lexical retrieval (used by `VespaKeywordRetriever`).
|
||||
- A rank profile named `semantic` that ranks with `closeness(field, embedding)` (used by `VespaEmbeddingRetriever`).
|
||||
|
||||
Field and rank profile names can be customized via the Document Store and Retriever constructors. See the [Vespa documentation](https://docs.vespa.ai/en/schemas.html) for details on writing schemas and rank profiles.
|
||||
|
||||
### Authentication
|
||||
|
||||
The `VespaDocumentStore` supports the authentication methods provided by `pyvespa`:
|
||||
|
||||
- **No authentication** for local development against an unsecured Vespa endpoint.
|
||||
- **mTLS** with a data plane certificate and key (via the `cert` and `key` parameters as [Secrets](../concepts/secret-management.mdx)).
|
||||
- **Bearer token** for Vespa Cloud token endpoints (via `vespa_cloud_secret_token` or the `VESPA_CLOUD_SECRET_TOKEN` environment variable).
|
||||
|
||||
The Vespa endpoint URL can be passed via the `url` parameter or the `VESPA_URL` environment variable:
|
||||
|
||||
```shell
|
||||
export VESPA_URL="http://localhost"
|
||||
```
|
||||
|
||||
For Vespa Cloud token authentication:
|
||||
|
||||
```shell
|
||||
export VESPA_URL="https://my-app.my-tenant.aws-us-east-1c.z.vespa-app.cloud"
|
||||
export VESPA_CLOUD_SECRET_TOKEN="my-secret-token"
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
Point the `VespaDocumentStore` at your deployed Vespa application and write Documents to it. The HTTP client is created lazily on first use:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.vespa import VespaDocumentStore
|
||||
|
||||
document_store = VespaDocumentStore(
|
||||
url="http://localhost",
|
||||
schema="doc",
|
||||
namespace="doc",
|
||||
content_field="content",
|
||||
embedding_field="embedding",
|
||||
metadata_fields=["category"],
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="Haystack integrates with Vespa for search.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
Document(
|
||||
content="Vespa supports lexical and vector retrieval.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see our [API docs](/reference/integrations-vespa#vespadocumentstore).
|
||||
|
||||
To compute embeddings for your Documents, you can use a Document Embedder, such as the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx).
|
||||
|
||||
### Metadata Fields
|
||||
|
||||
Vespa is strictly schema-bound: every metadata field that you want to feed or read back from Vespa must exist as a field in the deployed schema. Use the `metadata_fields` parameter to declare an allowlist of metadata keys to send to Vespa on write and to request back on read. Metadata keys that are not in this allowlist are kept on Documents in memory but are not stored in Vespa.
|
||||
|
||||
### Metadata Filtering
|
||||
|
||||
The `VespaDocumentStore` supports comparison operators (`==`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `not in`) and the logical operators `AND`, `OR`, and `NOT`. Filters are translated to Vespa's [YQL](https://docs.vespa.ai/en/query-language.html) where clauses whenever possible.
|
||||
|
||||
Filters on date-typed values are evaluated client-side in Python when YQL cannot express the comparison directly. For more details on filter syntax, refer to [Metadata Filtering](../concepts/metadata-filtering.mdx).
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`VespaEmbeddingRetriever`](../pipeline-components/retrievers/vespaembeddingretriever.mdx): A dense embedding-based Retriever that fetches Documents from Vespa using nearest-neighbor search and a configurable rank profile.
|
||||
- [`VespaKeywordRetriever`](../pipeline-components/retrievers/vespakeywordretriever.mdx): A lexical Retriever that fetches Documents from Vespa using a configurable rank profile (BM25 by default).
|
||||
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: "WeaviateDocumentStore"
|
||||
id: weaviatedocumentstore
|
||||
slug: "/weaviatedocumentstore"
|
||||
---
|
||||
|
||||
# WeaviateDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Weaviate](/reference/integrations-weaviate) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
|
||||
|
||||
</div>
|
||||
|
||||
Weaviate is a multi-purpose vector DB that can store both embeddings and data objects, making it a good choice for multi-modality.
|
||||
|
||||
The `WeaviateDocumentStore` can connect to any Weaviate instance, whether it's running on Weaviate Cloud Services, Kubernetes, or a local Docker container.
|
||||
|
||||
## Installation
|
||||
|
||||
You can simply install the Weaviate Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install weaviate-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
### Weaviate Embedded
|
||||
|
||||
To use `WeaviateDocumentStore` as a temporary instance, initialize it as ["Embedded"](https://weaviate.io/developers/weaviate/installation/embedded):
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate import WeaviateDocumentStore
|
||||
from weaviate.embedded import EmbeddedOptions
|
||||
|
||||
document_store = WeaviateDocumentStore(embedded_options=EmbeddedOptions())
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
You can use `WeaviateDocumentStore` in a local Docker container. This is what a minimal `docker-compose.yml` could look like:
|
||||
|
||||
```yaml
|
||||
---
|
||||
version: '3.4'
|
||||
services:
|
||||
weaviate:
|
||||
command:
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- '8080'
|
||||
- --scheme
|
||||
- http
|
||||
image: semitechnologies/weaviate:1.30.17
|
||||
ports:
|
||||
- 8080:8080
|
||||
- 50051:50051
|
||||
volumes:
|
||||
- weaviate_data:/var/lib/weaviate
|
||||
restart: 'no'
|
||||
environment:
|
||||
QUERY_DEFAULTS_LIMIT: 25
|
||||
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
|
||||
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
|
||||
DEFAULT_VECTORIZER_MODULE: 'none'
|
||||
ENABLE_MODULES: ''
|
||||
CLUSTER_HOSTNAME: 'node1'
|
||||
volumes:
|
||||
weaviate_data:
|
||||
...
|
||||
```
|
||||
|
||||
:::warning
|
||||
With this example, we explicitly enable access without authentication, so you don't need to set any username, password, or API key to connect to our local instance. That is strongly discouraged for production use. See the [authorization](#authorization) section for detailed information.
|
||||
|
||||
:::
|
||||
|
||||
Start your container with `docker compose up -d` and then initialize the Document Store with:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Weaviate Cloud Service
|
||||
|
||||
To use the [Weaviate managed cloud service](https://weaviate.io/developers/wcs), first, create your Weaviate cluster.
|
||||
|
||||
Then, initialize the `WeaviateDocumentStore` using the API Key and URL found in your [Weaviate account](https://console.weaviate.cloud/):
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate import (
|
||||
WeaviateDocumentStore,
|
||||
AuthApiKey,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
import os
|
||||
|
||||
os.environ["WEAVIATE_API_KEY"] = "YOUR-API-KEY"
|
||||
|
||||
auth_client_secret = AuthApiKey()
|
||||
|
||||
document_store = WeaviateDocumentStore(
|
||||
url="YOUR-WEAVIATE-URL",
|
||||
auth_client_secret=auth_client_secret,
|
||||
)
|
||||
```
|
||||
|
||||
## Authorization
|
||||
|
||||
We provide some utility classes in the `auth` package to handle authorization using different credentials. Every class stores distinct [secrets](../concepts/secret-management.mdx) and retrieves them from the environment variables when required.
|
||||
|
||||
The default environment variables for the classes are:
|
||||
|
||||
- **`AuthApiKey`**
|
||||
- `WEAVIATE_API_KEY`
|
||||
- **`AuthBearerToken`**
|
||||
- `WEAVIATE_ACCESS_TOKEN`
|
||||
- `WEAVIATE_REFRESH_TOKEN`
|
||||
- **`AuthClientCredentials`**
|
||||
- `WEAVIATE_CLIENT_SECRET`
|
||||
- `WEAVIATE_SCOPE`
|
||||
- **`AuthClientPassword`**
|
||||
- `WEAVIATE_USERNAME`
|
||||
- `WEAVIATE_PASSWORD`
|
||||
- `WEAVIATE_SCOPE`
|
||||
|
||||
You can easily change environment variables if needed. In the following snippet, we instruct `AuthApiKey` to look for `MY_ENV_VAR`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.auth import AuthApiKey
|
||||
from haystack.utils.auth import Secret
|
||||
|
||||
AuthApiKey(api_key=Secret.from_env_var("MY_ENV_VAR"))
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
[`WeaviateBM25Retriever`](../pipeline-components/retrievers/weaviatebm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`WeaviateEmbeddingRetriever`](../pipeline-components/retrievers/weaviateembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
Reference in New Issue
Block a user